Mechanics of Flite

Replacing JSON UDF calls with native function calls in MySQL 5.7

2016-10-03T14:22:00+00:00

If you use JSON UDFs in MySQL 5.6 there are a few things to consider before upgrading to MySQL 5.7. From a schema perspective, if you have JSON stored in text columns you can decide when to convert those columns to the new JSON datatype after you upgrade, but due to namespace collisions and format differences it’s probably not safe to upgrade without dropping some or all of the existing UDFs and updating your queries that use those UDFs.

Read on for details…

Namespace collisions

There are 15 functions in JSON UDFs version 0.4.0. Nine of those names are used by native JSON functions in MySQL 5.7, so it’s not possible to continue using those nine UDFs in MySQL 5.7. Therefore before you upgrade you should probably drop these UDFs:

DROP FUNCTION IF EXISTS json_append;
DROP FUNCTION IF EXISTS json_depth;
DROP FUNCTION IF EXISTS json_extract;
DROP FUNCTION IF EXISTS json_merge;
DROP FUNCTION IF EXISTS json_remove;
DROP FUNCTION IF EXISTS json_replace;
DROP FUNCTION IF EXISTS json_search;
DROP FUNCTION IF EXISTS json_set;
DROP FUNCTION IF EXISTS json_valid;

Path syntax

Depending on which version of MySQL JSON UDFs you are using, you may need to rewrite your queries to use the new path syntax. If you are using version 0.2 or 0.3 then you’re using this path syntax:

json_extract(doc, keypart1, keypart2, ...)

For example:

mysql> select json_extract('{"foo":"bar"}','foo');
+-------------------------------------+
| json_extract('{"foo":"bar"}','foo') |
+-------------------------------------+
| bar                                 |
+-------------------------------------+
1 row in set (0.00 sec)

mysql> select json_extract('{"parent":{"child":"hello"}}','parent','child');
+---------------------------------------------------------------+
| json_extract('{"parent":{"child":"hello"}}','parent','child') |
+---------------------------------------------------------------+
| hello                                                         |
+---------------------------------------------------------------+
1 row in set (0.00 sec)

The native JSON functions use this path syntax:

JSON_EXTRACT(json_doc, path[, path] ...)

For example:

mysql> select json_extract('{"foo":"bar"}','$.foo');
+---------------------------------------+
| json_extract('{"foo":"bar"}','$.foo') |
+---------------------------------------+
| "bar"                                 |
+---------------------------------------+
1 row in set (0.00 sec)

mysql> select json_extract('{"parent":{"child":"hello"}}','$.parent.child');
+---------------------------------------------------------------+
| json_extract('{"parent":{"child":"hello"}}','$.parent.child') |
+---------------------------------------------------------------+
| "hello"                                                       |
+---------------------------------------------------------------+
1 row in set (0.00 sec)

MySQL JSON UDFs version 0.4 had two copies of each UDF – one for each path syntax – so you only need to rewrite this part of your queries if you’re using the old path syntax. The functions using the old path syntax are in libmy_json_udf and the functions using the new path syntax are in libmy_json_udf_path.

Input differences

The MySQL 5.7 native JSON functions have some input differences as compared to the JSON UDF functions with the same name. One fundamental difference is the support for input strings which are not valid JSON. In general, the UDFs return NULL when the input is not valid JSON, whereas the native functions return an error. If your input is a column of JSON datatype then this is not a concern, since the datatype enforces the validity of the JSON and will not store invalid JSON. If the input is text you need to be more careful.

For example, none of these queries throw an error in MySQL 5.6 using JSON UDFs:

mysql> select json_extract('','foo');
+------------------------+
| json_extract('','foo') |
+------------------------+
| NULL                   |
+------------------------+
1 row in set (0.00 sec)

mysql> select json_extract('NOT JSON','foo');
+--------------------------------+
| json_extract('NOT JSON','foo') |
+--------------------------------+
| NULL                           |
+--------------------------------+
1 row in set (0.00 sec)

You can simply use COALESCE() if you want to replace the NULL output with the empty string:

mysql> select coalesce(json_extract('NOT JSON','foo'),'');
+---------------------------------------------+
| coalesce(json_extract('NOT JSON','foo'),'') |
+---------------------------------------------+
|                                             |
+---------------------------------------------+
1 row in set (0.00 sec)

But in MySQL 5.7 with native functions these queries throw an error:

mysql> select json_extract('','$.foo');
ERROR 3141 (22032): Invalid JSON text in argument 1 to function json_extract: "The document is empty." at position 0.
mysql> select json_extract('NOT JSON','$.foo');
ERROR 3141 (22032): Invalid JSON text in argument 1 to function json_extract: "Invalid value." at position 0.

One way around that is using a CASE statement along with the JSON_VALID() functions:

mysql> set @json = '';
Query OK, 0 rows affected (0.00 sec)

mysql> select (case when json_valid(@json) then json_extract(@json,'$.foo') else '' end);
+----------------------------------------------------------------------------+
| (case when json_valid(@json) then json_extract(@json,'$.foo') else '' end) |
+----------------------------------------------------------------------------+
|                                                                            |
+----------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> set @json = 'NOT JSON';
Query OK, 0 rows affected (0.01 sec)

mysql> select (case when json_valid(@json) then json_extract(@json,'$.foo') else '' end);
+----------------------------------------------------------------------------+
| (case when json_valid(@json) then json_extract(@json,'$.foo') else '' end) |
+----------------------------------------------------------------------------+
|                                                                            |
+----------------------------------------------------------------------------+
1 row in set (0.00 sec)

Since the above queries return empty string if they encounter invalid JSON there is no need to use COALESCE().

Output differences

The MySQL 5.7 native JSON functions return JSON objects, whereas the JSON UDFs return text. For example a call to the JSON_EXTRACT() UDF in MySQL 5.6 may return foo, but the same function in MySQL 5.7 would return "foo". Therefore in order to replace a call to the UDF version of a function like JSON_EXTRACT() with the corresponding native function call, you should wrap that call with the JSON_UNQUOTE() function.

Examples:

###MySQL 5.6

mysql> select json_extract('{"foo":"bar"}','foo');
+-------------------------------------+
| json_extract('{"foo":"bar"}','foo') |
+-------------------------------------+
| bar                                 |
+-------------------------------------+
1 row in set (0.00 sec)

###MySQL 5.7

mysql> select json_extract('{"foo":"bar"}','$.foo');
+---------------------------------------+
| json_extract('{"foo":"bar"}','$.foo') |
+---------------------------------------+
| "bar"                                 |
+---------------------------------------+
1 row in set (0.12 sec)

mysql> select json_unquote(json_extract('{"foo":"bar"}','$.foo'));
+-----------------------------------------------------+
| json_unquote(json_extract('{"foo":"bar"}','$.foo')) |
+-----------------------------------------------------+
| bar                                                 |
+-----------------------------------------------------+
1 row in set (0.00 sec)

While you are re-writing those function calls, consider using the -> (added in 5.7.9) and ->> (added in 5.7.13) operators, which are shortcuts for JSON_EXTRACT() and JSON_UNQUOTE(JSON_EXTRACT()) respectively. For example, these three statements are equivalent:

select JSON_UNQUOTE( JSON_EXTRACT(my_column, '$.some_key') )
from my_table;

select JSON_UNQUOTE(my_column -> '$.some_key')
from my_table;

select my_column->>'$.some_key'
from my_table;

Testing those out to confirm the output:

mysql> create temporary table if not exists json_test as select '{"foo":"bar"}' as json_val;
Query OK, 1 row affected (0.00 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> select json_unquote(json_extract(json_val,'$.foo')) from json_test;
+----------------------------------------------+
| json_unquote(json_extract(json_val,'$.foo')) |
+----------------------------------------------+
| bar                                          |
+----------------------------------------------+
1 row in set (0.03 sec)

mysql> select json_unquote(json_val->'$.foo') from json_test;
+---------------------------------+
| json_unquote(json_val->'$.foo') |
+---------------------------------+
| bar                             |
+---------------------------------+
1 row in set (0.00 sec)

mysql> select json_val->>'$.foo' from json_test;
+--------------------+
| json_val->>'$.foo' |
+--------------------+
| bar                |
+--------------------+
1 row in set (0.00 sec)

Speaking at Percona Live Amsterdam 2016

2016-09-21T15:33:00+00:00

I look forward to speaking at Percona Live Amsterdam 2016 just under two weeks from now.

Here is my talk:

Practical JSON in MySQL 5.7

To help organize my past (and future) blog posts about using JSON in MySQL, I am now tagging those posts with “JSON”, so you can find them here.

See you in Amsterdam!

Avoiding MySQL ERROR 1784 when replicating from 5.7 to 5.6

2016-07-22T16:50:00+00:00

Recently I upgraded some MySQL databases from 5.6 to 5.7, but – for boring reasons unique to my environment – I had to leave one replica on version 5.6. I knew there was a chance that the 5.7 -> 5.6 replication wouldn’t work, but I decided to try it out to see if (and why) it would fail. Once I upgraded the master, replication failed, so I checked the error log on the replica and found these messages:

[ERROR] Slave I/O: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF. Error_code: 1784
[ERROR] Slave I/O: Relay log write failure: could not queue event from master, Error_code: 1595

The error surprised me a little bit since I’m not using GTIDs in that replication topology. I asked around a bit, and Kenny Gryp hypothesized that I might be experiencing MySQL bug #74683, which was fixed in version 5.6.23. Since my replica was on 5.6.22, I decided to do an incremental 5.6 upgrade to see if that resolved the issue. I upgraded to 5.6.31 and replication started working.

YMMV and there are certainly bound to be 5.7 -> 5.6.31 replication scenarios that don’t work, but this was a simple fix for me. In hindsight it makes sense that replicating from a new major version to an older major version is more likely to work with a more recent minor version, so if I have to do this in the future I’ll make sure the replica is running the latest minor version before upgrading the master.

P.S. Thanks Kenny!

Column Level Privileges in MySQL

2016-06-30T21:33:00+00:00

Recently I experimented with column-level privileges in MySQL. Column-level privileges are fairly straightforward, but given how infrequently they are used I think there are a few areas worth discussing.

Here are a few high-level observations:

Users can execute INSERT and UPDATE statements that affect columns they don’t have privileges on, as long as they rely on implicit defaults
Since SQL is row-based, it doesn’t make sense to support column-level DELETE privileges, thus only SELECT, INSERT, and UPDATE are supported
You can grant privileges on multiple columns in one GRANT statement or multiple GRANT statements, the results are cumulative

Read on for more details on each type of column-level privilege, along with example queries.

SELECT

Users may only reference columns that they have explicity privileges on. This applies to the entire SELECT statement, not just the SELECT clause. If you try to reference a column that you do not have privileges on in the WHERE, GROUP BY, HAVING, or ORDER BY clause then you will get an error.

To illustrate this I created a table with two rows of sample data for testing:

mysql> create table good_stuff (
    ->   id int unsigned not null auto_increment primary key,
    ->   ctime timestamp default current_timestamp,
    ->   mtime timestamp default current_timestamp on update current_timestamp,
    ->   is_deleted tinyint not null default 0,
    ->   public varchar(255) null,
    ->   protected varchar(255) null,
    ->   private varchar(255) null
    -> ) engine = innodb;
Query OK, 0 rows affected (0.03 sec)

mysql> insert into good_stuff (id,public,protected,private) 
    -> values (DEFAULT,'Hello world!','Red','Secret');
Query OK, 1 row affected (0.00 sec)

mysql> insert into good_stuff (id,public,protected,private)
    -> values (DEFAULT,'Hi Scott','Blue','Boo Scott');
Query OK, 1 row affected (0.01 sec)

If I grant SELECT privileges on two columns in that table to a user named “scott”, then scott may select those two columns:

mysql> -- as root, grant SELECT privileges to scott
mysql> grant select (public,protected) on good_stuff to scott@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> -- as scott, test SELECT privileges
mysql> select public,protected from good_stuff;
+--------------+-----------+
| public       | protected |
+--------------+-----------+
| Hello world! | Red       |
| Hi Scott     | Blue      |
+--------------+-----------+
2 rows in set (0.00 sec)

But scott may not reference another column in the ORDER BY clause:

mysql> -- as scott, test SELECT privileges 
mysql> select public from good_stuff order by id;
ERROR 1143 (42000): SELECT command denied to user 'scott'@'localhost' for column 'id' in table 'good_stuff'

Table-level privileges take precendence, so granting table-level SELECT privileges to a user overrides any column-level SELECT privileges they may have.

If you happen to grant SELECT privileges on all columns in a table to a user, then the user is allowed to run SELECT * queries.

INSERT

Users may only explicitly insert data into columns for which they have the INSERT privilege. INSERT privileges do not rely on SELECT privileges, so it is possible to configure a user who may write data but not read it. Default values may be used, but only implicitly. If you try to explicitly reference a default value you will get an error.

If I give scott INSERT privileges on only the public column in my table, he can still insert a row as long as he only references that one column, and default values will be used for other columns (id, ctime, mtime, is_deleted).

mysql> -- as root, grant INSERT privileges to scott
mysql> grant insert (public) on good_stuff to scott@'%';
Query OK, 0 rows affected (0.00 sec)

For example this works:

mysql> -- as scott, test INSERT privileges 
mysql> insert into good_stuff (public) values ('Hi everybody');
Query OK, 1 row affected (0.01 sec)

And results in a row like this:

mysql> -- as root, select full row that scott inserted
mysql> select * from good_stuff where public = 'Hi everybody';
+----+---------------------+---------------------+------------+--------------+-----------+---------+
| id | ctime               | mtime               | is_deleted | public       | protected | private |
+----+---------------------+---------------------+------------+--------------+-----------+---------+
|  3 | 2016-06-30 20:37:39 | 2016-06-30 20:37:39 |          0 | Hi everybody | NULL      | NULL    |
+----+---------------------+---------------------+------------+--------------+-----------+---------+

These statements all fail even though they have the same intent, because they are explicitly referencing columns that scott does not have privileges to INSERT:

mysql> -- as scott, test INSERT privileges
mysql> insert into good_stuff (id,public) values (null,'Is it okay if I do this?');
ERROR 1143 (42000): INSERT command denied to user 'scott'@'localhost' for column 'id' in table 'good_stuff'
mysql> insert into good_stuff (id,public) values (DEFAULT,'Is it okay if I do this?');
ERROR 1143 (42000): INSERT command denied to user 'scott'@'localhost' for column 'id' in table 'good_stuff'

I would get the same results if I tried to use null or DEFAULT with the other columns that have defaults values (ctime, mtime, is_deleted).

UPDATE

UPDATE statements support column-level privileges much the same way as SELECT and INSERT. In order to explicitly update a column a user needs UPDATE privileges on that column, but columns can be set to default values implicitly. If you reference a column in the WHERE clause of an UPDATE, then you need SELECT privileges on that column.

First I grant UPDATE privileges on the public column to scott:

mysql> -- as root, grant UPDATE privileges to scott
mysql> grant update (public) on good_stuff to scott@'%';
Query OK, 0 rows affected (0.01 sec)

This is allowed:

mysql> -- as scott, test UPDATE privileges
mysql> update good_stuff set public = lower(public);
Query OK, 1 row affected (0.00 sec)
Rows matched: 4  Changed: 1  Warnings: 0

This is also allowed:

mysql> -- as scott, test UPDATE privileges
mysql> update good_stuff set public = upper(public) where protected = 'Blue';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Please note that the previous two statements will only work if scott has both SELECT and UPDATE privileges on the public column, since the existing value is being used in the UPDATE, so that requires both a read and a write.

These are not allowed because the WHERE clause references a column for which scott does not have SELECT privileges:

mysql> -- as scott, test UPDATE privileges
mysql> update good_stuff set public = upper(public) where id = 2;
ERROR 1143 (42000): SELECT command denied to user 'scott'@'localhost' for column 'id' in table 'good_stuff'
mysql> update good_stuff set is_deleted = DEFAULT;
ERROR 1143 (42000): UPDATE command denied to user 'scott'@'localhost' for column 'is_deleted' in table 'good_stuff'

Even though scott does not have privileges to update the mtime column, the on update current_timestamp is still invoked implicitly. To verify this I can select back the “Hi everybody” row I selected earlier to confirm that the mtime value changed:

mysql> -- as root, select full row that scott inserted
mysql> select * from good_stuff where id = 3;
+----+---------------------+---------------------+------------+--------------+-----------+---------+
| id | ctime               | mtime               | is_deleted | public       | protected | private |
+----+---------------------+---------------------+------------+--------------+-----------+---------+
|  3 | 2016-06-30 20:37:39 | 2016-06-30 21:15:03 |          0 | hi everybody | NULL      | NULL    |
+----+---------------------+---------------------+------------+--------------+-----------+---------+

DELETE

As mentioned above, column-level DELETE privileges are not supported in MySQL.

MySQL Keywords and Reserved Words

2016-05-24T16:05:00+00:00

I’ve seen some confusion on what constitutes a keyword or a reserved word in MySQL. The manual defines them, and has a complete list. Reserved words are a special subset of keywords, and you can’t use a reserved word as an identifier unless you quote it with backticks. I discourage the use of backticks to quote identifiers, because it allows you to use lots of words and characters in your identifier that you’ll probably regret later. For example:

mysql> create table `You will regret this!` (`(╯°□°）╯︵ ┻━┻` int);
Query OK, 0 rows affected (0.03 sec)

mysql> desc `You will regret this!`;
+--------------------------------+---------+------+-----+---------+-------+
| Field                          | Type    | Null | Key | Default | Extra |
+--------------------------------+---------+------+-----+---------+-------+
| (╯°□°）╯︵ ┻━┻                 | int(11) | YES  |     | NULL    |       |
+--------------------------------+---------+------+-----+---------+-------+
1 row in set (0.00 sec)

There is also an exception for table names. You can use a reserved word as a table name without quoting it with backticks as long as you prefix it with the schema name, but again I would discourage that. Here’s an example:

mysql> create table table (id int);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'table (id int)' at line 1
mysql> create table test.table (id int);
Query OK, 0 rows affected (0.04 sec)

Non-reserved keywords are a little bit trickier. Many of them probably should be reserved, but are not. For example, lots of data type names are not reserved, nor is the word “view”. Thus I can create a table like this without even having to use backticks:

create table view (
  bit bit,
  bool bool,
  boolean boolean,
  date date,
  datetime datetime,
  enum enum('enum'),
  text text,
  time time,
  timestamp timestamp,
  year year,
  comment int comment 'int'
);

Yuck. I think it’s a good practice to familiarize yourself with all of the non-reserved MySQL keywords and avoid using them as identifiers, but unfortunately there is nothing to prevent you from doing so.

Being Naive About Folder Paths

2016-04-29T13:41:00+00:00

Recently I worked on a feature that involved displaying a list of Folders using an auto-complete field. A simple enough task, but one that had a bit of complexity behind it. For starters, each organization in our system has their own “folder hierarchy”, or a tree structure of folders, that could be one to many layers deep. The second thing being that the UI we wanted – auto-complete – meant that we would need to “search” through the folders to find a match (based on the user’s input) as well as display the path information (where the folder lives) so that the user can differentiate folders if they happen to have the same or similar name. Since it was auto-complete, the searching also needed to be fast or else the user experience would be annoying.

The folder tree structure was stored in a MySQL DB table in a very typical self-referencing schema that represented hierarchical data. Each row of data could have a column that referenced the “parent” folder id and if there wasn’t one, then it was a “root” folder. Each row in the table could be represented by an object in Java called Folder. Getting folders by ID and getting them in a tree-like structure was pretty straight forward. However, the path-string wasn’t part of the deal.

My first thought on this was that the paths would need to be pre-computed ahead of time so that they wouldn’t need to be computed every time a user typed in a letter, an expensive and slow function. One way to do it would be to have some application code insert to a table that contained the folder paths whenever a folder was saved. Our resident MySQL expert, Ike Walker, suggested I look into the closure table pattern, discussed in Chapter 3, Naive Trees, of Bill Karwin’s book, “SQL Antipatterns”. Yes, this was exactly what I was thinking of! However, as I thought about it more there were some things I didn’t quite like about it. First, there was already a lot of existing data. Using Closure Tables would mean that I would need to script something to populate the table storing the folder paths, and run it at least once. The fact that I would need to script something like that also made me think about how will this table get updated? In the book “SQL Antipatterns”, a discussion thread example was used. In the case of a discussion thread, individual comments have the same kind of tree-structure that folders might have, but comments and replies typically don’t move around whereas folders do. Updating the folder paths would be required to keep in sync with the underlying tree, along with whole sections of a particular folder’s branch. The thought of having to write this updating code made me think twice about my initial approach.

##Caching to the rescue

So what does a developer do when there’s a problem with performance? Cache. So I thought: how can caching help me here? As I pointed out earlier, there was already a Java object – Folder – and a way to get the folders in a tree structure. If the Folder had a field to store the path, then I’m halfway there, right? But wait, doesn’t that mean I’d be storing the path in the DB again? Well, not if I set it on a transient field and set it only when the tree structure is getting built up. Hold on – doesn’t that mean that the paths still need to be calculated every time we search the tree? Here’s where the caching bit comes in. The folder tree would be traversed once and as it is recursing, the paths get computed and stored on a transient field of the Folder object. At the same time, the Folders are stored in a flat List that can be used for iterating and searching, etc. Next, the method which retrieves the folder list is cached and can be cached with the Organization’s ID in the cache-key so that each list can be kept separately. To make sure we don’t cache large lists of heavy Folder objects, I cached only lists of Folder IDs (not objects) and made the Folder objects also cached by their ID. Okay, then what about the updating part I talked about? The cached list of Folders would be invalidated whenever a Folder update occurred. It means that the paths would need to be re-computed again, but only once until the next update and that seemed reasonable enough.

In the end, I didn’t use the closure table, although I’m keen on using it if the opportunity arises. One of the key things to coming up with the solution was knowing not only the data-structure but also knowing the nature of the data (how it’s used and how often does it tend to change). In all fairness, Bill Karwin also points out frequent updating as a weakness/pain-point of the Closure Table pattern. So there you have it my friends, use those clousure tables if you can, and if you can’t, then hopefully the approach I used here will give you some ideas!

Time Zone Conversion in MySQL

2016-04-11T15:30:00+00:00

A wise man once said that the safest way to store date and time values in MySQL is to store Unix timestamps in an unsigned INT column. I agree with Baron, but most of us have to deal with DATE, DATETIME, and/or TIMESTAMP values sometimes, and it’s often useful to know how to convert values from one time zone to another.

##The CONVERT_TZ() Function

You can convert a temporal value in MySQL from one time zone to another using the CONVERT_TZ() function. As an example, say I want to watch the Boston Marathon in Melbourne, Australia but I’m afraid it may be on after my bed time. The elite men start at 10:25 AM on April 18, 2016. Boston’s time zone will be EDT that day, which is UTC-4. Melbourne’s time zone that day will be AEST, which is UTC+10. The following query will tell me what time the Boston Marathon starts in Melbourne:

mysql> select convert_tz('2016-04-18 10:25','-04:00','+10:00');
+--------------------------------------------------+
| convert_tz('2016-04-18 10:25','-04:00','+10:00') |
+--------------------------------------------------+
| 2016-04-19 00:25:00                              |
+--------------------------------------------------+
1 row in set (0.00 sec)

Therefore the marathon starts are 12:25 AM on Tuesday in Melbourne, which is past my bedtime.

It may seem a little silly to use CONVERT_TZ() with offsets like that rather than just adding 14 hours like this:

mysql> select '2016-04-18 10:25' + interval 14 hour;
+---------------------------------------+
| '2016-04-18 10:25' + interval 14 hour |
+---------------------------------------+
| 2016-04-19 00:25:00                   |
+---------------------------------------+
1 row in set (0.00 sec)

My argument for preferring CONVERT_TZ() is that it makes the query much more readable. In the first example it’s clear that I am converting a time from UTC-04:00 to UTC+10:00, whereas in the second example there’s no way to tell from the query why I’m adding 14 hours.

##Populating the Time Zone Tables

Using CONVERT_TZ() becomes more interesting using named time zones, but that only works if the system time zone tables are populated. For example if I try to find out what time the Boston Marathon starts in Melbourne using named time zones I may get a NULL value:

mysql> select convert_tz('2016-04-18 10:25','America/New_York','Australia/Melbourne');
+-------------------------------------------------------------------------+
| convert_tz('2016-04-18 10:25','America/New_York','Australia/Melbourne') |
+-------------------------------------------------------------------------+
| NULL                                                                    |
+-------------------------------------------------------------------------+
1 row in set (0.00 sec)

Why did I get NULL? Probably because my time zone tables are empty. Those tables are not populated by default.

Following the instructions in the reference manual I can populate those tables by running this command in the shell:

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql

Now I can use named time zones:

mysql> select convert_tz('2016-04-18 10:25','America/New_York','Australia/Melbourne');
+-------------------------------------------------------------------------+
| convert_tz('2016-04-18 10:25','America/New_York','Australia/Melbourne') |
+-------------------------------------------------------------------------+
| 2016-04-19 00:25:00                                                     |
+-------------------------------------------------------------------------+
1 row in set (0.00 sec)

Bear in mind that time zone rules change sometimes, so you may want to review the manual and develop a plan to stay up to date with those changes.

##Know What Time Zone MySQL is Using

My examples so far have used static datetime values. When dealing with values stored in a table or returned by MySQL functions it’s important to know the context of those values if I want to convert them. For example I need to know what time zone MySQL is using if I want to convert a datetime value stored in a table, or the output of a function like NOW() to a different time zone.

The relevant session and global variables are named time_zone and system_time_zone. You can access them like this:

mysql> select @@session.time_zone,@@global.time_zone,@@global.system_time_zone;
+---------------------+--------------------+---------------------------+
| @@session.time_zone | @@global.time_zone | @@global.system_time_zone |
+---------------------+--------------------+---------------------------+
| SYSTEM              | SYSTEM             | UTC                       |
+---------------------+--------------------+---------------------------+
1 row in set (0.00 sec)

If the time_zone variable is set to SYSTEM then the system_time_zone will be used.

Announcing MySQL BRO Schema

2016-04-01T17:00:00+00:00

When MySQL released performance_schema it made a lot of new useful information available, but it wasn’t particularly user-friendly. The introduction of the MySQL SYS schema made it a lot easier for developers and DBAs to get useful insights from performance_schema. I was talking to some brogrammers the other day, and while they admitted that SYS schema is “totally legit”, they wished that it “spoke their language”.

To that end, I am happy to announce MySQL BRO schema. It’s a natural complement to SYS schema, and makes it easier for brogrammers to work with performance_schema.

Read on for a few examples of the BRO schema in action.

The SYS schema has a view named user_summary. In the BRO schema the equivalent is the bro_summary view.

Here’s a sample query on the bro_summary view:

mysql> use bro
Database changed
mysql> select * from bro_summary;
+------------------+-------------+--------------------+------------------------+-------------+-------------+-------------------+----------------------+--------------------+---------------+
| bro              | bro_queries | bro_query_brotency | bro_query_avg_brotency | tabro_scans | file_ibros  | file_ibro_latency | current_bronnections | total_bronnections | unique_brosts |
+------------------+-------------+--------------------+------------------------+-------------+-------------+-------------------+----------------------+--------------------+---------------+
| brobi_wan_kenobi |      230732 | 6.24 m             | 1.62 ms                |           0 |     3375788 | 1.83 h            |                    2 |                  2 |             1 |
| broto_baggins    |      173767 | 1.18 m             | 405.89 us              |         135 |    11841856 | 13.95 h           |                   11 |                 17 |             3 |
| hulk_brogan      |        1549 | 1.29 s             | 830.87 us              |         509 |        2720 | 732.04 ms         |                    5 |                516 |             2 |
| tony_bromo       |     1064933 | 28.68 m            | 1.62 ms                |         136 |    32476670 | 22.55 m           |                    5 |                 56 |             7 |
| brute            |       76009 | 1.00 h             | 47.55 ms               |       37188 |   254239005 | 2.44 h            |                    1 |              37461 |             2 |
+------------------+-------------+--------------------+------------------------+-------------+-------------+-------------------+----------------------+--------------------+---------------+

Likewise the host_summary view from SYS schema is replaced with the brost_summary view in BRO schema:

mysql> select * from brost_summary;
+--------------------------------+-------------+--------------------+------------------------+-------------+------------+-------------------+----------------------+--------------------+-------------+
| brost                          | bro_queries | bro_query_brotency | bro_query_avg_brotency | tabro_scans | file_ibros | file_ibro_latency | current_bronnections | total_bronnections | unique_bros |
+--------------------------------+-------------+--------------------+------------------------+-------------+------------+-------------------+----------------------+--------------------+-------------+
| background                     |        NULL | NULL               | NULL                   |        NULL |       NULL | NULL              |                    0 |                  2 |           0 |
| ip-10-60-41-26.ec2.internal    |       55959 | 3.52 h             | 226.20 ms              |       13368 |   52416910 | 3.66 h            |                   23 |                 22 |          15 |
| ip-10-102-56-230.ec2.internal  |     1792854 | 2.17 h             | 4.36 ms                |        6703 |   39771784 | 4.14 h            |                    1 |                  7 |           5 |
| ip-10-30-14-233.ec2.internal   |      860733 | 1.20 h             | 5.03 ms                |       46829 |   10609581 | 53.39 m           |                   14 |                  4 |           3 |
| ip-10-21-17-24.ec2.internal    |      552149 | 5.84 m             | 634.79 us              |       77911 |   39970438 | 16.93 m           |                   13 |                  6 |          18 |
| ip-10-199-205-174.ec2.internal |        3481 | 4.09 h             | 4.23 s                 |      187574 |   10722269 | 1.67 h            |                   12 |                  5 |          11 |
| ip-10-139-129-225.ec2.internal |     1909218 | 3.57 h             | 6.73 ms                |      117428 |    1310063 | 2.46 h            |                   21 |                 11 |          15 |
| ip-10-154-79-189.ec2.internal  |     1643780 | 2.72 h             | 5.95 ms                |      203993 |   31900386 | 3.44 h            |                   15 |                 12 |          14 |
| localbrost                     |      787149 | 3.10 h             | 14.17 ms               |      118728 |  894439407 | 4.28 d            |                   11 |              39514 |           3 |
+--------------------------------+-------------+--------------------+------------------------+-------------+------------+-------------------+----------------------+--------------------+-------------+

This just scratches the surface of the insights you can get from the BRO schema. The truly brolific amount of information in BRO schema makes it absolutely bronafide.

Simplifying MySQL partition management using common_schema

2016-03-28T13:45:00+00:00

Maintaining partitioned tables in MySQL can be tedious. In a reporting database that uses monthly date partitions, I’ve found that I can simplify many frequent partition operations using the common_schema QueryScript language. Below I will describe two specific examples.

#Removing old data

I like to keep a 2-3 month window of data in some of my reporting tables. In order to maintain that window I periodically purge data more than 3 months old using the TRUNCATE PARTITION command. Since I want to apply the same purge logic to all tables matching a consistent naming pattern, I can use the common_schema foreach command to iterate over the tables based on an information_schema query and run the same TRUNCATE PARTITION command on all of the tables:

call common_schema.foreach(
  "select table_schema,table_name,partition_name 
    from information_schema.partitions 
    where table_schema = 'reporting' 
    and table_name like '%\_fact' 
    and table_rows != 0
    and partition_name <= concat('p',date_format(current_date() - interval 3 month,'%Y%m'))",
  "alter table ${1}.${2} truncate partition ${3};");

#Adding new partitions

Recently I discovered that some of the partitioned tables in this reporting database did not have any partitions for 2016, so all of this year’s data ended up in the “pmax” partition. In order to fix this I used the common_schema foreach command to iterate over another information_schema query and run a REORGANIZE PARTITION command on all of the tables:

call common_schema.foreach(
  "select p1.table_schema,p1.table_name,p1.partition_name
    from information_schema.PARTITIONS p1 
    where p1.PARTITION_NAME = 'pmax' 
    and p1.table_schema = 'reporting' 
    and p1.table_name like '%\_fact' 
    and exists (
      select NULL 
      from information_schema.partitions p2 
      where p2.table_Schema = p1.table_schema 
      and p2.table_name = p1.table_name 
      and p2.partition_name like 'p2015__'
    )
    and not exists (
      select NULL 
      from information_schema.partitions p2 
      where p2.table_Schema = p1.table_schema 
      and p2.table_name = p1.table_name 
      and p2.partition_name like 'p2016__'
    )",
    "ALTER TABLE ${1}.${2}
    REORGANIZE PARTITION ${3} INTO (
      PARTITION p201601 VALUES LESS THAN ('2016-02-01') ENGINE = InnoDB,
      PARTITION p201602 VALUES LESS THAN ('2016-03-01') ENGINE = InnoDB,
      PARTITION p201603 VALUES LESS THAN ('2016-04-01') ENGINE = InnoDB,
      PARTITION p201604 VALUES LESS THAN ('2016-05-01') ENGINE = InnoDB,
      PARTITION p201605 VALUES LESS THAN ('2016-06-01') ENGINE = InnoDB,
      PARTITION p201606 VALUES LESS THAN ('2016-07-01') ENGINE = InnoDB,
      PARTITION p201607 VALUES LESS THAN ('2016-08-01') ENGINE = InnoDB,
      PARTITION p201608 VALUES LESS THAN ('2016-09-01') ENGINE = InnoDB,
      PARTITION p201609 VALUES LESS THAN ('2016-10-01') ENGINE = InnoDB,
      PARTITION p201610 VALUES LESS THAN ('2016-11-01') ENGINE = InnoDB,
      PARTITION p201611 VALUES LESS THAN ('2016-12-01') ENGINE = InnoDB,
      PARTITION p201612 VALUES LESS THAN ('2017-01-01') ENGINE = InnoDB,
      PARTITION pmax VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB
  );");

Google Accelerated Mobile Pages and the &ltamp-ad&gt

2016-02-24T14:07:00+00:00

Google’s new Accelerated Mobile Pages project introduces a lot of new challenges for web page creation and design. It also presents some small challenges to Flite as an ad serving platform. The two ways in which the AMP project and its specfications interfere with our usual way of serving ads is through the restriction of written or external JavaScript and by serving ads through external iframes.

No JavaScript

Our platform, like many others, generally delivers its content to the page via a