Big Master Data PHP BLT #1

1. Big ”Master”Data
Masahiro Nagano
@kazeburo
PHP BLT#1

2. LT but Me
• Masahiro Nagano
• @kazeburo
• Site Reliability Engineer at Mercari, inc.
• ISUCON芸人
• a PHP newbie

3. What is Master Data?

4. “Master”Data
items
id
name
description
shipping_payer
shipping_method
condition
category_id
brand_id
...
payer
id
name
created
updated
...
method
id
name
created
updated
...
id
name
created
updated
...
condition
categories
id
name
disp_order
parent_id
created
updated
...
brands
id
name
disp_order
pronunciation
created
updated
...

5. Some time Master
Data cause N+1
• SELECT with JOIN
• Cache in File/Memory/Memcached
• Embed in PHP source code

6. Master Data we thought
array(
1 => array(
'id' => '1',
'created' => '2014-07-16 03:30:27',
'updated' => '2014-07-16 03:30:27',
'name' => 'Buyer',
'display_order' => '2',
),
2 => array(
'id' => '2',
'created' => '2014-07-16 03:30:27',
'updated' => '2014-07-16 03:30:27',
'name' => 'Seller',
'display_order' => '1',
),
);

7. Big Master Data
$ ls -lh
2.0M item_brands_master.php
509K item_categories_master.php

8. BMD is Heavy
$ cat bench.php
<?php
$time_start = microtime(true);
require_once '/item_brands_master.php';
$time_end = microtime(true);
$time = ($time_end - $time_start);
echo "#1 in $time secn";
$ php bench.php
#1 in 0.071656942367554 sec

9. “Use OPcache!!”
you say

10. How OPcache works
Process
shared mem
Copy!
Destroy!!
empty
full
php
php
php
php
php
php
DB
API
php
php
php
Request
Response

11. 5-10 msec to load
a 2.0MB php
from shared memory
According to strace/benchmark
sorry missed logs

12. Find way to reduce
file size

13. TSV!
split the line and create
an array on demand

14. Example
public static $keys = array (
0 => 'id',
1 => 'created',
2 => 'updated',
3 => 'name_ja',
4 => 'name_en',
5 => 'alternate_string',
6 => 'initials',
7 => 'display_order',
8 => 'is_deleted',
);
public static $rows = array (
1 => '1 2014-07-16 03:30:27 2014-07-16 03:30:27 104+ Octane
Boost 104+ Octane Boost 1 50 no',
2 => '2 2014-07-16 03:30:27 2014-07-16 03:30:27 20/10
Products 20/10 Products 2 50 no',
3 => '3 2014-07-16 03:30:27 2014-07-16 03:30:27 303 Products
303 Products 3 50 no',
4 => '4 2014-07-16 03:30:27 2014-07-16 03:30:27 3.1 Phillip
Lim 3.1 Phillip Lim 3 50 no',

15. File Size
$ ls -lh
2.0M item_brands_master.php
660K item_brands_master.php
require_once 70msec => 7msec (without OPCache)

16. Benchmarkfunction get($master, $id)
{
return $master::$rows[$id];
}
function get2($master, $id)
{
return array_combine($master::$keys,
explode("t",$master::$rows[$id]));
}
$time_start = microtime(true);
for ( $i=1; $i<3000; $i++ ) {
$val = get("ItemBrandsMaster",$i);
}
$time = 3000/(microtime(true) - $time_start);
echo "#1-2 in $time get/secn";
$time_start = microtime(true);
for ( $i=1; $i<3000; $i++ ) {
$val = get2("ItemBrandsMaster2",$i);
}
$time_end = microtime(true);
$time = 3000/(microtime(true) - $time_start);
echo "#2-2 in $time get/secn";

17. Result
$ php bench.php
#1-2 in 1374881.1188811 get/sec
#2-2 in 355409.33227884 get/sec
Seems Big speed down, but keeps fast enough.
Does not impact real application

18. Has been in production

19. END