27. Install
ColumnStore初期設定
# /usr/local/mariadb/columnstore/bin/postConfigure
Select the type of System Server install [1=single, 2=multi] (2) > 1
Enter System Name (columnstore-1) > columnstore-1
Select the type of Data Storage [1=internal, 2=external] (1) > 1
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm1' (1) > 1
===== Performing Configuration Setup and MariaDB ColumnStore Startup =====
...
Enter 'mcsmysql' to access the MariaDB ColumnStore SQL console
Enter 'mcsadmin' to access the MariaDB ColumnStore Admin console
31. Docker
MariaDB ColumnStore Docker, Vagrant, and Windows 10 Linux Setup -
(allows for evaluation on a PC or Mac)
https://mariadb.com/kb/en/library/columnstore-getting-started-mariadb-columnstore-
docker-vagrant-and-windows-/
• Mac & Windows でも動作検証可能
• 最小システム構成要件を満たしていなくても動作確認可能
• ただし…当然パフォーマンスは出ない
32. コンテナー取得
# docker run -d --name mcs1.1 -p 3307:3306 mariadb/columnstore:1.1
• dockerホストから接続できるように
3307から3306にポートフォワード指定
• 2018-09-16 時点でインストールされるのは v 1.1.6
docker for mac
MariaDB Columnstore 1.1 install
33. docker for mac
MariaDB Columnstore 1.1 install
dockerコンテナコンソール起動
# docker exec -it mcs1.1 bash
dockerホストから接続するユーザー作成
# mcsmysql –u root
MariaDB [(none)]> CREATE USER 'root'@'gateway' IDENTIFIED BY 'パスワード';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'gateway';
34. docker for mac
MariaDB Columnstore 1.1 install
dockerホストからColumnstoreに接続
# mysql -u root -p -P 3307 -h 127.0.0.1
• -h localhost だと ホストの mysql に接続しにいくので注意
• rpmパッケージインストール時同様、UTF-8設定を実施
• 最小システム構成要件を満たさない環境ではパフォーマンスに期待しない
38. 管理
mcsadmin Command
[myuser@srv1~]# mcsadmin
MariaDB Columnstore Admin Console enter 'help' for list of
commands enter 'exit' to exit the MariaDB Columnstore Command
Console use up/down arrows to recall commandsActive Alarm Counts:
Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0Critical Active
Alarms:
mcsadmin> quit
40. 管理
Viewing system status
[myuser@srv1~]# mcsadmin getSystemStatus
getsystemstatus Wed Aug 30 14:27:23 2017
System columnstore-1
System and Module statuses
Component Status Last Status Change
------------------ ------------------------- --------------------------------------
System ACTIVE Fri Jun 10 01:50:46 2016
Module pm1 ACTIVE Fri Jun 10 01:50:43 2016
41. 管理
Viewing process status
[myuser@srv1~]# mcsadmin getProcessStatus
getprocessstatus Wed Aug 30 14:27:23 2017
System columnstore-1
MariaDB Columnstore Process statuses
Process Module Status Last Status Change ProcessID
--------------------- ----------- ----------- ---------------------------------- -------------
ProcessMonitor pm1 ACTIVE Fri Jun 10 01:50:04 2016 2487
ProcessManager pm1 ACTIVE Fri Jun 10 01:50:10 2016 2673
…
42. 管理
Viewing network configuration
[myuser@srv1~]# mcsadmin getSystemNetworkConfig
getsystemstatus Wed Aug 30 14:27:23 2017
System columnstore-1
System and Module statuses
Component Status Last Status Change
------------------ ------------------------- --------------------------------------
System ACTIVE Fri Jun 10 01:50:46 2016
Module pm1 ACTIVE Fri Jun 10 01:50:43 2016
43. 管理
Viewing module configuration
[myuser@srv1~]# mcsadmin getModuleConfig
getmoduleconfig Wed Aug 30 14:27:23 2017
Module Name Configuration
Module 'um1' Configuration information
ModuleType = um
ModuleDesc = User Module #1
ModuleIPAdd NIC ID 1 = 10.100.7.80
ModuleHostName NIC ID 1 = srvhst2
…
50. インポート
cpimport 実行例
[myuser@srv1~]# cpimport sample table1 /tmp/table1.csv –s ',' –E '”'
Locale is : C
Column delimiter : ,
Enclosed by Character : ”
Using table OID 3278 as the default JOB ID
Input file(s) will be read from : /root/tmp
…
2017-08-31 16:16:37 (18403) INFO : For table sample.table1: 10000 rows
processed and 10000 rows inserted.
…
2017-08-31 16:16:37 (18403) INFO : Bulk load completed, total run time :
1.22974 seconds
DB TABLE File path option
53. インポート
INSERT SELECT, LOAD DATA INFILE
共通の注意点
• ロードするデータファイルサイズが大きい場合
バッファサイズ不足でロードに失敗する
• ERROR 1815 (HY000) at line 1 in file: 'table1.sql': Internal error: CAL0006: IDB-
2008: The version buffer overflowed. Increase VersionBufferFileSize or limit
the rows to be processed.
56. インポート
VersionBufferFileSize確認/変更
[myuser@srv1~]# cd /usr/local/mariadb/columnstore/bin/
[myuser@srv1~]# configxml.sh getconfig VersionBuffer VersionBufferFileSize
Current value of VersionBuffer / VersionBufferFileSize is 1GB
[myuser@srv1~]# configxml.sh setconfig VersionBuffer VersionBufferFileSize 10GB
Old value of VersionBuffer / VersionBufferFileSize is 1GB
VersionBuffer / VersionBufferFileSize now set to 10GB
section variablecommand
95. ディスクベースの結合
TempFilePath確認/変更
[myuser@srv1~]# cd /usr/local/mariadb/columnstore/bin/
[myuser@srv1~]# ./configxml.sh getconfig HashJoin TempFilePath
Current value of HashJoin / TempFilePath is /usr/local/mariadb/columnstore/tmp
[myuser@srv1~]# ./configxml.sh setconfig HashJoin TempFilePath {path}
Old value of HashJoin / TempFilePath is /user/local/…/tmp
HashJoin / TempFilePath now set to {path}
section variablecommand
98. debug.log
…
Sep 17 02:25:15 83665347d964 PrimProc[18651]: 15.019437 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3200;
/usr/local/mariadb/columnstore/data1/000.dir/000.dir/012.dir/128.dir/000.dir/FILE001.cdf:No such file or directory
Sep 17 02:25:15 83665347d964 PrimProc[18651]: 15.019940 |0|0|0| W 28 CAL0000: IDB-2039: Data file does not exist, please
contact your system administrator for more information.
Sep 17 02:38:43 83665347d964 ddlpackageproc[18851]: 43.789061 |12|58|0| D 23 CAL0041: Start SQL statement: alter table
flights2.flights drop column priority;|flights2|
Sep 17 02:38:44 83665347d964 ddlpackageproc[18851]: 44.354258 |12|58|0| D 23 CAL0042: End SQL statement
Sep 17 02:38:55 83665347d964 ddlpackageproc[18851]: 55.596063 |12|59|0| D 23 CAL0041: Start SQL statement: alter table
flights2.flights add column priority integer;|flights2|
Sep 17 02:38:55 83665347d964 ExeMgr[18718]: 55.607541 |2147483660|59|0| D 16 CAL0041: Start SQL statement: select
objectid,columnlength,datatype,dictobjectid,listobjectid,treeobjectid,columnposition,scale,prec,defaultvalue from syscolumn where
schema='flights2' and tablename='flights' and columnname='priority' --lookupOID/FE; ||
Sep 17 02:38:55 83665347d964 ExeMgr[18718]: 55.690064 |2147483660|59|0| D 16 CAL0042: End SQL statement
Sep 17 02:38:57 83665347d964 ddlpackageproc[18851]: 57.869577 |12|59|0| D 23 CAL0042: End SQL statement
Sep 17 02:40:58 83665347d964 ddlpackageproc[18851]: 58.459699 |12|60|0| D 23 CAL0041: Start SQL statement: alter table
flights2.flights drop column priority;|flights2|
Sep 17 02:40:58 83665347d964 ddlpackageproc[18851]: 58.946880 |12|60|0| D 23 CAL0042: End SQL statement
110. TPC-H Benchmark
InnoDBデータベース作成
MySQL Host 127.0.0.1
MySQL Port 3306
MySQL User root
MySQL User password rootのパスワード
MySQL Database tpch
Data Warehouse Storage engine innodb
Scale Factor 10
Virtual Users Build Schema 1
124. TPC-H Benchmark
クエリ書き換え
Query # Error
Query 2 ERROR 1815 (HY000): Internal error: IDB-3012: Scalar filter and semi join are not from the same
pair of tables.
Query 5 ERROR 1815 (HY000): Internal error: IDB-1003: Circular joins are not supported.
Query 17 ERROR 1815 (HY000): Internal error: IDB-3012: Scalar filter and semi join are not from the same
pair of tables.
Query 19 ERROR 1815 (HY000): Internal error: IDB-1000: 'lineitem' and 'part' are not joined.
ColumnStoreで実行する場合クエリを書き換える必要がある
133. TPC-H Benchmark
my.cnf 編集
# cd /usr/local/mariadb/columnstore/mysql
# vi my.conf
…
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 20GB
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
InnoDBでベンチマーク時のみ有効にする
135. TPC-H Benchmark
ベンチマーク実行
MySQL Host 127.0.0.1
MySQL Port 3306
MySQL User root
MySQL User password rootのパスワード
MySQL Database tpch or tpch_mcs
Data Warehouse Storage engine innodb
Scale Factor 10
Virtual Users Build Schema 1