ProxySQL - High Performance and HA Proxy for MySQL
•
9 likes•5,170 views
High Availability proxy designed to solve real issues of MySQL setups from small to very large production environments. Presentation at Percona Live Amsterdam 2015
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to ProxySQL - High Performance and HA Proxy for MySQL
Similar to ProxySQL - High Performance and HA Proxy for MySQL (20)
Recently uploaded
Recently uploaded (20)
ProxySQL - High Performance and HA Proxy for MySQL
- 1. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com ProxySQL High Performance & High Availability Proxy for MySQL René Cannaò rene@sysown.com
- 2. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com GA Stable and production ready version 0.2
- 3. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Genesis: main motivations rewrite queries cache frequent reads solving/finding the answer for DEVs vs DBAs
- 4. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Open Source proxies for MySQL MySQL Proxy HAProxy http://tinyurl.com/ptjmzde Now also MaxScale
- 5. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com MySQL Proxy Pros: customizable expandable acceptable performance Cons: CPU intensive Not really reliable Not maintained
- 6. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com HAProxy Pros: very stable high performance mature software Cons: Layer 7 proxy only for HTTP Doesn’t understand the MySQL Protocol Layer 4 for other services
- 7. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Proxy for rewrite and caching? MySQL Proxy with: Lua script External caching
- 8. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com ProxySQL Network database proxy that sits transparently between the application and the database to: empower the DBAs improve operation understand and improve performance
- 9. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com The right way! Designed by a DBA for DBAs Experience in a variety of setups Experience in very large production environments
- 10. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Features Some of the most interesting features: on-the-fly rewrite of queries caching reads outside the database server connection pooling and multiplexing complex query routing and read/write split load balancing real time statistics Monitoring High Availability and Scalability Seamless failover Firewall Query throttling query timeout runtime reconfiguration
- 11. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Hostgroups and Query Routing All backends are grouped into hostgroups Hostgroups have logical functionalities
- 12. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Hostgroups example #1 HostGroup0 (HG0): Write masters HostGroup1( HG1): Read slaves Read/Write split
- 13. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Hostgroups example #2 HG0: main write masters HG1: main read slaves HG2: reporting slaves HG3: ad-hoc queries slaves HG4: data warehouse write masters HG5: data warehouse read slaves HG6: remote site servers HG7: test servers
- 14. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Support for heterogeneous setups Hostgroups have logical functionalities No replication relationship between hostgroups ProxySQL is not replication aware by design, yet it monitors replication A query is routed to a single hostgroup
- 16. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Basic design , example #2 HG0 HG2 HG1 ProxySQL APP1 APP2 APP3
- 18. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries Processor Rewrite queries Defines what to cache Defines the hostgroup target
- 19. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries rules Complex rules to match incoming traffic. Regex on query username schemaname (a lot more in the roadmap) Rules can be chained
- 20. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries cache Caching on the wire Internal key/value storage In memory only Pattern based Expired by timeout
- 21. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries Cache Benchmark Benchmark on a 4 cores server
- 22. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries Rewrite Rewrite on the wire Regex match/replace Optionally cached
- 23. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries Rewrite Benchmark Ref: http://tinyurl.com/oan4528
- 25. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Users Authentication Credentials stored in the proxy User login always possible (even without backends) Max connections Security enhancement in roadmap: different credentials on backends and frontends internal mapping
- 26. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Hostgroup Manager and Connections Pool HG0 HG1 APP1 ProxySQL Queries cache Query Processor APP2 APP3 Thread#1 Thread#2 Thread#3 Users Auth Connection Pool Hostgroup Manager
- 27. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Hostgroups Manager Management of servers Track servers status Tightly integrated with the connections pool
- 28. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Connections Pool Reduced the overhead of creating new connections, and are recycled when not in use One to many connections Multiplexing & maximum connections Auto-reconnect and automatic re-execution of queries Failover management
- 29. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Multiplexing Reduce the number of connections against mysqld (configurable) Many clients connections (tens of thousands) can use few backend connections (few hundreds) Order by waiting time Few edge cases not handled correctly: open a feature request if you hit any
- 30. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Auto-reconnect and re-execution Automatic detection of failures Graceful handling Auto-reconnect when possible Pause until a backend becomes available Re-execution of queries
- 31. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Failover management Seamless switchover: http://www.proxysql.com/2015/09/proxysql-tutorial-seamles s-replication.html Managed by external process Switchover in less than 1 second
- 32. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Monitoring Module HG0 HG1 APP1 ProxySQL Queries cache Query Processor APP2 APP3 Thread#1 Thread#2 Thread#3 Users Auth Connection Pool Hostgroup Manager Monitoring
- 33. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Monitoring Module It monitors backends and collects metrics Monitors replication lag and shun hosts
- 34. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Admin Interface HG0 HG1 APP1 ProxySQL Queries cache Query Processor APP2 APP3 Thread#1 Thread#2 Thread#3 Users Auth Connection Pool Hostgroup Manager Admin Monitoring
- 35. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Admin Interface Allows runtime configuration Exports internal statuses It uses MySQL protocol Configuration possible from any client/tool using MySQL API
- 36. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Admin Interface mysql> show databases; +-----+---------+------------------+ | seq | name | file | +-----+---------+------------------+ | 0 | main | | | 2 | disk | /tmp/proxysql.db | | 3 | stats | | | 4 | monitor | | | 5 | myhgm | | +-----+---------+------------------+ 5 rows in set (0.00 sec)
- 37. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Admin Interface mysql> SHOW TABLES FROM main; +-------------------+ | tables | +-------------------+ | mysql_servers | | mysql_users | | mysql_query_rules | | global_variables | | mysql_collations | | debug_levels | +-------------------+ 6 rows in set (0.01 sec)
- 38. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Admin Interface mysql> SHOW TABLES FROM stats; +--------------------------------+ | tables | +--------------------------------+ | stats_mysql_query_rules | | stats_mysql_commands_counters | | stats_mysql_processlist | | stats_mysql_connection_pool | | stats_mysql_query_digest | | stats_mysql_query_digest_reset | | stats_mysql_global | +--------------------------------+ 7 rows in set (0.00 sec)
- 39. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Admin Interface mysql> SHOW TABLES FROM monitor; +----------------------------------+ | tables | +----------------------------------+ | mysql_server_connect | | mysql_server_connect_log | | mysql_server_ping | | mysql_server_ping_log | | mysql_server_replication_lag_log | +----------------------------------+ 5 rows in set (0.00 sec)
- 40. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries cache rules proxysql-admin> SELECT match_pattern , negate_match_pattern neg , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NULL ORDER BY rule_id ; +---------------------+------+-------+------+ | match_pattern | neg | hs_id | ttl | +---------------------+------+-------+------+ | ^SELECT | 1 | 0 | -1 | | s+FORs+UPDATEs*$ | 0 | 0 | -1 | | .* | 0 | 1 | 30 | +---------------------+------+-------+------+
- 41. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Queries rewrite rules proxysql-admin> SELECT match_pattern m, replace_pattern r , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NOT NULLG m: ^SELECT(| DISTINCT) c FROM sbtest WHERE id BETWEEN (.*) AND (.*) ORDER BY c r: SELECT c FROM sbtest WHERE id BETWEEN 2 AND 3 hs_id: 1 ttl: 30
- 43. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Location, location, location ProxySQL sits between the application and the database systems Where is exactly the best location?
- 44. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Basic design HG0 HG2 HG1 APP APP ProxySQL Very low latency using Unix Domain Socket
- 48. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Hostgroup reconfiguration HG0 HG1 1 43 ProxySQLAPP 2 Remove host : all the connections to server2 are terminated 1 Add host : New connections to HG1 can use also server1
- 49. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Failover 2 phases process: remove host add host Fully supported Seamless failover in under 1 second
- 50. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Failover : 2 phases task HG0 HG1 1 43ProxySQLAPP 2 1.Connections to an empty hostgroup are put on hold 2 2.Connections to HG0 are resumed Connections to HG1 were never stopped 1. remove host 2. add host
- 51. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Distributed failover Multiple ProxySQL are available in a network Failover managed by an external process that: remove host from each ProxySQL instance add new host into each ProxySQL instance Manager is not part of ProxySQL. Ex: MHA or MySQL Utilities
- 52. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com APP1 ProxySQL APP2 ProxySQL Distributed failover : remove host HG0 1 HG1 32 4 MHA
- 53. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com APP1 ProxySQL APP2 ProxySQL Distributed failover : add host HG0 1 HG1 3 4 MHA 2
- 54. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com New features in roadmap Improved security (SSL support, SQL injection detection, frontend/backend users, etc) ; GUI; Prepared statements; Support for multiple backends (postgres, redis, mongo) ; Enhanced monitoring and statistics;
- 55. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Try it! Source code on GitHub: https://github.com/sysown/proxysql Binaries on GitHub: https://github.com/sysown/proxysql-binaries Forum: https://groups.google.com/forum/#!forum/proxysql Tutorial on: http://www.proxysql.com
- 56. @proxysql http://proxysql.com https://github.com/sysown/proxysql/sysown.com Please contact me Thanks! Feedback, feature requests, contribution: rene.cannao@gmail.com rene@sysown.com @rene_cannao @proxysql