People love to talk about scale. Some vendors pitch that their systems easily support 100,000 simultaneous calls, or 500 calls per second, etc. The reality is, in the real world, people’s behaviors vary and the feature sets they use can cut these numbers down quickly. For example, ask that same vendor claiming 100,000 simultaneous calls if it can be done while call recording, call statistics and other features are turned on at the same time, and you’ll usually get a very different, cautious, qualified response. In this presentation, we'll show you how to set up your infrastructure to support 100,000 simultaneous calls.

1. PRESENTED BY:
Tuning Kazoo to
10,000+
handsets
Luis Azedo

2. @kazoocon@kazoocon
About the Topic
• People love to talk about scale.
• Customer’s usage patterns are different.
• No cookie-cutter solution.

3. @kazoocon@kazoocon
Design Overview
KAZOO

4. @kazoocon@kazoocon
General linux considerations
• File Descriptors
• Sockets
• Processes
• Limits
• Logs

5. #kazoocon14
Let’s Talk About
Each Component

6. @kazoocon@kazoocon
Couch
• Overprovisioning is good
• Plan for # of accounts you’ll have in
1-2 years
• Z, N, W parameters
• Set these when you first start
• Z = Number of Datacenters / Zones
• N = Number of copies
• W = Number of writes before it’s
“good enough” to move on
• Db tool can be used later to redo

7. @kazoocon@kazoocon
Freeswitch
• Bare metal vs VMs / docker
• 5 x 200 calls vs 1 x 1000 calls
• Sqlite vs mysql (memory locks)
• mod_kazoo filters

8. @kazoocon@kazoocon
Kamailio
• TCP Workers
• Shared Memory
• db_text vs mysql (in memory)
• Configure Zones
• AMQP workers / consumers
• Use ASYNC when possible

9. @kazoocon@kazoocon
Kamailio
#!substdef "!MY_AMQP_URL_1!kazoo://user:pass@amqp-01.lx1.com:5672!g"
#!substdef "!MY_AMQP_URL_2! kazoo://user:pass@amqp-02.lx1.com:5672!g”
#!substdef "!MY_AMQP_URL_3! kazoo://user:pass@amqp-03.lx1.com:5672!g”
#!substdef "!MY_AMQP_URL_4!zone=lx2;kazoo://user:pass@amqp-01.lx2.com:5672!g"
#!substdef "!MY_AMQP_URL_5! zone=lx2;kazoo://user:pass@amqp-02.lx2.com:5672!g”
#!substdef "!MY_AMQP_URL_6! zone=lx2; kazoo://user:pass@amqp-03.lx2.com:5672!g”
#!substdef "!MY_AMQP_URL_7!zone=lx3;kazoo://user:pass@amqp.lx3.com:5672!g”
modparam("kazoo", "amqp_consumer_processes", 8);
modparam("kazoo", "amqp_consumer_workers", 32);
Zone Configuration

10. @kazoocon@kazoocon
Kamailio
route[KAZOO_AUTHORIZATION]
{
xlog("L_INFO", "$ci|log|check $adn");
$var(amqp_payload_request) = '{"Event-Category" : "directory" , "Event-Name" : "authn_req", "Method" : "REGISTER", "Auth-Nonce" :
"$adn", "Auth-Realm" : "$fd", "Auth-User" : "$fU", "From" : "$fu", "To" : "$tu" }';
$var(amqp_routing_key) = "authn.req." + $(fd{kz.encode});
$avp(kz_timeout) = 2500;
if(kazoo_async_query("callmgr", $var(amqp_routing_key), $var(amqp_payload_request), "KAZOO_AUTHORIZATION_OK",
"KAZOO_AUTHORIZATION_ERROR") != 1) {
xlog("L_INFO", "$ci|log|failed to send Kazoo query for authentication credentials for $Au $si:$sp");
append_to_reply("Retry-After: 60rn");
send_reply("500", "Retry Later");
exit;
}
}
route[KAZOO_AUTHORIZATION_OK]
{
$var(password) = $(kzR{kz.json,Auth-Password});
$var(nonce) = $adn;
$xavp(ulattrs=>custom_channel_vars) = $(kzR{kz.json,Custom-Channel-Vars});
xlog("L_INFO", "$ci|log|authenticating $Au via Kazoo query response");
route(CHECK_AUTHORIZATION);
}
route[KAZOO_AUTHORIZATION_ERROR]
{
xlog("L_INFO", "$ci|log|failed to query Kazoo for authentication credentials for $Au $si:$sp");
append_to_reply("Retry-After: 60rn");
send_reply("500", "Retry Later");
exit;
}

11. @kazoocon@kazoocon
Whapps
3 Servers
2 Servers
One Server crossbar, registrar, callflow, fax
crossbar, callflow, fax
crossbar
callflow,
fax
registrar
registrar

12. @kazoocon@kazoocon
REST
Smtp
Couch
HAProxy

13. @kazoocon@kazoocon
Rabbit
• Thresholds
• Granular statistics (10% gain)
• Cluster rabbit
• Federation & Zones

14. @kazoocon@kazoocon
Rabbit Clustering
• High available queues
• Publish & consume
from any node
• Policies for replication
• LAN Only

15. @kazoocon@kazoocon
Rabbit Federation
• Federation plugin
primary Zone a Zone b
• Kazoo Federation
• Granular control

16. @kazoocon@kazoocon
REGISTER 401 REGISTER AUTHN_REQ COUCH AUTHN_RESP USRLOC REG_SUCCESS ECALLMGR
registration scenario
2 kamailio , 2 x kazoo registrar
instances with 4 listeners, 2 x
rabbit (clustered)
In this scenario, we are showing
5,800+ registrations per second.
Most phones register every 360
seconds, so:
360 x 5800 = 2,088,000 phones
(THEORETICAL!!!)

17. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy

18. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch

19. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH

20. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps

21. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit

22. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
Kamailio
FreeSWITCH

23. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
Kamailio
FreeSWITCH
apps apps
apps
apps
Rabbit

24. @kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
Kamailio
FreeSWITCH
apps apps
apps
apps
Rabbit
Kamailio
FreeSWITCH
Rabbit
apps
HAProxy
Start a New Zone

25. @kazoocon@kazoocon
The World & Beyond
“A thing called
open-source
kazoo platform
actually make it
possible.”

26. Thank You!
@kazoocon
Thank You!
@kazoocon