Hear the tale of how we revamped an open source API management platform (API Umbrella) to meet the growing needs of the U.S. federal government. Over the past year, we've transitioned from a stack running on Node.js and Elasticsearch to a more efficient stack utilizing Lua/OpenResty and Hadoop/Apache Kylin. We'll dive into why we underwent this migration, and why you may or may not want to consider these technologies in other cases. We'll detail how we rolled out these significant architecture changes without anybody noticing, and also touch on the US government's increasing adoption of open source.

1. Re-architecting
the Government
with Lua and Hadoop
Nick Muerdter @nickblah
National Renewable Energy Laboratory
Gluecon
2016-05-26
★ ★ ★
★ ★ ★

3. Chapter 1
★ ★ ★
Origins

5. 2010:
APIs!

6. API Management

7. 2013:
💬
with other
agencies

8. Open Sourcing
API Umbrella

9. api.data.gov

10. Chapter 2
★ ★ ★
A Journey from
NodeJS to Lua

11. Custom sauce
(was NodeJS)

12. Internet
nginx
API
nginx
Redis
NodeJS
NodeJS
NodeJS
API
…

13. Why Change?
• Operational simplicity
• Efficiency & speed
• Benefits of hindsight

14. OpenResty
(nginx + LuaJIT)

15. Fun & useful things
about OpenResty

16. 1. Async/event-driven
(but without
callbacks/promises)

17. NodeJS Callbacks
var pg = require('pg');
pg.connect({ database: 'mydb' }, function(err, client, done) {
client.query('INSERT INTO people(name) VALUES('Jane')', function(err, res) {
client.query('SELECT first_name FROM people', function(err, res) {
done();
});
});
});

18. Lua
local pgmoon = require("pgmoon")
local pg = pgmoon.new({ database = "mydb" })
pg:connect()
local res, err = pg:query("INSERT INTO people(name) VALUES('Jane')")
local res, err = pg:query("SELECT name FROM people")

19. 2. Shared memory
dictionaries

20. 3. The nginx we know
and love

21. 4. It’s fast!

22. How to pull this off?

23. API Compatibility

24. 💖
Integration Tests

25. Internet
nginx
API
nginx
Redis
NodeJS
NodeJS
NodeJS
API
…

26. Internet
API
nginx + Lua
API
…

27. Proxy Overhead
• NodeJS: ~10ms
• OpenResty: ~1ms
• 50% less CPU use
*Benchmarking caveats…

28. It may not all be 🍑s

29. Chapter 3
★ ★ ★
Growing with
Hadoop, Kylin, and SQL

30. Log Data & Analytics

31. request_at: 2016-05-25T20:02:53.752Z
request_method: GET
request_scheme: https
request_host: developer.nrel.gov
request_path: /alt-fuel-stations/v1.json
request_query: fuel_type=ELEC&limit=10
user_id: ad2d94b6-e0f8-4e26-b1a6-1bc6b12f3d76
request_user_agent: curl/7.33.0
request_ip_country: US
request_ip_region: CO
request_ip_city: Golden
response_status: 200
response_time: 40
…
…

32. ElasticSearch

33. Apache Kylin

34. Data Cubes
• Pick the dimensions (columns) you
want to aggregate on.
• Define measurements (counts,
sums, etc).
• For every possible combination of
dimensions, measurements are pre-
aggregated.

35. Dimensions: host
Measurements: count
Raw Rows: 150
host count
developer.nrel.gov 50
api.nasa.gov 100

36. SELECT host, COUNT(*)
FROM logs
GROUP BY host;

37. Dimensions: host, path
Measurements: count
Raw Rows: 150
host path count
developer.nrel.gov /alt-fuel 30
developer.nrel.gov /solar 20
api.nasa.gov /photos 40
api.nasa.gov /asteroids 50
api.nasa.gov /mars 10

38. SELECT path, COUNT(*)
FROM logs
WHERE host = 'developer.nrel.gov'
GROUP BY path;

39. SELECT path, COUNT(*)
FROM logs
WHERE user_agent = 'curl'
GROUP BY path;

40. ANSI SQL is a lovely thing
• If Kylin can’t answer a query from its
pre-aggregations, we can fallback
to slower Hadoop queries against
the raw dataset.
• Presto gives an ANSI SQL interface
against the raw dataset.

41. But I could
pre-aggregate
myself…

42. Query Performance
• ElasticSearch: 5s, 10s, 60s, 2m…
• Kylin: ~1s
*Benchmarking caveats…

43. It’s not all 🍑s

44. ★ ★ ★
Epilogue
★ ★ ★

45. Simplifying your stack

46. Scaling efficiently

47. Find the right tools

48. Perhaps some other
things for your toolbelt:
OpenResty/Lua
Kylin
API Umbrella

49. Sleeping Material
• Lua
– https://github.com/NREL/api-umbrella/issues/86
– https://github.com/NREL/api-umbrella/pull/183
• Kylin
– https://github.com/18F/api.data.gov/issues/235

50. ★ ★ ★
Questions?
★ ★ ★
Nick Muerdter
nick.muerdter@nrel.gov
@nickblah