Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Neo4j for Cloud Management at Scale

Cloud system configurations and their dependencies can quickly grow into the thousands of virtual machine, network and storage components. Once software is included, the number of components can easily rise into six figures.


Frequent releases using continuous integration and deployment tools makes a repository of these components and relationships absolutely critical to cloud system integrity and quality of service no matter what cloud management tools you use.


Systems configurations are more naturally represented using a graph database than the relational representations used by traditional IT management products.


Our talk will explore how we use Neo4J to create a live, active, self-updating repository service, containing nearly all virtual hardware, network and software components and their dependencies, enabling continuous deployment in any cloud environment at scale.

  • Login to see the comments

Neo4j for Cloud Management at Scale

  1. 1. David Brian Ward CEO, Telegraph Hill Software www.telegraphhillsoftware.com 535 Mission Street San Francisco CA 94105 Neo4j for Cloud Management at Scale
  2. 2. Headline text will go right here A Renaissance Of Database Innovation
  3. 3. Headline text will go right here Big ITIL Framework, Asset Heavy IT A shared CMDB as the single system of record is a key ITIL best practice for good reason.
  4. 4. Headline text will go right here Pre-DevOps, Pre-Cloud Ops Engineering Would you build an IT edifice like a cathedral today? Origins of DevOps Culture: • Open Source • Cloud Computing • Need for Dev Speed
  5. 5. Headline text will go right here Early Cloud, Early DevOps Days Shopping at the Open Source Bazaar – A new high quality DevOps tool every week?
  6. 6. Headline text will go right here On Sale: DevOps for 90% Off! A Graph Database Can Make It Possible!
  7. 7. Headline text will go right here Evolving our DevOps Approach Cloud DevOps should combine an integration hub architecture with deployment automation enabling the evolution of infra, apps and tools and their migration.
  8. 8. Headline text will go right here MacGyver and Neo4J for Business Critical DevOps
  9. 9. • API Integration with infra, tools and apps • A batch scheduler for near real time config • Data aggregation across services • Monitoring and alerting integration • Java script execution • An extensible UI and dashboards • Infrastructure mapping • Tools for building engineering, operations, infosec, QA and finance use cases • High availability and multi-site MacGyver Service Basics
  10. 10. • Perfect for capturing/modeling interdependencies • Cypher’s ad hoc query capability can’t be beat • Easy to extend, build more relationships and layers incrementally • Great join/traversal capability • Flexible and scalable vs rigid; dataset can easily evolve and grow in terms of complexity and structure • Easy to consume • Natural for infrastructure mapping and enterprise architecture • JSON native, FTW Neo4J is Perfect for MacGyver
  11. 11. “Do we have a single point of failure among any of our services?”
  12. 12. WHO ARE OUR CUSTOMERS? Continuous integration Test automation Release packaging High-availability/failover Server, network and environment provisioning Monitoring App performance monitoring Cloud cost management Audit  Change management,  Continuous deployment,  Service discovery  Micro-service cloud migration  Self healing systems  System and security maintenance  Operational cost and capacity management  Blue/green deployment,  API management and security  And more every day MacGyver Use Cases at Lending Club
  13. 13. WHO ARE OUR CUSTOMERS? Network: • A10 • ASA (adaptive security appliance) • DNS • F5 Software Code and Deployment: • Artifactory • Github • Jenkins • Springframework Authentication and Access: • AACS (advanced access content system) • Ldap • Microsoft active directory & sso Cloud Service Providers: • Aws • Cloudstack • Vsphere User Interface & Dashboards • Leftronic • Grafana (graphite) Notification and Collaboration • Hipchat • Pagerduty • Smtp Operational Documentation • Jira/Confluence Database Platforms • Mongo • JDBC (oracle, mysql) Performance Monitoring • Catchpoint • L7 • Newrelic MacGyver Integrations at Lending Club System Config and Management • Nimble • Purestorage • Puppet • Saltstack Monitoring and logs • Signalfx • Splunk MacGyver tools and services • ‘health check’ / bootstrap • Micro-service registry
  14. 14. Headline text will go right here DevOps Evolution with MacGyver Business Critical DevOps must anticipate the evolution of infra, apps and tools. E.g., • Physical -> Vsphere -> AWS • Monolithic -> Micro Services • Jenkins -> AWS Code Deploy • Bonus: Vendor Independence
  15. 15. WHO ARE OUR CUSTOMERS?MacGyver Micro Services – Service Discovery Problem: Keeping track of many rapidly-changing services Solution: All app servers phone home to MacGyver and are stored in Neo4j as ‘App Instance’ nodes. Deployment and release automation assure a real time database of deployed services. New services get auto-discovered by MacGyver. • Low maintenance • Easy scalability • Low latency ad hoc query capability
  16. 16. WHO ARE OUR CUSTOMERS?MacGyver Micro Services – Deployment Problem: Highly manual and tedious releases Difficult to answer questions like: –What pool should I deploy to? –Is the most recent revision ‘live’ right now? –Are live pool revisions in sync in different environments? Solution: Utilize app check-ins and Neo4j to expose info about live and dark pools, enabling us to automate deployments, and build on our existing monitoring automation.
  17. 17. Application Pools
  18. 18. Deployment and Release Automation •Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group “Live” Pool “Dark” Pool
  19. 19. Deployment and Release Automation •Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group
  20. 20. Deployment and Release Automation •Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group “Draining” Pool “Live” Pool
  21. 21. Deployment and Release Automation •Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group Pool Cut-over
  22. 22. Virtual Service Deployed
  23. 23. WHO ARE OUR CUSTOMERS? 280 production (micro)services 4338 hosts (+501 AWS EC2 instances) 2 sites (LAS & SJC) 4 environments (dev, demo, stage, prod) 108,917 graph nodes 5% #Lines of code compared to Netflix OSS MacGyver Size and Scale at Lending Club
  24. 24. WHO ARE OUR CUSTOMERS?MacGyver UI views
  25. 25. WHO ARE OUR CUSTOMERS?MacGyver UI views
  26. 26. WHO ARE OUR CUSTOMERS?MacGyver UI views
  27. 27. WHO ARE OUR CUSTOMERS?
  28. 28. Headline text will go right here What’s Next? Reactive DevOps As with the F35 cockpit, the sky is no longer the limit.
  29. 29. CLOSING THOUGHTS • MacGyver provides an Integration Architecture that enables scalability, enrichment, evolution and migration • MacGyver and Neo4J enables you to evolve your infrastructure using best of breed components, all the while running your business critical systems with integrity. • MacGyver dramatically reduces the amount of software development required for even the most sophisticated DevOps use cases. • Perfect for managing your hybrid infrastructure while staying ahead of Dev.
  30. 30. Headline text will go right here What’s Next? Reactive DevOps As in the F35 cockpit, the sky is no longer the limit.
  31. 31. CLOSING THOUGHTS • MacGyver provides an Integration Architecture that enables scalability, enrichment, evolution and migration • MacGyver and Neo4J enables you to evolve your infrastructure using best of breed components, all the while running your business critical systems with integrity. • MacGyver dramatically reduces the amount of software development required for even the most sophisticated DevOps use cases. • Perfect for managing your hybrid infrastructure while staying ahead of Dev? We know so.
  32. 32. QUESTIONS? FOR MORE INFO… https://github.com/if6was9/macgyver https://github.com/if6was9/neorx Ashley Sun – asun@lendingclub.com, @ashleycsun Rob Schoening - rschoening@lendingclub.com, @rschoening David Ward – david.ward@thpii.com Sarah Lewis – sarah.lewis@thpii.com, info@thpii.com
  33. 33. Extra slides
  34. 34. www.telegraphhillsoftware.com 535 Mission Street San Francisco CA 94105 Using Neo4j for Cloud Management at Scale
  35. 35. App Instances MacGyv er Neo4j <<update app instance>> MacGyv er
  36. 36. Virtual Servers MacGyv er Neo4j Load Balancer<<REST API Polling>>
  37. 37. Virtual Servers MacGyv er Neo4j <<update virtual server>> Load Balancer<<REST API Polling>>
  38. 38. App Instance App Instance App Instance App Instance Deployment and Release Automation MacGyv er Neo4j Load Balancer <<PUT>> <<REST API>> <<REST API>>

×