The way we think about data and databases must adapt to fit with dynamic 'cloud' infrastructure and Continuous Delivery. The need for rapid deployments and feedback from software changes combined with an increase in complexity of modern distributed systems and powerful new tooling are together driving significant changes to the way we design, build, and operate software systems. These changes require new ways of writing code, new team structures, and new ownership models for software systems, all of which in turn have implications for data and databases. In this talk, we will look at the factors driving increased deployability, the pattern of microservices as a way to improve deployability, changes to data models that microservices bring, and changes to team structures and responsibilities required to make these new approaches effective in a Continuous Delivery context.

1. Continuous Delivery for databases: microservices, team structures, and Conway's Law
Matthew Skelton, Skelton Thatcher Consulting
BBC:developconference,
Thursday 13thNovember 2014, London, UK
#bbcdevelop

2. Continuous Delivery
Databases
Microservices
Team structures
Conway’s Law

3. Developers & Database Administrators (DBAs)

5. Siege
http://galleryhip.com/castle-siege-wallpaper.html

6. http://theattractivearts.com/2013/11/life-after-love/

7. Shared context and language for Developers and DBAs to facilitate a move to microservicesand
Continuous Delivery

8. BUILDING BLOCKS
Scale (magnification)
Distributed transactions
Event streams

9. OUTCOMES
(New software architectures)
New team topologies
New responsibilities
(Continuous Delivery)

10. Matthew Skelton
•Building & operating commercial software systems since 1998
•Cybernetics + Neuroscience + Music
•control engineering
•psychology
•‘network’ and group interactions
@matthewpskelton

11. Skelton Thatcher
Introduction and Services Overview

12. FORCES / CONTEXT

13. ‘Cloud’ changes the way we mustdesign, deliver, and operateour software systems

14. Changes for ‘cloud’

15. FORCES
Deployability
(Elasticity)
(Conway’s Law)

16. DEPLOYABILITY
Single application
Single user journey
Single feature

17. DATABASE DEPLOYABILITY
Minimize changes in Production
Reduce accidental complexity
Archive, distinguish, and split data
Name things transparently
Source Business Intelligence from a data warehouse
Value more highly the need for change
Avoid Production-only tooling and config
http://bit.ly/DatabaseDeployability

18. MICROSERVICES
Single application
Single user journey
Single feature

19. DATABASE DEPLOYABILITY
End the siege mentality

20. BUILDING BLOCKS

21. BUILDING BLOCKS
Distributed transactions
Scale (magnification)
Event streams

27. BETWEEN ORGANISATIONS
Asynchronous data reconciliation
Eventual consistency

29. BETWEEN MICROSERVICES
Asynchronous data reconciliation
Eventual consistency

30. Familiar patterns,
different ‘magnification’

31. EVENT STREAMS
Event sourcing
NoSQL, CQRS, Reactive
Pub-Sub messaging

32. EVENT STREAMS
RDBMS database transaction log

33. Familiar patterns,
different ‘magnification’

34. ‘Fractal architecture’

35. FAMILIAR PATTERNS
Distributed transactionsEvent streamsAsynchronous data reconciliation
Eventual consistency

36. CHANGES

37. CHANGES
Team topology
Responsibilities

38. Conway’s Law

39. Mel Conway, 1968
“organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”
http://www.melconway.com/Home/Conways_Law.html

40. Ruth Malan, 2008
“if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”
http://traceinthesand.com/blog/2008/02/13/conways-law/

41. Topologies and responsibilities

48. There is no single ‘right’ team topology, but there are several ‘bad’ topologies for any one organisation

49. Make the team topology & responsibilities match the required software architecture

50. RECAPITULATION

51. Both Devsand DBAs know that: Distributed transactions do not scale wellEvent streams are the foundation of reliable systems

52. Shared concepts: Distributed transactionsEvent streamsAsynchronous data reconciliationEventual consistency

53. Familiar concepts, different scale & context

54. New team topologies needed

55. http://theattractivearts.com/2013/11/life-after-love/

56. We did not really mention:
Microservicestechnologies
Team topologies in depth
…

57. Continuous Delivery for databases
We need finer-grained, more frequent and isolated deployments
Microservicesallow these more fine-grained deployments
We need to support the microservicedatabase model: a larger number of smaller databases with eventual consistency and asynchronous (back office) data reconciliation
This is what we've been doing at an organisational level anyhow (Master Data, inter- bank reconciliation) so we apply the same principles at a Business Service level
This needs team restructuring to be effective (Conway's Law)
What team topologies might work well?

58. Continuous Delivery for databasesFurther details (article, slides): http://bit.ly/CDforDB

60. Further reading
Build Quality In
buildqualityin.com
Continuous Delivery & DevOps experience reports
Forewords by Dave Farley and Patrick Debois
70% of royalties go to Code Club (@CodeClub)
Contributors include several presenters at BBC:develop2014:
-Amy Phillips (Songkick)
-Phil Wills (Guardian)
Discount for #bbcdevelop!
https://leanpub.com/buildqualityin/c/BBCdevelop2014

61. Further reading
Ruth Malan (@ruthmalan) –Conway’s Law round-up: http://www.ruthmalan.com/Journal/2014/2014JournalMay.htm#Conways_Law
Allan Kelly on Conway’s Law: https://vimeo.com/channels/londoncd/85378217
Common database deployment blockers and Continuous Delivery headaches (Simple Talk) http://bit.ly/DatabaseDeployability
Team Topologies: http://bit.ly/DevOpsTopologies
Matt Hilbert on ‘Next Generation DBA’: http://bit.ly/NextGenDBA
Chris Richardson ‘Microservices: Decomposing Applications for Deployabilityand Scalability’ http://www.slideshare.net/chris.e.richardson/microservices-decomposing-applications-for- deployability-and-scalability-jax

62. Thanks
BBC for inviting the talk
Audience for listening, asking questions
People at Red Gate Software for joining up some of the dots

63. Thank you!
http://skeltonthatcher.com/
enquiries@skeltonthatcher.com
@SkeltonThatcher
+44 (0)20 8242 4103