DevOps for the DBA

1. DEVOPS FOR THE DBA GRANT FRITCHEY PRODUCT ADVOCATE REDGATE SOFTWARE

2. GOALS • Discuss tools, process and the fundamental changes in culture necessary to make your database development and deployment a part of a high functioning DevOps team

3. scarydba.com grant@scarydba.com @gfritchey Grant Fritchey youtube.com/c/GrantFritchey

4. WHAT IS DEVOPS?

5. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system- oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.

6. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.

7. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.

8. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people(and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.

9. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people(and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.

10. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people(and culture), and seeks to improve collaborationbetween operations and development teams. DevOps implementations utilize technology – especially automation toolsthat can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.

11. DEVOPS •An agile culture that better supports collaboration between people that uses automation and tooling in support of process.

12. WHO IS IN DEVOPS

13. WHO IS IN DEVOPS

14. WHAT DEVOPS IS NOT

15. WHY DEVOPS? Improved communications Increasing efficiency Faster development Fewer failures Lower costs

16. DEVOPS ADOPTION

17. WHY DEVOPS FOR THE DBA?

18. HOLDING BACK THE OCEAN Applying Versions and Patches Missing Backups Bad Code Incorrect Structures Misconfigured Servers Change You!

19. SOME JUSTIFICATION

20. FUD IS THE PROBLEM DevOps only works for new systems ‘SA’ for developers NoOps DevOps is incompatible with security and compliance Agile means no design Elimination of process

21. PROTECTION

22. DBA INTEGRATION

23. DATA PAIN POINTS

24. WHY DEVOPS FOR THE DBA Improved communications Better integration with dev, ops and the business Added protections through superior processes Better support for the organization Faster and safer deployments

25. A CULTURE OF COMMUNICATION

26. DEVOPS FUNDAMENTALS Communication is everything All code in source control Shift Left Automated Tests Automated Deployments Monitoring

27. COMMUNICATION IS CORE TO DEVOPS Eliminating barriers Mitigate, reduce, and eliminate choke points Successful implementation of DevOps involves all the teams

28. ORGANIZATIONAL CULTURE “We can’t change” “We’ve always done it this way” “Fine, until something goes wrong” “As long as it doesn’t impact the schedule”

29. CULTURAL SHIFT Transparency What do you do? How do you do it? Write it all down Collaboration Tear down the wall We’re all on the same team Clarity Definitions matter Language matters

30. MANAGEMENT WITHOUT DIRECT MANAGEMENT BUY-IN AND SUPPORT, THIS ALL COLLAPSES

31. DEVOPS FOR THE DBA

33. SOURCE CONTROL IS A BACKUP Every database is code More backups is better Recovery is hard An “undo” button in a database

34. “SHIFT LEFT” ADDS PROTECTION EARLIER TESTING FASTER FAILURE FREQUENCY OF DEPLOYMENT SELF-SERVICE MEANS LEARNING

35. AUTOMATED TESTING PROTECTS PRODUCTION • When something breaks deploying in dev, build a test • When something breaks deploying in CI, build a test • When something breaks deploying to QA, build a test • Every test improves protection for production • Testing combined with “shift-left” improves production safety

36. FREQUENT DEPLOYMENTS MEANS PRACTICE No script should run first in production Every deployment is a test Practice is fundamental to mastery “Shift-left” makes practice deployments mean more All this adds to protection for production

37. MONITORING ENSURES BEHAVIOR Understand behavior prior to deployment Know when deployments occur Understand behavior after a deployment Monitoring drives innovation Protection starts with monitoring

39. DEFINING YOUR PROCESS

40. SOME CAVEATS No two systems are the same What works for me, may not work for you Compliance drives some decisions Business needs drive others

41. DATABASE CHALLENGES DATA PERSISTENCE SIZE OF DATA VOLUME OF CHANGES LACK OF MONITORING

42. BUILDING BLOCKS Environments Databases Applications Data movement Compliance

43. ENVIRONMENTS Development Laptop Shared Development Development Integration Continuous Integration Quality Assurance Testing Financial Testing Performance Testing Pre- Production Staging Production

44. DATABASES One database or many Linked servers Replication Availability Groups Data size

45. APPLICATONS One application or many Linked or unlinked deployments Linked or unlinked testing Deployment methodologies

46. DATA MOVEMENT Shift Left SSIS Replication Compliance Analysis Reporting

47. COMPLIANCE Test data Development data Clean data Generated data Hybrid

48. BUILDING A DEPLOYMENT PROCESS WORK BACKWARDS FROM PRODUCTION IF IT’S NOT IN SOURCE CONTROL, IT DOESN’T EXIST ENSURE COMPLIANCE IS ALWAYS COVERED TEST WHAT BREAKS

49. BUILDING A DEPLOYMENT PROCESS Production deployment script Test script on copy of production Generate script for production from trunk Merge code in source control Add changes to source control Make changes to code Get code from source control Branch code in source control

50. TOOLS IN SUPPORT OF AUTOMATION

51. OPERATING SYSTEMS WINDOWS LINUX

52. LANGUAGES SQL POWERSHELL JAVA/JAVASCRIPT BASH

53. CONTAINERS OPEN CONTAINER INITIATIVE (OCI) DOCKER

54. NATIVE TOOLS VISUAL STUDIO SQL SERVER MANAGEMENT STUDIO AZURE DATA STUDIO

55. FLOW CONTROL CHEF JENKINS AZURE DEVOPS OCTOPUS TEAMCITY GITLAB READYROLL

56. SOURCE CONTROL Subversion/SVN Git Azure DevOps Team Foundation Services AWS Code Commit Helix Core

57. DATABASE TOOLING, THE MAGIC Microsoft SQL Server Data Tools Redgate SQL Change Automation DBMaestro DbUp Datical Apex/Idera DevOps Flyway

58. HOW THE MAGIC WORKS State • Compare between database & database • Compare between database & source control • Compare between ? & ? Migrations • Scripts build on scripts • Which build on scripts Hybrid • Mix & match

59. EXERCISE

60. SCENARIO ONE • We have a single database under development, but we use linked servers for reporting. How do we: • A) Automate our processes across multiple environments • B) Prevent use of production data in linked servers

61. SCENARIO TWO • We have multiple teams developing within a single database. How do we: • A) Support completely independent development and release cycles • B) Ensure that functionality from one team doesn’t affect another • C) Get each teams functionality to the other teams

62. SCENARIO THREE • Our system consists of four different, interrelated databases. Each one is on it’s own for development and release, but, we need to have all four while we develop. How do we: • A) Keep four databases on every developers machine • B) Ensure that different development teams don’t break others code

63. SCENARIO FOUR • We’re pretty happily chugging along with our DevOps processes, however, since moving to Azure SQL Database and implementing automated tuning, we’re seeing indexes in production that don’t exist elsewhere. How do we: • A) See what has changed prior to deployment • B) Ensure that we can incorporate these changes into our own processing

64. SOURCE CONTROL

65. SOURCE CONTROL SOFTWARE THAT ENABLES DEVELOPMENT THROUGH COLLABORATION AND CHANGE TRACKING

66. SOURCE CONTROL Subversion/SVN Git Azure DevOps Team Foundation Services AWS Code Commit Helix Core

67. SOURCE CONTROL FUNCTIONALITY Keep track of code changes Maintain a history of changes Multiple people working simultaneously Isolate code through branching

68. SOURCE CONTROL FUNCTIONALITY Merge code from different branches on command Show conflicts on merges in order to resolve them Allow reversion to a previous state Works within your toolset

69. GIT Local version control Central version control Branch/Merge core design concepts Very broad adoption and support It just works

70. GIT TERMINOLOGY Master Branch Pull/Sync Commit Merge Push Pull Request

71. GITFLOW • Sourced: A successful Git branching model

72. GITHUB FLOW • Sourced: Git Workflows That Work

73. MY SUGGESTION master Online Feature Branch Local Feature Branch Pull/Sync Commit after testing Push Push Pull Request and Merge

74. A SHORT DISCUSSION ON DATABASE BRANCHING • You can’t!

75. A SHORT DISCUSSION ON DATABASE BRANCHING ONE DATABASE PER BRANCH ON AN INSTANCE ONE INSTANCE PER BRANCH ONE CONTAINER PER BRANCH JUST BE READY TO REBUILD DATABASES AS NEEDED

76. DEMONSTRATION

77. AUTOMATED TESTING

78. WHY TESTING Things go wrong Mistakes are made Complexity leads to error

79. GENERAL TESTING PRACTICES TEST EARLY AND OFTEN TESTING STARTS IN DEVELOPMENT SLOWLY ADD TESTS OVER TIME VALIDATE STANDARDS DOCUMENT YOUR INTENTIONS

80. HOW DO WE TEST? Automatically Define a known starting state • Generated test data • Modified production data • Always work within Compliance requirements Test data must be provided

81. WHAT DO WE TEST WITH? tSQLt Microsoft Unit Testing Projects DbUnit Custom Scripts

82. WHAT DO WE TEST? CODE COVERAGE DATA MOVEMENT THINGS THAT BREAK

83. DEMONSTRATION

84. AUTOMATING DEPLOYMENTS

85. WHAT ARE WE AUTOMATING Getting code into source controlGetting Retrieving code from source controlRetrieving Generating an artifact (aka SQL)Generating

86. DATABASE TOOLING, THE MAGIC Microsoft SQL Server Data Tools Redgate SQL Change Automation DBMaestro DbUp Datical Apex/Idera DevOps Flyway

87. STATE VS. MIGRATIONS State • Compare between database & database • Compare between database & source control • Compare between ? & ? Migrations • Scripts build on scripts • Which build on scripts Hybrid • Mix & match as needed

88. STATE: STRENGTHS & WEAKNESSES Strengths • Simple to understand • Merge is easy • Tools generate scripts • Visible history Weaknesses • Domain of problems not solved • Tool ordering of changes can be problematic • Tool choices for changes can be an issue

89. MIGRATIONS: STRENGTHS & WEAKNESSES Strengths • All changes work • Control over method and changes • All domain problems handled Weaknesses • Harder to understand and work with • History is hard to assemble • Merges may be difficult • All changes in Dev occur in Prod

90. THE NEED FOR AN ARTIFACT Separate the concept of a build from a deploy Build generates an artifact Deploy uses the artifact to create/update a database Automate both

91. CONTINUOUS INTEGRATION FREQUENT BUILDS FREQUENT DEPLOYMENTS FREQUENT TESTS FAIL FAST FOR AN IMMEDIATE FEEDBACK LOOP

92. CONTINUOUS INTEGRATION • Get your code into source control • Pick a tool for the build • Pick a flow control tool • Build your CI process • You’ve now started down the DevOps path

93. FLOW CONTROL CHEF JENKINS AZURE DEVOPS OCTOPUS TEAMCITY

94. DEMONSTRATION

95. CONTINUOUS DELIVERY

96. CONTINUOUS DELIVERY DELIVER FUNCTIONALITY INTO PRODUCTION QUICKLY, SAFELY, AND CONSTANTLY

97. ENVIRONMENTS Development Laptop Shared Development Development Integration Continuous Integration Quality Assurance Testing Financial Testing Performance Testing Pre- Production Staging Production

98. FLOW CONTROL CHEF JENKINS AZURE DEVOPS OCTOPUS TEAMCITY

99. ENVIRONMENTAL DIFFERENCES SECURITY COMPLIANCE SECONDARY PROCESSES

100. PRODUCTION DRIFT There will ALWAYS be production fixes Prod changes must go to VCS Drift may block deployments Tooling detect drift Track versions of the db yourself Drift detection is needed

101. ROLLBACKS Once changes committed, can't be rolled back Old version of code Be prepared Pre-write rollback scripts Have old code versions ready (snapshot production) Be ready to restore

102. ROLLING BACK CODE The previous version of procs/functions/views can be used Double check logical function with the rest of the application Can we roll back independently? Is application code rollback needed? Previous version from production Previous released version from VCS Auto-generate rollback code

103. ROLLBACK TABLE AND DATA CHANGES Undoing data changes is complex and risky Pre-write scripts Include DML changes Test in intermediate environments May need to execute manually Copy old data prior to changes Use scripts to restore old data

104. ROLL FORWARD Faster • Faster than a rollback Safer • Safer than a rollback Easier • Easier than a rollback Harder • Harder to convince management to support

105. POST RELEASE Any deployment impacts the system Lack of impact is information Monitoring provides a baseline of resource usage Compare the baseline to new resource and business use

106. DEMONSTRATION

107. EXERCISE

108. SCENARIO ONE • We have a single database under development, but we use linked servers for reporting. How do we: • A) Automate our processes across multiple environments • B) Prevent use of production data in linked servers

109. SCENARIO TWO • We have multiple teams developing within a single database. How do we: • A) Support completely independent development and release cycles • B) Ensure that functionality from one team doesn’t affect another • C) Get each teams functionality to the other teams

110. SCENARIO THREE • Our system consists of four different, interrelated databases. Each one is on it’s own for development and release, but, we need to have all four while we develop. How do we: • A) Keep four databases on every developers machine • B) Ensure that different development teams don’t break others code

111. SCENARIO FOUR • We’re pretty happily chugging along with our DevOps processes, however, since moving to Azure SQL Database and implementing automated tuning, we’re seeing indexes in production that don’t exist elsewhere. How do we: • A) See what has changed prior to deployment • B) Ensure that we can incorporate these changes into our own processing

112. BONUS SCENARIO • We’re using state-based delivery of our databases. Because we have multiple development streams, we have multiple QA systems. We have an emergency release. How do we: • A) Manage getting our code from dev to QA to preprod to production

113. WRAP UP

115. DEVOPS FUNDAMENTALS One Team More Backups Added Protection Automated Protection Practice Deployments Monitoring

116. GOALS • Discuss tools, process and the fundamental changes in culture necessary to make your database development and deployment a part of a high functioning DevOps team

117. RESOURCES • State of Database Devops 2019 • The Phoenix Project • Redgate University • Microsoft Learning Resources for DevOps • Microsoft DevOps Resource Center • Steve Jones • Kendra Little

118. scarydba.com grant@scarydba.com @gfritchey Grant Fritchey youtube.com/c/GrantFritchey

119. PHOTO CREDITS • Programmer • Cat and Dog • Operations • Dba running with scissors • Crowd/Everyone • Peggy in the office • Culture of Communication • Tools • Management • Rube Goldberg • Exercise • Hope • Source control (safe) • Demonstration • Automatic • Delivery • Wrapping