Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

DevOps in Silos

Download to read offline

PASS DevOps VC 09/2020

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

DevOps in Silos

  1. 1. DevOps in a Vacuum of Silos Kellyn Gorman Sr. Cloud Solution Architect, SME for Oracle on Azure Microsoft
  2. 2. Who am I Kellyn Pot’Vin-Gorman • Database Administrator for last couple decades • DevOps Engineer at last two companies • Currently SME for Oracle on Azure at Microsoft in the Customer Architecture and Engineering Team, (CAE) • Blog at DBAkevlar.com • Yes, my husband is on the same team I’m on- get over it, we like working together. • Live in an RV, renovating a floating home in Portland, OR with my husband- no the kids aren’t invited...
  3. 3. DevOps is Difficult Enough… • No one talks about the additional challenge of silo’d teams and what it can do to a DevOps initiative. • My goal is to: • Use a real use case,(without any names ) • Discuss the cultural challenges toppled by: • Silos • Location • Project • How we were able to overcome and build out a full DevOps Solution
  4. 4. Goal To design a new product which met requirements from customers. Must be cloud and out of initial development in current datacenter Ability to deploy upwards of 200 customer environments a year
  5. 5. Environment • New product based off of some examples on-prem • Oracle database on Linux • Windows server • .Net application • No real workload to measure for migration • No previous Oracle workloads in Azure, only SQL Server • Tight timeline to deploy for customers • Varying sizes, depending on what modules customer purchased which could effect the size of the deployment greatly.
  6. 6. Deployment Architecture
  7. 7. Challenges • Teams involved were dispersed across the globe • Teams were incredibly siloed • Had their own set of tools, applications, scripting languages, etc. • Often disagreements on who’s tool or script set would be used • Current deployments were done in a very serialized steps, with sign off from each team, often manual steps to next team via email. • Lack of communication.
  8. 8. First Challenge- the workload • Isolating what a sample workload, even a small one would look like was important. • Sizing out and then attempting to identify what size each combination of modules would produce and then tagging each size. • Once sized out, created a spreadsheet to identify the needs for: • vCPU • Memory • MB/s • IOPS • Storage
  9. 9. With this Data Came up with a set of “t-shirt size” combinations: Small Medium Large Extra-large Identified what Azure VMs would meet these requirements and storage to meet IO needs.
  10. 10. Second Challenge- The Image • This is IaaS in Azure with Oracle • Linux VM image had to be chosen: • Oracle Linux was first choice • Changed to RedHat Linux after Cloud team identified need to use Azure monitoring services with Linux, which is supported by RedHat, as is Oracle. • Oracle installation of 18c was chosen • Installed Oracle binaries and as much ASM, DataGuard, etc. that can be available to an image. • Built out Perfected Image and then used Image catalog to then deploy from just as we would with a marketplace image, but at a customer’s level of availability. • Deters from someone having one-off images and having to recreate the image each time. https://docs.microsoft.com/en-us/azure/virtual-machines/linux/image-builder
  11. 11. Where They Were At: • Marketplace Image with Installed OS • DevOps team was attempting to automate OS install with PowerShell cmdlets. • Install Linux support libraries • Install Oracle, ASM, DataGuard 18c • Switch to DBA Team • Build out database • Test out build • Created script as system was built to perform same steps using BASH from CLI for database, but both install and test steps together.
  12. 12. Current Status of Automation Bash Scripts Power Shell Database Automation 1st Infrastructure DevOps Automation
  13. 13. First Impasse • Weeks to get the server deployed to the DBA team involvement. • Upon asking, discovered the steps that were involved: • Ticket system to request server • Team to deploy server required very specific information for Linux and system was designed for Windows servers, leaving many back and forth discussions. • No ETA or deadline requirements. • No image existed; the group created the VM through the Azure portal individually. • If they used any automation, it was from ARM templates to deploy a server with a market image. • Used deprecated Powershell commands to perform the deployment
  14. 14. Azure CLI Introduced • As the BASH scripts from the DBA team were quite mature: • Migrated the infrastructure deployment to the BASH script using AZ commands • These could be ported to other DevOps tools as the process matured. • Already had working examples from my other projects that I’d created. • By placing the infrastructure into the database deployment, it removed the ticketing system from the scenario. • Brought on a second DevOps team that had more mature methods and tools to work with the DBA team. • This team also came with a scrum master to assist in directing some of the project workload.
  15. 15. Current Status of Automation Bash Scripts Terraform Database Automation 2nd DevOps Automation Power Shell 1st Infrastructure DevOps Team
  16. 16. Second Impasse • Ownership of the VM Image • First infrastructure DevOps team still owned the image. • Upon request, wasn’t successful assigning ownership of image to the second DevOps team. • The First infrastructure DevOps team continued to “mature” the deprecated Powershell commands, wanting to incorporate the DBA’s BASH scripts vs. turning over the image. • Multiple meetings were required to get the right managers to assign ownership to correct team of the VM image, the automation and how to proceed forward to meet the deadline for customer engagements.
  17. 17. Current Status of Automation Bash Scripts Terraform Database Automation 2nd DevOps Automation Power Shell 1st Infrastructure DevOps Team
  18. 18. Third Impasse • Who Owns the Application Tier?? • Discovered this was still being worked on by the Product Team and the first Infrastructure DevOps Team. • Introduced the Product Team to the Second DevOps team and quickly got them working together. • Thanked the first Infrastructure DevOps team for their contributions and was able to move the application tier to the same team as the database. • Began to identify the stage of automation- all in Ansible.
  19. 19. Current Status of Automation Bash Scripts Ansible Terraform Database Automation Application Automation 2nd DevOps Automation
  20. 20. All Scripting Was Using Azure CLI • This simplified the process of taking the current ansible from the application automation and move it to Terraform. • A decision was made, since everything was in Azure, to use Azure DevOps and this simplified many of the tools, not showing preference for any one pre- existing tool. This would be done in small steps. Application automation was first to be absorbed
  21. 21. Current Status of Automation Bash Scripts Ansible Terraform Azure DevOps Database Automation Application Automation 2nd DevOps Automation Goal
  22. 22. Terraform Deployment of Oracle in Azure Demo
  23. 23. Phase II Start Deploying and Evolve • Product now has customers • Move from manual deployments to automating with current, three products: Bash scripts, called by Terraform and Azure DevOps. • Evolved bash scripts to complete automation. These scripts would be retained, but migrate the database tier steps into Azure DevOps.
  24. 24. Current Status of Automation Bash Scripts Terraform Azure DevOps Database Automation 2nd DevOps Automation Goal
  25. 25. Evolve and Eliminate • Refactor the final Terraform scripts into Azure DevOps • Create a simple Ctrl-M interface for a user to fulfill the inputs to the Azure DevOps and script arguments.
  26. 26. Current Status of Automation Terraform Azure DevOps 2nd DevOps Automation Goal User Interface Evolve
  27. 27. Final Impasse • Backups and Data Refreshes were put off from being tested • Slow response time and “IO throttling” was evident • Product was already priced out and large requirement for table level refreshes were impacting success of product • Commvault was brought in to use their snapshot and table level refresh options ONLY for small customers • Re-educate customers on how best work in the cloud • Snapshot database creation for trouble-shooting or table level restores.
  28. 28. Success of Project Siloed teams were re-aligned to work across teams. Automated and incorporated full DevOps practice for product that’s most in demand for company Product is leveraged by large companies nationwide 30 deployments this year, upwards of 300, (10X) for 2021. Profit margin increased with new Oracle in Azure cloud solution 5 teams of 120 individuals was decreased to 1 team of 7 to maintain.
  29. 29. Q&A
  30. 30. Thank you Kellyn Gorman Twitter: @DBAKevlar Email: kegorman@microsoft.com

PASS DevOps VC 09/2020

Views

Total views

232

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×