New DevOps for the DBA

© 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential.
Kellyn Pot’Vin-Gorman | Technical Intelligence Manager| October, 2017
The Revolution
DevOps to DataOps

© 2017 Delphix. All Rights Reserved. Private and Confidential.
2
Kellyn Pot’Vin-Gorman
Technical Intelligence Manager, Delphix
• Multi-platform DBA, (Oracle, MSSQL, MySQL,
Sybase, Postgres…..)
• Oracle ACE Director, (Alumni)
• Oak Table Network
• APEX Women in Technology Award, CTA 2014
• STEM education with Raspberry Pi and Python
• President, Rocky Mtn Oracle User Group
• Liaison, Denver SQL Server User Group
• DevOps author, instructor and presenter.
• Author, blogger, (http://dbakevlar.com)

3
• https://en.wikipedia.org/wiki/DevOps
The Source
DevOps is a short for DEVelopment and IT OPerationS. The term is used to
refer to a set of practices that emphasize the collaboration and communication
of both software developers and information technology (IT) professionals while
automating the process of software delivery and infrastructure changes. It aims
at establishing a culture and environment where building, testing, and releasing
software can happen rapidly, frequently, and more reliably

The Life of a DBA
 Database management, automating where able
 Provide environments and assist with projects
 Secure database environments
 Optimize data access
 Collaborate to solve business challenges

The Life of a DevOps Engineer
 Automation through scripting
 Orchestration between environments, including data sources
 Security
 Optimize projects, improving on each pass
 Collaborate and Communicate

6
Tools Empower?
• DevOps requires agility
• Agility requires more automation
and tools to ease demands
• Even the history of DevOps
includes tools
• What tools are part of DevOps for
the DBA?

7
How DBAs Risk the Change to DevOps
• Database Administrators heavily influence decisions.
• We’re sometimes identified with manual processing as a way to measure our value.
• Introduction of the cloud, especially and SaaS, (Software as a Service) has been seen as
competitive to our role.
Still, we’re essential to a successful change, or many are working around us…

8
Why Should We Master Tools?
• We Should be part of the development process
from the beginning
• We perform code reviews
• We check in and release code
• We write code, too.
• We create automation scripts.
• We monitor and optimize

Tools of the Development
• Oracle SQL Developer
• DBArtisan
• Redgate SQL Toolbelt and SQL Prompt
• SQL Automation PL/SQL Developer

10
DevOps Terms
• Build Automation
• Configuration Management
• Continuous Delivery/Rapid Deploy
• Functional Testing/Unit Testing/Test Driven
Development
• Monitoring/Alerting
• Packages/Containers/Virtualization
• Release Coordination/Orchestration
https://xebialabs.com/glossary/

11
Method to this Madness
• Identify Category/Methodology
• Identify tools
• Some tools will bridge more than one
category or methodology
• Dig into unique or interesting specifics
about the tool

12
Build Automation
• Maven
• Gradle
• Packer
• Cmake
• ANT
DBAs are a natural fit, due to their
history with scripting and automation
experience for backups, monitoring and
alerting.
Build links: Maven, Gradle, Make, Packer, MSBuild

13
MS Build
MS Build 15 came out in March 2017 and
part of Visual Studio
• No longer installed in the Global Assembly
cache.
• XML based
• Project file made up of items, properties
and tasks- and of course, targets.

Continuous Delivery
DBAs are much slower to
embrace continuous delivery.
The risk is high, with it’s “fail
first, fail fast” approach. Yet
we are its greatest asset.

15
Jenkins for CI/CD
Jenkins is an open source automation server-
• Easiest executed via a jetty servlet
• Has an easy to access URL, (commonly host:8080)
• Automation is done via individual projects.
• Has numerous plugins to support environments
• Supports multiple scripting languages, including shell, batch commands and Ant scripts.
• Supports robust revision control and compatible with version control systems like CVS and
Subversion.
• Supports notifications like qmail and sendmail.

16
Jenkins Groovy Script Example
// prepare variables for subscript execution:
String subscript = "cdmserver-monitor-instance-status.groovy";
println "executing subscript ${subscript}"
// variables to be passed to the subscript
status = (action == "start" ? "started" : "stopped")
def s = ScriptlerConfiguration.getConfiguration().getScriptById(subscript)
File scriptSrc = new File(ScriptlerManagment.getScriptDirectory(),
s.getScriptPath());

17
Jenkins Plugin for Delphix
https://github.com/delphix/delphix-jenkins-plugin

18
Configuration Management
Concept & Development Testing & Verification
Operations & Support Deployment to Production

19
Ansible
• Learn about Playbooks
• Powerful collection of steps, (aka plays)
• Plays consist of a list of tasks
• Handlers allow for remote operations
• YAML Syntax
• Security configurations based on hosts and users
• Ability to run as secondary users, (privilege
escalation)
• Huge support community and best practices

20
Example of an Ansible Call
- name: Transfer and execute a script.
hosts: server
remote_user: test_user
sudo: yes
tasks:
- name: Transfer the script
copy: src=test.sh dest=/home/test_user
mode=0777
- name: Execute the script
command: sh /home/test_user/test.sh

21
Puppet
• Review, automate, deploy and invest in all your
software.
• Control and enforce consistency across environments
and platforms.
• Share, test and enforce changes on-premise and in the
cloud.
Similar to Puppet
• Chef
• Jenkins

22
Rapid Deploy
Software Content
Enablement Service
Need or
Focus Area
All Tiers of Specific
Environment

23
Release Coordination/Orchestration
A demand for release tools and orchestration was at times a direct result to DBAs hesitation to risk
production with a release. This helped automate and bypass the DBA to get code and features into
production.
• Docker Hub
• Artifactory
• Zephyr
• Octopus Deploy
Deployment Links- RapidDeploy, ElasticBox, JuJu

24
Octopus Deploy
• Categorized as Continuous Integration Tool
• Release and deploy
• Promote to subsequent environments
• Cross tier/application, etc.

Automation and Orchestration, DB Style
• Idera SQL Admin Toolset
• Redgate DML Automation
• Automic
• CA Automation

Source Control and Repositories
• Git
• Bitbucket
• GitLab
• Subversion
• Mercurial

27
GitHub
Git is an open-source version control system that was started by
Linus Trovalds, (the creator of Linux)
• If you aren’t using it now, start. Yes, as a DBA…
• Great option to highlight your scripting work
GitHub Alternatives
• Subversion
• CVS
• Mercurial

28
Creating a GitHub Repository Project

29
Github Repository Example

30
Source/Version Control, but Data?
We all know about version
control with code, but how
are we, as DBAs doing
version control with data?
This is a concept that hasn’t
been discussed much and
DataOps is the new way of
bridging some of this
challenge.

31
The Goals of All Tools are the Same
• Shorter Delivery Cycles
• Higher Success Rate on Releases
• Shorten Time to Address Issues
• Continued Improvement in
Communication
• Increased Revenue

32
Let’s Talk Data Gravity and It’s Impact
• Data is HEAVY.
• Relational databases are heavily weighted and can
be very impacting to DevOps implementations.

33
Oh DBA, to Embrace Virtualization…
A technical approach in which users and applications do
not use physical machines, but simulated systems
running on actual, “real” hardware. Virtualization can be
used to eliminate resource usage and enable savings for
databases, network, file and application management,
along with server infrastructure.

Older Cloning Methods

Virtualize
Compress and Dedup

36
Why Database Virtualization is Different
• Many tools take considerable time to recover.
• DevOps automate an “undo” for development and
testing that includes data rewind.
• Include a self-service tool that will allow for rewind
without DBA intervention.
• Allows for data version control and DataOps, the
next step in DevOps

37
Virtualization Tools
• Delphix
• Veritas Velocity
• Oracle Thin Cloning Features
• Actifio

▶▶▶
Virtualize and Deployed▶ ▶ ▶
600GB
Virtualization 101 via Delphix
Storage Pool for Delphix
QA
DEV PATCH TEST
PRODUCTION
Database/App Tier
1 TB
1 TB
Read From Production
Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon
parameters) TEST
Read AND Write

Virtualized Database, (VDB)

Data Virtualization: Space
Savings
Delphix Virtualization EngineValidated Sync Target
Dozen Virtual Database Copies,
(VDBs)

41
The Future is Now- Containers/Data Pods
Containers offer the ability to isolate application code
and/or the whole infrastructure stack into a package able
entity to ease deployment, even from the same kernel.
This is a powerful tool for DevOps to ease deployment for
complex tiers, applications and multiple data stores.
Data Pods is the next step, where a pod is created from
virtualized environments.

42
Docker
• Virtualization is important,
this is OS level
• Resource level isolation at
the kernel level
• Technology such as
OverlayFS, allows for
containers, which removes
overhead of other
virtualization technologies.
Docker Alternatives:
• Open Container
Initiative, (OCI)
• Kubernetes
• CoreOS
• Apache Mesos

Group Environments into Data Pods

Why Create Data Pods
• Development is done by project or feature vs. tier or
product.
• With the introduction to the cloud, the business
requires those that can do more at a global scale.
• Ease of management and maintenance
• Provide more value to the business

45
Data Pods
Database
Unstructured
Data
Application

Data Pods for Ease of Delivery
Create “Container”
Rsync (UNIX/Linux)
Robocopy (Windows)
Reporting
NFS
iSCSI
Development
NFS
iSCSI
Testing
NFS
iSCSI
Delphix Virtualization Engine
Same Storage Size as Production
Deployed using Jenkins

47
How Does This Improve Agile?
Finance
HR
CRM

48
Timeline in a Project
Waterfall- 5 Releases
January February March April
Agile with Virtualization- 12 Releases
January February March April
Database Refresh
Development Production Release
Testing Cycle
The Goal of
Agile
Data Holdup to Development
Cycle

Data Versioning
• Delphix Jet Stream
• Redgate SQL Data Compare
• Oracle SQL Developer

50
Branching and Bookmarking
• The ability to mark each iteration of development
with a bookmark
• Simplify to lock and deliver while testing a
consistent image via a virtual database, (VDB)
• If a something goes wrong, the ability to
“bookmark”, (and subsequent snapshot) to
deliver to development to address.

51
DevOps and Source Control

Administration Ease

Self-Service
Release 1.2 Release 1.3 Branch 1.3
Branch 1.1

Methodologies and Collaboration

55
Scrum
https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile

56
Lean/Kanban

57
Extreme Programming

58
Crystal
Alistair Cockburn said, “Crystal is a family of human-powered, adaptive,
ultra light, ‘stretch-to-fit’ software development methodologies.”
• People are the most important aspect of Crystal.
• Development is self-sufficient and self-organizing, (very different from
Scrum)
• Very lightweight
• Very little documentation, overhead management or reporting.

59
Feature Driven Development

60
Collaboration Practices can Mutate
• Methods- Scrum, KanBan, Extreme Programming, Continuous
Delivery
• Official hybrid approaches like Scrumban and KanPlan
• Numerous other combinations created from any of the
originals.

• All IaaS solutions provide encryption in-flight and encryption at-
rest
• But encryption doesn’t protect data as much as it needs to be .
• Europe already requires data masking, not just data encryption for
any confindential data, (GDPR):
• http://ec.europa.eu/justice/data-protection/article-
29/documentation/opinion-recommendation/files/2014/wp216_en.pdf
Confidential data

• Encryption is reversible data obfuscation, which is very different
from masking data.
• Data masking is non-reversible.
• It solves the issue at the data level.
• Is authentication and authorization in non-production in
compliance with security goals?
• All organizations will soon need to review if critical data in non-
production environments be accessible to developers, testers and
users.
Confidential data

Encryption and Masking
• Optim
• Delphix Masking
• Dataveil
• CA Test Data Manager
• Oracle Encryption and Masking

• Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information
renders it useless from a security standpoint
• Resolves both the technical and personal responsibility issue.
• The data can be masked before it moves to non-production, removing
unnecessary risk.
Why Masking is Part of the Answer

Confidential data
Production
Non-Production
Exposure

Confidential data
Production
Non-Production
Exposure
Encryption
Masking
Solution

Auditing Solutions
• Idera SQL Secure and SQL Compliance
Manager
• Redgate SQL Source Control
• Oracle Auditing Features

Data Causes Friction
• If those managing the data sources aren’t included and working well with
those needing that data to produce features and products, friction is the
result.
• DBAs are consistently working with archaic utilities that guarantee
outcome, rarely speed of results.
A DBA is only as good as their last backup.

What is Data Gravity?
Data Gravity is the ability for data to attract
applications, services, etc. As with the laws
of physical gravity, data, due to its mass,
will attract all else that has less mass.

When you consider…
An estimate of a 10% increase in data
accessibility will result in more than $65 million
in additional net income, you can see why data
gravity is a big deal.

How much data are we talking about?
• More data has been created in just the last
two years than the previous history of
humanity
• 1.7MB of new information created every
second per human on the planet.
• Introduction of big data often has same
development pain points.

74
Know Your Tools
Incredible variation of tools and knowing tool classifications. Use
the right tool for the job and recognize that each tools has its
own “flavor.”

Remove the Roadblocks
• Remove the data as the roadblock.
• Use virtualization and containers to simplify.
• Self-service portals for developers and testers to
refresh and develop they way they do in an agile
environment.
• Tools with an interface made for Agile and
Development data at its focus.

76
Review- DBA Focus Areas
• Update skills, (scripting, monitoring, automating) to remove bottlenecks
• Enhance monitoring utilities that take infrastructure and development tools into
overall view
• Recognize a need to become more skilled in scripting and become
knowledgeable of DevOps tools.
• Data is EVERYWHERE! Recognize the integral part Database Administrators
play in DevOps and learn more about the next generation- DataOps!
Don’t fear the future, embrace it.
DBA
DevOpsSkills

77
The Periodic Table of DevOps Tools
https://xebialabs.com/periodic-table-of-devops-tools/

78
Read- The Phoenix Project and the DevOps Handbook!
Delphix with SQL Server- the Basics
https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-
sources/managing-sql-server-environments/overview-of-setting-up-sql-server-
environments
Delphix Upgrade Workflow: https://community.delphix.com/delphix/topics/tip-of-the-day-
upgrading-a-sql-server-dsource
Upgrading the Dsource after an Upgrade: https://docs.delphix.com/docs/delphix-
administration/sql-server-environments-and-data-sources/virtualizing-databases-using-
delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-topics/upgrading-
a-dsource-after-a-sql-server-upgrade
Delphix in the Cloud
https://www.delphix.com/solutions/cloud-migration-virtual-data
References and Tips

© 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential.
Kellyn Pot’Vin-Gorman
Technical Intelligence Manager
kellyn@delphix.com
http://dbakevlar.com

New DevOps for the DBA

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to New DevOps for the DBA

Similar to New DevOps for the DBA (20)

More from Kellyn Pot'Vin-Gorman

More from Kellyn Pot'Vin-Gorman (20)

Recently uploaded

Recently uploaded (20)

New DevOps for the DBA

Editor's Notes