Best Practices in Disaster Recovery Planning and Testing

CONFIDENTIAL – DO NOT DISTRIBUTE. Copyright © 2014 Axcient, Inc. All Rights Reserved.
Best Practices in
DR Planning and Testing
Paul F Kirvan, CISA, FBCI
Independent BC/DR Consultant
Member of the Board and Secretary
The Business Continuity Institute USA Chapter

Agenda
1. Introduction
2. Plan Components
3. Mistakes and Pitfalls to Avoid
4. DR Technology Options
5. Tips for Planning DR Tests
6. Summary
7. Q&A

Why is DR Important?
• Accepted way to ensure that critical data, IT systems and
networks can be recovered in an emergency
• Ensures that corporate business objectives can be
achieved, despite a disruption
• Increasingly accepted by management as a strategy for
keeping the business operational

Quick Poll
Do you currently have a Disaster Recovery plan in place?
a. Yes, I have a comprehensive DR plan at my company
b. Yes, but needs more work
c. No, but would like to get one ready
d. No, and have no plans to create one

Quick Poll
Do you currently have a Disaster Recovery plan in place?

What Do You Need?
A good disaster recovery plan needs:
• Support from senior management
• Funding approved by management
• Structured plan framework
• Access to qualified staff
• Access to relevant information
• Documentation and testing

What’s Your Goal with the Plan?
Build disaster recovery plans and associated
documentation based on a structured framework that is
consistent with good practices and standards.

DR Plan Activities
• Data gathering, interviews, analysis
• DR standards and good practice, emergency response
procedures, data backup and recovery procedures,
system recovery and restart processes, plan templates
• Tests to ensure that plan procedures and processes work
as designed
• Maintenance activities to keep plans up to date and
accurate

Standards and Good Practice
• Standards – NFPA 1600:2010; ISO 24762:2008; ISO
27031:2011; NIST 800-34
• Regulations – NASD 2510/3520; NYSE 446
• Good Practice – BCI Good Practice Guidelines, FFIEC
Handbook
• Corporate DR policies

What You Need to Identify
• DR objectives of the systems, networks or other IT assets
(e.g., uninterrupted operation, max downtime 4.0 hrs)
• Risks and/or threats to the achievement of the DR
objectives
• Define and document the processes and procedures
needed to recover and reactivate the IT assets
• Identify preventive measures to mitigate DR risks to an
acceptable level

Plan Components
The following pages list the typical components found in an IT
disaster recovery plan. There may be some variations based on
your organization’s requirements, but generally the following items
should be included.

Plan Components
A good DR plan usually includes the following
components:
• Company DR policies
• DR plan documents
• Business impact analysis reports
• Risk assessment reports
• Exercise results
• IT DR procedures (in the plan)
• Supporting documents (e.g., data backup process, off-site storage
process, vendor contracts, diagrams, maintenance contracts, training
plans)

Plan Components
 If there’s an existing plan, use it as a starting point
 Define plan scope, purpose, authority
 Define a policy statement
 Define management approval and funding
 Identify planning and response teams
 Identify critical IT resources
 Identify risks and their impact on IT assets
 Determine recovery time objectives (RTOs)
 Determine recovery point objectives (RPOs)

Plan Components
 Preventive controls (e.g., backup power)
 Response and recovery strategies
 Data backup and recovery methods, compared to existing data
storage and retrieval procedures
 Potential use of alternate IT sites, e.g., a backup data center,
collocated data center, the cloud
 Potential use of hot sites, cold sites
 Potential use of alternate work (e.g., office) sites, and the technology
needs for those sites

Plan Components
 Process for equipment replacement
 Process for obtaining spare parts
 Staff roles and responsibilities in a disaster
 Event notification procedures
 Damage assessment procedures
 Process and criteria for plan activation
 Identify who is authorized to declare a disaster
 Recovery / failover procedures
 System restart / failback procedures
 Resumption of business procedures

Plan Components
 Step-by-step procedures for recovery of
 IT operations
 Desktop systems
 Data
 Hardware
 Operating systems
 Applications
 Databases
 LANs and WANs
 Voice and VoIP systems
 Servers

 Step-by-step procedures for recovery of
 Web sites
 Mainframes
 Distributed systems
 Wireless technology
 Specialized systems
 Information security
 User access
 Physical security
 Vital records
Plan Components

Plan Components
 Step-by-step procedures for
 Alerting first responder organizations
 Alerting family members
 Alerting primary/alternate vendors
 Alerting staff, senior management
 Alerting clients, stakeholders
 Escalating recovery efforts
 Help desk support
 Using call trees
 Activating automated notification systems
 Activating conference bridges

Plan Components
 Links to emergency management and incident response plans,
business continuity plans
 Process for exercising DR plans
 Process for creating a DR awareness program
 Process for DR team training
 Process for DR training of employees
 Process for communicating with the media
 Designated company spokesperson

The next set of slides provides a sample DR plan
outline. While most plans will be different, this outline
includes the most common plan components and is
consistent with standards and good practice.
Plan Components

Plan Components
DR Plan Outline - 1
• Revision History
• Table of Contents
• Emergency Response Actions
‐ Assembly Points
‐ Emergency Call-in Number
‐ Key Personnel Contact Info
‐ Notification Calling Tree
‐ External Contacts
‐ External Contacts Calling Tree

Plan Components
DR Plan Outline - 2
• Policy Statement
• Objectives
• Plan Overview
• Plan Updating
• Plan Documentation Storage
• Backup Strategies
• Emergency Response
‐ Plan Triggering Events
‐ Assembly Points

Plan Components
DR Plan Outline - 3
• Activation of Emergency Management Team
• Technology Services Team
• Emergency Alert, Escalation and DRP Activation
• DR Procedures and Actions
‐ Contact with Employees
‐ Backup Staff
‐ Recorded Messages / Updates
‐ Alternate Recovery Facilities / Hot Site

Plan Components
DR Plan Outline - 4
• Personnel and Family Notification
• Communications with Media, Key Stakeholders
• Media and Key Stakeholders Contact
• Media and Key Stakeholders Team
• Rules for Dealing with Media, Key Stakeholders
• Insurance Requirements
• Financial and Legal Issues
‐ Financial Assessment
‐ Financial Requirements

Plan Components
DR Plan Outline - 5
• Legal Actions
• DR Plan Exercising
• Appendix A – Technology DR Plans
‐ Production Environment
‐ Private Cloud Environment
‐ Internal IT Environment at HQ
‐ Local Area Network (LAN)
‐ Voice over IP (VoIP) System
‐ Remote Connectivity / VPN

Plan Components
DR Plan Outline - 6
• Appendix B – Forms and Reports
‐ Management of DR Activities Forms
‐ Communications and Reporting Form
‐ Disaster Recovery Incident Recording Form
‐ Disaster Recovery Activity Report Form
‐ Mobilizing the Disaster Recovery Team Form
‐ Mobilizing the Business Recovery Team Form
‐ Monitoring Business Recovery Progress Form
‐ Business Process/Function Recovery Form

Mistakes and Pitfalls to Avoid
(the not-so-obvious things)

 Failure to obtain senior management support
 No budget (i.e., no plan)
 Lack of upfront research (e.g., risks, RTO/RPO)
 Lack of documentation (e.g., assume native knowledge will be
available)
 No step-by-step procedures (assume you know what to do first,
second, who to call, etc.)
 No plan testing (e.g., rolling the dice)
 No regular plan reviews and updates

 No DR team training (nobody knows what to do)
 Assume that IT staff knows what to do
 Assume that IT staff will be available in an emergency
 Assume that backup and recovery procedures will work when needed
 Assume that systems and networks will work properly when in backup
or recovery mode
 Assume that backed-up data will be available when needed

DR Technology Options

Quick Poll
What technologies are you currently using for Disaster Recovery?
a. Local backup to disk or tape
b. Cloud backup
c. Server replication (either locally or to off-site facility)
d. Hybrid technology with local and cloud protection
e. Collocation of data center
f. Other

Quick Poll
What technologies are you currently using for Disaster Recovery?

 Data backup and recovery to an alternate site, e.g., backup data
center
 Application backup and recovery to an alternate site, e.g., backup
data center
 Off-site data storage using a third-party firm
 Redundant components, e.g., servers, storage devices, network
components
 Diversely run networks, e.g., alternate service using a different carrier
and different paths
 System failover / failback technologies to rapidly recover and restart
disrupted systems

Current Process New Cloud Options
Application backup and recovery
to an alternate site or data center
Application backup and recovery to
the cloud
File/data/database backup and
recovery to an alternate site /
data center
File/data/database backup and
recovery to the cloud
Server backup and recovery via
failover to an alternate site / data
center
Server backup and recovery via
failover to the cloud
Cloud-based solutions have become very popular as
primary and alternate backup and recovery strategies.

Current Process New Cloud Options
Recover the minimum
configuration of servers,
applications, network resources if
it’s necessary to relocate to an
alternate office site
“Office virtualization”, which has
server failover, access to IP
addresses and Active Directory in
the cloud; this means rapid office
recovery and minimum downtime
Conduct DR plan tests using a
local, on-site environment or
alternate backup data center
resource
Streamline DR tests using a cloud-
based and automated DR testing
environment
Traditional DR activities can be automated and streamlined
to encourage more testing and reduce risks from disruptions.

Tips for Planning DR Tests

Quick Poll
How often do you test your DR plan and/or the ability to recover from a
disaster?
a. I don’t test
b. Once a year
c. Two to four times a year
d. Every month
e. Not as often as I should

Quick Poll
How often do you test your DR plan and/or the ability to recover from a
disaster?

1. Decide what you want to test, e.g., data recovery, system failover to
a backup site
2. Determine if production systems will be negatively affected during
the test
3. Conduct the test in a non-production environment, e.g., R&D
4. Select test participants and alternates
5. Document step-by-step procedures for performing the test
6. Secure a conference room or suitably equipped work area for the
test
7. Schedule the test so as not to interfere with production activities

8. Notify all IT teams and groups of the test at least two weeks in
advance
9. Include a scribe / timekeeper
10. (If possible) Conduct a dry run to validate that the test procedures
will/should work
11. Complete the test, keeping notes of all actions performed, time
needed for each activity
12. Prepare an after-action report summarizing what worked, what didn’t
work and lessons learned
13. Update the DR plan based on test results

Summary
 Develop and document a plan .. follow it
 Senior management supports the plan
 Policies, procedures, metrics
 Document, document, document
 Test, test, test
 Maintenance and regular review

About Axcient
Leader in Recovery-as-a-Service
One SaaS Platform
Backup Disaster
Recovery
Business
Continuity
WAN
Optimization
Dedupe
vs.
Rapid Recovery
Physical & Virtual Application
Continuity
Cloud
Virtualization
True Cloud
Platform

For more information, visit axcient.com or call 800 715.2339
@Axcient linkedin.com/company/axcient axcient.com/facebook
Paul Kirvan, CISA, FBCI
Phone (908) 902-2586
Email pkirvan@msn.com

Best Practices in Disaster Recovery Planning and Testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices in Disaster Recovery Planning and Testing

Similar to Best Practices in Disaster Recovery Planning and Testing (20)

Recently uploaded

Recently uploaded (20)

Best Practices in Disaster Recovery Planning and Testing