SpringOne Platform 2019 Session Title: Singing the "Migration Song" with No Downtime Speaker: Ronak Banka, Pivotal Youtube: https://youtu.be/sgLt2hitL_I

1. Singing the ‘Migration Song’
with no downtime
Ronak Banka

2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
$whoami
Solution Architect @Pivotal
@ronakbanka
6 Years working with Cloud Foundry
2

3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
How many of you have been in this
Situation??

6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
WHY Migrate?
• Data center decommissioning
• Hardware refresh
• Additional capacity
6

7. Platform Team: SO!! What all stuff we have
to Migrate??
Infrastructure Services team: EVERYTHING!!
#springone@s1p

8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Impact Radius

9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
End Goal for Migration
Customer was looking to migrate All PCF foundations, across DEV, SIT,
QA and 2 Active/Active PROD ones, from Intel-based hosts to new AMD-
based hosts.
9

10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Goals
● Migrate foundations one by one
● No downtime introduced to end-users
● Minimal downtime to developers
1
0

11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Known Facts/Constraints
● Timeline was a big constraint
● Live vMotion was not possible from Intel to AMD hosts
1
1

12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Approaches considered
1. Backup and Restore
Impact: Longer downtime impacting developers.
1. Share datastore across clusters
Impact: Different storage ops practise, hard to pass change control
1. Storage vMotion + BOSH commands
Impact: Need to shutdown PCF & replace new Cluster names with old ones
1. Set new Cluster as another AZ in Foundation
Impact: Mix of migrating backup and restore + migrating vms across AZ’s
1
2

13. Selected Approach
Storage vMotion +
BOSH Commands
#springone@s1p

14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Existing Cluster 1 (Active)
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
Existing Cluster 2 (Active)
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
Global Load Balancer (DNS)

15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Existing Cluster 1 (Active)
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
New Cluster 1Existing Cluster 2 (Active)
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
Global Load Balancer (DNS)

16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Existing Cluster 1 (Active)
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
New Cluster 1Existing Cluster 2 (Active)
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
Global Load Balancer (DNS)

17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM
New Cluster 1

18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
BOSH hard stop on VMs with no persistent disks (deletes the VM)
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1

19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
BOSH stop on VMs with persistent disks (keeps the VM)
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1

20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
vMotion VMs with persistent disks over to new cluster
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1
VM
Disk
VM
Disk

21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Stop BOSH
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1
VM
Disk
VM
Disk

22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Configuration Changes?
● Cluster name, Network names & Datastore names of new cluster were changed
to match the old one
2
2

23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
vMotion BOSH over to new cluster
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1
VM
Disk
VM
Disk
BOSH + OpsMan

24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Start BOSH
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1
VM
Disk
VM
Disk
BOSH + OpsMan

25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Use BOSH to start all VMs
Existing Cluster 1
Migration sequence in high level
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
New Cluster 1
PAS
VM
Disk
VM
Disk
BOSH + OpsMan
VM VM

26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Migration sequence in high level
New Cluster 1
PAS
VM
Disk
VM
Disk
BOSH + OpsMan
VM VM
Global Load Balancer (DNS)
Existing Cluster 2 (Active)
BOSH + OpsMan
PAS
VM
Disk
VM
Disk
VM VM

27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Overall Steps
1. Execute bosh stop --hard on VMs with no persistent disks (on existing cluster)
2. Execute bosh stop on remaining VMs with persistent disks (on existing cluster)
3. VMotion remaining VMs with persistent disks from existing cluster to new cluster
4. Stop BOSH Director & Ops Manager VM
5. VMotion BOSH Director & Ops Manager from existing cluster to new cluster
6. Start up BOSH Director and
7. Rename cluster and datastore in new cluster
8. Start all deployments, starting with PAS VMs, followed by Service Instances
2
7

28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Retro
What worked
● No End User Impact
● Inter-dept. coordination
● Total time
○ Production : 2 x 13 (70:30) Hrs
○ Non Prod: 2 x 15 (70:30) Hrs
● Developer downtime: 0 Min
● Foundation Size
○ Production: 2 x 55 Diego cells
○ Non Prod: 2 x 80 Diego cells
2
8
What we learned
● Stretch cluster
● Cross arch migration needs
careful planning
● Prod and non-prod parity is must
● Test vMotion and XvMotion
thoroughly
● VM’s with persistent disks are
notorious (ex: ELK)

29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
#KUDOS
PEZ team at Pivotal
👏

30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
THANK YOU!!
Questions?