本議題是去年 ModernWeb'19 「Progressive Deployment & NoDeploy」的延伸。雖然已提倡 Testing in Production 多年,但至今願意或敢於實踐的團隊並不多,背後原因多是與文化及態度有些關係。
此次主要分享推廣過程中遇到的苦與甜,以及自己親力操刀幾項達成 Testing in Production, Deploy on Fridays 成就的產品。
4. 4/111
Deploy on ... Fridays
No
NoNoNO
Don't
Do
that
Image Credit : https://kknews.cc/pet/pqbxb8.html
Font Credit : https://www.dafont.com/theme.php?cat=109 (Defused)
技術在我們手上 世界就在我們手上
45. 45/111
Staging
Staging 環境必須與 Production 環境幾乎
相同,這十分困難,否則測試就不準了。
Trying to mirror your staging
environment to production is a fool’s
errand. Just give up.
嘗試將 Staging 視為 Production 是一個愚蠢的事。放棄吧。
Partial Credit : https://thenewstack.io/honeycombs-charity-majors-go-ahead-test-in-production/
52. 52/111
Chaos Engineering @ Netflix
這些是需要整合測試的故障模式示例,因為它們涉及服務之間的互動。在某
些情況下,可能只能在 Production 進行完全整合測試。在 Netflix ,根本不
可能完全重現整個架構並運行端到端測試。
當可以在 Testing 環境中重現整個系統時,我們仍然相信需要在 Production
運行實驗,因為在 Testing 環境中永遠不可能完全重現系統的所有方面。總
是存在差異,例如合併客戶端與真實客戶端相比的行為方式或 DNS 配置問
題。
Credit : Chaos Engineering IEEE Software (2016-05-05).pdf
Testing in Production
53. 53/111
技術在我們手上 世界就在我們手上
“
If you aren't testing in prod
you aren't testing in reality
”
如果不在正式環境測試,
你的測試就不會是真實的
Credit : https://twitter.com/mipsytipsy/status/1147863838647185408
Image Credit : https://store.steampowered.com/app/561970/REALITY/
54. 54/111
技術在我們手上 世界就在我們手上
“
Dev will match Prod,
if you develop on production
”
如果在正式環境上進行開發,
開發環境就會與正式環境相稱
Credit : Blue is the new green (2019-03-26).pdf
Image Credit : https://tw.voicetube.com/videos/62711
55. 55/111
Your aim won’t be perfect,
control over the blast radius.
你的瞄準並不完美,
請控制好爆炸半徑。
Image Credit : https://www.behance.net/gallery/57558081/FREE-BUSINESS-ICONS
Progressive Deployment
61. 61/111
Code Test Acceptance Deploy Release
Continuous Integration / 持續整合
Continuous Delivery / 持續交付
Continuous Deployment / 持續部署
Continuous Deployment + Feature Flag / 持續部署 + 功能標誌
技術在我們手上 世界就在我們手上
Operate
DevOps / SRE
Code Push ➡
Deployment
Developer
Production
Server User
Feature Flaged ➡
Release
62. 62/111
Feature Flag
01 所有程式皆為部署,並可控制開關,減少整合問題
All code is deployed, but control exposure, reduces integration issue.
02 控制細緻度可至特定層級或是開發分支
Flags provide runtime control down to individual user.
03 開發者可以增加或刪除功能,無需再次部署
Users can be added or removed with no redeployment
04 暗啟動
Enables dark launch
Credit : Progressive Deployment, Experimentation, Multitenancy, No Downtime, Cloud Security (2018-02-23).pdf
Image Credit : https://opensource.com/article/18/2/feature-flags-ring-deployment-model
63. Cost in Time
時間成本
hours
mins ~ hour
secs
~
mins
Applicability
適用性
2%
8%
90%
Toggle
Feature Flag
ReDeploy
Previous Version
Rollback
Revert Commit
Incident Recovery
事件恢復
65. 65/111
技術在我們手上 世界就在我們手上
01 Feature Flag with Story Mapping
功能標誌及使用者故事對照
02 Progressive Deployment
漸進式部署
03 Data(base) Synchronization Strategy
資料 ( 庫 ) 同步策略
04 Pull Requests to Pair/Mob Programming
從 Pull Requests 轉為 Pair/Mob Programming
05 Service Mesh
服務網格
Testing in Production, Deploy on Fridays
66. 66/111
技術在我們手上 世界就在我們手上
01 Feature Flag with Story Mapping
功能標誌及使用者故事對照
02 Progressive Deployment
漸進式部署
03 Data(base) Synchronization Strategy
資料 ( 庫 ) 同步策略
04 Pull Requests to Pair/Mob Programming
從 Pull Requests 轉為 Pair/Mob Programming
05 Service Mesh
服務網格
Testing in Production, Deploy on Fridays
67. 67/111
技術在我們手上 世界就在我們手上
Group = Beta
Group = Normal
If ( group = “beta” ) {
...
}
else {
...
}
Beta
Normal
By User Group
TimeURIIP AddressUser Group
68. 68/111
技術在我們手上 世界就在我們手上
IP = o.o.o.o
IP = x.x.x.x
If ( IP = “o.o.o.o” ) {
...
}
else {
...
}
Beta
By IP Address
TimeURIIP AddressUser Group
Normal
69. 69/111
技術在我們手上 世界就在我們手上
URI ~ ?flag=1
If ( URI ~ “flag=1” ) {
...
}
else {
...
}
Beta
Normal
By URI
TimeURIIP AddressUser Group
70. 70/111
技術在我們手上 世界就在我們手上
URI ~ 20201005
If ( URI ~“20201005”) {
...
}
else {
...
}
Beta
Normal
By Time
TimeURIIP AddressUser Group
71. 71/111
技術在我們手上 世界就在我們手上
Story Mapping
Code Test Acceptance Deploy Release Done
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
#18
Filter By
Group
#22
German
i18n
72. 72/111
技術在我們手上 世界就在我們手上
Story Mapping
Code Test Acceptance Deploy Release Done
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
Flag#3
#18
Filter By
Group
Flag#4
#22
German
i18n
Flag#5
73. 73/111
技術在我們手上 世界就在我們手上
Story Mapping
Code Test Acceptance Deploy Release Done
#19
Fix
Login Bug
#20
Fix
Email Bug
#21
Perf.
Tuning
#17
Sort By
Date
Flag#3
#18
Filter By
Group
Flag#4
#22
German
i18n
Flag#5
#17
Sort By
Date
Flag#3
75. 75/111
01 提交程式至主幹可能會影響其他功能
Commit code on trunk may affects other features.
02 未完成的功能可能也會部署,如果配置有誤可能將未完成的功能開啟
Unfinished features may be deployed, and turned on if the configuration is incorrect.
Image Credit : https://opensource.com/article/18/2/feature-flags-ring-deployment-model
Feature Flag
86. 86/111
技術在我們手上 世界就在我們手上
Image Credit : https://jaxenter.com/deployment-best-practices-172662.html
Code Push ➡
5 mins
⬅ Data Sync
? mins
Code Push ➡
15 mins
Code Push ➡
15 mins
⬅ Data Sync
? mins
⬅ Data Sync
? mins
Easy
Hard