Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016
•
3 likes•844 views
A General Framework for Communication-Efficient Distributed Optimization: Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In light of this, we propose a general framework, CoCoA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. Our framework enjoys strong convergence guarantees and exhibits state-of-the-art empirical performance in the distributed setting. We demonstrate this performance with extensive experiments in Apache Spark, achieving speedups of up to 50x compared to leading distributed methods on common machine learning objectives.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016
Similar to Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016 (20)
More from MLconf
More from MLconf (20)
Recently uploaded
Recently uploaded (20)
Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016
- 1. CoCoA A General Framework for Communication-Efficient Distributed Optimization Virginia Smith Simone Forte ⋅ Chenxin Ma ⋅ Martin Takac ⋅ Martin Jaggi ⋅ Michael I. Jordan
- 2. Machine Learning with Large Datasets
- 3. Machine Learning with Large Datasets
- 4. Machine Learning with Large Datasets image/music/video tagging document categorization item recommendation click-through rate prediction sequence tagging protein structure prediction sensor data prediction spam classification fraud detection
- 6. Machine Learning Workflow DATA & PROBLEM classification, regression, collaborative filtering, …
- 7. Machine Learning Workflow DATA & PROBLEM classification, regression, collaborative filtering, … MACHINE LEARNING MODEL logistic regression, lasso, support vector machines, …
- 8. Machine Learning Workflow DATA & PROBLEM classification, regression, collaborative filtering, … MACHINE LEARNING MODEL logistic regression, lasso, support vector machines, … OPTIMIZATION ALGORITHM gradient descent, coordinate descent, Newton’s method, …
- 9. Machine Learning Workflow DATA & PROBLEM classification, regression, collaborative filtering, … MACHINE LEARNING MODEL logistic regression, lasso, support vector machines, … OPTIMIZATION ALGORITHM gradient descent, coordinate descent, Newton’s method, …
- 10. Machine Learning Workflow DATA & PROBLEM classification, regression, collaborative filtering, … MACHINE LEARNING MODEL logistic regression, lasso, support vector machines, … OPTIMIZATION ALGORITHM gradient descent, coordinate descent, Newton’s method, … SYSTEMS SETTING multi-core, cluster, cloud, supercomputer, …
- 11. Machine Learning Workflow DATA & PROBLEM classification, regression, collaborative filtering, … MACHINE LEARNING MODEL logistic regression, lasso, support vector machines, … OPTIMIZATION ALGORITHM gradient descent, coordinate descent, Newton’s method, … SYSTEMS SETTING multi-core, cluster, cloud, supercomputer, … Open Problem: efficiently solving objective when data is distributed
- 14. Distributed Optimization reduce: w = w ↵ P k w
- 15. Distributed Optimization reduce: w = w ↵ P k w
- 16. Distributed Optimization “always communicate” reduce: w = w ↵ P k w
- 17. Distributed Optimization ✔ convergence guarantees “always communicate” reduce: w = w ↵ P k w
- 18. Distributed Optimization ✔ convergence guarantees ✗ high communication “always communicate” reduce: w = w ↵ P k w
- 19. Distributed Optimization ✔ convergence guarantees ✗ high communication “always communicate” reduce: w = w ↵ P k w
- 20. Distributed Optimization ✔ convergence guarantees ✗ high communication average: w := 1 K P k wk “always communicate” reduce: w = w ↵ P k w
- 21. Distributed Optimization ✔ convergence guarantees ✗ high communication average: w := 1 K P k wk “always communicate” “never communicate” reduce: w = w ↵ P k w
- 22. Distributed Optimization ✔ convergence guarantees ✗ high communication ✔ low communication average: w := 1 K P k wk “always communicate” “never communicate” reduce: w = w ↵ P k w
- 23. Distributed Optimization ✔ convergence guarantees ✗ high communication ✔ low communication ✗ convergence not guaranteed average: w := 1 K P k wk “always communicate” “never communicate” reduce: w = w ↵ P k w
- 24. Mini-batch
- 25. Mini-batch
- 26. Mini-batch reduce: w = w ↵ |b| P i2b w
- 27. Mini-batch reduce: w = w ↵ |b| P i2b w
- 28. Mini-batch ✔ convergence guarantees reduce: w = w ↵ |b| P i2b w
- 29. Mini-batch ✔ convergence guarantees ✔ tunable communication reduce: w = w ↵ |b| P i2b w
- 30. Mini-batch ✔ convergence guarantees ✔ tunable communication a natural middle-ground reduce: w = w ↵ |b| P i2b w
- 31. Mini-batch ✔ convergence guarantees ✔ tunable communication a natural middle-ground reduce: w = w ↵ |b| P i2b w
- 33. Mini-batch Limitations 1. ONE-OFF METHODS
- 34. Mini-batch Limitations 1. ONE-OFF METHODS 2. STALE UPDATES
- 35. Mini-batch Limitations 1. ONE-OFF METHODS 2. STALE UPDATES 3. AVERAGE OVER BATCH SIZE
- 38. Mini-batch Limitations 1. ONE-OFF METHODS 2. STALE UPDATES 3. AVERAGE OVER BATCH SIZE
- 39. Mini-batch Limitations 1. ONE-OFF METHODS Primal-Dual Framework 2. STALE UPDATES Immediately apply local updates 3. AVERAGE OVER BATCH SIZE Average over K << batch size
- 40. 1. ONE-OFF METHODS Primal-Dual Framework 2. STALE UPDATES Immediately apply local updates 3. AVERAGE OVER BATCH SIZE Average over K << batch size CoCoA-v1
- 41. 1. Primal-Dual Framework PRIMAL DUAL≥
- 42. 1. Primal-Dual Framework PRIMAL DUAL≥ min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) #
- 43. 1. Primal-Dual Framework PRIMAL DUAL≥ min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) # Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 44. 1. Primal-Dual Framework PRIMAL DUAL Stopping criteria given by duality gap ≥ min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) # Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 45. 1. Primal-Dual Framework PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice ≥ min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) # Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 46. 1. Primal-Dual Framework PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice Default in software packages e.g. liblinear ≥ min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) # Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 47. 1. Primal-Dual Framework PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice Default in software packages e.g. liblinear Dual separates across machines (one dual variable per datapoint) ≥ min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) # Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 49. 1. Primal-Dual Framework Global objective:
- 50. 1. Primal-Dual Framework Global objective: Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 51. 1. Primal-Dual Framework Global objective: Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 52. 1. Primal-Dual Framework Global objective: Local objective: Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 53. 1. Primal-Dual Framework Global objective: Local objective: max ↵[k]2Rn 1 n X i2Pk `⇤ i ( ↵i ( ↵[k])i) 1 n wT A ↵[k] 2 1 n A ↵[k] 2 Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 54. 1. Primal-Dual Framework Global objective: Local objective: max ↵[k]2Rn 1 n X i2Pk `⇤ i ( ↵i ( ↵[k])i) 1 n wT A ↵[k] 2 1 n A ↵[k] 2 Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 55. 1. Primal-Dual Framework Global objective: Local objective: Can solve the local objective using any internal optimization method max ↵[k]2Rn 1 n X i2Pk `⇤ i ( ↵i ( ↵[k])i) 1 n wT A ↵[k] 2 1 n A ↵[k] 2 Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 56. 2. Immediately Apply Updates
- 57. 2. Immediately Apply Updates for i 2 b w w ↵riP(w) end w w + w STALE
- 58. 2. Immediately Apply Updates for i 2 b w w ↵riP(w) end w w + w STALE FRESH for i 2 b w ↵riP(w) w w + w end w w + w
- 59. 3. Average over K
- 60. 3. Average over K reduce: w = w + 1 K P k wk
- 62. Can we avoid having to average the partial solutions? COCOA-v1 Limitations
- 63. Can we avoid having to average the partial solutions? ✔ COCOA-v1 Limitations
- 64. Can we avoid having to average the partial solutions? ✔ [CoCoA+, Ma & Smith, et. al., ICML ’15] COCOA-v1 Limitations
- 65. Can we avoid having to average the partial solutions? L1-regularized objectives not covered in initial framework ✔ [CoCoA+, Ma & Smith, et. al., ICML ’15] COCOA-v1 Limitations
- 66. Can we avoid having to average the partial solutions? L1-regularized objectives not covered in initial framework ✔ [CoCoA+, Ma & Smith, et. al., ICML ’15] COCOA-v1 Limitations
- 70. L1 Regularization Encourages sparse solutions
- 71. L1 Regularization Encourages sparse solutions Includes popular models: - lasso regression - sparse logistic regression - elastic net-regularized problems
- 72. L1 Regularization Encourages sparse solutions Includes popular models: - lasso regression - sparse logistic regression - elastic net-regularized problems Beneficial to distribute by feature
- 73. L1 Regularization Encourages sparse solutions Includes popular models: - lasso regression - sparse logistic regression - elastic net-regularized problems Beneficial to distribute by feature Can we map this to the CoCoA setup?
- 75. min w2Rd " P(w) := 2 ||w||2 + 1 n nX i=1 `i(wT xi) # Solution: Solve Primal Directly PRIMAL DUAL≥ Ai = 1 n xi max ↵2Rn " D(↵) := 2 kA↵k 2 1 n nX i=1 `⇤ i ( ↵i) #
- 76. Solution: Solve Primal Directly PRIMAL DUAL≥ min ↵2Rn f(A↵) + nX i=1 gi(↵i) min w2Rd f⇤ (w) + nX i=1 `⇤ i ( xT i w)
- 77. Solution: Solve Primal Directly PRIMAL DUAL Stopping criteria given by duality gap ≥ min ↵2Rn f(A↵) + nX i=1 gi(↵i) min w2Rd f⇤ (w) + nX i=1 `⇤ i ( xT i w)
- 78. Solution: Solve Primal Directly PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice ≥ min ↵2Rn f(A↵) + nX i=1 gi(↵i) min w2Rd f⇤ (w) + nX i=1 `⇤ i ( xT i w)
- 79. Solution: Solve Primal Directly PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice Default in software packages e.g. glmnet ≥ min ↵2Rn f(A↵) + nX i=1 gi(↵i) min w2Rd f⇤ (w) + nX i=1 `⇤ i ( xT i w)
- 80. Solution: Solve Primal Directly PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice Default in software packages e.g. glmnet Primal separates across machines (one primal variable per feature) ≥ min ↵2Rn f(A↵) + nX i=1 gi(↵i) min w2Rd f⇤ (w) + nX i=1 `⇤ i ( xT i w)
- 82. 50x speedup
- 84. CoCoA: A Framework for Distributed Optimization
- 85. CoCoA: A Framework for Distributed Optimization flexible
- 86. CoCoA: A Framework for Distributed Optimization flexible allows for arbitrary internal methods
- 87. CoCoA: A Framework for Distributed Optimization flexible efficient allows for arbitrary internal methods
- 88. CoCoA: A Framework for Distributed Optimization flexible efficient allows for arbitrary internal methods fast! (strong convergence & low communication)
- 89. CoCoA: A Framework for Distributed Optimization flexible efficient general allows for arbitrary internal methods fast! (strong convergence & low communication)
- 90. CoCoA: A Framework for Distributed Optimization flexible efficient general allows for arbitrary internal methods fast! (strong convergence & low communication) works for a variety of ML models
- 91. CoCoA: A Framework for Distributed Optimization flexible efficient general allows for arbitrary internal methods fast! (strong convergence & low communication) works for a variety of ML models 1. CoCoA-v1 [NIPS ’14]
- 92. CoCoA: A Framework for Distributed Optimization flexible efficient general allows for arbitrary internal methods fast! (strong convergence & low communication) works for a variety of ML models 1. CoCoA-v1 [NIPS ’14] 2. CoCoA+ [ICML ’15]
- 93. CoCoA: A Framework for Distributed Optimization flexible efficient general allows for arbitrary internal methods fast! (strong convergence & low communication) works for a variety of ML models 1. CoCoA-v1 [NIPS ’14] 2. CoCoA+ [ICML ’15] 3.ProxCoCoA+ [current work]
- 94. CoCoA: A Framework for Distributed Optimization flexible efficient general allows for arbitrary internal methods fast! (strong convergence & low communication) works for a variety of ML models 1. CoCoA-v1 [NIPS ’14] 2. CoCoA+ [ICML ’15] 3.ProxCoCoA+ [current work] CoCoA Framework
- 100. Impact & Adoption gingsmith.github.io/ cocoa/ Numerous talks & demos
- 101. Impact & Adoption ICML gingsmith.github.io/ cocoa/ Numerous talks & demos
- 102. Impact & Adoption ICML gingsmith.github.io/ cocoa/ Numerous talks & demos Open-source code & documentation
- 103. Impact & Adoption ICML gingsmith.github.io/ cocoa/ Numerous talks & demos Open-source code & documentation Industry & academic adoption
- 105. Empirical Results in Dataset Training (n) Features (d) Sparsity Workers (K) url 2M 3M 4e-3% 4 kddb 19M 29M 1e-4% 4 epsilon 400K 2K 100% 8 webspam 350K 16M 0.02% 16
- 107. A First Approach:
- 108. A First Approach: CoCoA+ with Smoothing
- 109. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers
- 110. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer
- 111. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer k↵k1 + k↵k2 2
- 112. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer k↵k1 + k↵k2 2
- 113. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 k↵k1 + k↵k2 2
- 114. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 k↵k1 + k↵k2 2
- 115. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 k↵k1 + k↵k2 2
- 116. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 k↵k1 + k↵k2 2
- 117. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 k↵k1 + k↵k2 2 X
- 118. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 k↵k1 + k↵k2 2 X CoCoA+ with smoothing doesn’t work
- 119. A First Approach: CoCoA+ with Smoothing Issue: CoCoA+ requires strongly-convex regularizers Approach: add a bit of L2 to the L1 regularizer Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 k↵k1 + k↵k2 2 X CoCoA+ with smoothing doesn’t work Additionally, CoCoA+ distributes by datapoint, not by feature
- 121. Better Solution: ProxCoCoA+ Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465
- 122. Better Solution: ProxCoCoA+ Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 ProxCoCoA+ 0.6030
- 123. Better Solution: ProxCoCoA+ Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 ProxCoCoA+ 0.6030
- 124. Better Solution: ProxCoCoA+ Amount of L2 Final Sparsity Ideal (δ = 0) 0.6030 δ = 0.0001 0.6035 δ = 0.001 0.6240 δ = 0.01 0.6465 ProxCoCoA+ 0.6030
- 125. Convergence
- 126. Convergence Assumption: Local -Approximation For , we assume the local solver finds an approximate solution satisfying: ⇥ ⇥ 2 [0, 1) E ⇥ G 0 k ( ↵[k]) G 0 k ( ↵? [k]) ⇤ ⇥ ⇣ G 0 k (0) G 0 k ( ↵? [k]) ⌘
- 127. Convergence Theorem 1. Let have -bounded supportL T ˜O ⇣ 1 1 ⇥ ⇣ 8L2 n2 ⌧✏ + ˜c ⌘⌘ gi Assumption: Local -Approximation For , we assume the local solver finds an approximate solution satisfying: ⇥ ⇥ 2 [0, 1) E ⇥ G 0 k ( ↵[k]) G 0 k ( ↵? [k]) ⇤ ⇥ ⇣ G 0 k (0) G 0 k ( ↵? [k]) ⌘
- 128. Convergence Theorem 1. Let have -bounded supportL T ˜O ⇣ 1 1 ⇥ ⇣ 8L2 n2 ⌧✏ + ˜c ⌘⌘ gi Assumption: Local -Approximation For , we assume the local solver finds an approximate solution satisfying: ⇥ ⇥ 2 [0, 1) E ⇥ G 0 k ( ↵[k]) G 0 k ( ↵? [k]) ⇤ ⇥ ⇣ G 0 k (0) G 0 k ( ↵? [k]) ⌘ Theorem 2. Let be -strongly convexµ T 1 (1 ⇥) µ⌧+n µ⌧ log n ✏ gi