Architectural caching patterns for kubernetes

1. Architectural Caching Patterns
for Kubernetes
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast

2. About me
● Cloud-Native Team Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland

3. About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
● Products:
○ Hazelcast IMDG
○ Hazelcast Jet
○ Hazelcast Cloud
@Hazelcast
www.hazelcast.com

4. ● Introduction
● Caching Architectural Patterns
○ Embedded
○ Embedded Distributed
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

5. Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
Microservice World

6. Microservice World
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
cache
cache
cache
cache

7. Microservice World
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
cache
cache
cache

8. Microservice World
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
cache
cache
cache

9. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded
○ Embedded Distributed
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

10. 1. Embedded

11. Application
Kubernetes
Service
Cache
Application
Cache
Request
Embedded Cache
Kubernetes Pod
Kubernetes Pod

12. private ConcurrentHashMap<String, String> cache =
new ConcurrentHashMap<>();
private String processRequest(String request) {
if (cache.contains(request)) {
return cache.get(request);
}
String response = process(request);
cache.put(request, response);
return response;
}
Embedded Cache (Pure Java implementation)

13. private ConcurrentHashMap<String, String> cache =
new ConcurrentHashMap<>();
private String processRequest(String request) {
if (cache.contains(request)) {
return cache.get(request);
}
String response = process(request);
cache.put(request, response);
return response;
}
Embedded Cache (Pure Java implementation)

14. ● No Eviction Policies
● No Max Size Limit
(OutOfMemoryError)
● No Statistics
● No built-in Cache Loaders
● No Expiration Time
● No Notification Mechanism
Java Collection is not a Cache!

15. CacheBuilder.newBuilder()
.initialCapacity(300)
.expireAfterAccess(Duration.ofMinutes(10))
.maximumSize(1000)
.build();
Embedded Cache (Java libraries)

16. Embedded Cache (Java libraries)
CacheBuilder.newBuilder()
.initialCapacity(300)
.expireAfterAccess(Duration.ofMinutes(10))
.maximumSize(1000)
.build();

17. Caching Application Layer
@Service
public class BookService {
@Cacheable("books")
public String getBookNameByIsbn(String isbn) {
return findBookInSlowSource(isbn);
}
}

18. Caching Application Layer
@Service
public class BookService {
@Cacheable("books")
public String getBookNameByIsbn(String isbn) {
return findBookInSlowSource(isbn);
}
}
Be Careful, Spring uses ConcurrentHashMap by default!

19. Embedded Cache
Application
Kubernetes
Service
Cache
Application
Cache
Request
Kubernetes Pod
Kubernetes Pod

20. 1*. Embedded Distributed

21. Application
Application
Kubernetes
Service
Cache
Cache
Request
Hazelcast
Cluster
Embedded Distributed Cache
Kubernetes Pod
Kubernetes Pod

22. @Configuration
public class HazelcastConfiguration {
@Bean
CacheManager cacheManager() {
return new HazelcastCacheManager(
Hazelcast.newHazelcastInstance());
}
}
Embedded Distributed Cache (Spring with Hazelcast)

23. Hazelcast Discovery Plugins

24. Hazelcast Discovery Plugins

25. Embedded Distributed Cache
Application
Application
Kubernetes
Service
Cache
Cache
Request
Hazelcast
Cluster
Kubernetes Pod
Kubernetes Pod

26. Embedded Cache
Pros Cons
● Simple configuration /
deployment
● Low-latency data access
● No separate Ops Team
needed
● Not flexible management
(scaling, backup)
● Limited to JVM-based
applications
● Data collocated with
applications

27. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

28. 2. Client-Server

29. Application
Kubernetes
Service
Application
Request
Cache Server
StatefulSet
Client-Server Cache
Kubernetes Pod
Kubernetes Pod

30. Client-Server Cache
Application
Kubernetes
Service
Application
Request
Cache Server
StatefulSet
Kubernetes Pod
Kubernetes Pod

31. Client-Server Cache
Separate Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
Cache Server
StatefulSet
Kubernetes Pod
Kubernetes Pod

32. Client-Server Cache
Application
Kubernetes
Service
Application
Request
Cache Server
StatefulSet
Kubernetes Pod
Kubernetes Pod

33. Client-Server Cache
Application
Kubernetes
Service
Application
Request
Cache Server
StatefulSet
Kubernetes Pod
Kubernetes Pod

34. Client-Server Cache

35. Client-Server Cache

36. Client-Server Cache
Starting Hazelcast Cache Server
$ helm install hazelcast/hazelcast

37. Client-Server Cache
Hazelcast Client
@Configuration
public class HazelcastClientConfiguration {
@Bean
CacheManager cacheManager() {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().getKubernetesConfig()
.setEnabled(true);
return new HazelcastCacheManager(HazelcastClient
.newHazelcastClient(clientConfig));
}
}
Starting Hazelcast Cache Server
$ helm install hazelcast/hazelcast

38. Client-Server Cache
Separate Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
Cache Server
StatefulSet
Kubernetes Pod
Kubernetes Pod

39. 2*. Cloud

40. Application
Kubernetes
Service
Application
Request
Cloud (Cache as a Service)
Kubernetes Pod
Kubernetes Pod

41. Cloud (Cache as a Service)
Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
Kubernetes Pod
Kubernetes Pod

42. Cloud (Cache as a Service)
Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
Kubernetes Pod
Kubernetes Pod

43. Cloud (Cache as a Service)
Application
Kubernetes
Service
Application
Request
Kubernetes Pod
Kubernetes Pod

44. @Configuration
public class HazelcastCloudConfiguration {
@Bean
CacheManager cacheManager() {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().getCloudConfig()
.setEnabled(true)
.setDiscoveryToken("KSXFDTi5HXPJGR0wRAjLgKe45tvEEhd");
clientConfig.setGroupConfig(
new GroupConfig("test-cluster", "b2f984b5dd3314"));
return new HazelcastCacheManager(
HazelcastClient.newHazelcastClient(clientConfig));
}
}
Cloud (Cache as a Service)

45. Client-Server (Cloud) Cache
Pros
● Data separate from
applications
● Separate management
(scaling, backup)
● Programming-language
agnostic
Cons
● Separate Ops effort
● Higher latency
● Server network requires
adjustment (same region,
same VPC)

46. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

47. 3. Sidecar

48. Kubernetes Service
Request
Hazelcast
Cluster
Kubernetes POD
Application Container
Cache Container
Application Container
Cache Container
Kubernetes POD
Sidecar Cache

49. Sidecar Cache
Similar to Embedded:
● the same physical machine
● the same resource pool
● scales up and down together
● no discovery needed (always localhost)
Similar to Client-Server:
● different programming language
● uses cache client to connect
● clear isolation between app and cache

50. @Configuration
public class HazelcastSidecarConfiguration {
@Bean
CacheManager cacheManager() {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig()
.addAddress("localhost:5701");
return new HazelcastCacheManager(HazelcastClient
.newHazelcastClient(clientConfig));
}
}
Sidecar Cache

51. Sidecar Cache
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
containers:
- name: application
image: leszko/application
- name: hazelcast
image: hazelcast/hazelcast

52. Sidecar Cache
Pros Cons
● Simple configuration
● Programming-language
agnostic
● Low latency
● Some isolation of data and
applications
● Not flexible management
(scaling, backup)
● Data collocated with
application PODs

53. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar ✔
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

54. 4. Reverse Proxy

55. Application
Kubernetes
Service
Cache
Application
Request
Reverse Proxy Cache
Kubernetes Pod
Kubernetes Pod

56. Reverse Proxy Cache
http {
...
proxy_cache_path /data/nginx/cache
keys_zone=one:10m;
...
}

57. ● Only for HTTP
● Not distributed
● No High Availability
● Data stored on the disk
NGINX Reverse Proxy Cache Issues

58. 4*. Reverse Proxy Sidecar

59. Kubernetes Service
Request
Hazelcast
Cluster
Kubernetes POD
Application Container
Reverse Proxy Cache
Container
Application Container
Reverse Proxy Cache
Container
Kubernetes POD
Reverse Proxy Sidecar Cache

61. Good

62. Reverse Proxy Sidecar Cache
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby

63. Reverse Proxy Sidecar Cache
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby

64. Reverse Proxy Sidecar Cache
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
initContainers:
- name: init-networking
image: leszko/init-networking
containers:
- name: caching-proxy
image: leszko/caching-proxy
- name: application
image: leszko/application

65. Reverse Proxy Sidecar Cache
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby

67. Reverse Proxy Sidecar Cache (Istio)

68. Reverse Proxy Sidecar Cache (Istio)

69. Bad

71. Reverse Proxy Cache
@CacheEvict(value = "someValue", allEntries = true)
public void evictAllCacheValues() {}
Application Cache:

72. Reverse Proxy Cache
@CacheEvict(value = "someValue", allEntries = true)
public void evictAllCacheValues() {}
Application Cache:
Proxy Cache:
http {
...
location / {
add_header Cache-Control public;
expires 86400;
etag on;
}
}

73. Reverse Proxy (Sidecar) Cache
Pros Cons
● Configuration-based (no
need to change
applications)
● Programming-language
agnostic
● Consistent with containers
and microservice world
● Difficult cache invalidation
● No mature solutions yet
● Protocol-based (e.g. works
only with HTTP)

74. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar ✔
○ Reverse Proxy ✔
○ Reverse Proxy Sidecar ✔
● Summary
Agenda

75. Summary

76. application-aware?

77. application-aware?
early adopter?
no

78. application-aware?
Reverse Proxy
no
no
early adopter?

79. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
no
yes no
early adopter?

80. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
yes no
yes no
early adopter?

81. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
yes no
yes no
no
early adopter?

82. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
yes no
yes no
no
no
early adopter?

83. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
yes no
yes
yes no
no
no
early adopter?

84. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
cloud?
yes no
yes
yes
yes no
no
no
early adopter?

85. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
cloud?
Client-Server
yes no
yes
yes
yes no
no
no
no
early adopter?

86. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
cloud?
Client-Server
Cloud
yes no
yes
yes yes
yes no
no
no
no
early adopter?

87. Resources
● Code for this talk:
https://github.com/leszko/caching-patterns
● Hazelcast Sidecar Container Pattern:
https://hazelcast.com/blog/hazelcast-sidecar-container-pattern/
● Hazelcast Reverse Proxy Sidecar Caching Prototype:
https://github.com/leszko/caching-injector
● NGINX HTTP Reverse Proxy Caching:
https://www.nginx.com/resources/videos/best-practices-for-caching/

88. Thank You!
Rafał Leszko
@RafalLeszko