For Spring I/O Barcelona 2018 I explained how we spent the last 6 months building and running a set of Spring Cloud based microservices on AWS Elastic Container Service, their Docker Orchestrator.

1. Building and running Spring Cloud-based
microservices on AWS ECS
Joris Kuipers
@jkuipers

2. About Me
 Joris Kuipers
 @jkuipers
 CTO of application development,
hands-on architect and
fly-by-night Spring trainer
@ Trifork Amsterdam

3. Setting The Stage
“When we are born we cry that we are come to this great stage of fools.”
― William Shakespeare

4. Introduction
 Integration project for
Dutch Loteries
 Started Q4 2017
 Expose suite of new backend systems to clients
 Provide APIs based on custom domain model
 Encapsulate backend details
 Facilitate future replacement

5. Architecture
 Separate domains
(verticals)
and service types
(horizontals)
 Verticals:
specific games,
subscriptions, players, …

6. Project Setup
 Every service is Spring Cloud app
 Blocking HTTP for inter-service communication
 Too early for reactive
 Hystrix for circuit breaking
 Mono-repo using Gradle build
 More services than teams
 Easy to have shared libs
 Can extract later

7. Technology
 Services run in Docker Container on AWS ECS
 Consul for service discovery and shared config
 Provisioning with Terraform
 Use as-a-service for CI/CD

8. AWS ECS
 Elastic Container Service
 Docker container scheduling / orchestration
 Placement on EC2 hosts
 Adding / removing instances
 Includes Docker Registry:
ECR

9. Expectations
 Fairly late adopters of Cloud and Docker
 Some plain EC2 experience with AWS
 Some Kubernetes experience on private DC
 AWS is nr. 1 cloud: ECS will just work
 Without much custom ops
 Services and tasks: similar to K8S

10. Chapter 1: ECS vs EC2
“We are responsible for actions performed in response to
circumstances for which we are not responsible”
― Allan Massie

11. How ECS Works
 You configure and pay for EC2 hosts
 ECS is free service using those hosts
 Runs Container Agent as Docker container
 Dedicated AMI, or install yourself

12. Scaling ECS vs EC2
 With ECS, can scale nr of containers for a task
 Based on alarms like % CPU
 However, may require additional EC2 capacity
 Autoscaling no good for scaling down
 Don’t want to kill EC2 instances w/ active containers
 Drain frst
 Need to manually control this w/ lambda

13. Fargate
 Announced late 2017
 Containers 1st class citizen
 No need to manage your own EC2 instances
 Only available in US East initially
 No way to SSH into your instance anymore

14. Chapter 2: Consul
“Consul - in American politics, a person who having failed to
secure an office from the people is given one by the
Administration on condition that he leave the country.”
― Ambrose Bierce

15. HashiCorp’s Consul
 Service Discovery
 Configuration through K/V store
 Excellent Spring Cloud Support
 Just worked when first deploying 3 services
 Stopped working when adding more

16. ECS Networking
 Bridged networking: docker0 to EC2 host
 Communication between Dockers via
host IP and port
 Binding to host port dynamic
 We were using local IP and port
 Need to discover host IP and port at startup…

17. Discover Host
 IP can be accessed from URL
 http://169.254.169.254/latest/meta-data/local-ipv4/
if curl -s -q http://169.254.169.254 --connect-timeout 0.5 > /dev/null; then
export SPRING_CLOUD_CONSUL_DISCOVERY_IP_ADDRESS=
$( curl -s http://169.254.169.254/latest/meta-data/local-ipv4/ )
fi

18. Discover Port Binding
 Port binding is in metadata file
 JSON file, path available via env. Variable
 Feature since Nov. ’17: first enable
 Have to parse the JSON
# Wait until container metadata is complete
until grep "ContainerID" $ECS_CONTAINER_METADATA_FILE > /dev/null; do sleep 1; done
export SPRING_CLOUD_CONSUL_DISCOVERY_PORT=$( cat $ECS_CONTAINER_METADATA_FILE |
jq -r ".PortMappings[] .HostPort" )

19. Port Binding Discovery before Nov. ’17

20. Configuring Consul Instance ID
 Used to register service instance with Consul
 Needs to be unique per instance!
 Defaults to ${spring.application.name}:${server.port}
 But within Docker, our server port is always 8080
 Multiple instances overwrite each other
spring.cloud.consul.discovery.instance-id=
${spring.application.name}-${SPRING_CLOUD_CONSUL_DISCOVERY_PORT:${server.port}}

21. Consul as Config Server
 Built-in K/V store supported by Spring Cloud
 Individual properties, or file per service
 Supports shared configuration
 Idea: populate from Git repo (git2consul)
 Problem: passwords

22. Chapter 3: AWS Parameter Store
“In the absence of the gold standard, there is no way to
protect savings from confiscation through inflation. There is
no safe store of value.”
― Alan Greenspan

23. AWS Systems Manager
Parameter Store
 String K/V store, part of EC2
 Supports nested names
 Since June ’17
 Supports encryption of secrets
 Versioning
 Access controlled through IAM

24. AWS Parameter Store

25. Spring Cloud Parameter Store
support
 Looked like great solution
 Hierarchical paths allow config per env. / service &
shared config, like with Consul
 IAM Security, Java & Terraform API support, …
 But no Spring Cloud support, only open issue
 No support yet

26. Spring Cloud Config Clients
 Implement PropertySourceLocator
 CompositePropertySource: ordered list of
PropertySources for:
 Shared config (/application/…)
 Service-specific config (/<service-name>/…)
 Incl. per-profile config (/<service-name>,<profile>/…)
PropertySource<?> locate(Environment environment);

27. Implementing Parameter Store
Support
 PropertySourceLocator:
Mostly copied from Consul support
 No support for whole files
 Param Store value max. 4K chars
 Only real work: implement PropertySource
 Using AWS Java SDK
 AWSSimpleSystemsManagement client

28. public class AwsParamStorePropertySource extends EnumerablePropertySource<AWSSimpleSystemsManagement> {
private String context;
private Map<String, Object> properties = new LinkedHashMap<>();
public AwsParamStorePropertySource(String context, AWSSimpleSystemsManagement ssmClient) {
super(context, ssmClient);
this.context = context;
}
public void init() {
GetParametersByPathRequest paramsRequest = new GetParametersByPathRequest()
.withPath(context).withRecursive(true).withWithDecryption(true);
getParameters(paramsRequest);
}
@Override public String[] getPropertyNames() {
Set<String> strings = properties.keySet();
return strings.toArray(new String[strings.size()]);
}
@Override public Object getProperty(String name) { return properties.get(name); }
private void getParameters(GetParametersByPathRequest paramsRequest) {
GetParametersByPathResult paramsResult = source.getParametersByPath(paramsRequest);
for (Parameter parameter : paramsResult.getParameters()) {
String key = parameter.getName().replace(context, "").replace('/', '.');
properties.put(key, parameter.getValue());
}
if (paramsResult.getNextToken() != null) {
getParameters(paramsRequest.withNextToken(paramsResult.getNextToken()));
}
}
}

29. Wrapping up
 Add BootstrapConfiguration and type-safe
properties classes for Spring Cloud
 Configure per-environment support in
bootstrap.properties of every service:
 Contributed pull request
aws.paramstore.prefix=/config-${ENVIRONMENT}

30. Chapter 4: Service Discovery
“Mistakes are the portals of discovery.”
― James Joyce

31. Health Checks in ECS
 Started out with just external load balancer
 ECS needed internal one as well,
just for container health checks
 Might as well use for inter-service communication
 With Parameter Store, made Consul obsolete

32. AWS Advancements
 March 2018: ECS supports Docker health checks
 ELB no longer required
 Also March 2018: ECS Service Discovery
 Route 53 Auto Naming API
 Only returns healthy services
 No need for custom service registry or internal LB
 Requires awsvpc, not in all regions yet
 UPDATE: after this presentation, AWS announced this

33. AWSVPC
 “Cloud-native” Networking mode
 Alternative to bridged
 Introduced Nov. ’17
 Full networking features for ECS Tasks
 Security groups, network monitoring, direct IPs
 Optional using EC2, required for Fargate

34. AWSVPC Small Print

35. ECS Service Discovery
 Not using this yet
 Seems good candidate for Spring Cloud Support
 Discover all healthy instances
 Provide client-side load balancing through Ribbon
 Real benefits over internal LB
 Saves network hops and thus latency

36. Ribbon & Server-side Load Balancing
 Keeping Ribbon for now
 Might move to service discovery
 Just configure DNS name for downstream service
lotto.system-url=http://lotto-system
lotto-system.ribbon.listOfServers=lotto-system.nlogateway.vpc
@Bean @LoadBalanced
RestTemplate restTemplate(RestTemplateBuilder builder,
@Value("${lotto.system-url}") String serviceUrl)
{
return builder.rootUri(serviceUrl).build();
}

37. Ribbon & Zuul Configuration
 Using Ribbon facilitates Zuul proxy config as well
zuul:
ignoredServices: '*'
# proxy anything except for these, handled locally:
ignoredPatterns:
- /actuator/**
- /error
- /lotto/productOrder*
routes:
lotto-system:
path: /**
serviceId: lotto-system
Service, not URL

38. Chapter 4: Logging and Tracing
“To acquire knowledge, one must study;
but to acquire wisdom, one must observe.”
― Marilyn vos Savant

39. Logging
 Using logback-spring.xml for Logback config
 Allows accessing spring properties
 No need for reloading config
 Currently using logz.io
 ELK-based logging as a service provider
 Provides Logback appender

40. logback-spring.xml
<configuration>
<property name="environment" value="${ENVIRONMENT:-local}"/>
<include resource="org/sfw/boot/logging/logback/defaults.xml" />
<include resource="org/sfw/boot/logging/logback/console-appender.xml" />
<include resource="logzio-appender.xml" />
…
<root level="INFO">
<if condition='property("environment").equals("local")'>
<then><appender-ref ref="CONSOLE"/></then>
<else><appender-ref ref="LogzioAppender"/></else>
</if>
</root>
</configuration>

41. logzio-appender.xml
<included>
<springProperty name="service" source="spring.application.name/>
<springProperty name="version" source="gw.version" defaultValue="unknown"/>
<define name="awsRegion"
class="nlo.gateway.logging.AwsRegionPropertyDefiner"/>
<appender name="LogzioAppender"
class="io.logz.logback.LogzioLogbackAppender">
…
<additionalFields>environment=${environment};service=${service};
version=${version};aws_region=${awsRegion}</additionalFields>
</appender>
…
</included>

42. Logback PropertyProvider
import ch.qos.logback.core.PropertyDefinerBase;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
public class AwsRegionPropertyDefiner extends PropertyDefinerBase {
@Override
public String getPropertyValue() {
Region currentRegion = Regions.getCurrentRegion();
return currentRegion != null ? currentRegion.getName() : "-";
}
}

43. Distributed Tracing
 Spring Sleuth allows distribute tracing
 Propagating correlation ID per logical request
 Uses OpenZipkin’s Brave
 Instruments many Spring components

44. SLF4J MDC Integration

45. Brave vs. AMZN Trace IDs
 AWS LBs can add X-AMZN-Trace-ID header
 Format incompatible with Brave’s Trace ID
 Self=1-67891234-12456789abcdef012345678;Root=1-
67891233-abcdef012345678912345678
 However, easy to forward through Sleuth:
 Extracting and logging root trace ID slightly trickier
 Custom Slf4jCurrentTraceContext impl
 Override beans from SleuthLogAutoConfiguration
spring.sleuth.propagation-keys=X-Amzn-Trace-Id

46. Bootstrap Logging
 Initial bootstrap Logging not done via Logback
 Might prevent seeing cause of startup errors
 Enable Logging for ECS tasks, e.g. via docker-compose:
services:
lotto-experience:
logging:
driver: awslogs
options:
awslogs-group: ${AWSLOGS_GROUP}
awslogs-stream-prefix: lotto

47. Metrics
 Boot 2.0 ships with Micrometer integration
 Metrics façade: SLF4J for metrics
 Instruments many Spring components
 Using this with Datadog

48. Common Tags + Hystrix Metrics
@Bean
MeterRegistryCustomizer<MeterRegistry> commonTags(Environment env) {
Region awsRegion = Regions.getCurrentRegion();
return registry -> registry.config().commonTags(
"service", env.getProperty("spring.application.name),
"environment", env.getProperty("environment", "local"),
"version", env.getProperty("gw.version", "unknown-version"),
"aws_region", awsRegion != null ? awsRegion.getName() : "-"
);
}
@Bean HystrixMetricsBinder hystrixMetricsBinder() {
return new HystrixMetricsBinder();
}

49. Think About Cardinality
 Reduce possible values for metrics tags
 HTTP metrics include URI
@Bean
MeterFilter queryParameterStrippingMeterFilter() {
return MeterFilter.replaceTagValues("uri", url -> {
int i = url.indexOf('?');
return i == -1 ? url : url.substring(0, i);
});
}

50. Conclusion
 AWS Docker support less mature than expected
 Lots of improvements all the time, though
 Including Kubernetes support: EKS
 Spring Cloud provides tons of great features
 Central config, service discovery, tracing
 Easy to add custom implementations

51. THANKS!
Q & A
Joris Kuipers
@jkuipers