SpringOne Platform 2019 SB Payment Service (part of the SoftBank Group) is developing a next-generation, in-house payment system using Spring and Pivotal Application Service. In this session, we'll talk about how and why we introduced Pivotal Platform to the organization and provide an overview of the Spring Boot- and Spring Cloud-based architecture, as well as how we implemented CI/CD and designed for observability and high resilience. We'll also discuss the benefits introducing a platform has brought to both development and operations. This system is now running in production. We'd like to share the things we've done to reduce incidents in production and make operations stable.

2. 

3. We issue credit cards
"Softbank Card" to
consumers.
Credit Card Issuer
Payment Aggregator
We provide a comprehensive
payment platform that offers
various online payment solutions.
Credit Card Acquirer
We are the only payment aggregator
in Japan who accepts and processes
transactions made with major brands
(VISA/ mastercard/ UnionPay).
Softbank customers can also
pay for online purchases via
their phone bill as Japan’s
leading carrier company.
Carrier Billing Provider

4. We issue credit cards
"Softbank Card" to consumer.
Credit Card Issuer
Payment Aggregator
We provide a comprehensive
payment platform that offers
various online payment solutions.
Credit Card Acquirer
We are an only payment aggregator
in Japan that accepts and
processes transactions made with a
major brands(VISA/ mastercard/
UnionPay) as an acquirer.
We provide consumers to
pay for online purchases with
their phone bill as Japanese
leading carrier company.
Carrier Billing Provider

5. If you have any questions, we’d be
happy to answer them at the end of
our presentation

6. ●
●
●

8. We left all development of our services to
outside vendors.
There were zero in-house engineers writing code.
The development environment was not ready.

9. Problem
Team

10. Problem Developed Support Tools
Introduced ToolsTeam

11. Problem
Team
Developed Support Tools
Introduced Tools
3 Engineers joined!
Accelerated KAIZEN tasks by a team

12. ①
Problem
Team

13. ①
Problem Created dashboards
Introduced Tools
Elasticsearch
Logstash
Kibana
Team

14. ②
Problem
Team

15. ②
Problem Re-architected with Spring
Introduced Tools
Team

16. ②
Problem Re-architected with Spring
Introduced Tools
Team One more engineer joined!

17. ●
●
●

18. Merchants
Financial
Institutions
E-Commerce
Gaming
Education
RealEstate
Etc
E-Books/Movies
Unified
Payment Service
Tickets
Provides several payment methods as API to EC sites
Credit Card
Mobile Carrier
Convenience Store
Prepaid Card
Account Transfer
Point
Account Integration
当社Our Company
API
Development target
Online Payment Service

19. Merchants
Financial
Institutions
E-Commerce
Gaming
Education
RealEstate
Etc
E-Books/Movies
Unified
Payment Service
Tickets
Provides several payment methods as API to EC sites
Credit Card
Mobile Carrier
Convenience Store
Prepaid Card
Account Transfer
Point
Account Integration
当社Our Company
API
Development target
Online Payment Service
Number of adoption
111,742 stores
(As of May 2019)
Transaction Amount
$ 28 billion
(As of 2018)

20. Merchants
Financial
Institutions
E-Commerce
Game
Education
RealEstate
Etc
E-Book/Movie
Unified
Payment Service
Ticket
Provides several payment methods as API to EC sites
Credit Card
Mobile Carrier
Convenience Store
Prepaid Card
Account Transfer
Point
Account Integration
当社Our Company
API
Development target
Online Payment Service
40+ payment methods
supported

21. Merchants
Financial
Institutions
E-Commerce
Gaming
Education
RealEstate
Etc
E-Books/Movies
Unified
Payment Service
Tickets
Provides several payment methods as API to EC sites
Credit Card
Mobile Carrier
Convenience Store
Prepaid Card
Account Transfer
Point
Account Integration
当社Our Company
API
Development target
Online Payment Service
Located between merchant systems
and financial institution systems

22. Requirements for the new system
●
●
●
Before…
Every project was lead by external vendors
（A long path from estimation / contract /
requirement definition to
acceptance)

23. Requirements for the new system
●
●
●
今までは…
案件毎に開発ベンダさんのチカラを借りて構築
（見積もり/要件定義から検収まで長い道のり）
Outsourcing made it impossible to deliver
incrementally and quickly in the agile way.

24. Requirements for the new system
●
●
●
Speedy delivery and
Continuous improvement
through in-house development

25. Team
In-house dev PJ
One more engineer joined!
6-person team

26. Introduced Tools
Cloud Native Platform
based on Pivotal Platform

28. ・
・

29. ？？
・
・
・

30. ？？
What we needed
・
・

31. ●
●
●

32. ●
●
●
Buildpack converts source code to a container image.
Developers don’t need to write a Dockerfile.
cf push + Buildpack reduces extra works!

33. Team structure and responsibility boundary

34. Team structure and responsibility boundaryTeam structure and responsibility boundary
Networking
Storage
Servers
Virtualization
O/S
Middleware
Runtime
Platform Operators
2 people

35. Team structure and responsibility boundary
Networking
Storage
Servers
Virtualization
O/S
Middleware
Runtime
Data
Application
Application Developers
4 people
Platform Operators
2 people

36. Networking
Storage
Servers
Virtualization
O/S
Middleware
Runtime
Data
Application
Focus on building and
operating platforms
Application Developers
4 people
Platform Operators
2 people
Team structure and responsibility boundary

37. Networking
Storage
Servers
Virtualization
O/S
Middleware
Runtime
Data
Application
Focus on design and
implementation of
business code
Team structure and responsibility boundary
Application Developers
4 people
Platform Operators
2 people

38. Networking
Storage
Servers
Virtualization
O/S
Middleware
Runtime
Data
Application
12 Factor App The only contract is
“12 Factor App”.
No vendor lock in.
Application Developers
4 people
Team structure and responsibility boundary
Platform Operators
2 people

39. ➢
➢
➢
➢

40. syslog+TLS
Logstash
Elasticsearch
Kibana
cf pushConcourse
PrometheusGrafana
git push
cf create-service
cf bind-service

41. ➢
➢
➢
➢

42. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C

43. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
、 、 、
、

44. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Financial
Institution A
Financial
Institution B
Financial
Institution C
Merchant X
Merchant Y
Merchant Z

45. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Financial
Institution A
Financial
Institution B
Financial
Institution C
Merchant X
Merchant Y
Merchant Z
、 、
、

46. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Merchant X
Financial
Institution A
Financial
Institution B
Financial
Institution C
Merchant X
Merchant Y
Merchant Z
Each app is deployed on PAS as
a microservice

47. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Financial
Institution A
Financial
Institution B
Financial
Institution C
Merchant X
Merchant Y
Merchant Z
Each app is implemented
with Java and Spring Boot

48. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Financial
Institution A
Financial
Institution B
Financial
Institution C
Merchant X
Merchant Y
Merchant Z

49. API
Gateway
Service A
Service B
Service C
Financial
Institution A
Financial
Institution B
Financial
Institution C
（ ➡ ）
Merchant X
Merchant Y
Merchant Z

50. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
The systems of merchants system and financial
institutions are out of our control
（ ➡ ）

51. Hystrix
API
Gateway
Service A
Service B
Service C
Merchant A
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Introduced Hystrix Circuit Breaker
for inter-system communications
Hystrix
Hystrix
Hystrix
（ ➡ ）

52. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Without Circuit Breaker,
If a system outage happens in
financial institution A …
（ ➡ ）

53. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Slow Response, Timeout
（ ➡ ）

54. （ ➡ ）
API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
The failure is propagated to
Service A, blocking processes
and causing possible thread
exhaustion

55. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Failure propagated to API
Gateway causing blocked
processes, thread depletion
（ ➡ ）

56. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
（ ➡ ）
Failure propagated to API
Gateway causing blocked
processes, thread depletion

57. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
B and C are affected by the failure
of financial institution A.
（ ➡ ）

58. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix
（ ➡ ）
With Circuit Breaker
If a system outage happens in
financial institution A …

59. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Circuit Breaker prevents the failure
propagation.
No worry about the effect to other financial
institutions.
（ ➡ ）

60. API
Gateway
Service A
Service B
Service C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Circuit Breaker adds fault tolerance
and resiliency to the app
（ ➡ ）

61. Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix
（ ➡ ）

62. （ ➡ ）
Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix

63. （ ➡ ）
Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix
Introduced RabbitMQ + Spring Cloud
Stream for async processing

64. （ ➡ ）
Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix

65. （ ➡ ）
Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix
When a failure happens in the
merchant system

66. （ ➡ ）
Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Hystrix
The message will be diverted to a “Dead
Letter Queue” and requeued later

67. （ ➡ ）
Notification
Gateway
Receiver A
Receiver B
Receiver C
Merchant X
Merchant Y
Merchant Z
Financial
Institution A
Financial
Institution B
Financial
Institution C
Hystrix
Hystrix
Hystrix
Even if an outage happens in the
merchant, the Circuit Breaker will
prevent the failure propagation.
Hystrix

69. ⤴

70. ⤴

71. ➢
➢
➢
➢

72. https://concourse-ci.org/
Concourse

73. https://concourse-ci.org/
Concourse

75. Staging
Develop
Production
Push to a branch triggers the pipeline

76. Staging
Develop
Production

77. Unit test for each Java version
to detect issues on new versions earlier
Staging
Develop
Production

78. Staging
Develop
Production

79. Staging
Develop
Production

80. Develop Cycle by CI
Staging
Develop
Production

81. Staging
Develop
Production

82. Merging to master branch triggers
deploy to Nexus repository
Staging
Develop
Production

83. Production
One-click Deploy to production
from Nexus Repository using nexus concourse resource

84. API
Gateway
Service A
Service B
Service C
Financial
Institution B
Mock
Financial
Institution B
Mock
Financial
Institution B
Mock

85. ●
●

86. Problem
HTTP transmission processing is low throughput
Unexpected short of Circuit Breaker

88. Report html
cf push

89. Performance testing should be
repeated over and over during development
to ensure continuous improvement.

90. ➢
○
○
○

91. ※Metrics, tracing, and logging
https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

93. ➢
○
○
○

94. ・
・
・
・

95.
Preset Dashboard
Preset Alerts

96. rabbitmq
⇒
{
"targets":[
"10.0.1.1",
"10.0.2.1",
"10.0.3.1",
"10.0.1.2",
"10.0.2.2",
],
"labels":{
"__meta_bosh_deployment":"rabbitmq",
"__meta_bosh_job_process_name":"rabbitmq-server"
}
}
Targets are added
automatically

97. Org A
⇒
Promregator
SpaceA
CF Cloud
Controller
Devs don’t care
Prometheus
https://github.com/promregator/promregator
Considering migration
to Metrics Registrar

98. Micrometer
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
management:
endpoints:
web:
exposure:
include: "*"
base-path: /actuator

99. Also shows past changes
Visualize if each deployment is healthy
at this moment
Visualize if each VM is healthy
at this moment
Warning
Ops
Warning

100. Ops
・ ・

101. Dev
We can always check all metrics in
Grafana that we have ever collected
from logs
・CPU
・JVM Memory（per area）
・Thread
・GC（frequency、time）
・Classloader

102. Dev
Can always check metrics in Grafana
we have ever collected from logs
・CPU
・JVM Memory（per area）
・Thread
・GC（frequency、time）
・Classloader

103. Action is triggered by alerting

104. Ops
Dev
No outsourcing to
external monitoring center
org_name: OrgA
severity: fatal
org_name: OrgA
severity: /.*/
severity: fatal
severity: /.*/
Set route/receiver
using org_name,
severity as keys
Twilio call for emergency
(24 hour support)

105. Link to the Grafana dashboard
Detect RabbitMQ Dead Letter QueueCall

106. ・
・ ・ ・
・
・
・
・
・
Detect the decrease of
the service level
before user’s report

107. ➢
○
○
○

108. ・
・

110. Ops
Dev
Biz
ElastAlert
Praeco
Firehose
to syslog

111. Firehose-to
-syslog
Firehose-to
-syslog
Firehose-to
-syslog
Loggregator
Agent
Doppler
STD
OUT
STD
OUT
Considering migration
to Syslog Drain

112. ---
releases:
- name: syslog
version: "11.4.0"
url: "https://bosh.io/xxx?v=11.4.0"
sha1: "xxx"
addons:
- name: syslog
exclude:
jobs:
- name: syslog_forwarder
release: syslog
jobs:
- name: syslog_forwarder
release: syslog
properties:
syslog:
address: logstash.xxx.jp
port: 5514
transport: tcp
tls_enabled: true
permitted_peer: "*.xxx.jp"
ca_cert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

113. Ops
Dev
Biz

114. Biz

115. Dev
System Error
In a merchant
High latency
in a financial
institution

116. Ops

117. Ops
Dev
cf_org_name: OrgA
log_level: ERROR
cf_org_name: OrgA
log_level: WARN
@message: xxx
@message: xxx
Set notification
destination using
cf_org_name,
log_level as keys
Twilio call for emergency
(24 hour support)
ElastAlert

118. ➢
○
○
○

119. Service A
Merchant X
Financial
Institution A
API
Gateway
Receiver A
Notification
Gateway
Visualize

120. You can see a bottleneck at a glance across
multiple services.

121. Zipkin, Spring Cloud Sleuth
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
spring:
zipkin:
base-url: https://my-zipkin.example.com
service.name: my-application
sender.type: web

122. Ops
Dev
Biz
Share Elasticsearch with LoggingConfigure
instrumentations

123. # of transactions
Duration (Avg) Duratin (Percentile)

124. On-premise
Existing app
Existing
system team
Apps on PCF Dev
Something
happens in the
existing app?
Let me check

126. ➢
➢

127. Case1: Effect on Business members
Specialist of credit card system (NOT developer)
Biz
# of transactions
Duration (Avg) Duratin (Percentile)

128. Case1: Effect on Business members
Specialist of credit card system (NOT developer)
Something wrong...
It should be shorter...
Biz

129. Specialist of Credit Card System (Not developer)
Jump to Zipkin
Case1: Effect on Business members
This process affects?
Biz

130. Dev
Case1: Effect on Business members
Biz member can detect the problem.
I’ll check now.
This transaction seems
to be something wrong.Biz

131. Case2: Detect of abnormal trend

132. Case2: Detect of abnormal trend
With long time period,
delayed on specific time slot.
22:00 22:00 22:00
No problem

133. Case2: Detect of abnormal trend
No transaction finished.
Transaction might be locked.

134. On-premise
Existing app
Apps on PCF
Case2: Detect of abnormal trend
Delays on a specific time slot
Re-considered the number of
parallels and timeout.

135. Case2: Detect of abnormal trend
Our system was improved
by detecting slight abnormality

136. Improves apps’ operation efficiency
dramatically

138. ●
●
●
●

139. Before After
Release
Improvement
Release Work Manual work One click
Release Quality Human error occurs No mistakes
Release Time 45 min 5 min
Use of
Cloud
Scaleout operation Manual work One click
Container
Orchestration
- Leave it to the
platform
Auto-restart Self-made tools Leave it to the
platform

140. ●
●
●

141. A platform cannot be built by only relying on
outsourced vendors.
It’s possible to build and operate a platform by
taking ownership in-house.
A powerful platform allows a small engineering
team to focus on application development.