How we realized SOA by Python at PyCon JP 2015

1. How we realize SOA by Python
In PyCon JP 2015

2. #PyConJP_C

3. SOA meaning
Service Oriented Architecture

4. What's SOA
SOA is an approach to create a system
based on small servers separated for small
functions.

5. SOA
WebApp
Search
Auth
Other
Images

6. https://www.nginx.com/blog/building-microservices-using-an-api-gateway/

7. So... is the SOA comfy?
• Yes.
• But... no silver bullet.

8. Pros
• Separated responsibilities
• Rapid integration
• Flexible scaling

9. Cons
• Lots of servers
● To create
● To deploy
● To monitor

10. Can't realize SOA with legacy style

11. • Direct transmitting
• Manual deploying
• Manual testing

12. So how we realize SOA

13. Today you can learn
• Case study of SOA by Python
• Practices to manage lots of services

14. Today you can't lean
• Step by step guide
• So much about Python's core

15. Agenda
About Our case Summary
Architecture Creating MonitoringTesting Deploying

16. Who I am
• Hiroki Kiyohara (hirokiky)
• BePROUD Inc
• Developer, Consultant, Trainer
• Admin of djangoproject.jp

17. Who we are
• News paper company
● Over 400,000 paid user (Web)
• Migrating to in-house production

18. Our case

19. Architecture

20. Architecture

21. Architecture
REST API

22. Architecture Separated from Web

23. API Gateway
• Proxy for back-end servers
• Handling permissions

24. Creating

25. Creating
• Need to create many services
• Share docs

26. Django
• Easy to create
• Many libraries
• Community and knowledge

27. DjangoRestFramework
• Framework on Django to create REST API
• Useful modules
• Smart Auth/Authz framework

28. DRF: Serializer
• Django's Form for REST API
● Nested
● Array of objects

29. Serializer is really simple

30. Validating by serializer
>>> from search.serializers import BulkPostSerializer
>>> BulkPostSerializer(data={
... "name": "hiroki",
... "posts": [
... {"title": "PyCon JP 2015",
... "body": "This is awesome event"},
... {"title": "How we realize SOA",
... "body": "Python is awesome"},
... ]
... })
>>> serializer.is_valid()
True
>>> serializer.validated_data
OrderedDict([('name', 'hiroki'), ('pos...

31. cookiecutter
• Template of repository
• Share best practices

32. .
├── README.md
├── project_name
│ ├── manage.py
│ └── project_name
│ ├── settings
│ │ ├── __init__.py
│ │ └── test.py
│ ├── urls.py
│ └── wsgi.py
├── .elasticbeanstalk/
├── .coveragerc
├── .gitignore
├── circle.yml
├── Dockerfile
├── Dockerrun.aws.json
├── requirements.txt
├── setup.cfg
└── tox.ini

33. OK so... created
• Many applications
• Share docs

34. django-rest-swagger
• API docs from docstring and code
• Demo-able API docs

35. django-rest-swagger

36. django-rest-swagger

37. Creating
• Django
• DjangoRestFramework
• cookiecutter
• tox

38. Testing

39. Testing
• Handy unit tests
• E2E tests

40. I know you won't run. MEE TOO
• Complex
• Slow
• Useless

41. Speed up by django's setting
• On memory sqlite
• Dummy or local cache
• Light hasher
Refer Two scoops of Django

42. Use nice tools
• tox
• flake8
• ...

43. tox
• Just type `tox`
• Run several tests in separated virtualenv

44. [tox]
envlist = py34, flake8
skipsdist = True
setupdir = ./myprj/
[testenv:py34]
deps = coverage
-rrequirements.txt
setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test
commands =
coverage erase
coverage run myprj/manage.py test myprj
coverage report
[testenv:flake8]
basepython = python3.4
deps = flake8
commands = flake8 myprj

45. [tox]
envlist = py34, flake8
skipsdist = True
setupdir = ./myprj/
[testenv:py34]
deps = coverage
-rrequirements.txt
setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test
commands =
coverage erase
coverage run myprj/manage.py test myprj
coverage report
[testenv:flake8]
basepython = python3.4
deps = flake8
commands = flake8 myprj
Installing dependencies

46. [tox]
envlist = py34, flake8
skipsdist = True
setupdir = ./myprj/
[testenv:py34]
deps = coverage
-rrequirements.txt
setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test
commands =
coverage erase
coverage run myprj/manage.py test myprj
coverage report
[testenv:flake8]
basepython = python3.4
deps = flake8
commands = flake8 myprj
Using settings for test

47. [tox]
envlist = py34, flake8
skipsdist = True
setupdir = ./myprj/
[testenv:py34]
deps = coverage
-rrequirements.txt
setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test
commands =
coverage erase
coverage run myprj/manage.py test myprj
coverage report
[testenv:flake8]
basepython = python3.4
deps = flake8
commands = flake8 myprj
Running tests with coverage

48. [tox]
envlist = py34, flake8
skipsdist = True
setupdir = ./myprj/
[testenv:py34]
deps = coverage
-rrequirements.txt
setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test
commands =
coverage erase
coverage run myprj/manage.py test myprj
coverage report
[testenv:flake8]
basepython = python3.4
deps = flake8
commands = flake8 myprj
Static analysis

49. Just...
$ tox

50. flake8
• Code style check
• Static analysis

51. flake8
• Don't review about syntax
• Remove tiny and messy code bugs

52. And more
• responses
• testfixtures
• factory-boy

53. Of cause, DO NOT
• Accessing outer services
• Setting middle wares locally

54. E2E testing
• Using requests
• Checking connection of each services

55. Locust.io
• Load tests by Python
• Distributed clients but aggregated report

56. Test
• Faster Django setting
• tox
• flake8
• locust.io

57. Deploying

58. Reduce costs to deploy
• Automated deploy by ElasticBeanstalk
• Master deploying from CI

59. Auto deploy
• Master branch is on development env
• Deploying from CI tool

60. Deploying

61. Deploying
Deploy It

62. Deploying
GREEN

63. Deploying
Push

64. Deploying
Deploy

65. Deploying
OK Pulling

66. Deploying
Deployed

67. Also from CLI or Web console
• Just type `$ eb deploy`
• Or through the Web

68. Deploy
• GitHub
• CircleCI
• Docker
• ElasticBeanstalk

69. Monitoring

70. A lots of servers to monitor
• Many services, lots servers
• Immutable infrastructure

71. Monitoring tools

72. Sentry
• Event (log) aggregation platform
• Manage console of each events

75. Sentry
• Many events but one notification
• Marked as `Resolved`, it'll come again

76. Fluentd
• Aggregating access log
• S3 and BigQuery

77. NewRelic
• Resource monitoring for containers
• And APM is awesome

78. NewRelic APM
• Analyzing Python application
• Which progress is slow?

80. Rundeck
• Job scheduler
• Better crond

81. Rundeck

82. Monitor
• Sentry
• Fluentd
• NewRelic
• Rundeck

83. So...

84. Will SOA be a silver bullet?

85. No

86. Pros
• Separated responsibilities
• Rapid integration
• Flexible scaling

87. But you need to do
• Easy creating
• Handy testing
• Auto deploying
• Relief monitoring

88. What should we do next?
• Non-blocking
• Nicer container platform

89. Thank you
• For listening
• For this nice event