Submit Search
Upload
Lessons Learned: Using Concourse In Production
•
11 likes
•
10,662 views
Shingo Omura
Follow
Lessons Learned: Using Concourse In Production
Read less
Read more
Software
Report
Share
Report
Share
1 of 30
Download now
Download to read offline
Recommended
Production Bioinformatics, emphasis on Production
Production Bioinformatics, emphasis on Production
Chris Dwan
Introduction to yocto
Introduction to yocto
Alex Gonzalez
Linux Network Stack
Linux Network Stack
Adrien Mahieux
Yocto Project introduction
Yocto Project introduction
Yi-Hsiu Hsu
Understanding of linux kernel memory model
Understanding of linux kernel memory model
SeongJae Park
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
Jarod Wang
linux kernel overview 2013
linux kernel overview 2013
Rohit Pratap Singh
Jagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratch
linuxlab_conf
Recommended
Production Bioinformatics, emphasis on Production
Production Bioinformatics, emphasis on Production
Chris Dwan
Introduction to yocto
Introduction to yocto
Alex Gonzalez
Linux Network Stack
Linux Network Stack
Adrien Mahieux
Yocto Project introduction
Yocto Project introduction
Yi-Hsiu Hsu
Understanding of linux kernel memory model
Understanding of linux kernel memory model
SeongJae Park
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
Jarod Wang
linux kernel overview 2013
linux kernel overview 2013
Rohit Pratap Singh
Jagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratch
linuxlab_conf
Course 102: Lecture 25: Devices and Device Drivers
Course 102: Lecture 25: Devices and Device Drivers
Ahmed El-Arabawy
X / DRM (Direct Rendering Manager) Architectural Overview
X / DRM (Direct Rendering Manager) Architectural Overview
Moriyoshi Koizumi
I2C Drivers
I2C Drivers
SysPlay eLearning Academy for You
Pod density comparison: VMware vSphere with Tanzu vs. a bare-metal approach ...
Pod density comparison: VMware vSphere with Tanzu vs. a bare-metal approach ...
Principled Technologies
Expert Day 2019 - SUSE Linux Enterprise 15
Expert Day 2019 - SUSE Linux Enterprise 15
SUSE
Getting started with BeagleBone Black - Embedded Linux
Getting started with BeagleBone Black - Embedded Linux
Emertxe Information Technologies Pvt Ltd
Introduction to OpenStack Cinder
Introduction to OpenStack Cinder
Sean McGinnis
Launch the First Process in Linux System
Launch the First Process in Linux System
Jian-Hong Pan
Introduction to char device driver
Introduction to char device driver
Vandana Salve
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Cesar Rodriguez
Embedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernel
Emertxe Information Technologies Pvt Ltd
Enabling new client operating systems in Uyuni. AlmaLinux as an example.
Enabling new client operating systems in Uyuni. AlmaLinux as an example.
Uyuni Project
Linux Networking Explained
Linux Networking Explained
Thomas Graf
Linux systems - Linux Commands and Shell Scripting
Linux systems - Linux Commands and Shell Scripting
Emertxe Information Technologies Pvt Ltd
Course 102: Lecture 19: Using Signals
Course 102: Lecture 19: Using Signals
Ahmed El-Arabawy
Ansible: Infrastructure as Code for OpenShift
Ansible: Infrastructure as Code for OpenShift
Ignacio Sánchez Ginés
Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0
Emertxe Information Technologies Pvt Ltd
U Boot or Universal Bootloader
U Boot or Universal Bootloader
Satpal Parmar
Kubernetes CRI containerd integration by Lantao Liu (Google)
Kubernetes CRI containerd integration by Lantao Liu (Google)
Docker, Inc.
Ixgbe internals
Ixgbe internals
SUSE Labs Taipei
LAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMG
Linaro
Kubernetes 101
Kubernetes 101
Stanislav Pogrebnyak
More Related Content
What's hot
Course 102: Lecture 25: Devices and Device Drivers
Course 102: Lecture 25: Devices and Device Drivers
Ahmed El-Arabawy
X / DRM (Direct Rendering Manager) Architectural Overview
X / DRM (Direct Rendering Manager) Architectural Overview
Moriyoshi Koizumi
I2C Drivers
I2C Drivers
SysPlay eLearning Academy for You
Pod density comparison: VMware vSphere with Tanzu vs. a bare-metal approach ...
Pod density comparison: VMware vSphere with Tanzu vs. a bare-metal approach ...
Principled Technologies
Expert Day 2019 - SUSE Linux Enterprise 15
Expert Day 2019 - SUSE Linux Enterprise 15
SUSE
Getting started with BeagleBone Black - Embedded Linux
Getting started with BeagleBone Black - Embedded Linux
Emertxe Information Technologies Pvt Ltd
Introduction to OpenStack Cinder
Introduction to OpenStack Cinder
Sean McGinnis
Launch the First Process in Linux System
Launch the First Process in Linux System
Jian-Hong Pan
Introduction to char device driver
Introduction to char device driver
Vandana Salve
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Cesar Rodriguez
Embedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernel
Emertxe Information Technologies Pvt Ltd
Enabling new client operating systems in Uyuni. AlmaLinux as an example.
Enabling new client operating systems in Uyuni. AlmaLinux as an example.
Uyuni Project
Linux Networking Explained
Linux Networking Explained
Thomas Graf
Linux systems - Linux Commands and Shell Scripting
Linux systems - Linux Commands and Shell Scripting
Emertxe Information Technologies Pvt Ltd
Course 102: Lecture 19: Using Signals
Course 102: Lecture 19: Using Signals
Ahmed El-Arabawy
Ansible: Infrastructure as Code for OpenShift
Ansible: Infrastructure as Code for OpenShift
Ignacio Sánchez Ginés
Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0
Emertxe Information Technologies Pvt Ltd
U Boot or Universal Bootloader
U Boot or Universal Bootloader
Satpal Parmar
Kubernetes CRI containerd integration by Lantao Liu (Google)
Kubernetes CRI containerd integration by Lantao Liu (Google)
Docker, Inc.
Ixgbe internals
Ixgbe internals
SUSE Labs Taipei
What's hot
(20)
Course 102: Lecture 25: Devices and Device Drivers
Course 102: Lecture 25: Devices and Device Drivers
X / DRM (Direct Rendering Manager) Architectural Overview
X / DRM (Direct Rendering Manager) Architectural Overview
I2C Drivers
I2C Drivers
Pod density comparison: VMware vSphere with Tanzu vs. a bare-metal approach ...
Pod density comparison: VMware vSphere with Tanzu vs. a bare-metal approach ...
Expert Day 2019 - SUSE Linux Enterprise 15
Expert Day 2019 - SUSE Linux Enterprise 15
Getting started with BeagleBone Black - Embedded Linux
Getting started with BeagleBone Black - Embedded Linux
Introduction to OpenStack Cinder
Introduction to OpenStack Cinder
Launch the First Process in Linux System
Launch the First Process in Linux System
Introduction to char device driver
Introduction to char device driver
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Embedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernel
Enabling new client operating systems in Uyuni. AlmaLinux as an example.
Enabling new client operating systems in Uyuni. AlmaLinux as an example.
Linux Networking Explained
Linux Networking Explained
Linux systems - Linux Commands and Shell Scripting
Linux systems - Linux Commands and Shell Scripting
Course 102: Lecture 19: Using Signals
Course 102: Lecture 19: Using Signals
Ansible: Infrastructure as Code for OpenShift
Ansible: Infrastructure as Code for OpenShift
Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0
U Boot or Universal Bootloader
U Boot or Universal Bootloader
Kubernetes CRI containerd integration by Lantao Liu (Google)
Kubernetes CRI containerd integration by Lantao Liu (Google)
Ixgbe internals
Ixgbe internals
Similar to Lessons Learned: Using Concourse In Production
LAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMG
Linaro
Kubernetes 101
Kubernetes 101
Stanislav Pogrebnyak
GeoServer Developers Workshop
GeoServer Developers Workshop
Jody Garnett
Why kernelspace sucks?
Why kernelspace sucks?
OpenFest team
Road to sbt 1.0 paved with server
Road to sbt 1.0 paved with server
Eugene Yokota
Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)
Eugene Yokota
OpenDataPlane Testing in Travis
OpenDataPlane Testing in Travis
Dmitry Baryshkov
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
GR8Conf
Leveraging Android's Linux Heritage at AnDevCon IV
Leveraging Android's Linux Heritage at AnDevCon IV
Opersys inc.
Parallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-Mode
Akihiro Suda
HPC on OpenStack
HPC on OpenStack
Erich Birngruber
Rejekts 24 EU No GitOps Pain, No Platform Gain
Rejekts 24 EU No GitOps Pain, No Platform Gain
Łukasz Piątkowski
Deep dive - Concourse CI/CD and Pipelines
Deep dive - Concourse CI/CD and Pipelines
Syed Imam
Ci for android OS
Ci for android OS
Jarek Potiuk
Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned
RightScale
Leveraging Android's Linux Heritage at AnDevCon VI
Leveraging Android's Linux Heritage at AnDevCon VI
Opersys inc.
Leveraging Android's Linux Heritage at AnDevCon V
Leveraging Android's Linux Heritage at AnDevCon V
Opersys inc.
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
Ambassador Labs
LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205
Linaro
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
Yaniv cohen
Similar to Lessons Learned: Using Concourse In Production
(20)
LAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMG
Kubernetes 101
Kubernetes 101
GeoServer Developers Workshop
GeoServer Developers Workshop
Why kernelspace sucks?
Why kernelspace sucks?
Road to sbt 1.0 paved with server
Road to sbt 1.0 paved with server
Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)
OpenDataPlane Testing in Travis
OpenDataPlane Testing in Travis
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
Leveraging Android's Linux Heritage at AnDevCon IV
Leveraging Android's Linux Heritage at AnDevCon IV
Parallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-Mode
HPC on OpenStack
HPC on OpenStack
Rejekts 24 EU No GitOps Pain, No Platform Gain
Rejekts 24 EU No GitOps Pain, No Platform Gain
Deep dive - Concourse CI/CD and Pipelines
Deep dive - Concourse CI/CD and Pipelines
Ci for android OS
Ci for android OS
Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned
Leveraging Android's Linux Heritage at AnDevCon VI
Leveraging Android's Linux Heritage at AnDevCon VI
Leveraging Android's Linux Heritage at AnDevCon V
Leveraging Android's Linux Heritage at AnDevCon V
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
Recently uploaded
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
Jhone kinadey
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
Delhi Call girls
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Steffen Staab
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
anilsa9823
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
Fatema Valibhai
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
kalichargn70th171
Software Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
Arshad QA
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
mohitmore19
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
kellynguyen01
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
anilsa9823
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
Wave PLM
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
MyIntelliSource, Inc.
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
ICS
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
Willy Marroquin (WillyDevNET)
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
ABDERRAOUF MEHENNI
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
OnePlan Solutions
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
ThousandEyes
Recently uploaded
(20)
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Software Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
Lessons Learned: Using Concourse In Production
1.
Using Concourse in
Production - Lessons Learned - Shingo Omura(@everpeace) omura@chatwork.com ChatWork, Inc. Concourse Meetup #5 2017/03/13
2.
© ChatWork All
rights reserved.© ChatWork All rights reserved. Outline ● About ChatWork ● Our Context From the Point of View of Infrastructure ● Our Use Case ● Good parts ● Pipeline Tips ● Small Bad parts (expect to improve) 2
3.
© ChatWork All
rights reserved. Group Chat File Sharing Task Management Video Conference About ChatWork ~Group Chat for Global Teams~ 3
4.
© ChatWork All
rights reserved. ChatWork is growing rapidly ● 127,000 organizations ○ number of users is not opened ● 205 countries or regions ● 6 languages supported as of 2017/02 4
5.
Our Context From the
Point of View of Infrastructure
6.
© ChatWork All
rights reserved. New Infrastracture Project (1/2) ● Current Infra ○ EC2 based apps, deploy servers(for capistorano) ○ Jenkins servers for CI/CD ● Pain points ○ Ops team doesn’t scale ■ release always have to be done with Infra team members ○ AWS env and Jenkins are hard to sandboxing ■ part of aws resouces are managed by terraform, but not all ■ deployment flow is hard to develop and testing 6
7.
© ChatWork All
rights reserved. New Infrastracture Project (2/2) ● Next Infra ○ Kubernetes and Helm with Dockerized apps ○ Concourse CI for CI/CD ● Benefits ○ Kubernetes accelarate DevOps ■ App team can fully manage their deployment cycle by themselves. ■ minikube is really helpful for local dev environemnt. ■ kubernetes team can focus on reliability of Kuberentes. ○ Concourse CI does too! ← Today’s Focus ■ reduces operational load ■ helps agile development of deployment/testing process ● Status ○ Using from new messaging backend (released the last december) ○ Current system is planned to migrate to this next infra 7
8.
Our Use Case
9.
© ChatWork All
rights reserved. Overview of deployment system ● Concourse is deployed by concourse-aws ○ maintained by @mumoshu (my-colleague) and @everpeace (me) ● Branching model is Gitlab flow with Environment Branches ● chatwork-notify-resource for notification staging branch staging environment production environment master branch push im age build and deploy helm package build and deploy helm packagepush image pull image pull image notify 9
10.
© ChatWork All
rights reserved. Our build pipeline environment can be split by ‘groups’ notification resource 10
11.
© ChatWork All
rights reserved. Our build pipeline test&build jobs deploy jobs rollback jobs 11
12.
Good Parts Learned
13.
© ChatWork All
rights reserved. Good Parts ● concourse.ci is extemely well-documented ○ You can start trying concourse in 5 min. ■ virtualbox and vagrant: just ‘vagrant up’!! ■ docker-compose support!! ○ easty to write pipelines thanks to comrehensive reference ● easy to deploy & version up (thanks to concourse-aws :-P ) ○ initial deploy: 3 steps ■ ‘build-amis.sh’ → edit ‘cluster.yml’ → ‘concourse-aws up’ ○ version up: similar 3 steps ■ ‘build-amis.sh’(new version) → edit ‘cluster.yml’(new ami) → ‘concourse-aws up’ 13
14.
© ChatWork All
rights reserved. Good Parts (cont.) ● Concourse frees us from ”plugin hell” ○ all resource is provided by docker image ○ task environment can be injected by docker image too ○ no need to manage backups of CI servers!! ● Multi tenancy ‘team’ support ■ multiple team can share CI server resources ■ but isolated appropriately ■ each app team can have controll in their team ● Various authentication scheme support ■ concourse need not to have user database ■ we use github authentication 14
15.
© ChatWork All
rights reserved. ● easy to develop pipelines ○ Pipeline developed & tested in local env can be deployed directly to production concourse ■ Concourse CI’s pipeline is stateless and reproductive ■ Concourse & Kubernetes both supports local env (minikube & concourse vagrant box) Good Parts (cont.) 15
16.
© ChatWork All
rights reserved. Good Parts (cont.) ● easy to extend/custom ○ easy to develop custom resource. ■ you only need to develop 3 commands(check, in, out) whose returns json objects. ■ language agnostic! you can choose your own language!! ○ easy to prepare task environment ■ when you need some task environment in which some toolkit is installed, you just push docker image to any repository and specify the image to your task definition task.yml --- platform: linux image_resource: type: docker-image source: repository: /yourown/image tag: '1.1' 16
17.
Pipeline tips Learned
18.
© ChatWork All
rights reserved. Pipeline tips: summary ● Use groups for large pipeline ● Use aggregate for running in parallel (useful for resources) ● Use “[ci skip]” keyword to commit message when Concourse commits/push to git repo ● on_success/on_failure hook is useful for notification ● input_mapping/output_mapping is useful for shared task definition ● use attempts for deployment task due to intermittent network failure ● @making’s trick is helpful for build caches(sbt, ivy, maven) 18
19.
© ChatWork All
rights reserved. Pipeline Tips ● Use groups for large pipeline to group many jobs ● Use aggregate for multiple resources (useful for resources) pipeline.yml groups: - name: master jobs: - job-for-master - name: production jobs: - job-for-production pipeline.yml plan: - aggregate: - get: app-repo trigger: true - get: tool-repo - get: sbt-ivy-cache those 3 get runs in parallel 19
20.
© ChatWork All
rights reserved. Pipeline Tips ● Use “[ci skip]” keyword to commit message when Concourse commits/push to git repo ○ git resource skip commits with [ci skip] keywords ○ It’s really useful when ■ back merge: “merging release branch to develop branch” ● the commit is wanted to skip CI process ■ the commit bumping versions ● when using sbt, version number is embedded to repo 20
21.
© ChatWork All
rights reserved. ● on_success/on_failure hook is useful for notification Pipeline Tips pipeline.yml - task: deploy-write-api-to-dev-kube file: foo/task.yml on_success: task: chatwork-notification file: tasks/notify_chatwork.yml on_failure: task: chatwork-notification file: tasks/notify_chatwork.yml on_failure on_success 21
22.
© ChatWork All
rights reserved. ● input_mapping/output_mapping is useful for shared task definition Pipeline Tips pipeline.yml - task: test-pull-request file: pull-request/ci/tasks/unit.yml input_mapping: { repo: pull-request } - task: unit file: master/ci/tasks/unit.yml input_mapping: { repo: master } ci/tasks/unit.yml --- platform: linux image_resource: type: docker-image source: repository: yourown/toolbox inputs: - name: repo run: path: /bin/bash args: - repo/ci/tasks/unit.sh 22
23.
© ChatWork All
rights reserved. ● use attempts for deployment task due to intermittent network failure Pipeline Tips pipeline.yml ... - task: deploy-write-api-to-dev-kube file: ..snip../deploy-to-kube-helm.yml attempts: {{attempts}} attempts=3 23
24.
© ChatWork All
rights reserved. ● @making’s trick is helpful for build caches(sbt, ivy, maven) ○ prepare own cache docker image repo (anywhere) ○ archives cache files as rootfs.tar and push it directly to the image repo ○ related issue is now open: Caching directories between runs of a task #230 Pipeline Tips 24
25.
Small Bad Parts (expect
to improve)
26.
© ChatWork All
rights reserved. Small Bad Parts (expect to improve) ● No fine-grained authorization (No role based aaccess control) ○ every team member can take full controll in the team ○ ‘fly get-pipeline’ exposes all creadentials embedded in pipelines ○ We sometime want to split ■ people who can write/read pipeline ■ people who can just view logs and trigger jobs (no rights to change pipelines but can just operate the pipeline) ○ related issues are open ■ Credential management #19 ■ Individual/fine-grained access control #23 26
27.
© ChatWork All
rights reserved. Small Bad Parts (expect to improve)(cont.) ● No parameterized job ○ we would like to deploy specific feature branch to shared dev environment ○ How could do this with Concourse?? Any Idea?? ○ git-multibranch-resource could achive similar thing ■ branch name convention which will be deployed to shared dev env should be agreed ○ Perhaps `fly exec` prompts user input? 27
28.
© ChatWork All
rights reserved. Small Bad Parts (expect to improve)(cont.) ● No Docker Compose in task ○ the issue is now open: Docker Compose support in Task definitions #324 ■ integration test task with app & local db containers ● FYI: various improvements are disscued in https://github.com/concourse/design-notes/issues 28
29.
Thank you for
Listening!!
30.
We’re Hiring!!! Search “ChatWork”
in Wantedly https://www.wantedly.com/companies/chatwork/projects
Download now