Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Serverless orchestration and automation with Cloud Workflows

118 views

Published on

Join this session to understand how Cloud Workflows resolves challenges in connecting services, HTTP based service orchestration and automation. We are going to dive deep how serverless HTTP service automation works to automate step engines. Based on practical examples we will demonstrate the built-in decision and conditional executions, subworkflows, support for external built-in API calls, and integration with any Google Cloud product without worrying about authentication. We are going to cover Marketing, Retail, Industrial and Developer possibilities, such as event driven marketing workflow execution, or inventory chain operations, generating and automatic state machines, or orchestrate DevOps workflows and automating the Cloud.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Serverless orchestration and automation with Cloud Workflows

  1. 1. Serverless orchestration and automation with GCPWorkflows December 2020 talk for GDG Cloud Portland Márton Kodok / @martonkodok Google Developer Expert at REEA.net
  2. 2. ● Among the Top 3 romanians on Stackoverflow 185k reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery + Redis database engine expert Slideshare: martonkodok Twitter: @martonkodok StackOverflow: pentium10 GitHub: pentium10 Serverless orchestration and automation with GCP Workflows @martonkodok About me
  3. 3. 1. Challenges in connecting services 2. What is Workflows? - HTTP based service orchestration and automation 3. Introduction to Workflows - automate complex processes 4. Practical use cases 5. Automate, orchestrate and provide reliable line-of-business automation 6. Conclusions Agenda Serverless orchestration and automation with GCP Workflows @martonkodok
  4. 4. Connectivity - should be easy, but in reality you need to figure out ● Common connection format ● Make the connections ● Parse the results ● Decisions and conditional step executions ● Error handling, logging ● Retries ● Scaling up and down to zero ● Authentication Challenges in connecting services Serverless orchestration and automation with GCP Workflows @martonkodok
  5. 5. Meet Workflows https://cloud.google.com/workflows Serverless orchestration and automation with GCP Workflows @martonkodok
  6. 6. Workflows in Google Cloud portfolio Introduction Orchestrate any Google Cloud API, SaaS API or private APIs. Serverless Compute External API’s Google API’s etc... Workflows - orchestrate & integrate SaaS API’s Private API’s Other Clouds
  7. 7. Step-Automation-as-a-Service - Serverless HTTP service automation Declarative workflow language (YAML, JSON) Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020 Built-in decision and conditional executions expression formulas, operation on var Subworkflows similar to routine in a programming language with input/return var Support for external API calls out of the box support outside of Google Cloud Integrates with any Google Cloud product without worrying about authentication What is GCP Workflows? Serverless orchestration and automation with GCP Workflows @martonkodok
  8. 8. OAuth, OIDC, Secret Manager integration Enterprise Security Keep your workflows secure X Authenticated Invocations Authenticated calls to Google Cloud services Integration with Secret Manager Encryption at rest and in transit External API 1 External API 2 External API
  9. 9. Code example @martonkodok - callMyFunction: call: http.get args: url: https://us-central1-project123.cloudfunctions.net/... query: metric: NoOfTrs result: metricResult - saveResult: switch: - condition: ${ metricResult.body.TrNo > 100 } call: http.post args: url: https://mydeployment.notify.... body: Metric: ${ metricResult.body.TrNo } getMetric TrNo>100? End YesNo {“metric”:”NoOfTrs”} Notify
  10. 10. hello.yaml Anatomy of a Cloud Workflow! Serverless orchestration and automation with GCP Workflows @martonkodok workflowRevisionId: 000001-de1 argument: 'null' startTime: '2020-10-16T20:19:34.448323739Z' name: projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c- 4065-a696-940bc7658c33 result: '"Hello, Cloud Workflows!"' endTime: '2020-10-16T20:19:34.845536315Z' state: SUCCEEDED Deploying gcloud beta workflows deploy hello --source=hello.yaml Executing gcloud beta workflows execute hello --data={“var”:”value”} Describing gcloud beta workflows executions describe-last
  11. 11. Sample Workflows Serverless orchestration and automation with GCP Workflows @martonkodok
  12. 12. E-commerce invoice generation with Workflows Steps orchestration Reliable execution, with error handling and retries Orchestration microservices or other API’s Create an invoice Generate PDF Send PDF via email
  13. 13. Sending reminders to accounts with overdue payments Process array elements Execute steps conditionally For each customer: Get a list of customers Send reminder overdue ? Yes
  14. 14. IT management automation Combine automation with scheduler Wait for service checks Orchestrate work across Compute Engine, PubSub, Stackdriver and other Google Cloud Products 9 AM trigger Start a Compute Engine VM Log the event App Started? No Wait 60 seconds Notify the team
  15. 15. Code Examples Serverless orchestration and automation with GCP Workflows @martonkodok
  16. 16. HTTP Post Sequence two steps HTTP Post combined with Secret Manager credentials Switch block Working with subworkflows Code Examples Serverless orchestration and automation with GCP Workflows @martonkodok
  17. 17. http_post.yaml Making an external HTTP POST request Serverless orchestration and automation with GCP Workflows @martonkodok { "archived":false, "created_at":"2020-10-16T17:40:17+0000", "id":"bit.ly/35452TM", "link":"https://bit.ly/35452TM", "long_url":"<truncated>", }
  18. 18. wikipedia.yaml Sequence two steps to get data from Wikipedia Serverless orchestration and automation with GCP Workflows @martonkodok
  19. 19. switch.yaml Switch block Serverless orchestration and automation with GCP Workflows @martonkodok 9 13 14 9 13 14
  20. 20. subworkflow.yaml Subworkflow Serverless orchestration and automation with GCP Workflows @martonkodok call call def
  21. 21. retries.yaml Retries Serverless orchestration and automation with GCP Workflows @martonkodok Retries HTTP status codes [429, 502, 503, 504], connection error, or timeout
  22. 22. Orchestrate and automate the Cloud Serverless orchestration and automation with GCP Workflows @martonkodok
  23. 23. Using Workflows to load Cloud Storage files into BigQuery - Cloud Workflows - Cloud Storage - BigQuery Full guide article on: martonkodok.medium.com Cloud Storage BigQueryCloud Workflows Authenticated Invocations foreach
  24. 24. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Build Shell command Serverless Secure Connect Exec command Firewall Compute Engine Cloud IAP Tunnel Cloud Workflows
  25. 25. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Workflow Steps Cloud Workflows Execution Authorize Cloud Build Roles and Permissions Cloud IAM Shell Service Account Start VM Compute Engine Submit Stop VM Compute Engine Firewall Compute Engine Cloud IAP Tunnel wait wait
  26. 26. Conclusions Serverless orchestration and automation with GCP Workflows @martonkodok
  27. 27. Reliable workflow execution - execute workflows for enterprise business apps Low latency of execution - no cold starts Built-in error handling out of the box error handling with configurable retry policies Passing variable built-in JSON parsing and expression-based variable manipulation Rich runtime iterating through an array, embedded steps for readability Secret Manager integration out of the box Cloud Logging out of the box integration with Cloud Logging Reading from Firestore read/write an entry using Yaml syntax Benefits of Cloud Workflows Serverless orchestration and automation with GCP Workflows @martonkodok
  28. 28. “Automate, orchestrate and provide reliable line-of-business automation. Serverless orchestration and automation with GCP Workflows @martonkodok Google Cloud Workflows
  29. 29. “Enables kids to build their first serverless product by using only YAML language. Serverless orchestration and automation with GCP Workflows @martonkodok Google Cloud Workflows
  30. 30. Easy to build/operate Scales out Does not lose state Handles errors/timeouts Out-of-the-box support of Cloud APIs Auditable Developer friendly Serverless orchestration and automation with GCP Workflows @martonkodok
  31. 31. The possibilities are endless Marketing Retail IndustrialandIoT Developer Event driven marketing workflow execution Relay conversions to customer profiles in external services Workflow based emails, discounts, promotions Order management Inventory chain operations Data gathering and processing Synchronize systems Generate state machines Verify equipment lifecycle Workflow based maintenance needs Digitalization of internal policies Automate the Cloud Shell-script replacement Orchestrate devops workflows @martonkodok
  32. 32. Thank you. Q&A. Slides available on: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.

×