Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

•

7 likes•5,025 views

This was my talk at QConNY 2018, which was part of the container orchestration track, about how Chick-fil-A uses Kubernetes at the Edge in our restaurants and how we have engineered some solutions to solve problems that are unique to our scale.

Technology

Milking the most
out of thousands of
Kubernetes clusters

What to expect from the session
• Intro
• How is CFA using K8s?
• What does our
architecture look like?
• How are we
engineering around
K8s for our business?
• Q&A

AT PEAK HOUR
1 sandwich every 16 seconds
1 box of nuggets every 25 seconds
1 order of waffle fries every 14 seconds
1 car through the drive thru every 22 seconds
267 total transactions

Chick-fil-A Architecture (2017)
MSGing
Web
Server
Local
AuthEdge
Cloud
Event
Fwding
Apps
…
Local Persistence/Storage
Connectivity
Analytics Management
Things
OAuth Server MQTT
Edge Tools

Chick-fil-A Architecture (Today)
MSGing
Local
Auth
Edge
Cloud
Event
Fwd
Apps
…
Local Persistence/Storage
Connectivity
Analytics Management
Things
OAuth Server MQTT Fleet

Why Containers? Why Kubernetes?
Idea Code
Production
Code
Value
Impact
Optimize for

North American Data Centers
Google
Cloud
AWSAzure

North American Data Centers
Google
Cloud
AWSAzure
Cloud-fil-A

Restaurant “Data Centers”
Intel: Quadcore processor, 8 GB RAM, SSD

Engineering Around K8s
• How we build and repair bare
metal clusters
• SRE Lessons Learned
• How we deploy applications to
thousands of clusters

Challenges of Bare Metal K8s clustering at scale
• Goal: #code2prod
• Simple enough for a non-
technologist to install
• Manageable remotely
• Automated device discovery
and self-clustering
• Self healing & HA

How we Bare Metal Cluster K8s at scale
Highlander Hooves Up
TOOLS
Sherlock FleetRKEImage
PROCESS

Bootstrapping Clusters
• Highlander
– Node coordination and
clustering leader election
using UDP
– Execute clustering (RKE)
– Swap KubeDNS for CoreDNS
– Base OAuth identity
negotiation
– Controller Pods (control
plane activity/Istio)

Initializing Clusters
What we considered
• Kops = love it, no bare metal
• Kubespray = slow + brittle
• kubeadmin = maybe in the future
• RKE = fairly simple, works for us
Future State?
• Stick w/ RKE, Kubeadmin, or roll our own to meet our needs

Resetting Cluster State
• Requirement: Need to be
able to re-image remotely
• Solution: Overlay FS + HAMS
– Manages wiping clusters
and restoring to base

Hooves Up
• Self-healing AWS SSM
Registration
• Free even for non-AWS
deployments
• Able to do remote
commands and patch
reporting/management

Lessons learned
• Use K8s feature set and don’t reinvent the wheel
• MVP. MVP. MVP.
• Ensure aggregated and searchable logging
• Deep health checks are a must --> Use /healthz
• Every service needs “/metrics”
endpoint

How do we deploy to our restaurants?
• Large number of
deployment targets
• Complex success/fail
criteria
• Array of application types
• What approaches did we
consider?
kubectl
/

Introducing Fleet
• Design Goals
– Simple to use / reason about
– Use declarative approach
– Support for variety of deployment
models (canary, blue/green)
– Rollout over flexible time period
– Sane rollback behaviors
– Leverage standard k8s API
– Full visibility

Fleet Ecosystem Components
• Fleet Client
– Git webhook, REST call, CLI
• Fleet Server API
– Code generation for
deployment, service,
ingress files
– Git management for cluster
repositories
– Deployment status tracking
• Atlas
– Repository of deploy-ready,
k8s compliant application
files
• Vessel
– Deployed on cluster, git
pull, kubectl apply, report
status
• Dashboards

Application Configuration
HTTP POST Request

Where you can find us
www.linkedin.com/in/brian-chambers
www.linkedin.com/in/calebrhurd
@brianchambers21
@calebrhurd
https://medium.com/@cfatechblog
https://github.com/chick-fil-a

What's hot

Taking a Crawl-Walk-Run Approach to Office 365 Retention - Ottawa SPUG (no de...

Joanne Klein

Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Ethan Guo | Current 2022 Back in 2016, Apache Hudi brought transactions, change capture on top of data lakes, what is today referred to as the Lakehouse architecture. In this session, we first introduce Apache Hudi and the key technology gaps it fills in the modern data architecture. Bridging traditional data lakes and warehouses, Hudi helps realize the Lakehouse vision, by bringing transactions, optimized table metadata to data lakes and powerful storage layout optimizations, moving them closer to cloud warehouses of today. Viewed from a data engineering lens, Hudi also plays a key unifying role between the batch and stream processing worlds, by acting as a columnar, server-less ""state store"" for batch jobs, ushering in what we call the incremental processing model, where batch jobs can consume new data, update/delete intermediate results in a Hudi table, instead of re-computing/re-write entire output like old-school big batch jobs. Rest of talk focusses on a deep dive into the some of the time-tested design choices and tradeoffs in Hudi, that helps power some of the largest transactional data lakes on the planet today. We will start by describing a tour of the storage format design, including data, metadata layouts and of course Hudi's timeline, an event log that is central to implementing ACID transactions and concurrency control. We will delve deeper into the practical concurrency control pitfalls in data lakes, and show how Hudi's hybrid approach combining MVCC with optimistic concurrency control, lowers contention and unlocks minute-level near real-time commits to Hudi tables. We will conclude with code examples that showcase Hudi's rich set of table services that perform vital table management such as cleaning older file versions, compaction of delta logs into base files, dynamic re-clustering for faster query performance, or the more recently introduced indexing service that maintains Hudi's multi-modal indexing capabilities.

Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...

HostedbyConfluent

Learn how to utilize AWS services for serverless processing, such as AWS Lambda and the Amazon API Gateway, to facilitate secure development and deployment of a HL7 FHIR interface. Using the AWS code tools for development, such as AWS CodeBuild and AWS CodeCommit, we walk through several message examples and show you how to address common API development needs and deploy a sample interface for HL7 FHIR on AWS. We use AWS X-Ray to take a look inside the sample interface and examine the path of calls. Also, hear from Cerner on their experience implementing a HL7 FHIR gateway on AWS.

AWS Serverless Interface Building and Cerner's FHIR Experience (HLC401) - AWS...

Amazon Web Services

MQ, ETL and ESB middleware are often used as integration backbone between legacy applications, modern microservices and cloud services. This introduces several challenges and complexities like point-to-point integration or non-scalable architectures. This session discusses how to build a completely event-driven streaming platform leveraging Apache Kafka’s open source messaging, integration and streaming components to leverage distributed processing, fault-tolerance, rolling upgrades and the ability to reprocess events. Learn the differences between a event-driven streaming platform leveraging Apache Kafka and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

confluent

How to Lock Down Apache Kafka and Keep Your Streams Safe

confluent

Introduction to Apache Kafka

Jeff Holoman

ClearPass Guest Overview

Aruba, a Hewlett Packard Enterprise company

Using ClickHouse for Experimentation

Gleb Kanterov

jmp206 - Lotus Domino Web Services Jumpstart

Bill Buchan

Apache NiFi in the Hadoop Ecosystem

DataWorks Summit/Hadoop Summit

Is your wireless network more secure than your wired network? In this session, we'll discuss how to use industry standard techniques to provide secure wired access. This includes using policies and RADIUS/RADIUS CoA to ensure that ports used for Wi-Fi APs, IoT devices, printers and IP phones are protected against unwanted use by employees, guests, and contractors. The days of dedicated ports assigned to VLANs are over! By using a 'universal ports tied to policies’ model, network access is based on dynamic enforcement rules. These techniques work across popular wired infrastructure from HPE, Cisco, and others.

Securing the LAN Best practices to secure the wired access network

Aruba, a Hewlett Packard Enterprise company

A Kafka journey and why migrate to Confluent Cloud?

confluent

Netflix Studio spent 8 Billion dollars on content in 2018. When the stakes are so high, it is paramount to track changes to the core studio metadata, spend on our content, forecasting and more to enable the business to make efficient and effective decisions. Embracing a Kappa architecture with Kafka enables us to build an enterprise grade message bus. By having event processing be the de-facto paved path for syncing core entities, it provides traceability and data quality verification as first class citizens for every change published.This talk will also get into the nuts and bolts of the eventing and stream processing paradigm and why it is the best fit for our use case, versus alternative architectures with similar benefits We will do a deep dive into the fascinating world of Netflix Studios and how eventing and stream processing are revolutionizing the world of movie productions and the production finance infrastructure.

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

confluent

We have covered the need for CDC and the benefits of building a CDC pipeline. We will compare various CDC streaming and reconciliation frameworks. We will also cover the architecture and the challenges we faced while running this system in the production. Finally, we will conclude the talk by covering Apache Hudi, Schema Registry and Debezium in detail and our contributions to the open-source community.

Building robust CDC pipeline with Apache Hudi and Debezium

Tathastu.ai

[IGNITE2018] [BRK2495] What’s new in Microsoft Information Protection solutio...

☁️ Gustavo Magella

Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of _temporal_ join semantics. For joins, timestamps are as important as the actual data and it is important to understand how they impact the join result. In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall ""time engine"" of the Kafka Streams query runtime and explain its relationship to operator semantics. To allow developers to apply their knowledge on temporal join semantics, we provide best practices, tip and tricks to ""bend"" time, and configuration advice to get the desired join results. Last, we give an overview of recent, and an outlook to future, development that improves joins even further.

Temporal-Joins in Kafka Streams and ksqlDB | Matthias Sax, Confluent

HostedbyConfluent

Cisco Live Brksec 3032 - NGFW Clustering

ib_cims

We’re feeling the growing pains of maintaining a large data platform. Last year we went from 50 to 150 unique data feeds by adding them all by hand. In this talk we will share the best practices developed to handle our 300% increase in feeds through self service. Having self-service capabilities will increase your teams velocity and decrease your time to value and insight. * Self service data feed design and ingest * configuration management * automatic debugging * light weight data governance

Data Ingest Self Service and Management using Nifi and Kafka

DataWorks Summit

Fiware IoT_IDAS_intro_ul20_v2

FIWARE

Dataflow with Apache NiFi

DataWorks Summit/Hadoop Summit

What's hot (20)

Taking a Crawl-Walk-Run Approach to Office 365 Retention - Ottawa SPUG (no de...

Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...

AWS Serverless Interface Building and Cerner's FHIR Experience (HLC401) - AWS...

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

How to Lock Down Apache Kafka and Keep Your Streams Safe

Introduction to Apache Kafka

ClearPass Guest Overview

Using ClickHouse for Experimentation

jmp206 - Lotus Domino Web Services Jumpstart

Apache NiFi in the Hadoop Ecosystem

Securing the LAN Best practices to secure the wired access network

A Kafka journey and why migrate to Confluent Cloud?

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Building robust CDC pipeline with Apache Hudi and Debezium

[IGNITE2018] [BRK2495] What’s new in Microsoft Information Protection solutio...

Temporal-Joins in Kafka Streams and ksqlDB | Matthias Sax, Confluent

Cisco Live Brksec 3032 - NGFW Clustering

Data Ingest Self Service and Management using Nifi and Kafka

Fiware IoT_IDAS_intro_ul20_v2

Dataflow with Apache NiFi

Similar to Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

Did your company start down the path of building a cloud native platform using Kubernetes with the goal of enabling developers to innovate faster and increase productivity, but then run into challenges keeping it operating in an optimal way? In this session, Weaveworks will discuss how to migrate from self-managed Kubernetes on EC2 to a GitOps managed Shared Services Platform (SSP) on EKS. A SSP built on EKS and managed with Weave GitOps provides developers and operators with common workflows to update both applications and infrastructure. With every change in version control, full audit trails are available, and security is enforced. While at the same time enabling easier rollbacks and faster mean-time-to-recovery (MTTR). In short, a Weave GitOps managed SSP increases developer velocity while boosting stability. How to operate a hybrid Kubernetes architecture, using managed EKS in the AWS Cloud and EKS-Distro on premises. How to structure your infrastructure repository to efficiently manage multiple teams. How to use Kubernetes RBAC to provide secure cluster multi-tenancy. How to use GitOps to promote releases across a hybrid set of independent clusters. How to accomplish data and operational sovereignty.

Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS

Weaveworks

OneAPI Series 2 Webinar - 9th, Dec-20

Tyrone Systems

In 2015, Google open sourced the core of their internal container clustering system under the name Kubernetes. Teams that previously relied upon IaaS and PaaS to run their applications quickly adopted Kubernetes instead. Today, only a few years later, Kubernetes is key to many companies and runs applications with literally billions of users. Kubernetes has become the de facto standard for deploying and running cloud native applications. We’ll give an overview of what Kubernetes is today and share our experiences from using Kubernetes in an ecormmerce and an IoT application. The future of Kubernetes could not look better. The Kubernetes ecosystem is growing, allowing to provision professionally managed databases directly within the cluster, running functions in a serverless-fashion, and even allowing us to host the code, the build pipeline and the application itself on Kubernetes. In the future, there might be only one Kubernetes to rule them all.

One Kubernetes to rule them all (ZEUS 2019 Keynote)

Simon Harrer

Lc3 beijing-june262018-sahdev zala-guangya

Sahdev Zala

Deploying commercial applications that meet their expected business needs is challenging due to the differences between how business goals are specified and how the system is evaluated. Furthermore, business goals are dynamic, requiring deployment to change constantly over time. Such difficulties make it costly to maintain application quality as the underlying infrastructure is not always fast enough to keep up with business changes. Nowadays, serverless opens a new approach to build application. By abstracting out the deployment details, serverless application can be implemented with minimum deployment efforts. Serverless also reduces maintenance cost with auto-scaling and pay-as-you-go. Such abilities make us believe that by adopting serverless, we can build application that can meet and quickly adapt to business goals. However, simply writing applications with serverless is not sufficient. Due to best-effort invocation mechanisms and the lack of application structure awareness, serverless performance is highly variable and often fails to support applications with rigorous quality of service requirements. In this study, we aim to mitigate such limitations by coupling serverless deployment with business needs. In particular, we define an Serverless Service-Level Objective (SLO) interface that allows developers to describe their application structure and business goals in terms of software-level objectives. We implement an SLO enforcer, which uses this information in combination with the system performance metrics to decide a proper serverless deployment and resource allocation for meeting business goals. The Serverless SLO leverages blueprint model, which allow developers to describe applications' architecture and runtime characteristics needs, to map application description to serverless function deployment on the top of Knative. We deploy our proposed system on KinD, a tool to run Kubernetes cluster over our local Docker container, and evaluate it with different system configurations. Evaluation results showed that SLO definition and enforcement helps serverless application use resources in accordance with business goals.

Service-Level Objective for Serverless Applications

alekn

Data(?)Ops with CircleCI

Jinwoong Kim

Cost is often the conversation starter when customers think about moving to the cloud. AWS helps lower costs for customers through its “pay only for what you use” pricing model, frequent price drops, and pricing model choice to support variable & stable workloads. In this session, you will learn about the financial considerations of owning and operating a traditional data center or managed hosting provider versus utilizing AWS. We will detail our TCO methodology and showcase cost comparisons for some common customer use-cases. We’ll also cover a few AWS cost optimization areas, including Spot and Reserved Instances, EC2 Auto Scaling, and consolidated billing. Presenter: Amit Sharma, Solution Architect, Amazon Internet Services Krishnenjit Roy, Director IT Operations, Freshdesk

Managing Your Cloud Assets

Amazon Web Services

01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware

VMUG IT

Couchbase Connect 2016

Michael Kehoe

Kubernetes Community Growth and Use Case

Chris Gaun

Customer Sharing: HTC - What is in AWS Cloud for me?

Amazon Web Services

Kubernetes is a deep and complex technology that is evolving fast with new functionality and a growing ecosystem of cloud-native solutions. While the public cloud delivers an almost frictionless user experience, configuring and managing a production Kubernetes environment is an enormous technical challenge for the majority of enterprises that choose to do so on premises. Without the right approach, operationalizing Kubernetes in the data center can take upwards of 6 months, jeopardizing developer productivity and speed-to-market. In this webinar, you’ll learn from Nutanix cloud native experts on how to fast-track your way to operationalizing a production-ready Kubernetes environment on-prem. Specifically, we’ll talk about: How containerized applications use IT resources (and why legacy infrastructure isn’t built for Kubernetes); The main advantages of running Kubernetes on prem (as part of a multi-cloud strategy); Key aspects of Kubernetes lifecycle management that greatly benefit from automation.

Simplify Your Way To Expert Kubernetes Management

DevOps.com

BayInfotech (BIT) ACI Portfolio

Maulik Shyani

Introducing Kubernetes Concepts And Architecture PowerPoint Presentation Slides. This readily available open-source architecture PPT infographics well explains the concept of containers. You can also depict the architecture of containers and microservices with the help of a visually appealing PPT slideshow. Our content-ready containers PPT slideshow allow you to showcase the reasons for opting for Kubernetes by an organization. Depict the roadmap for installing Kubernetes in the organization in a presentable manner by using this slide design. The major advantages of Kubernetes, such as the stability of application run, improving productivity, and many more can be presented in this slide deck. Cover 30 60 90 days plan to implement Kubernetes in the organization with this thoroughly researched PowerPoint templates. Discuss the key components of Kubernetes with a diagram using this modern-designed cluster architecture PowerPoint layouts. Describe each element’s functionality using these PowerPoint visuals. Hence manage the clusters efficiently by downloading Kubernetes architecture PPT slides. https://bit.ly/3p6xEoS

Driving Digital Transformation With Containers And Kubernetes Complete Deck

SlideTeam

Database as a Service (DBaaS) on Kubernetes

ObjectRocket

Amazon Elastic Compute Cloud (Amazon EC2) provides resizable compute capacity in the cloud and makes web scale computing easier for customers. Amazon EC2 provides a wide variety of compute instances suited to every imaginable use case, from static websites to high performance supercomputing on-demand, available via highly flexible pricing options. Amazon EC2 works with Amazon Elastic Block Store (Amazon EBS) and Auto Scaling to make it easy for you to get the performance and availability you need for your applications. This session will introduce the key features and different instance types offered by Amazon EC2, demonstrate how you can get started and provide guidance on choosing the right types of instance and purchasing options.

Introduction to Amazon EC2

Amazon Web Services

This presentation provides the basic concepts of the Kubernetes for Beginners. 1) Introduction of Kubernetes Before Kubernetes What is Kubernetes What Kubernetes can do? What Kubernetes can't do? Features of Kubernetes Kubernetes Architecture Kubernetes vs Docker Swarm Kubernetes 7 use cases ... 2) Kubernetes Component What is Kubelet? What is Kubectl? What is Kubeadm? 3) Nodes in Kubernetes What is a node in Kubernetes? Master node Worker node 4) Kubernetes Development Process What is blue green deployment? How to automate the deployment? 5) Networking in Kubernetes Kubernetes networking model Ingress networking in Kubernetes 6) Security Measures in Kubernetes Best security measures in Kubernetes

Introduction of Kubernetes - Trang Nguyen

Trang Nguyen

AWS Sydney Summit 2013 - Big Data Analytics

Amazon Web Services

oci-container-engine-oke-100.pdf

NandiniSinghal16

Mathias Kriegel - IT Operations, Desigual Joan Anton Sances - Software Architect, Desigual Desigual, a $1-billion-dollar fashion retailer headquartered in Barcelona, operates over 500 stores worldwide. The company is on a digital transformation journey touching every aspect of the customer experience. In this session, IT Operations and Software Architecture teams, will explain how Desigual built an in-store “assistant shopping” that transformed the customer experience adopting modern architecture models leveraging Docker Enterprise for containerization. In the session, you’ll learn: ● How Desigual is leveraging containers with Docker Enterprise, micro services, API´s, CI/CD and hybrid cloud to create an excellent customer experience. ● How to use a container platform to accelerate time-to-market for new applications. ● How Desigual changed its traditional IT operational model, focusing on bringing a PaaS like model for Developer teams, and what they learned along the way. ● How Dev and Ops teams aligned together in the process. ● How Developer productivity increased by adopting modern architecture models.

DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...

Docker, Inc.

Similar to Chick-fil-A: Milking the most out of thousands of kubernetes clusteres (20)

Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS

OneAPI Series 2 Webinar - 9th, Dec-20

One Kubernetes to rule them all (ZEUS 2019 Keynote)

Lc3 beijing-june262018-sahdev zala-guangya

Service-Level Objective for Serverless Applications

Data(?)Ops with CircleCI

Managing Your Cloud Assets

01 - VMUGIT - Lecce 2018 - Fabio Rapposelli, VMware

Couchbase Connect 2016

Kubernetes Community Growth and Use Case

Customer Sharing: HTC - What is in AWS Cloud for me?

Simplify Your Way To Expert Kubernetes Management

BayInfotech (BIT) ACI Portfolio

Driving Digital Transformation With Containers And Kubernetes Complete Deck

Database as a Service (DBaaS) on Kubernetes

Introduction to Amazon EC2

Introduction of Kubernetes - Trang Nguyen

AWS Sydney Summit 2013 - Big Data Analytics

oci-container-engine-oke-100.pdf

DCEU 18: Desigual Transforms the In-Store Experience with Docker Enterprise C...

Recently uploaded

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

caitlingebhard1

The presentation was made in “Web3 Fusion: Embracing AI and Beyond” is more than a conference; it's a journey into the heart of digital transformation. The conference a provided a platform where the future of technology meets practical application. This three-day hybrid event, set in the heart of innovation, served as a gateway to the latest trends and transformative discussions in AI, Blockchain, IoT, AR/VR, and their collective impact on the information space.

AI in Action: Real World Use Cases by Anitaraj

AnitaRaj43

In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.

The Zero-ETL Approach: Enhancing Data Agility and Insight

Safe Software

Portal Kombat : extension du réseau de propagande russe

中央社

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

In today's digital world, trust is key to customer relationships, but keeping it is a huge challenge. Customers are well-informed and empowered, quick to change brands if their trust is broken, even if it costs them more. This puts a lot of pressure on organizations to handle trust and safety issues with great care and transparency. The challenge, however, is real. Fragmented solutions have left privacy, legal, and security teams in a perpetual cycle of catch-up, struggling to update privacy notices, manage customer data rights, and answer lengthy security questionnaires—all while trying to prove ROI to the business. It's a thankless job, filled with repetition, tedious tasks, and constant interdepartmental coordination. Combine this with fast regulatory changes and the quick evolution of AI, and it becomes overwhelming. Join this webinar to learn more about TrustArc's new innovative solution Trust Center, the only unified, no-code online hub for trust and safety information built for privacy, security, compliance, and legal teams. Trust Center streamlines your path to compliance, shortens the pre-sales cycle, and reduces both legal and regulatory risks, saving time, effort, and cost. This webinar will review: - Why companies are building unified Trust Centers for a robust privacy program. - How unified Trust Centers streamline sales cycles, ensure regulatory compliance, and reduce operational bottlenecks. - How compliance, legal, security, GRC, and privacy teams benefit from a unified Trust Center in terms of needs, pains, and outcomes. - How TrustArc Trust Center saves time and work while reducing legal, reputational, and compliance risk by effectively managing policies, notices, terms, and disclosures, and providing real-time updates on subprocessors.

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...

WSO2

Design and Development of a Provenance Capture Platform for Data Science

Paolo Missier

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

rightmanforbloodline

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

ERP Contender Series: Acumatica vs. Sage Intacct

BrainSell Technologies

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Overview of Hyperledger Foundation

Hyperleger Tokyo Meetup

Working together SRE & Platform Engineering

Marcus Vechiato

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

Introduction to use of FHIR Documents in ABDM

Kumar Satyam

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx

FIDO Alliance

DBX First Quarter 2024 Investor Presentation

Dropbox

Recently uploaded (20)

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

AI in Action: Real World Use Cases by Anitaraj

The Zero-ETL Approach: Enhancing Data Agility and Insight

Portal Kombat : extension du réseau de propagande russe

Introduction to Multilingual Retrieval Augmented Generation (RAG)

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...

Design and Development of a Provenance Capture Platform for Data Science

WSO2's API Vision: Unifying Control, Empowering Developers

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

Platformless Horizons for Digital Adaptability

ERP Contender Series: Acumatica vs. Sage Intacct

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Overview of Hyperledger Foundation

Working together SRE & Platform Engineering

CNIC Information System with Pakdata Cf In Pakistan

Introduction to use of FHIR Documents in ABDM

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx

DBX First Quarter 2024 Investor Presentation

Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

1. Milking the most out of thousands of Kubernetes clusters

2. What to expect from the session • Intro • How is CFA using K8s? • What does our architecture look like? • How are we engineering around K8s for our business? • Q&A

3. Internet of Things: Why?

4. AT PEAK HOUR 1 sandwich every 16 seconds 1 box of nuggets every 25 seconds 1 order of waffle fries every 14 seconds 1 car through the drive thru every 22 seconds 267 total transactions

5. Chick-fil-A Architecture (2017) MSGing Web Server Local AuthEdge Cloud Event Fwding Apps … Local Persistence/Storage Connectivity Analytics Management Things OAuth Server MQTT Edge Tools

6. Chick-fil-A Architecture (Today) MSGing Local Auth Edge Cloud Event Fwd Apps … Local Persistence/Storage Connectivity Analytics Management Things OAuth Server MQTT Fleet

7. Why Containers? Why Kubernetes? Idea Code Production Code Value Impact Optimize for

8. Accelerate

9. North American Data Centers Google Cloud AWSAzure

10. North American Data Centers Google Cloud AWSAzure Cloud-fil-A

11. Restaurant “Data Centers” Intel: Quadcore processor, 8 GB RAM, SSD

12. Engineering Around K8s • How we build and repair bare metal clusters • SRE Lessons Learned • How we deploy applications to thousands of clusters

13. Challenges of Bare Metal K8s clustering at scale • Goal: #code2prod • Simple enough for a non- technologist to install • Manageable remotely • Automated device discovery and self-clustering • Self healing & HA

14. How we Bare Metal Cluster K8s at scale Highlander Hooves Up TOOLS Sherlock FleetRKEImage PROCESS

15. Bootstrapping Clusters • Highlander – Node coordination and clustering leader election using UDP – Execute clustering (RKE) – Swap KubeDNS for CoreDNS – Base OAuth identity negotiation – Controller Pods (control plane activity/Istio)

16. Initializing Clusters What we considered • Kops = love it, no bare metal • Kubespray = slow + brittle • kubeadmin = maybe in the future • RKE = fairly simple, works for us Future State? • Stick w/ RKE, Kubeadmin, or roll our own to meet our needs

17. Resetting Cluster State • Requirement: Need to be able to re-image remotely • Solution: Overlay FS + HAMS – Manages wiping clusters and restoring to base

18. Hooves Up • Self-healing AWS SSM Registration • Free even for non-AWS deployments • Able to do remote commands and patch reporting/management

19. Lessons learned • Use K8s feature set and don’t reinvent the wheel • MVP. MVP. MVP. • Ensure aggregated and searchable logging • Deep health checks are a must --> Use /healthz • Every service needs “/metrics” endpoint

20. How do we deploy to our restaurants? • Large number of deployment targets • Complex success/fail criteria • Array of application types • What approaches did we consider? kubectl /

21. Introducing Fleet • Design Goals – Simple to use / reason about – Use declarative approach – Support for variety of deployment models (canary, blue/green) – Rollout over flexible time period – Sane rollback behaviors – Leverage standard k8s API – Full visibility

22. Fleet Ecosystem Components • Fleet Client – Git webhook, REST call, CLI • Fleet Server API – Code generation for deployment, service, ingress files – Git management for cluster repositories – Deployment status tracking • Atlas – Repository of deploy-ready, k8s compliant application files • Vessel – Deployed on cluster, git pull, kubectl apply, report status • Dashboards

23. Sample Templates

24. Fleet Walk Thru/Demo PLACEHOLDER

25. Application Configuration HTTP POST Request

26. K8s config example

27. Atlas

28. Fleet Walk Thru/Demo PLACEHOLDER

29. Where you can find us www.linkedin.com/in/brian-chambers www.linkedin.com/in/calebrhurd @brianchambers21 @calebrhurd https://medium.com/@cfatechblog https://github.com/chick-fil-a

Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Chick-fil-A: Milking the most out of thousands of kubernetes clusteres

Similar to Chick-fil-A: Milking the most out of thousands of kubernetes clusteres (20)

Recently uploaded

Recently uploaded (20)

Chick-fil-A: Milking the most out of thousands of kubernetes clusteres