Prometheus and Thanos

•

2 likes•1,270 views

CloudOps2005

This document discusses using Thanos for long term storage of metrics data collected by Prometheus at Ticketmaster. It describes how Thanos provides global visibility of metrics across datacenters, high availability of data through replication, and increased data retention compared to using just Prometheus by storing data in object storage like S3. The architecture includes Thanos query and store nodes, compaction of data, and integration with Prometheus through the store API. Thanos was chosen over alternatives like Cortex due to its simpler deployment model and minimal dependencies.

Technology

Ticketmaster confidential. Do not distribute.
Ticketmaster confidential. Do not distribute.
Prometheus +
Thanos
Thanos long term storage for Prometheus

Ticketmaster confidential. Do not distribute.
About me...
• Rocking it at Ticketmaster
• Student in Master of Business Administration (MBA) Strategic project management
• Bachelor of Applied Science (B.A.Sc.) in Computer Science
• Devops, Cloud, Kubernetes, Kafka, etc.
• Sport lover (Hockey, Dek Hockey, Ultimate Frisbee, Football…)
2
SECTION

Ticketmaster confidential. Do not distribute.
Where Prometheus & Thanos are in the CNCF landscape...
3
Prometheus

Ticketmaster confidential. Do not distribute.
Insert photography and crop to size of grey box.
(24.25 in. x 7.21 in.)
What is Prometheus?
4
Prometheus
Prometheus was the second project to be graduated of the Cloud Native Computing Foundation.

Ticketmaster confidential. Do not distribute.
Prometheus Architecture

Ticketmaster confidential. Do not distribute.
How we are using Prometheus at Ticketmaster?
• We are using the Prometheus-Operator made by CoreOS (RedHat (IBM))
• We have created a Helm chart that created common pulling jobs, settings and exporters
• Scrape EC2 instances based on Tags, Kubernetes
• Exporters like cloudwatch-metrics, kafka, blackbox
• Ingresses
• Thanos
• Federations
6
Prometheus

Ticketmaster confidential. Do not distribute.
Why moving away from the federation?
• Calculate the right ingestion rate of our scrape job is not really easy
• When the disk is full
• Prometheus stop working
• We needed to delete the PVC to recreated it bigger.
• Long term storage is costly SSD == $$$$
• Single point of failure
• Availability
• Operator error
• Hardware failure
• Rollout
7
Prometheus

Ticketmaster confidential. Do not distribute.
Solution

Ticketmaster confidential. Do not distribute.
Goals
9
Thanos
- Easy Deployment model
- Minimal number of dependencies
- Minimal baseline cost
Have a global view Seamless
integration with
Prometheus
Increase retentionHave a HA in place

Ticketmaster confidential. Do not distribute.
Global view
10
Thanos

Ticketmaster confidential. Do not distribute.
Global view + HA
11
Thanos

Ticketmaster confidential. Do not distribute.
Increase retention (Persist data)
12
Thanos

Ticketmaster confidential. Do not distribute.
Increase retention (Querying)
• A series is made up of one or more “chunks”
• A chunk contains ~120 samples each
• Chunks can be retrieved through HTTP byte range queries
Example:
• 1000 series @ 30s scrape interval
• Query 1 year
• 8.7 million chunks/range queries
• Chunks of the same series are aligned
• Similar series are aligned due to same metric name
This reduce request count by 4=6 orders of magnitude.
8.7 million requests turned into O(20) requests
•
13
Thanos

Ticketmaster confidential. Do not distribute.
Increase retention (Querying)
14
Thanos
group chunk same series group by same metric name

Ticketmaster confidential. Do not distribute.
Compaction
15
Thanos

Ticketmaster confidential. Do not distribute.
Full Architecture
16
Thanos

Ticketmaster confidential. Do not distribute.
Deployment Model( Example)
• Federation through Store API
17
Thanos

Ticketmaster confidential. Do not distribute.
Cost
• Store + Query node + Compaction ~ Savings on Prometheus side (+/- 0)
• Fewer SSD space on Prometheus side (Savings)
• Basically we are only paying for your data stored in S3/GCS/etc + requests
18
Thanos

Ticketmaster confidential. Do not distribute.
Cortex
19
Alternatives

Ticketmaster confidential. Do not distribute.
Cortex - Reason that we didn’t choose Cortex
• Documentation is not really good
• Need to maintain another dataset (NoSQL DB)
• Interfere with the datapath
20
Alternatives

Ticketmaster confidential. Do not distribute.
Question?

Ticketmaster confidential. Do not distribute.

What's hot

Getting Started Monitoring with Prometheus and Grafana

Syah Dwi Prihatmoko

Prometheus (Prometheus London, 2016)

Brian Brazil

This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana

MeetUp Monitoring with Prometheus and Grafana (September 2018)

Lucas Jellema

Server monitoring using grafana and prometheus

Celine George

The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.

OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...

NETWAYS

Monitoring with prometheus

Kasper Nissen

Prometheus

wyukawa

Monitoring With Prometheus

Knoldus Inc.

Prometheus and Grafana

Lhouceine OUHAMZA

Improve monitoring and observability for kubernetes with oss tools

Nilesh Gule

Intro to open source observability with grafana, prometheus, loki, and tempo(...

LibbySchulze

Prometheus 101

Paul Podolny

Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io

DevOpsDays Tel Aviv

VictoriaMetrics and Grafana Mimir are time series databases with support of mostly the same protocols and APIs. However, they have different architectures and components, which makes the comparison more complicated. In the talk, we'll go through the details of the benchmark where I compared both solutions. We'll see how VictoriaMetrics and Mimir are dealing with identical workloads and how efficient they’re with using the allocated resources. The talk will cover design and architectural details, weak and strong points, trade-offs, and maintenance complexity of both solutions.

Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx

RomanKhavronenko

Cloud Native PostgreSQL

EDB

Prometheus - Intro, CNCF, TSDB,PromQL,Grafana

Sridhar Kumar N

Benchmarking is hard. Benchmarking databases, harder. Benchmarking databases that follow different approaches (relational vs document) is even harder. But the market demands these kinds of benchmarks. Despite the different data models that MongoDB and PostgreSQL expose, many organizations face the challenge of picking either technology. And performance is arguably the main deciding factor. Join this talk to discover the numbers! After $30K spent on public cloud and months of testing, there are many different scenarios to analyze. Benchmarks on three distinct categories have been performed: OLTP, OLAP and comparing MongoDB 4.0 transaction performance with PostgreSQL's. What would be faster, MongoDB or PostgreSQL?

MongoDB vs. Postgres Benchmarks

EDB

Distributed tracing using open tracing & jaeger 2

Chandresh Pancholi

Monitoring With Prometheus

Agile Testing Alliance

Custom DevOps Monitoring System in MelOn (with InfluxDB + Telegraf + Grafana)

Seungmin Yu

What's hot (20)

Getting Started Monitoring with Prometheus and Grafana

Prometheus (Prometheus London, 2016)

MeetUp Monitoring with Prometheus and Grafana (September 2018)

Server monitoring using grafana and prometheus

OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...

Monitoring with prometheus

Prometheus

Monitoring With Prometheus

Prometheus and Grafana

Improve monitoring and observability for kubernetes with oss tools

Intro to open source observability with grafana, prometheus, loki, and tempo(...

Prometheus 101

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io

Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx

Cloud Native PostgreSQL

Prometheus - Intro, CNCF, TSDB,PromQL,Grafana

MongoDB vs. Postgres Benchmarks

Distributed tracing using open tracing & jaeger 2

Monitoring With Prometheus

Custom DevOps Monitoring System in MelOn (with InfluxDB + Telegraf + Grafana)

Similar to Prometheus and Thanos

Presto at Tivo, Boston Hadoop Meetup

Justin Borgman

Implementing a canonical IoT backend in Azure with Azure Stream Analytics

Marco Parenzan

Apache Tajo: Query Optimization Techniques and JIT-based Vectorized Engine

DataWorks Summit

Hadoop Summit 2014: Query Optimization and JIT-based Vectorized Execution in ...

Gruter

Jon Bratseth (VP Architect) @ Verizon Media: The big data world has mature technologies for offline analysis and learning from data, but have lacked options for making data-driven decisions in real time. When it is sufficient to consider a single data point model servers such as TensorFlow serving can be used but in many cases you want to consider many data points to make decisions. This is a difficult engineering problem combining state, distributed algorithms and low latency, but solving it often makes it possible to create far superior solutions when applying machine learning. This talk will explain why this is a hard problem, show the advantages of solving it, and introduce the open source Vespa.ai platform which is used to implement such solutions in some of the largest scale problems in the world including the world's third largest ad serving system.

Big data serving: Processing and inference at scale in real time

Itai Yaffe

Token Design as Optimization Design

Trent McConaghy

Practical Data Science Workshop - Recommendation Systems - Collaborative Filt...

Chris Fregly

Monitoring your Python with Prometheus (Python Ireland April 2015)

Brian Brazil

Starboard Solutions - 3PL

Stephen Johanson

A Production Quality Sketching Library for the Analysis of Big Data

Databricks

AWS Roadshow Herbst 2013: Datenanalyse und Business Intelligence

AWS Germany

Introduction to Data streaming - 05/12/2014

Raja Chiky

SnappyData at Spark Summit 2017

Jags Ramnarayan

Apache Spark 2.0 offers many enhancements that make continuous analytics quite simple. In this talk, we will discuss many other things that you can do with your Apache Spark cluster. We explain how a deep integration of Apache Spark 2.0 and in-memory databases can bring you the best of both worlds! In particular, we discuss how to manage mutable data in Apache Spark, run consistent transactions at the same speed as state-the-art in-memory grids, build and use indexes for point lookups, and run 100x more analytics queries at in-memory speeds. No need to bridge multiple products or manage, tune multiple clusters. We explain how one can take regulation Apache Spark SQL OLAP workloads and speed them up by up to 20x using optimizations in SnappyData. We then walk through several use-case examples, including IoT scenarios, where one has to ingest streams from many sources, cleanse it, manage the deluge by pre-aggregating and tracking metrics per minute, store all recent data in a in-memory store along with history in a data lake and permit interactive analytic queries at this constantly growing data. Rather than stitching together multiple clusters as proposed in Lambda, we walk through a design where everything is achieved in a single, horizontally scalable Apache Spark 2.0 cluster. A design that is simpler, a lot more efficient, and let’s you do everything from Machine Learning and Data Science to Transactions and Visual Analytics all in one single cluster.

SnappyData, the Spark Database. A unified cluster for streaming, transactions...

SnappyData

Break out of The Box - Part 2

Karl-Henry Martinsson

Storm makes it easy to write and scale complex realtime computations on a cluster of computers, doing for realtime processing what Hadoop did for batch processing. Storm guarantees that every message will be processed. And it’s fast — you can process millions of messages per second with a small cluster. Best of all, you can write Storm topologies using any programming language. Storm was open-sourced by Twitter in September of 2011 and has since been adopted by many companies around the world. Storm has a wide range of use cases, from stream processing to continuous computation to distributed RPC. In this talk I'll introduce Storm and show how easy it is to use for realtime computation.

Jan 2012 HUG: Storm

Yahoo Developer Network

If you are storing records with a timestamp in your database, it is very likely a time series database can make your life easier. However, time series databases are still the great unknown for a large part of the tech community. In this talk, I will show you what use cases they are good for, what they give you that you cannot get from a traditional database, and when it is a good idea (and when it is not) to use them. For the demos, we will be using QuestDB, the fastest open-source time series database.

Your Timestamps Deserve Better than a Generic Database

javier ramirez

AWS re:Invent 2016: Amazon CloudFront Flash Talks: Best Practices on Configur...

Amazon Web Services

The Data Platform at Twitter supports engineers and data scientists running batch jobs on Hadoop clusters that are several 1000s of nodes, and real-time jobs on top of systems such as Storm. In this presentation, I discuss the overall Data Platform stack at Twitter. In particular, I talk about enabling real-time and batch analytics at scale with the help of Scalding, which is a Scala DSL for batch jobs using MapReduce, Summingbird, which is a framework for combined real-time and batch processing, and Tsar, which is a framework for real-time time-series aggregations.

Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale

Sriram Krishnan

Organizations often need to quickly analyze large amounts of data, such as logs generated from a wide variety of sources and formats. However, traditional approaches require a lot of time and effort designing complex data transformation and loading processes; and configuring data warehouses. Using AWS, you can start querying your datasets within minutes. In this session you will learn how you can deploy a managed Presto environment in minutes to interactively query log data using standard ANSI SQL. Presto is a popular open source SQL engine for running interactive analytic queries against data sources of all sizes. We will talk about common use cases and best practices for running Presto on Amazon EMR.

Interactively Querying Large-scale Datasets on Amazon S3

Amazon Web Services

Similar to Prometheus and Thanos (20)

Presto at Tivo, Boston Hadoop Meetup

Implementing a canonical IoT backend in Azure with Azure Stream Analytics

Apache Tajo: Query Optimization Techniques and JIT-based Vectorized Engine

Hadoop Summit 2014: Query Optimization and JIT-based Vectorized Execution in ...

Big data serving: Processing and inference at scale in real time

Token Design as Optimization Design

Practical Data Science Workshop - Recommendation Systems - Collaborative Filt...

Monitoring your Python with Prometheus (Python Ireland April 2015)

Starboard Solutions - 3PL

A Production Quality Sketching Library for the Analysis of Big Data

AWS Roadshow Herbst 2013: Datenanalyse und Business Intelligence

Introduction to Data streaming - 05/12/2014

SnappyData at Spark Summit 2017

SnappyData, the Spark Database. A unified cluster for streaming, transactions...

Break out of The Box - Part 2

Jan 2012 HUG: Storm

Your Timestamps Deserve Better than a Generic Database

AWS re:Invent 2016: Amazon CloudFront Flash Talks: Best Practices on Configur...

Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale

Interactively Querying Large-scale Datasets on Amazon S3

More from CloudOps2005

Defense in Depth: Securing your new Kubernetes cluster from the challenges th...

CloudOps2005

Human No, Machine Yes: Welcome to the CDF with Incremental Confidence

CloudOps2005

The Salmon Algorithm Spawning with Kubernetes

CloudOps2005

Own your Destiny in the Cloud - Ian Rae - Cloud Native Day Montreal 2019

CloudOps2005

Plateformes et infrastructure infonuagique natif de ville de Montréall

CloudOps2005

Using Rook to Manage Kubernetes Storage with Ceph

CloudOps2005

Kafka on Kubernetes

CloudOps2005

Kubernetes: Crossing the Chasm

CloudOps2005

Distributed Logging with Kubernetes

CloudOps2005

Kubernetes Security with Calico and Open Policy Agent

CloudOps2005

Advanced Deployment Strategies with Kubernetes and Istio

CloudOps2005

GitOps with ArgoCD

CloudOps2005

Kubernetes Services are sooo Yesterday!

CloudOps2005

Amazon EKS: the good, the bad, and the ugly

CloudOps2005

Kubernetes, Terraform, Vault, and Consul

CloudOps2005

SIG Multicluster and the Path to Federation

CloudOps2005

To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem

CloudOps2005

Operator SDK for K8s using Go

CloudOps2005

How to Handle your Kubernetes Upgrades

CloudOps2005

This year's first round of Kubernetes and Cloud Native meetups in Eastern Canada began with an update of the CNCF by Ayrat Khayretdinov, CNCF Ambassador and Solutions Architect at CloudOps. He explained the status of various projects and highlights from KubeCon + CloudNativeCon. To learn the basics of cloud native application modernization, sign up for one of our hands-on, three-day workshops on Docker and Kubernetes at https://www.cloudops.com/workshops/#DockerK8s

Kubernetes and Cloud Native Meetup - March, 2019

CloudOps2005

More from CloudOps2005 (20)

Defense in Depth: Securing your new Kubernetes cluster from the challenges th...

Human No, Machine Yes: Welcome to the CDF with Incremental Confidence

The Salmon Algorithm Spawning with Kubernetes

Own your Destiny in the Cloud - Ian Rae - Cloud Native Day Montreal 2019

Plateformes et infrastructure infonuagique natif de ville de Montréall

Using Rook to Manage Kubernetes Storage with Ceph

Kafka on Kubernetes

Kubernetes: Crossing the Chasm

Distributed Logging with Kubernetes

Kubernetes Security with Calico and Open Policy Agent

Advanced Deployment Strategies with Kubernetes and Istio

GitOps with ArgoCD

Kubernetes Services are sooo Yesterday!

Amazon EKS: the good, the bad, and the ugly

Kubernetes, Terraform, Vault, and Consul

SIG Multicluster and the Path to Federation

To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem

Operator SDK for K8s using Go

How to Handle your Kubernetes Upgrades

Kubernetes and Cloud Native Meetup - March, 2019

Recently uploaded

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

MINDCTI Revenue Release Quarter One 2024

MIND CTI

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Bhuvaneswari Subramani

Recently uploaded (20)

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Corporate and higher education May webinar.pptx

MINDCTI Revenue Release Quarter One 2024

Platformless Horizons for Digital Adaptability

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Artificial Intelligence Chap.5 : Uncertainty

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Why Teams call analytics are critical to your entire business

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

FWD Group - Insurer Innovation Award 2024

Vector Search -An Introduction in Oracle Database 23ai.pptx

MS Copilot expands with MS Graph connectors

Understanding the FAA Part 107 License ..

Six Myths about Ontologies: The Basics of Formal Ontology

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Prometheus and Thanos

1. Ticketmaster confidential. Do not distribute. Ticketmaster confidential. Do not distribute. Prometheus + Thanos Thanos long term storage for Prometheus

2. Ticketmaster confidential. Do not distribute. About me... • Rocking it at Ticketmaster • Student in Master of Business Administration (MBA) Strategic project management • Bachelor of Applied Science (B.A.Sc.) in Computer Science • Devops, Cloud, Kubernetes, Kafka, etc. • Sport lover (Hockey, Dek Hockey, Ultimate Frisbee, Football…) 2 SECTION

3. Ticketmaster confidential. Do not distribute. Where Prometheus & Thanos are in the CNCF landscape... 3 Prometheus

4. Ticketmaster confidential. Do not distribute. Insert photography and crop to size of grey box. (24.25 in. x 7.21 in.) What is Prometheus? 4 Prometheus Prometheus was the second project to be graduated of the Cloud Native Computing Foundation.

5. Ticketmaster confidential. Do not distribute. Prometheus Architecture

6. Ticketmaster confidential. Do not distribute. How we are using Prometheus at Ticketmaster? • We are using the Prometheus-Operator made by CoreOS (RedHat (IBM)) • We have created a Helm chart that created common pulling jobs, settings and exporters • Scrape EC2 instances based on Tags, Kubernetes • Exporters like cloudwatch-metrics, kafka, blackbox • Ingresses • Thanos • Federations 6 Prometheus

7. Ticketmaster confidential. Do not distribute. Why moving away from the federation? • Calculate the right ingestion rate of our scrape job is not really easy • When the disk is full • Prometheus stop working • We needed to delete the PVC to recreated it bigger. • Long term storage is costly SSD == $$$$ • Single point of failure • Availability • Operator error • Hardware failure • Rollout 7 Prometheus

8. Ticketmaster confidential. Do not distribute. Solution

9. Ticketmaster confidential. Do not distribute. Goals 9 Thanos - Easy Deployment model - Minimal number of dependencies - Minimal baseline cost Have a global view Seamless integration with Prometheus Increase retentionHave a HA in place

10. Ticketmaster confidential. Do not distribute. Global view 10 Thanos

11. Ticketmaster confidential. Do not distribute. Global view + HA 11 Thanos

12. Ticketmaster confidential. Do not distribute. Increase retention (Persist data) 12 Thanos

13. Ticketmaster confidential. Do not distribute. Increase retention (Querying) • A series is made up of one or more “chunks” • A chunk contains ~120 samples each • Chunks can be retrieved through HTTP byte range queries Example: • 1000 series @ 30s scrape interval • Query 1 year • 8.7 million chunks/range queries • Chunks of the same series are aligned • Similar series are aligned due to same metric name This reduce request count by 4=6 orders of magnitude. 8.7 million requests turned into O(20) requests • 13 Thanos

14. Ticketmaster confidential. Do not distribute. Increase retention (Querying) 14 Thanos group chunk same series group by same metric name

15. Ticketmaster confidential. Do not distribute. Compaction 15 Thanos

16. Ticketmaster confidential. Do not distribute. Full Architecture 16 Thanos

17. Ticketmaster confidential. Do not distribute. Deployment Model( Example) • Federation through Store API 17 Thanos

18. Ticketmaster confidential. Do not distribute. Cost • Store + Query node + Compaction ~ Savings on Prometheus side (+/- 0) • Fewer SSD space on Prometheus side (Savings) • Basically we are only paying for your data stored in S3/GCS/etc + requests 18 Thanos

19. Ticketmaster confidential. Do not distribute. Cortex 19 Alternatives

20. Ticketmaster confidential. Do not distribute. Cortex - Reason that we didn’t choose Cortex • Documentation is not really good • Need to maintain another dataset (NoSQL DB) • Interfere with the datapath 20 Alternatives

21. Ticketmaster confidential. Do not distribute. Question?

22. Ticketmaster confidential. Do not distribute.

Prometheus and Thanos

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Prometheus and Thanos

Similar to Prometheus and Thanos (20)

More from CloudOps2005

More from CloudOps2005 (20)

Recently uploaded

Recently uploaded (20)

Prometheus and Thanos