Leveraging Microservices and Apache Kafka to Scale Developer Productivity

•

0 likes•395 views

Presenter: Mohinish Shaikh, Software Engineer, Confluent Streaming Audio Podcast: Scaling Developer Productivity with Apache Kafka https://cnfl.io/scaling-developer-productivity-with-apache-kafka

Technology

Leveraging
Microservices &
Apache Kafka® to Scale
Developer Productivity
Mohinish Shaikh
@mohinishbasha
https://www.linkedin.com/in/mohinishbasha/

How ...
What ...
Why ...
of Developer
productivity

development ...
process ...
operations ...
engineering ...
devops ...
sre ...
pipelines ...
engineering services ...
platform infrastructure ...
whoami

Developer productivity platform
Quality & Test
Automation
Packaging
Commits
Monitoring, Logging,
Tracing
Infrastructure
Management &
Automation
CI/ CD across Stage, QA,
Integ, Perf, Production
* Productivity
Intelligence
Builds, Releases &
Deployments
Observability
Security, Compliance &
Audit
Developer Productivity
Platform

. . . anything &
everything between
commit to production
and beyond ...

common
tools …
systems ..
frameworks ...
infrastructure …
standards …
architectural patterns ...
automation …
workflows …
workloads ...

Support
polyglot applications ...
quality ...
reliability ...
efficiency ...
environments ...
speed ...
time to market ...
roi ...

products expansion ...
lob ...
complementary services ...
bigger markets ...
partners ...
community ...
compliance ...
laws ...
geography ...
languages ...
culture ...

if ...
sales ...
promises ...
stuff ...
to ...
customers …
Developer Productivity ...
helps the ...
organization ...
Deliver with speed

nasa helios 2 set a record
speed of 157078 mph in 1976

Cohesive ...
Hyperconverged ...
self serviced ...
self aware ...
Agnostic ...
Pluggable ...

as organizations grow over time,
things become
monolithic in nature & silo’d

building a productivity
platform that scales while
avoiding monolithic traps &
technical debt is the
challenge

pipelines
containerize tooling ..
source code analysis ..
unit, functional, perf tests ..
builds, releases, packaging ..
deployments, infrastructure ..
automated ..
repeatable ..
de-coupled ..
systems ..
services ..

Learn, adapter, evolve
using
a productivity
intelligence system.

Mgmt. Infrastructure
Productivity platform
DB2
DB1
KV
DWH
Builds
Provisionin
g
KV2
KV3
App2-v2
App5 App7
App1-v2
Application Cluster
App8
DWH
Tests
Kafka
Kafka
Deploymen
ts
DB3
SCM
Productivity Intelligence

Before: Many Ad Hoc Pipelines
Search Security
General Application Architecture
Service Discovery/ KV Logging, Tracing Operational Metrics
Hadoop
Product1: App Data
Warehouse
Provisioning Builds Deployments
Databases
Storage
Interfaces
Product2: App
Databases
Storage
Interfaces
Artifacts
Governance
Test Automation
IaaS
Compliance
Security

Kafka Streams
Kafka Streams
After: Productivity Platform with Kafka
✓ cohesive ✓ agnostic ✓ Self serviceable
Search Security
General Application Architecture
Service Discovery/ KV
Monitoring, Logging,
Tracing
Kafka
Provisioning Builds Deployments
Hadoop Product1: App Product2: App
Data
Warehouse
pipelines running productivity micro services
✓ hyper-converged
Governance
Artifacts
Test automation
Immutable IaaS
Security
Compliance

Management Cluster: future
build container
Tools Service pod
Native Kafka Java
Applications
build container
Management Pipelines Management Cluster
Provisioning
Artifacts
Test Infrastructure
Builds, Release &
Packaging
Deployments
Tools container
Tools Service pod
Native Kafka Java
Applications
Tools container
Productivity Intelligence

Code philosophy – Plan, Apply, Destroy
State: Maintain state of automation & infrastructure at all times.
1 2
Plan
generate a plan of
execution
Execute
Apply changes based
on generated plan.
3
Destroy
At any given point in
time tools should be
able to roll back
KV
Configuration
Keep configuration out
of tooling.

Q & A.
Mohinish Shaikh
@mohinishbasha
https://www.linkedin.com/in/mohinishbasha/

What's hot

Course Objectives In this three-day hands-on course, you will learn how to build an application that can publish data to, and subscribe to data from, an Apache Kafka cluster. You will learn the role of Kafka in the modern data distribution pipeline, discuss core Kafka architectural concepts and components, and review the Kafka developer APIs. As well as core Kafka, Kafka Connect, and Kafka Streams, the course also covers other components in the broader Confluent Platform, such as the Schema Registry and the REST Proxy. For more information, please visit www.confluent.io/training/

Confluent Developer Training

confluent

Haitao Zhang, Uber, Software Engineer + Yang Yang, Uber, Senior Software Engineer Kafka Consumer Proxy is a forwarding proxy that consumes messages from Kafka and dispatches them to a user registered gRPC service endpoint. With Kafka Consumer Proxy, the experience of consuming messages from Apache Kafka for pub-sub use cases is as seamless and user-friendly as receiving (g)RPC requests. In this talk, we will share (1) the motivation for building this service, (2) the high-level architecture, (3) the mechanisms we designed to achieve high availability, scalability, and reliability, and (4) the current adoption status. https://www.meetup.com/KafkaBayArea/events/273834934/

Uber: Kafka Consumer Proxy

confluent

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us. Things I will be sharing as part of the talk: 1. Establishing the use case/problem we were trying to solve (why we needed mTLS) 2. Building a Certificate Authority with open source tools (with self-signed Root CA) 3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers 4. Hot reloading regenerated certificates on Kafka brokers without downtime 5. What we built to rotate the self-signed root CA without downtime as well across the board 6. Monitoring and alerts on TTL of certificates 7. Performance impact of using TLS (along with why TLS affects kafka’s performance) 8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier 9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020

confluent

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...

confluent

Many enterprises have a large technical debt in legacy applications hosted in on-premises data centers. There is a strong desire to modernize and move to a cloud-based infrastructure, but the world won’t stop for you to transition. Existing applications need to be supported and enhanced; data from legacy platforms is required to make decisions that drive the business. On the other hand, data from cloud-based applications does not exist in a vacuum. Legacy applications need access to these cloud data sources and vice versa. Can an enterprise have it both ways? Can new applications be built in the cloud while existing applications are maintained in a private data center? Monsanto has adopted a cloud-first mentality—today most new development is focused on the cloud. However, this transition did not happen overnight. Chrix Finne and Bob Lehmann share their experience building and implementing a Kafka-based cross-data-center streaming platform to facilitate the move to the cloud—in the process, kick-starting Monsanto’s transition from batch to stream processing. Details include an overview of the challenges involved in transitioning to the cloud and a deep dive into the cross-data-center stream platform architecture, including best practices for running this architecture in production and a summary of the benefits seen after deploying this architecture.

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform

confluent

Westpac Bank Tech Talk 1: Dive into Apache Kafka

confluent

Badai Aqrandista, Confluent, Senior Technical Support Engineer This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way! https://www.meetup.com/apache-kafka-sydney/events/279651982/

Common issues with Apache Kafka® Producer

confluent

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Kafka Streams for Java enthusiasts

Slim Baltagi

Understanding Apache Kafka® Latency at Scale

confluent

Apache Kafka® in Industrial Environments

confluent

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)

confluent

We use machine learning to delve deep into the internals of how systems like Kafka work. In this talk I'll dive into what variables affect performance and reliability, including previously unknown leading indicators of major performance problems, failure conditions and how to tune for specific use cases. I'll cover some of the specific methodology we use, including Bayesian optimization, and reinforcement learning. I'll also talk about our own internal infrastructure that makes heavy use of Kafka and Kubernetes to deliver real-time predictions to our customers.

What's inside the black box? Using ML to tune and manage Kafka. (Matthew Stum...

confluent

Even though Kafka is scalable by design, proper handling of over one petabyte of data a day requires much more than Kafka’s scalability. Several challenges present themselves in a data centric business at this scale. These challenges include capacity planning, provisioning, message auditing, monitoring and alerting, rebalancing workloads with changes in traffic patterns, data lineage, handling service degradation and system outages, optimizing cost, upgrades, etc. In this talk we describe how at Pinterest we tackle some of these challenges and share some of the key lessons that we learned in the process. Specifically we will share how we: * Automate Kafka cluster maintenance * Manage over 150K partitions * Manage upgrade lifecycle * Track / troubleshoot thousands of data pipelines

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...

confluent

Intro to AsyncAPI

confluent

What do you do when you've two different technologies on the upstream and the downstream that are both rapidly being adopted industrywide? How do you bridge them scalably and robustly? At Wework, the upstream data was being brokered by Kafka and the downstream consumers were highly scalable gRPC services. While Kafka was capable of efficiently channeling incoming events in near real-time from a variety of sensors that were used in select Wework spaces, the downstream gRPC services that were user-facing were exceptionally good at serving requests in a concurrent and robust manner. This was a formidable combination, if only there was a way to effectively bridge these two in an optimized way. Luckily, sink Connectors came to the rescue. However, there weren't any for gRPC sinks! So we wrote one. In this talk, we will briefly focus on the advantages of using Connectors, creating new Connectors, and specifically spend time on gRPC sink Connector and its impact on Wework's data pipeline.

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...

confluent

Namely is a late-stage startup that builds HR, Payroll and Benefits software for mid-sized businesses. Over the years, we've ended up with a number of monolithic and legacy applications covering overlapping domain concepts, which has limited our ability to deliver new and innovative features to our customers. We need a way to get our data out of the monoliths to decouple our systems and increase our velocity. We've chosen Kafka as our way to liberate our data in a reliable, scalable and maintainable way. This talk covers specific examples of successes and missteps in our move to Kafka as the backbone of our architecture. It then looks to the future - where we are trying to go, and how we plan on getting, both from the short term and long term perspectives. Key Takeaways: - Successful and unsuccessful approaches to gradually introducing Kafka to a large organization in a way that meets the short and long term needs of the business. - Successful and unsuccessful patterns for using Kafka. - Pragmaticism versus purisim: Building Kafka-first systems, and migrating legacy systems to Kafka with Debezium. - Combining event driven systems with RPC based systems. Observability, alerting and testing. - Actionable steps that you can take to your organization to help drive adoption.

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019

confluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. Many organizations understand the use cases around their data – fraud detection, quality of service and technical operations, user behavior analysis, for example – but are not necessarily data infrastructure experts. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes an hour of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. Attendees will leave this session knowing not just which open source projects go into a system such as this, but how they work together, what tradeoffs and decisions need to be addressed, and how to present a single general purpose data platform to multiple applications. This session should be attended by data infrastructure engineers and architects planning, building, or maintaining similar systems.

Building an Event-oriented Data Platform with Kafka, Eric Sammer

confluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

ksqlDB: A Stream-Relational Database System

confluent

What's hot (20)

Confluent Developer Training

Uber: Kafka Consumer Proxy

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Common issues with Apache Kafka® Producer

Kafka Streams for Java enthusiasts

Understanding Apache Kafka® Latency at Scale

Apache Kafka® in Industrial Environments

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)

What's inside the black box? Using ML to tune and manage Kafka. (Matthew Stum...

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...

Intro to AsyncAPI

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

Building an Event-oriented Data Platform with Kafka, Eric Sammer

ksqlDB: A Stream-Relational Database System

Similar to Leveraging Microservices and Apache Kafka to Scale Developer Productivity

Devops architecture

Ojasvi Jagtap

This talk was given at KCD Munich - July 17 2023 Abstract “Kubernetes is a platform for building platforms. It’s a better place to start: not the endgame”, tweeted by Kelsey Hightower in November 2017. 6 years later the Cloud Native Community is faced with 159 different CNCF projects to choose from. Entering CNCF can be overwhelming! Cloud Native Platform Engineering with white papers, best practices and reference architectures are here to convert this dilemma into an opportunity. Internal Developer Platforms (IDP) are being built as we speak enabling organizations to harness the power of Kubernetes as a self-service platform. Join this talk with Andreas Grabner, CNCF Ambassador, and get some insights on tooling, use cases and best practices so we can all fulfill the idea that Kelsey put out years ago.

KCD Munich - Cloud Native Platform Dilemma - Turning it into an Opportunity

Andreas Grabner

We'll show you how to take your application and launch it quickly on a variety of AWS infrastructure. You'll learn how to leverage CodeStar, CodeBuild, CodeDeploy, and Cloud9 to provide your startup with reliable, flexible, and cost efficient build pipelines in minutes. This will set your technical teams up for faster deploys and consistent development environments allowing you to focus on your product, not your deployment process. This is a key pain point for early stage startups, learn how to solve it before it starts to impact your team's productivity.

AWS18 Startup Day Toronto- Launching your Application the Amazon Way

Amazon Web Services

Diese Präsentation beinhaltet Erfahrungen, Empfehlungen und Planungs-Gedanken, die man beachten sollte, wenn man multiple IBM Cloud Paks auf der Container Platform OpenShift installieren / deployen möchte. Es beschreibt die Grundlagen zu "common services", auch "foundational services" genannt, die als Basis-Services die Lauffähigkeit dieser Cloud Paks auf OpenShift erläutert und wie man Cloud Paks auch logisch trennen kann auf OpenShift worker nodes über taints und node selectors.

IBM BP Session - Multiple CLoud Paks and Cloud Paks Foundational Services.pptx

Georg Ember

DevOps : Integrate, Deliver and Deploy continuously with Visual Studio Team S...

BAINIDA

MLOps in action

Pieter de Bruin

Dev ops using Jenkins

Synergetics Learning and Cloud Consulting

Continuous Delivery with a PaaS Application

Mark Rendell

Applications are moving to cloud and containers to boost reliability and speed delivery to production. However, if we use the same old approaches to testing, we'll fail to achieve the benefits of cloud. But what do we really need to change? We know we need to automate tests, but how do we keep our automation assets from becoming obsolete? Automatically provisioning test environments seems close, but some parts of our applications are hard to move to cloud.

Modernizing Testing as Apps Re-Architect

DevOps.com

by Nick Brandaleone, Solutions Architect AWS Join us to learn about continuous integration, continuous delivery, and DevOps. The AWS Developer Tools have been designed based on the tools used by Amazon engineers to rapidly and reliably deliver products and features to customers. We’ll provide overviews of the services and best practices followed by a hands-on workshop to help you learn how to automate your software release processes, deploy application code, and monitor your application and infrastructure performance.

DevOps on AWS - Building Systems to Deliver Faster

Amazon Web Services

NET Aspire - NET Conf IL 2024 - Tamir Dresher.pdf

Tamir Dresher

Microsoft recently released Azure DevOps, a set of services that help developers and IT ship software faster, and with higher quality. These services cover planning, source code, builds, deployments, and artifacts. One of the great things about Azure DevOps is that it works great for any app and on any platform regardless of frameworks. In this session, I will provide a hands on workshop guiding you through getting started with Azure Pipelines to build your application. Using continuous integration and deployment processes, you will leave with clear understanding and skills to get your applications up and running quickly in Azure DevOps and see the full benefits that CI/CD can bring to your organization.

Azure DevOps in Action

Callon Campbell

Mobile Apps are not stand alone applications running on a mobile device anymore. Apps today are complex systems with back-ends hosted on clouds, with application servers, databases, API calls to external systems, and of course a powerful app running on a mobile device. Mobile App development and deployment is further complicated with todays need for supporting multiple mobile devices, with multiple OSes, multiple versions of the OSes, multiple form factors and varied network, CPU, GPU and memory specs. DevOps - the new and growing movement addresses these development and deployment challenges. The goal of DevOps is to align Dev and Ops by introducing a set of principles and practices such as continuous integration and continuous delivery. Mobile apps take the need for these practices up a level due to their inherent distributed nature. Multi-platform mobile apps need even more care in applying DevOps principles as there are multiple platforms to be targeted, each with its own requirements, quirks, and nuanced needs. This talk will introduce attendees to the basic practices of DevOps and then take a look at the DevOps challenges specific to cross-platform Mobile apps and present Best Practices to address them.

Dev ops for cross platform mobile modeveast 12

Sanjeev Sharma

Often times, developers and auditors can be at odds. The agile, fast-moving environments that developers enjoy will typically give auditors heartburn. The more controlled and stable environments that auditors prefer to demonstrate and maintain compliance are traditionally not friendly to developers or innovation. In this session, we will look at the intersection where developers and auditors meet by discussing techniques, tools, and concepts that appeal to both. Topics covered will include shared responsibility, using compartmentalization and microservices for scope control, immutable infrastructure, and continuous security testing.

Keeping Developers and Auditors Happy in the Cloud

Amazon Web Services

Growing Adoption of Open Source in Enterprises

WSO2

20171122 aws usergrp_coretech-spn-cicd-aws-v01

Scott Miao

Netflix Cloud Architecture and Open Source

aspyker

DevOps Evolution - The Next Generation ?

Marc Hornbeek

Simplify and Scale Enterprise Spring Apps in the Cloud | March 23, 2023

VMware Tanzu

Azure DevOps työkalut - Roundtable 14.3.2019

Janne Mattila

Similar to Leveraging Microservices and Apache Kafka to Scale Developer Productivity (20)

Devops architecture

KCD Munich - Cloud Native Platform Dilemma - Turning it into an Opportunity

AWS18 Startup Day Toronto- Launching your Application the Amazon Way

IBM BP Session - Multiple CLoud Paks and Cloud Paks Foundational Services.pptx

DevOps : Integrate, Deliver and Deploy continuously with Visual Studio Team S...

MLOps in action

Dev ops using Jenkins

Continuous Delivery with a PaaS Application

Modernizing Testing as Apps Re-Architect

DevOps on AWS - Building Systems to Deliver Faster

NET Aspire - NET Conf IL 2024 - Tamir Dresher.pdf

Azure DevOps in Action

Dev ops for cross platform mobile modeveast 12

Keeping Developers and Auditors Happy in the Cloud

Growing Adoption of Open Source in Enterprises

20171122 aws usergrp_coretech-spn-cicd-aws-v01

Netflix Cloud Architecture and Open Source

DevOps Evolution - The Next Generation ?

Simplify and Scale Enterprise Spring Apps in the Cloud | March 23, 2023

Azure DevOps työkalut - Roundtable 14.3.2019

More from confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

confluent

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

confluent

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

confluent

Confluent Partner Tech Talk with Synthesis

confluent

The Future of Application Development - API Days - Melbourne 2023

confluent

The Playful Bond Between REST And Data Streams

confluent

More from confluent (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams

Recently uploaded

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

AXA XL - Insurer Innovation Award Americas 2024

The Digital Insurer

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

DBX First Quarter 2024 Investor Presentation

Dropbox

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Recently uploaded (20)

AWS Community Day CPH - Three problems of Terraform

Cyberprint. Dark Pink Apt Group [EN].pdf

Manulife - Insurer Transformation Award 2024

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

2024: Domino Containers - The Next Step. News from the Domino Container commu...

[BuildWithAI] Introduction to Gemini.pdf

Artificial Intelligence Chap.5 : Uncertainty

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

AXA XL - Insurer Innovation Award Americas 2024

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

DBX First Quarter 2024 Investor Presentation

MS Copilot expands with MS Graph connectors

Exploring the Future Potential of AI-Enabled Smartphone Processors

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Ransomware_Q4_2023. The report. [EN].pdf

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Corporate and higher education May webinar.pptx

Leveraging Microservices and Apache Kafka to Scale Developer Productivity

1. Leveraging Microservices & Apache Kafka® to Scale Developer Productivity Mohinish Shaikh @mohinishbasha https://www.linkedin.com/in/mohinishbasha/

2. Next ... Microservices Kafka

3. How ... What ... Why ... of Developer productivity

4. development ... process ... operations ... engineering ... devops ... sre ... pipelines ... engineering services ... platform infrastructure ... whoami

5. trustme ... i am an engineer

6. Developer productivity platform Quality & Test Automation Packaging Commits Monitoring, Logging, Tracing Infrastructure Management & Automation CI/ CD across Stage, QA, Integ, Perf, Production * Productivity Intelligence Builds, Releases & Deployments Observability Security, Compliance & Audit Developer Productivity Platform

7. . . . anything & everything between commit to production and beyond ...

8. common tools … systems .. frameworks ... infrastructure … standards … architectural patterns ... automation … workflows … workloads ...

9. Support polyglot applications ... quality ... reliability ... efficiency ... environments ... speed ... time to market ... roi ...

10. products expansion ... lob ... complementary services ... bigger markets ... partners ... community ... compliance ... laws ... geography ... languages ... culture ...

11. if ... sales ... promises ... stuff ... to ... customers … Developer Productivity ... helps the ... organization ... Deliver with speed

12. nasa helios 2 set a record speed of 157078 mph in 1976

13.

14. Cohesive ... Hyperconverged ... self serviced ... self aware ... Agnostic ... Pluggable ...

15. as organizations grow over time, things become monolithic in nature & silo’d

16.

17.

18. building a productivity platform that scales while avoiding monolithic traps & technical debt is the challenge

19.

20.

21.

22. pipelines containerize tooling .. source code analysis .. unit, functional, perf tests .. builds, releases, packaging .. deployments, infrastructure .. automated .. repeatable .. de-coupled .. systems .. services ..

23. Cohesive ... Hyperconverged ... self serviced ... self aware ... Agnostic ... Pluggable ...

24. Learn, adapter, evolve using a productivity intelligence system.

25. Mgmt. Infrastructure Productivity platform DB2 DB1 KV DWH Builds Provisionin g KV2 KV3 App2-v2 App5 App7 App1-v2 Application Cluster App8 DWH Tests Kafka Kafka Deploymen ts DB3 SCM Productivity Intelligence

26. Before: Many Ad Hoc Pipelines Search Security General Application Architecture Service Discovery/ KV Logging, Tracing Operational Metrics Hadoop Product1: App Data Warehouse Provisioning Builds Deployments Databases Storage Interfaces Product2: App Databases Storage Interfaces Artifacts Governance Test Automation IaaS Compliance Security

27. Kafka Streams Kafka Streams After: Productivity Platform with Kafka ✓ cohesive ✓ agnostic ✓ Self serviceable Search Security General Application Architecture Service Discovery/ KV Monitoring, Logging, Tracing Kafka Provisioning Builds Deployments Hadoop Product1: App Product2: App Data Warehouse pipelines running productivity micro services ✓ hyper-converged Governance Artifacts Test automation Immutable IaaS Security Compliance

28. Management Cluster: future build container Tools Service pod Native Kafka Java Applications build container Management Pipelines Management Cluster Provisioning Artifacts Test Infrastructure Builds, Release & Packaging Deployments Tools container Tools Service pod Native Kafka Java Applications Tools container Productivity Intelligence

29. Code philosophy – Plan, Apply, Destroy State: Maintain state of automation & infrastructure at all times. 1 2 Plan generate a plan of execution Execute Apply changes based on generated plan. 3 Destroy At any given point in time tools should be able to roll back KV Configuration Keep configuration out of tooling.

30.

31. Q & A. Mohinish Shaikh @mohinishbasha https://www.linkedin.com/in/mohinishbasha/

Leveraging Microservices and Apache Kafka to Scale Developer Productivity

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Leveraging Microservices and Apache Kafka to Scale Developer Productivity

Similar to Leveraging Microservices and Apache Kafka to Scale Developer Productivity (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Leveraging Microservices and Apache Kafka to Scale Developer Productivity