The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

•Download as PPTX, PDF•

4 likes•7,255 views

All Things Open

Michael Laing New York Times - Systems Architect Tuesday, Oct 20th 10:30 am - Back-End Dev

Michael Laing
Architect, Edge Engineering
New York Times
michael.laing@nytimes.com

• nyt⨍aбrik (my project)

A Global Mesh with a Memory
Millions of users
Dozens of
internal clients

• nyt⨍aбrik (my project)

by wmasterj and Fabrice TIERCELIN - Wikipedia

Messages

Async
Scope

Time
P C
Idle
Idle
Idle

Time
P C CP
Idle
Idle
Idle
Idle
latency

• nyt⨍aбrik (my project)

Database
Semantics
For access

GW GW GW GW GW GW
Region A Region B

Push Pull
snd rcv

Written in C

Push Pull
snd rcv

Push Pull snd rcv
NSQ

NSQ

Push Pull
snd rcv
NSQ
What if…

GW GW GW GW GW GW
Region A Region B
Then:

Region A Region B
GW GW GW GW GW GW
Becomes:

• nyt⨍aбrik (my project)

NSQ

Distributed Graphs at Scale with Cassandra
and Titan
● Structured Knowledge
Representation
● Index-free adjacency:
Like memory pointers,
but in disk
● Navigation between
nodes in constant time.
● Graph != No schema

Michael Laing
Architect, Edge Engineering (soon to include OLAP w Spark Streaming)
New York Times
michael.laing@nytimes.com

More Related Content

What's hot

TiDB Introduction - San Francisco MySQL Meetup

TiDB Introduction - San Francisco MySQL Meetup

TiDB Introduction - San Francisco MySQL Meetup

DSD-INT 2014 - OpenGIS Workshop - Gridded data, Giorgio Santinelli, Deltares

DSD-INT 2014 - OpenGIS Workshop - Gridded data, Giorgio Santinelli, Deltares

DSD-INT 2014 - OpenGIS Workshop - Gridded data, Giorgio Santinelli, Deltares

Trondheim Eclipe Day 2015 and 2016

Trondheim Eclipe Day 2015 and 2016

Trondheim Eclipe Day 2015 and 2016

Matthew Gerring

Saratov open it teach talk. Дамир Яраев: Введение в Apache Cassandra (В ходе презентации Дамир расскажет, когда и почему стоит переходить с проверенных временем реляционных баз данных на ставшие модными в последнее время решения на базе NoSQL. В качестве примера рассмотрит колоночную NoSQL базу данных Apache Cassandra)

Введение в Apache Cassandra

Введение в Apache Cassandra

Введение в Apache Cassandra

Debugging and Profiling Rails Application

Debugging and Profiling Rails Application

Debugging and Profiling Rails Application

IPTC News Exchange Working Group 2013 Autumn Meeting

IPTC News Exchange Working Group 2013 Autumn Meeting

IPTC News Exchange Working Group 2013 Autumn Meeting

Quantum Gaussian Processes - Gawel Kus

Quantum Gaussian Processes - Gawel Kus

Quantum Gaussian Processes - Gawel Kus

Advanced-Concepts-Team

SWT Final Project Presentation

SWT Final Project Presentation

SWT Final Project Presentation

TiDB + Mobike by Kevin Xu (@kevinsxu)

TiDB + Mobike by Kevin Xu (@kevinsxu)

TiDB + Mobike by Kevin Xu (@kevinsxu)

Strings, C# and Unmanaged Memory

Strings, C# and Unmanaged Memory

Strings, C# and Unmanaged Memory

Michael Yarichuk

Introducing MagnetoDB, a key-value storage sevice for OpenStack

Introducing MagnetoDB, a key-value storage sevice for OpenStack

Introducing MagnetoDB, a key-value storage sevice for OpenStack

CEDAR & PRELIDA Preservation of Linked Socio-Historical Data

CEDAR & PRELIDA Preservation of Linked Socio-Historical Data

CEDAR & PRELIDA Preservation of Linked Socio-Historical Data

PRELIDA Project

Third openCypher Implementers Group Meeting: Status Update

Third openCypher Implementers Group Meeting: Status Update

Third openCypher Implementers Group Meeting: Status Update

OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal...

OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal...

OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal...

OpenNebula Project

Presented at FOSDEM 2021: https://fosdem.org/2021/schedule/event/k8sconfigmgmtlandscape/ “We are all YAML engineers now” as Bob Walker said in 2018 in Ghent. So we now need something to manage these millions of lines of YAML. There’s the “Kubernetes application management tools” list by Bryan Grant with over 120 tool aiming at config management for Kubernetes. This area is evolving fast, and as k8s users we need some guidance to make informed decisions on what to use. In this talk I’ll describe the problem I need to solve and take a look into what we can learn from the previous generation of cfg mgmt tools. Then, will show some of the new tools and methods. Will not go into details of each solution, but rather compare different approaches and discuss which is good for specific needs.

Kubernetes Config Management Landscape

Kubernetes Config Management Landscape

Kubernetes Config Management Landscape

Tomasz Tarczyński

OpenCloud - A Research Cloud

OpenCloud - A Research Cloud

OpenCloud - A Research Cloud

In this talk Kim-Norman Sahm and Alexander Trost dive into the challenges of storage for containerized applications on Kubernetes. We'll see how the current state is and how Rook can help with that. We are going to especially look at Ceph run through Rook here, but nonetheless trying not to lose sight of the whole picture. There is a lot to keep in mind storage as is, but everything gets more complex with storage for containers. From what type of storage to how much and how "safe" it should, all questions that should be asked and most of them which should be answered as well. Rook's project site https://rook.io/ Kim-Norman Sahm is CTO of Cloudical. He also works as Executive Cloud Architect at Cloudical. Previously, he was OpenStack Cloud Architect at T-Systems (operational services GmbH) and noris network AG. He is an expert of the technologies OpenStack, Ceph and Kubernetes (CKA). Alexander Trost works as a DevOps Engineer at Cloudical Deutschlang GmbH and is a Certified Kubernetes Administrator (CKA). He is one of four maintainers of the Rook.io Project and is engaged in several more open source projects, for example a Prometheus exporter for Dell Hardware (Dell OMSA Metrics), k8s-vagrant-multi-node an easy local multi node Kubernetes environment, and others. Besides Containers and Kubernetes he is expert on Software Defined Storage, Golang and Continuous Integration (with GitLab CI). He passionately enjoys working on open source projects, such as Rook, Ancientt and other projects.

Rook: Storage for Containers in Containers – data://disrupted® 2020

Rook: Storage for Containers in Containers – data://disrupted® 2020

Rook: Storage for Containers in Containers – data://disrupted® 2020

data://disrupted®

Seventh openCypher Implementers Group Meeting: Status Update

Seventh openCypher Implementers Group Meeting: Status Update

Seventh openCypher Implementers Group Meeting: Status Update

Presto Bangalore Meetup1 Event Listeners@qubole

Presto Bangalore Meetup1 Event Listeners@qubole

Presto Bangalore Meetup1 Event Listeners@qubole

Redis Day TLV 2018 - RediSearch Aggregations

Redis Day TLV 2018 - RediSearch Aggregations

Redis Day TLV 2018 - RediSearch Aggregations

What's hot (20)

TiDB Introduction - San Francisco MySQL Meetup

TiDB Introduction - San Francisco MySQL Meetup

TiDB Introduction - San Francisco MySQL Meetup

DSD-INT 2014 - OpenGIS Workshop - Gridded data, Giorgio Santinelli, Deltares

DSD-INT 2014 - OpenGIS Workshop - Gridded data, Giorgio Santinelli, Deltares

DSD-INT 2014 - OpenGIS Workshop - Gridded data, Giorgio Santinelli, Deltares

Trondheim Eclipe Day 2015 and 2016

Trondheim Eclipe Day 2015 and 2016

Trondheim Eclipe Day 2015 and 2016

Введение в Apache Cassandra

Введение в Apache Cassandra

Введение в Apache Cassandra

Debugging and Profiling Rails Application

Debugging and Profiling Rails Application

Debugging and Profiling Rails Application

IPTC News Exchange Working Group 2013 Autumn Meeting

IPTC News Exchange Working Group 2013 Autumn Meeting

IPTC News Exchange Working Group 2013 Autumn Meeting

Quantum Gaussian Processes - Gawel Kus

Quantum Gaussian Processes - Gawel Kus

Quantum Gaussian Processes - Gawel Kus

SWT Final Project Presentation

SWT Final Project Presentation

SWT Final Project Presentation

TiDB + Mobike by Kevin Xu (@kevinsxu)

TiDB + Mobike by Kevin Xu (@kevinsxu)

TiDB + Mobike by Kevin Xu (@kevinsxu)

Strings, C# and Unmanaged Memory

Strings, C# and Unmanaged Memory

Strings, C# and Unmanaged Memory

Introducing MagnetoDB, a key-value storage sevice for OpenStack

Introducing MagnetoDB, a key-value storage sevice for OpenStack

Introducing MagnetoDB, a key-value storage sevice for OpenStack

CEDAR & PRELIDA Preservation of Linked Socio-Historical Data

CEDAR & PRELIDA Preservation of Linked Socio-Historical Data

CEDAR & PRELIDA Preservation of Linked Socio-Historical Data

Third openCypher Implementers Group Meeting: Status Update

Third openCypher Implementers Group Meeting: Status Update

Third openCypher Implementers Group Meeting: Status Update

OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal...

OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal...

OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal...

Kubernetes Config Management Landscape

Kubernetes Config Management Landscape

Kubernetes Config Management Landscape

OpenCloud - A Research Cloud

OpenCloud - A Research Cloud

OpenCloud - A Research Cloud

Rook: Storage for Containers in Containers – data://disrupted® 2020

Rook: Storage for Containers in Containers – data://disrupted® 2020

Rook: Storage for Containers in Containers – data://disrupted® 2020

Seventh openCypher Implementers Group Meeting: Status Update

Seventh openCypher Implementers Group Meeting: Status Update

Seventh openCypher Implementers Group Meeting: Status Update

Presto Bangalore Meetup1 Event Listeners@qubole

Presto Bangalore Meetup1 Event Listeners@qubole

Presto Bangalore Meetup1 Event Listeners@qubole

Redis Day TLV 2018 - RediSearch Aggregations

Redis Day TLV 2018 - RediSearch Aggregations

Redis Day TLV 2018 - RediSearch Aggregations

Similar to The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

The Current Messaging Landscape

The Current Messaging Landscape

The Current Messaging Landscape

Great Wide Open

Among the resources offered by Wikimedia is an API providing low-latency access to full-history content, in many formats. Its results are often the product of computationally intensive transforms, and must be pre-generated and stored to meet latency expectations. Unsurprisingly, there are many challenges to providing low-latency access to such a large data-set, in a demanding, globally distributed environment. This presentation covers the Wikimedia content API and its use of Apache Cassandra as storage for a diverse and growing set of use-cases. Trials, tribulations, and triumphs, of both a development and operational nature will be discussed.

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Most Cassandra usages take advantage of its exceptional performance and ability to handle massive data sets. At PagerDuty, we use Cassandra for entirely different reasons: to reliably manage mutable application states and to maintain durability requirements even in the face of full data center outages. We achieve this by deploying Cassandra clusters with hosts in multiple WAN-separated data centers, configured with per-data center replica placement requirements, and with significant application-level support to use Cassandra as a consistent datastore. Accumulating several years of experience with this approach, we've learned to accommodate the impact of WAN network latency on Cassandra queries, how to horizontally scale while maintaining our placement invariants, why asymmetric load is experienced by nodes in different data centers, and more. This talk will go over our workload and design goals, detail the resultant Cassandra system design, and explain a number of our unintuitive operational learnings about this novel Cassandra usage paradigm.

PagerDuty: Span the WAN? Yes you can!

PagerDuty: Span the WAN? Yes you can!

PagerDuty: Span the WAN? Yes you can!

DataStax Academy

2015 03-31 dom-code_cassandra

2015 03-31 dom-code_cassandra

2015 03-31 dom-code_cassandra

Freek van Polen

Learn from HomeAway Hadoop Development and Operations Best Practices

Learn from HomeAway Hadoop Development and Operations Best Practices

Learn from HomeAway Hadoop Development and Operations Best Practices

Hundreds of millions of people use Quora to find accurate, informative, and trustworthy answers to their questions. As it so happens, counting things at scale is both an important and a difficult problem to solve. In this talk, we will be talking about Quanta, Quora's counting system built on top of HBase that powers our high-volume near-realtime analytics that serves many applications like ads, content views, and many dashboards. In addition to regular counting, Quanta supports count propagation along the edges of an arbitrary DAG. HBase is the underlying data store for both the counting data and the graph data. We will describe the high-level architecture of Quanta and share our design goals, constraints, and choices that enabled us to build Quanta very quickly on top of our existing infrastructure systems.

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

When machine learning models are productionized, they are commonly formed as workflows with multiple tasks, managed by a task scheduler such as Airflow, Prefect. Traditionally each task within the same workflow uses similar computing frameworks (e.g. Python, Spark, and PyTorch) in the same backend computing environment (e.g. AWS EMR, Google DataProc) with globally fixed settings (e.g. instances, cores, memory). In complicated use cases, such traditional workflows create large resource and runtime inefficiency, hence it is highly desired to use different computing frameworks in the same workflow in different computing environments. Such workflows can be named as superworkflows. Fugue is an open-sourced abstraction layer on top of different computing frameworks and creates uniform interfaces to use these frameworks without dealing with the complexities associated with them. To this end, Fugue can be viewed as a superframework. In addition, Kubernetes (K8S) is a container orchestration system, and it is easy to create different computing environments (e.g. Spark, PyTorch) with different docker images as everything is containerized in K8S. It is natural to combine K8S and Fugue to create superworkflows for complicated machine learning problems. In this talk, we use a popular graph neural network named Node2Vec as an example to illustrate how to create an efficient superworkflow using Fugue and K8S on very large graphs with hundreds of millions of vertices and edges. We also demonstrate how to partition the whole Node2Vec process into multiple tasks based on their complexities and parallelism. Benchmark testing is conducted for comparing performance and resource efficiency. Finally, it is easy to generalize this superworkflow concept to other deep learning problems.

Superworkflow of Graph Neural Networks with K8S and Fugue

Superworkflow of Graph Neural Networks with K8S and Fugue

Superworkflow of Graph Neural Networks with K8S and Fugue

KubeCon US 2021 - Recap - DCMeetup

KubeCon US 2021 - Recap - DCMeetup

KubeCon US 2021 - Recap - DCMeetup

"Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community!"

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

HostedbyConfluent

Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community!

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Yaroslav Tkachenko

As your organization rapidly grows in scale, so do the amount of challenges. Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc. Coming up with an architecture that can handle all of the data flows in a universal, simple way is key. This talk is about Wix's Kafka based global data architecture and platform. How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use. All the while offering various tools and features for adapting to growing scale and insuring high resilience.

Kafka based Global Data Mesh at Wix

Kafka based Global Data Mesh at Wix

Kafka based Global Data Mesh at Wix

Natan Silnitsky

Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...

Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...

Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...

Seattle Apache Flink Meetup

Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...

Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...

Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...

S2

Voldemort : Prototype to Production

Voldemort : Prototype to Production

Voldemort : Prototype to Production

Event description: Why Tangent Works chooses Julia: The Two Language Problem TIM: Automatic Model Building for Energy Industry Julia and its major differences to other technical computing languages (R, Matlab, ...) - Why is vectorized code fast? - Why is it not as fast as it could be? Speaker: Ján Dolinský, Tangent Works (www.tangent.works) Language of the event: Julia, Slovak & English ------------------------------------ PyData Bratislava [Python Data Enthusiasts and Users, Data Scientists & Statisticians of all levels from Slovakia] ------------------------------------ -- This meetup group is for Data Scientists, Statisticians, Economists and Data Enthusiasts using Python for data analysis and data visualization. The goals are to provide Python enthusiasts a place to share ideas and learn from each other about how best to apply the language and tools to ever-evolving challenges in the vast realm of data management, processing, analytics, and visualization. -- PyData is a group for users and developers of data analysis tools to share ideas and learn from each other. We gather to discuss how best to apply Python tools, as well as those using R and Julia, to meet the evolving challenges in data management, processing, analytics, and visualization. PyData groups, events, and conferences aim to provide a venue for users acrossall the various domains of data analysis to share their experiences and their techniques. PyData is organized by NumFOCUS.org, a 501(c)3 non-profit in the United States. ------------------------------------ Our Facebook group here: https://www.facebook.com/groups/1813599648877946/ Our Twitter account here: https://twitter.com/PyDataBA Our LinkedIn group here: https://www.linkedin.com/groups/13506080 All materials from previous meetups on GitHub here: https://github.com/GapData/PyDataBratislava Recordings of previous meetups on our YouTube here: https://www.youtube.com/watch?v=XYpKpmapqjI&list=PLISV6olKXnd9pE-KPtPgwwLe6qPXvb9K7 ------------------------------------ Organizers: GapData Institute (https://www.gapdata.org/) (GDI) is a nonprofit nonpartisan research institution harnessing power of data & wisdom of economics for public good. || Data. Think. Change. || -- NumFOCUS (http://www.numfocus.org/) is a 501(c)(3) nonprofit that supports and promotes world-class, innovative, open source scientific computing. The mission of NumFOCUS is to promote sustainable high-level programming languages, open code development, and reproducible scientific research. We accomplish this mission through our educational programs and events as well as through fiscal sponsorship of open source data science projects. We aim to increase collaboration and communication within the scientific computing community.

TIM: Large-scale Energy Forecasting in Julia

TIM: Large-scale Energy Forecasting in Julia

TIM: Large-scale Energy Forecasting in Julia

GapData Institute

Streaming process with Kafka Connect and Kafka Streams

Streaming process with Kafka Connect and Kafka Streams

Streaming process with Kafka Connect and Kafka Streams

NoSQL Intro with cassandra

NoSQL Intro with cassandra

NoSQL Intro with cassandra

Overview of Lincoln Paper Design

Overview of Lincoln Paper Design

Overview of Lincoln Paper Design

Similar to The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka (20)

The Current Messaging Landscape

The Current Messaging Landscape

The Current Messaging Landscape

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

Wikimedia Content API: A Cassandra Use-case

PagerDuty: Span the WAN? Yes you can!

PagerDuty: Span the WAN? Yes you can!

PagerDuty: Span the WAN? Yes you can!

2015 03-31 dom-code_cassandra

2015 03-31 dom-code_cassandra

2015 03-31 dom-code_cassandra

Learn from HomeAway Hadoop Development and Operations Best Practices

Learn from HomeAway Hadoop Development and Operations Best Practices

Learn from HomeAway Hadoop Development and Operations Best Practices

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

Superworkflow of Graph Neural Networks with K8S and Fugue

Superworkflow of Graph Neural Networks with K8S and Fugue

Superworkflow of Graph Neural Networks with K8S and Fugue

KubeCon US 2021 - Recap - DCMeetup

KubeCon US 2021 - Recap - DCMeetup

KubeCon US 2021 - Recap - DCMeetup

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Kafka based Global Data Mesh at Wix

Kafka based Global Data Mesh at Wix

Kafka based Global Data Mesh at Wix

Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...

Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...

Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...

Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...

Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...

Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...

S2

Voldemort : Prototype to Production

Voldemort : Prototype to Production

Voldemort : Prototype to Production

TIM: Large-scale Energy Forecasting in Julia

TIM: Large-scale Energy Forecasting in Julia

TIM: Large-scale Energy Forecasting in Julia

Streaming process with Kafka Connect and Kafka Streams

Streaming process with Kafka Connect and Kafka Streams

Streaming process with Kafka Connect and Kafka Streams

NoSQL Intro with cassandra

NoSQL Intro with cassandra

NoSQL Intro with cassandra

Overview of Lincoln Paper Design

Overview of Lincoln Paper Design

Overview of Lincoln Paper Design

More from All Things Open

Presented at the ATO RTP Meetup Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally Title: Building Reliability - The Realities of Observability Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

All Things Open

Presented at the ATO RTP Meetup Presented by Peter Zaitsev, Founder of Percona Title: Modern Database Best Practices Abstract: There are now more Database choices available for developers than ever before - there are general purpose databases and specialized databases, single node and distributed databases, Open Source, Proprietary databases and databases available exclusively in the cloud. In this presentation we will cover the best practices of choosing database(s) for your applications, best practices as it comes to application development as well as managing those databases to achieve best possible performance, security, availability at the lowest cost.

Modern Database Best Practices

Modern Database Best Practices

Modern Database Best Practices

All Things Open

All Things Open 2023 Presented at All Things Open 2023 Presented by Deb Bryant - Open Source Initiative, Patrick Masson - Apereo Foundation, Stephen Jacobs - Rochester Institute of Technology, Ruth Suehle - SAS, & Greg Wallace - FreeBSD Foundation Title: Open Source and Public Policy Abstract: New regulations in the software industry and adjacent areas such as AI, open science, open data, and open education are on the rise around the world. Cyber Security, societal impact of AI, data and privacy are paramount issues for legislators globally. At the same time, the COVID-19 pandemic drove collaborative development to unprecedented levels and took Open Source software, open research, open content and data from mainstream to main stage, creating tension between public benefit and citizen safety and security as legislators struggle to find a balance between open collaboration and protecting citizens. Historically, the open source software community and foundations supporting its work have not engaged in policy discussions. Moving forward, thoughtful development of these important public policies whilst not harming our complex ecosystems requires an understanding of how our ecosystem operates. Ensuring stakeholders without historic benefit of representation in those discussions becomes paramount to that end. Please join our open discussion with open policy stakeholders working constructively on current open policy topics. Our panelists will provide a view into how oss foundations and other open domain allies are now rising to this new challenge as well as seizing the opportunity to influence positive changes to the public’s benefit. Topics: Public Policy, Open Science, Open Education, current legislation in the US and EU, US interest in OSS sustainability, intro to the Open Policy Alliance Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Open Source and Public Policy

Open Source and Public Policy

Open Source and Public Policy

All Things Open

Presented at All Things Open 2023 Presented by Ashpak Shaikh & Lucy Shen - Intuit Title: Weaving Microservices into a Unified GraphQL Schema with graph-quilt Abstract: The magic of GraphQL is that it provides data access through a single endpoint—clean and easy. But as the number of GraphQL microservices your tech stack depends on starts to grow, that single-endpoint purpose becomes a new multi-endpoint problem. Ideally, we would have an orchestrator that could aggregate schemas from multiple microservices into a unified GraphQL schema and route the requests to the appropriate microservice. Enter graph-quilt, an open source Java library that provides recursive schema stitching and Apollo Federation style schema composition. In this talk, we’ll walk through our GraphQL journey and show you how to use graph-quilt to simplify your data orchestration needs. We will also share our open sourced reference implementation of a highly performant graph-quilt gateway currently being used in production here at Intuit, where we’ve had incredible success in scaling the gateway with 50+ microservices and 150+ clients. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

All Things Open

Presented at All Things Open 2023 Presented by Phil Nash - Sonar Title: The State of Passwordless Auth on the Web Abstract: Can we get rid of passwords yet? They make for a poor user experience and users are notoriously bad with them. The advent of WebAuthn has brought a passwordless world closer, but where do we really stand? In this talk we'll explore the current user experience of WebAuthn and the requirements a user has to fulfil to authenticate without a password. We'll also explore the fallbacks and safeguards we can use to make the password experience better and more secure. By the end of the session you'll have a vision of how authentication could look in the future and a blueprint for how to build the best auth experience today. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

All Things Open

Presented at All Things Open 2023 Presented by Phil Nash - Sonar Title: Total ReDoS: The dangers of regex in JavaScript Abstract: Regular expressions are complicated and can be hard to learn. On top of that, they can also be a security risk; writing the wrong pattern can open your application up to denial of service attacks. One token out of place and you invite in the dreaded ReDoS. But how can a regular expression cause this? In this talk we’ll track down the patterns that can cause this trouble, explain why they are an issue and propose ways to fix them now and avoid them in the future. Together we’ll demystify these powerful search patterns and keep your application safe from expressions that behave in a way that is anything but regular. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

All Things Open

Presented at All Things Open 2023 Presented by Karl Mozurkewich - Storj Title: What Does Real World Mass Adoption of Decentralized Tech Look Like? Abstract: We delve into the transformative potential of decentralized technology. Beginning with a brief overview of the rise of centralization with the advent of the internet and the counter-shift marked by blockchain we explore the intrinsic characteristics of decentralized and distributed systems, such as trustless operations, peer-to-peer networks, and enterprise application scalability. Various sectors, including finance, supply chains, media and entertainment, data science and cloud infrastructure are on the brink of disruption. The societal implications are vast, with the potential for greater individual empowerment, a greener planet and more viable resource utilization, but concerns about data security persist. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

All Things Open

Presented at All Things Open 2023 Presented by Anastasia Lalamentik - Kaleido Title: How to Write & Deploy a Smart Contract Abstract: In this talk, Anastasia Lalamentik, Full Stack Engineer at Kaleido, will walk through how Ethereum smart contracts work and go over related concepts like gas fees, the Ethereum Virtual Machine (EVM), the block explorer, and the Solidity programming language. This is vital to anyone who wants to build a blockchain app and is a great introduction to blockchain technology for newcomers to the space. By the end of the talk, attendees will better understand how to: - Write a simple smart contract - Deploy their smart contract to an Ethereum test network through the latest tools like Hardhat and the MetaMask wallet - Test interactions with their deployed smart contract and ensure that everything is working properly Additionally, participants will get to interact with Anastasia's deployed smart contract at the end of the talk. Anastasia’s past talks have attracted and have been attended by a diverse group of participants with a range of experience in the space. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

All Things Open

Presented at All Things Open 2023 Presented by Paul Brebner - Instaclustr (by Spot by NetApp) Title: Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Abstract: In this talk we’ll build a Drone delivery application, and then use it to do some Machine Learning “on the fly”. In the 1st part of the talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming data). With up to 2,000 (simulated) drones and deliveries in progress at once this application generates a vast flow of spatio-temporal data. In the 2nd part of the talk, we'll use this platform to explore Machine Learning (ML) over streaming and drifting Kafka data with TensorFlow to try and predict which shops will be busy in advance. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

All Things Open

Presented at the All Things Open 2023 Inclusion and Diversity in Open Source Event Presented by Efraim Marquez-Arreaza - Red Hat Title: DEI Challenges and Success Abstract: In today's world, many companies and organizations have Diversity, Equity and Inclusion (DEI) communities. Red Hat Unidos is a DEI community focused on advocating for the Hispanic/Latine community. In this talk, we would like to share our challenges and success during the past 4-years and plans for the future. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

DEI Challenges and Success

DEI Challenges and Success

DEI Challenges and Success

All Things Open

Presented at All Things Open 2023 Presented by Lydia Cupery - HubSpot Title: Scaling Web Applications with Background Jobs: Takeaways from Generating a Huge PDF Abstract: Do you need to perform time-consuming or CPU-intensive processes in your web application but are concerned about performance? That’s where background jobs come in. By offloading resource-intensive tasks to separate worker processes, you can improve the scalability of your web application. In this talk, I'll share my experience of using background jobs to scale our web application. I'll discuss the challenges my team faced that led us to adopt background jobs. Then, I'll share practical tips on how to design background jobs for CPU-intensive or time-consuming processes, such as generating huge PDFs and batch emailing. I'll wrap up by going over the performance and cost tradeoffs of background jobs. I'll use Typescript, Express, and Heroku as examples in this talk, but the concepts and best practices that I'll share are applicable to other languages and tools. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Scaling Web Applications with Background

Scaling Web Applications with Background

Scaling Web Applications with Background

All Things Open

Presented at All Things Open 2023 Presented by Robert Aboukhalil - CZI Title: Supercharging tutorials with WebAssembly Abstract: sandbox.bio is a free platform that features interactive command-line tutorials for bioinformatics. This talk is a deep-dive into how sandbox.bio was built, with a focus on how WebAssembly enabled bringing command-line tools like awk and grep to the web. Although these tools were originally written in C/C++, they all run directly in the browser, thanks to WebAssembly! And since the computations run on each user's computer, this makes the application highly scalable and cost-effective. Along the way, I'll discuss how WebAssembly works and how to get started using it in your own applications. The talk will also cover more advanced WebAssembly features such as threads and SIMD, and will end with a discussion of WebAssembly's benefits and pitfalls (it's a powerful technology, but it's not always the right tool!). Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

All Things Open

Presented at All Things Open 2023 Presented by K.S. Bhaskar - YottaDB LLC Title: Using SQL to Find Needles in Haystacks Abstract: Database journal files capture every update to a database. A database of a few hundred GB can generate GBs worth of journal files every minute at busy times. Troubleshooting and forensices, especially of rare and intermittent problems, such as which process made what update and when, is an exercise of finding needles in haystacks. A similar problem exists with syslogs. A solution is to load the journal files and syslogs into a database, and use SQL to query the database. Bhaskar will present and demonstrate this with a 100% FOSS stack. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

All Things Open

Presented at All Things Open 2023 Presented by Wes Widner - Automox Title: Configuration Security as a Game of Pursuit Intercept Abstract: In this session we will take a look at the emerging field of cloud security posture management and how we can approach the problem space using a class of board games known as pursuit/intercept. Using the game Scotland Yard as a visual illustration we'll explore the cognitive and technical limitations that all CSPM systems face and what you should look for when evaluating the strengths and weakness of CSPM vendors and approaches. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

All Things Open

Presented at All Things Open 2023 Presented by Carol Huang & Mike Fix - Stripe Title: Scaling an Open Source Sponsorship Program Abstract: We already know this: the open-source ecosystem needs further monetary investment from the companies that benefit most from it. Likewise, companies say they want to participate in these initiatives, but find it hard to dedicate resources to open source funding when there isn’t a clear ROI. This talk discusses how the Open Source Program Office at Stripe built a scalable, sustainable open source sponsorship model that aligns internal company incentives with those of open source maintainers and the community at large. We go over the unique “platformization” of our OSPO that allowed us to create multiple funding models, such as BYOB (Bring Your Own Budget), and share lessons learned from this experience as well as other OSPOs. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

All Things Open

Presented at All Things Open 2023 Presented by Arundeep Nagaraj - Amazon Web Services (AWS) Title: Build Developer Experience Teams for Open Source Abstract: Open Source has become the default strategy for many IT organizations and Enterprises. However, the constant challenge with Open Source leaders of these organizations has been - How is my product's developer experience? Is this the right metric to track? How can I scale my team to support our products better? How can I add automation to scale redundant workflows? If my product involves working with developers, how can I scale to the complexity of the requests and reduce Engineering bandwidth? The challenges within support of open source products continues to magnify depending on the end user persona whether they are consumers or contributors to your product. Consumers utilize your product, SDK's and API's and are blocked with using it or run into issues, whereas contributors are advanced users of your software that understands the codebase to provide a meaningful contribution back to the product. The answer to the above is to look at Open Source support as a first-class citizen of your corporate support strategy. To employ the right level of developer focused support as opposed to traditional infrastructure based support is key to scale to the amount of developers using your product. Supporting customers in the open involves more than pure support - building customer / developer experiences (DX) in the open (across platforms and communities) that pivots over the ability of your product's users or developers to be focused on the end-to-end value add. This helps with your active developer growth and retention of users. Key Takeaways: - IT leaders of Open Source will learn to employ strategies to build a DX team that engages on multiple platforms - Work on identifying accurate metrics for product and organization - Innovate on platforms such as Discord to build a bot and a dashboard - Ability to leverage customer feedback and iterate over the customer success flywheel - Distinguish between DX and Developer Advocacy (DA) Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

All Things Open

Presented at All Things Open 2023 Presented by Danny McCormick - Google Title: Deploying Models at Scale with Apache Beam Abstract: Apache Beam is an open source tool for building distributed scalable data pipelines. This talk will explore how Beam can be used to perform common machine learning tasks, with a heavy focus on running inference at scale. The talk will include a demo component showing how Beam can be used to deploy and update models efficiently on both CPUs and GPUs for inference workloads. An attendee can expect to leave this talk with a high level understanding of Beam, the challenges of deploying models at scale, and the ability to use Beam to easily parallelize their inference workloads. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

All Things Open

Presented at All Things Open 2023 Presented by Peter Czanik - One Identity Title: Sudo – Giving access while staying in control Abstract: Sudo is used by millions to control and log administrator access to systems, but using the default configuration only, there are plenty of blind spots. Using the latest features in sudo let you watch some previously blind spots and control access to them. Here are four major new features, which arrived since the 1.9.0 release, allowing you see your blind spots: - configuring a working directory or chroot within sudo often makes full shell access redundant - JSON-formatted logs give you more details on events and are easier to act on - relays in sudo_logsrvd make session recording collection more secure and reliable - you can log and control sub-commands executed by the command run through sudo Let us take a closer look at each of these. Previously, there were quite a few situations where you had to give users full shell access through sudo. Typical examples include when you need to run a command from a given directory, or running commands in a chroot environment. You can now configure the working directory or the chroot directory and give access only to the command the user really needs. Logging is a central role of sudo, to see who did what on the system. Using JSON-formatted log messages gives you even more information about events. What is even more: structured logs are easier to act on. Setting up alerting for suspicious events is much easier when you have a single parser to configure for any kind of sudo logs. You can collect sudo logs not only by local syslog, but also by using sudo_logsrvd, the same application used to collect session recordings. Speaking of session recordings: instead of using a single central server, you can now have multiple levels of sudo_logsrvd relays between the client and the final destination. This allows session collection even if the central server is unavailable, providing you with additional security. It also makes your network configuration simpler. Finally, you can log sub-commands executed from the command started through sudo. You can see commands started from a shell. No more unnoticed shell access from text editors. Best of all: you can also intercept sub-commands. These are just a few of the most prominent features helping you to watch and control previous blind spots on your systems. See these and other possibilities in action in some live demos during our presentation. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

All Things Open

Presented at All Things Open 2023 Presented by Christine Abernathy - F5, Inc. Title: Fortifying the Future: Tackling Security Challenges in AI/ML Applications Abstract: As Artificial Intelligence (AI) and Machine Learning (ML) applications continue to surge, it is crucial to be aware of and address the security risks associated with these technologies. In this talk, Christine will explore AI/ML failure modes, threats, and mitigation strategies. She will guide you through the fundamentals of ML models then introduce you to key security challenges such as adversarial attacks, data poisoning, model inversion, model stealing, and membership inference attacks, using real-world examples to demonstrate their potential impact. Christine will also discuss privacy and ethical considerations in ML, touching upon techniques like federated learning and shedding light on the current regulatory landscape surrounding security risks. If you are developing AI/ML applications or incorporating AI/ML components into your technology stack, check out this talk. You will walk away with a deeper understanding of the current AI/ML security landscape and a toolkit to help you address these risks, enabling you to build safer, more secure, and privacy-aware applications. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

All Things Open

Presented at All Things Open 2023 Presented by Carlos Santana - AWS Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you! We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance. By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

All Things Open

More from All Things Open (20)

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

Modern Database Best Practices

Modern Database Best Practices

Modern Database Best Practices

Open Source and Public Policy

Open Source and Public Policy

Open Source and Public Policy

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

DEI Challenges and Success

DEI Challenges and Success

DEI Challenges and Success

Scaling Web Applications with Background

Scaling Web Applications with Background

Scaling Web Applications with Background

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Recently uploaded

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

Choosing the right accounts payable services provider is a strategic decision that can significantly impact your business's financial performance and operational efficiency. By considering factors such as expertise, range of services, technology infrastructure, scalability, cost, and reputation, businesses can make informed decisions and select a provider that aligns with their unique needs and objectives. Partnering with the right provider can streamline accounts payable processes, drive cost savings, and position your business for long-term success. https://katprotech.com/accounts-payable-and-purchase-order-automation/

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Katpro Technologies

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Recently uploaded (20)

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

1.

2. Michael Laing Architect, Edge Engineering New York Times michael.laing@nytimes.com

3.

4.

5. • nyt⨍aбrik (my project)

6.

7.

8.

9.

10.

11.

12.

13. A Global Mesh with a Memory Millions of users Dozens of internal clients

14.

15.

16.

17.

18.

19. • nyt⨍aбrik (my project)

20.

21.

22.

23. by wmasterj and Fabrice TIERCELIN - Wikipedia

24.

25.

26.

28. Async Scope

29.

30. Time P C Idle Idle Idle

31. Time P C CP Idle Idle Idle Idle latency

32. • nyt⨍aбrik (my project)

33.

34.

35.

36.

37.

38.

39.

40. Database Semantics For access

41.

42.

43. GW GW GW GW GW GW Region A Region B

44. Push Pull snd rcv

45.

46.

47.

48.

49. Written in C

50. Push Pull snd rcv

51.

52. Push Pull snd rcv NSQ

54. Push Pull snd rcv NSQ What if…

55. GW GW GW GW GW GW Region A Region B Then:

56. Region A Region B GW GW GW GW GW GW Becomes:

57.

58.

59.

60.

61. • nyt⨍aбrik (my project)

62.

63.

65.

66.

67.

68.

69. Distributed Graphs at Scale with Cassandra and Titan ● Structured Knowledge Representation ● Index-free adjacency: Like memory pointers, but in disk ● Navigation between nodes in constant time. ● Graph != No schema

70. Michael Laing Architect, Edge Engineering (soon to include OLAP w Spark Streaming) New York Times michael.laing@nytimes.com

Editor's Notes

Back to 1964
Not me Nice shoes though GE 225 – 8KB
Rethink Flatten Global Reliable Manageable
Mesh Resilient Balancing
Nodes w Roles Self organizing
Messaging everywhere
GLobal
Messaging everywhere
Header = Metadata: routing, timestamp, source
Messaging everywhere
Designed to fail fast
Designed to fail fast
Designed to fail fast
Designed to fail fast
Rethink Flatten Global Reliable Manageable
Rethink Flatten Global Reliable Manageable
Rethink Flatten Global Reliable Manageable
Rethink Flatten Global Reliable Manageable
Rethink Flatten Global Reliable Manageable
This powers The New York Times Semantic Platform: mechanism for accessing all of our knowledge about our indexing concepts: people, places organizations and descriptors. This is an example of a more complex traversal. Let’s say we have a concept called Ebola Virus, and we want to retrieve all the ancestors of this concepts. That means all concepts that have a broader meaning than Ebola Virus. So in theory, we need to iterate through adjacent edges and vertices until we can’t find any other broader term.