Incrementally streaming rdbms data to your data lake automagically

•

1 like•641 views

Incrementally streaming rdbms data to your data lake automagically using Apache NiFi to load Oracle data to Apache Hive, Apache Kudu, Apache Impala, Apache HDFS

Technology

APACHECON @HOME
Spt, 29th – Oct. 1st 2020

•
•
•
•
Incrementally Streaming RDBMS Data

5
Future of Data - Princeton
@PaasDev
https://www.meetup.com/futureofdata-princeton/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

6
Speakers
John Kuchmek
Senior Solutions Engineer

7
Speakers
Tim Spann
Principal DataFlow Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB
Princeton NJ Future of Data Meetup
https://github.com/tspannhw
https://www.datainmotion.dev/

8
JDBC Database to Apache Kudu / JDBC Database to HDFS and Hive

9
Trillions of Messages to SQL Databases and Data Warehouses
PROCESS DELIVER

10
https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html

11
Reference Architecture
Files to RDBMS

‘QueryRecord’ Processor
https://medium.com/@abdelkrim.hadjidj/democratizing-niﬁ-record-processors-with-automatic-schemas-inferenc
e-4f2b2794c427

ADVANCED XML PROCESSING
https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html
https://pierrevillard.com/2018/06/28/niﬁ-1-7-xml-reader-writer-and-forkrecord-processor/

• Example
• Flat ﬁles on an FTP
server named by date
• Downloads ﬁle
• HTTP REST API
endpoint
• Invokes API and
downloads data
• Legacy/Remote DB
• Performs SQL queries
1
5
DBCP Connection Pool to remote SQL Server
ExecuteSQLRecord processor

•
1
6
https://www.datainmotion.dev/2019/03/implementing-streaming-use-case-from.html

INGEST RDBMS TABLES
https://community.cloudera.com/t5/Community-Articles/Incrementally-Streaming-RDBMS-Data-to-Your-Hadoop-DataLake/ta-p/247927

https://dzone.com/articles/lets-build-a-simple-ingest-to-cloud-data-warehouse

What's hot

Stable and robust ETL pipelines are a critical component of the data infrastructure of modern enterprises. ETL pipelines ingest data from a variety of sources and must handle incorrect, incomplete or inconsistent records and produce curated, consistent data for consumption by downstream applications. In this talk, we’ll take a deep dive into the technical details of how Apache Spark “reads” data and discuss how Spark 2.2’s flexible APIs; support for a wide variety of datasources; state of art Tungsten execution engine; and the ability to provide diagnostic feedback to users, making it a robust framework for building end-to-end ETL pipelines.

Building Robust ETL Pipelines with Apache Spark

Databricks

Dremio, une architecture simple et performance pour votre data lakehouse. Dans le monde de la donnée, Dremio, est inclassable ! C’est à la fois une plateforme de diffusion des données, un moteur SQL puissant basé sur Apache Arrow, Apache Calcite, Apache Parquet, un catalogue de données actif et aussi un Data Lakehouse ouvert ! Après avoir fait connaissance avec cette plateforme, il s’agira de préciser comment Dremio aide les organisations à relever les défis qui sont les leurs en matière de gestion et gouvernance des données facilitant l’exécution de leurs analyses dans le cloud (et/ou sur site) sans le coût, la complexité et le verrouillage des entrepôts de données.

Vue d'ensemble Dremio

Modern Data Stack France

Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. In this talk, learn how to operationalize ML across the full lifecycle with Databricks Machine Learning.

Learn to Use Databricks for the Full ML Lifecycle

Databricks

Siligong.Data - May 2021 - Transforming your analytics workflow with dbt

Jon Su

Modernize & Automate Analytics Data Pipelines

Carole Gunst

Join operations in Apache Spark is often the biggest source of performance problems and even full-blown exceptions in Spark. After this talk, you will understand the two most basic methods Spark employs for joining DataFrames – to the level of detail of how Spark distributes the data within the cluster. You’ll also find out how to work out common errors and even handle the trickiest corner cases we’ve encountered! After this talk, you should be able to write performance joins in Spark SQL that scale and are zippy fast! This session will cover different ways of joining tables in Apache Spark. Speaker: Vida Ha This talk was originally presented at Spark Summit East 2017.

Optimizing Apache Spark SQL Joins

Databricks

Introducing Databricks Delta

Databricks

As organizations pursue Big Data initiatives to capture new opportunities for data-driven insights, data governance has become table stakes both from the perspective of external regulatory compliance as well as business value extraction internally within an enterprise. This session will introduce Apache Atlas, a project that was incubated by Hortonworks along with a group of industry leaders across several verticals including financial services, healthcare, pharma, oil and gas, retail and insurance to help address data governance and metadata needs with an open extensible platform governed under the aegis of Apache Software Foundation. Apache Atlas empowers organizations to harvest metadata across the data ecosystem, govern and curate data lakes by applying consistent data classification with a centralized metadata catalog. In this talk, we will present the underpinnings of the architecture of Apache Atlas and conclude with a tour of governance capabilities within Apache Atlas as we showcase various features for open metadata modeling, data classification, visualizing cross-component lineage and impact. We will also demo how Apache Atlas delivers a complete view of data movement across several analytic engines such as Apache Hive, Apache Storm, Apache Kafka and capabilities to effectively classify, discover datasets.

Apache Atlas: Governance for your Data

DataWorks Summit/Hadoop Summit

Airbnb has a wide variety of ML problems ranging from models on traditional structured data to models built on unstructured data such as user reviews, messages and listing images. The ability to build, iterate on, and maintain healthy machine learning models is critical to Airbnb’s success. Many ML Platforms cover data collection, feature engineering, training, deploying, productionalization, and monitoring but few, if any, do all of the above seamlessly. Bighead aims to tie together various open source and in-house projects to remove incidental complexity from ML workflows. Bighead is built on Python and Spark and can be used in modular pieces as each ML problem presents unique challenges. Through standardization of the path to production, training environments and the methods for collecting and transforming data on Spark, each model is reproducible and iterable. This talk covers the architecture, the problems that each individual component and the overall system aims to solve, and a vision for the future of machine learning infrastructure. It’s widely adapted in Airbnb and we have variety of models running in production. We have seen the overall model development time go down from many months to days on Bighead. We plan to open source Bighead to allow the wider community to benefit from our work.

Bighead: Airbnb’s End-to-End Machine Learning Platform with Krishna Puttaswa...

Databricks

Data Vault Modeling and Methodology introduction that I provided to a Montreal event in September 2011. It covers an introduction and overview of the Data Vault components for Business Intelligence and Data Warehousing. I am Dan Linstedt, the author and inventor of Data Vault Modeling and methodology. If you use the images anywhere in your presentations, please credit http://LearnDataVault.com as the source (me). Thank-you kindly, Daniel Linstedt

Data Vault Overview

Empowered Holdings, LLC

Introduction SQL Analytics on Lakehouse Architecture

Databricks

Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi

Timothy Spann

Apache NiFi in the Hadoop Ecosystem

DataWorks Summit/Hadoop Summit

Power BI Architecture

Arthur Graus

Session #1 - Melissa Coates: "What You Need to Know to Administer Power BI" YouTube channel: https://www.youtube.com/channel/UCOAWiig6JH1i8MqcniEVbTg LinkedIn: https://www.linkedin.com/groups/8933736/  Melissa Coates Owner of Coates Data Strategies. Microsoft Data Platform MVP. Data architect with a background in data warehousing and business intelligence. Her current professional focus is enterprise-level Power BI governance, deployment, security, and administration. As the owner of Coates Data Strategies, Melissa produces training and consults to help companies strengthen and sustain their data-driven initiatives. Melissa is a big supporter of the technical community, and has been a Microsoft Data Platform MVP since 2013. Topic: "What You Need to Know to Administer Power BI". The Power BI administrator is a very high privilege role. Some administration activities apply consistently for every organization, whereas others depend on how Power BI is being used for self-service and corporate business intelligence initiatives. Each organization's needs related to security, governance, auditing, and data management influence the scope of responsibilities for a Power BI administrator.

Power BI On AIR - Melissa Coates: "What You Need to Know to Administer Power BI"

Bohdan Maherus

How to build a successful Data Lake

DataWorks Summit/Hadoop Summit

Data Lakes have been built with a desire to democratize data - to allow more and more people, tools, and applications to make use of data. A key capability needed to achieve it is hiding the complexity of underlying data structures and physical data storage from users. The de-facto standard has been the Hive table format addresses some of these problems but falls short at data, user, and application scale. So what is the answer? Apache Iceberg. Apache Iceberg table format is now in use and contributed to by many leading tech companies like Netflix, Apple, Airbnb, LinkedIn, Dremio, Expedia, and AWS. Watch Alex Merced, Developer Advocate at Dremio, as he describes the open architecture and performance-oriented capabilities of Apache Iceberg. You will learn: • The issues that arise when using the Hive table format at scale, and why we need a new table format • How a straightforward, elegant change in table format structure has enormous positive effects • The underlying architecture of an Apache Iceberg table, how a query against an Iceberg table works, and how the table’s underlying structure changes as CRUD operations are done on it • The resulting benefits of this architectural design

Apache Iceberg: An Architectural Look Under the Covers

ScyllaDB

Talk @ ScaleUp 360° AI Infrastructures DACH, 2021: Data scientists spend 80% and more of their time searching for and preparing data. This talk explains Snowflake’s Platform capabilities like near-unlimited data storage and instant and near-infinite compute resources and how the platform can be used to seamlessly integrate and support the machine learning libraries and tools data scientists rely on.

Actionable Insights with AI - Snowflake for Data Science

Harald Erb

Lecture 6: Infrastructure & Tooling (Full Stack Deep Learning - Spring 2021)

Sergey Karayev

Intro to Delta Lake

Databricks

What's hot (20)

Building Robust ETL Pipelines with Apache Spark

Vue d'ensemble Dremio

Learn to Use Databricks for the Full ML Lifecycle

Siligong.Data - May 2021 - Transforming your analytics workflow with dbt

Modernize & Automate Analytics Data Pipelines

Optimizing Apache Spark SQL Joins

Introducing Databricks Delta

Apache Atlas: Governance for your Data

Bighead: Airbnb’s End-to-End Machine Learning Platform with Krishna Puttaswa...

Data Vault Overview

Introduction SQL Analytics on Lakehouse Architecture

Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi

Apache NiFi in the Hadoop Ecosystem

Power BI Architecture

Power BI On AIR - Melissa Coates: "What You Need to Know to Administer Power BI"

How to build a successful Data Lake

Apache Iceberg: An Architectural Look Under the Covers

Actionable Insights with AI - Snowflake for Data Science

Lecture 6: Infrastructure & Tooling (Full Stack Deep Learning - Spring 2021)

Intro to Delta Lake

Similar to Incrementally streaming rdbms data to your data lake automagically

Let's build a simple ingest to cloud datawarehouse with low code

Timothy Spann

Using the FLaNK Stack for edge ai (apache mxnet, apache flink, apache nifi, a...

Timothy Spann

Sparkler Presentation for Spark Summit East 2017

Karanjeet Singh

A web crawler is a bot program that fetches resources from the web for the sake of building applications like search engines, knowledge bases, etc. In this talk, Karanjeet Singh and Thamme Gowda will describe a new crawler called Sparkler (contraction of Spark-Crawler) that makes use of recent advancements in distributed computing and information retrieval domains by conglomerating various Apache projects like Spark, Kafka, Lucene/Solr, Tika, and Felix. Sparkler is extensible, highly scalable, and high-performance web crawler that is an evolution of Apache Nutch and runs on Apache Spark Cluster. https://github.com/USCDataScience/sparkler

Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...

Spark Summit

Sparkler at spark summit east 2017

Thamme Gowda

Cisco event 6 05 2014v3 wwt only

Arthur_Hansen

The Briefing Room with Dr. Robin Bloor and Teradata RainStor Live Webcast October 13, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=012bb2c290097165911872b1f241531d Hadoop data lakes are emerging as peers to corporate data warehouses. However, successful data management solutions require a fusion of all relevant data, new and old, which has proven challenging for many companies. With a data lake that’s been optimized for fast queries, solid governance and lifecycle management, users can take data management to a whole new level. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor as he discusses the relevance of data lakes in today’s information landscape. He’ll be briefed by Mark Cusack of Teradata, who will explain how his company’s archiving solution has developed into a storage point for raw data. He’ll show how the proven compression, scalability and governance of Teradata RainStor combined with Hadoop can enable an optimized data lake that serves as both reservoir for historical data and as a "system of record” for the enterprise. Visit InsideAnalysis.com for more information.

First in Class: Optimizing the Data Lake for Tighter Integration

Inside Analysis

Creating a Science-Driven Big Data Superhighway

Larry Smarr

Global Big Data Conference is offering 3 day extensive bootcamp on Big Data. This is a fast paced, vendor agnostic, technical overview of the Big Data landscape. No prior knowledge of databases or programming is assumed. Big Data Bootcamp is targeted towards both technical and non-technical people who want to understand the emerging world of Big Data, with a specific focus on Hadoop, NoSQL & Machine Learning. Attendees will experience real Hadoop clusters and the latest Hadoop distributions.

Spark training in austin tx

19scottmiller

The blessing and curse of today's database market? So many choices! While relational databases still dominate the day-to-day business, a host of alternatives has evolved around very specific use cases: graph, document, NoSQL, hybrid (HTAP), column store, the list goes on. And the database tools market is teeming with activity as well. Register for this special Research Webcast to hear Dr. Robin Bloor share his early findings about the evolving database market. He'll be joined by Steve Sarsfield of HPE Vertica, and Robert Reeves of Datical in a roundtable discussion with Bloor Group CEO Eric Kavanagh. Send any questions to info@insideanalysis.com, or tweet with #DBSurvival.

Horses for Courses: Database Roundtable

Eric Kavanagh

The more time the data science teams spend on model training, the less business value is added because no value is created until that model is deployed in production. Traditional HDD-based systems are not suitable for training, which is very IO intensive due to complex transformations that are involved during data preparation. Moreover, Training is not a one-time process. Trends and patterns in the data keep changing rapidly, hence models need to be retrained to address drift issues to continually improve performance in production. Data scientists often experiment with thousands of models, and speeding up the process has significant business implications. In this talk, we will cover how you can accelerate an AI/ML pipeline by speeding up data loads using the Aerospike database which leverages its hybrid memory architecture to achieve sub-millisecond read/writes. In a hybrid memory architecture the index is stored in-memory (not persisted), and data is stored on persistent storage (SSD) and read directly from the disk. Disk I/O is not required to access the index. For time-sensitive and high throughput use cases such as fraud detection, you need a transactional database at the edge that can handle high-velocity ingestion and support millions of IOPS. The events are then streamed downstream to your AL/ML platform for training or your inference server for predictions. We will share the reference architecture of a highly performant AI/ML training and inference pipelines consisting of Apache Pulsar, Apache Spark 3.0, Aerospike database, and its Spark and Pulsar connectors. This architecture can be extended to other use cases that demand low latency and high throughput while not blowing your budget.

Add Horsepower to AI/ML streaming Pipeline - Pulsar Summit NA 2021

StreamNative

In this deck, Pradeep Dubey from Intel discusses AI & The Virtuous Cycle of Compute. Traditionally, there has been a division of labor between computers and humans where all forms of number crunching and bit manipulation are left to computers, whereas intelligent decision-making is left to us humans. We are now at the cusp of a major transformation that can disrupt this balance. This disruption is triggered by an unprecedented convergence of massive compute with massive data, and some recent algorithmic advances. This confluence has the potential to spur a virtuous cycle of compute. Deep Learning was recently scaled to obtain 15PF performance on the Cori supercomputer at NERSC. Cori Phase II features over 9600 KNL processors. It can significantly impact how we do computing and what computing can do for us. In this talk, I will discuss some of the application-level opportunities and system-level challenges that lie at the heart of this intersection of traditional high performance computing with emerging data-intensive computing. Dr. Pradeep Dubey is an Intel Fellow and Director of Parallel Computing Lab (PCL), part of Intel Labs. His research focus is computer architectures to efficiently handle new compute- and data-intensive application paradigms for the future computing environment. Watch the Video: https://wp.me/p3RLHQ-hRv Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

AI & The Virtuous Cycle of Compute

inside-BigData.com

The Pacific Research Platform: Building a Distributed Big Data Machine Learni...

Larry Smarr

Java One 2017: Open Source Big Data in the Cloud: Hadoop, M/R, Hive, Spark an...

Frank Munz

The PRP and Its Applications

Larry Smarr

Data Con LA 2020 Description Coming from a grand belief of data democratization, I believe that in order for any team to be successful collaborators, it has to be data centric and data should be accessible to all. *To ensure that your non software or software engineering centric team has maximum efficiency, data should be visible, data lake should be accessible. *Form a database for analytics summaries, talk about the different technologies(SQL, NoSQL) cost of deployment, need, team driven structure. Build an API for this database for external/inter team crosstalk. *Build analytics and visual layer on top of it. Flask/Django/Node, etc.., to enable the team to have high visibility in their analysis, and to ensure a higher turnaround of data. *Talk about an easy way of enabling the team to run code, could be local/cloud, JupyterHub is a great way of doing so, talk about the tremendous value added in that and the potential it enables *Talk about the common tools user for version control/CICD/Coding technologies, etc.. *Finally summarize the value of the mixture of all these tools and technologies in order to ensure the maximum efficiency. Speaker Nawar Khabbaz, Rivian, Data Engineer

Enabling Data centric Teams

Data Con LA

The right architecture is key for any IT project. This is especially the case for big data projects, where there are no standard architectures which have proven their suitability over years. This session discusses the different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Streaming Analytics architecture as well as Lambda and Kappa architecture and presents the mapping of components from both Open Source as well as the Oracle stack onto these architectures. The right architecture is key for any IT project. This is valid in the case for big data projects as well, but on the other hand there are not yet many standard architectures which have proven their suitability over years. This session discusses different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Event Driven architecture as well as Lambda and Kappa architecture. Each architecture is presented in a vendor- and technology-independent way using a standard architecture blueprint. In a second step, these architecture blueprints are used to show how a given architecture can support certain use cases and which popular open source technologies can help to implement a solution based on a given architecture.

Big Data Architectures

Guido Schmutz

The Future of Data Science

sarith divakar

Fundamentals Big Data and AI Architecture

Guido Schmutz

Hw09 Protein Alignment

Cloudera, Inc.

Similar to Incrementally streaming rdbms data to your data lake automagically (20)

Let's build a simple ingest to cloud datawarehouse with low code

Using the FLaNK Stack for edge ai (apache mxnet, apache flink, apache nifi, a...

Sparkler Presentation for Spark Summit East 2017

Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...

Sparkler at spark summit east 2017

Cisco event 6 05 2014v3 wwt only

First in Class: Optimizing the Data Lake for Tighter Integration

Creating a Science-Driven Big Data Superhighway

Spark training in austin tx

Horses for Courses: Database Roundtable

Add Horsepower to AI/ML streaming Pipeline - Pulsar Summit NA 2021

AI & The Virtuous Cycle of Compute

The Pacific Research Platform: Building a Distributed Big Data Machine Learni...

Java One 2017: Open Source Big Data in the Cloud: Hadoop, M/R, Hive, Spark an...

The PRP and Its Applications

Enabling Data centric Teams

Big Data Architectures

The Future of Data Science

Fundamentals Big Data and AI Architecture

Hw09 Protein Alignment

More from Timothy Spann

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Timothy Spann

Gen AI on Enterprise Cloud Apache NiFi Milvus Apache Kafka Apache Flink Cloudera Machine Learning Cloudera DataFlow https://medium.com/@tspann/building-a-milvus-connector-for-nifi-34372cb3c7fa https://www.meetup.com/futureofdata-princeton/events/300737266/ https://lu.ma/q7pcfyjn?source=post_page-----34372cb3c7fa--------------------------------&tk=TTyakY If you're interested in working with Generative AI on the cloud, this virtual workshop is for you. Tim Spann from Cloudera and Yujian Tang from Zilliz will cover how you can implement your own GenAI workflows on the cloud at enterprise scale. 9:00 - 9:05: Intro 9:05 - 9:15: What is Milvus 9:15 - 9:25: Cloudera Development Platform 9:25 - 10:00: Demo Location https://www.youtube.com/watch?v=IfWIzKsoHnA https://github.com/tspannhw/SpeakerProfile https://www.linkedin.com/in/yujiantang/

Generative AI on Enterprise Cloud with NiFi and Milvus

Timothy Spann

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Timothy Spann

Real-Time AI Streaming - AI Max Princeton

Timothy Spann

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines https://www.youtube.com/watch?v=Yeua8NlzQ3Y https://www.conf42.com/Large_Language_Models_LLMs_2024_Tim_Spann_generative_ai_streaming Adding Generative AI to Real-Time Streaming Pipelines Abstract Let’s build streaming pipelines that convert streaming events into prompts, call LLMs, and process the results. Summary Tim Spann: My talk is adding generative AI to real time streaming pipelines. I'm going to discuss a couple of different open source technologies. We'll touch on Kafka, Nifi, Flink, Python, Iceberg. All the slides, all the code and GitHub are out there. Llm, if you didn't know, is rapidly evolving. There's a lot of different ways to interact with models. That enrichment, transformation, processing really needs tools. The amount of models and projects and software that are available is massive. Nifi supports hundreds of different inputs and can convert them on the fly. Great way to distribute your data quickly to whoever needs it without duplication, without tight coupling. Fun to find new things to integrate into. So what we can do is, well, I want to get a meetup chat going. I have a processor here that just listens for events as they come from slack. And then I'm going to clean it up, add a couple fields and push that out to slack. Every model is a little bit of different tweaking. Nifi acts as a whole website. And as you see here, it can be get, post, put, whatever you want. We send that response back to flink and it shows up here. Thank you for attending this talk. I'm going to be speaking at some other events very shortly. Transcript This transcript was autogenerated. To make changes, submit a PR. Hi, Tim Spann here. My talk is adding generative AI to real time streaming pipelines, and we're here for the large language model conference at Comp 42, which is always a nice one, great place to be. I'm going to discuss a couple of different open source technologies that work together to enable you to build real time pipelines using large language models. So we'll touch on Kafka, Nifi, Flink, Python, Iceberg, and I'll show you a little bit of each one in the demos. I've been working with data machine learning, streaming IoT, some other things for a number of years, and you could contact me at any of these places, whether Twitter or whatever it's called, some different blogs, or in person at my meetups and at different conferences around the world. I do a weekly newsletter, cover streaming ML, a lot of LLM, open source, Python, Java, all kinds of fun stuff, as I mentioned, do a bunch of different meetups. They are not just in the east coast of the US, they are available virtually live, and I also put them on YouTube, and if you need them somewhere else, let me know. We publish all the slides, all the code and GitHub. Everything you need is out there. Let's get into the talk. Llm, if you didn't know, is rapidly evolving. While you're typing down the things that you use, it

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

Timothy Spann

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

Timothy Spann

28March2024-Codeless-Generative-AI-Pipelines https://www.meetup.com/futureofdata-princeton/events/299440871/ https://www.meetup.com/real-time-analytics-meetup-ny/events/299290822/ ******Note***** The event is seat-limited, therefore please complete your registration here. Only people completing the form will be able to attend. ----------------------- We're excited to invite you to join us in-person, for a Real-Time Analytics exploration! Join us for an evening of insights, networking as we delve into the OSS technologies shaping the field! Agenda: 05:30-06:00: Pizza and friends 06:00- 06:40: Codeless GenAI Pipelines with Flink, Kafka, NiFi 06:40- 07:20 Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders 07:20-07:30 QNA Codeless GenAI Pipelines with Flink, Kafka, NiFi | Tim Spann, Cloudera Explore the power of real-time streaming with GenAI using Apache NiFi. Learn how NiFi simplifies data engineering workflows, allowing you to focus on creativity over technical complexities. I'll guide you through practical examples, showcasing NiFi's automation impact from ingestion to delivery. Whether you're a seasoned data engineer or new to GenAI, this talk offers valuable insights into optimizing workflows. Join us to unlock the potential of real-time streaming and witness how NiFi makes data engineering a breeze for GenAI applications! Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders | Viktor Gamov, StarTree Explore how industry leaders like LinkedIn, Uber Eats, and Stripe are mastering real-time data with Viktor as your guide. Discover how Apache Pinot transforms data into actionable insights instantly. Viktor will showcase Pinot's features, including the Star-Tree Index, and explain why it's a game-changer in data strategy. This session is for everyone, from data geeks to business gurus, eager to uncover the future of tech. Join us and be wowed by the power of real-time analytics with Apache Pinot! ------- Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more.

28March2024-Codeless-Generative-AI-Pipelines

Timothy Spann

https://princetonacm.acm.org/tcfpro/ 18th Annual IEEE IT Professional Conference (ITPC) Armstrong Hall at The College of New Jersey Friday, March 15th, 2024 | 10:00 AM to 5:00 PM IT Professional Conference at Trenton Computer Festival IEEE Information Technology Professional Conference on Friday, March 15th, 2024 TCFPro24 Building Real-Time Generative AI Pipelines Building Real-Time Generative AI Pipelines In this talk, Tim will delve into the exciting realm of building real-time generative AI pipelines with streaming capabilities. The discussion will revolve around the integration of cutting-edge technologies to create dynamic and responsive systems that harness the power of generative algorithms. From leveraging streaming data sources to implementing advanced machine learning models, the presentation will explore the key components necessary for constructing a robust real-time generative AI pipeline. Practical insights, use cases, and best practices will be shared, offering a comprehensive guide for developers and data scientists aspiring to design and implement dynamic AI systems in a streaming environment. Tim will show a live demo showing we can use Apache NiFi to provide a live chat between a person in Slack and several LLM models all orchestrated with Apache NiFi, Apache Kafka and Python. We will use RAG against Chroma and Pinecone vector data stores, Hugging Face and WatsonX.AI LLM, and add additional context with NiFi lookups of stocks, weather and other data streams in real-time. Timothy Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

TCFPro24 Building Real-Time Generative AI Pipelines

Timothy Spann

2024 Build Generative AI for Non-Profits

Timothy Spann

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipelines https://www.meetup.com/futureofdata-newyork/events/298660453/ Unlocking Financial Data with Real-Time Pipelines (Flink Analytics on Stocks with SQL ) By Timothy Spann Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. I will be utilizing NiFi 2.0 with Python and Vector Databases. Timothy Spann Principal Developer Advocate, Cloudera Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. https://twitter.com/PaaSDev https://www.linkedin.com/in/timothyspann/ https://medium.com/@tspann https://github.com/tspannhw/FLiPStackWeekly/

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

Timothy Spann

Conf42-Python-Building Apache NiFi 2.0 Python Processors https://www.conf42.com/Python_2024_Tim_Spann_apache_nifi_2_processors Building Apache NiFi 2.0 Python Processors Abstract Let’s enhance real-time streaming pipelines with smart Python code. Adding code for vector databases and LLM. Summary Tim Spann: I'm going to be talking today, be building Apache 9520 Python processors. One of the main purposes of supporting Python in the streaming tool Apache Nifi is to interface with new machine learning and AI and Gen AI. He says Python is a real game changer for Cloudera. You're just going to add some metadata around it. It's a great way to pass a file along without changing it too substantially. We really need you to have Python 310 and again JDK 21 on your machine. You got to be smart about how you use these models. There are a ton of python processors available. You can use them in multiple ways. We're still in the early world of Python processors, so now's the time to start putting yours out there. Love to see a lot of people write their own. When we are parsing documents here, again, this is the Python one I'm picking PDF. Lots of different things you could do. If you're interested on writing your own python code for Apache Nifi, definitely reach out and thank.

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Timothy Spann

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM Abstract In this talk, we’ll discuss how to use Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg to process and analyze stock data. We demonstrated the ingestion, processing, and analysis of stock data. Additionally, we illustrated how to use an LLM to generate predictions from the analyzed data. Karin Wolok Developer Relations, Dev Marketing, and Community Programming @ Project Elevate Karin Wolok's LinkedIn account Karin Wolok's twitter account Tim Spann Principal Developer Advocate @ Cloudera Tim Spann's LinkedIn account Tim Spann's twitter account https://www.conf42.com/Python_2024_Karin_Wolok_Tim_Spann_nifi__kafka_risingwave_iceberg_llm

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

Timothy Spann

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines https://www.aicamp.ai/event/eventdetails/W2024022214 apache nifi llm generative ai gen ai ml dl machine learning apache kafka apache flink postgresql python AI Meetup (NYC): GenAI, LLMs, ML and Data Feb 22, 05:30 PM EST Welcome to the monthly in-person AI meetup in New York City, in collaboration with Microsoft. Join us for deep dive tech talks on AI, GenAI, LLMs and machine learning, food/drink, networking with speakers and fellow developers Agenda: * 5:30pm~6:00pm: Checkin, Food/drink and networking * 6:00pm~6:10pm: Welcome/community update * 6:10pm~8:30pm: Tech talks * 8:30pm: Q&A, Open discussion Tech Talk: Searching and Reasoning Over Multimedia Data with Vector Databases and LMMs Speaker: Zain Hasan (Weaviate LinkedIn) Abstract: In this talk, Zain Hasan will discuss how we can use open-source multimodal embedding models in conjunction with large generative multimodal models that can that can see, hear, read, and feel data(!), to perform cross-modal search(searching audio with images, videos with text etc.) and multimodal retrieval augmented generation (MM-RAG) at the billion-object scale with the help of open source vector databases. I will also demonstrate, with live code demos, how being able to perform this cross-modal retrieval in real-time can enables users to use LLMs that can reason over their enterprise multimodal data. This talk will revolve around how we can scale the usage of multimodal embedding and generative models in production. Tech Talk: Codeless Generative AI Pipelines Speaker: Timothy Spann (Cloudera LinkedIn) Abstract: Join us for an insightful talk on leveraging the power of real-time streaming tools, specifically Apache NiFi, to revolutionize GenAI data engineering. In this session, we’ll explore how the integration of Apache NiFi can automate the entire process of prompt building, making it a seamless and efficient task. Speakers/Topics: Stay tuned as we are updating speakers and schedules. If you have a keen interest in speaking to our community, we invite you to submit topics for consideration: Submit Topics Sponsors: We are actively seeking sponsors to support our community. Whether it is by offering venue spaces, providing food/drink, or cash sponsorship. Sponsors will have the chance to speak at the meetups, receive prominent recognition, and gain exposure to our extensive membership base of 20,000+ local or 300K+ developers worldwide. Venue: Microsoft NYC - Times Square, 11 Times Square, New York, NY 10036 Room Name: Central Park West 6501 Community on Slack/Discord - Event chat: chat and connect with speakers and attendees - Sharing blogs, events, job openings, projects collaborations Join Slack (search and join the #newyork channel) | Join Discord

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

Timothy Spann

DBA Fundamentals Group: Continuous SQL with Kafka and Flink 20-Feb-2024 In this talk, I will walk through how someone can set up and run continuous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas, and publishing data. We will then cover consuming Kafka data, joining Kafka topics, and inserting new events into Kafka topics as they arrive. This basic overview will show hands-on techniques, tips, and examples of how to do this. Tim Spann Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

Timothy Spann

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

Timothy Spann

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines Unlocking Financial Data with Real-Time Pipelines Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. Key Points to be Covered: Introduction to Real-Time Data Pipelines: a. The limitations of traditional batch processing in the financial domain. b. Understanding the need for real-time data processing. Apache Flink: Powering Real-Time Stream Processing: a. Overview of Apache Flink and its role in real-time stream processing. b. Use cases for Apache Flink in the financial industry. c. How Flink enables fast, scalable, and fault-tolerant processing of streaming financial data. Apache Kafka: Building Resilient Event Streaming Platforms: a. Introduction to Apache Kafka and its role as a distributed streaming platform. b. Kafka's capabilities in handling high-throughput, fault-tolerant, and real-time data streaming. c. Integration of Kafka with financial data sources and consumers. Apache NiFi: Data Ingestion and Flow Management: a. Overview of Apache NiFi and its role in data ingestion and flow management. b. Data integration and transformation capabilities of NiFi for financial data. c. Utilizing NiFi to collect and process financial data from diverse sources. Iceberg: Efficient Data Lake Management: a. Understanding Iceberg and its role in managing large-scale data lakes. b. Iceberg's schema evolution and table-level metadata capabilities. c. How Iceberg simplifies data lake management in financial institutions. Real-World Use Cases: a. Real-time fraud detection using Flink, Kafka, and NiFi. b. Portfolio risk analysis with Iceberg and Flink. c. Streamlined regulatory reporting leveraging all four technologies. Best Practices and Considerations: a. Architectural considerations when building real-time financial data pipelines. b. Ensuring data integrity, security, and compliance in real-time pipelines. c. Scalability an

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Timothy Spann

Building Real-time Travel Alerts In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts. We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development. Let's get streaming. Apache Flink Apache Kafka Apache NiFi FLaNK Stack Tim Spann Big Data Conference Europe 2023

Building Real-Time Travel Alerts

Timothy Spann

JConWorld: Continuous SQL with Kafka and Flink In this talk, I will walk through how someone can setup and run continous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas and publishing data. We will then cover consuming Kafka data, joining Kafka topics and inserting new events into Kafka topics as they arrive. This basic over view will show hands-on techniques, tips and examples of how to do this. Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber

JConWorld_ Continuous SQL with Kafka and Flink

Timothy Spann

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines https://dssconf.pl/en/#agenda-section Integrating LLM with Streaming Data Pipelines Timothy Spann, Principal Developer Advocate, Cloudera APACHE NIFI, APACHE FLINK, APACHE KAFKA, LLM, HUGGINGFACE, REST, STREAMING LESS In this talk and demo I will walk through how to add LLMs to your streaming pipelines by integration through Apache NiFi. https://github.com/tspannhw/FLaNK-watsonx.ai Cloudera streaming llm generative ai slack to slack

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines

Timothy Spann

Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo

Timothy Spann

More from Timothy Spann (20)

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Generative AI on Enterprise Cloud with NiFi and Milvus

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Real-Time AI Streaming - AI Max Princeton

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

28March2024-Codeless-Generative-AI-Pipelines

TCFPro24 Building Real-Time Generative AI Pipelines

2024 Build Generative AI for Non-Profits

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Building Real-Time Travel Alerts

JConWorld_ Continuous SQL with Kafka and Flink

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines

Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo

Recently uploaded

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Architecting Cloud Native Applications

WSO2

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

The presentation was made in “Web3 Fusion: Embracing AI and Beyond” is more than a conference; it's a journey into the heart of digital transformation. The conference a provided a platform where the future of technology meets practical application. This three-day hybrid event, set in the heart of innovation, served as a gateway to the latest trends and transformative discussions in AI, Blockchain, IoT, AR/VR, and their collective impact on the information space.

AI in Action: Real World Use Cases by Anitaraj

AnitaRaj43

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Introduction to use of FHIR Documents in ABDM

Kumar Satyam

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard37

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Recently uploaded (20)

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

How to Troubleshoot Apps for the Modern Connected Worker

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

MINDCTI Revenue Release Quarter One 2024

Architecting Cloud Native Applications

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Why Teams call analytics are critical to your entire business

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

AI in Action: Real World Use Cases by Anitaraj

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Introduction to use of FHIR Documents in ABDM

Platformless Horizons for Digital Adaptability

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Vector Search -An Introduction in Oracle Database 23ai.pptx

JohnPollard-hybrid-app-RailsConf2024.pptx

Apidays New York 2024 - The value of a flexible API Management solution for O...

AWS Community Day CPH - Three problems of Terraform

Incrementally streaming rdbms data to your data lake automagically

1. APACHECON @HOME Spt, 29th – Oct. 1st 2020

2. APACHECON NA Spt, 28th – Oct. 2nd 2020

3. APACHECON NA Spt, 28th – Oct. 2nd 2020

4. • • • • Incrementally Streaming RDBMS Data

5. 5 Future of Data - Princeton @PaasDev https://www.meetup.com/futureofdata-princeton/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ...

6. 6 Speakers John Kuchmek Senior Solutions Engineer

7. 7 Speakers Tim Spann Principal DataFlow Field Engineer @PaasDev DZone Zone Leader and Big Data MVB Princeton NJ Future of Data Meetup https://github.com/tspannhw https://www.datainmotion.dev/

8. 8 JDBC Database to Apache Kudu / JDBC Database to HDFS and Hive

9. 9 Trillions of Messages to SQL Databases and Data Warehouses PROCESS DELIVER

10. 10 https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html

11. 11 Reference Architecture Files to RDBMS

12. 12 Messages to Databases

13. ‘QueryRecord’ Processor https://medium.com/@abdelkrim.hadjidj/democratizing-niﬁ-record-processors-with-automatic-schemas-inferenc e-4f2b2794c427

14. ADVANCED XML PROCESSING https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html https://pierrevillard.com/2018/06/28/niﬁ-1-7-xml-reader-writer-and-forkrecord-processor/

15. • Example • Flat ﬁles on an FTP server named by date • Downloads ﬁle • HTTP REST API endpoint • Invokes API and downloads data • Legacy/Remote DB • Performs SQL queries 1 5 DBCP Connection Pool to remote SQL Server ExecuteSQLRecord processor

16. • 1 6 https://www.datainmotion.dev/2019/03/implementing-streaming-use-case-from.html

17. INGEST RDBMS TABLES https://community.cloudera.com/t5/Community-Articles/Incrementally-Streaming-RDBMS-Data-to-Your-Hadoop-DataLake/ta-p/247927

18. https://dzone.com/articles/lets-build-a-simple-ingest-to-cloud-data-warehouse