Cassandra Lunch #23: Lucene Based Indexes on Cassandra

•

0 likes•379 views

During this week's Cassandra Lunch we covered packaged and DIY methods for Lucene-based indexes on Cassandra; as well as some pros and cons for using Lucene Based Indexes on Cassandra. Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-23-lucene-based-indexes-on-cassandra/ Cassandra Lunch Recording: https://youtu.be/Z0NXWmZAB8s Additional Resources: https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search/searchTOC.html https://github.com/instaclustr/cassandra-lucene-index https://github.com/strapdata/elassandra https://github.com/tjake/Solandra Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Follow Us and Reach Us At: Anant: https://www.anant.us/ Cassandra.Link: https://cassandra.link/ Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

Technology

Version 1.0
Lucene Based Indexes on Cassandra
An Anant Corporation Story.

Types Covered + Pros and Cons
● Packaged
● DIY
● Pros
● Cons

DSE Search / Solr
● https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search
/searchTOC.html

Cassandra Lucene Index
● instaclustr/cassandra-lucene-index: Lucene based secondary indexes for
Cassandra
a. https://github.com/instaclustr/cassandra-lucene-index
● Mutations to Cassandra -> Mutations on Disk (for that node)

Elessandra / ElasticSearch
● strapdata/elassandra: Elassandra = Elasticsearch + Apache Cassandra
a. https://github.com/strapdata/elassandra
● Cassandra + Elasticsearch
a. Mutations to Cassandra -> Mutations to Elasticsearch

Event -> CQRS -> Cassandra + Index
● Write
○ Event / Command goes into an Event Source repository (Kafka, SQL
Table, etc. )
○ Command Processor processes it into CQL / Elasticsearch or SOLR or
Amazon ... Algolia
● Request
○ Event / Command goes into an Event Source repository (Kafka, SQL
Table, etc. )
○ Command Processor goes to index / ﬁnds the data, goes to Cassandra ,
gets the data, returns.
○ Query the index --

Cassandra -> Batch -> Index
● Writes to Cassandra
● Every now and then - Index to ???
● Cassandra + Spark

Serverless Function -> Cassandra + Index

Apache Niﬁ (Lucene) -> Niﬁ Processor -> Cassandra + Index

Pros
● extremely rich search capabilities
● Geospatial
● synonym
● fuzzy logic search
● Typos
● Stemming
● packaged elastic/solr/lucene -> shorter latency
● separate index -> better separation of concerns and speed

Cons
● pure lucene -> reinventing the wheel of what Solr/ElasticSearch
● external elastic/solr/ ?? -> longer latency between ﬁnding the data / getting
the data
● packaged elastic/solr/lucene -> don't expect it to solve all your problems
● consistency issues (if DIY)
● lucene is memory heavy
● lucene is disk heavy

Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037

Resources
● tjake/Solandra: Solandra = Solr + Cassandra

To support 6 million on-demand rides per day, a lot has to happen in near-real time. Latency translates into missed rides and monetary losses. Grab relies data streaming in Apache Kafka, with Scylla to tie it all together. This presentation details how Grab uses Scylla as a high throughput, low-latency aggregation store to combine multiple Kafka streams in near real-time, highlighting impressive characteristics of Scylla and how it fared against other databases in Grab’s exhaustive evaluations.

Scylla Summit 2022: An Odyssey to ScyllaDB and Apache Kafka

ScyllaDB

Will LaForest is the Public Sector CTO for Confluent. In his current position, Will evangelizes the benefits of Apache Kafka, event-driven data in motion architecture, and open-source software is addressing mission challenges in the Government. He has spent 25 years wrangling data at massive scale. His technical career spans diverse areas from software engineering, NoSQL, data science, cloud computing, machine learning, and building statistical visualization software but began with code slinging at DARPA as a teenager. Will holds degrees in mathematics and physics from the University of Virginia. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...

HostedbyConfluent

Apache Hudi is a data lake platform, that provides streaming primitives (upserts/deletes/change streams) on top of data lake storage. Hudi powers very large data lakes at Uber, Robinhood and other companies, while being pre-installed on four major cloud platforms. Hudi supports exactly-once, near real-time data ingestion from Apache Kafka to cloud storage, which is typically used in-place of a S3/HDFS sink connector to gain transactions and mutability. While this approach is scalable and battle-tested, it can only ingest data in mini batches, leading to lower data freshness. In this talk, we introduce a Kafka Connect Sink Connector for Apache Hudi, which writes data straight into Hudi's log format, making the data immediately queryable, while Hudi's table services like indexing, compaction, clustering work behind the scenes, to further re-organize for better query performance.

Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry

confluent

Cloud native data platform

Li Gao

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Realtime Business Platform Architecture Review

Anant Corporation

In this talk we’ll present how at GetYourGuide we’ve built from scratch a completely new ETL pipeline using Debezium, Kafka, Spark and Airflow, which can automatically handle schema changes. Our starting point was an error prone legacy system that ran daily, and was vulnerable to breaking schema changes, which caused many sleepless on-call nights. As most companies, we also have traditional SQL databases that we need to connect to in order to extract relevant data. This is done usually through either full or partial copies of the data with tools such as sqoop. However another approach that has become quite popular lately is to use Debezium as the Change Data Capture layer which reads databases binlogs, and stream these changes directly to Kafka. As having data once a day is not enough anymore for our bussiness, and we wanted our pipelines to be resilent to upstream schema changes, we’ve decided to rebuild our ETL using Debezium. We’ll walk the audience through the steps we followed to architect and develop such solution using Databricks to reduce operation time. By building this new pipeline we are now able to refresh our data lake multiple times a day, giving our users fresh data, and protecting our nights of sleep.

Using Hazelcast in the Kappa architecture

Oliver Buckley-Salmon

Introduction to Data Engineer and Data Pipeline at Credit OK

Kriangkrai Chaonithi

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

HostedbyConfluent

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

Databricks

ASPgems - kappa architecture

Juantomás García Molina

Lambda architecture

Mario Alexandro Santini

Lambda architecture: from zero to One

Serg Masyutin

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Lambda architecture @ Indix

Rajesh Muppalla

Apache Spark StreamingBartosz Jankiewicz

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Databricks

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

HostedbyConfluent

Enforcing format, changing schema, introducing privacy filters have always been a challenge with the classical Kafka-API. In this talk we'll cover how to extend existing applications with webassembly, allowing developers to change the shape of data at runtime, per application without creating additional topics. By leveraging WebAssembly, we can extend the capabilities of the Kafka-API beyond what it was initially imagined. Come and learn about the future of the Kafka-API

Large Scale Lakehouse Implementation Using Structured Streaming

Databricks

Business leads, executives, analysts, and data scientists rely on up-to-date information to make business decision, adjust to the market, meet needs of their customers or run effective supply chain operations. Come hear how Asurion used Delta, Structured Streaming, AutoLoader and SQL Analytics to improve production data latency from day-minus-one to near real time Asurion’s technical team will share battle tested tips and tricks you only get with certain scale. Asurion data lake executes 4000+ streaming jobs and hosts over 4000 tables in production Data Lake on AWS.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Superset druid realtime

arupmalakar

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Databricks

AI is fundamentally transforming how we live and work. Zalando is a data driven company. We deliver an optimal customer experience that drives engagement. We continue to improve this experience by leveraging the latest technologies and machine learning techniques — such as building a cutting edge cloud based infrastructure to support our operations at scale. We provide our data scientists across Zalando with the means to implement artificial intelligence use cases, leveraging data from all parts of our company and the best machine learning techniques from across the industry. Apache Spark delivered through Databricks is at the core of this strategy. In this keynote, I’ll share our AI journey thus far, and share how we are exploring ways to unify data through A.I. with Spark and Databricks.

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Uber Geo spatial data platform at DataWorks Summit

Zhenxiao Luo

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

confluent

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Anant Corporation

In Cassandra Lunch #58, Rahul Singh will be leading a presentation covering a Cassandra topic we're sure you won't want to miss. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/l4mAt3MDVZA Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cesare Cugnasco

What's hot

Modern ETL Pipelines with Change Data Capture

Databricks

Using Hazelcast in the Kappa architecture

Oliver Buckley-Salmon

Introduction to Data Engineer and Data Pipeline at Credit OK

Kriangkrai Chaonithi

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

HostedbyConfluent

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

Databricks

ASPgems - kappa architecture

Juantomás García Molina

Lambda architecture

Mario Alexandro Santini

Lambda architecture: from zero to One

Serg Masyutin

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

Lambda architecture @ Indix

Rajesh Muppalla

Apache Spark StreamingBartosz Jankiewicz

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Databricks

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

HostedbyConfluent

Large Scale Lakehouse Implementation Using Structured Streaming

Databricks

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Superset druid realtime

arupmalakar

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Databricks

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Uber Geo spatial data platform at DataWorks Summit

Zhenxiao Luo

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

confluent

What's hot (20)

Modern ETL Pipelines with Change Data Capture

Using Hazelcast in the Kappa architecture

Introduction to Data Engineer and Data Pipeline at Credit OK

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

ASPgems - kappa architecture

Lambda architecture

Lambda architecture: from zero to One

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

Lambda architecture @ Indix

Apache Spark Streaming

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

Large Scale Lakehouse Implementation Using Structured Streaming

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Superset druid realtime

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

Uber Geo spatial data platform at DataWorks Summit

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

Similar to Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Anant Corporation

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cesare Cugnasco

Cassandra at Pollfish

Pollfish

Pollfish is a survey platform which provides access to millions of targeted users. Pollfish allows easy distribution and targeting of surveys through existing mobile apps. (https://www.pollfish.com/). At pollfish we use Cassandra for difference use cases, eg. for application data store to maximize write throughput when appropriate and for our analytics project to find insights in application generated data. As a medium to accomplish our success so far, we use the Datastax's DSE 4.6 environment which integrates Appache Cassadra, Spark and a hadoop compatible file system (CFS). We will discuss how we started, how the journey was and the impressions gained so far along with some tips learned the hard way. This is a result of joint work of an excellent team here at Pollfish.

Cassandra at Pollfish

Stavros Kontopoulos

Apache Cassandra Lunch #96: Apache Cassandra Change Data Capture (CDC) Strate...

Anant Corporation

In Cassandra Lunch #96, Rahul Singh, CEO of Anant, will discuss different ways to get change data into and out of Cassandra using a few different strategies which could work out for your platform. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/HjgqAhsCQ64 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Building Enterprise Search Engines using Open Source Technologies

Rahul Singh

Enterprise Search is a challenging problem for most organizations. Public search technologies such as Google can index content and use link popularity to rank content in addition to the basic keyword matches. Enterprise Search is different. Sometimes it requires specially designed indexes as well as several processing steps. At the U.S. Patent & Trademark Office, part of the Department of Commerce, a team of professionals is building the next generation of search tools using open source technologies. Like any large undertaking, it’s not a simple plug and play project. Main topics to be covered in this talk: + Architectures for Large Scale Enterprise Search + Leveraging Apache Cassandra & Spark + Customizing / Configuring Apache SolR and Indexing + Writing a custom Parser for SolR in Scala

Building Enterprise Search Engines using Open Source Technologies

Anant Corporation

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Red Hat Developers

We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.

Big Data Technologies

Anant Corporation

Avoiding Pitfalls for Cassandra.pdf

Cédrick Lunven

Cassandra Summit 2014: Interactive OLAP Queries using Apache Cassandra and Spark

DataStax Academy

Presenter: Evan Chan, Principal Software Engineer at Socrata Inc. How do you rapidly derive complex insights on top of really big data sets in Cassandra? This session draws upon Evan's experience building a distributed, interactive, columnar query engine on top of Cassandra and Spark. We will start by surveying the existing query landscape of Cassandra and discuss ways to integrate Cassandra and Spark. We will dive into the design and architecture of a fast, column-oriented query architecture for Spark, and why columnar stores are so advantageous for OLAP workloads. I will present a schema for Parquet-like storage of analytical datasets on Cassandra. Find out why Cassandra and Spark are the perfect match for enabling fast, scalable, complex querying and storage of big analytical data.

Kindling: Getting Started with Spark and Cassandra

DataStax Academy

Speaker(s): Erich Ess, CTO at SimpleRelevance An introduction to using Spark and integrating Spark with Cassandra to perform analytics and data processing. If you are using Cassandra, then you almost certainly have a large amount of data which you want upon which some form of processing needs to be done. Spark is a new distributed computing platform which makes writing big data analytics incredibly easy and it integrates with Cassandra with minimal effort. In this presentation, I will show how to get Spark setup with Cassandra and how to interact with Cassandra through Spark. Also covered will be architectural details of Spark such as how it handles failure recovery along with the main programming concepts needed to implement simple through complex processes on Spark. Then I will walk through a demonstration of using Spark to perform ETL on a dataset and save the data into Cassandra and then use Spark to do analytics on the data in Cassandra. Time permitting I will explain some things we are using Spark for at SimpleRelevance and cover Spark Streaming.

Kafka spark cassandra webinar feb 16 2016

Hiromitsu Komatsu

Kafka spark cassandra webinar feb 16 2016

Hiromitsu Komatsu

Apache Solr! Enterprise Search Solutions at your Fingertips!

Murshed Ahmmad Khan

Spark Introduction

DataStax Academy

TupleJump: Breakthrough OLAP performance on Cassandra and Spark

DataStax Academy

Apache Cassandra is rock-solid and widely deployed for OLTP and real-time applications, but it is typically not thought of as an OLAP database for analytical queries. This talk will show architectures and techniques for combining Apache Cassandra and Spark to yield a 10-1000x improvement in OLAP analytical performance. We will then introduce a new open-source project that combines the above performance improvements with the ease of use of Apache Cassandra, and compare it to implementations based on Hadoop and Parquet. First, the existing Cassandra Spark connector allows one to easily load data from Cassandra to Spark. We'll cover how to accelerate queries through different caching options in Spark, and the tradeoffs and limitations around performance, memory, and updating data in real time. We then dive into the use of columnar storage layout and efficient coding techniques that dramatically speed up I/O for OLAP use cases. Cassandra features like triggers and custom secondary indexes allow for easy data ingestion into columnar format. Next, we explore how to integrate this new storage with Spark SQL and its pluggable data storage API. Future developments will enable extreme analytical database performance, including smart caching of column projections, a columnar version of Spark's Catalyst execution planner, and how vectorization makes for fast cache- and GPU-friendly calculations - see Spark's Project Tungsten. FiloDB is a new open-source database using the above techniques to combine very fast Spark SQL analytical queries with the ease of use of Cassandra. We will briefly cover interesting use cases, such as: * Easy exactly-once ingestion from Kafka for streaming and IoT applications * Incremental computed columns and geospatial annotations. We'll discuss how FiloDB improves aggregations needed for choropleth maps over standard PostGIS solutions.

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

Evan Chan

11. From Hadoop to Spark 2/2

Fabio Fumarola

In these slides is given an overview of the different parts of Apache Spark. We analyze spark shell both in scala and python. Then we consider Spark SQL with an introduction to Data Frame API. Finally we describe Spark Streaming and we make some code examples. Topics:spark-shell, pyspark, HDFS, how to copy file to HDFS, spark transformations, spark actions, Spark SQL (Shark), spark streaming, streaming transformation stateless vs stateful, sliding windows, examples

Breakthrough OLAP performance with Cassandra and Spark

Evan Chan

Similar to Cassandra Lunch #23: Lucene Based Indexes on Cassandra (20)

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cassandra at Pollfish

Apache Cassandra Lunch #96: Apache Cassandra Change Data Capture (CDC) Strate...

Building Enterprise Search Engines using Open Source Technologies

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Big Data Technologies

Avoiding Pitfalls for Cassandra.pdf

Cassandra Summit 2014: Interactive OLAP Queries using Apache Cassandra and Spark

Kindling: Getting Started with Spark and Cassandra

Kafka spark cassandra webinar feb 16 2016

Apache Solr! Enterprise Search Solutions at your Fingertips!

Spark Introduction

TupleJump: Breakthrough OLAP performance on Cassandra and Spark

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

11. From Hadoop to Spark 2/2

Breakthrough OLAP performance with Cassandra and Spark

More from Anant Corporation

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Anant Corporation

Kono.IntelCraft.Weekly.AI.LLM.Landscape.2024.02.28.pdf

Anant Corporation

Data Engineer's Lunch 96: Intro to Real Time Analytics Using Apache Pinot

Anant Corporation

NoCode, Data & AI LLM Inside Bootcamp: Episode 6 - Design Patterns: Retrieval...

Anant Corporation

Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers. GPT Automation: What it is and How it Works How Time-Saving GPT Automation Can Improve Your Business Cost-Effective GPT Automation: How it Can Save Your Business Money Using GPT Automation for Customer Service: Benefits and Best Practices The Power of GPT Automation for Content Creation Data Analysis Made Easy with GPT Automation Top GPT-3 Automation Tools for Businesses The Ethical Considerations of GPT Automation Overcoming Bias in GPT Automation: Best Practices The Future of GPT Automation: Trends and Predictions Since we focus on "no code" here, we'll explore the tools that are already out there such as ChatGPT plugins for Chrome, OpenAI GPT API, low-code/no-code platforms like Make/Integromat and Zapier, existing apps like Jasper/Rytr, and ecosystem tools like Everyprompt. We'll also discuss the resources available for those interested in learning more about GPT, including other people’s prompts.

Automate your Job and Business with ChatGPT #3 - Fundamentals of LLM/GPT

Anant Corporation

Episode 3: The LLM / GPT / AI Prompt / Data Engineer Roadmap In this episode, we'll discuss the history, fundamentals, and the different flavors of LLMs available, beyond GPT/ChatGPT. This is a dry run of a session that will be on a LLM Bootcamp ( Fill out the survey on the link if you are interested in an in-person vs. virtual session) Intro / Fundamentals of LLM LLM Foundations History of LLMs Tuning, Training, or "In Context Learning" with LLMs What is "Prompt Engineering" Case for Augmenting LLMs Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers. GPT Automation: What it is and How it Works How Time-Saving GPT Automation Can Improve Your Business Cost-Effective GPT Automation: How it Can Save Your Business Money Using GPT Automation for Customer Service: Benefits and Best Practices The Power of GPT Automation for Content Creation Data Analysis Made Easy with GPT Automation Top GPT-3 Automation Tools for Businesses The Ethical Considerations of GPT Automation Overcoming Bias in GPT Automation: Best Practices The Future of GPT Automation: Trends and Predictions Since we focus on "no code" here, we'll explore the tools that are already out there such as ChatGPT plugins for Chrome, OpenAI GPT API, low-code/no-code platforms like Make/Integromat and Zapier, existing apps like Jasper/Rytr, and ecosystem tools like Everyprompt. We'll also discuss the resources available for those interested in learning more about GPT, including other people’s prompts.

YugabyteDB Developer Tools

Anant Corporation

In Apache Cassandra Lunch #131: YugabyteDB Developer Tools, we discussed third party developer tools that are compatible with YugabyteDB. We talked about using Yugabyte Developer Tools for data visualization and schema management. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Developer tools play a critical role in simplifying and streamlining database development and management. They allow developers and administrators to be more productive, reducing the time and effort required to create and maintain database schemas, write SQL queries, test database performance, and enable collaboration. Developer tools also make it possible to track changes over time, improving the ability to manage the entire development lifecycle.

Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap

Anant Corporation

In this episode we'll discuss the different flavors of prompt engineering in the LLM/GPT space. According to your skill level you should be able to pick up at any of the following: Leveling up with GPT 1: Use ChatGPT / GPT Powered Apps 2: Become a Prompt Engineer on ChatGPT/GPT 3: Use GPT API with NoCode Automation, App Builders 4: Create Workflows to Automate Tasks with NoCode 5: Use GPT API with Code, make your own APIs 6: Create Workflows to Automate Tasks with Code 7: Use GPT API with your Data / a Framework 8: Use GPT API with your Data / a Framework to Make your own APIs 9: Create Workflows to Automate Tasks with your Data /a Framework 10: Use Another LLM API other than GPT (Cohere, HuggingFace) 11: Use open source LLM models on your computer 12: Finetune / Build your own models Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers.

Machine Learning Orchestration with Airflow

Anant Corporation

In Data Engineer’s Lunch #89: Machine Learning Orchestration with Airflow, we discussed using Apache Airflow to manage and schedule machine learning tasks. By following the best practices of ML Ops, teams can streamline their ML workflows and build scalable, efficient, and accurate models that deliver real-world business value. Properly implemented ML Ops can help organizations stay ahead of the curve and achieve their goals in the fast-paced world of machine learning. Apache Airflow is an open-source tool for scheduling and automating workflows. Airflow allows you to define workflows in Python, with tasks defined as Python functions that can include Operators for all sorts of external tools. This makes it easy to automate repeated processes and define dependencies between tasks, creating directed-acyclic-graphs of tasks that can be scheduled using cron syntax or frequency tasks. Airflow also features a user-friendly UI for monitoring task progress and viewing logs, giving you greater control over your data pipeline.

Cassandra Lunch 130: Recap of Cassandra Forward Talks

Anant Corporation

If you didn't attend, you don't want to miss a much shorter synopsis of what was covered and get some thoughts from us as to why they are important. We'll talk about the main topics of the event. 1. ACID transactions on Cassandra by Aaron Ploetz, Datastax 2. Apache Flink with Apache Cassandra at Satyajit Thadeswar, Netflix 3. Durable Execution built on Apache Cassandra by Loren Sands-Ramshaw, Temporal 4. Switching from Mongo to Cassandra with Mongoose & new Stargate JSON API, Valeri Karpov 5. Cloud Native and Realtime AI/ML with Patrick Mcfadin and Davor Boncaci, Datastax

Data Engineer's Lunch 90: Migrating SQL Data with Arcion

Anant Corporation

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...

Anant Corporation

Cassandra Lunch 129: What’s New: Apache Cassandra 4.1+ Features & Future

Anant Corporation

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Anant Corporation

As the demand for real-time data processing continues to grow, so too do the challenges associated with building production-ready applications that can handle large volumes of data and handle it quickly. In this talk, we will explore common problems faced when building real-time applications at scale, with a focus on a specific use case: detecting and responding to cyclist crashes. Using telemetry data collected from a fitness app, we’ll demonstrate how we used a combination of Apache Kafka and Python-based microservices running on Kubernetes to build a pipeline for processing and analyzing this data in real-time. We'll also discuss how we used machine learning techniques to build a model for detecting collisions and how we implemented notifications to alert family members of a crash. Our ultimate goal is to help you navigate the challenges that come with building data-intensive, real-time applications that use ML models. By showcasing a real-world example, we aim to provide practical solutions and insights that you can apply to your own projects. Key takeaways: An understanding of the common challenges faced when building real-time applications at scale Strategies for using Apache Kafka and Python-based microservices to process and analyze data in real-time Tips for implementing machine learning models in a real-time application Best practices for responding to and handling critical events in a real-time application

Data Engineer's Lunch #85: Designing a Modern Data Stack

Anant Corporation

CL 121

Anant Corporation

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Anant Corporation

Apache Cassandra Lunch 120: Apache Cassandra Monitoring Made Easy with AxonOps

Anant Corporation

In this lunch, Johnny will show us how easy it is to start monitoring your Cassandra cluster in minutes. He will explain the various aspects and features of Cassandra that need to be monitored, how to do it, and most importantly why! Approaches for backups and Cassandra repairs will be discussed and explored in detail. Learn how AxonOps significantly reduces the complexity and overhead when looking after Cassandra and ensures your Cassandra cluster is reliable and resilient. Experienced developer, DevOps, architect, and AxonOps co-founder, Johnny Miller, has worked with a wide variety of companies – from small start-ups to large enterprises. He has been working with Cassandra for many years and has a deep understanding of the challenges facing modern companies looking to adopt Apache Cassandra.

Apache Cassandra Lunch 119: Desktop GUI Tools for Apache Cassandra

Anant Corporation

In Apache Cassandra Lunch #119, Rahul Singh will cover a refresher on GUI desktop/web tools for users that want to get their hands dirty with Cassandra but don't want to deal with CQLSH to do simple queries. Some of the tools are web-based and others are installed on your desktop. Since the beginning days of Cassandra, a lot has changed and there are many options for command-line-haters to use Cassandra.

Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache...

Anant Corporation

Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness

Anant Corporation

More from Anant Corporation (20)

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Kono.IntelCraft.Weekly.AI.LLM.Landscape.2024.02.28.pdf

Data Engineer's Lunch 96: Intro to Real Time Analytics Using Apache Pinot

NoCode, Data & AI LLM Inside Bootcamp: Episode 6 - Design Patterns: Retrieval...

Automate your Job and Business with ChatGPT #3 - Fundamentals of LLM/GPT

YugabyteDB Developer Tools

Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap

Machine Learning Orchestration with Airflow

Cassandra Lunch 130: Recap of Cassandra Forward Talks

Data Engineer's Lunch 90: Migrating SQL Data with Arcion

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...

Cassandra Lunch 129: What’s New: Apache Cassandra 4.1+ Features & Future

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Data Engineer's Lunch #85: Designing a Modern Data Stack

CL 121

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Apache Cassandra Lunch 120: Apache Cassandra Monitoring Made Easy with AxonOps

Apache Cassandra Lunch 119: Desktop GUI Tools for Apache Cassandra

Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache...

Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness

Recently uploaded

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

НАДІЯ ФЕДЮШКО БАЦ «Професійне зростання QA спеціаліста»

QADay

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Bits & Pixels using AI for Good.........

Alison B. Lowndes

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

КАТЕРИНА АБЗЯТОВА «Ефективне планування тестування ключові аспекти та практ...

QADay

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Knowledge engineering: from people to machines and back

Elena Simperl

Quantum Computing: Current Landscape and the Future Role of APIs

Vlad Stirbu

Recently uploaded (20)

The Art of the Pitch: WordPress Relationships and Sales

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Mission to Decommission: Importance of Decommissioning Products to Increase E...

GraphRAG is All You need? LLM & Knowledge Graph

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

PHP Frameworks: I want to break free (IPC Berlin 2024)

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

НАДІЯ ФЕДЮШКО БАЦ «Професійне зростання QA спеціаліста»

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

How world-class product teams are winning in the AI era by CEO and Founder, P...

Bits & Pixels using AI for Good.........

Connector Corner: Automate dynamic content and events by pushing a button

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Accelerate your Kubernetes clusters with Varnish Caching

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

КАТЕРИНА АБЗЯТОВА «Ефективне планування тестування ключові аспекти та практ...

Neuro-symbolic is not enough, we need neuro-*semantic*

Knowledge engineering: from people to machines and back

Quantum Computing: Current Landscape and the Future Role of APIs

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

1. Version 1.0 Lucene Based Indexes on Cassandra An Anant Corporation Story.

2. Types Covered + Pros and Cons ● Packaged ● DIY ● Pros ● Cons

3. Packaged

4. DSE Search / Solr ● https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search /searchTOC.html

5. Cassandra Lucene Index ● instaclustr/cassandra-lucene-index: Lucene based secondary indexes for Cassandra a. https://github.com/instaclustr/cassandra-lucene-index ● Mutations to Cassandra -> Mutations on Disk (for that node)

6. Elessandra / ElasticSearch ● strapdata/elassandra: Elassandra = Elasticsearch + Apache Cassandra a. https://github.com/strapdata/elassandra ● Cassandra + Elasticsearch a. Mutations to Cassandra -> Mutations to Elasticsearch

7. DIY

8. Event -> CQRS -> Cassandra + Index ● Write ○ Event / Command goes into an Event Source repository (Kafka, SQL Table, etc. ) ○ Command Processor processes it into CQL / Elasticsearch or SOLR or Amazon ... Algolia ● Request ○ Event / Command goes into an Event Source repository (Kafka, SQL Table, etc. ) ○ Command Processor goes to index / ﬁnds the data, goes to Cassandra , gets the data, returns. ○ Query the index --

9. Cassandra -> Batch -> Index ● Writes to Cassandra ● Every now and then - Index to ??? ● Cassandra + Spark

10. Cassandra Triggers -> Index

11. Serverless Function -> Cassandra + Index

12. Apache Niﬁ (Lucene) -> Niﬁ Processor -> Cassandra + Index

13. Pros ● extremely rich search capabilities ● Geospatial ● synonym ● fuzzy logic search ● Typos ● Stemming ● packaged elastic/solr/lucene -> shorter latency ● separate index -> better separation of concerns and speed

14. Cons ● pure lucene -> reinventing the wheel of what Solr/ElasticSearch ● external elastic/solr/ ?? -> longer latency between ﬁnding the data / getting the data ● packaged elastic/solr/lucene -> don't expect it to solve all your problems ● consistency issues (if DIY) ● lucene is memory heavy ● lucene is disk heavy

15. Strategy: Scalable Fast Data Architecture: Cassandra, Spark, Kafka Engineering: Node, Python, JVM,CLR Operations: Cloud, Container Rescue: Downtime!! I need help. www.anant.us | solutions@anant.us | (855) 262-6826 3 Washington Circle, NW | Suite 301 | Washington, DC 20037

16. Resources ● tjake/Solandra: Solandra = Solr + Cassandra

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Similar to Cassandra Lunch #23: Lucene Based Indexes on Cassandra (20)

More from Anant Corporation

More from Anant Corporation (20)

Recently uploaded

Recently uploaded (20)

Cassandra Lunch #23: Lucene Based Indexes on Cassandra