Alluxio in MOMO

•

2 likes•1,005 views

08/2018

HDFS Federation
Alluxio
HDFS-NS1HDFS-NS2 HDFS-NS3 HDFS-NS4
Kafka
Yarn (Data Operation System)
MapReduce Tez Spark Flink TensorFlow
CPU GPU Memory
Oozie Dr.elephantMorse Hive
HBase
Kylin
MPIXGB
Phoenix

Alluxio
•
•
•
•
•
• Ad-hoc
• , HDFS
• 1500+

ü
ü

Spark Driver
Alluxio
Alluxio
HDFS
ü

Yarn
Spark
Alluxio
Yarn
Spark
Yarn
Spark
Alluxio(HDD)
Yarn
Spark
-
ü

SQL1
Avg (sec)
SQL2
Avg (sec)
SQL3
Avg (sec)
SQL4
Avg (sec)
ü

Alluxio
•
•
•
•
•
• Ad-hoc
• , HDFS
• 1500+

Ad-hoc ETL
-
ü
ü

- - -
-
Node
YARN NodeManager
Alluxio Worker
Spark Executor
HDFS DataNode
CPU MEM Alluxio
Node
YARN NodeManager
Alluxio Worker
Spark Executor
HDFS DataNode
CPU MEM Alluxio
Node
YARN NodeManager
Alluxio Worker
Spark Executor
HDFS DataNode
CPU MEM Alluxio
Node
YARN NodeManager
Spark Driver
HDFS DataNode
CPU MEM Alluxio
Node
YARN NodeManager
HDFS DataNode
CPU MEM Alluxio
Node
YARN ResourceManager
CPU MEM Alluxio
Request CPU, MEM, Alluxio Resource
Run with Alluxio Resource
Tez ApplicationMaster
1.
2.
ü

-
ü
ü

-
C
Level 3: replicas = 3
Level 2: replicas = 2
Level 4: replicas = 4
Level 6: replicas = 6
Level 5: replicas = 5
Level 7: replicas = 7
-3
) : 13 2 -3
(3 1 1 3 1 -3
ü
ü

L3: block_E
L2: block_F
L4: block_D
L6: block_B
L5: block_C
L7: block_A
L3: block_E
L2: block_F
L4: block_D, block_C
L6: block_A
L5: block_B
L7:
Minor clean

-
L3: block_E
L2: block_F
L4: block_D, block_C
L6: block_A
L5: block_B
L7:
L3: block_D, block_C
L2: block_E, block_F
L4: block_B
L6:
L5: block_A
L7:
Major clean

-
client
OnlineReconfigManager
Configuration reload
OnlineReconfigObserver C
OnlineReconfigObserver A
OnlineReconfigObserver B
OnlineReconfigObserver C
2: Reconfig
1: Register
3: Notify
OnlineReconfigObserver Interface:
needUpdate()
onNotification()
OnlineReconfigManager class:
(de)registerOberver()
notifyAll()
ü
ü

-
Worker
toRemoved blocks
[a,b,c,d,e,f,…,z]
BlockRemover Thread Pool
BlockRemover A
[a,b,c,d,e,f,….,z]
BlockRemover B
[ ]
BlockRemover B
[ ]
ü
ü

-
Worker
toRemoved blocks
[a,b,c,d,e,f,…,z]
BlockRemover Thread Pool
BlockRemover A
[a,b,c,d,e,f,…]
BlockRemover B
[ l, m, n,…. ]
BlockRemover B
[ t, u, v, …. ]
ü

-
ü

Alluxio
•
•
•
•
•
• Ad-hoc
• , HDFS
• 1500+

ü Alluxio Feature ü ü

-
Block 1
Block 2
Block 3
Block 4
Block 5
Block 3
ü

- -
ü

-
ü

-
-
ü
ü
ü

-
ü

ü Master Architecture

ü Worker Architecture

&
!
MOMO:5488358

More Related Content

What's hot

はじめてのGlusterFS

Php perf

Project update - heat (up to pike-1)

Project update - heat (up to pike-1)

Project update - heat (up to pike-1)

Deep Visibility for Production Microservices

Deep Visibility for Production Microservices

Deep Visibility for Production Microservices

Heat project onboarding

Heat project onboarding

Heat project onboarding

OpenStack - heat on boarding

OpenStack - heat on boarding

OpenStack - heat on boarding

Automated Hadoop Cluster Construction on EC2

Automated Hadoop Cluster Construction on EC2

Automated Hadoop Cluster Construction on EC2

Docker at OpenDNS

Docker at OpenDNS

Docker at OpenDNS

In this talk you can expect to learn what OCI containers are, how to build them and why you may want them. The first part will be a brief introduction to OCI containers followed by the motivation behind our use-case at the OpenStack/Magnum project and the Container Service at CERN. How we leverage OCI containers and why we chose them to offer container infrastructure to our users, meaning running kubernetes, etcd, flanneld, OpenStack-specific daemons, CERN-specific tools, the docker daemon and cri-o. The second part will be a shallow dive on how to run and build OCI containers from scratch and most importantly how to populate the famous config.json file, the heart of the OCI configuration. This part will include examples on how to use docker, runc, rkt, atomic and buildah.

How to build and run oci containers

How to build and run oci containers

How to build and run oci containers

Spyros Trigazis

System Capa Planning_DBA oracle edu

System Capa Planning_DBA oracle edu

System Capa Planning_DBA oracle edu

MongoDB Replication Cluster

MongoDB Replication Cluster

MongoDB Replication Cluster

Anuchit Chalothorn

[Open Infrastructure & Cloud Native Days Korea 2019] 커뮤니티 버전의 OpenStack 과 Ceph를 활용하여 대고객서비스를 구축한 사례를 공유합니다. 유연성을 확보한 기업용 클라우드 서비스 구축 사례와 높은 수준의 보안을 요구하는 거래소 서비스를 구축, 운영한 사례를 소개합니다. 또한 이 프로젝트에 사용된 기술 스택 및 장애 해결사례와 최적화 방안을 소개합니다. 오픈스택은 역시 오픈소스컨설팅입니다. #openstack #ceph #openinfraday #cloudnative #opensourceconsulting

Ceph issue 해결 사례

Ceph issue 해결 사례

Ceph issue 해결 사례

Open Source Consulting

Dockertaipei 20150528-dockerswarm

Dockertaipei 20150528-dockerswarm

Dockertaipei 20150528-dockerswarm

Cloud foundry on kubernetes

Cloud foundry on kubernetes

Cloud foundry on kubernetes

[OpenInfra Days Korea 2018] (Track 4) - Backend.AI: 오픈소스 머신러닝 인프라 프레임워크

[OpenInfra Days Korea 2018] (Track 4) - Backend.AI: 오픈소스 머신러닝 인프라 프레임워크

[OpenInfra Days Korea 2018] (Track 4) - Backend.AI: 오픈소스 머신러닝 인프라 프레임워크

OpenStack Korea Community

Etcd terraform by Alex Somesan

Etcd terraform by Alex Somesan

Etcd terraform by Alex Somesan

Maarten van der Hoef

Anton Moldovan "Building an efficient replication system for thousands of ter...

Anton Moldovan "Building an efficient replication system for thousands of ter...

Anton Moldovan "Building an efficient replication system for thousands of ter...

PostgreSQL Replication with Bucardo

PostgreSQL Replication with Bucardo

PostgreSQL Replication with Bucardo

Robertus M. S. E. Kusumaning Negoro

RediSearch Mumbai Meetup 2020

RediSearch Mumbai Meetup 2020

RediSearch Mumbai Meetup 2020

⚡️ Vikram Sahu

Jk rubyslava 25

Jk rubyslava 25

Jk rubyslava 25

What's hot (20)

はじめてのGlusterFS

Php perf

Project update - heat (up to pike-1)

Project update - heat (up to pike-1)

Project update - heat (up to pike-1)

Deep Visibility for Production Microservices

Deep Visibility for Production Microservices

Deep Visibility for Production Microservices

Heat project onboarding

Heat project onboarding

Heat project onboarding

OpenStack - heat on boarding

OpenStack - heat on boarding

OpenStack - heat on boarding

Automated Hadoop Cluster Construction on EC2

Automated Hadoop Cluster Construction on EC2

Automated Hadoop Cluster Construction on EC2

Docker at OpenDNS

Docker at OpenDNS

Docker at OpenDNS

How to build and run oci containers

How to build and run oci containers

How to build and run oci containers

System Capa Planning_DBA oracle edu

System Capa Planning_DBA oracle edu

System Capa Planning_DBA oracle edu

MongoDB Replication Cluster

MongoDB Replication Cluster

MongoDB Replication Cluster

Ceph issue 해결 사례

Ceph issue 해결 사례

Ceph issue 해결 사례

Dockertaipei 20150528-dockerswarm

Dockertaipei 20150528-dockerswarm

Dockertaipei 20150528-dockerswarm

Cloud foundry on kubernetes

Cloud foundry on kubernetes

Cloud foundry on kubernetes

[OpenInfra Days Korea 2018] (Track 4) - Backend.AI: 오픈소스 머신러닝 인프라 프레임워크

[OpenInfra Days Korea 2018] (Track 4) - Backend.AI: 오픈소스 머신러닝 인프라 프레임워크

[OpenInfra Days Korea 2018] (Track 4) - Backend.AI: 오픈소스 머신러닝 인프라 프레임워크

Etcd terraform by Alex Somesan

Etcd terraform by Alex Somesan

Etcd terraform by Alex Somesan

Anton Moldovan "Building an efficient replication system for thousands of ter...

Anton Moldovan "Building an efficient replication system for thousands of ter...

Anton Moldovan "Building an efficient replication system for thousands of ter...

PostgreSQL Replication with Bucardo

PostgreSQL Replication with Bucardo

PostgreSQL Replication with Bucardo

RediSearch Mumbai Meetup 2020

RediSearch Mumbai Meetup 2020

RediSearch Mumbai Meetup 2020

Jk rubyslava 25

Jk rubyslava 25

Jk rubyslava 25

Similar to Alluxio in MOMO

Hadoop 1.0 is a significant milestone in being the most stable and robust Hadoop release tested in production against a variety of applications. It offers improved performance, support for HBase, disk-fail-in-place, Webhdfs, etc over previous releases. The next major release, Hadoop 2.0 offers several significant HDFS improvements including new append-pipeline, federation, wire compatibility, NameNode HA, further performance improvements, etc. We describe how to take advantages of the new features and their benefits. We also discuss some of the misconceptions and myths about HDFS.

Strata + Hadoop World 2012: HDFS: Now and Future

Strata + Hadoop World 2012: HDFS: Now and Future

Strata + Hadoop World 2012: HDFS: Now and Future

Odsc workshop - Distributed Tensorflow on Hops

Odsc workshop - Distributed Tensorflow on Hops

Odsc workshop - Distributed Tensorflow on Hops

Hadoop Architecture_Cluster_Cap_Plan

Hadoop Architecture_Cluster_Cap_Plan

Hadoop Architecture_Cluster_Cap_Plan

Hadoop Operations - Best practices from the field

Hadoop Operations - Best practices from the field

Hadoop Operations - Best practices from the field

Cloud, Opensource, OPNFV and CI/CD for VNFs

Cloud, Opensource, OPNFV and CI/CD for VNFs

Cloud, Opensource, OPNFV and CI/CD for VNFs

You know, for search. Querying 24 Billion Documents in 900ms

You know, for search. Querying 24 Billion Documents in 900ms

You know, for search. Querying 24 Billion Documents in 900ms

Apache Solr on Hadoop is enabling organizations to collect, process and search larger, more varied data. Apache Spark is is making a large impact across the industry, changing the way we think about batch processing and replacing MapReduce in many cases. But how can production users easily migrate ingestion of HDFS data into Solr from MapReduce to Spark? How can they update and delete existing documents in Solr at scale? And how can they easily build flexible data ingestion pipelines? Cloudera Search Software Engineer Wolfgang Hoschek will present an architecture and solution to this problem. How was Apache Solr, Spark, Crunch, and Morphlines integrated to allow for scalable and flexible ingestion of HDFS data into Solr? What are the solved problems and what's still to come? Join us for an exciting discussion on this new technology.

Ingesting hdfs intosolrusingsparktrimmed

Ingesting hdfs intosolrusingsparktrimmed

Ingesting hdfs intosolrusingsparktrimmed

Setting up a big data platform at kelkoo

Setting up a big data platform at kelkoo

Setting up a big data platform at kelkoo

Fabrice dos Santos

Facing enterprise specific challenges – utility programming in hadoop

Facing enterprise specific challenges – utility programming in hadoop

Facing enterprise specific challenges – utility programming in hadoop

ZFS for Databases

ZFS for Databases

ZFS for Databases

Intro to hadoop

Intro to hadoop

Intro to hadoop

This session will use Novell Open Enterprise Server 2 SP2 to demonstrate how to cluster critical services—from NSS and Novell iPrint to Novell GroupWise, AFP and beyond. We'll cover the new features of Novell Cluster Services in the latest release of Novell Open Enterprise Server, and we'll show you how you can ensure consistency by using AutoYaST to build your nodes. This will be a practical session, so be prepared for a few thrills and spills along the way! Speakers: Tim Heywood CTO NDS 8 Mark Robinson CTO Linux NDS8

Practical Tips for Novell Cluster Services

Practical Tips for Novell Cluster Services

Practical Tips for Novell Cluster Services

Introduction and HDInsight best practices

Introduction and HDInsight best practices

Introduction and HDInsight best practices

Ashish Thapliyal

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insur...

Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insur...

Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insur...

02 Hadoop deployment and configuration

02 Hadoop deployment and configuration

02 Hadoop deployment and configuration

Subhas Kumar Ghosh

Hug syncsort etl hadoop big data

Hug syncsort etl hadoop big data

Hug syncsort etl hadoop big data

Stéphane Heckel

Since mid-2016, Spark-as-a-Service has been available to researchers in Sweden from the Rise SICS ICE Data Center at www.hops.site. In this session, Dowling will discuss the challenges in building multi-tenant Spark structured streaming applications on YARN that are metered and easy-to-debug. The platform, called Hopsworks, is in an entirely UI-driven environment built with only open-source software. Learn how they use the ELK stack (Elasticsearch, Logstash and Kibana) for logging and debugging running Spark streaming applications; how they use Grafana and InfluxDB for monitoring Spark streaming applications; and, finally, how Apache Zeppelin can provide interactive visualizations and charts to end-users. This session will also show how Spark applications are run within a ‘project’ on a YARN cluster with the novel property that Spark applications are metered and charged to projects. Projects are securely isolated from each other and include support for project-specific Kafka topics. That is, Kafka topics are protected from access by users that are not members of the project. In addition, hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.afka topics are protected from access by users that are not members of the project. We will also discuss the experiences of our users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and our novel solutions for helping researchers debug and optimize Spark applications.

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

RAPIDS: GPU-Accelerated ETL and Feature Engineering

RAPIDS: GPU-Accelerated ETL and Feature Engineering

RAPIDS: GPU-Accelerated ETL and Feature Engineering

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

OpenStack Korea Community

Similar to Alluxio in MOMO (20)

Strata + Hadoop World 2012: HDFS: Now and Future

Strata + Hadoop World 2012: HDFS: Now and Future

Strata + Hadoop World 2012: HDFS: Now and Future

Odsc workshop - Distributed Tensorflow on Hops

Odsc workshop - Distributed Tensorflow on Hops

Odsc workshop - Distributed Tensorflow on Hops

Hadoop Architecture_Cluster_Cap_Plan

Hadoop Architecture_Cluster_Cap_Plan

Hadoop Architecture_Cluster_Cap_Plan

Hadoop Operations - Best practices from the field

Hadoop Operations - Best practices from the field

Hadoop Operations - Best practices from the field

Cloud, Opensource, OPNFV and CI/CD for VNFs

Cloud, Opensource, OPNFV and CI/CD for VNFs

Cloud, Opensource, OPNFV and CI/CD for VNFs

You know, for search. Querying 24 Billion Documents in 900ms

You know, for search. Querying 24 Billion Documents in 900ms

You know, for search. Querying 24 Billion Documents in 900ms

Ingesting hdfs intosolrusingsparktrimmed

Ingesting hdfs intosolrusingsparktrimmed

Ingesting hdfs intosolrusingsparktrimmed

Setting up a big data platform at kelkoo

Setting up a big data platform at kelkoo

Setting up a big data platform at kelkoo

Facing enterprise specific challenges – utility programming in hadoop

Facing enterprise specific challenges – utility programming in hadoop

Facing enterprise specific challenges – utility programming in hadoop

ZFS for Databases

ZFS for Databases

ZFS for Databases

Intro to hadoop

Intro to hadoop

Intro to hadoop

Practical Tips for Novell Cluster Services

Practical Tips for Novell Cluster Services

Practical Tips for Novell Cluster Services

Introduction and HDInsight best practices

Introduction and HDInsight best practices

Introduction and HDInsight best practices

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insur...

Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insur...

Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insur...

02 Hadoop deployment and configuration

02 Hadoop deployment and configuration

02 Hadoop deployment and configuration

Hug syncsort etl hadoop big data

Hug syncsort etl hadoop big data

Hug syncsort etl hadoop big data

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

RAPIDS: GPU-Accelerated ETL and Feature Engineering

RAPIDS: GPU-Accelerated ETL and Feature Engineering

RAPIDS: GPU-Accelerated ETL and Feature Engineering

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

More from Alluxio, Inc.

As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Optimizing Data Access for Analytics And AI with Alluxio

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Speed Up Presto at Uber with Alluxio Caching

Speed Up Presto at Uber with Alluxio Caching

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Correctly Loading Incremental Data at Scale

Correctly Loading Incremental Data at Scale

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Data Infra Meetup | Uber's Data Storage Evolution

Data Infra Meetup | Uber's Data Storage Evolution

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jordan Plawner (Global Director of Artificial intelligence Product Management and Strategy, @Intel) ChatGPT and other massive models represents an amazing step forward in AI, yet they do not solve real-world business problems. We will survey how the AI ecosystem has worked non-stop over this last year to take these all-purpose multi-task models and optimize them to they can be used by organizations to address domain specific problems. We will explain these new AI-for-the-real world techniques and methods such as fine tuning and how can be applied to deliver results which are highly performant with state-of-the-art accuracy while also being economical to build and deploy everywhere to enhance products and services.

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sally (Mihyoung) Lee (Senior Staff Engineer, TLM, @Uber) Machine learning models power Uber’s everyday business. However, developing and deploying a model is not a one-time event but a continuous process that requires careful planning, execution, and monitoring. This session will highlight Uber’s practice on the machine learning lifecycle to ensure high model quality.

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio Monthly Webinar Oct. 3, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Greg Palmer (Lead Solutions Engineer) Model training requires extensive computational and GPU resources. When training models on AWS, loading data from S3 often becomes a major bottleneck, wasting valuable GPU cycles. Optimizing data loading can greatly reduce GPU idle time and increase GPU utilization. In this webinar, Greg Palmer will discuss best practices for efficient data loading during model training on AWS. He will demonstrate how to use Alluxio on EKS as a distributed cache to accelerate PyTorch training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs. What you will learn: - The challenges of feeding data-hungry GPUs in the cloud - How to accelerate model training by optimizing data loading on AWS - The reference architecture for running PyTorch jobs with Alluxio cache on EKS while reading data from S3, with benchmark results of training ResNet50 and BERT - How to use TensorBoard to identify bottlenecks in GPU utilization

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

More from Alluxio, Inc. (20)

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Optimizing Data Access for Analytics And AI with Alluxio

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Speed Up Presto at Uber with Alluxio Caching

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Correctly Loading Incremental Data at Scale

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Data Infra Meetup | Uber's Data Storage Evolution

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Recently uploaded

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Recently uploaded (20)

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Alluxio in MOMO

1.

2.

3.

5. HDFS Federation Alluxio HDFS-NS1HDFS-NS2 HDFS-NS3 HDFS-NS4 Kafka Yarn (Data Operation System) MapReduce Tez Spark Flink TensorFlow CPU GPU Memory Oozie Dr.elephantMorse Hive HBase Kylin MPIXGB Phoenix

6. Alluxio • • • • • • Ad-hoc • , HDFS • 1500+

8. Spark Driver Alluxio Alluxio HDFS ü

9. Yarn Spark Alluxio Yarn Spark Yarn Spark Alluxio(HDD) Yarn Spark - ü

10. SQL1 Avg (sec) SQL2 Avg (sec) SQL3 Avg (sec) SQL4 Avg (sec) ü

12. Alluxio • • • • • • Ad-hoc • , HDFS • 1500+

13. Ad-hoc ETL - ü ü

14. - - - - Node YARN NodeManager Alluxio Worker Spark Executor HDFS DataNode CPU MEM Alluxio Node YARN NodeManager Alluxio Worker Spark Executor HDFS DataNode CPU MEM Alluxio Node YARN NodeManager Alluxio Worker Spark Executor HDFS DataNode CPU MEM Alluxio Node YARN NodeManager Spark Driver HDFS DataNode CPU MEM Alluxio Node YARN NodeManager HDFS DataNode CPU MEM Alluxio Node YARN ResourceManager CPU MEM Alluxio Request CPU, MEM, Alluxio Resource Run with Alluxio Resource Tez ApplicationMaster 1. 2. ü

16. - C Level 3: replicas = 3 Level 2: replicas = 2 Level 4: replicas = 4 Level 6: replicas = 6 Level 5: replicas = 5 Level 7: replicas = 7 -3 ) : 13 2 -3 (3 1 1 3 1 -3 ü ü

17. L3: block_E L2: block_F L4: block_D L6: block_B L5: block_C L7: block_A L3: block_E L2: block_F L4: block_D, block_C L6: block_A L5: block_B L7: Minor clean

18. - L3: block_E L2: block_F L4: block_D, block_C L6: block_A L5: block_B L7: L3: block_D, block_C L2: block_E, block_F L4: block_B L6: L5: block_A L7: Major clean

19. -

20. - client OnlineReconfigManager Configuration reload OnlineReconfigObserver C OnlineReconfigObserver A OnlineReconfigObserver B OnlineReconfigObserver C 2: Reconfig 1: Register 3: Notify OnlineReconfigObserver Interface: needUpdate() onNotification() OnlineReconfigManager class: (de)registerOberver() notifyAll() ü ü

21. - Worker toRemoved blocks [a,b,c,d,e,f,…,z] BlockRemover Thread Pool BlockRemover A [a,b,c,d,e,f,….,z] BlockRemover B [ ] BlockRemover B [ ] ü ü

22. - Worker toRemoved blocks [a,b,c,d,e,f,…,z] BlockRemover Thread Pool BlockRemover A [a,b,c,d,e,f,…] BlockRemover B [ l, m, n,…. ] BlockRemover B [ t, u, v, …. ] ü

24. Alluxio • • • • • • Ad-hoc • , HDFS • 1500+

25. ü Alluxio Feature ü ü

26. - Block 1 Block 2 Block 3 Block 4 Block 5 Block 3 ü

29. - - ü ü ü

31. ü Master Architecture

32. ü Worker Architecture

33. & ! MOMO:5488358