Free Code Friday: Genome Resequencing with Spark, Part 1

•

4 likes•673 views

Spark is a powerful new tool for processing large volumes of data quickly across a cluster of networked computers. Typical bioinformatics workflow requirements are well-matched to Spark’s capabilities. However, Spark is not commonly used because many legacy bioinformatics applications make assumptions about their computing environment. These assumptions present a barrier to integrating the tools into more modern computing environments. These barriers are quickly coming down. ADAM is a software library and set of tools built on top of Spark that make it easy work with file formats commonly used for genome analysis like FastQ, BAM, and VCF. In this presentation, we’ll explore how a step that is common to many bioinformatics workflows, sequence alignment, can done with Bowtie and ADAM inside a Spark environment to quickly align short reads to a reference genome. A complete code example is demonstrated and provided at https://github.com/allenday/spark-genome-alignment-demo

Technology

© 2015 MapR Technologies 1© 2015 MapR Technologies

© 2015 MapR Technologies 2
Let’s kick of a build while we talk…
• Follow the instructions at
• https://github.com/allenday/spark-genome-alignment-demo
• While that’s cooking…
• What’s this all about?

© 2015 MapR Technologies 3
Alignment
Reference
Sequences
Aligned
Reads Downstream
Applications…
DNA Reads

© 2015 MapR Technologies 4
Alignment
Reference
Sequences
DNA Reads
Aligned
Reads Downstream
Applications…
Align()

© 2015 MapR Technologies 5
Possible Align() Outcomes
Unaligned
DNA Reads
Reference
Sequences
Single
Location
Reads
Multiple
Location
Reads
Unlocatable
Reads
Align()

© 2015 MapR Technologies 6
Many-to-Many Relationship Between Reads and Locations
• Read1
• Read2
• Read3
• Read4
• NULL
• LocationA
• LocationB
• LocationC
• LocationD
• LocationA
• NULL
• LocationE

© 2015 MapR Technologies 7
Parallelizing Alignment
Unaligned
DNA Reads
Locations
Locations
Locations
Part1Part2Part3
Aligned
DNA Reads
Align() Concat() Sort() Etc…Split()

© 2015 MapR Technologies 8
Using HPC+SAN has Bottlenecks (SGE, PBS, Condor, Etc)
Part1Part2Part3
Volume Read
Bottleneck
Volume Write
Bottleneck
Read & Write
Bottleneck

© 2015 MapR Technologies 9
Using Spark Eliminates Bottlenecks
Align() Concat() Sort()Split()

© 2015 MapR Technologies 10
How to Do it? FastQ => BAM
Prepare Environment
• Get Input Reference Sequences in FastA format (one time step)
• Index Reference Sequences with Bowtie (one time step)
Get Input Data
• Get Input Unaligned Reads in FastQ format (per input data set)
Do the Work
• Convert Reads to ADAM Format with Spark (per input data set)
• Use Spark Pipe to Align the Reads (per input data fragment)
• Write the Alignments to SAM or BAM format (per input data,
optional)

© 2015 MapR Technologies 11
Let’s look at some code…
• Let’s check on the build we kicked off earlier.
• And the driver script:
• cat $DEMO/bin/bowtie_pipe_single.scala
• I said before that it’s possible to be 100% compatible with legacy
tools by writing out to SAM or BAM.
• You might not want to do this…

© 2015 MapR Technologies 12
Benefits of Keeping Data in Spark / ADAM
• ADAM file sizes are about 20% smaller than BAM.
• ADAM files are faster for common operations (filters, range
queries)
• ADAM files sort more quickly than `samtools sort …`. This can
be a huge time savings

© 2015 MapR Technologies 14
• Some other commands built-in, take a look:
• $ADAM_HOME/bin/adam-submit
• $ADAM_HOME/bin/adam-submit count_kmers
$DEMO/build/data/reads.sam /tmp/count_kmers 3
• head /tmp/count_kmers/part-00000
• All of this is easier without leaving the Spark shell
• And it’s faster, too…

© 2015 MapR Technologies 15
And it’s faster, too… Using Spark Eliminates Bottlenecks
Align() Concat() Sort()Split()

© 2015 MapR Technologies 16
Thanks! Questions?
@allenday, @mapr
aday@mapr.com
linkedin.com/in/allenday

What's hot

May 2021 Spark Testing ... or how to farm reputation on StackOverflow

Adam Doyle

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

PivotalOpenSourceHub

ODI is an incredible and flexible development tool that goes beyond simple data integration. But most of its development power comes from outside-the-box ideas. * Did you ever want to dynamically run any number of “OS” commands using a single ODI component? * Did you ever want to have only one data store and loop different sources without the need of different ODI contexts? * Did you ever want to have only one interface and loop any number of ODI objects with a lot of control? * Did you ever need to have a “third command tab” in your procedures or KMs to improve ODI powers? * Do you still use an old version of ODI and miss a way to know the values of the variables in a scenario execution? * Did you know ODI has four “substitution tags”? And do you know how useful they are? * Do you use “dynamic variables” and know how powerful they can be? * Do you know how to have control over you ODI priority jobs automatically (stop, start, and restart scenarios)?

Incredible ODI tips to work with Hyperion tools that you ever wanted to know

Rodrigo Radtke de Souza

Breaking data

Terry Bunio

Data science lifecycle with Apache Zeppelin

DataWorks Summit/Hadoop Summit

Spark Uber Development Kit

Jen Aman

Dagster @ R&S MNT

Simon Späti

Spark Summit EU talk by Yiannis Gkoufas

Spark Summit

Wrapper formulas saltconf 2016

Charles Rodetsky

During the last year, the team at IBM Research at Ireland has been using Apache Spark to perform analytics on large volumes of sensor data. These applications need to be executed on a daily basis, therefore, it was essential for them to understand Spark resource utilization. They found it cumbersome to manually consume and efficiently inspect the CSV files for the metrics generated at the Spark worker nodes. Although using an external monitoring system like Ganglia would automate this process, they were still plagued with the inability to derive temporal associations between system-level metrics (e.g. CPU utilization) and job-level metrics (e.g. job or stage ID) as reported by Spark. For instance, they were not able to trace back the root cause of a peak in HDFS Reads or CPU usage to the code in their Spark application causing the bottleneck. To overcome these limitations, they developed SparkOScope. Taking advantage of the job-level information available through the existing Spark Web UI and to minimize source-code pollution, they use the existing Spark Web UI to monitor and visualize job-level metrics of a Spark application (e.g. completion time). More importantly, they extend the Web UI with a palette of system-level metrics of the server/VM/container that each of the Spark job’s executor ran on. Using SparkOScope, you can navigate to any completed application and identify application-logic bottlenecks by inspecting the various plots providing in-depth timeseries for all relevant system-level metrics related to the Spark executors, while also easily associating them with stages, jobs and even source code lines incurring the bottleneck. They have made Sparkoscope available as a standalone module, and also extended the available Sinks (mongodb, mysql).

SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...

Databricks

Components of openEHR based EHRs

Aastha Madaan

Agenda: • Brief overview of Spark provided spark-shell, spark-submit • Overview of Spark ContextOverview of Zeppelin and Jupyter notebooks for Spark • Introduction to IBM Spark Kernel • Introduction to Cloudera Livy and Spark JobServer Github Link: Previous meetups:- 1) Introduction to Resilient Distributed Dataset and deep dive Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-and-resilient-distributed-dataset-basics-and-deep-dive Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/225159947/ Video: https://www.youtube.com/watch?v=MkeRWyF1y_0 Github: https://github.com/SatyaNarayan1/spark_meetup 2) Introduction to Spark DataFrames/SQL and Deep dive Slides: http://www.slideshare.net/sachinparmarss/deep-dive-spark-data-frames-sql-and-catalyst-optimizer Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/226419828/ Video: https://www.youtube.com/watch?v=h71MNWRv99M Github: https://github.com/parmarsachin/spark-dataframe-demo 3) Apache Spark - Introduction to Spark Streaming and Deep dive Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-to-spark-streaming-and-deep-dive-57671774 Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/227008581/ Video: Github: https://github.com/agsachin/spark-meetup Looking forward to have a great interactive session. Do provide feedback.

Interactive Analytics using Apache Spark

Sachin Aggarwal

Apache edgent

Soma Shekarchi

SparkR + Zeppelin

felixcss

CloudStack currently provides a variety bespoke high availability mechanisms for resources such as virtual machines, hosts, and virtual routers. Each of these implementations duplicates the HA check/recovery cycle, as well as, concurrency, persistence, and clustering required manage high available for any CloudStack resource. The High Availability Resource Management Service has been developed to consolidate these concerns -- providing a robust, extensible HA mechanism. Using this service, plugins only need to define health check, activity check, and fence operations.

When the Cloud is a Rockin: High Availability in Apache CloudStack

John Burwell

Michael Heemskerk The launch of JIRA, Confluence and Stash Data Center opens up both new opportunities and new challenges for add-on developers. Data Center installations consist of multiple clustered nodes that serve thousands of users. Data Center add-ons need to cope with distributed state, high load, large numbers of users, issues, pages or repositories. Join Michael Heemskerk, Stash architect, to learn all about the challenges, best practices and frameworks that are available for add-on developers that want to make their add-on ready for Data Center.

AtlasCamp 2015: Get your add-on in shape for Data Center

Atlassian

This technical tutorial is designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. The session starts with a quick introduction to Redis and the capabilities Redis provides. It will cover the basic data types provided by Redis and the module system. Using an ad serving use case, Griffith will look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. You will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark and then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will also be discussed, focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these features. By the end of the session, you should feel confident building a prototype/proof-of-concept application using Redis and Spark. You’ll understand how Redis complements Spark, and how to use Redis to serve complex, ML-models with high performance.

Getting Ready to Use Redis with Apache Spark with Tague Griffith

Databricks

Building the next generation Spark SQL engine at speed poses new challenges to both automation and testing. At Databricks, we are implementing a new testing framework for assessing the quality and performance of new developments as they produced. Having more than 1,200 worldwide contributors, Apache Spark follows a rapid pace of development. At this scale, new testing tooling such as random query and data generation, fault injection, longevity stress, and scalability tests are essential to guarantee a reliable and performance Spark later in production. By applying such techniques, we will demonstrate the effectiveness of our testing infrastructure by drilling-down into cases where correctness and performance regressions have been found early. In addition, showing how they have been root-caused and fixed to prevent regressions in production and boosting the continuous delivery of new features.

Fast and Reliable Apache Spark SQL Engine

Databricks

Real time ETL processing using Spark streaming

datamantra

Is your job running slower than usual? Do you want to make sense from the thousands of Hadoop & Spark metrics? Do you want to monitor the performance of your flow, get alerts and auto tune them? These are the common questions every Hadoop user asks but there is not a single solution that addresses it. We at Linkedin faced lots of such issues and have built a simple self serve tool for the hadoop users called Dr. Elephant. Dr. Elephant, which is already open sourced, is a performance monitoring and tuning tool for Hadoop and Spark. It tries to improve the developer productivity and cluster efficiency by making it easier to tune jobs. Since its open source, it has been adopted by multiple organizations and followed with a lot of interest in the Hadoop and Spark community. In this talk, we will discuss about Dr. Elephant and outline our efforts to expand the scope of Dr. Elephant to be a comprehensive monitoring, debugging and tuning tool for Hadoop and Spark applications. We will talk about how Dr. Elephant performs exception analysis, give clear and specific suggestions on tuning, tracking metrics and monitoring their historical trends. Open Source: https://github.com/linkedin/dr-elephant

Dr. Elephant: Achieving Quicker, Easier, and Cost-Effective Big Data Analytic...

Spark Summit

What's hot (20)

May 2021 Spark Testing ... or how to farm reputation on StackOverflow

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

Incredible ODI tips to work with Hyperion tools that you ever wanted to know

Breaking data

Data science lifecycle with Apache Zeppelin

Spark Uber Development Kit

Dagster @ R&S MNT

Spark Summit EU talk by Yiannis Gkoufas

Wrapper formulas saltconf 2016

SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...

Components of openEHR based EHRs

Interactive Analytics using Apache Spark

Apache edgent

SparkR + Zeppelin

When the Cloud is a Rockin: High Availability in Apache CloudStack

AtlasCamp 2015: Get your add-on in shape for Data Center

Getting Ready to Use Redis with Apache Spark with Tague Griffith

Fast and Reliable Apache Spark SQL Engine

Real time ETL processing using Spark streaming

Dr. Elephant: Achieving Quicker, Easier, and Cost-Effective Big Data Analytic...

Viewers also liked

Scaling up genomic analysis with ADAM

fnothaft

ADAM—Spark Summit, 2014

fnothaft

The science driving genomic analyses is rapidly changing, but the operational problems of processing data from DNA sequencers quickly and reliably are not new. I present an analysis of the parallels in the fundamental limiting components of the '90s internet boom and the DNA sequencing boom that is currently underway, and illustrate how Hadoop, a proven application architecture used widely in BigData and commercial internet applications can be reused in the genomics sector.

Genomics isn't Special

Allen Day, PhD

Genomics Is Not Special: Towards Data Intensive Biology

Uri Laserson

Strata Big Data Science Talk on ADAM

Matt Massie

Genome Analysis Pipelines with Spark and ADAM

Allen Day, PhD

We are at a time where biotech allow us to get personal genomes for $1000. Tremendous progress since the 70s in DNA sequencing have been done, e.g. more samples in an experiment, more genomic coverages at higher speeds. Genomic analysis standards that have been developed over the years weren't designed with scalability and adaptability in mind. In this talk, we’ll present a game changing technology in this area, ADAM, initiated by the AMPLab at Berkeley. ADAM is framework based on Apache Spark and the Parquet storage. We’ll see how it can speed up a sequence reconstruction to a factor 150.

Lightning fast genomics with Spark, Adam and Scala

Andy Petrella

Viewers also liked (7)

Scaling up genomic analysis with ADAM

ADAM—Spark Summit, 2014

Genomics isn't Special

Genomics Is Not Special: Towards Data Intensive Biology

Strata Big Data Science Talk on ADAM

Genome Analysis Pipelines with Spark and ADAM

Lightning fast genomics with Spark, Adam and Scala

Similar to Free Code Friday: Genome Resequencing with Spark, Part 1

In this talk will show how Large Scale Data Analytics can be done with Spark and Cassandra on the DataStax Enterprise Platform. First we will give an overview of what is the Spark Cassandra Connector and how it enables working with large data sets. Then we will use the Spark Notebook to show live examples in the browser of interacting with the data. The example will load a large Movies Database from Cassandra into Spark and then show how that data can be transformed and analyzed using Spark.

Large Scale Data Analytics with Spark and Cassandra on the DSE Platform

DataStax Academy

20151015 zagreb spark_notebooks

Andrey Vykhodtsev

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

Bolke de Bruin

In a fast-moving business environment, finance leaders are successfully leveraging technology advancements to transform their finance organizations and generate value for the business. Oracle’s Enterprise Performance Management (EPM) applications are an integrated, modular suite that supports a broad range of strategic and financial performance management tools that help business to unlock their potential. Dell’s global financial environment contains over 10,000 users around the world and relies on a range of EPM tools such as Hyperion Planning, Essbase, Smart View, DRM, and ODI to meet its needs. This session shows the complexity of this environment, describing all relationships between those tools, the techniques used to maintain such a large environment in sync, and meeting the most varied needs from the different business and laws around the world to create a complete and powerful business decision engine that takes Dell to the next level.

Take a peek at Dell's smart EPM global environment

Rodrigo Radtke de Souza

Rust is for "Big Data"

Andy Grove

Visual Mapping of Clickstream Data

DataWorks Summit

Apache Tez – Present and Future

Jianfeng Zhang

Apache Tez – Present and Future

Rajesh Balamohan

DataFrames: The Extended Cut

Wes McKinney

IBMHadoopofferingTechline-Systems2015

Daniela Zuppini

20160524 ibm fast data meetup

shinolajla

Introduction to MySQL Cluster

Abel Flórez

PySpark Best Practices

Cloudera, Inc.

MySQL Cluster

Abel Flórez

The main objective of this workshop is to give the audience hands on experience with several Hadoop technologies and jump start their hadoop journey. In this workshop, you will load data and submit queries using Hadoop! Before jumping in to the technology, the Founders of DataKitchen review Hadoop and some of its technologies (MapReduce, Hive, Pig, Impala and Spark), look at performance, and present a rubric for choosing which technology to use when. NOTE: To complete hands on poriton in the time allotted, attendees should come with a newly created AWS (Amazon Web Services) Account and complete the other prerequisites found in the DataKitchen blog <http: />.

Open Data Science Conference Big Data Infrastructure – Introduction to Hadoop...

DataKitchen

Stream Processing Everywhere - What to use?

MapR Technologies

A web application’s attack surface is the combination of URLs it will respond to as well as the inputs to those URLs that can change the behavior of the application. Understanding an application’s attack surface is critical to being able to provide sufficient security test coverage, and by watching an application’s attack surface change over time security and development teams can help target and optimize testing activities. This presentation looks at methods of calculating web application attack surface and tracking the evolution of attack surface over time. In addition, it looks at metrics and thresholds that can be used to craft policies for integrating different testing activities into Continuous Integration / Continuous Delivery (CI/CD) pipelines for teams integrating security into their DevOps practices.

Monitoring Attack Surface to Secure DevOps Pipelines

Denim Group

12 Factor Scala

Joe Kutner

Spark SQL versus Apache Drill: Different Tools with Different Rules

DataWorks Summit/Hadoop Summit

Real Time Analytics with Dse

DataStax Academy

Similar to Free Code Friday: Genome Resequencing with Spark, Part 1 (20)

Large Scale Data Analytics with Spark and Cassandra on the DSE Platform

20151015 zagreb spark_notebooks

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

Take a peek at Dell's smart EPM global environment

Rust is for "Big Data"

Visual Mapping of Clickstream Data

Apache Tez – Present and Future

DataFrames: The Extended Cut

IBMHadoopofferingTechline-Systems2015

20160524 ibm fast data meetup

Introduction to MySQL Cluster

PySpark Best Practices

MySQL Cluster

Open Data Science Conference Big Data Infrastructure – Introduction to Hadoop...

Stream Processing Everywhere - What to use?

Monitoring Attack Surface to Secure DevOps Pipelines

12 Factor Scala

Spark SQL versus Apache Drill: Different Tools with Different Rules

Real Time Analytics with Dse

More from MapR Technologies

How Data-Driven Approaches are Changing Your Data Management Strategies Introducing data-driven strategies into your business model alters the way your organization manages and provides information to your customers, partners and employees. Gone are the days of “waterfall” implementation strategies from relational data to applications within a data center. Now, data-driven business models require agile implementation of applications based on information from all across an organization–on-premises, cloud, and mobile–and includes information from outside corporate walls from partners, third-party vendors, and customers. Data management strategies need to be ready to meet these challenges or your new and disruptive business models will fail at the most critical time: when your customers want to access it.

Converging your data landscape

MapR Technologies

How Rendezvous Architecture Improves Evaluation in the Real World In this addition of our machine learning logistics webinar series we build on the ideas of the key requirements for effective management of machine learning logistics presented in the Overview webinar and in Part I Workshop. Here we focus on model-to-model comparison & evaluation, use of decoy models and more. Listen here: http://info.mapr.com/machine-learning-workshop2.html?_ga=2.35695522.324200644.1511891424-416597139.1465233415

ML Workshop 2: Machine Learning Model Comparison & Evaluation

MapR Technologies

Self-Service Data Science for Leveraging ML & AI on All of Your Data

MapR Technologies

Enabling Real-Time Business with Change Data Capture

MapR Technologies

Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...

MapR Technologies

Having heard the high-level rationale for the rendezvous architecture in the introduction to this series, we will now dig in deeper to talk about how and why the pieces fit together. In terms of components, we will cover why streams work, why they need to be persistent, performant and pervasive in a microservices design and how they provide isolation between components. From there, we will talk about some of the details of the implementation of a rendezvous architecture including discussion of when the architecture is applicable, key components of message content and how failures and upgrades are handled. We will touch on the monitoring requirements for a rendezvous system but will save the analysis of the recorded data for later. Listen to the webinar on demand: https://mapr.com/resources/webinars/machine-learning-workshop-1/

ML Workshop 1: A New Architecture for Machine Learning Logistics

MapR Technologies

Join Ellen Friedman, co-author (with Ted Dunning) of a new short O’Reilly book Machine Learning Logistics: Model Management in the Real World, to look at what you can do to have effective model management, including the role of stream-first architecture, containers, a microservices approach and a DataOps style of work. Ellen will provide a basic explanation of a new architecture that not only leverages stream transport but also makes use of canary models and decoy models for accurate model evaluation and for efficient and rapid deployment of new models in production.

Machine Learning Success: The Key to Easier Model Management

MapR Technologies

Data warehouses have been the standard tool for analyzing data created by business operations. In recent years, increasing data volumes, new types of data formats, and emerging analytics technologies such as machine learning have given rise to modern data lakes. Connecting application databases, data warehouses, and data lakes using real-time data pipelines can significantly improve the time to action for business decisions. More: http://info.mapr.com/WB_MapR-StreamSets-Data-Warehouse-Modernization_Global_DG_17.08.16_RegistrationPage.html

Data Warehouse Modernization: Accelerating Time-To-Action

MapR Technologies

Live Tutorial – Streaming Real-Time Events Using Apache APIs

MapR Technologies

Deploying storage with a forklift is so 1990s, right? Today’s applications and infrastructure demand systems and services that scale. Customers require performance and capacity to fit the use case and workloads, not the other way around. Architects need multi-temperature, multi-location, highly available, and compliance friendly platforms that grow with the generational shift in data growth and utility.

Bringing Structure, Scalability, and Services to Cloud-Scale Storage

MapR Technologies

Churn prediction is big business. It minimizes customer defection by predicting which customers are likely to cancel a service. Though originally used within the telecommunications industry, it has become common practice for banks, ISPs, insurance firms, and other verticals. More: http://info.mapr.com/WB_PredictingChurn_Global_DG_17.06.15_RegistrationPage.html The prediction process is data-driven and often uses advanced machine learning techniques. In this webinar, we'll look at customer data, do some preliminary analysis, and generate churn prediction models – all with Spark machine learning (ML) and a Zeppelin notebook. Spark’s ML library goal is to make machine learning scalable and easy. Zeppelin with Spark provides a web-based notebook that enables interactive machine learning and visualization. In this tutorial, we'll do the following: Review classification and decision trees Use Spark DataFrames with Spark ML pipelines Predict customer churn with Apache Spark ML decision trees Use Zeppelin to run Spark commands and visualize the results

Live Machine Learning Tutorial: Churn Prediction

MapR Technologies

An Introduction to the MapR Converged Data Platform

MapR Technologies

IT budgets are shrinking, and the move to next-generation technologies is upon us. The cloud is an option for nearly every company, but just because it is an option doesn’t mean it is always the right solution for every problem. Most cloud providers would prefer that every customer be tightly coupled with their proprietary services and APIs to create lock-in with that cloud provider. The savvy customer will leverage the cloud as infrastructure and stay loosely bound to a cloud provider. This creates an opportunity for the customer to execute a multicloud strategy or even a hybrid on-premises and cloud solution. Jim Scott explores different use cases that may be best run in the cloud versus on-premises, points out opportunities to optimize cost and operational benefits, and explains how to get the data moved between locations. Along the way, Jim discusses security, backups, event streaming, databases, replication, and snapshots across a variety of use cases that run most businesses today.

How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...

MapR Technologies

Best Practices for Data Convergence in Healthcare

MapR Technologies

Changes in how business is done combined with multiple technology drivers make geo-distributed data increasingly important for enterprises. These changes are causing serious disruption across a wide range of industries, including healthcare, manufacturing, automotive, telecommunications, and entertainment. Technical challenges arise with these disruptions, but the good news is there are now innovative solutions to address these problems. http://info.mapr.com/WB_Geo-distributed-Big-Data-and-Analytics_Global_DG_17.05.16_RegistrationPage.html

Geo-Distributed Big Data and Analytics

MapR Technologies

MapR Product Update - Spring 2017

MapR Technologies

3 Benefits of Multi-Temperature Data Management for Data Analytics

MapR Technologies

Cisco & MapR bring 3 Superpowers to SAP HANA Deployments

MapR Technologies

You’re not the only one still loading your data into data warehouses and building marts or cubes out of it. But today’s data requires a much more accessible environment that delivers real-time results. Prepare for this transformation because your data platform and storage choices are about to undergo a re-platforming that happens once in 30 years. With the MapR Converged Data Platform (CDP) and Cisco Unified Compute System (UCS), you can optimize today’s infrastructure and grow to take advantage of what’s next. Uncover the range of possibilities from re-platforming by intimately understanding your options for density, performance, functionality and more.

MapR and Cisco Make IT Better

MapR Technologies

Evolving from RDBMS to NoSQL + SQL

MapR Technologies

More from MapR Technologies (20)

Converging your data landscape

ML Workshop 2: Machine Learning Model Comparison & Evaluation

Self-Service Data Science for Leveraging ML & AI on All of Your Data

Enabling Real-Time Business with Change Data Capture

Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...

ML Workshop 1: A New Architecture for Machine Learning Logistics

Machine Learning Success: The Key to Easier Model Management

Data Warehouse Modernization: Accelerating Time-To-Action

Live Tutorial – Streaming Real-Time Events Using Apache APIs

Bringing Structure, Scalability, and Services to Cloud-Scale Storage

Live Machine Learning Tutorial: Churn Prediction

An Introduction to the MapR Converged Data Platform

How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...

Best Practices for Data Convergence in Healthcare

Geo-Distributed Big Data and Analytics

MapR Product Update - Spring 2017

3 Benefits of Multi-Temperature Data Management for Data Analytics

Cisco & MapR bring 3 Superpowers to SAP HANA Deployments

MapR and Cisco Make IT Better

Evolving from RDBMS to NoSQL + SQL

Recently uploaded

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

DBX First Quarter 2024 Investor Presentation

Dropbox

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

Architecting Cloud Native Applications

WSO2

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Exploring Multimodal Embeddings with Milvus

Zilliz

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

Recently uploaded (20)

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DBX First Quarter 2024 Investor Presentation

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Boost Fertility New Invention Ups Success Rates.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Architecting Cloud Native Applications

Artificial Intelligence Chap.5 : Uncertainty

Why Teams call analytics are critical to your entire business

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Exploring Multimodal Embeddings with Milvus

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

AWS Community Day CPH - Three problems of Terraform

Corporate and higher education May webinar.pptx

Platformless Horizons for Digital Adaptability

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

CNIC Information System with Pakdata Cf In Pakistan

Free Code Friday: Genome Resequencing with Spark, Part 1

2. © 2015 MapR Technologies 2 Let’s kick of a build while we talk… • Follow the instructions at • https://github.com/allenday/spark-genome-alignment-demo • While that’s cooking… • What’s this all about?

6. © 2015 MapR Technologies 6 Many-to-Many Relationship Between Reads and Locations • Read1 • Read2 • Read3 • Read4 • NULL • LocationA • LocationB • LocationC • LocationD • LocationA • NULL • LocationE

10. © 2015 MapR Technologies 10 How to Do it? FastQ => BAM Prepare Environment • Get Input Reference Sequences in FastA format (one time step) • Index Reference Sequences with Bowtie (one time step) Get Input Data • Get Input Unaligned Reads in FastQ format (per input data set) Do the Work • Convert Reads to ADAM Format with Spark (per input data set) • Use Spark Pipe to Align the Reads (per input data fragment) • Write the Alignments to SAM or BAM format (per input data, optional)

11. © 2015 MapR Technologies 11 Let’s look at some code… • Let’s check on the build we kicked off earlier. • And the driver script: • cat $DEMO/bin/bowtie_pipe_single.scala • I said before that it’s possible to be 100% compatible with legacy tools by writing out to SAM or BAM. • You might not want to do this…

12. © 2015 MapR Technologies 12 Benefits of Keeping Data in Spark / ADAM • ADAM file sizes are about 20% smaller than BAM. • ADAM files are faster for common operations (filters, range queries) • ADAM files sort more quickly than `samtools sort …`. This can be a huge time savings

14. © 2015 MapR Technologies 14 • Some other commands built-in, take a look: • $ADAM_HOME/bin/adam-submit • $ADAM_HOME/bin/adam-submit count_kmers $DEMO/build/data/reads.sam /tmp/count_kmers 3 • head /tmp/count_kmers/part-00000 • All of this is easier without leaving the Spark shell • And it’s faster, too…

Free Code Friday: Genome Resequencing with Spark, Part 1

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Free Code Friday: Genome Resequencing with Spark, Part 1

Similar to Free Code Friday: Genome Resequencing with Spark, Part 1 (20)

More from MapR Technologies

More from MapR Technologies (20)

Recently uploaded

Recently uploaded (20)

Free Code Friday: Genome Resequencing with Spark, Part 1