Genomic Data Pipelines
•Download as PPTX, PDF•
1 like•600 views
This document discusses genomic-scale data pipelines. It introduces Dr. Denis Bauer and his transformational bioinformatics team. It describes how genomic data and research will grow exponentially to exabytes by 2025. It outlines genomic research workflows and challenges like processing, analyzing, and visualizing large variant call format (VCF) data. It presents two cloud data pipeline patterns used by the team: 1) A Spark server cluster pipeline for machine learning on large genomic datasets. 2) A serverless pipeline using AWS Lambda and Step Functions for scalable genomic searches.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Genomic Data Pipelines
Similar to Genomic Data Pipelines (20)
More from Lynn Langit
More from Lynn Langit (20)
Recently uploaded
Recently uploaded (20)
Genomic Data Pipelines
- 1. Dr. Denis Bauer & Lynn Langit Genomic-scale Data Pipelines
- 2. Denis Bauer, PhD Oscar Luo, PhD Rob Dunne, PhD Piotr Szul Team Aidan O’BrienLaurence Wilson, PhD Adrian White Andy Hindmarch Collaborators David Levy News Software Dan Andrews Kaitao Lai, PhD Arash Bayat John Hildebrandt Mia Chapman Ian Blair Kelly Williams Jules Damji Gaetan Burgio Lynn Langit Natalie Twine, PhD Prabha Pillay Transformational Bioinformatics | Denis C. Bauer | @allPowerde Transformational Bioinformatics Team
- 3. 1000 17 2000 0 500 1000 1500 2000 2500 Astronomy Twitter YouTube Big Data in 2025…Petabytes? 1000 17 2000 0 500 1000 1500 2000 2500 Astronomy Twitter YouTube Big Data in 2025…Petabytes? Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 4. 1 0.17 2 20 0 5 10 15 20 25 Astronomy Twitter YouTube Genomic GENOMIC Big Data in 2025 - Exabytes Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 5. Genome holds Blueprint for Every Cell Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 6. Affects Looks, Disease Risk, and Behavior Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 7. VCF Data Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 8. Transformational Bioinformatics | Denis C. Bauer | @allPowerde Genomic Research Workflow https://www.projectmine.com/about/ BigData Focus
- 9. Finding the Disease Gene(s) Spot the letter that is… • common amongst all affected • absent in all unaffected* * oversimplified cases controls Gene1 Gene2 Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 10. BMC Genomics 2015, 16:1052 PMID: 26651996 (IF=4) Cited 4 Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 11. Why Apache Spark? Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 12. Performance – Faster and More Accurate VariantSpark is the only method to scale to 100% of the genome Transformational Bioinformatics | Denis C. Bauer | @allPowerde low Accuracy high lowSpeedhigh
- 13. CloudDataPipelinePattern Business Problem Data Quality Candidate Technologies Build/Test MVPs Assemble Pipeline Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 14. Building a CloudDataPipeline Candidate Technologies • Ingest/Clean • Analyze/Predict • Visualize Build MVPs • Test • Iterate • Learn Assemble Pipeline • Combine pieces • Validate sections • Test at scale Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 15. Building a Cloud Data Pipeline Spark •IaaS, PaaS, SaaS Vendors •AWS, Azure, GCP… Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 16. Visualizing Machine Learning Results Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 17. Solving Important Questions… Cancer genomics? Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 18. DEMO: Who is a Bondi Hipster? Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 19. Supervised ML: Wide Random Forests Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 20. Scaling to 50 M variables and 10 K samples Transformational Bioinformatics | Denis C. Bauer | @allPowerde 100K trees: 5 – 50h AWS: ~$215.50 100K trees: 200 – 2000h AWS: ~ $ 8620.00 • Yarn Cluster • 12 workers • 16 x Intel CPUs • Xeon E5-2660@2.20GHz • 128 GB RAM • Spark 1.6.1 • 128 executors • 6GB / executor 0.75TB • Synthetic dataset Whole Genome Range GWAS Range
- 21. Future Directions for VariantSpark RF Mixed feature types Unordered Categorical Continuous Build Community Python API Non-Genomic Demos Transformational Bioinformatics | Denis C. Bauer | @allPowerde Implementation by
- 22. Try it out: VariantSpark Notebook Transformational Bioinformatics| Denis C. Bauer @allPowerde https://docs.databricks.com/spark/latest/training/variant-spark.html
- 23. Genome Editing can correct genetic diseases, ex. hypertrophic cardiomyopathy “Editing does not work every time, e.g. only 7 in 10 embryos were mutation free.” Aim: Develop computational guidance framework to enable edits the first time; every time Ma et al. Nature 2017 * * Controversy around the paper – stay tuned Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 24. Make Process Parallel and Scalable SPEED • Each search can be broken down into parallel tasks - each takes seconds SCALE • Researchers might want to search the target for one gene or 100,000 Scalability + Agility = Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 25. One of the first Serverless Applications in Research Transformational Bioinformatics | Denis C. Bauer | @allPowerde Featured in
- 27. X-Ray Tracing Demo of GT-Scan2 • Find performance bottlenecks • Fix and test Transformational Bioinformatics | Denis C. Bauer | @allPowerde Webapp Resources (S3, DynamoDB) Lambda
- 28. 25 50 75 getFastaSequence createJob targetScan offtargetScanStarter offtargetSearch targetIntersects targetTranscriptionIntersects targetW uScorer targetSgR N AScorer O nTargetScorer genom eC R ISPR functions runtime(s) Type base old GTScan2 X-Ray Analysis Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 29. Results – 4x Faster (80% improvement) Transformational Bioinformatics | Denis C. Bauer | @allPowerde 2 min 30 sec
- 30. Considering Services for GT-Scan2 • Use AWS Step Functions • Simplify workflow • Simplify task timeouts • Simplify task failures • Must evaluate costs • SNS vs. Step Functions Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 31. CloudDataPipelinePattern Problem Data Technologies MVPs Pipeline Search GTScan2 fastq, bed-> S3, NoSQL Ingest ETL, Analyze Viz S3 Lambda Lambda/API Gateway Serverless Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 32. Serverless Pipeline Pattern Lambda function 1 Lambda function 2 Lambda function 3 buckets with objects DynamoDB API Gateway Users Step Functions Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 33. CloudDataPipelinePattern Problem Data Technologies MVPs Pipeline Analyze GWAS vcf -> S3/Spark Ingest ETL Analyze Viz S3 -> Databricks DBFS Apache Spark Variant-Spark ML Notebook, SQL, R, Python Spark Server Cluster Transformational Bioinformatics| Denis C. Bauer @allPowerde
- 34. Spark Server Cluster Pipeline Pattern Transformational Bioinformatics | Denis C. Bauer | @allPowerde Jupyter Notebook Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 35. Cloud Genomic-Scale Data Pipelines • Problem # 1 – ML on Large Data • Solution: Spark-server cluster + custom machine learning • Problem #2 – Burstable Search • Solution: Serverless pipeline Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 36. Genomic-scale Data Pipelines Transformational Bioinformatics | Denis C. Bauer | @allPowerde Dr. Denis Bauer & Lynn Langit
Editor's Notes
- http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002195
- http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002195
- https://www.genome.gov/18016863/a-brief-guide-to-genomics/ https://www.thinglink.com/scene/617714375666434050
- http://images.wisegeek.com/woman-in-greek-tank-top-looking-at-thumb.jpg http://nborganics.com.au/index.php/product/herbs-coriander/
- http://www.internationalgenome.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-40/
- Using this instead?
- Denis: Do we need this ?
- https://academics.cloud.databricks.com/#notebook/170398/command/170419 – AND-- http://www.drjasonfox.com/
- https://databricks.com/blog/2017/07/26/breaking-the-curse-of-dimensionality-in-genomics-using-wide-random-forests.html
- http://www.nature.com/nature/journal/v462/n7276/fig_tab/nature08645_F1.html Bauer et al. Trends Mol Med. 2014 PMID: 24801560.
- https://www.gt-scan.net/ --AND- AMA with Dr, Bauer -- https://www.reddit.com/r/science/comments/5fiicm/science_ama_series_im_denis_bauer_a_team_leader/
- https://aws.amazon.com/xray/details/
- X-ray has helped us isolate the component in our GT-Scan2 pipeline that slows the overall execution time, and how replacing it with a more performant function increased runtime 4 fold (125s to 32s).
- X-ray has helped us isolate the component in our GT-Scan2 pipeline that slows the overall execution time, and how replacing it with a more performant function increased runtime 4 fold (125s to 32s).
- Recent team presentation - https://www.slideshare.net/AustralianNationalDataService/gtscan2-bringing-bioinformatics-to-the-cloud-may-tech-talk
- Quickly access a managed Spark cluster - AWS EC2 / spot instances Link to your data and perform whole genome analysis in real-time