PixieDust: an Open Source Library that simplifies and improves Jupyter Python Notebooks

1. @MargrietGr
Margriet Groenendijk
Developer Advocate for IBM Cloud Data Services
Python DBBUG meetup
Bristol
28 November 2016
PixieDust: an Open Source Library that simplifies
and improves Jupyter Python Notebooks

2. @MargrietGr
https://github.com/ibm-cds-labs/pixiedust

3. @MargrietGr
Run Spark locally in a Python notebook
https://www.continuum.io/downloads
http://spark.apache.org/downloads.html
Create a new kernel to use in a
Jupyter notebook

4. @MargrietGr
Spark installation
•http://spark.apache.org/downloads.html
•Spark release: 1.6.2
•package type: Pre-built for Hadoop 2.6
•mkdir dev
•cd dev
•tar xzf ~/Downloads/spark-1.6.2-bin-hadoop2.6.tgz
•ln -s spark-1.6.2-bin-hadoop2.6 spark
•mkdir dev/notebooks

5. @MargrietGr
• create directory ~/.ipython/kernels/pyspark1.6/
• create file kernel.json
• cd ~/dev/spark/conf
• cp spark-defaults.conf.template spark-defaults.conf
• add to end of spark-defaults.conf:
spark.driver.extraClassPath=<HOME DIRECTORY>/data/libs/*
{
"display_name": "pySpark (Spark 1.6.2) Python 2",
"language": "python",
"argv": [
"/Users/sparktest/miniconda2/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"env": {
"SPARK_HOME": "/Users/sparktest/dev/spark",
"PYTHONPATH": "/Users/sparktest/dev/spark/python/:/Users/sparktest/dev/spark/python/lib/py4j-0.9-src.zip",
"PYTHONSTARTUP": "/Users/sparktest/dev/spark/python/pyspark/shell.py",
"PYSPARK_SUBMIT_ARGS": "--master local[10] pyspark-shell",
"SPARK_DRIVER_MEMORY": "10G",
"SPARK_LOCAL_IP": "127.0.0.1"
}
}

6. @MargrietGr
Jupyter Notebooks in the Cloud
• Server-client application to edit and run
notebook documents via a web browser
• Cells with:
•Code
•Figures and tables
•Rich text elements
• Different kernels: Python, R, Scala,
Spark

7. @MargrietGr
http://datascience.ibm.com/

8. @MargrietGr
PixieDust: an Open Source Library that simplifies and
improves Jupyter Python Notebooks
• PackageManager
• Visualizations
• Cloud Integration
• Scala Bridge
• Extensibility
• Embedded Apps
https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for-
python-notebook/
@DTAIEB55

9. @MargrietGr
Install Spark packages or plain jars in your Notebook Python
kernel without the need to modify configuration file
Uses the GraphFrame Python APIs
Install GraphFrames Spark Package

10. @MargrietGr
One simple API: display()
Call the Options dialog
Panning/Zooming
options
Performance statistics

11. @MargrietGr
Easily export your data to csv, json, html, etc. locally on your laptop
or into a cloud-based service like Cloudant or Object Storage

12. @MargrietGr
Scala Bridge
Define a Python variable
Use the Python var in Scala
Define a Scala variable
Use the Scala var in Python

13. @MargrietGr
Easily extend PixieDust to create your own visualizations
using HTML/CSS/JavaScript
Customized
Visualization for
GraphFrame
Graphs

14. @MargrietGr
Encapsulate your analytics into compelling User
Interfaces better suited for Line of Business Users

15. @MargrietGr

16. @MargrietGr
Sign up for beta: http://datascience.ibm.com/features#machinelearning

17. @MargrietGr
https://developer.ibm.com/clouddataservices/author/
mgroenen/
Thanks!
Slides will be here:
http://www.slideshare.net/MargrietGroenendijk