2016/10/04のDB Analytics Showcaseでのプレゼンです。

1. Python / pandas
Sky

3. •
• Python 2000
(**)
• db tech showcase MongoDB
•
• FB: Ryuji Tamagawa
• Twitter : tamagawa_ryuji

5. 2015-2016

6. • Python / pandas
• Python / pandas

7. • Python
•
• Python
• NumPy, SciPy, matplotlib, pandas
• Python
• Python IPython, Jupyter notebook, Spyder, VisualStudio
• Python / pandas
• Python
• pandas
• Spark - PySpark DataFrame API
• matplotlib

8. Part 1 : Python

9. Python
•
• Google Guido
Google Google
1
•
NumPy, SciPy, matplotlib →
pandas
•
•
-2000
Linux
-2010 Web Trac
Google

10. Python
•
•
•
•
•
• ’Batteries included’

11. Python
• 2.x 3.x 32bit 64bit
64bit
• 2.x
• 3.x
3
• 2.x
3.x

12. • Ruby?
• R?
• Java?
• Scala?

13. Python
• Python ’CPython’ JIT
PyPy JVM Jython .Net IronPython
• CPython
• CPython 2
• C
• processing
PySpark

14. Python
• Python Linux Mac OS
Python Python
Mac
• Python pip 3.x 2.7.9 2.x
Python pip Linux Python
pip yum apt
• Python Anaconda Python
conda
• python 2016
http://qiita.com/y__sama/items/5b62d31cb7e6ed50f02c

15. NumPy, SciPy, matplotlib, pandas
•
• NumPy SciPy
• pandas
pandas pandas NumPy
• Anaconda Python

16. Python
•
scikit-learn http://
scikit-learn.org/stable/

17. Python
• TensorFlow
Python

18. Python
IPython
Jupyter, …
IDE
Spyder, Rodeo
Visual Studio, PyCharm,
PyDev

19. • IPython
•
•
• Anaconda

20.
• Jupyter Notebook
• Python
• IPython Notebook
Python
• Apache Zeppelin http://
zeppelin.apache.org

21. IDE
• R RStudio
• IDE
•
• 2 Spyder Rodeo
•
Spyder

22. •
• Visual Studio
• Eclipse PyDev
• PyCharm
•

23. Part 2 :
Python / pandas

24. Python / pandas
• pandas
• /
etc…
•
Spark
• pandas

25. processing
•
• 64bit Python +
GB
• Python 1
1 CPU
GIL
• processing
Jenkins
CPU/
Jenkins

26. 1 1.2 1000000
‘abc’ ’ ’
[1, 2, 3, ‘foo’, ‘bar’, ‘foo’]
(1, 2, 3, ‘foo’, ‘bar’, ‘foo’)
{‘k1’: ‘value1’, ‘k2’: ‘value2’}
set(1, 2, 3, ‘foo’, ‘bar’)

27. •
•
• split
s = ‘foo, bar, baz’
items = s.split(‘,’)
print items[0]
print items[-1]
print items[0][-2:]

28. • ,
• lambda map, reduce, filter
sList = [‘foo’, ‘bar’, ‘baz’]
lList = [len(s) for s in sList]
lList = map(lambda s:len(s), sList)
lDict = {s:len(s) for s in sList}
lList = []
for s in sList:
lList.append(len(s))
lDict = {}
for s in sList:
lDict[s] = len(s)

29. pandas
• pandas
•
matplotlib / seaborn
• NumPy
SciPy
Python
• pandas + matplotlib
OK pandas NumPy
NumPy /
SciPy
https://openbook4.me/projects/183

31. pandas
• pandas
DataFrame
• R
• RDB
2
• index Series Columns
Columns
Series Series SeriesIndex

32. IDE /
• IDE
• jupyter notebook

33. • http://sinhrks.hatenablog.com/entry/2015/01/28/073327
0 1
import pandas as pd
df[‘nValue’] = df[‘value’] / sum(df[‘value’])
id value color
sapporo 43 red
osaka 42 pink
matsumoto 40 green
id value color nValue
sapporo 43 red 0.344
osaka 42 pink 0.336
matsumoto 40 green 0.32
Python

34. pandas I/O
• CSV JSON RDB Excel
• column
• RDB
•
import pandas as pd
pd.read_csv(<filename>)
pd.read_json(<filename>)
pd.to_csv(<filename>)
pd.to_excel(<filename>)
#
pd.to_clipboard()

35. pandas.read_csv
• pandas CSV
•
•
• usecols :
• nrows :
• na_values : na
• parse_dates infer_datetime_format:
• chunksize :
• compression : zip CSV
pandas.read_csv(filepath_or_buffer, sep=', ',
delimiter=None, header='infer', names=None,
index_col=None, usecols=None, squeeze=False,
prefix=None, mangle_dupe_cols=True, dtype=None,
engine=None, converters=None, true_values=None,
false_values=None, skipinitialspace=False,
skiprows=None, skipfooter=None, nrows=None,
na_values=None, keep_default_na=True,
na_filter=True, verbose=False,
skip_blank_lines=True, parse_dates=False,
infer_datetime_format=False,
keep_date_col=False, date_parser=None,
dayfirst=False, iterator=False, chunksize=None,
compression='infer', thousands=None, decimal='.',
lineterminator=None, quotechar='"', quoting=0,
escapechar=None, comment=None,
encoding=None, dialect=None, tupleize_cols=False,
error_bad_lines=True, warn_bad_lines=True,
skip_footer=0, doublequote=True,
delim_whitespace=False, as_recarray=False,
compact_ints=False, use_unsigned=False,
low_memory=True, buffer_lines=None,
memory_map=False, float_precision=None)

36. Spark - PySpark DataFrame API
•
Python
• Spark PySpark
findSpark
Spark
• Python Spark API
DataFrame API
• Spark pandas
Spark
PySpark
Spark
node
Spark
node
Spark
node
Spark
node
driver

37. •
•
Apache Arrow
• Python / R
:
feather
• pandas 2.0, parquet for Python

38. Python / pandas

39. Questions ?