This document provides information about NumPy memory mapped arrays and Enthought Python Distribution (EPD). It discusses how memory mapped arrays allow large datasets to be accessed as NumPy arrays while residing in disk files rather than main memory. It also advertises EPD software and training courses for scientific computing in Python.

1. NumPy MemoryMapped Arrays May 22, 2009

2. A word from our sponsor…

4. Enthought Python Distribution (EPD) Explanations, demonstrations, and tips For subscribers to Enthought Python Distribution (EPD) and their guests. Presenters and Panelists will include Enthought experts and other leading community members.

5. Enthought Training Courses Python Basics, NumPy, SciPy, Matplotlib, Traits, TraitsUI, Chaco…

6. Upcoming Training Classes June 15 - 19, 2009 Introduction to Scientific Computing with Python Austin, Texas July, 2009 TBA August, 2009 TBA September 21 to 25, 2009 Introduction to Scientific Computing with Python Austin, Texas http://www.enthought.com/training/

7. Enthought Consulting Process Built with Python by a team of scientists, EPD provides a versatile and coherent platform for analysis and visualization .

8. Software Application Layers Python NumPy (Array Mathematics) SciPy (Scientific Algorithms) 3 rd Party Libraries wxPython VTK, etc. ETS (App construction) Traits, Chaco, Mayavi, Envisage, etc. Domain Specific GUI Applications Semiconductor, Fluid Dynamics, Seismic Modeling, Financial, etc.

9. Shell

10. Chaco: Interactive Graphics

12. Multiple Plug-ins. One Application

14. Database Access Compliance Tools Equipment Interface Scientific Algorithms UI Elements Testing Framework Scripting Interface Chaco Plotting Data Display Rich Client App (Geophysics, Finance, Etc)

15. NumPy

16. Array Data Structure

17. “Structured” Arrays name char[10] age int weight double Elements of an array can be any fixed-size data structure! EXAMPLE >>> from numpy import dtype, empty # structured data format >>> fmt = dtype([('name', 'S10'), ('age', int), ('weight', float) ]) >>> a = empty((3,4), dtype=fmt) >>> a.itemsize 22 >>> a['name'] = [['Brad', ,'Jill']] >>> a['age'] = [[33, ,54]] >>> a['weight'] = [[135, ,145]] >>> print a [[('Brad', 33, 135.0) ('Jill', 54, 145.0)]] 27 32 61 29 145.0 88.0 135.0 188.0 54 18 33 19 Jill Jennifer Susan Ron 187.0 137.0 202.0 154.0 Amy Brian George Henry 140.0 225.0 105.0 135.0 54 47 25 33 Fred John Jane Brad

18. Even Nested Datatypes

19. Nested Datatype dt = dtype([('time', np.uint64), ('size', np.uint32), ('position', [('az', np.float32), ('el', np.float32), ('region_type', np.uint8), ('region_ID', np.uint16)]), ('gain', np.uint8), ('samples', (np.int16,2048))]) data = np.fromfile(f, dtype=dt) If you only wanted to access a part of the file use a memory map

20. Virtual Memory http://en.wikipedia.org/wiki/Virtual_memory Memory mapped files are like intentional disk-based virutal memory

25. memmap -- write through behavior # Create a memory mapped "write through" file, overwriting it if it exists. In [66]: q=memmap('new_file.dat',mode='w+',shape=(2,5)) In [67]: q memmap([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], dtype=uint8) # Print out the contents of the underlying file. Note: It # doesn't print because 0 isn't a printable ascii character. In [68]: !cat new_file.dat # Now write the ascii value for 'A' (65) into our array. In [69]: q[:] = ord('A') In [70]: q memmap([[65, 65, 65, 65, 65], [65, 65, 65, 65, 65]], dtype=uint8) # Ensure the OS has written the data to the file, and examine # the underlying file. It is full of 'A's as we hope. In [71]: q.flush() In [72]: !cat new_file.dat AAAAAAAAAA

26. memmap -- copy on write behavior # Create a copy-on-write memory map where the underlying file is never # modified. The file must already exist. # This is a memory efficient way of working with data on disk as arrays but # ensuring you never modify it. In [73]: q=memmap('new_file.dat',mode='c',shape=(2,5)) In [74]: q memmap([[65, 65, 65, 65, 65], [65, 65, 65, 65, 65]], dtype=uint8) # Set values in array to something new. In [75]: q[1] = ord('B') In [76]: q memmap([[65, 65, 65, 65, 65], [66, 66, 66, 66, 66]], dtype=uint8) # Even after calling flush(), the underlying file is not updated. In [77]: q.flush() In [78]: !cat new_file.dat AAAAAAAAAA

27. Using Offsets # Create a memory mapped array with 10 elements. In [1]: q=memmap('new_file.dat',mode='w+', dtype=uint8, shape=(10,)) In [2]: q[:] = arange(0,100,10) memmap([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90], dtype=uint8) # Now, create a new memory mapped array (read only) with an offset into # the previously created file. In [3]: q=memmap('new_file.dat',mode='r', dtype=uint8, shape=6, offset=4) In [4]: q memmap([40, 50, 60, 70, 80, 90], dtype=uint8) # The number of bytes required by the array must be equal or less than # the number of bytes available in the file. In [3]: q=memmap('new_file.dat',mode='r', dtype=uint8, shape= 7 , offset=4) ValueError: mmap length is greater than file size new_file.dat new_file.dat

28. Working with file headers 64 bit floating point data… # Create a dtype to represent the header. header_dtype = dtype([('rows', int32), ('cols', int32)]) # Create a memory mapped array using this dtype. Note the shape is empty. header = memmap(file_name, mode='r', dtype=header_dtype, shape=()) # Read the row and column sizes from using this structured array. rows = header['rows'] cols = header['cols'] # Create a memory map to the data segment, using rows, cols for shape # information and the header size to determine the correct offset. data = memmap(file_name, mode='r+', dtype=float64, shape=(rows, cols), offset=header_dtype.itemsize) rows (int32) data header File Format: cols (int32)

29. Accessing Legacy Files header harr data darr dtype objects NumPy arrays binary file format = harr['format'] time = harr['time'] Explore in Python as NumPy Arrays a = darr['image'][:30,:40]

30. Strategy for creating new files!

31. Memmap Timings (3D arrays) All times in milliseconds (ms). Linux: Ubuntu 4.1, Dell Precision 690, Dual Quad Core Zeon X5355 2.6 GHz, 8 GB Memory OS X: OS X 10.5, MacBook Pro Laptop, 2.6 GHz Core Duo, 4 GB Memory 27 ms 3505 ms 11 ms 2103 ms read 7.4 ms 4.4 ms 4.6 ms 2.8 ms y slice 8.3 ms 1.8 ms 4.8 ms 1.8 ms x slice 0.02 ms 9.2 ms In Memory Linux downsample 4x4 z slice Operations (500x500x1000) 198.7 ms 0.02 ms 125 ms 18.7 ms 10 ms 13.8 ms Memory Mapped In Memory Memory Mapped OS X

32. Parallel FFT On Memory Mapped File 500 MB memory mapped data file split & assign rows Run parallel code on each processor

33. Parallel FFT On Memory Mapped File 1.0 11.75 1 3.5 3.36 4 1.9 6.06 2 2.50 Time (seconds) 8 Processors 4.7 Speed Up

36. EPD & EPD Webinars: http://www. enthought . com/products/epd . php Enthought Training: http://www. enthought .com/training/