The document discusses the data analysis tool Apache Pig. Pig is a platform for analyzing large datasets that uses a high-level language called Pig Latin for expressing data analysis programs. Pig is built on Hadoop and allows for easy programming while also providing optimization opportunities and extensibility. The document provides an overview of Pig internals including its data structures, basic operations like group, join and order, and architecture. Examples of Pig Latin code and use cases for Pig are also presented.

1. Pig : Data Analysis Tool in the Cloud Jeff Zhang zjffdu@gmail.com Committer of Pig in ASF

2. Agenda Background What is Pig Brief introduction of Pig internals Demo Q/A

4. Then, Pig’s Coming

5. What is Pig Apache Pig is a platform for analyzing large data sets that consists of a high-level language (PigLatin) for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Ease of programming Optimization opportunities Extensibility Built upon Hadoop

7. Operators in Pig-Latin Load - a = load ‘data’ usingPigStorage(‘’) as (f1:int ,f2:double,f3:chararray) Store - store a into ‘/test/output’ usingPigStorage(‘,’) Dump - dump a Filter - b = filter a by f1 > 0 and f2 == ‘java_one’ Foreach - b = foreach a generate f1, f3 Group - b= group a by f3; Join - b = Join a by f1, b by f1; Describe - describe b; ….

8. Data Structure in Pig Cell  field in database - Primitive types: int, long, float, double, bytearray, chararrar,nul - Complex types: map, tuple, databag Tuple row (1, 1.2, “java”) DataBag table or view { (1, 1.2, “java”), (2,2.3, “c++”) , (3,4.5,”c”) }

9. How to use Pig Grunt (Interactive Shell) Java API Other languages (in future)

10. Architecture of Pig Grunt (Interactive shell) PigServer (Java API) Parser (PigLatinLogicalPlan) PigContext Optimizer (LogicalPlan LogicalPlan) Compiler (LogicalPlan PhysiclaPlan  MapReducePlan) ExecutionEngine Hadoop

11. Three basic operations of Pig Group by Join Order

12. How Pig do Group by Data Source  Split  Mapper  Partition  Reducer (A,1) (B,2) (C,3) (A,1) (B,2) (C,3) (B,4) (B,5) (C,6) (A,7) (E,8) (D,9) (A,{(A,1),(A,7)}) (C,{(C,3),(C,6)}) (E,{(E,8)}) (B,4) (B,5) (C,6) (B,{(B,2),(B,4),(B,5)}) (D,{(D,9)}) (A,7) (E,8) (D,9)

13. How Pig do Join Data Source  Split  Mapper  Partition  Reducer (1,A1) (4,A4) (3,A3) (5,A5) (2,A2) (1,A1) (4,A4) (5,B5) (1,B1) ((1,A1),(1,B1)) ((3,A3),(3,B3)) ((5,A5),(5,B5)) (3,A3) (5,A5) (3,B3) (2,B2) (5,B5) (1,B1) (3,B3) (2,B2) (4,B4) ((2,A2)(2,B2)) ((4,B4),(4,B4)) (2,A2) (4,B4)

14. How Pig do Sort Data Source  Split  Mapper  Range Partition  Reducer (100) (200) (900) (50) (100) (200) (300) (400) (100) (200) (900) (50) (600) (800) (300) (400) (50) (600) (800) (600) (800) (300) (400)

15. UDF (User-Defined-Function) register myudf.jar; raw_data= load ‘/java_one/udf’ as (name:chararray); firstnames = foreachraw_datageneratemyudf.FirstName (name); storefirstnamesinto ‘/java_one/udf_output’; public class FirstNameextendsEvalFunc<String>{ @Override public String exec(Tuple input) throwsIOException { String name=input.get(0).toString(); …. returnfirstname; } }

16. What Storage Pig Supports HDFS Plain Text Binary format Customized format (XML, JSON, Protobuf, Thrift…) RDBMS(DBStorage) Cassandra (CassandraStorage) HBase(HBaseStorage)

17. What fields can Pig be applied Data Analysis Text Processing ETL Machine Learning

18. Who’s using Pig More: http://wiki.apache.org/pig/PoweredBy

19. References http://pig.apache.org (Pig official site) http://hadoop.apache.org (Hadoop official site) https://github.com/zjffdu/RAF-PIG (Rich API for Pig)

20. Demo

21. Thank you ! Q&A