DiffTool (difftool.org) is a UI based data comparison tool that can be used across data sources like RDBMS, Hadoop(Hive) or text files (CSV & JSON). DiffTool Compares datasets based on key(s), and some of the features include: controlling data volume using custom filters, Transforming columns using SQL expressions, The ability to scale using distributed architecture, And analyzing results intuitively with rich visualization.

1. Big Data Validation made
easy

2. • Validate data for the number one use case in big data
currently i.e. ETL offloading.
• Validate data to implement existing Process
improvements, optimizations and enhancements.
• Validate data loads/transformations in the Data Pipeline.
Why DiffTool ?

3. Common “Data Comparison”
Requirements ?
• Gather stats to analyze the differences.
• Compare a random sample instead of the full data.
• Apply Transformations before comparing.

4. Demo

5. How to use DiffTool ?
• Using the UI
• Integrate simple Rest API into the build process

6. Technical Challenges
• Automate testing with various data sources Hive,
JDBC(popular databases), CSV, Parquet, ORC, - Thanks
to Docker
• Even 30-35 seconds wait is not acceptable for interactive
applications.

7. Feedback