OC Big Data Monthly Meetup #6 - Session 1 - IBM

BigInsights — Technical Overview
OC Big Data Meetup
© 2014 IBM Corporation
Information Management
Lynn Hedegard
Technical Sales Specialist
West Region
15th of October, 2014
Real-Time CRM in the Social World (Meet Lisa)
Telco Customer Profile
Retailer Customer Profile
Lisa registers with
Retailer. Gives
Retailer & Telco
permissions to
“Opt In”
Intelligent Advisor
Platform
The “Intelligent
Advisor” platform
processes Lisa’s recent
on-line activity and
constructs a targeted
offer based on recent
behavior AND internal
marketing strategy
Product Catalog
Lisa receives a
message with an
offer reminding her
to stop by if she’s
in the area
While walking
past the store, Lisa
receives a promo
code for a product
we think she might
like
© 2014 IBM Corporation 3
Retailer Fan Page
Lisa “follows” a
friend’s post on
FB and clicks the
“Like” button on
an Item she likes
Lisa uses promo
code to purchase
product from offer
AND a few more
items that go with
the outfit ☺
IBM Big Data & Analytics

Problem Statement — Complex Environment
• The Local Environment is Complex:
• A single large retail store (1.5 million SKUs)
• Large manufacturing floor (~6 million parts)
• Vegas Casino (20 million card carrying customers)
• The Global Environment is Complex:
• The number of variables affecting business performance is huge.
• US citizens (source: google population)
• 300+ Million total
• (21M+ teenagers) + (40M+ in their 20’s) (that’s a lot of calls & text messages!)
• The interrelationships between these variables is very complex (e.g., N2 problem)
• Multiple customer touch points
• Multiple suppliers & distribution methods
• Market forces (cost of raw goods & services, pricing dynamics, supply/demand)
• Working Premise: Few people in the enterprise can make “good”
Operational Decisions — consistently & quickly
• Few people can “see” all the necessary data.
• Few people can “analyze” all the necessary data.
• Few people understand all the inter-relationships
Businesses can no
longer tolerate
inconsistent Business
Processes
between business variables.
IBM’s Big Data Reference Architecture — High Level
Big Data Reference Architecture
BI and
Reporting
Analytic Applications
Exploration
Visualization
Functional
App
Industry
App
Predictive
Analytics
Content
Analytics
IBM Big Data Platform
Systems
Management
Application
Development
Visualization
& Discovery
Accelerators
Hadoop
System
Stream
Computing
Data
Warehouse
Information Integration & Governance
An Enterprise Eco-System for Big Data
• Integration of all classes of Data Repositories
• Complete set of reusable analysis
components (i,e., Accelerators)
• Apply analysis to data in its native form (i.e.
in the repository)
• Data Exploration of data from myriad
repositories using a common interface
• Powerful Visualization Tools
• Eclipse based Development Environments
(e.g. DW, Hadoop, & Streaming Data)
• Management
• Enterprise Class Security & Data
Governance
•Workload Optimization
•Workload Scheduling
• Dynamic Reconfiguration
• Advanced Analytics

Application Accelerators Improve Time to Value
Finance Analytics
Streaming options trading
Insurance and banking DW
models
Telecommunications
CDR streaming analytics
Deep Customer Event Analytics
Text Analytics
Natural Language Processing
Multi-Language Support
Domain Specific
Social Data Analytics
Sentiment Analytics, Intent to
purchase
Machine Data Analytics
Operational data including logs
for operations efficiency
IBM’s Big Data / Analytics Reference Architecture
Streaming Computing
Real-Time Analytical Processing
Analytical Sources
Enhanced
Applications
Actionable
Insight
Decision
Management
Discovery
& Exploration
Modeling &
Predictive
Analytics
Analysis &
Reporting
Planning &
Forecasting
Content
Integrated
Data
Warehouse
Enterprise
Warehouse
Landing
Exploration &
Archive
Big Data
Repository
Deep
Analytics &
Modeling
Analytical
Appliances
Interactive
Analysis &
Reporting
Data
Marts
Shared Operational Information Analytics
Activity
Hub
Metadata
Catalog
Customer
Experience
New
Business
Model
Financial
Performance
Risk
Operations
& Fraud
IT
Economics
Governance
Event Detection and Action
Security & Business Continuity Management
Platforms
Data
Integration
Data Quality,
Xfrm & Load
Master &
Reference
Content
Hub
Data
Sources
New
Data Sources
Machine &
Sensor Data
Image &
Video
Enterprise
Content Data
Social Data
Internet
Data
Traditional
Data Sources
Third-Party
Data
Transactional
Data
Application
Data
Data Acquisition & Application Access

Merging the Traditional and Big Data Approaches
Big Data Approach
Iterative & Exploratory Analysis
IT Group
Delivers a platform to
enable creative
discovery
Business Users &
Data Scientists
Explore what questions
could be asked
Brand sentiment
Product strategy
Maximum asset utilization
Traditional Approach
Structured & Repeatable Analysis
Business Users
Determine what
question to ask
IT Group
Structures the
data to answer
that question
Monthly sales reports
Profitability analysis
Customer surveys
BigInsights
BigInsights

BigInsights: Value Beyond Open Source
Key differentiators
• Built-in text analytics
• Enterprise software integration
• SQL support
• Spreadsheet-style analysis
• Integrated installation of supported open
source and other components
• Web Console for admin and application access
• Platform enrichment: additional security,
performance features, GPFS (alternative file
system), . . .
• World-class support
• Full open source compatibility
Business benefits
• Quicker time-to-value due to IBM technology
and support
• Reduced operational risk
• Enhanced business knowledge with flexible
analytical platform
• Leverages and complements existing software
IBM’s
Value
Add
Open
Source
Components
Visualization & Exploration
Development Tool
Advanced Engines
Connectors
Workload Optimization
Administration & Security
IBM-certified
Apache Hadoop
and related projects
BigSheets
• Model “big data” collected from
various sources in spreadsheet-like
structures
• Filter and enrich content with
built-in functions
• Combine data in different
workbooks
• Visualize results through
spreadsheets, charts
• Export data into common formats
(if desired)
No programming knowledge needed!

Social Data Analytics Accelerator
Provides the ability to analyze large volumes of various
types of social media data with real-time processing
Social Data Analytics
Why should you care?
It enables clients to easily obtain insights necessary for:
–Effective/targeted Marketing Campaigns
–Timely product/marketing decisions
–Gaining competitive Intelligence
–Building customer retention and new customer acquisition
programs
Example Application : Movie Campaign Effectiveness
• Large Movie Studio wants to understand reaction of movie commercials around events (e.g., SuperBowl)
• Over 30 Million social media consumer profiles built and used in the analysis
• Real-time summary of insights correlated with the airing of the commercial
What does it do?
. . .
Big SQL
• Standard SQL syntax and data types
• Joins, unions, aggregates . . .
• VARCHAR, decimal, TIMESTAMP, . . .
• JDBC/ODBC drivers
• Prepared statements
• Cancel support
• Database metadata API support
• Secure socket connections (SSL)
• Optimization
• MapReduce parallelism
or…
• “Local” access for low-latency queries
• Varied storage mechanisms appropriate
for Hadoop ecosystem
• Integration
• Eclipse tools
• DB2, Netezza, Teradata (via LOAD)
• Cognos Business Intelligence
IBM Big Data Analytics

R Clients
Scalable
Statistic
s Engine
“End-to-end integration of R into IBM BigInsights”
Pull data
(summaries) to
R client
Data Sources
R Packages
1
2
3
Embedded R Execution
R Packages
Or, push R
functions
right on the
data
Big R
1. Explore, visualize, transform,
and model big data using
familiar R syntax and paradigm
2. Scale out R
• Partitioning of large data
(“divide”)
• Parallel cluster execution of
pushed down R code (“conquer”)
• All of this from within the R
environment (Jaql, Map/Reduce
are hidden from you
• Almost any R package can run in
this environment
3. Scalable machine learning
• A scalable statistics engine that
provides canned algorithms, and
an ability to author new ones, all
via R
• Mature System: “System T” text analytics engine embedded in IBM products
• Found in Lotus Notes, IBM e-discovery Analyzer, CCI, InfoSphere Warehouse,+++
• Almost a decade since initial release
• Extensible: User can customize Text Analytics Engine
• Toolkit: BigInsights Text Analytic Toolkit provides
• Developer tools
• Easy to use text analytics language
• Set of extractors for fast adoption
• Multilingual support, including support for DBCS languages
• AQL: BigInsights includes Annotator Query Language (AQL): SQL-like!
• Fully declarative text analytics language
• No “black boxes” or modules that can’t be customized.
• Tooling for easy customization because you are abstracted from the programmatic
• Competing solutions make use of locked up black-box modules that cannot be
customized, which restricts flexibility and are difficult to optimize for performance
Text Analytics Toolkit
details

BigInsights Enterprise Edition
IBM InfoSphere BigInsights
IIBBMM VVaalluuee AAdddd
OOppeenn SSoouurrccee
Analytics
of Data in
Motion
Machine
Learning
SSttrreeaammss
R
CCooggnnooss BBII
Data
Integration
BBooaarrddRReeaaddeerr
WWeebb CCrraawwlleerr
DDBB IImmppoorrtt
DDBB EExxppoorrtt
SSqqoooopp
FFlluummee
DDaattaaSSttaaggee
System
Mgmt
Dynamic
Configuration
Monitor
Workflow
Deploy
Applications
Flexible
Scheduler
GGuuaarrddiiuumm
DDaattaaEExxpplloorreerr
JJDDBBCC
NNeetteezzzzaa
DDBB22
Accelerator for
Machine Data
Analysis
BBiigg SSQQLL
PPiigg
Visualization and Discovery
Dashboards And
Visualizations
Deep Analytics
Accelerator for
Social Data
Analysis
IInnddeexxiinngg
JJaaqqll
BBiiggSShheeeettss
Text Processing
Engine Library
Text
Compression
Distributed
File Copy
ZZoooo KKeeeeppeerr HHCCaattaalloogg
HHbbaassee
Adaptive
Map Reduce
GGPPFFSS--FFPPOO
Integrated
Installer
Enhanced
Security
LLuucceennee
HHiivvee
MMaapp RReedduuccee
HHDDFFSS
OOoozziiee
Infrastructure
Parallel Processing
Engines
File Systems
Web
Console
Web Console

Welcome Tab: Your Starting Point
Tasks: Where and how to begin performing
common administrative or analytical tasks
Quick links to common functions
Learn more through external Web resources
Overview of Web Console Capabilities
• Manage BigInsights
• Inspect /monitor system
health
• Add / drop nodes
• Start / stop services
• Launch / monitor jobs
• Explore / modify file system
• Create custom dashboards
• . . .
• Launch applications
• Spreadsheet-like analysis tool
• Pre-built applications (IBM
supplied or user developed)
• Publish applications
• Monitor cluster, applications,
data, etc.

BigInsights Applications Catalog (Web Console)
• Browse available applications
• Manage and deploy applications (administrators only)
• Execute (or schedule execution of ) a deployed application
• Monitor job (application) status
• Link or chain applications for sequential execution
BigSheets
BigSheets

Why Did IBM Develop BigSheets?
A Browser-Based Analytics Tool For Business Users.
Business users need an intuitive non-technical
enterprise and Web data promotes new
business intelligence.
How can BigSheets help?
Spreadsheet-like interface enables
business users to gather and analyze
data easily.
Built-in “readers” can work with data in
several common formats (JSON arrays,
CSV, TSV, Web crawler output, . . . )
Users can combine and explore various
types of data to identify “hidden”
insights.
Why BigSheets?
approach for analyzing Big
Data.
Translating untapped data into
actionable business insights is a
common requirement.
Visualizing and drilling down into
• Ensure BigInsights Enterprise is running
Launch the Web console with URL http://host:port or
http://host:port/data/html/index.html
• Follow on-screen Task prompt or click on the BigSheets tab
Accessing BigSheets

BigSQL
BigSQL
. . .
Big SQL
• Standard SQL syntax and data types
• Joins, unions, aggregates . . .
• VARCHAR, decimal, TIMESTAMP, . . .
• JDBC/ODBC drivers
• Prepared statements
• Cancel support
• Database metadata API support
• Secure socket connections (SSL)
• Optimization
• MapReduce parallelism
or…
• “Local” access for low-latency queries
• Varied storage mechanisms appropriate
for Hadoop ecosystem
• Integration
• Eclipse tools
• DB2, Netezza, Teradata (via LOAD)
• Cognos Business Intelligence

Analyst Comments Regarding BigInsights
Analysts
Comments
BigInsights
The Forrester Wave™ - Hadoop Solutions Q1 2014
• Hadoop momentum is unstoppable
• It’s open source roots grow deeply and wildly into the enterprise. Its
refreshingly unique approach is transforming how companies process,
analyze and share big data
• Hadoop vendors face a cut-throat market
• The buying cycle is on the upswing, and Hadoop vendors know it.
Pure-play upstarts must capture market share quickly to make
investors happy; stalwart enterprise vendors need to avoid being
disintermediated; cloud vendors must make solutions cheaper.
• Hadoop is open, but vendors add differentiated features
• Hadoop is an Apache open-source project that anyone can download
for free. Vendors all support, extend and augment Apache Hadoop and
add differentiated features.

The Forrester Wave™ - Hadoop Solutions Q1 2014
Distributed computing platforms
not new to IBM
Advanced analytic tools
Global presence
Deep implementation services
Complete big data solution
Compelling roadmap
http://www.forrester.com/pimages/
rws/reprints/document/112461/oid/
1-PBE69P
The Forrester Wave is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave are trademarks of Forrester Research, Inc. The Forrester Wave
is a graphical representation of Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores, weightings, and comments.
Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect
judgment at the time and are subject to change.
InfoSphere BigInsights 3.0 – Worth a look!
Cloudera CDH5 HortonWorks HDP
2.1
MAP-R 3.1 Pivotal HD 2.0 Amazon Elastic
MapReduce
Capability IBM InfoSphere
BigInsights
Open Source Hadoop Components – PIG, Hive,
HBASE, Oozie, Avro etc ..
Big SQL – Rich, high-performance ANSI compliant
SQL on Hadoop
BigSheets – Spreadsheet style visualization tool for
business users
Text Analytics Accelerator – Simplified development
for text analytics (AQL)
Social Data Accelerator – Developer toolkit for social
media applications
Machine Data Accelerator – Developer toolkit for
building log analytics apps
Adaptive MapReduce– High-performance MR with
recoverable jobs
GPFS-FPO –POSIX, HDFS compatible file system
with enterprise features
IDE – ECLIPSE based integrated development
environment
Big R – full R language integration
Watson Explorer – search and index all data within
BigInsights

BigInsights On-Line Resources
BigInsights
On-Line
Resources
InfoSphere BigInsights 3.0 – QuickStart Edition
Free, no limit, non-production version of BigInsights
Big SQL, BigSheets, Text Analytics, Big R, management
console, development tools
Tutorials and education
Installable images or VM
• Single or multi-node clusters
• Over 53,000 downloads to date
http://IBM.co/QuickStart
http://www.ibm.com/developerworks/downloads/im/biginsightsquick/
http://www.ibm.com/software/data/infosphere/biginsights/quick-start/

External Hadoop Resource
• IBM.com/Hadoop
• Messaging aimed at Hadoop and open
source enthusiasts
• Extensive resources, links to other IBM
Big Data sites
External BigInsights Resource
• Developer.IBM.com/Hadoop
• Referred to as “Hadoop.dev”
• Site and resources tailored to technical
buyers and evaluators
Web Resources
BigSQL Value Add To Hadoop
• SQL on Hadoop without Compromise
• http://public.dhe.ibm.com/common/ssi/ecm/en/sww14019usen/SW
• New Big SQL Datasheet – Covers key value propositions
differentiation + HIVE 0.12 vs. Big SQL 3.0 benchmarks
(20x performance advantage on average)
• Key Big SQL advantages
• Enterprise features
• Compatibility
• Performance
• Federation
W14019USEN.PDF

IBM BigInsights on Cloud
• Enterprise Hadoop as a Service
Focus on analyzing data using BigInsights features including Big
SQL, BigSheets and text analytics rather than managing
infrastructure
• High performance hardware environment
Hadoop specific reference architecture implemented on dedicated
bare metal nodes
• Auto-provision BigInsights on nodes through a simple web
interface
InfoSphere BigInsights
Thank You

OC Big Data Monthly Meetup #6 - Session 1 - IBM

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to OC Big Data Monthly Meetup #6 - Session 1 - IBM

Similar to OC Big Data Monthly Meetup #6 - Session 1 - IBM (20)

More from Big Data Joe™ Rossi

More from Big Data Joe™ Rossi (11)

Recently uploaded

Recently uploaded (20)

OC Big Data Monthly Meetup #6 - Session 1 - IBM