Today’s leading digital businesses, such as Amazon, Google and Facebook, cite experimentation as the key to constantly improving their digital experiences. Yet many companies are still in the early stages of adopting a test and learn mindset when it comes to innovating on their own experiences. In this webinar, hear how HP has built a global center of excellence, structured their experimentation teams and meetings, and focused on technology, governance, and strategy on their way to building a culture of experimentation. You'll learn: - How we structured our Center Of Excellence, and why it's working for us at a global scale - Our emphasis on Data and Strategy, ensuring we make effective business decisions - How we’ve interlocked our technologies to give us insights and the edge on our Experiments - Our Partnership and Growth Story, on our way to grow experiments by 3x worldwide YoY

1. 1
Building A Culture Of
Experimentation At HP Inc.
Hear how HP has built a global center of excellence, structured their experimentation
teams, and focused on technology, governance, and strategy on their way to building
a culture of experimentation.

2. 2
Introductions
Neville Davey
Global Experimentation Lead
David Carlile
Customer Success Manager
Anmeen Leong
Americas Experimentation Lead

3. 3
Housekeeping
I. We are recording this webinar
II. You’ll receive the recording and slides
III. We’ll answer all questions at the end

4. 4
www.optimizely.com/digital-change-agents

5. 5
What do we do at Optimizely?

6. Revenue, share of wallet, funnel conversion, risk
mitigation, ops efficiency
What
We Do Next gen “Test and Learn” platform for enterprise-
wide digital experience optimization
Over 1B impressions daily
Replaces Digital Guesswork
With Evidence-based Optimization
Digital Experimentation
SaaS Platform
Apply the scientific method to “at-scale”
business decision making.
20X increase in Yield

7. What
We Enable
1500+
+51%
+19%
+28%
Improved subscription
acquisition
Users Scale and agility
of experiments; users
Improved Driver
Acquisition
increase in cross-sell
performance

8. Spent with Digital Media
19 hr
week
2018
Digital-Centric Customer
Spent with Digital Media
41
$396m
Mobile commerce Mobile commerce
$1 8t.
2008
Digital-Convenient Customer
hr
week
Your customer has fundamentally
changed how they engage

9. Wealthfront managed $1B
in assets 3x faster than Schwab
RETAIL
Amazon makes up 50 percent of the U.S.
e-commerce market. (Walmart 4%)
TRAVEL
Airbnb has more room listings than
top 5 hotel brands combined
HIGH TECH
Teams using Slack see an average
reduction in email volume of 48.6%
MEDIA
33 million people cut their cords in 2018
The Digital
Experience
Economy
Winning in digital
is a matter of life
or death in many
industries
How do win?
FINANCE

10. Experimentation: Testing variations of digital experiences
with real users and deploying what wins
+21%
-5%

11. Frontend UI Backend Business
Logic & Data
The anatomy of an experience
i.e. navigation,
search location
& visual treatment
Copy
Images &
Colors
Layout Search algorithms
Personalize content
based on previous
behavior
Recommendations
Make your headlines
more personal

12. Web / SPA Trusted results
Adoption &
scale
Integration with
key systems
Mobile
OTT
Governance
Process change,
implementation
support
Security &
compliance
Frontend UI
Optimizing these
experiences is
extremely challenging
Backend
Business Logic
& Data
Personalization
IOT / Voice
Marketing
Product &
Engineering
Support

13. Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
Marketing
Product &
Engineering
Support
The Optimizely Solution
Consumer grade
usability for
adoption &
velocity
Program
Management
Purpose- Built
Statistics Engine
& Machine
Learning
Open Platform
Data &
Integrations
Security &
Compliance
OTT
Voice/IOT
Mobile
Web / SPA
Frontend and
Server-side
Experimentation
& Personalization
Best in class
Services &
Success teams
SUCCESS ACCELERATIONEXPERIENCE COVERAGEDEVICE COVERAGE TRUSTED RESULTS ENTERPRISE SCALE

14. We’re Proud to Work With Great Global Enterprises
24 of the Fortune 100 have chosen Optimizely to drive their digital experience

16. 16
CULTURE OF
EXPERIMENTATION
WINNING
Setting The Stage at HP
Where We Were Two Years Ago
Where We Are Today
Where We Want To Be

17. To improve Omnichannel experiences that delight our customers and
drive sales through data-driven decision making.
HP’s Experimentation Vision

18. 18
BUSINESS VALUE
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
2015
US B2C Store
Quick Wins
2016-17
B2C Geo Expansion
B2B, Marketing & Support Sites
CoE Program started
2018
APJ & EMEA Expansion
Segment based personalization
HP’s EXPERIMENTATION JOURNEY

19. FY18 Major Milestones
Formalized Testing Program
(Ideation sessions, workflow, reporting)
Created Experimentation CoE
(Best practice sharing, promote test culture)
Expanded to All Regions
(1 to 3 regions, 10+ countries)
More Stakeholders
(3 Region Stores, HP.com, B2B, Support,
Instant Ink)
Increased Campaign Volume
(2x increase Y/Y)
Increased Value to HP
(>3x revenue increase Y/Y)
FY19 Target: 3x Campaign Volume

20. I. Our emphasis on a strong end-to-end program management foundation.
II. How we’ve interlocked our technologies to give us insights and the edge on our
Experiments.
III. How we structured our Center Of Excellence, and why it's working for us at a global scale
to increase campaigns by 3x.
IV. Examples of how experimentation has brought value to HP.
What we want to share with you today:

21. 21
Individual
Champion
2015
US
Level 1
Challenges
• Low Activity
• Lack of Knowledge / Value
• Credibility

22. 22
Individual
Champion
2015 2018
COE
US Americas
Asia
Pacific
Europe
B2B
Enterprise
Customer
Support
Instant
Ink
Full Service (Testing Council Model)
Enable (COE Model)
Level 1 Level 3
?

23. 23
Individual
Champion
2015 2018
COE
US Americas
Asia
Pacific
Europe
B2B
Enterprise
Customer
Support
Instant
Ink
Full Service (Testing Council Model)
Enable (COE Model)
Program Management
ü Strategy
ü Communications
ü Resources & Value
Technology & Data
ü Tech Stack
ü Data
ü Code Quality
Level 1 Level 3
Worldwide Growth
ü Program Structure
ü Service Model
ü Team Structure

24. 24
Program Management
Strategy
Communications
Resources & Value

25. 25
Individual
Champion
2015 2018
COE
US Americas
Asia
Pacific
Europe
B2B
Enterprise
Customer
Support
Instant
Ink
Full Service (Testing Council Model)
Enable (COE Model)
Program Management
ü Strategy
ü Communications
ü Resources & Value
Technology & Data
ü Tech Stack
ü Data
ü Code Quality
Level 1 Level 3
Worldwide Growth
ü Program Structure
ü Service Model
ü Team Structure

26. 26
Strategy
• Based on business goals
• Ideate and prioritize with stakeholders
• Informed by analytics and customer feedback
KPIs
• Vary by stakeholder
• eCommerce: CR, RPV
• Call/Chat Center: Call volume
• Marketing: Engagement, CTR

27. 27
A/B test was aimed to increase global nav referrals
and measure how increasing traffic to the site would
influence other key performance indicators.
Other
referrers
Lead-gen
start rate
Lead-gen starts
Global nav referrals
3D page visits
Lead-gen
start rate
Completion % per
step
# of steps
Typed /
bookmarked
3D Printer
Revenue
Leads generatedLead quality
Test KPI
BU Strategic Goal
Test Change

28. Prioritization
Create a meeting structure for US Store
Covers test cycle across multiple teams
28
Print Meeting
MEETINGS
Ideation
Results Sharing
Alignment Meeting PC Meeting
Print Category Combined Category PC Category
Print Meeting PC MeetingAlignment Meeting
Alignment Meeting
Reporting Meeting
Create tickets in JIRA
4x Meetings
1x Print Weekly
1x PC Weekly
1x Alignment Weekly
1x Reporting Weekly
& Alignment
Create tickets in JIRACreate tickets in JIRA

29. List of Tools used to coordinate the program
29
Ideation
Collect and gather test ideas
from all over the organization
Prioritization / Roadmap Test Plans Results Sharing
Prioritize based on impact, level of
effort, and initiatives
Document and explain the key
information in for your experiment
Share test results and insights
broadly with HP organization
Optimizely Program Management

30. Experimentation Hub (intranet site) for Reporting & Resources
30

31. Email Newsletter & Yammer for Communications
31

32. Evangelizing to execs and broader company:
Program Dashboard (OptimizelyPM, Tableau), Internal Webinar
32

33. 33
• Strategy based on business goals
• Communicate and align with stakeholder
• Communicate broadly, show value
Bottom Line

34. 34
Technology & Data
Tech Stack
Data
Code Quality

35. 35
Individual
Champion
2015 2018
COE
US Americas
Asia
Pacific
Europe
B2B
Enterprise
Customer
Support
Instant
Ink
Full Service (Testing Council Model)
Enable (COE Model)
Program Management
ü Strategy
ü Communications
ü Resources & Value
Technology & Data
ü Tech Stack
ü Data
ü Code Quality
Level 1 Level 3
Worldwide Growth
ü Program Structure
ü Service Model
ü Team Structure

36. Tech stack enables quality code and data
36
Development Task Automation Workflow
Optimizely Custom Snippet (cross-domain/project integration)
Optimizely for Dynamic Websites (single page application)
Quantitative Data Qualitative Data

37. Analytics Tools in Action (integrated with Optimizely)
37
Breakdown experiment sales by
product category
User feedback and pain points to inform
experimentation roadmap
User behavior in experiment

38. 38
Development Task Automation Workflow
Mohammad Alabandi
Lead Experimentation Developer
HP
Coming Soon to
Features
• Build outside Optimizely using code editor of choice
• Lints and optimizes javascript files, browser
compatibility
• Direct publish to Optimizely (via API)
Benefits
ü Velocity: Development speed increased 40%
ü Quality: Code errors reduced 20%
ü Open source: Shared across HP stakeholders

39. 39
Worldwide Growth
Program Structure
Service Model
Team Structure

40. 40
Individual
Champion
2015 2018
COE
US Americas
Asia
Pacific
Europe
B2B
Enterprise
Customer
Support
Instant
Ink
Full Service (Testing Council Model)
Enable (COE Model)
Program Management
ü Strategy
ü Communications
ü Resources & Value
Technology & Data
ü Tech Stack
ü Data
ü Code Quality
Level 1 Level 3
Worldwide Growth
ü Program Structure
ü Service Model
ü Team Structure

41. 41
Individual
Single team and single site
experience
Small focus, simple tests, build
momentum
Program Manager with access
to Dev, Designer, Etc.
Center Of Excellence
Central support team for
business units (own individual
experiences)
Enabling and supporting
individual teams, and sharing
best practices
Central expert team (Including
PM, Dev, Designer, Analyst, Etc.)
Testing Council
Shared ownership of overall
experiences and prioritization
Broader digital initiatives
requiring coordination
Testing Council for Analytics, UX,
Etc.
Definition
Characteristics
Team
Common Types of Program Structure

42. 42
Testing Council
Individual Center Of Excellence
Common Types of Program Structure
Decentralized
Centralized
One Many
Culture/Ownership
Number of Teams

43. 43
Testing Council
Individual Center Of Excellence
Common Types of Program Structure
Decentralized
Centralized
One Many
Culture/Ownership
Number of Teams

44. 44
Testing Council
Individual Center Of Excellence
Common Types of Program Structure
Decentralized
Centralized
One Many
Culture/Ownership
Number of Teams

45. FULL SERVICE
Per Quarter fee
ENABLE ONLY
Per Quarter fee
FREE TRIAL
One quarter
AB Testing platform
Onboarding / training
Monthly Unique Visitor Allocation
Out-of-the-box reporting
Test bandwidth
Advice throughout testing process, best practices
Full definition & set-up (Web / Mobile dev, UX, Analytics)
Launch test (with adjustments)
Full analysis and custom reporting
Insights, recommendations & iteration advice
Experiment revision(s)
Program Governance
1 to 1 Sessions
Community Sessions (Office Hours)
Playbook on launching campaigns/Wiki Site Access
1m (Funded) Unlimited (Paid) Unlimited (Paid)
Up to 5 tests Unlimited Unlimited
-
-
-
-
-
-
-
-
-
-
-
Most Popular Package
HP Service Plans: Stakeholders pick what’s best for them
COE Model
Testing Council Model

46. 46
Experimentation = Value for HP
Revenue Growth
Reduce Customer Interaction Cost
Enable Operational Excellence
Risk Mitigation

47. 47
DIGITAL EXPERIENCE OPTIMIZATION
Mitigate RiskEnable Operational
Excellence
Deliver Measurable
Revenue Growth
Reduce Customer
Interaction Costs

48. 48
DIGITAL EXPERIENCE OPTIMIZATION
Deliver Measurable
Revenue Growth

49. Problem: Gaming customers not aware of gaming offers
49
Original (Standard Ribbon) Variation (Personalized Ribbon)
Results
• +28% RPV for Gaming PCs (+13% RPV overall).
• Explore other customer segments for simple personalization.
1 1
“Value Prop” storewide ribbon
Sale Page / Weekly Deals Page
“Value Prop” storewide ribbon
Omen Sale Page
• Value Prop ribbon has personalized (gaming) sale messaging
• CTA goes to Omen Sale Page
• Value Prop ribbon has standard sale messaging
• CTA goes to standard sale page (sale / weekly deals)1 1
Action: Personalize the storewide ribbon messaging and landing page for “gaming inclined” visitors.

50. 50
DIGITAL EXPERIENCE OPTIMIZATION
Reduce Customer
Interaction Costs

51. Problem: Customers are calling the sales phone number for support calls
51
Original (“Order Status” Text) Variation (“Sales” Text)
Results
• +35% sales calls, -86% order status calls.
• Customers still able to check order status online via web link.
• Order Status link in position3.
• Phone number now in position1, including the “Sales” text.
• Order Status link in position1.
Action: Move “order status” link to less prominent position, add “sales” text to clarify phone number purpose
1 2 1
1
2
1

52. 52
DIGITAL EXPERIENCE OPTIMIZATION
Mitigate Risk

53. Problem: Companywide “HPID” initiative may increase checkout friction
53
Results
• Net effect: No harm to sales.
• High-ticket item shoppers less impacted than low-ticket item shoppers.
Original (Standard Checkout) Variation 1 (HPID Checkout)
• Logging in from Cart page sends user directly to Checkout page • Logging in from Cart page, user will see an intermediate HPID
password page, before proceeding to Checkout page
Action: Added additional step/page (HPID) in checkout flow prior to formal rollout, to gauge impact on conversion & CX

54. 54
DIGITAL EXPERIENCE OPTIMIZATION
Enable Operational
Excellence

55. Problem: Black Friday Sale Page not optimized for speed and CX
55
Original (Classic Webpage) Variation (Single Page Application)
Results
• +15% Conversion. Single Page Application domain rolled out for Black Friday and Cyber Monday sale pages.
• Contributed to double-digit Y/Y sale growth.
• HP Pre-Black-Friday Sale Page UI (identical)
• Faster page load times (in theory).
• Single page application, server side rendering.
• HP Pre-Black-Friday Sale Page UI
Action: Developed new single page application (newer technology) with faster page speed

56. 56
In Summary…

57. 57
Individual
Champion
2015 2018
COE
US Americas
Asia
Pacific
Europe
B2B
Enterprise
Customer
Support
Instant
Ink
Full Service (Testing Council Model)
Enable (COE Model)
Program Management
ü Strategy
ü Communications
ü Resources & Value
Technology & Data
ü Tech Stack
ü Data
ü Code Quality
Level 1 Level 3
Worldwide Growth
ü Program Structure
ü Service Model
ü Team Structure

58. Looking Ahead
58
Complexity vs Velocity
• How to balance creating more complex tests vs more volume of tests?
• Bottom line: what mix will drive the most value?
Personalization
• Current: broad customer segmentation, account-based segmentation
• Near future: increase use of 3P data, more granular personalization
BU Expansion • Reach more business units, ramp existing stakeholders
Server-Side Testing • Evaluate Optimizely Full Stack platform

59. 59
Q & A
If you questions for HP please submit in the chat box now!