The document discusses Salesforce's plans to enhance its streaming and enterprise messaging capabilities. It describes the limitations of the current streaming API and previews new "durable streaming" functionality that will allow events to be replayed. It also outlines Salesforce's vision for an "Enterprise Messaging" platform using an event-driven architecture and "Conduit" API to reliably deliver data changes in real time or from a replay log. A roadmap is provided showing phased rollout of these capabilities through 2016.
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
Durable Streaming and Enterprise Messaging
1. Durable Streaming and Enterprise Messaging
Jay Hurst
Director, Product Management
jhurst@salesforce.com
@extraidea
John Brock
SMTS – Enterprise Messaging
jbrock@salesforce.com
@_ johnbrock
2. Safe Harbor
Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed
or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-
looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any
statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new,
planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new
functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our
operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any
litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our
relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our
service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger
enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our
annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These
documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our
Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available
and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features
that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
3. Jay Hurst
Director - Product Management, Salesforce
jhurst@salesforce.com
@extraidea
4. John Brock
SMTS – Enterprise Messaging
jbrock@salesforce.com
@_ johnbrock
5. Streaming Data from Salesforce
Many customers are facing a set of challenges around pushing
changes out of Salesforce.
Raise your hand if you:
• Have used the Classic or Generic Streaming API in Salesforce
• Have tried to build synchronization or change detection based on
streaming, but are having issues
• Have a need for a reliable, durable flow of data in and out of
Salesforce
7. Enterprise Messaging
The goal is to support the concept of an Event as a first class object
• An event is an immutable, time-stamped set of values.
• Can represent multiple values
• Actions - "A report was generated", "An order has been placed"
• Measurements – "API limits are at 50%", "1MM records have been created"
• State Changes - "Account 0010000023234 field Name was updated to 'Acme, Inc.'"
• Allow a large number of events to flow through the platform
• Scale to millions of events per minute per pod
• API first design to support:
• Publishing events to a message channel
• Consuming events from a message channel
• Provide ability to replay events from a defined time-window
8. Where are we today?
Classic Streaming API
Based on the Bayeux/CometD protocol
• The client will open up a long polling connection to the Salesforce server and subscribe to a Push Topic
• The client must regularly reconnect to keep the session alive
• When a record meets the query, we send an event down the connection to the client
• The client then reconnects to the long polling connection and waits for the next message
Allows you to define a query and have Salesforce send an event to your listener whenever a record
is modified and meets that query
• The messages are sent in near real time
• The Streaming API is not loss-less
• If the client is not connected, the message will not be received, and cannot be replayed
9. Where are we going tomorrow (or shortly thereafter)?
Durable Streaming API
The first implementation of Enterprise Messaging will be to enhance the Streaming API
• Provide the ability for replay of Generic Streaming API events (Durable Streaming)
• Existing Streaming Channel pushes will generate the Conduit events
• Additional Event ID will be provided in message
• Event ID will be atomic and ever-increasing
• A new replay method will be available that will replay all events from a provided ID
• The replay will re-deliver all messages that are after the provided ID
• Event replay will be available for up to 24 hours
• Protocol (Bayeux/CometD) will remain and require changes only to take advantage of replay
• Clients will need to store and provide the Event ID
• If no event ID provided in subscribe, then the events will be delivered from the tip of the queue
• This will result in the existing behavior available today
17. Enterprise Messaging – Event Time Window
The Event Time Window allows for events to be
added in time order to the channel
• Events are added through a producer API and
retrieved from the channel via the consumer API
• The "window" will slide with time, and any events
within the window can be replayed
• Events are ordered
• Replay events from anywhere in event log window
18. Event Architecture – Conduit API
Conduit API Allows for Producing and Consuming Events
• Simple publisher and consumer API
19. Event Architecture – Conduit SPIs
Conduit SPI Allows a common API for multiple Time Ordered
Event Log implementations
• Conduit SPI interface allows multiple concrete event log
implementations
• Kafka, SQL, HBase, In Memory, etc...
• Provides a common API across all providers
• The SPIs will also allow for composing event logs into different SLAs
20. Streaming API v2
How do Conduit and Streaming v2 Interact
Runs as an embedded or stand-alone service
Client interface via CometD
Event processing pipeline
Conduit - SPI
• Use different data stores
• Plug and play
• Holds client information
• Optimizes event replay
• Plus much more!
24. Enterprise Messaging – Roadmap (Safe Harbor J)
Winter '16
• Durable Generic Streaming (Limited Pilot)
Spring '16
• Durable Generic Streaming GA
• Durable Classic Streaming Pilot
Summer '16
• Durable Classic Streaming GA
• Kafka-Backed Queues
• Conduit Producer API public implementations for Apex
• Conduit Consumer API public implementations for Apex
• Conduit API implementations for Workflow/Process Builder
25. What did we Learn?
Existing Features that begin to support messaging
• Classic Streaming API
• Generic Streaming API
Information on Durable Streaming
Demo of Durable Streaming
• Showed an example of Generic Durability
• Showed an example of Classic Durability
Roadmap for Event Messaging