Overview of Apache Taverna, an open source workflow system. See http://taverna.incubator.apache.org/

1. Apache Taverna
NERSC Workflow Day, Berkeley Lab, California 2015-02-20
http://taverna.incubator.apache.org/
Stian Soiland-Reyes
@soilandreyes
stain@apache.org
http://orcid.org/0000-0001-9842-9718
Donal Fellows
@donalfellows
dkf@apache.org
http://orcid.org/0000-0002-9091-5938
This work is licensed under a Creative Commons
Attribution 3.0 Unported License.

2. TavernaWorkflow Ecosystem
• Workflow Language — SCUFL2 (and t2flow)
• Workflow Engine —Taverna
• Used in…
– Taverna Command LineTool
– Taverna Server
– TavernaWorkbench
• Allied services
– myExperiment, workflow repository
– Service Catalographer, service catalog software
• Instantiated as BioCatalogue, BiodiversityCatalogue, …
NERSC Workflow Day 2

3. UI Plugins
Map of theTaverna Ecosystem
UI Plugins
Taverna
Workbench UI Plugins
Taverna
Command
Line Tool
UI PluginsTaverna APIs
UI PluginsUI Ps
Taverna
Engine
Activity
Plugins
Taverna
Core
Taverna
Server
UI
Plugins
Taverna
Player
Rubyclient
REST API
SOAP API
UI Plugins
Taverna
Online
UI Plugins
Taverna
Lite
Components Other
Servers
Workflow
Repository
Service
Catalogs
many
services…
Application-Specific Portals
3

4. Taverna In Use
Users, ScientificAreas, Projects
NERSC Workflow Day 4

5. Taverna UsersWorldwide
NERSC Workflow Day 5

6. Taverna Uses — Scientific Areas
• Biodiversity — BioVeL project
• Digital Preservation — SCAPE project
• Astronomy — AstroTaverna product
• SolarWind Physics — HELIO project
• In silico Medicine —VPH-Share project
NERSC Workflow Day 6

7. Biodiversity: BioVeL
• Virtual e-Laboratory for
Biodiversity
– Service and knowledge commons
– Supporting biodiversity research
– Integrating with third-party
applications
• For example, iPython Notebook
• Portal for running production-
grade workflows on users’ data
– Powered byTaverna Server
– Integration with major biodiversity
databases
– Interaction support made to
support
NERSC Workflow Day 7

8. Digital Preservation:
SCAPE
• Automated petabyte-scale
digital collection maintenance
– Century of scanned
newspapers
– Whole national radio/TV
output
– MajorWeb archives
• Processing engine powered by
Taverna
– Lift simple workflows to work
at collection level
– Metadata management
– Semantic annotations and
components for guided
workflow construction
NERSC Workflow Day 8

9. Astronomy:
AstroTaverna
• Taverna plugin: IVOA (Virtual Observatory)
– Astronomy data services and tools
• Example workflow:
– List of galaxy names → Look upVO
properties → Find similar/near galaxies →
Add bibliography
• VOTable support (select/merge/split/..)
– Later adapted by bioinformatics community
• Projects: CANUBE, Wf4Ever,VAMDC, ER-
Flow
• TavernaWorkbench used on the desktop:
– IVOA service registry user interface
– Integrated with standalone astronomy tools
(SAMPS protocol): Aladin,TOPCAT
NERSC Workflow Day 9

10. Astrophysics: HELIO
• Virtual laboratory for
SolarWind Science
– Observation catalogs
– Processing
– Data integration platform
• Taverna is workflow glue
– Taverna Server created to
support
– Workflows manage
catalog access
– Workflows manage data
processing
NERSC Workflow Day 10

11. Medicine and Physiology:
VPH-Share
• Platform for computer-aided
medicine
– Support for diagnosis and
treatment prognosis
• Osteoarthritis, Dementia, Liver
disease, Cardiovascular disease
– Driven by specially-configured
cloud instances
• Taverna is control and data
management layer
– Coordinates processing within
cloud instances
– User communication with
cloud instances viaTaverna
interactions
• Including complex 3D tasks
NERSC Workflow Day 11

12. Inside theTaverna Ecosystem
Introduction to theTavernaWorkflow Language and its Executors
NERSC Workflow Day 12

13. The Basics of aTavernaWorkflow
Input Ports (data in)
SOAP processor (web service call)
XML handling processors
Data Links (connect processors)
Output Ports (data out)
13
Get concept suggestions from term
Eelke van der Horst
http://www.myexperiment.org/workflows/4590.html
NERSC Workflow Day

14. TavernaWorkflows
• Describe how data flows between processing nodes
– Control dependencies also supported
• Processing service nodes of various kinds
– Invoke programs (local or on cluster or grid or …)
– Call services (SOAP or REST)
– Read from and write to databases
– Transfer data
– Interact with the user
• Built-in parallelism and iteration
– Processes lists of data in parallel
• Large data usually handled by reference
– Avoids having to transfer it where not necessary
NERSC Workflow Day 14

15. TavernaWorkflows can get
complex…
NERSC Workflow Day 15
BioVeL Population Model Construction and Analysis
Maria Paula Balcázar-Vargas, Jonathan Giddy and Gerard Oostermeijer
http://www.myexperiment.org/workflows/3684.html

16. Managing Workflow Complexity
• Subworkflows
– Put smaller workflows within larger ones
– Like using a user-defined function in a
programming language
– Can hide contents of subworkflow
• Components
– “Black box” (but implemented with subworkflow)
– Semantically-annotated; described behaviour
– Like using a library in a programming language
NERSC Workflow Day 16

17. Taverna Engine
• Executes (“enacts”)Taverna Workflows
• Pushes data through system in parallel
– Subject to limits described in workflow
• Processor nodes invoked when their data
becomes available
– Turn inputs into outputs
• Captures detailed trace of what happened
(“provenance”)
– FollowsW3C PROV specification
NERSC Workflow Day 17

18. TavernaCommand LineTool
• Simple wrapper roundTavernaWorkflow
Engine
• Inputs as simple files
• Outputs as directory structure
• Provenance packaged in Research Object
– ZIP Archive
– Inputs, Outputs, Intermediate values
– Workflow, Provenance, Overall metadata
NERSC Workflow Day 18

19. Taverna Server
• ExtendsWorkflow Engine to work for
multiple simultaneous users
– Isolates workflows from each other
– Allows asynchronous usage
– Manages resources
– Clients can be in any language, not just Java
• Designed to sit behind a Portal
– User interfaces are domain-specific
NERSC Workflow Day 19

20. Taverna ServerArchitecture
20
Tomcat Container
+ CXF Framework
Taverna Server
Webapp
Common System
Model
PerUserFileManager
Web
Portal
Ruby
Client
Per-RunTavernaWorkflowEngine
Processing
Service
Catalog
Services
Storage
Services
TavernaWorkbench
(forthcoming)
Deployment
Host
Common
Management
Model
Selected
Notification
Endpoints
Management
Interface
(separate auth)
NERSC Workflow Day

21. TavernaWorkbench
• IDE forTaverna
Workflows
• Design
workflows
• Run workflows
• Analyze
workflows
• Access
workflow
repository
NERSC Workflow Day 21

22. Taverna Online
Web IDE forTaverna
NERSC Workflow Day 22

23. The Future ofTaverna
ApacheTaverna and Future Releases
NERSC Workflow Day 23

24. • Non-profit organization, forming a
community of open-source software
projects.
• Strong emphasis on openness, collaboration
and a consensus-based development
process.
• Examples:
– Apache HTTP Server,Tomcat, Maven, Hadoop,
OpenOffice, Subversion
NERSC Workflow Day 24

25. Why ApacheTaverna?
• Open development: Everything on mailing
list
• Engagement: Encourage developer
involvement – not just making plugins
• Independence:ApacheTaverna is an
independent project
– Not a “Manchester thing”
• Shared ownership: equal participation
• Sustainability: self-managed community
NERSC Workflow Day 25

26. Apache Incubator
Gradually becoming an Apache project
• Intellectual Property assigned to ASF
– License changed to Apache License 2.0
• Infrastructure change – everything at
*.apache.org
• Community building – growing developer
base
• Mentoring on the “ApacheWay” by
volunteers from other Apache projects
NERSC Workflow Day 26

27. Taverna Releases
• Current stable release: Taverna 2.5
– Command Line (2.5.1), Server (2.5.4), Workbench (2.5.1)
• http://www.taverna.org.uk/download/
• Taverna 3 Release plan:
– ApacheTaverna Language
• API for workflow definitions
– ApacheTaverna Engine & Command Line
• Can also run workflows fromTaverna 2Workbench
– ApacheTaverna Server
– ApacheTavernaWorkbench
NERSC Workflow Day 27

28. TryTaverna!
• GetTaverna:
– http://taverna.org.uk/download/
• Documentation:
– http://www.taverna.org.uk/documentation/taver
na-2-x/
• Code:
– http://taverna.incubator.apache.org/code/
• Getting involved:
– http://taverna.incubator.apache.org/community/
NERSC Workflow Day 28