Modern Enterprise integration Strategies
•Download as PPTX, PDF•
16 likes•10,765 views
This document explores the history of the evolution of integration technologies in the enterprise from EAI to micro-services.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Modern Enterprise integration Strategies
Similar to Modern Enterprise integration Strategies (20)
More from Jesus Rodriguez
More from Jesus Rodriguez (20)
Recently uploaded
Recently uploaded (20)
Modern Enterprise integration Strategies
- 2. Agenda • About US • The enterprise integration landscape • Enterprise application integration servers • Enterprise service bus • Integration platform as a service • New trends in enterprise integration
- 4. Types of Integration Platforms Early 2000s Enterprise Application Integration Platforms 2005+ Enterprise Service Bus 2011+ Integration Platform as a Service 2014+ Microservices Architecture
- 6. Goals • Based on the traditional enterprise integration patterns architecture • Enable integration between different enterprise line of business systems • Facilitate long running processes • Support for B2B standards like EDI or Rosettanet
- 7. Technological Foundation • Orchestration engine • Business rules platform • Business activity monitoring • B2B interfaces
- 8. Strengths • Implementing long running processes • Supporting B2B processes • Rapid integration with line of business systems
- 9. Challenges • Expensive to implement • Lack of extensibility • Lack of support for modern architecture styles like microservices or cloud computing
- 10. Platforms • Microsoft BizTalk Server • Tibco Businessworks • Oracle SOA suite • IBM WebSphere Process Server • Software AG • …….
- 11. Enterprise Architect Standpoint • Implementing enterprise app integration server is an expensive endeavor both technologically and financially • Require on-premise infrastructure • Development using proprietary workflow and orchestration tools • Training • Challenge hiring talent
- 12. EAI Example: BizTalk Server
- 13. BizTalk Server Capabilities Management and Operations RFID Platform Business Rule Framework Business to Business Integration Business Activity Monitoring Messaging Orchestration Tools AppFabric Connect Integration
- 14. BizTalk Server Integration Scenario Suppliers Application 1 3 2 4 Inventory Application Enterprise Resource Planning (ERP) Application Microsoft BizTalk Server 2010 IT Pros and Developers Manage and Operate Design and Implement RFI 5 EDI Messaging Messaging Assign Supplier Policy Filter Tags Policy Re-stock Orders Orchestration Read Shipment RFID Tags Up-to-date KPIs in BAM Business Users Messaging
- 15. BizTalk Server Runtime Architecture www.devscope.net Receive Port Orchestration XML EDI or Flat File XML EDI or Flat File Send Port Send Adapter Send Pipeline MessageBox Mapping TO: NWTraders (Flat file format) FROM: Fabrikam (XML format) Mapping FROM: Contoso (Flat file format) TO: Fabrikam (XML format) Receive Location Receive Adapter Receive Pipeline
- 17. Goals • Provide integration capabilities in service oriented solutions • Enable a lighter, messaging-centric model to integrate services in a SOA architecture
- 18. Technological Foundation • Messaging channels • Services • Topics, queues • Routing rules
- 19. Strengths • Lightweight messaging between services • Abstract infrastructure capabilities such as routing and transformation in a SOA solution • Enable traditional messaging patterns such as persistent messaging, reliable messaging, retransmissions, multi-cast, uni-cast, publish- subscribe etc
- 20. Challenges • Lack of B2B support • Scalability challenges with centralized messaging models • Fundamentally based on .NET and J2EE programming models • Lack of support for modern architecture styles such as APIs, micro- services etc
- 21. Platforms • MuleSoft • WSO2 • BizTalk ESB Toolkit • Oracle ESB • WebSphere ESB
- 22. ESB Example: Mule ESB
- 23. ONN
- 24. ESBFrameworks Apache ServiceMix Sun (Oracle) OpenESB JBoss ESB Fuse ESB Mule ESB
- 25. Messaging Framework •RoutingConnectivity (HTTP, JMS, FTP) • Cloud connect (Salesforce, Facebook,Twitter) Web Services (SOAP, REST)
- 26. Flow Message source Message processor Component Transformer Filter Router Endpoint Subflow – Private Flow
- 28. 3 <mule xmlns:jdbc="http://www.mulesoft .org/schema/mule/jdbc" xmlns:sftp="http://www.mulesoft .org/schema/mule/sftp" xmlns:mulexml="http://www.muLesoft .org/schema/mul 4 http://ww1uulesoft .org/schema/mule/xml http://ww11.mulesoft .org/schema/mule/xmL/current/mule-xmL.xsd 5 http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd 6 http://www.mulesoft.org/schema/mule/jdbc http://www.mulesoft.org/schema/mule/jdbc/current/mule-jdbc.xsd 7 http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd 8 http://www.mulesoft .org/schema/mule/core http://www.mulesoft .org/schema/mule/core/current/mule.xsd 9 http://111111.mulesoft .org/schema/mule/scripting http://111111.mulesoft .org/schema/mule/scripting/current/mule-scripting.xsd "> 10 <flow name="VcuPreApprSFTP" doc:name="VcuPreApprSFTP"> 11 <jdbc:inbound-endpoint queryKey="GetPADataForVCU"queryTimeout="-1" pollingFrequency="600000"connector-ref="NySQL"doc:name="GetPADataForVCU"> 12 <jdbc:querykey="GetPADataForVCU"value="call sp_GetPreApprovedinfo(173,${last.approved.hours})"></jdbc:query> 13 </jdbc:inbound-endpoint> 14 <logger message="####Payload #(message.payload)" level="INFO"doc:name="Logger"></logger> 15 <choice doc:name="Choice"> 16 <when expression="payload( 'resultsetl '].size()==@"> 17 <processor-chain> 18 <logger message="NoData Found for VCUPA" level="INFO"doc:name="NoData Found"></logger> 19 </processor-chain> 20 </when> 21 <othemise> 22 <processor-chain> 23 <componentclass="com.chromeriver.mule.export.preapprove.component.VCUPreApproveDataCompoent" doc:name="VCUPreApproveDataComponent"><Icomponent 24 <mulexml:object-to-xml-transformer doc:name="Object to XNL"></mulexml:object-to-xml-transformer> 25 <sftp:outbound-endpoint exchange-pattern="one-way" outputPattern="PreAppr_#{function:datestamp:yyyyfYl.ddHHmmss].xml" host="${ftp.panam.host}" po 26 </processor-chain> 27 </othemise> 28 </choice> 29 <rollback-exception-strategy doc:name="Rollback Exception Strategy"></rollback-exception-strategy> 30 </flow> 31 32 33 </mule>
- 29. flo VcuPreApprSFTP @::' ------------------------------------------------ Processor Chain @::' f lGetPADataForVCU Logger Choice NoDataFound Deffault·- - - - - - -:------------------ ProcessorChain @::' Ill ) aObject toXML SFTPtoVCUFolder I IVCUPreApproveDataComponent
- 30. Filter: Select 'aEndpoints • Hfti'lpl'. w - • CS)IMAP Q JMS HTIP- >SOAP - -- > • ->· Je EliScopes fillComponentscustom-interceptor WsMatter Transformers <X> mlByte-AT.1t fu "Se1ializable mlByte Arrayto String mlEmailto String ( ? ,Filters (? .Flow Control Error Handling (e Cloud Connectors Iii!Amazon SQS G D CMIS 0 Facebook M essage Flow Global Elements Configuration XM L
- 31. Enterprise Architect Standpoint • Implementing a ESB platform should be part of a broader SOA strategy • An ESB implementation will require professional services and training
- 32. Integration Platform as a Service
- 33. Goals • Deliver integration in a cloud computing model • Enable seamless integration with SaaS platforms • Leverage cloud infrastructure to run and scale integration solutions • Facilitate cloud-to-on-premise integration models
- 34. Technological Foundation • Adapters • Messaging channels • Services • Topics, queues • Routing rules • On-Premise agents
- 35. Strengths • Deep integration with modern SaaS platforms • Based on simpler architectures than its predecessors
- 36. Challenges • Lack of support for complex orchestrations and choreographies • Not designed for pure on-premise integration scenarios • Security, regulatory and compliance challenges
- 37. Platforms • Azure BizTalk Services • MuleSoft CloudHub • IBM Castiron Live • Dell Boomi
- 38. iPaaS Example: Azure BizTalk Services
- 40. Bridges in BizTalk Services
- 41. Transformation
- 42. Property
- 43. Routing
- 44. Hybrid Connectivity Server Explorer (Visual Studio) PowerShell CmdLets Management Service REST API Lob Relay (Service Host)
- 45. Enterprise Architect Standpoint • Implementing an iPaaS should be part of a broader cloud strategy • iPaaS should be focused on on-premise to cloud integration scenarios
- 47. Some Thoughts • The industry has evolved from centralized to federated models • Cloud infrastructures are called to play an important role in integration solutions • SOA is being replaced with new architecture models based on APIs and Microservices • Modern integration strategies should take into consideration mobile and IOT solutions
- 48. Microservices
- 49. Principles Of Microservices Modelled Around Business Domain Culture Of Automation Hide Implementation Details Decentralise All The Things Isolate Failure Deploy Independently Highly Observable
- 51. Principles of Microservices Integration • Focus on messaging brokering capabilities • Routing • Transformation • Protocol adaptation • Integration infrastructure optimized fro working with hundreds of endpoints • Highly dynamic, continuously changing in
- 55. Applications Become Services Service interface Data Service implementation SOAP, JMS, XML…
- 56. Services in the Network Controlled Interconnection and Re-use Data CRM Data ERP Data Risk
- 57. A Service Bus Data CRM Data ERP Data Risk The Service or Message Bus
- 58. OSS Other Corporate SVs BSS Enterprise Orchestration BPM BRMS Application REMOTE DATA CENTER Deployment Option C REMOTE DATA CENTER Deployment Option B REMOTE DATA CENTER Deployment Option A Application Application Application Application Orchestration BPM BRMS DOMAIN ESB Platform Application Application Application Orchestration BPM BRMS DOMAIN ESB Platform Application External Mashups External Mashups Hosted Services (e.g., google, facebook) BPM BRMS Java Service Delivery Orchestration Identity & Access Management (Common) MAIN ESB – A API Gateway Internal Mashups (e.g., web 2.0) ESB Gateway Invocation Module API Directory External Users MAIN ESB – B MAIN ESB – CORPORATE SVCs Big Data Profile DB Service Events Mgmt, Processing & Continuous Intelligence Other Modules We’ve All Been Here
- 59. Changing Constraints 2005: Yahoo Maps API 2005: Flickr API 2007: First iPhone & SDK
- 60. Service Orientation? Meet the Smartphone
- 61. SOA Strict contracts Basic Connectivity Few, known developers, Strictly managed Slow, unreliable, segmented networks APIs Loose contracts Minimal “stack” Self-service 1000s of developers O(10^4) users, external Always connected apps Different Sets of Requirements
- 62. Changes in Technology Why OAuth and not WS-Security? Why JSON and not XML? Why the focus on HTTP? Why add Developer on-boarding? Why add Analytics?
- 63. The metaphor gets us to think differently. Rapid iteration Programmability from anything Loose documentation YAGNI APIs as Design Metaphor
- 64. Thanks
Editor's Notes
- Messaging: Inbound, Outbound, Durable Messaging, Message Security, Message Assembly/Disassembly, Pluggable Adapters, Message Security, Message Assembly/Disassembly, Pluggable Adapters, Enterprise Single Sign-On, Service Enablement • Web Services based messaging (WS* standards)Windows® Communication Foundation (WCF) Adapters • WCF-BasicHttp • WCF-WsHttp • WCF-NetTcp • WCF-NetMsmq Orchestration: Flow Control, Port Bindings, Scopes, Design Patterns • Asynchronous Response Correlation • Aggregator • Composed Message Processor • Dynamic Router • Parallel Convoy • Scatter and Gather • Sequential Convoy • Splitter Business to Business Integration: Trading Partner Management, EDI Support, AS2 Support, Industry Accelerators • SWIFT • HL7 • RosettaNet Business Rule Framework: Business Rule Engine, Business Rule Authoring Business Activity Monitoring: Observation Model Design, Event Observation Engine, Reporting Management & Operations: Application Management, Tracking Configuration & Reporting, Host Management, Application Deployment, Best Practices Analysis, MOM Monitoring, Data Management AppFabric Connect Integration: Line of Business Applications (LOB) activities (Consume LOB operations as Windows Workflow Foundation (WF) activities), Mapper Activity (Transformation as a WF activity) RFID Platform Tools • Schema Editor • Flat File Schema Wizard • Mapper • Pipeline Designer • Orchestration Designer • WCF Service Publishing Wizard • Administration Console • MOM Management Pack • Best Practices Analyzer • Business Rule Composer • Policy Deployment Wizard • Orchestration Designer for Business Analysts • Observation Model Designer • Tracking Profile Editor • BAM Portal • BAM Manager • EDI User Friendly Schema Viewer • EDI Document Instance Generation/Validation • Trading Partner Portal • RFID Manager • RFID Tray • RFID Event Simulator • RFID Client Console
- Neste cenário, um aplicativo de inventário, talvez a correr num mainframe, determina que o stock de um item é baixa e emite um pedido para pedir mais desse item. Ocorrem as seguintes etapas: The request is sent to a BizTalk Server 2010 application The BizTalk application requests a purchase order (PO) from the organization’s Enterprise Resource Planning (ERP) application. The ERP application, which might be running on a UNIX system, sends back the requested PO. The BizTalk application informs a fulfillment application, built on the Microsoft .NET Framework, that the item should be ordered Neste exemplo, cada aplicação comunica por meio de um protocolo diferente, utilizando os formatos de mensagens específicas de cada aplicação. Isto significa que o mecanismo de mensagens do BizTalk deve ser capaz de se comunicar com cada aplicativo no seu protocolo de comunicação e formato nativo e também converter as mensagens para o protocolo e formato exigido pelos demais sistemas.
- The two main services in BizTalk Server 2010, the messaging engine and the orchestration engine O objetivo do BizTalk é processar mensagens. Toda a comunicação dentro e entre um servidor de BizTalk são baseados na troca de mensagens. Por esse motivo, o mecanismo de mensagens é essencial para todas as operações do BizTalk. The BizTalk messaging engine performs the following tasks: Receives inbound messages Parses inbound documents to identify their specific formats Extracts key identifiers and identifies applicable routing rules Delivers documents to their respective destinations including ports or orchestrations Tracks documents Message database The MessageBox database is a Microsoft SQL Server™ database that is used by BizTalk to store and route messages to orchestrations and send ports. When a message arrives in the MessageBox database, the metadata associated with the message is matched and evaluated to determine the services that subscribe to messages of this type. Publish-Subscribe Model BizTalk Server implements a publish-subscribe model for the routing of messages. In the publish-subscribe model, message providers (publishers) submit messages to a central store (the MessageBox), where subscribers (send ports and orchestrations) can subscribe to specific messages. After a message of interest is received by the MessageBox, it is sent to all subscribers
- Sources: -- Where the message comes from -- Every bridge has one HTTP source -- Multiple sources can be configured for a bridge endpoint Destinations: -- Where the message is sent -- Determined by the routing filters Pipelines: -- Pipeline template plus configuration -- Template composed of stages; stages composed of activities -- Stages and activities execute on the request path, as well as response path
- In the beginning there were apps. Actually, before the beginning there were digital computers, and then mass storage to store system state And pretty soon people wanted to set out formally codified rules that modified the state in formal ways. Data and logic. We used to call these “two tiered systems”. Nobody says that any more. What was the first system you worked on Greg? …. I did a bunch of numerical programming in the engineering program I was part of. And also I was a hobbyist so I wrote code just for fun. But the first job I got paid for was a two-tiered app. It was in my college years, a dBase II project. dBase was a sort of integrated environment with the data and logic all melded together. It was a order and customer management system for an investment brokerage company. I paid a lot of bills working on that system. A ton of people did. The world ran on these kinds of systems.
- Applications were useful, so they proliferated. And it was good. But then the problem became interconnecting these things. The system that handled loan origination needed to integrate with the system that was used for business planning, or portfolio management. And that’s a problem we’ve never really 100% solved. That problem continues to this day, just on different scales. About loan origination - I just talked with a company that has a twelve year old system that does loan origination. They’ve run their business on this system for more than a decade. Custom built. They also built a portfolio management system – once the borrower signs for the loan, it becomes part of the portfolio. It needs to be managed, checked, verified. But: The two systems don’t share a common customer model. Also - There’s no feedback mechanism in the portfolio management back to origination. If there are patterns, if for example loans made for mini-golf places are more risky than loans made for car washes, there’s no way to feed that information into the risk rating during the origination process. This is typical. Happens everywhere. Even in “modern” enterprises.
- There were a number of different approaches taken for interconnecting different systems. Numerous. But in 2002 to 2005, we all settled on SOA. What’s your view on SOA, Greg? Is it real? Useful? ….(basically, yes. It got overhyped and over-productized. But it works as a model)…. SOA is an approach for modeling large systems as services. In contrast to objects. Anyone remember CORBA? DCOM? Those were distributed object models. Very popular. Objects were everywhere. OOPSLA was born. With the distributed objects model, cooperating applications were remote objects, and programmers could interconnect by invoking methods on remote objects. SOA’s different. Rather than invoking methods on objects, SOA said these remote things (applications) were really services, and we would send them messages. This was a profound switch. SOA’s Only Architecture
- At that point, we modeled each standalone system as a service. And each service accepts one or more different sorts of messages. The list of messages a service could receive could be called the “contract” – and it was rigidly specified in the form of WSDL. We, as an industry, spent a lot of time focusing on the messages, describing them formally, describing how to map in-memory representations to on-the-wire, textual representations of messages. Trying to be very thorough. XML Schema. XML Data Binding. JAXB. XML namespaces. The effort we as an industry put into figuring out how to handle these messages. An astounding amount of effort. It amazes me. The XML Infoset. An attribute on an XML element can be namespace qualified. Remember that? A namespace with a trailing slash is different than a namespace with no trailing slash. Namespaces look like URLs but aren’t really. ----- “Services” can also be clients of other services. Extending that model we have a connected web of services. They all interoperate by sending and receiving messages according to the defined contracts. It was all about re-use of systems, with tight control. Primarily system-to-system integration. And while all of this was happening, the web was maturing.
- The term “Bus” is an abstraction, an analogy. It refers to the hardware bus – the bus that moves data between CPUs and memory. But it can be more than an abstraction. It can be concrete. Companies deploy ESB products, to facilitate the control and management of the SOA interconnections. Logging, controlling the access.
- The price of enterprise network has plunged steadily and continues to decline. It used to be expensive to acquire and provision a 10mb ethernet switch. Now 1gb ethernet is yesterday, and 10gbe is standard. Network is plentiful. Compute has followed a similar trajectory. It continues to drop in cost. Storage likewise. There was a time when people actually had to “reserve time” on a shared computer. Now, compute power is cheap and getting cheaper. And at the same time, the iphone Here’s the thing: SOA was designed with the old constraints in mind. The formality of SOA messages was necessary, because compute was still scarce, and network was relatively scarce or expensive. Proper system architecture had to consider the cost and availability of these resources, which meant we needed asynchronous queues in front of systems, so they would not be overloaded. This implied that the messages on those queues needed a great deal of formality, so they could be stored with all their context, indefinitely. This way, when re-constituted from the queue, everything was still there. Time of origination. Identity of the sender. Identities of other intermediaries. Response path. Many people shied away from using HTTP as a transport inside corp networks for system-to-system connection, because of the overhead of an http request, as compared to TCP. Have you ever heard of :”Chargeback”? There was a time when shared resources on a corporate network were metered, and if your system used those resources, your department was charged for that usage. Like a toll road. But we don’t have those constraints now. Now most enterprises have so much network capacity, it makes no sense within a corporation to charge for it. It’s like air. It also makes no sense to reject HTTP for its “overhead” in most casts. And there is less imperative to introduce asynchrony. Compute power is so inexpensive that most requests can be fulfilled RIGHT NOW, and systems can connect to each other directly. It’s more efficient this way.
- One more huge change: Mobile computing.
- APIs address a set of requirements that are different than those addressed by SOA. If you think of iterations in IT architecture, Some might say SOA was a necessary step for us to take. At this point though, we’ve got requirements being driven by the ubiquity of mobile computers, and the opportunity to be accessible in new ways to people who carry those computers with them at all times of the day and night. SOA wasn’t designed for that. There are a bunch of hard feature/functions that come from that. Rate limiting Oauth Auditing Analytics Quota management Versioning Developer enablement and onboarding
- Design Metaphors are just ways of thinking about modeling a problem. For a while we used Objects as a metaphor. EJBs –distributed objects SOA – services Now, APIs are the metaphor we use. It’s THAT important. The metaphor gets us to think differently about framing problems. It’s not just lip service. Thinking differently guides the solutions we build. What’s different? The Focus on Programmability. It’s in the NAME, fer pete’s sake. The loose contracts. They’re described in human documentation form, or they’re not described at all. JSON for universal access. A simple data format. YAGNI. Loose contracts means things can change often, and they do! Change leads us to iterations, which begets agility. Change of the interface is simply not something SOA was aimed at. In fact the mantra for SOA was was: change the implementation, and keep the interface exactly the same.