Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (18)
Similar to Cloud integration patterns, technologies & trends
Similar to Cloud integration patterns, technologies & trends (20)
More from Richard Seroter
More from Richard Seroter (8)
Recently uploaded
Recently uploaded (20)
Cloud integration patterns, technologies & trends
- 1. CLOUD INTEGRATION PATTERNS, TECH & TRENDS richard seroter | seroter.wordpress.com | @rseroter
- 5. TIGHT FIT
- 9. RPC
- 10. CHALLENGES
- 11. EXAMPLE #1
- 12. Demo Architecture Contoso. Ltd line of business system web server Service Bus Relay
- 13. MESSAGING
- 14. CHALLENGES
- 15. EXAMPLE #2
- 16. Demo Architecture BizTalk Services Web Sites
- 17. EXAMPLE #3
- 18. Demo Architecture Service Bus Topics Contoso. Ltd BizTalk Server Web Sites db server
- 19. SHARED DATABASE
- 20. CHALLENGES
- 21. DATA INTEGRATION
- 22. FILE TRANSFER
- 23. CHALLENGES
- 24. EXAMPLE #1
- 28. CHALLENGES
- 29. EXTEND DIRECTORY
- 30. CHALLENGES
- 31. EXAMPLE #1
- 32. Demo Architecture Web Sites Contoso. Ltd Active Directory AD FS
- 33. EXAMPLE #2
- 36. CHALLENGES
- 37. EXAMPLE #3
- 38. Demo Architecture Windows Azure ACS Web Sites
- 40. POINT TO SITE
- 41. CHALLENGES
- 42. SITE TO SITE
- 43. CHALLENGES
- 44. DIRECT CONNECT
- 45. CHALLENGES
- 46. WRAP UP
- 47. BE A GUIDE
- 48. richard seroter | seroter.wordpress.com | @rseroter THANK YOU
Editor's Notes
- Photo attribution: http://www.flickr.com/photos/ogimogi/2253657555/WHERE ARE WE?Cloud usage increasing rapidlyAccording to new data from Gartner, a mere 38% of enterprises are running cloud services today. But a whopping 80% expect to adopt cloud services within the next 12 months,including 55% of those that currently don't have any cloud services running. A mere 6% expect to decrease their investments in cloud services over the next few years. Easier than ever to create apps in the cloud. Providers offer different types of tools that can be used by BAs or developershttp://blogs.forrester.com/james_staten/13-05-16-hybrid_cloud_future_too_late?And about one third of the developers who responded to our Forrsights Q1 2013 Developer Survey said they have already deployed applications to the public cloud.Twenty-five percent also admitted to putting application integrations in placeWhere you know it or not, odds are you’re company is leveraging cloud assetsTREND: Many cloud providers now offer a free or cheap tier so even easy to on-ramp
- Photo attribution: http://www.flickr.com/photos/djbiesack/2570996397/WHERE ARE WEThe ease by which we can deploy cloud apps – and doing so without IT oversight – means that many applications are silos that don’t share enterprise data, user profiles, business logic, and more. Often, you either reproduce it entirely in the cloud app, or maintain a kludgy link between cloud and on-premises.
- Photo attribution: http://www.sxc.hu/photo/1214218WHAT’S THE PROBLEMWhy do people love using the cloud? Agility, broad network access, pay-as-you, speed. Why don’t they? Integration!Far from a 100% success rate on cloud projects, and surveys show that a good portion of them fail because of poorly thought out integration A survey published in Integration Developer Newsshows that 92% of CIO’s see the benefit of Cloud solutions but are concerned that data will be trapped in one vendor’s service, unavailable to the remainder of the organization. Cloud may be the best thing to happen to integration developers in a decade; massive opportunity for tangible value
- Photo attribution: http://www.sxc.hu/photo/595985WHERE WE WANT TO BEThe cloud isn’t an “if” but “when” for a portion of your enterprise portfolioHowever, all your assets – regardless of where they are hosted – should be interconnected so that you can maximize productivity, security, and reuse.Cloud seems to be where we finally realize the vision of service oriented architecture since each “service” resides in a different placeWe are faced with three new trends in integration: ground to cloud, cloud to cloud, cloud to ground
- Source: http://www.flickr.com/photos/shanepope/2375499336 | http://www.flickr.com/photos/adesigna/3237575990 | http://www.flickr.com/photos/28221478@N02/2941688941 | http://www.flickr.com/photos/bump/3806115100Four key areas of focus today, with discussion of eachApplicationDataNetworkIdentityIf you achieve integration across these four dimensions, you stand a great chance of realizing maximum benefit from cloud applications and environments
- Discuss a handful of patterns, challenges, and technologies
- Photo attribution: http://www.flickr.com/photos/inju/337085332/in/photostream/RPCAbout exposing application interfaces to allow other systems to interact with itWhy?Want to share logic and data among independently designed applicationsInformation lookup patternCharacteristicsData stays with sourceService oriented (encapsulated, abstracted, interoperable, reusable)Often synchronous in nature, meaning that the caller waits for a responseDon’t often need reliable intermediary as calling system receives exceptions and can retrySome cases for service busTrendsCloud platforms offer SOAP/REST endpointsSome also let you do custom code for invoking external services
- Challenges in the cloud?Coupling – source is tied functionally and temporally to the other applicationHTTP oriented – most cloud-based RPC is done via HTTP, so may not be able to use protocols like TCPLatency – inherent delay in cross geo boundaries, and waiting for responsesNetwork access – can internal systems reach cloud systems, and vice versa?Authentication – often want fine-grained controls, but can be tricky if calling app and target app use different security schemesScalability – cloud-enabling an existing app may introduce unexpected load and RPC doesn’t scale particularly well anyway!Connectivity – related to coupling, how do you handle downtime in the target app? Circuit breaker pattern?Transactions – considered the enemy of scalability, and tough to do across network boundariesHiding the network – big complaint against RPC is that it can make services seem local when they aren’t; encourages chatty behavior or local design
- NEEDWant access to information in on-premises systemIdeally in real time, and not replicatedWant to access it at the sourceCan use the Service Bus Relay to expose local services
- Photo attribution:http://www.flickr.com/photos/mrjoro/41626558/Async messagingTransferring bits of data (messages) frequently, immediately, reliably, and asynchronouslyWhy?Share data between systems in responsive and scalable wayNon-blocking; caller can continue on and optionally get an acknowledgement laterCaller may not care where the data goesSupports disconnected appsMost of our life experiences work this way: getting coffee, sending email, even depositing a check at the ATMConsiderationsPub-sub messaging (route based on content, recipients filter it out, broadcast to many, combine messages, split messages up)Therefore can be stateful or stateless
- Challenges in the cloudLocation of reliable intermediary, and is this a poll or push?Limited support – not a lot of commercial products support async input, let alone async outputReliability – if doing straightasync communication without a broker, can lose messagesNetwork access – how are we accessing on-prem and cloud systems?Debugging – async, brokered data can be trickier to trace as there are more hops, identity/format/structure/protocol transfersEventual consistency – brokers process data as they can handle it, so information is not instantly replicated but EVENTUALLY everything will be consistentProvider limits – if using a cloud-based broker/queue, many providers impose limits as to how often you can poll; leads to different design
- NEEDHave message needed by one of many partnersNot a single queueWhat if I have a customer who wants to place an order and based on the product, we want one of three suppliers to fulfill it?Could bake that RPC logic into the web app, but that increases coupling, and doesn’t scale well as we add new providers or if there is a burst in ordersWant to route data to reliable queue for each supplierBizTalk Services is brand new, announced this weekRuns in Windows Azure and acts as a cloud brokerNOT a workflow engine, but a routing engineMultiple sources (services, FTP), lots of destinations (services, queues, etc)
- NEEDPrevious case we saw a 1 to 1 routing; only a single recipientWhat if we want many parties to have access to the same message?Service Bus Topics give us a way to send a message to one place, and each “subscriber” gets a copySubscribers choose which messages they wantIn this demo, we’ll send a “customer signup” message to a topic, and not only pull it from a cloud based app, but also route it through BizTalk Server 2013
- Photo attribution: http://www.flickr.com/photos/dogsbylori/6501704501/Shared databaseIntegrate apps by having them store their data in a shared databaseSystems access the data from that shared locationWhy?Sharing via ETL or even messaging isn’t timely enoughDependent systems are consistent at the same time (although this may depend on technology)Single view of data, single data access strategyNo replication neededConsiderationsCould be for reporting, reference data, or even transactional dataTechnology choice
- Cloud challengesCoupling – VERY tight; connected to shared data source!Commercial support – few commercial products let you swap out their DB for anotherScalability – single point of failure (potentially) and may have to figure out how to scale to much wider loadHTTP – cloud DBs often expose REST-only endpoints that may not work against existing appsAuthorization – can you use same identity directory, or do fine-grained access control?Authentication – which security scheme being used to determine who users are?Provider limits – sizes, bandwidth, supported featuresSchema agreement – need ALL parties to agree on data definitionsRDBMS vsNoSQL – decide on best tech for the situation; may choose RDBMS for master data and NoSQL for external-facing cache
- Discuss one pattern, challenges, and technologies
- Photo attribution:http://www.sxc.hu/photo/1125238File transferSystems produce files (data extracts) that others want to consume; typically produced at regular intervalsWhy?No need for real-time replicationHave large data sets needed outside the source systemCannot access the source system (no query interfaces)Need data quality operations like complex transformation, address standardization, de-duping, other cleansing activitiesEndpoint is a data warehouseConsiderations…
- Cloud challengesCloud adapters – if integrating with SaaS systems or cloud environments, may want adapters specifically built for those endpointsTime delay – one of biggest downsides; hard to do rapid sharingNetwork access – how to push data to internal systems?Bandwidth costs – many providers let you shove data into their cloud apps for free, but charge you for extractHTTP – most cloud endpoints are HTTP, and classic ETL tools aren’t optimized for itAuthentication – securely access multiple data repositoriesExcess data sharing – one of biggest reasons people don’t like this; end up sharing more than you should and make copies that could be stolenMaintenance – ETL tools aren’t know for their simplicity, so small changes can take a while to deploy
- NEEDWhat if we have data in cloud LOB system and need it in another?One great tools is from Informatica which offers a cloud-based ETL IDE and an on-premises agent that actually executes the ETLData doesn’t pass through a cloud broker, so less concern
- Discuss 3 patterns, challenges, and technologies
- Photo attribution: http://www.sxc.hu/photo/371772Single credentialsUse “application” credentials that aren’t tied to a specific userWhy?Generic access to resource where personalization or – to a less extent – privacy isn’t a factorConsider lists of products, other reference dataWe technically do this all the time when we have IIS app pools run as a specific user, etc. Not a *bad* thing, but use with cautionWe saw this earlier when Salesforce talked to the on-premises service via the Service BusConsiderationsMore or less anonymous
- Cloud challengesAuditing – logging/auditing isn’t particularly useful as you don’t have an easy way to may consumers to usersAlso can’t throttle individual users easily (short of doing things with source IPs)Leakage – if credentials are shared (assuming callers still provide these shared credentials vs anon access that is impersonated on server)Authorization – can’t authorize specific actionsPersonalization – no built-in way to give results relevant to youLeast privilege – forces least privilege practices as you can’t be sure who is accessing the resource
- Photo attribution: http://www.sxc.hu/photo/1328142Extending directoryAllow access to on-premises identity directory to cloud resourcesCloud do this through a cloudy replica, or by directly extending on-premises directoryWhy?Allow outside users to authenticate with enterprise credentialsAvoid asking users to constantly create/maintain dedicated credentials for each cloud/mobile appSingle password policy, audit trailsConsiderations
- Cloud challengesLatency – can add log-in latency depending on where the directory (or replicas) resideHTTP – apps need HTTP-based access to directoriesSync lag – if using a cloud replica, could have a period after adding/changing/deleting users where the replica is out of syncLoad – is the on-prem directory capable of handing a 4x load increase if mobile clients, etc are now authenticating against it?Attack surface – technically exposing more endpoints for malicious targetingMobile connectivity – support for offline clients, token cachingInteroperability – need to be able to support many client types as cloud likely not homogenous
- NEEDMay want to expose directory directlyAD FS runs on Windows Server and provides interoperable endpoints for authenticating usersCloud apps, regardless of platform, can use this to figure out who users are, and personalize their user experience
- NEEDHave cloud app that wants to reuse existing enterprise credentials instead of coming up with own authentication/authorization schemeWindows Azure Active DirectoryCreate a directory in the cloudCan sync with on-premises directories and create a cloudy cache for Office 365 apps, or custom apps
- Photo attribution: http://www.sxc.hu/photo/584383Trusted 3rd partyRely on external identity provider; “bring your own identity”Federate with their IdP, and let them bring their ownWhy?Don’t make customers/partners/users set up YET another identity, or HAVE to reside in your own identity repositoryThink of partner integration where you don’t want to create accounts in your directory just for partnersReduce friction and give people option to use existing identitiesConsiderationsThis typically addresses *authentication* not *authorization*
- Cloud challengesVerified identity – anyone can create a Live/Yahoo/Google/FB ID, all this proves is that you are the credentials, not that you are the personManagement – aren’t enforcing PW policies, or easily auditing; also need to have mapping from web identity to application rolesMonitoring – no real way to tap into failed attempts, etcSLA – dependent on others; may have to provide backup login optionsLatency – relying on performance of web IdPs
- NEEDImagine having a website that needs user authentication for non-employeesCould use an ASP.NET identity provider (custom), or, easily use Azure ACS to federate with leading web IdPs
- Discuss 3 patterns, challenges, and technologies
- Photo attribution: http://www.sxc.hu/photo/908059Point to site VPNVPN that connects an individual machine to a remote environmentWhy?FlexibleUser-based securityCan do two-factor with cert + credentialsConsiderationsWindows Azure now offers this
- Cloud challengesPerformance may suffer (many conditions affect this)Not usable everywhere (some network connections have outbound filters that prevent VPN connections)Requires end-user system configuration (or end-user savvy)Customer pays for bandwidth
- Photo attribution: http://www.sxc.hu/photo/1156469Site to site VPNConnect two sites with a persistent connectionWhy?No special services to orderFlexibilityGood securityPerformance is pretty good, although this can depend on quality of endpoint hardware and the internet pathConsiderations
- Cloud challengesComplex to set up (not all customers are savvy)No guarantee of performance – just because it’s fast one day, doesn’t mean it will be the next, and in general this is not resolvable by configuration changes (ie out of anyone’s control)Customer pays for bandwidth
- Photo attribution: http://www.sxc.hu/photo/419970Direct connectIncludes technologies like cross-connects (where you physically link cages in a shared DC), point-to-point network services, MPLS networks, or an WAN technologyWhy?Super low latencyConsistent performanceRemove bandwidth costsMost reliable (internet connections do not affect performance)Can still use point-to-point direct VPN as a backupConsiderations
- Cloud challengesExpenseLead-time to set upConsulting expenses (ie P/S activity for Tier 3)Possible vendor lock-in (ie contract lengths with terminations penalties)Inflexible – if you provision a 100Mbps link, you can’t scale it up to 1gbps easily (but you can work around this – ie provision a gigiabit link with a 100mbps committed rate – which will have cost impacts)Customer savvy required (network engineering)
- Photo attribution: http://www.sxc.hu/photo/1373194Organizations NEED expertise in an exploding area of attentionCloud offers some awesome agility, and don’t have to throw away existing, established patterns. Reuse your knowledge, but understand new caveats AND new technologies