How Vanguard and Bloomberg Use AWS PrivateLink (NET323) - AWS re:Invent 2018

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Vanguard and Bloomberg Use AWS PrivateLink N E T 3 2 3 Ilya Epshteyn Principal Solutions Architect Amazon Web Services Barry Sheward Chief Enterprise Architect Vanguard Cory Albert Global Head of Cloud Strategy Bloomberg

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda AWS PrivateLink overview Vanguard’s use of AWS PrivateLink as part of micro account strategy Bloomberg’s use of AWS PrivateLink for real-time data (B-PIPE)

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. VPC Peering The Internet On-Premises VPC VPN AWS Direct Connect Availability Zone A Availability Zone B Instance C 10.1.3.33/24 Instance A 10.1.1.11/24 Instance B 10.1.2.22/24 Instance D 10.1.4.44/24 Public Subnet Public Subnet Private Subnet Private Subnet NAT VGW IGW EIP: 54.1.13.43=10.1.1.11 NAT Gateway AWS network primer (prior to AWS PrivateLink)

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. VPC endpoint gateway type • Limited support—Amazon S3 and Amazon DynamoDB only • Gateway endpoints not accessible from on-prem network natively (requires somewhat complex proxy setup) • Available only for AWS services

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. VPC peering • Designed for use cases with broad, bi-directional network trust • Not intended for fine-grained microservices trust model • Maximum of 125 peering connections per VPC by design • VPCs cannot have overlapping CIDR blocks

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Connection always initiated by the service user Brings services into your VPC and on-premise network via AWS private network Service owner only exposes a service concept without any network complexity AWS PrivateLink enables a secure and scalable model for sharing services

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PrivateLink for AWS Services, Enterprises, and Partners 18 AWS Services (and growing) AWS KMS Amazon Kinesis AWS STS Amazon SNS Amazon EC2 Systems Manager Amazon EC2 APIs Amazon API Gateway Amazon CloudWatch AWS Direct Connect VPN Connection Your Shared Services in Another AWS Account and VPC AWS Partners / Marketplace corporate data center

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Natively accessible from on-prem networks AWS Partners / Marketplace 18 AWS Services (and growing) AWS KMS Amazon Kinesis AWS STS Amazon SNS Amazon EC2 Systems Manager Amazon EC2 APIs Amazon API Gateway Amazon CloudWatch Your Shared Services in Another AWS Account and VPC AWS Direct Connect VPN Connection corporate data center

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10.0.0.0/16 Consumer VPC Private 2a 10.0.16.0/20 Private 2b 10.0.32.0/20 10.0.16.1 10.0.32.2 10.0.0.0/16 Provider VPC Public 2a 10.0.1.0/20 Destination Target 10.0.0.0/16 Local 0.0.0.0/0 IGW Destination Target 10.0.0.0/16 Local Private 2a 10.0.16.0/20 Private 2b 10.0.32.0/20 10.0.16.1 10.0.32.1

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10.0.0.0/16 Consumer VPC Private 2a 10.0.16.0/20 Private 2b 10.0.32.0/20 10.0.16.1 10.0.32.2 10.0.0.0/16 Provider VPC Public 2a 10.0.1.0/20 Private 2a 10.0.16.0/20 Private 2b 10.0.32.0/20 10.0.16.1 10.0.32.1 corporate data center

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. VPC 10.0.0.0/16 Public 2a 10.0.16.0/20 Private 2a 10.0.144.0/20 Amazon KMS (Provider) Destination Target 10.0.0.0/16 Local 0.0.0.0/0 IGW Destination Target 10.0.0.0/16 Local 10.0.158.56 10.0.128.238 kms.us-east-1.amazonaws.com vpce-042260d8dadad476a-0vjawe46.kms.us-east-1.vpce.amazonaws.com vpce-042260d8dadad476a-0vjawe46-us-east-1a.kms.us-east-1.vpce.amazonaws.com vpce-042260d8dadad476a-0vjawe46-us-east-1b.kms.us-east-1.vpce.amazonaws.com Endpoint-specific DNS and Default service DNS Endpoint-specific DNS and Default service DNS with “Enable Private DNS feature” (recommended)

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. VPC peering and VPC endpoints • Broad-based network trust • Connect VPCs, not services • Inter-region connectivity • Fine-grained trust between services • Service provider and consumer • Scalable to thousands of consumers VPC peering VPC endpoints with AWS PrivateLink

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Key benefits • Private IP addresses used to connect to external services • Same reliable and scalable technology used to access AWS services, Enterprise microservices, or third-party solutions • Support for overlapping addresses and reduced management points • Service owner only exposes a service concept • Connection always initiated by the service user • Accessible from VPC or from on-prem (DX or VPN – NEW) • Growing support by AWS services

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vanguard—Background Began operations – May 1, 1975 in Valley Forge, PA One of the world's largest investment companies, offering a large selection of low-cost mutual funds, ETFs, advice, and related services Wall ST

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vanguard’s account strategy—2016 AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC AWS Account VPC DC1 DC2 DCx

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Issues with the 2016 approach AWS Account AWS Account Subnet Network ACL Availability Zone - A Subnet Network ACL Availability Zone - B Subnet Subnet Network ACL Network ACL CIDR: 192.168.0.1/26 Subnet Network ACL Availability Zone - A Subnet Network ACL Availability Zone - B Subnet Subnet Network ACL Network ACL CIDR: 192.168.1.0/24 CIDR: 192.168.0.0/28 CIDR: 192.168.0.48/28 CIDR: 192.168.0.32/28 CIDR: 192.168.0.16/28 CIDR: 192.168.1.0/26 CIDR: 192.168.1.64/26 CIDR: 192.168.1.128/28 CIDR: 192.168.191/28

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vanguard’s micro account strategy AWS Organizational Unit AWS Account AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Organizational Unit AWS Account AWS Organizational Unit AWS Account AWS Account AWS Organizational Unit AWS Account AWS Organizational Unit AWS Account AWS Account AWS Organizational Unit AWS Account AWS Organizational Unit AWS Account AWS Account AWS Account AWS Account AWS Account AWS Account AWS Account AWS Account AWS Account syslevel division account type

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. IAM in micro accounts—STS IdP User memberOf Description Inan RootOU IAM Admin Bob DevLOB#1OU LOB DevOps Alice ProdOU Prod Support IAM for Enterprises: How Vanguard Has Matured Their IAM Controls to Support a Micro Account Strategy

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vanguard cloud registry service DCx AWS Account Transit Account AWS Account AWS Organizational Unit AWS Account - SvcConsumer Subnet Availability Zone - A Subnet Availability Zone - B CIDR: 172.31.0.0/16 Endpoints Endpoints Endpoint Service Endpoint VCRS Endpoint Service VCRS Endpoint VCRS Endpoint VCRS Endpoint Endpoint Service #2 Endpoint Service #1 Endpoint Service #3 AWS Account - SvcProvider Subnet Availability Zone - A Subnet Availability Zone - B CIDR: 172.31.0.0/16

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vanguard cloud registry service—Building AWS PrivateLinks AWS Account – Svc Provider CIDR: 172.31.0.0/16 Subnet – AZ A Subnet – AZ B Subnet – AZ C Subnet – AZ D AWS Account – Svc Consumer CIDR: 172.31.0.0/16 Subnet – AZ A Subnet – AZ B Subnet – AZ C Subnet – AZ D SvcProvider SvcConsumer 2. Endpoint Creation SvcConsumer

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Access to micro accounts AWS Account rolerole AWS Account role role AWS Account Bastion Account rolerole AWS Account Subnet Availability Zone - A Subnet Availability Zone - B CIDR: 172.31.0.0/16 Account role Amazon EC2 systems manager

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Future vision Ephemeral accounts Three Rs of enterprise security1 • Rotate • Repave • Repair applied to AWS Accounts Supports ZeroAccess Fit-for-purpose accounts Handle special cases, for example, custom address ranges, VPC peering Standard build mechanism (VCRS) Both Use AWS CloudFormation post-account creation AWS Account 1 https://builttoadapt.io/the-three-r-s-of-enterprise-security-rotate-repave-and-repair-f64f6d6ba29d

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bloomberg B-PIPE at a glance: The solution  What is B-PIPE? Consolidation, distribution, and access via a common API Bloomberg Customers 330 Exchanges 5,000 Pricing Contributors 35 Million Instruments 110 Countries 80 Billion ticks/day 15k Customer Locations 2+ Servers per Location 2+ Routers per Location Fault Tolerant Connectivity

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bloomberg B-PIPE at a glance: Our customers • Who leverages B-PIPE data? • Capital markets professionals: Small hedge funds to international banks • Front office applications used to • Assess risk • Manage portfolios • Make informed decisions • What drives customer buying decisions? • Total cost of ownership • Trust: managed service w/highly-specialized support • Optimized: Reliability, scalability, flexibility • These same “ilities” are driving their cloud migration . . .

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our customer’s path to optimized Pre-2017: Customers subscribed to data on-prem and published it to applications residing in the cloud Jun 2017: Zero footprint offerings provide data directly to applications. However, a reliance on the internet causes performance, reliability, and scalability concerns Nov 2018: B-PIPE in AWS is introduced as a cloud-optimized solution PublicCloud ApplicationsTickerplants API Infrastructure BLOOMBERG Parsers Customer Prem B-PIPE B-PIPE App App App App 3rd Party Content PublicCloud Applications Customer Prem Tickerplants Distribution BLOOMBERG Parsers blp api 3RD Party Content Tickerplants Distribution BLOOMBERG Parsers PublicCloud Customer VPC Customer A Office Bloomberg VPC Apps blp api BPIPE 3RD Party Content

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Requirements for an optimized solution • Must be a “no compromise” offering • Content: depth and breadth • Volumes of data consumed • Resiliency • Latency • Must continue to be a managed solution • Monitoring the health of the data path • SW upgrades • Entitlements management • API consistency (BLPAPI) whether cloud, on-prem, etc.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. B-PIPE optimized: Getting to the cloud (US East) Bloomberg Global Network Content Providers Content Providers Content Providers BloombergVPC Cust A VPC B-PIPE Cust B B-PIPE Cust A B-PIPE Cust B BLPAPI Cust A App BLPAPI Cust B App B-PIPE Cust A • Bloomberg ingests, normalizes, and distributes data globally • Distribution extends to AWS US East 1 • B-PIPE endpoints are deployed on EC2 instances in a Bloomberg managed VPC • Customer applications remain in their own VPC • Applications connect to B-PIPE using AWS PrivateLink • Result: Customers no longer need to host infrastructure to obtain reliable market data Cust B VPC

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. B-PIPE optimized: Inside the cloud (US East) Deployment for a Single Customer Location • 10 gig Direct Connects • Bloomberg AFN’s optimize BW utilization • B-PIPE service runs on EC2 instances • AZ’s provide resiliency • Bloomberg provisions B-PIPE via NLB • Provisioned customers create VPC endpoints to the NLB • Optional customer private DNS using Route 53

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Results from the lab Scenario Results %-tiles (September 24-8)*  DIFF 50% tile = ~0 ms  DIFF 99% tile = ~0 ms * * Ranges selected solely due to AWS presentation due dates.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Up Next • Business • Continue to work with early adopters • Prepare for general US release • Expand offering to meet customer demand globally • Technology • Develop and test multi-tenant solutions • Auto Scale w/Load Balancing • Expand the use of serverless

37. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Barry Sheward, Chief Enterprise Architect, barry_p_sheward@vanguard.com Cory Albert, Head of Cloud Strategy, Bloomberg Enterprise Data calbert3@bloomberg.net