CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta

Cloud Architectures & Platforms
CWIN17, September 27th 2017
Presenter : SHUVADEEP DUTTA, Insights Platform, Insights & Data

2Copyright © Capgemini 2017 All Rights Reserved 2Copyright © Capgemini 2017. All Rights Reserved
AGENDA
Platform Architecture Views
Cloud Platform Security
Platform Logging
Platform DevOps

Future Looking Data Lake – Conceptual Architecture

Big Data and Analytics Platform Logical Architecture – Cloud & Technology Agnostic View

Big Data and Analytics Cloud Platform – MS Azure / AWS Native Services

Big Data and Analytics Cloud Platform – Custom Built Stack (Opensource Hortonworks)

Cloud Deployment Considerations
Data topology, governance & security capabilities need to be evaluated taking into consideration the key considerations to define a
target state hybrid cloud platform architecture
Focus Area Consideration
Data Location Location of data storage in alignment with regional regulatory compliance directives
Analytics Use Cases Type of analytical workloads going to be executed on the data repository located at a specific region / country
Data Access, Authentication and Authorization Mechanism of data and underlying platform resource access based on specific user role, user location, time of access etc.
Network Latency Volume of data transfer over network ensuring appropriate bandwidth and SLAs being met
Data Ingestion Type of data ingestion mechanisms used to transfer data from on-premise to cloud and vice-versa
Security and Regulatory Compliance Implementation of controls and mechanisms to meet platform security (perimeter, data and application), regional regulatory
compliance directives such as PCI, SOX etc.
Platform Governance Platform wide metadata management, audit logging, master and reference data management capabilities; considerations
for bottom-up / top-down /Hybrid approaches: Data Catalog driven data discovery and knowledge sharing (bottom-up) vs.
Data Governance Council (Data Stewardship) driven information asset management and knowledge sharing (Top-down)
Platform Infrastructure Considerations for High Availability, Fault Tolerant and Disaster Recovery capabilities
Considerations from platform performance and scalability based on specific Big Data Analytics workload use cases
Considerations for Platform resource management and orchestration
Data and Application Portability Considerations for data and application portability across different platform environments e.g. On-premise, Private, Public
cloud

The information contained in this presentation is proprietary.
Copyright © 2016 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.capgemini.com
8Copyright © Capgemini 2017. All Rights Reserved
Capgemini’s DaaS (Data as a Service) Solution Framework
Meet all data and analytics management service needs from data ingestion, preparation, discovery, through till data analysis using opensource or commercial tools
Leverage client’s current investments and integrate with technology of client choice or extend / customize existing framework capabilities

9Copyright © Capgemini 2017 All Rights Reserved
Cloud Platform Security

Big Data and Analytics Platform – Cloud Security Framework
Physical Asset Security
Management
Protection for physical
assets and locations
including networks and
data centers
Cloud Governance
Cloud specific
security governance
including directory
synchronization and
geo locational
support
Information Asset
Protection
Protection of data at
rest or in transit
Governance &
Compliance
Security governance,
maintenance of
security policy, audit
and compliance
Threats and
Vulnerability
Management
Management of
vulnerabilities and
mitigations with
Network and
endpoint protection
Incident
Management
Management and
responding to
expected and
unexpected events
Identity and Access
Management
Authentication of
users and
management of
identity

Cloud Security Capability Framework – Shared Responsibility Model

AWS VPC Security Reference Architecture
VPC Security Architecture Scenarios
Scenario 1 : VPC with a Single Public Subnet only
• Instances run in a private, isolated section of the AWS cloud with direct access to
the Internet
• Network ACLs and security groups can be used to provide strict control over
inbound and outbound network traffic to EC2 instances
Scenario 2 : VPC with Public and Private Subnets and NAT
• In addition to public subnet, a private subnet is added whose instances are not
addressable from the Internet
• Instances in the private subnet can establish outbound connections to the Internet
via the public subnet using Network Address Translation (NAT Gateway or Instance)
Scenario 3 : VPC with Public and Private Subnets and hardware VPN access
• IPsec VPN connection between Amazon VPC and customer data center, while also
providing direct access to the Internet for public subnet instances in Amazon VPC
• VPN appliance on customer corporate data center side
Scenario 4 : VPC with Private Subnet only and hardware VPN access
• Instances run in a private, isolated section of the AWS cloud with a private subnet
whose instances are not addressable from the Internet
• Private subnet is connected to customer data center via an IPsec VPN tunnel

AWS VPC Security Reference Architecture – Security Groups
 VPC security groups to firewall each EC2 instance
 Each instance can be in up to 5 security groups
 Separate security groups for applications and management
 Security groups are stateful with ingress and egress rules
 Max. 50 rules per security group
 VPC Router will allow any subnet to route to another in VPC
 Network Access Control Lists are used to restrict internal VPC traffic
 Elastic load balancers are used to distribute traffic between instances
 Elastic load balancers are also placed in security groups
 Platform security can scale up and down with solution; instances can be added into security groups during launch time
 Use NAT instances to provide internet connectivity for Private Subnets; allow backend servers to route to AWS APIs – Ex. storing logs in S3
bucket or using DynamoDB, SES
 Access AWS API endpoints through the Internet Gateway like S3, SES, DynamoDB, SNS etc.

AWS Identity and Access Management
 Securely control access to AWS services and resources
 Fine grained control of user permissions, resources and actions
 Support for RunInstances
 Multi Factor Authentication – Hardware token or Smartphone Apps
 Segregation of roles using IAM
AWS Account Owner (Master)
Network
Management
Security
Management
Server
Management
Storage
Management

AWS Identity and Access Management Role Based Security

AWS Identity and Access Management using Enterprise’ existing Active Directory
Flow
1
2
3
4
The enterprise user accesses the identity broker
application
The identity broker application authenticates the users
against the corporate identity store
The identity broker application has permissions to
access the AWS Security Token Service (STS) to
request temporary security credentials
Enterprise users get a temporary URL that gives them
access to the AWS APIs or the Management Console
AWS Identity Federation with Temporary Security Credentials

AWS Data Storage Security – Capabilities
 AWS S3 Capabilities –
 Access controls at bucket and object level
 Cryptographic capabilities such as SSL for data in motion, Server/Client side encryption, MD5 checksums
 AWS Redshift Capabilities –
 Full disk encryption; CloudHSM to store keys
 Back-up access logs to S3 for analysis
 Security groups and VPC for deployment; data loading using SSL from S3 with restricted access to S3
 SSL encryption for data accessed over internet
 AWS RDS Capabilities –
 Restricted access to RDS instances using Security groups and IAM permissions
 Data encryption (Data at rest and in motion)
 Automatic patching for minor updates
 AWS DynamoDB Capabilities –
 Fine grained security access to columns and rows using IAM role and access policies
 AWS EBS Volume Capabilities –
 Option to use own encryption or commercial solutions Ex. Windows BitLocker or Linux LUKS for encrypted volumes and TrueCrypt for containers; Commercial : Safenet Protect-V,
Trend Secure Cloud etc.

Securing AWS Applications Process Flow

Platform Logging

Platform Monitoring – Centralized Troubleshooting, Security, Audit and Monitoring

Platform Monitoring – Solution Options
Shippers Queue Logstash Elasticsearch
Elasticsearch
Kibana
Log Parsing Indexing & Curation Dashboards & ReportsRedis, RabbirMQ, KafkaSyslog, Rsyslog,
Logstash, Fluentd etc.
JSONEvents Extract
Option 1 – Custom Built using ELK stack (Elasticsearch, Logstash and Kibana)
Option 2 – SaaS Solution – Loggly (alternate leading tool : Sumologic)
Shippers
Syslog, Rsyslog,
Logstash, Fluentd,
cloud plug-ins etc.
Application Framework
INGEST PROCESS INDEX
Search &
Other
Services
• Cloud-based SaaS for easy central log collection, aggregation,
management
• Easy set-up
• Dynamic parsing - Real-time, JSON support, parsing/tagging,
self-documenting
• Regular Expressions based
• Dashboards, pre-configured and customizable, shareable
• Anomaly Detection
• Alerts that can be sent to HipChat, Slack, PagerDuty, HTTP
endpoints, others
• JIRA Software integration, point-and click ticket creation
without leaving Loggly

Platform DevOps

DevOps Reference Architecture for Big Data Analytics Workloads

The information contained in this presentation is proprietary.
Copyright © 2015 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.capgemini.com
About Capgemini
With more than 145,000 people in over 40 countries, Capgemini is one of the world's foremost providers of
consulting, technology and outsourcing services. The Group reported 2014 global revenues of EUR 10.573
billion.
Together with its clients, Capgemini creates and delivers business and technology solutions that fit their
needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its
own way of working, the Collaborative Business Experience™, and draws on Rightshore®, its worldwide
delivery model
Learn more about us at www.capgemini.com.

CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta

Similar to CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta (20)

More from Capgemini

More from Capgemini (20)

Recently uploaded

Recently uploaded (19)

CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta