Auditing Isolation Segments and Application Security Groups

David M. Zendzian - dzendzian@pivotal.io
Steve White - swhite@pivotal.io
Day 3: Security Auditing and
Compliance

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
This presentation contains statements which are intended to outline the general direction of certain of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding the pre-release of Pivotal offerings, future updates or other
planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. All software releases are on an “if and when available” basis
and are subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. Any purchasing decisions
should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's
offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward-looking information in this presentation.
This presentation contains statements relating to Pivotal’s expectations, projections, beliefs, and prospects which are "forward-looking statements” and
by their nature are uncertain. Words such as "believe," "may," "will," "estimate," "continue," "anticipate," "intend," "expect," "plans," and similar expressions
are intended to identify forward-looking statements. Such forward-looking statements are not guarantees of future performance, and you are cautioned
not to place undue reliance on these forward-looking statements. Actual results could differ materially from those projected in the forward-looking
statements as a result of many factors. All information set forth in this presentation is current as of the date of this presentation. These forward-looking
statements are based on current expectations and are subject to uncertainties, risks, assumptions, and changes in condition, significance, value and
effect as well as other risks disclosed previously and from time to time by us. Additional information we disclose could cause actual results to vary from
expectations. Pivotal disclaims any obligation to, and does not currently intend to, update any such forward-looking statements, whether written or oral,
that may be made from time to time except as required by law.

Agenda
■ R1 - Segmentation / Secure Architecture
■ R2 - Standard / hardened conﬁgurations
■ R3 - Secure Storage
■ R4 - Secure Transmission
■ R5 - AntiVirus
■ R6 - Secure Development Practices
■ R7/R8 - Access Control
■ R10 - Logging and Monitoring
■ R11 - Security Scanning/Testing
(Not covering R9-Physical Security, R12-Policies)

Preparing for the Audit

● While some of the items discussed in this presentation overlap assessments
like PCI vs Penetration Testing, this presentation will focus on assessments
and not Penetration Testing practices of Pivotal Platforms
● MFA Jump host with PAM (Privileged Access Management) recording all
commands used on host
● Don’t manually add users (implies no ssh to Opsman VM)
● Do not share accounts like BBR or other admin accounts as you lose
traceability (or very diﬃcult to trace)
Pre-Audit preparation

Pre-Audit Preparation
Audit user accounts
● Opsman audit read-only account - Not very useful (auditor often unfamiliar with
platform, but still can be used to see conﬁgurations)
● Auditor working with administrator to review conﬁg
○ Sanitized export of system manifests
■ $om deployed-manifests (if om CLI is installed)
■ $bosh deployments; $bosh manifest; $bosh releases;
$bosh cloud-config

● What is “In-Scope” for the audit
○ OpsMan / Director / Infrastructure Network / PAS-PKS Control plane
○ PCI / HIPAA / .. isolation segment or entire platform
○ IaaS
○ Services / Data Stores
○ Identify which deployed applications are in-scope for the audit
● Cloud Native Policies and Procedures
○ Have your company policies been updated for Pivotal Platform cloud
native environments including continuous compliance requirements.
○ Policies are the business responsibility, we will not be covering those
control requirements in this presentation.
Audit Scoping

Scope - PAS components / Isolation Segments
● PAS Subnet includes PAS
components that are typically
in-scope as they control the
platform.
● Without Isolation segments all
Diego Brains are in-scope as they
are in the in-scope network.
● With Isolation segments, any Diego
Brains in the PAS network are still
in-scope because the in-scope PAS
components are in the same
in-scope network.

● OpsMan and Director are not BOSH managed
○ If doing authenticated scans using BOSH added users it will only add
those users to BOSH managed VMs
○ FIM / AV / IPSec / Compliance Scanner are only on BOSH managed VMs
● OpsMan is a unique host for the platform due to on-boot requirements and
configurations needed to bootstrap and manage the platform
● Maintaining an up to date diagram is difficult due to the constantly changing
environment. There should be a company provided diagram based off of the
Pivotal reference architecture that documents the architecture of the platform.
The diagram should be based on inventory information that is covered in
section R2 below.
Unique challenges for auditing Pivotal Platforms

R1 - Segmentation / Secure Architecture
“Install and maintain a ﬁrewall conﬁguration to protect cardholder data“

IaaS Segmentation
● Pivotal Platforms should be segmented from the rest of the corporate
infrastructure.
● Application Ingress traﬃc should be restricted to the Load Balancer or
whatever is in front of the provided services (GoRouter / Service Mesh).
● Access to the Jump Box should be restricted to those who have access rights.
● Egress traﬃc should be restricted to that which is necessary for the platform to
operate.
● Proxies are recommended for use on egress if they are in use by the company
for existing data-center solutions.

Isolation Segments
Compute Isolation allows for different
compute placement & configuration
Routing Isolation via Dedicated subnet,
firewall and load balancer in addition to
segmentation at IaaS.
Organization and space can be assigned
to isolation segment instead of the
default shared multi-tenant segment
Share a single PCF control plane across
discrete, isolated application planes

• Egress rules that define where traffic can be sent.
• Define protocols, ports, and IP addresses
• Staging and Running ASGs can be configured
• BETA - Dynamic egress groups allow egress rules per application
Application Security Groups / Dynamic Egress

Granular Isolation
App-deﬁned container to container network policies

Auditing IaaS Segmentation Controls
● IaaS segmentation - Audit/validate using existing well-known practices
○ IaaS Security Groups
○ IaaS and business ﬁrewalls
○ Router conﬁguration / ACL

Auditing Isolation Segmentation Controls
https://docs.pivotal.io/pivotalcf/2-6/customizing/installing-pcf-is.html
● Use OpsMan to confirm Isolation segmentation installation and configuration
○ Confirm “Enable Silk Policy Enforcement” is enabled
○ Confirm “Router Sharding Mode” is configured for Isolation Segment Only
○ Confirm “Configure System Logging” is enabled to syslog system components
● IaaS firewall configuration for isolation of isolation segmentation
○ https://docs.pivotal.io/pivotalcf/2-6/adminguide/routing-is.html#config-firewall
review IaaS rules have default deny and configured for services in link
● From command line audit
○ $ cf isolation-segments
○ $ cf org ORG-NAME
○ $ cf space SPACE-NAME
● There may be multiple ISO Segments; perform the above for all of them

Auditing C2C Segmentation Controls
● Container to Container configuration
○ $cf network-policies
■ source is the name of the app that sends traffic.
■ destination is the name of the app that will receive traffic.
■ protocol is one of the following: tcp or udp.
■ ports are the ports at which to connect to the destination app. The
allowed range is from 1 to 65535. You can specify a single port, such as
8080, or a range of ports, such as 8080-8090.
■ destination space is the space of the destination app.
■ destination org is the org of the destination app.

Isolation Segmentation and C2C
● Container to Container overlay network can span into isolation segments
● If using C2C and Isolation segment you will need to audit all C2C conﬁgurations
and ensure that none of them are spanning into spaces that are part of isolation
segments

Auditing Egress Segmentation Controls
● Egress sets scope of “connected systems”
○ If foundation is dedicated with good foundational egress controls that may be
“good enough”
○ If there either one is not true then ASG/DE are needed - ex: foundation IaaS
“allows” access to entire enterprise or other “holes” in controls around
foundation
○ Dynamic Egress “default deny” is layered under ASG default group - need to
ensure default allow all asg is removed
● Dynamic Egress (list destinations and policies)
○ policies enforced by app GUID so need to identify app GUIDs in-scope
○ $cf curl /networking/v1/external/destinations -X GET
○ $cf curl /networking/v1/external/egress_policies -X GET
○ $cf security-groups

Auditing ASG Segmentation Controls
● ASGs are applied by configuring ASG sets differentiated by scope, platform-wide
or space specific, and lifecycle, staging or running
● Binding an ASG does not affect started apps until you restart them
● Make sure ASGs are defined, and the default allow all rule has been unbound
○ $cf security-groups
■ List all security groups
○ $cf security-group SECURITY_GROUP
■ Display all rules of a security group
○ $cf staging-security-groups
■ All ASGs applied to the platform-wide staging ASG set
○ $cf running-security-groups
■ All ASGs applied to the platform-wide running ASG set

ASG Sample results
Ensure audited application org and space ASG deﬁnitions do not have an allow all rule and only have
deﬁned egress that is necessary for the app. [
{
"protocol": "icmp",
"destination": "0.0.0.0/0",
"type": 0,
"code": 0
},
{
"protocol": "tcp",
"destination": "0.0.0.0/0",
"log": false,
"description": "Allow All"
}
]
EX: Make sure you don’t see something like this --->>

R2 - Standard / hardened conﬁgurations
“Do not use vendor-supplied defaults for system passwords and other security
parameters”

Stemcells and Buildpacks
Stemcells
Versioned OS image
Bare minimum OS skeleton
No information about software that will be installed
Exactly the same for all infrastructure
Updates published by Pivotal
• Monthly for Low/Med CVEs
• As fast as possible for High
Extensively hardened
Based on industry best practices from CIS and NIST
NOTE: Passwords and secrets are customized at installation, each Pivotal Platform installation has unique
passwords and secrets, there are no “vendor default” passwords in a deployed Pivotal platform.
Buildpacks
Framework and runtime support for apps
Examine apps for dependencies and how to
conﬁgure apps for bound services
Automatically detected and used to compile
or prepare app for launch
Can be customized if needed by the
developer
Deployed and logged in a consistent way
Provides control and auditability over what’s
running at any given time

Auditing Conﬁguration/Inventory
● Run Pivotal Compliance Scanner to demonstrate the VMs are conﬁgured
according to industry standard guidelines.
○ https://docs.pivotal.io/addon-compliance-tools
● Use $ bosh vms or BBR to get a snapshot of the running environment.
● $ cf apps and $cf app APP can be used to identify details about apps.
● CF Butler can also greatly assist with this.
○ https://github.com/pacphi/cf-butler

R3 - Secure Storage
“Protect stored cardholder data”

At-Rest Encryption
● Use IaaS at-rest encryption methods for underlying storage.
● Credhub database is encrypted with a user-provided key and random seed.
○ There are multiple credhub services within the system. For applications
encrypting customer data this would be the Credhub Service Broker.
There are also BOSH, PAS and Runtime Credhubs which are used for the
platform and will be reviewed in R7 for platform credentials.
○ Admins and Developers for a space have permission to bind Credhub
service broker instances to applications. All Credhub service broker
services created are globally available. Be sure to audit for applications
that may be bound to service broker instances incorrectly.
● HSMs can be used to provide the encryption key for the Credhub database.
○ Currently support Luna HSMs
○ nCipher nShield HSM will soon be available and is in testing now

Auditing Storage
● Review/conﬁrm IaaS at-rest encryption methods for underlying storage.
○ If using terraform / platform automation review those scripts as well.
● Validate that Credhub database is encrypted, sample a few columns.
● PAS
○ Determine if using external database or internal - if external use credentials
when creating that to run the query.
■ OpsMan / PAS / CredHub - Options PAS or External
■ OpsMan / PAS / Databases - Options Internal MySQL or External

Auditing Storage
● If auditing the BOSH credhub for validating storage of passwords is encrypted
○ $ bosh ssh director
○ $ psql -u -p
● If it’s internal, BOSH ssh to instance and run the query
○ $bosh ssh database
○ $mysql -u XXX -p XXX credhub
● Run SQL query to view encrypted columns
○ mysql> select * from encrypted_value limit 5;

R4 - Secure Transmission
“Encrypt transmission of cardholder data across open, public networks”

TLS Component Certiﬁcate Source
Load Balancer Enterprise root CA
Gorouter Enterprise root CA
App PCF root CA

TLS - Platform Components

Auditing Transmission Encryption
● Opsman PAS conﬁguration (Networking tab)
○ Minimum TLS
● Where TLS is terminated
● HAProxy and mTLS (if used)

Auditing Application Transmission Encryption
OpsMan PAS Conﬁguration (Application Containers tab)
● Ensure mTLS is used between GoRouter and app containers
● Ensure in-scope apps aren’t using TCP routing or if they are, they have their
own mechanism for TLS

Validating Transmission Encryption
● SSH to GoRouter / Diego cell VM and use $tcpdump to validate transmission is
encrypted.
○ On diego cell find the IP of the app GUID being evaluated and capture
■ $less /var/vcap/data/container-metadata/store.json
| json_pp to find the IP
■ $tcpdump -v -XX -i any src host <IP_of_app>
○ On GoRouter, if you see unencrypted traffic, monitor a full session and
capture the application URL to see if it is the application being audited.
■ $ tcpdump -w outputfile.pcap -S0
■ Load outputfile.pcap into wireshark or ngrep and search for GET
request in an unencrypted session to ensure it’s the application being
audited.

R5 - AntiVirus
“Protect all systems against malware and regularly update anti-virus software or
programs”

Pivotal Anti-Virus
(artist formerly known as ClamAV Add-on for PCF)
● Antivirus for VMs and container ﬁle system
● Scan on-access and/or via a schedule
● Conﬁgurable update mirror
● Alerts sent to syslog

Auditing Anti-Virus (schedule)
● Verify scheduled scans are not disabled (Anti-Virus Conﬁguration tab)

Auditing Anti-Virus (definition files)
● Verify definition files are updated automatically
○ Reivew the last 20 lines of the update log for each PAS vm:
■ bosh -e <env> -d <deployment> ssh -c "sudo tail -20
/var/vcap/sys/log/antivirus/freshclam.log"
○ Repeat for each deployment in-scope for the audit

Auditing Anti-virus (logging)
● Validate syslog forwarding is turned on (details further on under R10) and
review syslog target to ensure messages are received from AV
● If syslog forwarding is not used, review the following ﬁles on the VMs
○ /var/vcap/sys/log/antivirus/freshclam.log
○ /var/vcap/sys/log/antivirus/clamd.log
○ /var/vcap/sys/log/antivirus/clamdscan.log

R6 - Secure Development Practices
“Develop and maintain secure systems and applications”

Remove bad software
● Malicious software
● Unauthorized changes
● Configuration drift
Patching Inconsistent Disrupt CnC/Exfil
● Open files/locks
● Kernel updates
● Failed patches
● Remove point of
presence on internal
network
● Remove staged data
● Return to golden image
● Must be architected
properly
● No downtime to
applications
● Minimal impact to
platform functions
No Downtime
Repave don’t Patch (the infrastructure)

Continuous Delivery Pipeline Example
CI
Production
Arbitrary Jobs
Compliance checks
Service tickets
Performance tests
Security validation
Monitoring
Security scans
Chaos engineering
Blue/Green deploys
Canary analysis
A/B testing
Test-driven dev
Iterative coding/ﬁxing
Frequent integration

Auditing Platform
● Monitor https://pivotal.io/security or the RSS feed linked from that page for
awareness of new vulnerabilities
● Review platform automation pipelines to ensure:
○ Repaves are performed regularly (ideal is at least weekly, minimum
monthly)
○ All production applications are restaged (not just restarted) monthly to
ensure they are using the latest buildpacks
○ Updated stemcells are consistently applied to production within 30 days
from release
● Run the Pivotal Compliance Scanner and review the results to demonstrate
compliance with recommended security conﬁgurations

Auditing Platform (stemcells & buildpacks)
● $bosh stemcells shows all the stemcells uploaded to the system and
indicate which ones(s) are currently deployed
○ Compare versions with release information from PivNet to validate none of
the deployed stemcells are older than 30 days
● cf butler is the best way to audit the buildpacks and versions used by
currently deployed applications. Without cf butler:
○ Identify droplets used by in-scope apps $cf v3-droplets APP_NAME
○ Find buildpack info from droplet using cf curl
○ $cf curl /v3/droplets/[GUID] GUID is the droplet GUID above
● Review the buildpack versions in-use by the running applications and ensure
they are the most recent

Auditing Platform (PKS/k8s)
● Review platform automation pipelines, repaves, and stemcells as mentioned
previously for PAS as these items are the same
● Evaluate how containers are built and the automation involved to determine
how automated/programmatic it is
● Run vulnerability scans and conﬁguration scans on the images in the repo used
by the running apps to evaluate security

R7/R8 - Access Control
“Restrict access to cardholder data by business need to know”
“Identify and authenticate access to system components”

Cloud Foundry platform users are developers and operators using platform applications
E.g.: Apps Manager or the cf CLI
There are three ways to store platform user proﬁle/credentials:
• Internal store - User information is stored in the UAA database
• LDAP - User information is stored in an LDAP server
• Enterprise Identity Provider - User information is stored in an external
service like ADFS/SAML Provider (recommended)
Cloud Foundry Platform Users

CredHub Mitigates the Risk of Leaked Credentials
CredHub delivers centralized
management of platform and
application creds.
● Credentials are the bedrock for trust in the
cloud.
● CredHub’s goal: deliver cradle-to-grave
management of credentials (create, access
control, distribution, rotation, logging)
● Manages passwords, certiﬁcates, ssh keys,
RSA keys, and arbitrary values (strings and
JSON blobs).
● All credentials are encrypted w/a key that
rotates (HSM support in OSS & PCF)
● CredHub Service Broker for oﬀ-platform
services
● Cert based app identity

PAS User-level RBAC
● Platform operators have broad access to
support day-to-day health and
conﬁguration of the platform
● All applications reside within a Space,
and each Space is within an Org
● Collaborators share an org's resource
quota plan, applications, services
availability, and custom domains
● Using standard roles, users are granted
permissions at the Org and/or Space
level to meet the unique needs of each
customer

Auditing Users (and Roles) PAS
● Review the OpsManager SAML and LDAP setting tabs (found under the Settings
menu in the drop-down from the logged in username in OpsMan)
○ Ensure either SAML (preferred) or LDAP settings are completed
○ Review the “SAML Admin Group” or “LDAP RBAC Admin Group Name”
○ Conﬁrm this is the appropriate group to have platform admin role
○ If for some reason local users are used, review password security settings
and use $uaac target <OPS_MAN/uaa> to target the OpsMan UAA
instance, $uaac token to login and $uaac users to list users
● PAS Tile - Review the “Authentication and Enterprise SSO Tab”
○ Either SAML (preferred) or LDAP should be conﬁgured, not local users
○ Also audit local users using $cf org-users and $cf space-users

Auditing Users (and Roles) PKS
● Review the OpsManager SAML and LDAP setting tabs (found under the Settings
menu in the drop-down from the logged in username in OpsMan) and review the
OpsMan users as described earlier for PAS.
● PKS Tile - Review the “UAA Tab”
○ Either SAML (preferred) or LDAP should be conﬁgured, not local users
○ Audit local users using UAA as described for OpsMan, but targeting the PKS
UAA server rather than the OpsMan UAA server and add the following:
○ $uaac group mappings to see admin roles mapped to external groups
○ $uaac clients to see admin roles mapped to automation client IDs
○ $kubectl get clusterroles --all-namespaces to review all
ClusterRoleBindings
○ $kubectl get roles --all-namespaces to review all RoleBindings

R10 - Logging and Monitoring
“Track and monitor all access to network resources and cardholder data”

Forward All Platform Logs
Logs should be forwarded to a central platform for storage and analysis
Conﬁgure forwarding at all three platform layers
• Operations Manager (syslog forwarder)
• PAS Platform (syslog forwarder)
• Apps - Loggregator (nozzles and/or drains)
Activity logging/auditing for privileged users will require 3rd party tools

R11 - Security Scanning/Testing
“Regularly test security systems and processes”

Pivotal Add-Ons
ClamAV
Antivirus for VMs
Scan on-demand or via a
schedule
Conﬁgurable update mirror
Alerts sent to syslog
Helps comply with PCI DSS
and other standards
File Integrity Monitoring
Default policy setup to
monitor a set of critical
system directories.
Alerts sent to syslog
Helps comply with PCI DSS
and other standards
IPsec
Network layer security
strongSwan implementation
of IPsec
Encrypts IP data ﬂow
between hosts

Partner Add-Ons

Auditing Isolation Segments and Application Security Groups

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Auditing Isolation Segments and Application Security Groups

Similar to Auditing Isolation Segments and Application Security Groups (20)

More from VMware Tanzu

More from VMware Tanzu (20)

Recently uploaded

Recently uploaded (20)

Auditing Isolation Segments and Application Security Groups