HERE Technologies enables people, enterprises, and cities around the world to harness the power of location. In this session, you learn how HERE uses JFrog Artifactory with Amazon EFS to deliver close to a million downloads and uploads per day to its CI/CD environment. We walk you through HERE’s AWS process for handling development at scale, and we discuss lessons learned and best practices for success throughout.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
C a s e S t u d y : L e a r n H o w H E R E U s e s J F r o g A r t i f a c t o r y
w i t h A m a z o n E F S t o S u p p o r t M i l l i o n s o f A r t i f a c t s f o r
C I / C D S y s t e m s U s e d b y T h o u s a n d s o f A c t i v e
D e v e l o p e r s
S u r e s h P r e m – P r i n c i p a l S y s t e m s E n g i n e e r , H E R E
Y o a v L a n d m a n – C T O a n d C o - F o u n d e r , J F r o g
Y o n g K i m – S t o r a g e B u s i n e s s D e v e l o p m e n t , A W S
N o v e m b e r 2 7 , 2 0 1 7
S T G 3 1 4

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda:
Amazon Storage and EFS Overview
JFrog and Artifactory Overview
Case Study: HERE Technologies
Reference and Other Sessions
Q&A

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What do you think about
when choosing a storage solution?
EconomicsInterface, semantics,
and permissions
model
Features
and performance

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What do we mean by interface, semantics,
and permissions model?
Interface: How you access your data
Semantics: How your data is represented and updated
Permissions model: How you manage user/group access

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File
Three types of storage
ObjectBlock

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File storage is the most common storage
type for application developers
Interface: Appears as a native file system on a client accessing it, OS APIs,
accessible from multiple clients concurrently
Semantics: Strong data consistency, locking, directory structure
Permissions model: Standard OS model for specifying read/write/execute
permissions for users logged in to the OS

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provides storage as a FS
supporting standard OS
interface, semantics, perms
Multi-attach
How does file compare to the other
types of storage?
Provides access to data
via a simple API
commonly over the
Internet
Provides a virtual disk
that can be formatted to
store data in a file system
or directly as blocks
Single-attach

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File
How does performance compare?
Object
Block
Latency
Scale

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Before EFS…DIY file storageAZ-a
Clients
Storage
Volumes
AZ-b
Storage
Volumes
File Server
File Server

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EFS
A fully managed file service

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key features of Amazon EFS
ScalableSimple Elastic
Highly
durable
Highly
available
Fully
managed
Highly
performant

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Designed for a wide spectrum of needs
Scale-out jobs Metadata-intensive jobs
Analytics
M&E workflows
Web serving
Content management
Database backups
Enterprise apps
& messaging
Dev tooling
Home directories
Container storage
Low latency and serial I/OHigh throughput and parallel I/O

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EFS customers
Web serving + Content management
Media and entertainment workflowsDatabase backups
Big data and analytics
Enterprise applications
Developer tools Home directories Container storage

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security model and security features
Control
network traffic
using VPC security
groups and network
ACLs
Control file and
directory access
using POSIX
permissions
Control administrative
access (API access)
using AWS Identity and
Access Management (IAM)
(action-level and
resource-level
permissions)
Encrypt
data at rest
using keys
managed in AWS
KMS

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EFS economics
$0.30/GB-Month (US Regions)
$0.33/GB-Month (EU Ireland)
$0.36/GB-Month (EU Frankfurt)
$0.36/GB-Month (AP Sydney)
No minimum commitments
or up-front fees
No need to provision
storage in advance
No other fees, charges,
or billing dimensions

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Before EFS…Costs of DIY file storageAZ-a
Clients
Storage
Volumes
AZ-b
Storage
Volumes
File Server
File Server
EC2 instance
costs
Inter-AZ data
transfer costs
EBS volume
costs

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TCO example: For storing 500 GB,
Amazon EFS is 75% less than DIY
EFS cost: (500GB * $0.30/GB-month) = $150 per month
For DIY, you might provision 600 GB of EBS (i.e., ~85% utilization):
Storage (2x 600 GB EBS gp2 volumes): $120 per month
Compute (2x m4.xlarge instances): $350 per month
Inter-AZ data transfer costs (est.): $130 per month
Total $600 per month

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where is EFS available today?
• US West (Oregon)
• US East (N. Virginia)
• US East (Ohio)
• EU (Ireland)
• EU (Frankfurt)
• Asia Pacific (Sydney)
More coming soon!

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data movement
OnlineOffline
Data security
and management
Complete set of data building blocks
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS Cloud Formation
AWS Lambda
Amazon Macie
AWS QuickSight

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introducing EFS File Sync
Connect to data in existing
on-prem or in-cloud file systems
Copy data 5x faster data than
standard Linux copy tools
Fast and simple way to copy file systems to Amazon EFS
EFS file systemEFS File SyncFile system
Set up and manage easily
from the AWS Console

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
JFrog and Artifactory Overview

22. Yoav Landman
CTO and Co-founder at JFrog
@_yoav_
22 © 2017 JFrog

23. Binaries
The
People
23 © 2017 JFrog

24. Managing Binaries
24 © 2017 JFrog

25. JFrog: From GIT to K8S
25 © 2017 JFrog

26. JFrog Artifactory and AWS
JFrog Hosted
Artifactory Cloud
Self-hosted
Install on your own AWS resources
Dedicated Multitenant
© 2017 JFrog26

27. JFrog Artifactory, Universal Binary Repository
“The
DevOps
database”
Universal
+50k
Devs
+15k
Request
Per/Sec
10’s TB
Daily
Transfer
Scaling
© 2017 JFrog27

28. Managing Artifactory Storage
• Checksum-based storage
• Append-only
• Garbage-collected
File A: 46b34
File B: a64ff7
/user-a/repo-z/package-y/file-x
/org-c/repo-m/package-n/file-k
/user-m/repo-w/package-t/file-f
© 2017 JFrog28

29. Extensible Storage Architecture
Binary providers
Simple interface
•
Powerful composition
Eventual
Sharding
Failure
Detection
Retries
Caching
Remoting
Replica-
Aware
© 2017 JFrog29

30. Binary Provider Composition
Redundancy=2
© 2017 JFrog30

31. Repository Replication
© 2017 HEREAWS re:Invent 201731
SAN FRANCISCO
LONDON MOSCOW
BANGALORE

32. Push Replication
© 2017 JFrog32
* Lost events recovery: scheduled task

33. Pull Replication
© 2017 JFrog33
* Lost events recovery: scheduled task/on demand proxy cache-in

34. Never Be Down
• Super devops
ops
users
Automation
© 2017 JFrog34

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Case Study: HERE Technologies

36. HERE Technologies
AWS re:Invent 2017
Suresh Prem
Principal Systems Engineer
November 28, 2017

37. Suresh Prem
HERE Technologies
Principal Systems Engineer
2011 - Present
Contact: Suresh.Prem@here.com
© 2017 HEREAWS re:Invent 201737

38. © 2017 HEREAWS re:Invent 201738
On Demand Economy
9.5 B ARRobotics Quantum Computing
Megacities Artificial Intelligence Sustainability
Neural
Networks
Data Analytics
Location
Location
Location
Location
Location
Location
Location
Location
Location
Location
Location
Location

39. © 2017 HEREAWS re:Invent 201739
200
Countries mapped
4of5
In-car navigation
systems in Europe
and North America
use HERE maps
30+
Years of
experience
transforming
location
technology
7,000+Employees in 56 countries
focused on delivering the world’s best
map and location technologies
HERE Maps on board of
100Mvehicles and counting
3D data points
per second per car
700,000
collecting
data for
our
maps
400
HERE cars
28TB map data
collected
perday

40. Cloud First Strategy
© 2017 HEREAWS re:Invent 201740
Customer and Advanced AWS Partner (APN) for over 8 years.

41. Artifactory High Availability Architecture
AWS re:Invent 201741
Load Balancer
Artifactory 1
Artifactory 2
Artifactory 3
File Store
Database
© 2017 HERE

42. HERE Artifactory with EBS
© 2017 HERE42
Repository
Consumption
Users
2 EC2 InstancesDirect Connect ELB
RDS
CI/CD DevOps
Tools
DuplicateInputs
AWS US-East
AWS EU-West
Disaster Recovery
Artifactory HA Implementation
EBSEC2
AWS re:Invent 2017

43. HERE Artifactory with EFS
AWS re:Invent 201743
Repository
Consumption
Users
2 EC2 InstancesDirect Connect ELB
S3EFSRDS
CI/CD DevOps
Tools
DuplicateInputs
AWS US-East
AWS EU-West
Disaster Recovery
Artifactory HA Implementation
© 2017 HERE

44. Consolidation of various repository managers to Artifactory
© 2017 HEREAWS re:Invent 201744
CI/CD
DevOps Tool 1
Repository
Manager 1
Repository
Manager 2
CI/CD
DevOps Tool 2
HERE Artifactory
Instance

45. © 2017 HEREAWS re:Invent 201745
Amazon Elastic File System is an
Investment in Innovation
No timeout issues Auto scaling Pay for usage
Scalable managed service Static DNS for client access

46. Performance Comparison – AWS EFS vs NFS vs S3
© 2017 HEREAWS re:Invent 201746
62
32
45
24
27
24
Deploy within AWS Deploy from On-Premises Datacenter
EFS (MB/s) NFS (MB/s) S3 (MB/s)

47. © 2017 HEREAWS re:Invent 2017
1.2 Million
Artifacts stored
using
750K+
Downloads
per day
21 TB
of EFS space
1,000+Active users
50K + Uploads
per day
99.5%
SLA
Agreement
300+Repositories
47

48. EFS Metrics via Cloudwatch
© 2017 HEREAWS re:Invent 201748

49. EFS Metrics via Cloudwatch
© 2017 HEREAWS re:Invent 201749

50. HERE Artifactory Backup Solution
© 2017 HEREAWS re:Invent 201750
CI/CD DevOps Tools Artifactory Amazon S3
copy artifacts
Plugin
AWS cli cp
• 1
Artifactory
Move after 6 days
Amazon S3 Amazon Glacier
AWS cli sync
last weekly backup
Amazon EFS
Retention: 365 days
Retention: 1 weekBackup daily
and weekly

51. © 2017 HEREAWS re:Invent 201751
Artifactory with EFS
An enterprise tool enabling speed, agility and innovation.

52. Thank you.
© 2017 HEREAWS re:Invent 201752
Contact: Suresh.Prem@here.com

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to learn more: aws.amazon.com/efs
Feature blogs
Whitepapers
Ref architectures
TCO calculator
10-minute tutorials
Documentation

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
re:Invent EFS Sessions
Monday, 11:30AM STG314: Case Study: Learn How HERE Uses JFrog Artifactory with Amazon EFS
Tuesday, 12:15PM STG401: This Is My Architecture - Lightning Round: Most Innovative Storage Solutions
Tuesday, 2:00PM STG327: Chalk Talk: Learn How to Maximize Throughput to EFS by Using Parallel Data Access Methods
Tuesday, 2:30PM STG324: Learn to Build a Cloud-Scale WordPress Site That Can Keep Up with Unpredictable Changes
Tuesday, 4:45PM STG307: Deep Dive on Amazon Elastic File System (Amazon EFS)
Wednesday, 12:15PM STG325: Case Study: Learn How SiriusXM & Digital ReLab Leveraged Amazon EFS for their Media Workflows
Wednesday, 4:00PM STG310: Files in AWS: Overcoming Storage Challenges for Common File Use Cases - Special Guest Celgene
Thursday, 12:15PM STG307-R: Deep Dive on Amazon Elastic File System (Amazon EFS) [REPEAT]
Thursday, 3:15PM STG315: Case Study: Learn How Thomson Reuters Uses Amazon EFS to Deliver Billions of Pieces of Content
Thursday, 3:15PM STG321: Amazon EFS: Leverage the Power of a Distributed Shared File System in the Cloud [WORKSHOP]
Thursday, 5:30PM STG326: How to Build a Scalable & Highly Available Messaging Service with IBM & TIBCO Using Amazon EFS

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
For Enterprise Storage Engineers
• Learn how to architect and
manage highly available
solutions on AWS storage
services
• Advance toward AWS
certifications
• Help your organization migrate
to the cloud faster
Online at www.aws.training
• Access 100+ new digital
training courses including
advanced training on storage
• Deep Dives on S3, EFS, and EBS
• Migrating and Tiering Storage
to AWS (Hybrid Solutions)
At re:Invent
• Visit Hands-on Labs at the
Venetian
• Attend a proctored
“Introduction to EFS” Spotlight
Lab on Thursday at 3pm at the
Venetian
• Meet Storage experts at the Ask
the Experts in Hands-on Labs
room at the Venetian
New Storage Training

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!