Docker. Does it matter for Java developer ?

•

29 likes•11,403 views

Do you know, being a Java dev, how to manage development environments with less effort? How to achieve continuous delivery using immutable server concept? How to manage set up a cloud within your workstation and many more? It might be the case you know, I bet it's much more easier to do with Docker.

Software

● just baked father
● SA at EPAM Systems
● primary skill is Java
● hands-on-coding with Groovy, Ruby
● trying to get in touch with Erlang/Elixir
● passionate about agile, clean code and devops
Izzet Mustafayev@EPAM Systems
@webdizz webdizz izzetmustafaiev
http://webdizz.name

Agenda
● what is docker?
● concepts
● how to
● dev env
● demo time
● what’s next
● summary
● q&a

Docker https://www.docker.com/
Docker - An open platform for
distributed applications for
developers and sysadmins.
● Develop an app with any language and any
toolchain
● Ship the “Dockerized” app and
dependencies anywhere
● Scale, move between data centers, update
with zero downtime and more

Automation
● Build automation
● Provisioning automation
● Configuration automation
● Delivery Automation

● Environment as a code
● Prohibition of ad-hoc changes
Immutability

Containerisation
● Container as deployment artifact
● Environment agnostic
● New version - new container
● All dependencies built in

Installation
# apt-get update
# wget -qO- https://get.docker.com/ | sh
Assumption: Ubuntu

Dockerfile
FROM phusion/baseimage:0.9.12
RUN apt-get -y update
&& apt-get -y install software-properties-common python-software-
properties
&& add-apt-repository ppa:webupd8team/java
&& apt-get clean
RUN apt-get -y update
&& echo oracle-java8-installer shared/accepted-oracle-license-v1-
select true | /usr/bin/debconf-set-selections
&& apt-get -y install oracle-java8-installer
&& apt-get clean
&& update-alternatives --display java
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
1
3
4
5

Command Line
$ docker build -t webdizz/baseimage-java8 .
$ docker images
$ docker run -d --name=java8
webdizz/baseimage-java8
$ docker ps
$ docker logs java8
$ docker stop java8
$ docker start java8
$ docker top java8

Docker Hub
● User accounts and Authentication
● Public namespaces
● Automated builds
● Registry for images with checksums
● Meta-data store (comments, stars, list public
repositories)

Official images for Java...
● Clojure
● Tomcat
● Jruby
● Maven
● Glassfish
● Elasticsearch
● Jenkins
● SonarQube
● And more non-official

Docker-compose is our way
● YAML syntax
● Configuration inclusion/inheritance
● Comprehensive CLI
● Bash completion

Docker-compose YAML
● image
● command
● links
● ports/expose
● volumes
● environment
● ...

Docker-compose CLI
● build
● up
● logs
● ps
● run
● start/stop
● scale
● ...

Service Discovery
● Configuration registry
● DNS
● Health checking
Tools
● Consul
● Etcd
● Zookeeper

Resource Management
● Scheduling
● Task execution
● Auto scale
Tools
● Mesos
● Flynn
● Fleet

Monitoring
● Failures and Outages are expected for
distributed apps
● Business aware metrics matters
Tools
● Dropwizard Metrics
● Netflix Servo
● cAdvisor

Logging
● Critical for operation
● Even more critical for distributed operation
● Remains critical with operation in Docker
Tools
● Elasticsearch Logstash Kibana
● Elasticsearch Fluentd Kibana

Continuous Delivery
● Container as deployable artifact
● New container for new application version
● Start/stop proper version of container for proper
version of application
Tools
● Ansible
● Docker Swarm

Summary
● Easy and fast building of images
● Easy to share in your team
● Easy to scale
● Easy to work with new technologies
● Of course learning curve
● Production like dev env*

● http://docs.docker.com/
● https://hub.docker.com/
● http://www.fig.sh/
● https://mesosphere.com/
● http://www.consul.io/
● https://flynn.io
● https://github.com/coreos/fleet
● http://goo.gl/NEAfF2
● http://goo.gl/gfi94z
References

Izzet Mustafayev@EPAM Systems
@webdizz webdizz izzetmustafaiev
http://webdizz.name

What's hot

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

PivotalOpenSourceHub

#GeodeSummit Keynote: Creating the Future of Big Data Through 'The Apache Way"

PivotalOpenSourceHub

Application development has come a long way. From client-server, to desktop, to web based applications served by monolithic application servers, the need to serve billions of users and hundreds of devices have become crucial to today's business. Typesafe Reactive Platform helps you to modernize your applications by transforming the most critical parts into microservice-style architectures which support extremely high workloads and allow you to serve millions of end-users.

Typesafe Reactive Platform: Monitoring 1.0, Commercial features and more

Legacy Typesafe (now Lightbend)

At Capital One, we use Spark to detect Fraud. Recently we have started implementing real-time fraud detection using machine learnt models. One of Capital One’s fraud detection micro services was an early adopter of Structured Streaming. As part of this implementation, the micro service ran into several roadblocks. In this talk, we describe those roadblocks and how we got around them. Caching of lookup data Dilemma: Store state in Spark vs store state in database Retrieve state from database efficiently Non-homogenous data sources Aggregations in the stream Checkpointing fumbles Checkpointing performance and instabilities

Whirlpools in the Stream with Jayesh Lalwani

Databricks

Microservices. Streaming data. Event Sourcing and CQRS. Concurrency, routing, self-healing, persistence, clustering… You get the picture. The Akka toolkit makes all of this simple for Java and Scala developers at Amazon, LinkedIn, Starbucks, Verizon and others. So how does Akka provide all these features out of the box? Join Hugh McKee, Akka expert and Developer Advocate at Lightbend, on an illustrated journey that goes deep into how Akka works–from individual Akka actors to fully distributed clusters across multiple datacenters.

Akka A to Z: A Guide To The Industry’s Best Toolkit for Fast Data and Microse...

Lightbend

Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available. In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like: * The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers. * The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements. * How to address the particular needs of stateful services in a model that natively favors stateless, transient services.

Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS

Lightbend

"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer at H2O.ai. Jakub Hava enjoys dealing with problems and learning new programming languages. At H2O.ai, Kuba works on Sparkling Water project and its integration with the rest of the H2O.ai ecosystem. The speech includes a live demo showing how to create a Sparkling Water pipeline with H2O.ai's XGBoost model — no terminal needed, all we need is Jupyter!

"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at ...

Provectus

Full Stack Reactive In Practice

Lightbend

Kafka On YARN (KOYA): An Open Source Initiative to integrate Kafka & YARN

DataWorks Summit

Organizations like Starbucks, HPE, and PayPal (see our customers) have selected the Akka toolkit for their enterprise scale distributed applications; and when it comes to squeezing out the best possible performance, the secret is using two particular modules in tandem: Akka Cluster and Akka Streams. In this webinar by Nolan Grace, Senior Solution Architect at Lightbend, we look at these two Akka modules and discuss the features that will push your application architecture to the next tier of performance. For the full blog post, including the video, visit: https://www.lightbend.com/blog/akka-at-enterprise-scale-performance-tuning-distributed-applications

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Lightbend

Apache Zeppelin & Cluster

Jongyoul Lee

Non-blocking IO to tame distributed systems ー How and why ChatWork uses async...

TanUkkii

From banks to airlines to credit rating agencies, security continues to be a major focus for organizations across various industries. As the newspapers show, it’s heavily damaging to enterprises when security vulnerabilities in their code, infrastructure, or open source frameworks/libraries get exploited. The good news is that your Scala development team now has a powerful ally for securing their applications. Co-developed by the Fortify team along with Lightbend, the upcoming Fortify for Scala Plugin is the only Static Application Security Testing (SAST) solution to use the official Scala compiler. This plugin automatically identifies code-level security vulnerabilities early in the SDLC, so you can confidently and reliably secure your mission-critical Scala-based applications. In this webinar by Seth Tisue, Scala Committer and Senior Scala Engineer at Lightbend, and Poonam Yadav, Product Manager for Fortify at Micro Focus, you will learn about: * Some of the more than 200 vulnerabilities that the Fortify plugin for Scala can catch and help you resolve, * How the plugin works to analyze, identify and provide actionable recommendations, * How to integrate it into your modern DevOps environment, * Why this plugin was co-developed by Lightbend and the Fortify team, and how it benefits your organization’s security professionals / CISO office.

Developing Secure Scala Applications With Fortify For Scala

Lightbend

Introducing Kubernetes

VikRam S

Walmart.com generates millions of events per second. At WalmartLabs, I’m working in a team called the Customer Backbone (CBB), where we wanted to upgrade to a platform capable of processing this event volume in real-time and store the state/knowledge of possibly all the Walmart Customers generated by the processing. Kafka streams’ event-driven architecture seemed like the only obvious choice. However, there are a few challenges w.r.t. Walmart’s scale: – the clusters need to be large and the problems thereof. – infinite retention of changelog topics, wasting valuable disk. – slow stand-by task recovery in case of a node failure (changelog topics have GBs of data) – no repartitioning in Kafka Streams. As part of the event-driven development and addressing the challenges above, I’m going to talk about some bold new ideas we developed as features/patches to Kafka Streams to deal with the scale required at Walmart. – Cold Bootstrap: Where in case of a Kafka Streams node failure, how instead of recovering from the change-log topic, we bootstrap the standby from active’s RocksDB using JSch and zero event loss by careful offset management. – Dynamic Repartitioning: We added support for repartitioning in Kafka Streams where state is distributed among the new partitions. We can now elastically scale to any number of partitions and any number of nodes. – Cloud/Rack/AZ aware task assignment: No active and standby tasks of the same partition are assigned to the same rack. – Decreased Partition Assignment Size: With large clusters like ours (>400 nodes and 3 stream threads per node), the size of Partition Assignment of the KS cluster being few 100MBs, it takes a lot of time to settle a rebalance. Key Takeaways: – Basic understanding of Kafka Streams. – Productionizing Kafka Streams at scale. – Using Kafka Streams as Distributed NoSQL DB.

Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit London 2019

confluent

Speakers: Joe Beda, Co-founder and CTO, Heptio + Gwen Shapira, Principal Data Architect, Confluent With the rapid adoption of microservices, there is a growing need for solutions to manage deployment, resources and data for fleets of microservices. Kubernetes is a resource management framework for containers that is rapidly growing in popularity. Apache Kafka is a streaming platform that makes data accessible to the edges of an organization. It's no wonder the question of running Kafka on Kubernetes keeps coming up! In this online talk, Joe Beda, CTO of Heptio and co-creator of Kubernetes, and Gwen Shapira, principal data architect at Confluent and Kafka PMC member, will help you navigate through the hype, address frequently asked questions and deliver critical information to help you decide if running Kafka on Kubernetes is the right approach for your organization. You will: -Get an introduction to the basic concepts you need to know as you plan to deploy services on Kubernetes. -Learn which parts of the Kafka ecosystem fit Kubernetes like a glove, and which require special attention. -Pick up useful tips for getting started. -See why Confluent Platform for Kubernetes is the simplest solution to deploying and orchestrating Kafka on Kubernetes, using container images and a Kubernetes operator. Watch the recording: https://videos.confluent.io/watch/yoZcuazDjDDTcj1sRnaD3J?.

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

confluent

Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security. A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka. In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

HostedbyConfluent

Lspe

Arpit Tak

Open up your user base to the data! Almost everybody knows how to search. This talk describes through an interactive demo based on open source Hue how users can graphically search their data in Hadoop with Apache Solr. The session will detail how to get started with data indexing in just a few clicks and then explore several data analysis scenarios. The open source Hue search dashboard builder, with its draggable charts and dynamic interface lets any non-technical user look for documents or patterns. Attendees of this talk will learn how to get started with interactive search visualization in their Hadoop cluster.

Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue

gethue

(Mike Graham + Dan Carroll, Comcast) Kafka Summit SF 2018 Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant. In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.

Inside Kafka Streams—Monitoring Comcast’s Outside Plant

confluent

What's hot (20)

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

#GeodeSummit Keynote: Creating the Future of Big Data Through 'The Apache Way"

Typesafe Reactive Platform: Monitoring 1.0, Commercial features and more

Whirlpools in the Stream with Jayesh Lalwani

Akka A to Z: A Guide To The Industry’s Best Toolkit for Fast Data and Microse...

Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS

"Introduction to Sparkling Water" — Jakub Hava, Senior Software Engineer, at ...

Full Stack Reactive In Practice

Kafka On YARN (KOYA): An Open Source Initiative to integrate Kafka & YARN

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Apache Zeppelin & Cluster

Non-blocking IO to tame distributed systems ー How and why ChatWork uses async...

Developing Secure Scala Applications With Fortify For Scala

Introducing Kubernetes

Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit London 2019

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

Lspe

Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue

Inside Kafka Streams—Monitoring Comcast’s Outside Plant

Viewers also liked

Akka in Practice: Designing Actor-based Applications

NLJUG

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

Stephane Manciot

Reactive programming on Android

Tomáš Kypta

Reactive Streams and RabbitMQ

mkiedys

Tachyon and Apache Spark

rhatr

In this presentation you will learn how to leverage the features introduced in Akka Persistence: opt-in at-least-once delivery semantics between actors and the ability to recover application state after a crash. Both are implemented by storing immutable facts in a persisted append-only log. We will show you how to create persistent actors using command and event sourcing, replicate events with reliable communication, scale out and improve resilience with clustering.

Resilient Applications with Akka Persistence - Scaladays 2014

Björn Antonsson

xPatterns is a big data analytics platform as a service that enables a rapid development of enterprise-grade analytical applications. It provides tools, api sets and a management console for building an ELT pipeline with data monitoring and quality gates, a data warehouse for ad-hoc and scheduled querying, analysis, model building and experimentation, tools for exporting data to NoSql and solrCloud cluster for real-time access through low-latency/high-throughput apis as well as dashboard and visualization api/tools leveraging the available data and models. In this presentation we will showcase one of the analytical applications build on top of xPatterns for our largest customer for that runs xPatterns in production on top a data warehouse consisting of several hundreds TB of medical, pharmacy and lab data records consisting of tens of billions of records. We will showcase the xPatterns components in the form of APIs and tools employed throughout the entire lifecycle of this application. The core of the presentation is the evolution of the infrastructure from the Hadoop/Hive stack to the new BDAS Spark, Shark, Mesos and Tachyon, with lessons learned and demos.

xPatterns ... beyond Hadoop (Spark, Shark, Mesos, Tachyon)

Claudiu Barbura

Micro services, reactive manifesto and 12-factors

Dejan Glozic

Viewers also liked (8)

Akka in Practice: Designing Actor-based Applications

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

Reactive programming on Android

Reactive Streams and RabbitMQ

Tachyon and Apache Spark

Resilient Applications with Akka Persistence - Scaladays 2014

xPatterns ... beyond Hadoop (Spark, Shark, Mesos, Tachyon)