A quick introduction to BigTop's VM and Docker provisioner

1. BigTop
VM and Docker
Provisioner
Evans Ye
2015.2.17

2. Apache Bigtop is a project for…
• Packaging
• Packages Hadoop ecosystem
• Testing
• integration/interoperability testing framework
• Virtualization
• vagrant/docker based hadoop provisioning
• Deployment
• puppet recipes for included components

3. What does vagrant-puppet do?

4. 1) Create a set of VMs by vagrant box
2/17/2015
Vagrant Box
(Centos, Ubuntu,…)
Virtualbox
VMs

5. 2) Install puppet agents
2/17/2015

6. 3) Deploy Hadoop by bigtop puppet
(masterless)
2/17/2015
puppet apply -d
--modulepath="bigtop-deploy/puppet/modules:/etc/puppet/modules"
bigtop-deploy/puppet/manifests/site.pp
puppet apply…
puppet apply…
puppet apply…

7. One click auto deployment
2/17/2015
$ cd bigtop-deploy/vm/vagrant-puppet-vm
$ vagrant up

8. And the docker-puppet?

9. Similar to vagrant-puppet
but spin up a cluster on top of Linux
containers
2/17/2015
$ cd bigtop-deploy/vm/vagrant-puppet-docker
$ ./docker-hadoop.sh --build-image --create 3

10. Some prerequisites (see each README.md)
vagrant-puppet docker-puppet
2/17/2015 Confidential | Copyright 2013 TrendMicro Inc. 2

11. Use cases
• for BigTop
• to do bigtop integration tests
• to test bigtop puppet recipes
• integrates packaging, deployment, and testing all into one package
(demonstrate the BigTop life cycle)
• for Contributors & Users
• run round trip test locally (from source code to a testing cluster)
• easy-to-get Hadoop cluster environment
2/17/2015 Confidential | Copyright 2013 TrendMicro Inc. 2

12. Execute BigTop smoke-tests
• vagrant-puppet
• README.md
• vagrantconfig.yaml
• set run_smoke_tests: true
• Vagrantfile
• docker-puppet
• ./docker-hadoop.sh --smoke-tests
2/17/2015 Confidential | Copyright 2013 TrendMicro Inc. 2