Short description

1. Large scale log collection
Guided by
Professor Simon Shim
Team #14
Gaurav Bhardwaj <009297431>
Vaibhav Bhor <009313434>
Sumant Murke <009303879>
Amod Rege <009259692>
CMPE 283: VIRTUALIZATION
TECHNOLOGIES

2. 1. Project Overview
2. Objective
3. Project Part-2
4. Project Part-1 (DRS-DPM)
5. Screenshots
6. Lessons learnt
7. Conclusion
AGENDA

3. Objective
 Manage and test Virtual Machines
 Simulate DRS- DPM functionality
 Develop large scale analysis tool, which collects VM as well as
Host performance data.
 Understand need to Gather and Analyze log Data
 To come up with a framework which provides complete solution
for virtual Machine log file collection & analysis.

4. Design

6. Components
 Agent
 Collector
 Aggregator
 Local storage (mongoDB)
 Central storage (MySQL)
 Visualization

7. Agent
 Uses Java VI api to collect system metrics
 Collects Host as well as Virtual Machine stats
 Writes to a text file every 5 secs
 Takes following parameter VM Name, vHost
Name , y/n
 VM Name => Name of Virtual Machine it has to
monitor,
 y=> to collect stats for both vHost as well as VM,
 n=> to collect only VM stats
 Vhost-Name => Name of vHost it has to monitor
Java -jar Agent.jar “vHost Name” “vm Name”

8. Agent flow

9. Parsing file using LogStash
 LogStash reads log file written by agent,
 For every append in log file it detects and generates
an event, parses each line of log file and stores it in
mongoDB.
 Conf file(logshipper.conf) supplied to LogStash
 Input {file=> ”*.log”}
 Filter {filter=>json}
 Output {output=> mongoDB }
bin/logstash -f logshipper.conf

10. Collector
 Takes IP of all agents
 Connects to local storage of each VM
 Pulls data in a round robin manner
 Clears data from mongoDB after reading
 Stores in MySQL
 Configuration file for connection information
 Automated run every 5 min using crontab
Python collector.py “conf file”

11. Aggregator & Central DB design
 24 hour
 1 hour
 5 minute data
 VM and vHost stats
 Schema

12. DRS-DPM (Part-1)
Initialize the environment and get number of VM's
and host's.
Initialize standard variables vmCount and
hostCount.
If number of virtual machines is greater than
vmCount.
If new machine is powered on.
Move newly added virtual machine to host with
minimum load.
End if
End if
If number of host machines is greater than
hostCount.
If cpu load of new host is less than 30%

13. Is our design good ?
 Agents: will not append will re-write to file
 DataBase (mongoDB)
 Collector:
 Collects data, stores it in MySQL and removes it from
local Storage
 Can connect to as many client specified in conf file
 Aggregator purges main table
 DataBase (MySQL): Aggregator clears the main
table
 Visualization module is totally decoupled from
server and storage

14. Visualization approach
Library
We used canvas.js a Javascript library for
visualization.
 CanvasJS
Used canvas.js to plot the graphs.
We used canvas.js since it is easy to use and
provides different types of visualization.
Data Source: MySQL Database
MySQL database was used from which data
was plotted on the graph.
MySQL was used to get data in structured
format and then plotted on the graph.

15. Output Graphs

16. Output Graphs

17. Output Graphs

18. Output Graphs

19. Output Graphs

20. Tools & Technology
 Agents
- Java VI api
 Collectors
- Python script automated with CRONTAB
 Log file parsing
- LogStash with mongoDB plugin
 Stress api
Manually increase CPU, IO and RAM consumption
 stress --cpu 2 --io 1 --vm 1 --vm-bytes 128M --timeout 10s --verbose
 Visualization tools
 CanvasJS JavaScript Library
 JSP & HTML5
 Programming languages
- Java, Python, JavaScript
 Utilities
 Putty , winscp
 Database
 MySQL
 mongoDB

21. Lessons learnt
 Using VI java api
 Concept behind DRS-DPM.
 Never clone a vHost
 Not every Virtual Machine is Linux
 Automation using CRONTAB
 ESX log files awareness
 Designing systems
 Working with SQL and No-SQL databases and
understanding their usage context

22. THANK YOU...