A presentation I did on what, why, how, and benefits of centralized logging in the Enterprise. This presentation was focused on implementing centralized logging in a environment that is mostly .NET/Windows.
3. How does logging
provide value?
We can gather information on the system while it’s in
operation. We can use this information to provide more
value to the business.
4. More Value?
Information gathered from logs allows you to
proactively take actions that will keep systems
providing the most value to the business.
5. Talk in circles much?
Centralized logging provides visibility into the
operational efficiency of the system and process.
7. How is it done?
Logs for most/all systems are
shipped, parsed and stored in a
central location for monitoring and
decision making.
8. Um…sounds like a bunch of
work.
IIS Logs
SQL Logs
Router Logs
Etc.
9. Initial focus is on application logging for in-house applications.
10. Where we are…
System logs
file to
directory
System logs
file to
database
?
System logs
file to
directory
Where many are…we’re not alone.
11. Where we want to go.
Shipper
Systems
Parsing, Indexing, St
orage
Logger
Viewer
App
A system that has the ability to store and analyze log
data from any system that provides it.
19. Windows Event Log
• Low Cost – Built in.
• Built in UI
• Can forward logs to a central server
20. Windows Event Log
• Event entries stored in memory.
• Have to configure logging per server.
• Service interface for other clients to
hook into not available.
21. Log to central SQL database
• Low Cost – SQL in
house, create an
database/table.
22. Log to central SQL database
• Relational (Schema based) format
challenging to use for
unstructured data.
• logging straight to database
could degrade application
performance.
32. Inverted Index
“In computer science, an inverted index (also referred to as postings file or inverted file) is anindex data
structure storing a mapping from content, such as words or numbers, to its locations in adatabase file, or in a
document or a set of documents. The purpose of an inverted index is to allow fast full text searches, at a cost of
increased processing when a document is added to the database.” via http://en.wikipedia.org/wiki/Inverted_index
Word
Word
Word
Page
Page
Page
Page
Word
Regular Index
Inverted Index
33. Documented Oriented
• No need to define schema upfront
• Can store, index and quickly search unstructured data.
• Schema can be defined per type for customization of
the indexing process.
35. Easy to scale
• Distributed by nature.
• Indexes broken down by shards with 0 or more
replicas.
36. Easy Level 1 Rest API
• Well documented and straight forward api, which
makes it easy to build a client for it.
37. What’s Needed recap
•
•
•
•
Server for Elastic Search (Windows/Linux)
Redis Server
Windows service to index logs from queue.
Modify Enterprise Library Logger to send to queue.
38. In closing
Gathering the data is one part. Logging enough
information and knowing what questions your are
looking to answer is another part.