Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. Configure

2. KIBANA4SOLR
Markus Klose
mk@shi-gmbh.com
Head of Training & Consulting

3. Agenda - State Of the Art Logging with Kibana4Solr
•
•
•
•
•
•
Me / SHI GmbH
Motivation of Kibana4Solr
What is Kibana
Project
– What is Kibana4Solr
– Implementation / Architecture
– Pitfalls
– Prospect
Kibana4Solr
– Use cases
– Demo
Q &A

4. Markus Klose / SHI GmbH
•
Head of Training & Consulting
– Project management
– Requirements Engineering
– Consulting
•
•
LucidWorks Certified Lucene / Solr Trainer
LucidWorks Certified Lucene / Solr Developer
•
mk@shi-gmbh.com

5. SHI GmbH - Company
•
•
History
– Founded in 1994
– Since 1995 focus on Search Technologies
– Since 2000 focus on Open Source Solutions
– Since 2011 Gold Partnership with LucidWorks
– Since 2013 Partnership with elasticsearch
Complete Software Service Provider
– Consulting, Engineering, Training, Support
– Open Source Projects
– 5 Consultants, 4 Trainers, 20 Developers
•
100+ years of experience in working with search technologies
•
200+ customers out of different branches

6. SHI GmbH – Search Services
Consulting
Realisation

Search Strategy Consulting

Conception of Search Applications

Search Implementation

Execution of Proof of Concepts


Search Usability Studies
Programming (Extensions, Portals,
Specialized Applications,
Embedded, …)

Architecture & Design Reviews

Sizing & Scaling

Relevancy Tuning

Search Semantics & Lingustics
360° Services
Trainings

Classroom and On-site
Support

Apache Solr Basics, Unleashed &
Under The Hood

Development Support (ExpertLink)

Apache Mahout (planned)

Service Level Agreements (SLA)

Apache Hadoop (planned)

Providing Search Consultants and
Developers

7. Motivation 1
•
In customer projects there is always the need of monitoring data/content and user
data
– „what are the most searched terms?“
– „what hit was selected?“
– „are there searches with no hits?“
– „how are my products categorized?“
– „how many of my products are almost not in stock?“
– …
•
„…is there nothing out of the box?...“ -> NO not with Solr

8. Motivation 2
•
Use and combine existing tools to
offer all our customer a good
analyzing infrastructure
•
Good experience with kibana in our
elasticsearch projects
 idea: enable kibana to use a solr index

9. What is Kibana
•
“Kibana is an open source (Apache Licensed), browser based analytics and
search interface to Logstash and other timestamped data sets stored in
ElasticSearch ... Kibana strives to be easy to get started with, while also being
flexible and powerful”
 visualize logs and time-stamped data
•
Seamless integration with Logstash, Apache Flume, Fluentd and others
logs
Logstash
Elasticsearch
Kibana

10. What is Kibana
•
Create custoum Dasbords BY SEARCHING with
– bar, line diagrams
– scatter plots
– pie charts
– maps

11. What is kibana4solr
•
combines the Power of Kibana and Solr
•
Translator elasticsearch <-> solr
•
•
open source
deployable as war file in all major application servers and servlet containers
What is kibana4solr NOT
•
•
No indexing / crawling tool
No complete monitoring tool

12. The project – kibana4solr / SEMKit
•
•
•
•
•
•
•
•
06/2013 - Idea kibana4solr
07/2013 - POC
08/2013 - Implementation
– Table
09/2013 – idea SEMKit
– Refactoring !!!
– Facetting (query…)
TODAY - kibana4solr/SEMKit is still in progress
11/2013 - more features (histogram facets)
11/2013 - GitHub
…
06/2013
Today

13. basic concept of kibana4solr
Elasticsearch
Kibana
solr4kibana
Solr

14. the bigger picture
•
•
SEMKit – Search Engine Migration Kit
– Framework for smooth search
engine migrations
– Connectors are plugable
– More Connectors can be
integrated easily
kibana4solr is one configuration of the
SEMKit
– Elasticsearch <-> Solr
Solr
ES
FAST
…
FAST
…
SEMKit
Solr
ES

15. Pitfalls & Challanges
•
Minior
– Settings in Kibana
– Settings in Solr
– Tomcat settings
– To much refactoring because the idea change within project
•
Major
– Different syntax and feature spectrum, e.g.
• labels on facets
• …

16. Kibana4Solr Use Cases
•
•
•
Business Aware Monitoring
– Search tracking
– User Tracking - Trends, Conversions, Cancelation
Perfomance Monitoring
– Backend Tracking
– Network Tracking
Data Analysis
– Measure the Development of one Product
– Extract knwoledge form social data
– Sediment Analysis
all that in REAL TIME and NOW with one tool and based on a SOLR INDEX

17. Prospects
•
Kibana4Solr
– Logs to Solr
– RealTimeLogging & analysis
•
SEMKit
– Search Engine Migrations
• Fast2Solr
• …
– Multi-Channel-Monitoring
• Kibana4MySQL
• Kibana4SAP
• …

18. Demo

19. Resources
•
Kibana4Solr/SEMKit Websites
– GitHub
– SHI GmbH
– @SHIEngineers
•
Kibana Websites
– http://www.elasticsearch.org/overview/kibana/
– http://rashidkpc.github.io/Kibana/index.html
•
Q &A

20. KIBANA4SOLR
Markus Klose
mk@shi-gmbh.com
Head of Training & Consulting

21. checklist
•
•
•
Project
Kibana4solr
Spellcheck
•
Links?
(10 slides)
(prepare demonstration)