A journey in everyday queries

1. Kibana:
Real-World Examples
IT Meeting, 17th July 2020

2. Discover UI

3. Discover UI

4. Discover UI

5. Discover UI

6. UI elements:
● Time Picker
● Query Bar
● Result Area
○ Histogram
○ Table
● Index selector
● Available fields
Discover UI

7. UI elements:
● Expanded document
○ Table
○ JSON
Discover UI

8. Kibana Search Types

9. ● Free Text
Queries all fields including _source field
Less performant
Example: GDPR was here. Run. Run away from here
● Field Level
Queries for values of specific fields
More performant/accurate
Example: message:GDPR AND host.name:batch-5
● Filters
Add conditional filters based on fields in log
Always additive
Kibana Search Types

10. Query samples (1)

11. Sample #$i=1:
GDPR was here
Query samples
KQL

12. Sample #$i:
GDPR was here
1. Every word is a search term
2. It assumes implicit OR operator
Query samples
KQL

13. Sample #$i:
GDPR was here
We can translate it like this:
GDPR OR was OR here
Query samples
KQL

14. Sample #++$i:
"GDPR was here"
Query samples
KQL

15. Sample #++$i:
message:"GDPR was here"
Query samples
KQL Kibana supports fields autocomplete

16. Sample #++$i:
source.geo.city_name:Milan OR
source.geo.city_name:Rome
Query samples
KQL

17. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND url.original:/cbr/ws/utils/proxy-list
Query samples
KQL

18. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND http.response.status_code >= 200
AND http.response.status_code < 300
Query samples
KQL Supported operators: : < <= > >=

19. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND url.original:/cbr/api/*
Query samples
KQL

20. Sample #++$i:
NOT source.geo.country_iso_code:IT
Query samples
KQL

21. Sample #++$i:
NOT source.geo.country_iso_code:IT
_exists_:source.geo.country_iso_code
Query samples
KQL

22. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND user_agent.*:Firefox
Query samples
KQL

23. Sample #++$i:
http.response.status_code:*
Query samples
KQL Find all docs where the field exists

24. Kibana Query Language

25. It supports:
● Terms or phrase search
● Rage operators
● Wildcard
● AND, OR, NOT operator
● Grouping (override precedence)
KQL docs is here https://sal.cr/2WaLutc.lnk
Kibana Query Language

26. Advanced Kibana Search Types

27. ● Wildcard/Fuzziness
? to replace a single character
* to replace multiple characters
Example: GD?R
● Proximity
Query word/phrases that are further apart or in a different order
Example: "GDPR here"~1
● Boosting
Manually specify relevance ranking of returned documents
Example: message:Facile^4 mutui^0.1 Cbr^8
Advanced Kibana Search Types

28. ● Ranges
Ranges can be specified for date, numeric, or string fields
Example: http.response.status_code:[200 TO 299]
● Boolean operators
Must +, must not -, AND, OR, NOT
● Regular Expression (Regexp Queries)
Uses regexp term queries (pattern matching)
Domain-specific regexp library (Elastic)
Wrap with forward slashes (/collaboratori/)
Example: url.original://assicurazioni-[a-z]+/preventivo.html/
Advanced Kibana Search Types

29. Query samples (2)

30. Sample #++$i:
/[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/
Query samples
Lucene

31. Sample #++$i:
message:/[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/
Query samples
Lucene

32. Sample #++$i:
url.original://api/email/.*/
Query samples
Lucene

33. Sample #++$i:
url.original://api/email/status/[0-9]+/
Query samples
Lucene

34. Sample #++$i:
/https://[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}:443/api/
Query samples
Lucene

35. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND http.response.status_code:[200 TO 299]
Query samples
Lucene

36. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND http.response.status_code:[200 TO *]
Query samples
Lucene

37. Sample #++$i:
destination.domain:collaboratori.facile.it-443
AND http.response.status_code:[200 TO 300}
Query samples
Lucene

38. Sample #++$i:
destination.domain:collaboratori.facile.it*
AND host.name:web-1?
Query samples
Lucene

39. Sample #++$i:
message:GDRP~
Query samples
Lucene

40. Sample #++$i:
message:GDRP~1
Query samples
Lucene

41. Sample #++$i:
message:"GDPR away"~5
Query samples
Lucene

42. Sample #++$i:
message:"GDPR away"~5
Query samples
Lucene

43. Sample #++$i:
"Sending request" -NOTIFICATION +ivass
Query samples
Lucene

44. Fuzziness

45. Query: collaborator~
Fuzziness uses the Damerau-Levenshtein distance to find all terms with a
maximum of two changes, where a change is the insertion, deletion or
substitution of a single character, or transposition of two adjacent
characters.
The default edit distance is 2, but an edit distance of 1 should be sufficient
to catch 80% of all human misspellings.
More about Damerau-Levenshtein distance https://sal.cr/2ZZooHb.lnk
Fuzziness

46. Proximity searches

47. Fuzzy queries can specify a maximum edit distance for
characters in a word.
A proximity search allows us to specify a maximum edit
distance of words in a phrase.
Proximity searches

48. ● Query: "Simpson Homer"
It expects all of the terms in exactly the same order
● Query: "Simpson Homer"~5
It’s a proximity query
The closer the text in a field is to the original order specified in the query
string, the more relevant that document is considered to be.
"Homer Simpson" is more relevant than "Homer Jay Simpson".
Proximity searches

49. Boolean operators

50. The preferred operators are
● + this term must be present
● - this term must not be present
● all other terms are optional
For example, this query:
assicurazione +zuzu -MUTUI_ANAGRAFICA_ELABORAZIONE
states that:
● zuzu must be present
● MUTUI_ANAGRAFICA_ELABORAZIONE must not be present
● assicurazione is optional, its presence increases the relevance
Boolean operators

51. Lucene Query Language

52. Lucene Query Language
Our query: "GDPR was here"
GET /_search
{
"query": {
"query_string" : {
"query": ""GDPR was here""
}
}
}

53. Lucene Query Language
Our query: "GDPR was here"
GET /_search
{
"query": {
"query_string" : {
"query": ""GDPR was here""
}
}
}

54. Query string syntax supports:
● Field names
● Wildcards
● “Not very” regular expressions
● Fuzziness
● Proximity searches
● Ranges
● Boosting
● Boolean operators
Mixing fuzzy and wildcard operators is not supported
Check query string query docs here https://sal.cr/2ZnEBqB.lnk
Query string syntax

55. ● Elasticsearch uses Apache Lucene's regular expression engine to parse
queries
● Lucene’s regular expression engine does not use the Perl Compatible
Regular Expressions (PCRE) library
For example:
● Lucene’s regular expression engine does not support anchor operators, such
as ^ (beginning of line) or $ (end of line). To match a term, the regular
expression must match the entire string
Regular expression syntax docs is here https://sal.cr/3h2S11h.lnk
“Not very” regular expressions

56. Adding filters using UI

57. Adding filters using UI

58. Adding filters using UI

59. Fields visualization

60. Fields visualization

61. View surrounding documents

62. View surrounding documents

63. A naive approach for search
on Kibana

64. A naive approach for search on Kibana
Choose index

65. A naive approach for search on Kibana
Choose index Set time range

66. A naive approach for search on Kibana
Choose index Set time range

67. A naive approach for search on Kibana
Choose index Set time range
Free text search

68. A naive approach for search on Kibana
Choose index Set time range
Free text search

69. A naive approach for search on Kibana
Choose index Set time range
Free text search
Field search

70. A naive approach for search on Kibana
Choose index Set time range
Free text search
Field search Add filters

71. 1. Choose index
2. Set time range
3. Start with free text search
4. Refine search with field matching and adding filters
5. Inspect surrounding documents
A naive approach for search on Kibana

72. Thanks for your attention