Splunk can be used to analyze log data from an online gaming company to help identify issues causing customer complaints. The demo shows how to ingest sample log data, perform searches to find error codes and pages, create alerts, and generate statistics and reports on the data. Dynamic field extraction, pivoting, and over 140 search commands allow transforming and analyzing the data in various ways. Results can be saved as dashboards and applications for ongoing monitoring and insights.
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
Getting Started with Splunk Enterprise - Demo
1. Getting Started with Splunk
Page 1 of 11
Before Demo
Things to do prior to demo:
1. Download latest version of Splunk from http://www.splunk.com/download
2. Download http://docs.splunk.com/images/Tutorial/tutorialdata.zip
Notables
The idea is to introduce the main features of Splunk iteratively as you do the demo, explaining what they are
and how they work, not just simply point them out.
[http://docs.splunk.com/images/e/e5/6.2tutorial_startsearching2.png]
Also, please remember to sprinkle in Splunk’s typical value props as you demo: Real-Time architecture, agile
statistics and reporting, schema on the fly, raw data stored – nothing filtered in the event, time-based series,
etc.
At a high level the customer has been receiving a lot of customer complaints when engaging in their online
sales portal for video games. They would like visibility into what causes issues and be alerted when they do
occur. Total demo time should be about 30 min or less.
Demo
1. Install latest version of Splunk.
2. Start up the new installed instance of Splunk.
If not installing on a Windows based machine please mention it’s installed as a Service. Its start
up status can be modified there.
On *nix platforms the following command can be run to ensure Splunk starts at boot time:
$SPLUNK_HOME/bin/splunk enable boot-start
3. Ingest Tutorial Data
Click Path – Screen Action Say - Description Display
Emulate thatyou download
tutorialdata.zipandshow it’s
contents.
Docs Website where youcan
getthisdata.
Splunkprovidesanonlinetutorial for
gettingdataintoSplunk. It includesa
sample datafile thatcomesfromthe
fictitious“ButterCupGames,Inc.”:A
worldwide game companysellingit’s
productsthroughit’sonline store.
Thiszipfile iscollectionof webaccess
logs,securitylogs,andvendorsales
2. Getting Started with Splunk
Page 2 of 11
generatedfromwebsite
infrastructure.
Loginto newlyinstalledSplunk
instance onyour laptop.
There are twowaysto get to the
appropriate menu. Afteryoulogin
yousee the option “AddData”
-> Settings -> Add Data
OR
-> Settings -> Data Input
There are twomore optionsunderthe
Settingsmenu. Yousee the “Add
Data” image again,but underData
youalso have “Data Inputs”. I’m
goingto clickon “AddData”
ClickUploadaftersayingthe
following:
Besidesuploading datayoualsosee
youcan monitorfiles, use Windows
ManagementInstrumentation(WMI),
TCP/UDP,Scripts,and Modularinput
for external datasources. Splunk’s
Universal Forwarderallowsyouto
securelyandefficientlyforwarddata
fromremote server. We are goingto
choose the uploadoption.
SelectFile -> browse to
tutorialdata -> www3 ->
access.log
OR
Clickand Drag
tutorialdata/www3/access.log
to “Drop your data file here”
ClickgreenNextbuttonattop
of the screen.
Once selected,Splunkwillshow me a
sample of myeventsandmake a best
guessonthe type of data, the
timestamp,and determine if the data
issingle line ormulti-line.Ican
override Splunk’sbestguess,ordefine
my ownsettingsforhow I wantthe
data to be treated withthe user
interface. Thisone time configuration
providessupportfora varietyof out-
of-box source typeswhilegivingyou
the flexibilitytodefine anew source
type basedonany customsourcesyou
may have.
3. Getting Started with Splunk
Page 3 of 11
ClickBack on the Web
Browser. Click and drag
tutorialdata.zip into“Drop
your data file here”.
ClickgreenNextbuttonattop
of the screen
You can eveningestcompressedfiles.
So I letsuploadthe tutorialdata.zipfile
to “bulk”uploadall of this data.
ClickReviewtop of the screen
aftersaying->
Here you can setadditional input
parametersforthisdata.
Sourcetype:TellsSplunkwhatkindof
data youhave,allowingSplunk
categorize yourdata soyou can
searchit easily.
Host: Name of machine whichthe
data originated
Index:A logical container/destination
for yourdata.
→ Apps→ Searching&
Reporting
Withdata ingested we can
immediatelybegintoSearchourdata
and gainmeaningful insight.
4. Search Basics
Letssay you are a Web Site Administrator. Yourecentlyreceivedusercomplaintsthatthatwebpagesare failingandnot
returningcontentwhenitshould. Let’suse Splunktosearchthisdata,to not onlydetermine problems thathappened
but factorsassociatedwithorcontributingtoit.
Search Bar -> * Do thisbefore
yousay ->
At topof the screenyouhave a Search
Bar, similartowhat youwoulduse if
searchingthe Internet. I can simply
type whatI’d like tosearchfor.
Notice thatwhenyoudo searchit’s
across all yourdata, structured,
unstructured,andverylikely
heterogeneous.
Search Bar -> buttercupgames
(returns36,819 events)
SimilartoGoogle,Ican use whole
words,suchas typinginthe word
“buttercupgames”. Pleasenotice that
as youtype Splunkdisplays“Matching
terms”justbelow the searchbar.
Splunkalsodisplaysdifferentwaysto
use searchto return events. Splunk’s
goal is to enable ourcustomerstouse
4. Getting Started with Splunk
Page 4 of 11
manyof the skillstheyalreadyhave
whensearching,makingiteasytodo
while providingquicktime tovalue.
Executingthe search for
“buttercupgames”returnsevents
containingthatword. Splunk returns
eventscontainingthatterm,
highlightingthe terminthe events
returned.
Search Bar -> buttercupgames
403
(returns282 events)
Letssay a customermade a valid
requestbutbuttercupgames web
service simplyfailedtorespond,the
webserverwouldrespondwitha403
code. Expandour search for403.
Whensearchingfortwo termsthe
ANDis implied.
Search Bar -> buttercupgames
403 OR 404
(returns1013 events)
Maybe a webpage resource was
missing,thatwouldbe encodedasa
404, so we can lookfor either403 OR
404
Search Bar -> buttercupgames
40*
(returns5268 events)
Insteadof OR maybe we use a
wildcard,searchfor40*. Notice that
returnstermsstartingwith“40”,
dramatically increasingourresultset.
You can see terms408, 404, 406, etc.
highlighted.
5. Getting Started with Splunk
Page 5 of 11
5. Time Picker
Search Bar -> buttercupgames
403
→ Time Drop Down
All eventsinSplunkare time-based.
Keyingoff time isanotherwayto
enable efficientsearching. Splunk
providesaTime-Picker, givingyou
flexibilitytosearchreal-time data,
relative time ranges,suchasprevious
businessweek,last30 minutes,orall
time. You can alsodefine specific
time or date ranges.
Highlightthe histogram Belowthe searchbar isa histogram
displayingthe frequencyof events.
Thiscan be veryhelpful if I’mlooking
for gapsor spikesincertaintypesof
events. Ialsohave the optionof
zoominginonthe histogramor
focusingall the waydownto
milliseconds.
6. Extracted Fields
Expandon the first event. The real secretsauce to Splunkisit’s
abilitytorecognize andextract
informationcontainwithinthe events.
Splunkwill automaticallypull outany
key/value pairs,IPaddresses,time
and date fields,aswell ascommon
formatssuch as comma,tab delimited
fieldsinacsv file. Splunkdoesthis
while retainingthe entire raw event.
Clickon the value of status –
Add to search
By clickingonany givenvalue youcan
Addit or Exclude itfroma search, or
evenstarta new searchbasedonthat
value. Addingitto our existing
searchyou’ll notice akey-value pair
addedto our search. Now,insteadof
justsearchingfora giventerminany
eventyoucan furtherrefine your
searchto eventswho’sgiven
extractedfieldcontainsaspecific
value.
6. Getting Started with Splunk
Page 6 of 11
Highlightlefthandpane On the left,Ican see ALL the fields
that were dynamicallyextractedand
are available tome forsearchingand
reportingpurposes. Splunkalso
showsthe numberof unique valuesit
foundforeverygivenfielditfound.
-> Smart Mode You can alsoadjustSplunk’s
“discoverymode”forfielddata
extractionduringsearches.
7. Dynamic Field Extraction
a Expand an Event, click on Event Actions -> Extract Fields
b Extract the field after status code, should be the response size.
It will miss some, no worries, just highlight the values missed in events and add to extraction.
8. Alerts
-> Save As -> Alert Now,if a serverrefusestorespondsto
a usersrequest,status=403, Splunk
can detectthat and alertusto it.
ClickReal Time, Provide a
Title,then clickNext
I can choose to constantlymonitorfor
thisinreal-time orto schedule this
alertbasedon a varietyof
frequencies.
7. Getting Started with Splunk
Page 7 of 11
Splunkprovidesflexiblenotification
options,allowingyouto assign
severity tothe Alert, toautomatically
distribute anemail (orSMStext) and
include outputinline orasan
attachment. You can even domore
advancedactionslike runa script
native tothe OS Splunkrunsfor such
thingsas mitigationorremediation.
8. Getting Started with Splunk
Page 8 of 11
9. Statistics and Reporting
Search Bar -> buttercupgames
status=403
→ Statisticstab
Splunkmakesgatheringstatisticsand
reportinga snap. Let’sclickon the
“Statistics”tab underneaththe search
bar.
-> Quick Reports You are presentedwiththreedifferent
options. QuickReportsletsyouclick
on anyfieldfora listof quickreports.
-> uri_path To findoutthose webpages
associatedwiththe serverfailingto
respondtoa request,clickon
uri_path. Here Splunkprovidesyou
withdifferentreportingoptions:Top
Values,TopValuesbyTime,etc.
-> uri_path -> Top values ClickingonTopValuesIget a break
downof the total numberof events
associatedwiththe serverfailingto
respond,brokendownperuri_pathor
webpage.
->Bar -> Pie Reportingisagile,youcaneasily
modifythe reportingvisualization.
Maybe we wanta pie chart insteadof
a bar chart,no problem.
-> Save As -> Dashboard Panel
Fill inDashboardtitle
Enable “SharedinApp”for
DashboardPermissions
Panel Title “FailuresbyWeb
Page”
ClickSave
9. Getting Started with Splunk
Page 9 of 11
ClickViewDashboard
Search -> Search Bar ->
buttercupgames
Clickon Statisticstab, choose
Pivot. ClickOK.
Splunkalsoallowsyoutobuildtables
and visualizationsusingmultiple fields
and metricswithoutwritingsearches.
Pivotautomaticallygeneratesdata
modelsbasedonyourdata, allowing
youto pivotaroundyour data to
extractstatisticsandreports.
-> + (Nextto time filter),
status, then match = 403
We can selectthe status attribute,
thenenterina value of 403
SplitRows -> + -> uri_path,
Add to Table
We can splitthe numberof server
refusedresponsesbyuri_path.
Clickon Horizontal Bar Chart
on reporting panel.
We can leave the datain tabularform
or simply choose another
visualization.
-> Save As -> Dashboard Panel
-> Existing.
10. Getting Started with Splunk
Page 10 of 11
Make sure the previous
Dashboard“ServerFailures”
appears. Provide aModel Title
and ID. ClickSave -> View
Dashboard.
10. Command Language
Search -> Search Bar ->
buttercupgames| stats count
by status
Save As -> Report
There are nearly140 search
commandsthat can be appliedto
data. Whensearchingdata you
isolatedeventsyouare interestedin,
thenapplySplunk’scommandsto
transformdata, furtherreduce data
sets,generate statisticsorperform
analyticsonthe data. To dothiswe
applya pipe character(|) thenissue
the desiredcommand.
I.e.If we wantedto geta countof all
statuscodesfoundinour data, we can
use the stats command,countall
eventswithstatusvalues,thensplit
the count by individual statusvalues.
Letssave that as a report:
Search Bar ->
buttercupgames | stats
count by status | where
status=403 OR status=404
Save As -> Dashboard
Maybe we’re onlyinterestedin403’s
and 404’s? Noproblemletsjustuse a
WHERE commandto isolate those.
Cool. Let’smake thispart of a
dashboard.
11. Getting Started with Splunk
Page 11 of 11
Search Bar ->
buttercupgames | stats
count, sparkline by uri_path
| where status=403 OR
status=404
In additiontocountingyoucan see
trendinginformationbyaddinga
sparkline commandtostats. Heck,
letsevenbreakitdownbywebpage
too. We’ll addthatour dashboard.
Search Bar ->
buttercupgames NOT
status=200| timechart
count by uri_path
What if we want to see anyresponse
that wasn’tOK, basicallyanynon200
code,and we like tosee an actual
time distributionof those. The
timechartfunctionhandlesthatquite
nicely. Because Splunk’s reportingis
agile youcan easilychange the
visualizationonthe fly. Maybe want
stack our valuesfora givenday.
Totallycool! Letssave that to our
dashboard.
Search Bar ->
buttercupgames NOT
status=200 | iplocation
clientip | geostats count
Want to see where yourclientsare
locatedencounteringthese browser
requestsproblems,justuse the
iplocationandgeostatscommands.
Definitelywanttoaddthat to my
dashboard!
These are justa few waysto applythe
over140 commandsavailable in
Splunk.
11. Splunk Applications
https://splunkbase.splunk.com The collectionsof these savedreports,
dashboards,alerts,inputsettings,etc.
iswhat Splunkreferstoas an
Application. There are over600
readilyavailable applicationsonthe
splunkbase website–meantto
provide youinstantvalue byproviding
prebuilddashboards,alerts,and
reportsfor yourdatasources.