This document discusses how to analyze search engine crawler logs to optimize a website for crawling. It recommends checking which search engine robots crawl the site, which pages they visit most frequently, the status codes they receive, and any errors. It also provides tips for technical improvements like infrastructure optimization and content guidelines like disallowing thin or duplicate content from being crawled. The overall goal is to allocate the search engine's crawl budget efficiently to increase organic traffic.

1. GetINSIDE THE SEARCH
ENGINE CRAWLER HEAD
Filip Podstavec

4. Dr.

5. Supermodel gynecologist

6. PSYCHOLOGISTSPSYCHOLOGISTSPSYCHOLOGISTS

11. https://orig14.deviantart.net/9386/f/2015/100/e/8/mixels_cc__1_dizzy_robot_
by_supercoco142-d8p7kph.png

12. Goal:
FASTEST

13. FASTEST
MOST ACCESSIBLE
Goal:

14. FASTEST
MOST ACCESSIBLE
RICHEST
Goal:

15. Why?

16. Higher crawl rate
Benefits

17. Higher crawl rate
More indexed pages
Benefits

18. Higher crawl rate
More indexed pages
Make website faster
Benefits

19. Higher crawl rate
More indexed pages
Make website faster
Cleaner architecture
Benefits

20. Higher crawl rate
More indexed pages
Make website faster
Cleaner architecture
Fixed error pages
Benefits

21. January 2016 April 2016 July 2016 October 2016 January 2017
Organic traffic boost
Start of crawl budget optimization

22. Higher crawl rate !
Direct ranking signal
=

25. How?

26. Make diagnosis

27. Medical History
12.6.2010 | Filip Podstavec | sick | height:
183cm | 120/60
17.9.2010 | Filip Podstavec | sick (again) |
height: 184cm | 120/60
06.4.2014 | Filip Podstavec | diarrhea | height:
184cm | 120/60
Filip Podstavec | aneurisma|
0

30. Medical History
12.6.2010 | Filip Podstavec | sick | height:
183cm | 120/60
17.9.2010 | Filip Podstavec | sick (again) |
height: 184cm | 120/60
06.4.2014 | Filip Podstavec | diarrhea | height:
184cm | 120/60
Filip Podstavec | aneurisma|
0

32. 155.62.100.122 - - [25/Aug/2017:08:22:55
-0400] “GET /category/mktfest/ HTTP/1.1” 200
“-” “Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)”

33. 155.62.100.122
[25/Aug/2017:08:22:55 -0400]
GET /category/mktfest/ HTTP/1.1
200
Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

34. IP (WHO)
Timestamp (WHEN)
Method and requested URL (WHAT)
Status code (SUCCESSFULLY OR NOT)
User-agent (DETAILS)

35. “The biggest source
of website URLs”

36. Web
Logs

37. Web
Logs

38. Web
Logs

39. The most accurate source about
crawl budget

40. What is the crawl budget?

41. Crawl budget is...

42. How does a Googlebot allocate
the crawl budget for your website?

43. How much time it allocates:
Authority * Amount of acquired new info

44. How do you use that time:
Speed

45. How do you care
about future crawling:
Internal linking

46. “People choose the paths that
grant them the greatest rewards
for the least amount of effort. “
Dr. House / David Shore

47. “Crawlers choose the paths that
grant them the greatest rewards
for the least amount of effort.“
Filip Podstavec

48. “Crawlers choose the paths that
grant them the greatest rewards
for the least amount of effort.“
Filip Podstavec

49. Do I have logs?
Yes, but I don’t know about that
No
Yes

50. mktfest_com.log (146.55 GB)
podstavec_cz.log (11.07 GB)

51. Tools for log analysis

52. Static
Real-time
vs.

53. Static

54. < 50MB (Small Files)

55. 50MB - 10GB (Medium Files)
Google BigQuery OpenRefine
Screaming Frog
Log Analyzer

56. > 10 GB (Large Files)
Google BigQuery

57. Real-time

59. Real-time logs

60. = 20 minutes
= 20$ / m
+
Server

61. Bit.ly/elkresources

62. What to check in logs:
(diagnosis)

63. #1 Which search engine robots do crawl
my website and how much?

64. Bot requests in last 7 days
Googlebot
Bingbot
Slurp
SeznamBot
Other

65. Bot requests in last 7 days
#1 Bot requests
Googlebot
Bingbot
Slurp
SeznamBot
Other
Googlebot
Bingbot
Slurp
SeznamBot
Other

66. DESKTOP : MOBILE : IMAGE
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.
google.com/bot.html)
Googlebot-Image/1.0
​Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safa-
ri/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

67. #2 Googlebot agents
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.
google.com/bot.html)
Googlebot-Image/1.0
​Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safa-
ri/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

68. #3 Which pages do they visit
with the highest frequency?

69. Most visited URLs by Googlebot

70. Most visited URLs by Googlebot
#3 Most visited URLs

71. Most crawled URL?

72. .txt

73. #4 Which status codes and how much
of them does a Googlebot crawl?

74. 200
404
301
302
500

75. #4 Pie chart of Googlebot status codes
200
404
301
302
500
200
404
301
302
500

76. 200
404
301
302
500

77. Where/when/why do they
request error pages?

78. Googlebot hits with status code higher than 399

79. #5 Googlebot errors

80. #6 User errors

81. Does somebody crawl
my website?

82. Of your traffic are bots!
61.8%

87. #7 IP Requests
#8 Googlebot IPs

89. What can you do?
(content point of view)

90. Content

91. is

93. What can you do?
(technical point of view)

96. 503Service unavailable

98. “Be in touch with somebody
responsible for infrastructure!”
Filip Podstavec

99. Communicate
with your
sysadmin

100. https://twitter.com/dimensionmedia/status/877513185238151168

101. DISALLOW

102. “Not all of your URLs should
be crawled or indexed”

103. Links

104. URL1 URL2
URL3
URL4
URL5

105. <a href=”a/”>
/a/ /a/a/ /a/a/a/ /a/a/a/a/

106. Sanitize the pagination

108. Rel=”prev”
<link rel=”prev” href=”http://www.example.com/page1” />
Rel=”next”
<link rel=”next” href=”http://www.example.com/page3” />
Noindex,follow
<meta name=”robots” content=”noindex, follow”>

109. Sanitize the filter combinations

110. http://edition.cnn.com/videos/tech/2016/08/26/black-hole-breakthrough-lee-pkg.cnn

111. 10 filters
25 variants

112. 95 367 431 640 625
Variants

114. 3.8 Years

115. How to fix that?
Create rules like:

116. Disallow filters without search
volume (price, etc.)
Example: Eshop.tld/notebooks/price-0-1000/
Robots.txt block: Disallow: */price

117. Disallow more than one used filter
from each segment
Example: Eshop.tld/notebook/acer,apple,lenovo/
Robots.txt block: Disallow: /*,*,*

118. Delete unused filters by users

119. Use pseudo checkbox
Example: <input><label><a>Select link</a></label>
Example URL: https://notebooky.heureka.cz/

120. Avoid thin content crawling

122. Disallow: /directory/
<meta name=”robots”
content=”noindex”>

123. Sometimes speed matters

124. Google PageSpeed Insights

125. Google Lighthouse

126. https://varvy.com/

127. Keep your sitemap clean

129. Are you happy?

132. Thank you!
Filip Podstavec
THE MAIN CONSTRUCTER OF MARKETING MINER
FILIP@MARKETINGMINER.COM