The basics of SEO are technical accessibility, relevance, quality, and authority. Or: can it be crawled, does it meet a keyword need, and is it trustworthy? In each of these areas, we need to build on solid foundational understanding, and find the areas where advanced understanding will give us an edge. Will’s recent research has shown common gaps in understanding, and highlighted interesting advanced topics. In this wide-ranging session, he guarantees you’ll learn something, and you’ll come away with training guidelines for the basics.

1. SearchLove London 2019
Will Critchlow (@willcritchlow), CEO, Distilled

2. The design flaw that almost wiped out a NYC skyscraper [New Yorker]

3. Shout out to
@DianeHartley who
was the student, who
didn’t realise until
years later that her
call had made any
difference

8. Some
purely
theoretical

9. Some
useful and
practical
Some
purely
theoretical

10. Twitter thread reference

16. User-agent: *
Disallow: /mydir/

17. /mydir/

18. User-agent: googlebot
Disallow: /mydir/

19. /mydir/

20. User-agent: *
Disallow: /mydir/
User-agent: googlebot
Disallow: /secret/

21. /mydir/

22. User-agent: *
Disallow: /mydir/
User-agent: googlebot
Disallow: /secret/

24. WRONG

25. /mydir/

26. User-agent: *
Disallow: /mydir/
User-agent: googlebot
Disallow: /secret/

27. User-agent: *
Disallow: /mydir/
User-agent: googlebot
Disallow: /secret/

32. User-agent: *
Disallow: /a/
User-agent: googlebot
Disallow: /b/

33. Adsbot-Google

34. User-agent: *
Disallow: /a/
User-agent: googlebot
Disallow: /b/

37. User-agent: *
Disallow: /a/
User-agent: googlebot
Disallow: /b/

38. not

39. User-agent: *
Disallow: /a/
User-agent: googlebot
Disallow: /b/

41. User-agent: *
Disallow: /a/
User-agent: googlebot
Disallow: /b/

42. *

43. both

44. User-agent: Adsbot-Google
Disallow: /a/
Disallow: /b/

48. googlebot/1.2
googlebot*
googlebot
Source: https://developers.google.com/search/reference/robots_txt

49. User-agent:
googlebot/1.2

51. THIS IS WRONG

52. Luckily released their
parser with source:
“The library is slightly
modified (i.e. some internal
headers and equivalent
symbols) production code
used by Googlebot”

53. /*static*/ absl::string_view RobotsMatcher::ExtractUserAgent(
absl::string_view user_agent) {
// Allowed characters in user-agent are [a-zA-Z_-].
const char* end = user_agent.data();
while (absl::ascii_isalpha(*end) || *end == '-' || *end == '_') {
++end;
}
return user_agent.substr(0, end - user_agent.data());
}
Source: open source robots.txt parser

54. /*static*/ absl::string_view RobotsMatcher::ExtractUserAgent(
absl::string_view user_agent) {
// Allowed characters in user-agent are [a-zA-Z_-].
const char* end = user_agent.data();
while (absl::ascii_isalpha(*end) || *end == '-' || *end == '_') {
++end;
}
return user_agent.substr(0, end - user_agent.data());
}
Source: open source robots.txt parser

55. // Allowed characters in user-agent
are [a-zA-Z_-].
Source: open source robots.txt parser

56. User-agent:
googlebot/1.2

57. User-agent:
googlebot/three

58. User-agent:
googlebot-1.2

59. User-agent:
googlebo

60. User-agent:
google

61. User-agent:
googlebotthethird

62. Mozilla/5.0 AppleWebKit/537.36
(KHTML, like Gecko; compatible;
Googlebot/2.1;
+http://www.google.com/bot.html)
Chrome/W.X.Y.Z Safari/537.36
Source: updating the user agent of Googlebot

63. Mozilla/5.0 AppleWebKit/537.36
(KHTML, like Gecko; compatible;
Googlebot/2.1;
+http://www.google.com/bot.html)
Chrome/W.X.Y.Z Safari/537.36
Source: updating the user agent of Googlebot
The “token”

66. https://support.google.com/webmasters/answer/6062596?hl=en

71. https://support.google.com/webmasters/answer/1061943?hl=en

72. https://support.google.com/webmasters/answer/1061943?hl=en

81. ./robots robots.txt googlebot-image /c/
Source: open source robots.txt parser

82. ./robots robots.txt googlebot-image /c/
Source: open source robots.txt parser

83. ./robots robots.txt googlebot-image /c/
Source: open source robots.txt parser

84. ./robots robots.txt googlebot-image /c/
Source: open source robots.txt parser

85. ./robots robots.txt googlebot-image /c/
user-agent 'googlebot-image' with URI
'/c/': ALLOWED
Source: open source robots.txt parser

86. ./robots robots.txt googlebot-image /c/
user-agent 'googlebot-image' with URI
'/c/': ALLOWED
Source: open source robots.txt parser

87. ./robots robots.txt googlebot-image /c/
user-agent 'googlebot-image' with URI
'/c/': ALLOWED
Source: open source robots.txt parser
��

88. Docs Online
tool
Open
source

89. Docs Online
tool
Open
source
Googlebot/1.2

90. Docs Online
tool
Open
source
Rule ordering

91. Docs Online
tool
Open
source
Other gbots

92. Docs Online
tool
Open
source
Other gbots
��

93. Docs Online
tool
Open
source

94. Official Google announcement

95. Gary Illyes at Pubcon

96. Docs Online
tool
Open
source

97. ===
User-Agent: googlebot
Disallow: /foo.png
===
Target:
/foo.png
Bot:
Googlebot-Image

98. ===
User-Agent: googlebot
Disallow: /foo.png
===
Target:
/foo.png
Bot:
Googlebot-Image
Results:
Documentation: SHOULD BE BLOCKED

99. ===
User-Agent: googlebot
Disallow: /foo.png
===
Target:
/foo.png
Bot:
Googlebot-Image
Results:
Documentation: SHOULD BE BLOCKED
Online Tool: BLOCKED

100. ===
User-Agent: googlebot
Disallow: /foo.png
===
Target:
/foo.png
Bot:
Googlebot-Image
Results:
Documentation: SHOULD BE BLOCKED
Online Tool: BLOCKED
Open Source Parser: ALLOWED

101. ===
User-Agent: googlebot
Disallow: /foo.png
===
Target:
/foo.png
Bot:
Googlebot-Image
Results:
Documentation: SHOULD BE BLOCKED
Online Tool: BLOCKED
Open Source Parser: ALLOWED
Actual Googlebot: BLOCKED

102. Docs Online tool
Open source
Treatment of
googlebot-image
googlebot-news
etc
��
Actual gbot

107. Twitter threads: reference 1 and reference 2

108. 19 50

109. 100 19 50

115. @PeterSokolowski via Lisa Schneider from Merriam Webster at SearchLove

116. OK. But.
1. We appropriated
this language

117. THE TAIL IS LITERALLY LONG

118. And 2. It has a meaning in the context of a long tail
strategy

119. A plan to have huge numbers of pages indexed
with differentiated content and enough
authority to rank widely

122. Low
volume
Lots of
words
Uncom-
petitive

123. Target uncompetitive queries can be a strategy
for smaller businesses.
Low
volume
Lots of
words
Uncomp-
etitive

126. x%

129. WRONG

132. Thanks to @KaneJamison for the reference link
Collection
frequency (cf)
Document
frequency (df)
try 10,422
insurance

133. Thanks to @KaneJamison for the reference link
Collection
frequency (cf)
Document
frequency (df)
try 10,422
insurance 10,440

134. Thanks to @KaneJamison for the reference link
Collection
frequency (cf)
Document
frequency (df)
try 10,422 8,760
insurance 10,440

135. Thanks to @KaneJamison for the reference link
Collection
frequency (cf)
Document
frequency (df)
try 10,422 8,760
insurance 10,440 3,997

136. Thanks to @KaneJamison for the reference link
Inverse
collection
frequency (icf)
Inverse
document
frequency (idf)
try 0.000096 0.000114
insurance 0.000096 0.000250

137. Thanks to @KaneJamison for the reference link
Inverse
collection
frequency (icf)
Inverse
document
frequency (idf)
try 0.000096 0.000114
insurance 0.000096 0.000250

139. Twitter thread reference

145. y

149. 0

150. WRONG

151. 0.25
0.25
0.25
0.25

152. 0.25
0.25
0.25
0.25
0.15 / # pages

155. y

158. y

166. Twitter thread reference - more detail in @tomanthony’s presentation

167. google.com

168. google.com
googleusercontent.com

172. RIGHT

173. httpOnly

175. WRONG

177. distilled.net

178. <script
src="https://example.com/widget.js">
</script>

179. 🍪🍪🍪

181. WRONG

189. Twitter search:
[#SEOquiz from:willcritchlow]
Or go here to find the thread that pulls them all together.

192. ● Citicorp Center
● Spider web
● Ice cream
● Lightning
● Robot
● Gate
● St. John’s
● Who wants to be a millionaire
● Fire
● Long tail
● Lisa Schneider
● Confused
● Hay stack
● Graph
● Cookies
● London