SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs

DEVELOPER
THINKING FOR
SEOS
MICHAEL KING
MANAGING DIRECTOR, IPULLRANK
@IPULLRANK

Baby Sign Language
…and how it relates to technical SEO

IPULLRANK.COM @ IPULLRANK
So, I’m Having a Baby (She is too)…
It’s a girl!

Last Night I Learned Infants Can Learn Sign Language
…

Which is Great Because They Act like Drunks Otherwise
…

Which is Astonishingly
Similar to the
Perception of
Marketers

SEOs generally know what
they want, just not how to
effectively ask for it.

As of Late People Started Saying You Don’t Need to Be Technical

Yet, the Web is More Technical than Ever Before

The modern SEO needs to
think like a developer
because we are in a technical
SEO renaissance.

Lucky for You This Is The Point Where I have My Own Agency
Shameless plug.

Changes in Web
Technologies
What is pushing this renaissance?

JavaScript is Not Going Away
Nearly every site on the web is using JavaScript to perform some type of transformation on the page. It’s
obvious that Google needed to solve for this.

Growth of AngularJS
AngularJS, the single page
application, MVW
framework continues to
grow dramatically in the top
million websites.

AngularJS is Popular in Most Verticals
It’s likely that you’ll run into an AngularJS site soon enough in your optimization duties

ReactJS is Growing Too
Similarly, Facebook’s MVW
framework is growing as
well. Although, it has a built
in feature for making sure
that it renders proper HTML,
but it’s still a framework you
should know how to
optimize for.
Check out this post https://builtvisible.com/react-js-seo/

Ironically, React (Facebook) is SEO-ready, Angular (Google) is Not
Angular 1 did not have anything to make it easier to crawl. React has ReactDOMServer.renderToString to allow React to serve
fully rendered HTML from the server.

HTTP/2 – Perhaps the Most Important Thing
All you need to know right now is that it’s fast as hell.

Adoption is Super Low For Now Though

We Need More from Our
SEO Tools
Many of them are far behind search engines

So, You’re #1? Cool Story, Bro.
Where they
at tho?

On What Device, OS and Location?
Different browser and OS configurations will show you different results. So what truly is a mobile
ranking?
+ + = Ranking #3+
Nexus 5 Android NYCChrome
Safari
+ + = Ranking #1+
iPhone 6 iOS NYC

Are we measuring for context
and actionability or just
vanity?

Please Give Me Position 0 in My Rankings

Please Give Me My Results in Context of Paid
#1 is really #8
in this SERP

304 Response Codes
The Not-Modified response code tells clients that they do not need to download a page again if it has already been downloaded.

Googlebot Respects the 304
What is cloaking when Google accepts 304 response codes and does not revisit that content after
indexation?

What is Cloaking in the Adaptive/Responsive Era?
Many people show less or more content based on the context of the browser/user, so what is
cloaking now?

Google accepts directives in
HTTP headers.

X-Robots-Tag HTTP Header
Many of the popular crawling tools do not look for this in the response headers.

Hreflang HTTP Header
I have not seen a tool that looks for this HTTP header. However, this implementation is preferred over the many lines
of hreflang tags on every page.

Rel-Canonical HTTP Header
Many tools do not look for the rel-canonical HTTP header.

All of these HTTP headers are
better than clogging up the
<head> of the code.

JavaScript Redirects for Mobile
Google allows client side redirects for automation redirection of mobile to desktop and vice versa. Many crawling tools do not
account for this.

Many SEO tools do not
account for these things

Chrome DevTools Does
Natively in Chrome’s Developer Tools you can see all of the HTTP Request and
Response Headers in the Network tab section.

Geolocation Emulation is Built-in to Chrome
If you’re running into issues with finding local rankings now that Google has gotten rid of
nearEquals you can just change your location on the browser level.

Remote Debugging
You can also see this for a specific mobile devices with the remote debugging features of Chrome.

Crawlability
What can be crawled at this point?

Google is Crawling Headless
Make no mistake that Google has not been crawling with just text driven crawlers for a long time.

Adam Audette Said It So You KNOW It’s True
Adam ran a few tests a little under a year ago further proving out these capabilities

Also an Ex-Googler Confirmed It
A former Google developer said the headless crawling was his primary project from 2006 – 2010.

I Said it in 2011

JsCrawlability.com
Eric Wu has been doing tests to get a better determination of what it is that Google can actually see.
http://www.jscrawlability.com

But Mike, what does it
matter?

Well I Captured Video of Headless Googlebot
It’s a question of what does Googlebot do on your site. Using LuckyOrange I captured Googlebot sessions from Search Console’s Fetch and
Render and then I submitted it to the index. They load the page and leave. The mouse never moves, the headless browser never came back.

Get a Free Trial of LuckyOrange and Try it Yourself
This code will only place LuckyOrange on the page if a JS-enabled Googlebot comes to your site.
My test did not show what Googlebot from the wild does when it’s crawling.

Log Files Went out of Style?
No, log file analysis has not gone out of style. In fact, it’s more valuable than ever before.
LOOK AT YOUR LOG FILES!

Log File Analysis
We parse the logs and query them in MySQL, but you can use any number of tools for this.

Screaming Frog Log File Analyzer
The Screaming Frog team rolled out an awesome new tool recently for log file analysis.

Search Console’s Crawl Stats are Mediocre
I personally find what Google gives us to be pretty much useless. Don’t less this take the place of
going through your logs.

Log Files Help You Spot What Really Happens
I had a client come last year that was unsure whether it was Penguin or changes in their offline media
buying habits had led to the downturn in traffic. Layering the Googlebot logs made it clear.
0
50000000
100000000
150000000
200000000
250000000
0
500000
1000000
1500000
2000000
2500000
Unknown organic organic home organic dyor
organic er organic home,dyor,er organic cushion organic net of home
organic ritani organic diamonds direct home paid brand sessions
tv spend web sessions (as reported by adaptv) Top 800 Monthly SV impressions
Googlebot Visits
Penguin Update

Social Shares Influence Crawl Patterns
We find that there is a higher correlation with crawling and social shares than crawling and links. Commonly
held SEO knowledge would make you believe this is entirely dictated by links.
0
1000000
2000000
3000000
4000000
5000000
6000000
0
100000
200000
300000
400000
500000
600000
700000
Googlebot Visits Links Social Shares FB Twitter G+

Prerendering is the Best Practice
Using a solution like prerender.io is the best practice.

Prerender Can Cause Problems
However, we’ve seen Prerender setups misidentify Googlebot as a human users and serve the wrong the
AngularJS versions.
0
20000
40000
60000
80000
100000
120000
Googlebot Perceived as Human

Prerender Setup Serving Non-200
Prepping the prerender cache can cause outages and lead to pages falling in and out of the index.
0
2000
4000
6000
8000
10000
12000
14000
16000
206
301
302
304
403
404
405
410
500
503
504
206
301
302
304
403
404
410
503
504
206
301
302
304
404
405
410
500
503
504
206
301
302
304
401
404
405
410
500
503
504
206
301
302
304
403
404
405
410
502
503
504
1 2 3 4 5
-
hit
miss
pass

Also Prerendering is Not a Requirement
We got rid of prerender on 2 sections of the site and saw the traffic from Organic Search increase YoY.
Google can index it just fine.

AJAX Crawling Deprecated
Google has gotten rid of the AJAX crawling scheme because they don’t need it anymore. They have perfected their
ability to crawl headless at reasonable scale. They say don’t block the CSS or JS because it’s a browser.

They Say Don’t Prerender Just for Google

Prerender Does Speed Up Crawling
There is certainly value in prerendering for speed, but then that makes me wonder, why not just not
use Angular?

A Better Idea Is a Library that Doesn’t Need Prerender
Let’s stop going after the shiny object and go with the things that are built to work, shall we? Check
out MeteorJS – http://www.meteor.com

Or go with Angular2
Brad Green is the Google engineering product manager for AngularJS. He says Angular2 is better in
every way.

Don’t Forget Google Wants Progressive Enhancement
Thinking with the IoT in mind, progressive enhancement still makes sense. However, frameworks
like AngularJS are completely contradictory in that they don’t offer graceful degradation.

Scraping
The backbone of a lot of our analysis as SEOs

Most Tools use cURL or Similar
There are many similar libraries, but many SEO tools are built on library called cURL for managing HTTP requests. This
is a text based crawler.

This is a fundamental flaw of
many SEO tools because they
are not constructing the DOM
and its CSS/JS
transformations. Search
Engines are.

PhantomJS – For Scraping the Modern Web
There are many headless browsers out there such as Selenium, but PhantomJS is the go to.
http://www.phantomjs.org

HorsemanJS – The Easy PhantomJS
I prefer HorsemanJS which is a NodeJS module that is incredibly simple. http://www.horsemanjs.org

In-Browser Scraping
I wrote a post on how you can scrape using your browser’s Console.

Scrape the Unscrapable
Nothing is in unscrapable when you’re using a browser.

Multi-Page In-browser Scraping
If you want to scrape multiple pages check out ArtooJS https://medialab.github.io/artoo/

WebScraper.io
WebScraper is another high-powered in-browser scraping tool. No code required. http://www.webscraper.io

Or You Can Make Screaming Frog Headless
You can make Screaming Frog headless by placing a proxy with a headless browser in front of it. From my initial findings, there
is not much different aside from the order it identifies URLs and its ability to scrape content from pages.

Here’s the Code
Here’s the node code for how I did it. HorsemanJS is not the best tool for the proxy, but it’s great for a proof
of concept.
Screaming Frog pings for
robots.txt a lot. Make sure you
handle it with a cURL request
rather than a headless one.
For speed, don’t load images
Make sure you’re setting the
right headers before your return
the HTML. HorsemanJS does not
return headers.

PHP Wrapper for PhantomJS
For those of you that prefer PHP, the PHP PhantomJS wrapper is pretty decent as well.
http://jonnnnyw.github.io/php-phantomjs/
For speed, don’t load images

Content & Linking
You knew I wasn’t going to get up here and not talk
about these things, right?

Google Looks at Entities First
According to Paul Haahr, Google looks at entities first and it appears that we as an industry have talked
about this in theory, but not truly worked it into the optimization workflow.

AlchemyAPI
I encourage you to work AlchemyAPI into your keyword research and content optimization process.

Term Relevance
There’s also been a lot of talk about co-relevance, topic modeling and TF*IDF. This is a great
opportunity for technical elements to influence content creation.

In Fact Just Do Everything Cyrus Says
https://moz.com/blog/7-advanced-seo-concepts

Moz’s New Tool Will Help
Moz has made the related keywords concept more accessible by adding it to Moz Pro.
Searchmetrics also has a similar solution.

Onpage.Org’s TF-IDF Tool is Also Great
http://onpage.org

John Mueller Says 302s Pass PageRank
Don’t let Google mislead you about 302s vs. 301s.

These Rankings Say He’s Wrong
We had a client with millions for 302s and all we did was change them to 301s
June August September
HyperTraffic (1-3) 1272 1,283 1,473
Traffic (4-10) 979 983 837
Striking Distance (11-20) 167 161 110
Emerging (21-40) 106 103 98
Developmental (41-100) 156 90 87
Unranked 0 60 75
0
200
400
600
800
1000
1200
1400
1600
301 redirects

This Traffic Improvement Says He’s Wrong
Traffic went up pretty dramatically despite seasonality.
August 27th
301 redirects
11,039,188
10,455,517
11,348,902
12,049,481
11,250,148
12,902,847
11,040,192
13,158,296
11,287,790 12,203,998
12,508,164
13,158,296
14,041,313
15,609,390
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
18,000,000
April May Jun Jul Aug Sep Oct
Organic Visits
2014 2015

But. We need to stop
saying ‘just do 301
redirects.’

Instead Do Optimal 301 Redirects
Pattern matching on the .htaccess level is always faster than a list of redirects and always better than doing
it on the code/page level. Developers will respect that you understand what you’re talking about.
ReWriteRule url_pattern
file_reference [R=301, L]
redirect 301 /relative/path/to/file.php
http://www.yoursite.com/path/to/new/file.php
>

Internal Linking Structures
Internal Linking Structures are still an incredibly powerful asset. I’d like to see more tools like what Portent has built to visualize
them and understand where there is opportunity. https://www.portent.com/blog/seo/seo-force-directed-diagrams.htm

Searchmetrics Does a Great Job of This
I wish all the link indices had a stronger focus on internal linking structure and visualizing where there are
opportunities to flow more link equity throughout the site. Searchmetrics’ Site Structure tool does this well.

Paul’s Method is Good for Rolling Your Own
http://searchengineland.com/improve-internal-linking-calculate-internal-pagerank-r-246883

If Possible, Always Update your internal links on the database level
WordPress Example
UPDATE wp_posts SET post_content =
REPLACE (post_content,
'http://www.oldsiteurl.com',
'http://www.newsiteurl.com');

Google is becoming the
presentation layer of the web.
Help them do it.

Schema.org / JSON-LD
Schema.org is a lot more reasonable to implement now that they are allowing JSON-LD. Being in knowledge
boxes and such yields a lot more traffic than it takes away.

Page Speed
Google’s next big play

Google Expects Ludicrously Speedy
Google has an expectation that the above the fold content on the page will load within 1 second. You basically get 400 ms to
make that happen.

The Critical Rendering Path

DevTools Is Your Friend
In the Timeline section, DevTools gives you a point by point break down of each operation and how it impacts the site’s load
time. This is how you can determine what exactly is slowing down your pages.

How Your Code Is Laid Out Matters A LOT

This Is what PageSpeed Insights Helps With

External Resources Also Slow Down the Page
Chartbeat is typically a huge culprit and using rel-dns-prefetch can speed this up considerably.

AMP is a Spec to Enforce the Critical Rendering Path
Sound familiar?

Responsive Sites are
SLOOOOOOOW
One of the key things you
can do, aside from optimizing
for the critical rendering
path, is conditional loading.

PreBrowsing Directives
Browsers have preloading directives which allow the page to download elements in the background for later
viewing or within the same session.

Rel-Preconnect
Rel-preconnect will resolve the DNS, start the TCP handshake and negotiate the TLS tunnel
beforehand, effectively shaving hundreds of milliseconds to several seconds in page load.

When to Use Rel-Preconnect
When you see a long time for connections to happen or a lot of idle time, it’s a good idea to use rel-preconnect.

How Rel-Prerender Works
It loads the page in an invisible tab to make the page appear instantly.

Not to Be Confused with Prerender.io

Google Uses It in the SERPs
This could potentially be a way to definitively identify navigational queries. h/t @yoast

I Sent Thousands of Headless Visits

Rel-Prerender’d Pages Generally Perform Better

Use it With the GA API
Get the pageviews of the
ga:pagePath based on the
ga:previousPagePath set to
the current page that you’re
on.
You’ll get a list of the pages
your users are most likely to
visit next. Set one of these as
the rel-prerender.

Code.
Inject rel-prerender into the
page programmatically.
You could also session the
URLs the user is going to to
ensure you are always
prerendering a new page.

It Improved Our Site Speed 68.35%
One line of code did this magic.

Don’t use it on mobile.
Analytics packages aside
from Google Analytics may
show fake sessions.

Defer JS Load with Page Visibility API
…

Rel-Preload for Same Page Preloading
I haven’t played with it yet, but Ian says they’re using it on the Portent site.
https://www.smashingmagazine.com/2016/02/preload-what-is-it-good-for/

How to Go Forward
Things you should know and do to be prepared for
the technical renaissance

Understand the Document Object Model
https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction

Critical Rendering Path
https://developers.google.com/web/fundamentals/performance/critical-rendering-path/analyzing-crp?hl=en

Page Speed
https://www.portent.com/blog/design-dev/ultimate-site-speed-guide-what-impacts-site-speed.htm

Log File Analysis
https://builtvisible.com/log-file-analysis/

Understand SEO for JavaScript Frameworks
https://builtvisible.com/javascript-framework-seo/

Learn DevTools
http://ipullrank.com/6-things-you-should-know-about-in-chrome-devtools/

Accelerated Mobile & Instant Pages
Even if you’re not a publisher, I encourage you to watch these projects as I expect that Google and Facebook will grow the specs
dramatically once they’re adopted.

Read Up On HTTP/2
https://moz.com/blog/http2-a-fast-secure-bedrock-for-the-future-of-seo

Get Back to Testing All The Things

Let’s stop chasing the
content train and get back to
making experiences that
perform.

THANK YOU
Michael King
Managing Director, iPullRank
mike@ipullrank.com
http://ipullrank.com
@iPullRank

SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs

Similar to SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs (20)

More from Distilled

More from Distilled (20)

Recently uploaded

Recently uploaded (20)

SearchLove Boston 2016 | Mike King | Developer Thinking for SEOs