Mobile-First Preparedness- what we've learned from crawling the top 1 million websites.

@jondmyers @DeepCrawl
Mobile-First Preparedness- what we've learned from
crawling the top 1 million websites.
Jon Myers – Chief Growth Officer
Source: http://glooadvertising.com.au/is-your-website-mobile-ready/

Google is our world…

2 Trillion searches per year!

15% of searches new each day!

60% of search are on Mobiles!

Really??...

Sooooooo 2015! And before!

Google is thinking about it...

Helpful to see…

1 page at a time could take a while!…
@jondmyers @DeepCrawlSource: http://shop.adamjk.com/product/1-page-at-a-time

History of my devices…

History of Mobile…
WAP
HTML
1999 2007 2009 2014 2015 2017
Separate Mobile
Pages (Mobile Site
or Dynamic
Delivery)
Responsive
Design
Deep App
Linking
AMP &
Progressive
Web Apps
Mobile-
first
Indexing

Time to get SH*T done…
@jondmyers @DeepCrawlSource: https://www.successimpulse.com/products/less-talk-more-action?variant=44575800839

Heavy Data ahead…
@jondmyers @DeepCrawlSource: https://betanews.com/2015/09/14/8-early-warning-signs-of-problems-in-your-data-governance-plan/

Being Majestic…
Source: https://lafleur.marketing/blog/six-mad-men-marketing-tips-don-draper/

What we did, why? And only we could…
@jondmyers @DeepCrawlSource: https://markfisherfitness.com/what-could-have-been/

History of Mobile SEO Configuration…
Desktop
Mobile
Responsive
AMP pages
Mobile first
One version for desktop devices
One to rule them all - one version designed to work equally good
on desktop and mobile
Dedicated light weight version designed for a fast loading
Dedicated mobile pages served on a separate URL e.g. m.domain
or dynamically served on the same URL
Mobile becomes the PRIMARY version.

Types of Mobile Configuration…
1. No Mobile Configuration (desktop only)
2. Responsive
3. Dynamic
4. Desktop + Dedicated Mobile
5. Desktop + AMP
6. Responsive/Dynamic + AMP
7. Desktop + Dedicated Mobile + AMP
8. Responsive + Dedicated Mobile + AMP
9. Responsive + Dynamic + Dedicated Mobile + AMP
10. Mobile/AMP only
11. Mobile + AMP

Impact of Mobile-first…
-CRITICAL- - Desktop only (Heavily Affected)
-HIGH PRIORITY- - Desktop + Dedicated Mobile (Affected)
-HIGH PRIORITY- - Dynamic (Affected e.g. Content Issues)
-LOW PRIORITY- - Responsive (Not affected)
Impact of Mobile-first on different Mobile Configurations:

Responsive Dynamic Separate
Mobile
Would expect Google to use both
user-agents on the same URL to
validate that the same content is
returned.
Google needs to crawl with both user
agents to validate the mobile version.
Hint: Use the Vary HTTP header!
Google needs to crawl the dedicated
mobile URLs with a mobile user agent to
validate the pages and confirm the
content matches the desktop pages.
How does Google crawl different configurations?

Responsive design is the optimal situation…
“Responsive design is pretty
much the optimal situation
because regardless of which
device we use to crawl the site
we will see the same thing.
We’ll get the same content and
the structured data, the same
videos, the same images and the
same internal links. It’s all
there.”
Reference: https://youtu.be/7Aq9bFdfMuE?t=54m53s

How does the world look?...
Source: http://abcnews.go.com/topics/news/doomsday-clock.htm

How are sites set up for mobile?...

Mobile Configuration Breakdown…
Breakdown of all sites where mobile configuration could be identified
Significant no. dynamic sites
– work to ensure meta data
is kept consistent across
different versions of page.
Responsive design dominates -
Path of least resistance
moving to Mobile-first Index
Separate Mobile unpopular –
high maintenance to ensure
desktop/mobile equivalence
15,000 sites
190,000 sites
795,000 sites

Separate Mobile Sites – Are they up to scratch?...
Desktop
Page
Mobile
Page
Rel=canonical
61% of Separate Mobile sites have correct canonical to desktop page.

Desktop
Page
Mobile
Page
No canonical
21% of Separate Mobile sites had no canonical to desktop equivalent.

Desktop
Page
Mobile
Page
Other
page?
18% of Separate Mobile sites had canonicals pointing to wrong page

Separate Mobile Sites – Desktop/Mobile Equivalence…
Of 15,000 Separate Mobile sites returning 200 response codes:
Desktop
Page
Mobile
Page
Matching title &
meta descriptions
Desktop
Page
Mobile
Page
Matching meta
descriptions
Desktop
Page
Mobile
Page
Matching title
7.7%
7.9%
5.8%

Separate Mobile Issues…
1%
of separate mobile
sites had noindex
meta tag
25%
of all Separate Mobile sites
returned a non-200 status code
21% returned 3xx redirection
2% Separate Mobile sites returned
4xx errors
1% returned 5xx errors

Google pushing for responsive web…
https://webmasters.googleblog.com/2017/09/how-to-move-from-
m-dot-urls-to.html
Separate Mobile
Google needs to crawl the dedicated
mobile URLs with a mobile user agent to
validate the pages and confirm the
content matches the desktop pages.

Order of Mobile-first Indexing…
“If you have a vary header
which returns a different
page for a mobile user agent,
Google will use that as the
mobile page instead of the
responsive page.”
Reference:https://www.youtube.com/watch?v=8NyZypIfO
zI&t=36m31s

Dynamic configurations…
Initial analysis found ?% of websites that serve
different content based on user agent do not have
a Vary: User-agent header.
Dynamic – same URL, different HTML
dependent on user agent

Let’s talk about speed…
Source: https://play.google.com/store/apps/details?id=com.ea.game.nfs14_row

Fetch times – The need for speed…
Percentage of sites split by fetch time
Fetch time includes time taken to fetch URL and display the HTTP response. This doesn’t
include the time taken to request or run any associated resources (such as images or scripts) on
the page.

But are the top million sites really that fast?...

Fetch times – Configuration breakdown…
Dynamically servedResponsive Separate Mobile
Average fetch time by configuration (secs):
Separate mobile sites have the slowest fetch times.
Responsive sites have fastest fetch times of
three mobile configurations.

Fetch time with mobile desktop user agents…
19% of sites had a
fetch time between 1-
2 seconds.
68% of sites had a
fast fetch time – below
one second.
13% of sites had a
slow fetch time
exceeding 2 seconds

What does Google have to say about site performance?...
Source: https://www.thinkwithgoogle.com/about/

Google research – Performance per vertical…
Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/
Consumer sites
have faster more
responsive web
servers.
Business and
finance lagging
behind.
Technology my
arse!

Google research – Performance per vertical…
Google says to aim for 2.4
seconds load time for a
page!!!???!!

Lets Talk Page Size…

Google research – Page Size…
According to Google, sites
across all verticals are on
average larger than the
500kb recommendation.
Again, technology sites
aren’t leading the way.

What did we find out about HTML size?...
94%<=200k
5%200k - 500k

Accelerated Mobile Pages…
Percentage of sites with AMP version of homepage:
Responsive
0.63%
Dynamic
0.61%
Separate Mobile
0.86%

What about HTTPS?...

HTTPS – How many have made the switch?...
HTTPS
(22%)
HTTP
(78%)
Has HTTPS reached critical mass? Our data
says there’s a long way to go for the top one
million sites.
This figure is much lower than currently reported
Mozcast – 68% of first page results have SSL
encryption.
Source: http://mozcast.com/features

HTTPS – Adoption across mobile configurations…
Dynamically served sites appear
to be particularly slow to adopt
HTTPS.
Surprisingly Separate Mobile sites
leading the way for the switch to
HTTPS.

HTTPS doesn’t end there…
Watch out for non-secure form fields (we
found a lot of them) as Chrome is now
flagging these with security warnings.
12.7% incorrectly configured sites with links
between protocols and some sites with mixed
content e.g. a HTTPS page with reference to a
script on an HTTP URL.

Carving up Configurations by Country…

Splitting the Majestic Million by TLDs…
Percentage of sites per TLD

.edu & .gov are leading the way with responsive design…

…and .co.uk TLDs aren’t far behind…
ResponsiveDynamic Separate

What next? Plenty of issues…
Non Secure Form Fields
Pages without Valid Canonical
Tag
Failed URLs
Empty Pages
Max Fetch Time
Thin Pages
Missing H1 Tags
Non-301 Redirects
Pages with Duplicate Body
Duplicate Pages
Malformed URLs
High External Linking
Max URL Length
Missing Titles
Pages with Duplicate Titles
No Descriptions & No Snippets
Unauthorised Pages
5xx Errors
Max Links
Broken Pages (4xx Errors)
Max Description Length
Short Titles
Unlinked Paginated Pages
Hreflang to Non-200 URLs
Non-rel Alted AMP Pages
All Broken Links
Non-reciprocal Mobile/AM
Duplicate Description Sets
Max Content Size
Uncategorised HTTP Response Codes
Max Title Length
Duplicate Body Sets
Duplicate Page Sets
Pages with Duplicate Descriptions
Canonical to Non-200
Max Redirections
Max HTML Size
Short Descriptions
Redirect Loops
Duplicate Title Sets
And more…
Non-200 Mobile/AMP
Excessive Redirects In (Admin Only)
Mobile Links Out Mismatch
Mobile Links In Mismatch
Mobile Word Count Mismatch
Mobile Content Mismatch
Duplicate Pages including Primary

In a nutshell…

Separate Mobile sites are poorly configured

Dynamic sites need to check they have Vary HTTP response header…

HTTPS adoption relatively low amongst top million sites…

Fetch time is surprisingly low???…

Watch this space for an even deeper dive into the data…
Source: http://dive-bohol.com/

But what does the future hold?...

Go beyond Responsive with PWA’s…
Progressive Web Apps is:
• Progressive
• Responsive
• App-like
• And more...
Progressive Web Apps can fall back to AMP.
Hint… first load as AMP.
You should think about it now.
Progressively Web App is THE FUTURE.

Jon Myers
DeepCrawl
Chief Growth Officer

Mobile-First Preparedness- what we've learned from crawling the top 1 million websites.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (15)

Similar to Mobile-First Preparedness- what we've learned from crawling the top 1 million websites.

Similar to Mobile-First Preparedness- what we've learned from crawling the top 1 million websites. (20)

Recently uploaded

Recently uploaded (20)

Mobile-First Preparedness- what we've learned from crawling the top 1 million websites.