How Much Content is Not Indexed in Google in 2019?

Bartosz Góralewicz
@bart_goralewicz

Google Hangouts
(August 23rd 2019)

JavaScript SEO is not dying.
It's getting even more complex
Is JavaScript SEO dying?

There are so many things that you
can do wrong, and it takes a lot of
experience to be able to debug
and find out and improve things

HTML HTMLHTML
Everything is JS

HTML
Even HTML
pages CAN
be rendered.

HTML
Even HTML
pages CAN
be rendered.
BUT

We found out
that JavaScript
of non-JavaScript
websites.
is a problem

We found out
that JavaScript
with non-JavaScript
websites.
is a problem

JavaScript SEO evolved from “JS
websites” to all websites.

Timeframe (of JS indexing)
or the lack thereof

It is 2019 and our
research shows that
there are hundreds
of thousands of
domains not fully
indexed.
Even after… Months of
publishing the content

Even after… Months of
publishing content.
It is 2019 and our
research shows that
there are hundreds
of thousands of
domains not fully
indexed.

How rendering works with Google

Bartosz: So, you look at the
difference between the initial
HTML, and, then, if after
rendering you see extra content?

Bartosz: So, you look at the
difference between the initial
HTML, and, then, if after
rendering you see extra content?
Martin Splitt: Yeah.

The Heuristics
There are certain heuristics, that,
if we see after a while, like, oh, this
page, actually, the renderer does not
diff as much or doesn’t diff, it looks
the way before

There are certain heuristics, that,
if we see after a while, like, oh, this
page, actually, the renderer does not
diff as much or doesn’t diff, it looks
the way before
The Heuristics
I still haven’t fully, like, grasped
what exactly triggers the
heuristics

… those heuristics are FAR from perfect though.

Pretty much every website, when
we see them for the first time, goes
to rendering. So there’s no indexing
before it hasn’t been rendered.”
ALL NEW SITES
GET RENDERED

But what is
a new website exactly?
What is a new website?

What if
a new website doesn’t have
any comments yet?

Experiments on staging? Don't go this way

We decided to experiment
with how good Google is
with their heuristics.

Benchmarking Google's heuristics:
Our own JS Budget Experiment (2019)

3 domains with <a href> links in the initial HTML response

3 other domains with <a href> links generated with external JavaScript files

Each domain runs on a different IP (UK)

Each domain runs on a different IP (UK)
Content generated using Articoolo

Crawler budget experiment 2017

List of experiment websites
Time needed to reach the 7th level of nested content
(server logs)
htmlcrawling.party
first 6 levels: 6 minutes. We had to wait as much as 6
days for the 7th level.
jscrawling.party 17 minutes
htmlcrawling.wine 4 minutes
jscrawling.wine
We had to wait 1 day for Google to reach the 2nd level,
but then it went through the remaining levels within
minutes.
jscrawling.pizza 8 minutes
htmlcrawling.pizza 7 minutes
Results – 2019

I was right
Results – 2019
List of experiment websites
Time needed to reach the 7th level of nested content
(server logs)
htmlcrawling.party
first 6 levels: 6 minutes. We had to wait as much as
6 days for the 7th level.
jscrawling.party 17 minutes
htmlcrawling.wine 4 minutes
jscrawling.wine
We had to wait 1 day for Google to reach the 2nd level,
but then it went through the remaining levels within
minutes.
jscrawling.pizza 8 minutes
htmlcrawling.pizza 7 minutes

Crawler budget experiment 2019

Google in 2017 vs Google in 2019.
0:1

Good job,
Google!
0:1

Ok, Let's do another experiment!
0:1

HTML to JS ratio experiment
v
3 types
of test pages

HTML to JS ratio experiment- results

Raw data:

Raw data: Most of the JavaScript content indexed after
a half-hour (!)

a half-hour (!)
With the exception of 5 URLs.

a half-hour (!)
But here, Google didn’t even crawl it,
so it’s not related to rendering at all

a half-hour (!)
After 4 hours: 29/30
JavaScript content indexed.

a half-hour (!)
After 4 hours: 29/30
JavaScript content indexed. After 8 hours: 30/30
JavaScript content indexed.

0:2

Good job,
Google!
0:2

Ok, Let's do another experiment!
0:2

JSSEO.expert v1 - 2017
V2 Inline
GSC fetch
URL Indexed?
Content
cached
Links
Content
Indexed?
Test URL
Inline Inline

JSSEO.expert v2 - results
V2 Inline
GSC fetch
URL Indexed?
Content
cached
Links
Content
Indexed?
Test URL
Inline Inline

0:3

Good job,
Google!
0:3

But what about popular websites having some
content generated by JS?
Martin Splitt was right
about the new websites.

Martin Splitt was right
about the new websites.
Let's find out!Can Google deal with them?
But what about popular websites having some
content generated by JS?

The State of JavaScript Indexing
Experiment 2019

National Geographic - with JS
v

National Geographic - NO JS
v
100%of JS content indexed

ASOS - NO JS
v
100%of JS content indexed

v
But not every website
is lucky enough

82%
0%
0%
0%
42%
73%
Percentage
of the
JavaScript
content
indexed

2 WAVES
O F I N D E X I N G
T I M E F R A M E

Percentage of the JavaScript
content not indexed after 14 days
4%
0%
66%
30%
99,5%

4%
0%
66%
30%
99,5%
Percentage of the JavaScript
content not indexed after 14 days

Blame Google or say that JS is evil

Every JavaScript SEO issue we saw
was 100% self-induced

2 WAVES
O F I N D E X I N G - T I M E L I N E

I expect, eventually rendering,
crawling and indexing will come
closer together
We are going like: ”Oh, all right, we
are gonna skip rendering.” It is not
as frequently happening anymore.
(...) Many websites even if they do
not run JavaScript, they might still
go through the render phase,
because it doesn’t make a difference
as much

Crawling
and indexing
to come
together
I expect, eventually rendering,
crawling and indexing will come
closer together
We are going like: ”Oh, all right, we
are gonna skip rendering.” It is not
as frequently happening anymore.
(...) Many websites even if they do
not run JavaScript, they might still
go through the render phase,
because it doesn’t make a difference
as much

Now it's time for
the BIG news!

Onely Made for Geeks – free toolset
OMFG

OMFG
www.onely.com/tools

OMFG
www.onely.com/tools
WARNING – still an early alpha version.

www.onely.com/tools
Fully built within the last 3 weeks 
WARNING – still an early alpha version.
OMFG

TGIF - The Google Indexing Forecast

WWJD – What Would JavaScript Do?

WWJD – BBC & meta tags vs. rendering

TL;DR – Too Long Didn’t Render

Chrome
Mobile-
Friendly Tester

A LOT of cool FREE tools coming soon!

JavaScript SEO is important. But other
parts of technical SEO are important too.

I mentioned The Guardian has some issues with JavaScript SEO.

But Google indexes their HTML content really fast.

Sample of 1300 URLs.
Number of pages indexed after 1 day: 1280 = 98.5%

What about the rest?
Sample of 1300 URLs.
Number of pages indexed after 1 day: 1280 = 98.5%

But other websites aren't as lucky
as The Guardian.

Medium is...medium*
A quick check of 100 URLs of Medium...
*Tomek’s joke :D

Only 70% of them are indexed in Google.
*Tomek’s joke :D
Medium is...medium*

Spoiler alert: 50% of the indexed URLs has JS content indexed.
That's the...medium value**.
Only 70% of them are indexed in Google.
*Tomek’s joke :D** 
Medium is...medium*

Nomoregunsusa.com
case study
vv

Cloaking in 2019
www.onely.com
@bart_goralewicz
THANK YOU

How Much Content is Not Indexed in Google in 2019?

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to How Much Content is Not Indexed in Google in 2019?

Similar to How Much Content is Not Indexed in Google in 2019? (20)

Recently uploaded

Recently uploaded (20)

How Much Content is Not Indexed in Google in 2019?