[MozCon 2019] Fixing the Indexability Challenge: A Data-Based Framework

Fixing the Indexability Challenge:
A Data-Based Framework
Areej AbuAli
#MozCon 2019
slideshare.net/areejabuali@areej_abuali

I’m here to talk to you about a framework
that I came up with to fix a client’s
indexability challenge

But more importantly…
(And I only realised this after I’d finished preparing this talk)

I’m also here to talk to you about how
“Technical Problems are People Problems”

I first met this client back in 2017

They’re a job aggregator site

89%YoY decrease in organic visibility
0
2000
4000
6000
8000
10000
12000

Organic Traffic
Y1 X
Y2 Y
Change -46%

0
100,000
200,000
300,000
400,000
500,000
600,000
Competitor OrganicVisibility
Client indeed.co.uk monster.co.uk reed.co.uk

And their site was so massive;
we struggled to crawl it

To make this work, we need to fix the
fundamentals first

So this talk is about my 18-month
relationship with this client

It’s about what worked, what didn’t work
and what I would have done differently

Links Tech Content
I started working on a comprehensive audit

I ended up with a 70-page document

There was a total of 50 recommendations

Some of the main things included were…

72% of backlinks came from only
3 referring domains

Their on-page content was full of
duplication and missing the basics

Canonical tags were not set up correctly

And their internal linking structure
was a nightmare

Every recommendation was outlined using a
traffic light system

And every section was split into
Problem, Effect & Solution

We had a half-day audit handover meeting
where I walked them through all
of our recommendations

Yet I couldn’t help but feel it
was not enough…

Everything I recommended up till
now was solid

But I had a gut feeling that due to the
nature of their site…

These recommendations wouldn’t
quite cut it

I had to go back to the drawing board

The Findings I Should’ve Found The First Time
Round But Didn’t So I’m Choosing To Call It
Supplementary Findings To Sound
...

They’re a job aggregator site – in essence, they’re a
job search engine

That means that every single search
conducted could, potentially, be
crawled and/or indexed
if it wasn’t built right

That’s equivalent to an infinite number
of potential searches!

Which part was their site
not getting right?

I knew that it was impossible to crawl

The one time I tried to fully crawl the site,
it returned over 2.5 million URLs

And I could only crawl it by excluding
massive sections

And if their pages couldn’t be crawled,
then they would never be
indexed properly…

So there were three problems that
needed to be fixed

It was apparent that there were no rules in
place to help direct robots

Unique URLs were created, using up all
possible filter combinations

This can create a potentially-unlimited
number of URLs

And all the pages looked exactly the
same – they just had a list of jobs!

Google was wasting crawl budget by
crawling duplicate thin pages and
attempting to index them

I needed to create a customised framework
that instructs bots on what to do and
what not to do

The job aggregator industry seemed to be
doing one of two things

Limiting indexable pages
à miss out on ranking opportunity

Limiting indexable pages
à miss out on ranking opportunity
Not limiting indexable pages
à wind up with reduced link equity

My framework was going to do neither

Instead it would use search volume data
to determine which pages are
valuable to search engines

So how exactly would this work?

It starts off by passing
the search query
through a keyword
combination script

This script outputs different combinations
for the search conducted

It does that by changing the order of
keywords to see all possible combinations

Digital Marketing Manager London

Digital Marketing Manager London
London Digital Marketing Manager
Digital Marketing London Manager

These combinations will increase based
on the search query and filters applied

Even though Google will regard most
of these combinations as the same…

The script will help avoid duplicating
pages that have different versions
of the same thing

It then searches the
database to see
if this job is available

If (Job # Available)
Load a page stating so
and no-index it

If (Job = Available)
Search the keyword
database for all
keyword combinations
from the script

Fetch search volume
data for these
keyword combinations

For search volume data, we recommended
using keywordtool.io API

If (SearchVolume > 50)
A page is then created
using the highest SV
keyword combination
that is both indexed
and crawled

If (SearchVolume < 50)
Load page for users
but no-index it

Your search volume cut-off can be updated
at anytime and is based on what makes
sense for your industry

There is always the possibility of
errors occurring

What if there’s a tie-breaker?
(Several keyword combinations having the
same search volume)

Create an indexed crawlable page based
on the keyword used in the
user search query

What if the API was down?
(And you’re unable to generate search volume data)

Load a no-index page for usability
and don’t store the query in the database

As for their internal linking structure…

This was the status of their header

We also provided internal linking
recommendations for their footer and
job advertisement pages

And an exact breakdown of their filter
system on job search pages

As for their sitemaps – they
had none

We recommended creating and
splitting them up

Blog Sitemap
Job Advertisements Sitemap
Job Results Sitemap
Ancillary Sitemap

The final step was to help them sort out
their content

Even if they were only indexing high
search volume pages; their
content was very thin

Their chances to rank would
still be minimal

They had the same H1,Title Tag & Meta
Description for every filtered
indexed page

Most competitors automatically generate
optimised meta tags

They needed to do the same for
indexable filter pages

Other than a company page and a
handful of blog posts, there were
no core content pages

So we performed in-depth keyword research
and opportunity analysis to see what
content generates traffic

And we provided a content audit
and strategy to go with it

My ‘Aha!’ Moment felt complete

This was the piece of the puzzle
that was missing

The client confirm that they
implemented everything

The first thing that caught my attention
was their site went from over
2.5M crawled pages to 20K

Which initially felt like good news…

Until I realised that their traffic had
declined…

Remember this?
Organic Traffic
Y1 X
Y2 Y
Change -46%

Organic Traffic
Y1 X
Y2 Y
Y3 Z
Change -86%

The Findings I’m Rushing To Find In A Panic
To Prove That They Haven’t Implemented
My Recommendations Accurately,
Hence I ...

I went through the original list of
50 recommended actions

At that point, there were 29 that had
not been implemented

I also discovered ten new issues that were
affecting their indexability

Google was choosing to only index
20% of the submitted sitemap

Googlebot will choose to visit their site less
if their indexability is not in check

The client explained that they’ve added
canonical tags and felt that was enough

They’re relying on Googlebot to:
1) Crawl the pages

1) Crawl the pages
2) Find the canonical tags

1) Crawl the pages
2) Find the canonical tags
3) Then choose to ignore them

Canonical tags are simply hints for bots

Google might decide to ignore the
tags and pick other pages to index

Almost 80K pages had been indexed
despite not being submitted
via the sitemap

And some pages were included in the
sitemap but not indexed

/job/finance-manager-liverpool

/job/finance-manager-liverpool?index=
57271b7c~4&utm_campaign=job-
detail&utm_source=search-
result&utm_medium=website

Because this page was the one
discovered via internal links

Over 5K similar pages with parameters
were getting indexed

Your main goal is to maximise crawl budget

You cannot use canonical tags as a
sticking plaster to fix that

This implementation was still incomplete

I had to change my way of conveying
this message

I put a stop to the endless stream of emails
and scheduled a face-to-face meeting

We reviewed each and every single
remaining task and discussed them in detail

Behold the wonders of Google Sheets!

We re-prioritised tasks and put
estimated completion dates

It was not an easy meeting but
it felt productive

On a personal level, I discovered that I
suffer from Imposter Syndrome

This was my constant state of mind

I was working closely with the CTO
throughout this project

I felt he didn’t trust me or my knowledge

So, what would I have done differently?

If I could go back in time, I would realise
what the actual problem was

All technical problems are people problems

The SEO recommendations were solid

Getting it implemented was the hard part

As a tech SEO, the most you can do
is to influence priorities

In this instance, I didn’t manage to
persuade him to implement
the recommendations

I also learned that the way I’ve been
doing SEO audits is plain wrong

I always focused on delivering a set
of comprehensive actions

Instead, maybe I should just deliver
a SINGLE recommendation

And once that’s implemented…

Then, and only then, will I recommend
another

And maybe I shouldn’t recommend
Nice-To-Do’s…

Until there are only Nice-To-Do’s
left to do

Because they are simply a distraction from
the main problem

This talk does not have a happy ending…

It is not a successful case study…

There’s no upward visibility graph and
page one rankings for me to show off…

It’s about a framework I created that
fixes indexability issues that I’m proud of
and I know in my gut *works*

So I wanted to share it with you

Because I can see this applied across
many sites in plenty of industries

And I’d love forYOU to implement it

So I’m going to share my full
methodology with you

Getting the basics right is so fundamental

If you can do nothing else,
just do the tech.

Areej AbuAli
Slides: slideshare.net/areejabuali
Tweet things @areej_abuali

References
(1) http://3.bp.blogspot.com/_zKFo7OwbIp8/TLkK8i5FrlI/AAAAAAAAEeY/oeXBnenxnWU/s1600/uphill3_l.jpg
(2) https://s...

[MozCon 2019] Fixing the Indexability Challenge: A Data-Based Framework