Restructuring Websites to Improve Indexability

slideshare.net/areejabuali
RESTRUCTURING WEBSITES TO
IMPROVE INDEXABILITY
AREEJ ABUALI // VERVE SEARCH //
@areej_abuali

I’m here to talk to you about a framework
that I came up with to fix a client’s
indexability challenge

But more importantly…
(And I only realised this after I’d finished preparing this talk)

I’m also here to talk to you about how
“Technical Problems are People Problems”

I first met this client back in 2017

They’re a job aggregator site

My CEO introduced me, saying:
“Areej is the bestTech SEO you’ll find.”

Needless to say, the pressure was on!

They wanted to work with us because
we’re known to produce cool
creative campaigns

And we’re good at outreach
and link building

But we said that from some quick analysis,
it was too early for that…

“Your website is struggling”

89% decrease in organic visibility

Organic Traffic
2017 X
2016 Y
Change -45.6%

And your site is so massive;
we struggled to crawl it

On the other hand, here’s where
your competitors are…

To make this work, we need to fix the
fundamentals first

So, let’s spend 6 months purely on tech

Once that’s sorted, then we can work
on creative campaigns together

This talk is about my 18-month
relationship with this client

It’s about what worked, what didn’t work
and what I would have done differently

Links Tech Content
I started working on a comprehensive audit

I ended up with a 70-page document

There was a total of 50 recommendations

Some of the main things included were…

72% of backlinks came from only
3 referring domains

Their on-page content was full of
duplication and missing the basics

Canonical tags were not set up correctly

And their internal linking structure
was a nightmare

Every recommendation was outlined using a
traffic light system

We had a half-day audit handover meeting
where I walked them through all
of our recommendations

Yet I couldn’t help but feel it
was not enough…

Everything I recommended up till
now was solid

But I had a gut feeling that due to the
nature of their site…

These recommendations wouldn’t
quite cut it

I had to go back to the drawing board

The Findings I Should’ve Found The First Time
Round But Didn’t So I’m Choosing To Call It
Supplementary Findings To Sound
...

They’re a job aggregator site – in essence, they’re a
job search engine

That means that every single search
conducted could, potentially, be
crawled and/or indexed
if it wasn’t built right

That’s equivalent to an infinite number
of potential searches!

In my first six months in SEO,
I used the terms ‘Crawling’ and ‘Indexing’
interchangeably

Until someone on Twitter called me
out on it, and said…

“You do realise they’re not the same thing,
right?”

So let’s make sure we’re all on
the same page

Crawling (V)
The process of gathering information
from billions of webpages through
their link structure.

Indexing (V)
To add crawled webpages to
the search engine index.

Which part was their site
not getting right?

I knew that it was impossible to crawl

The one time I tried to fully crawl the site,
it returned over 2.5 million URLs

I could only crawl it by excluding
massive sections

And if their pages couldn’t be crawled,
then they would never be
indexed properly…

So there were three problems that
needed to be fixed

It was apparent that there were no rules in
place to help direct robots

This can create a potentially-unlimited
number of URLs

Google was wasting crawl budget by
crawling duplicate thin pages and
attempting to index them

I needed to create a customised framework
that instructs bots on what to do and
what not to do

The job aggregator industry seemed to be
doing one of two things

Limiting indexable pages
à miss out on ranking opportunity

Limiting indexable pages
à miss out on ranking opportunity
Not limiting indexable pages
à wind up with reduced link equity

My framework was going to do neither

Instead it would use search volume data
to determine which pages are
valuable to search engines

So how exactly would this work?

It starts off by passing
the search query
through a keyword
combination script

This script outputs different combinations
for the search conducted

It does that by changing the order of
keywords to see all possible combinations

Digital Marketing Manager London

Digital Marketing Manager London
London Digital Marketing Manager
Digital Marketing London Manager

These combinations will increase based
on the search query and filters applied

Even though Google will regard most
of these combinations as the same…

The script will help avoid duplicating
pages that have different versions
of the same thing

It then searches the
database to see
if this job is available

If (Job # Available)
Load a page stating so
and that is no-index
and no-crawl

If (Job = Available)
Search the keyword
database for all
keyword combinations
from the script

Fetch search volume
data for these
keyword combinations

For search volume data, we recommended
using keywordtool.io API

If (SearchVolume > 50)
A page is then created
using the highest SV
keyword combination
that is both indexed
and crawled

If (SearchVolume < 50)
Load page for users
but no-index and
no-crawl it

Your search volume cut-off can be updated
at anytime and is based on what makes
sense for your industry

There is always the possibility of
errors occurring

What if there’s a tie-breaker?
(Several keyword combinations having the
same search volume)

Create an indexed crawlable page based
on the keyword used in the
user search query

What if the API was down?
(And you’re unable to generate search volume data)

Load a no-index no-crawl page for
usability and don’t store the
query in the database

But I still knew that this was not enough…

What about their internal linking structure?

This was the status of their header

We also provided internal linking
recommendations for their footer and
job advertisement pages

And an exact breakdown of their filter
system on job search pages

As for their sitemaps – they
had none

We recommended creating and
splitting them up

Blog Sitemap
Job Advertisements Sitemap
Job Results Sitemap
Ancillary Sitemap

The final step was to help them sort out
their content

Even if they were only indexing high
search volume pages; their
content was very thin

Their chances to rank would
still be minimal

They had the same H1,Title Tag & Meta
Description for every filtered
indexed page

This was creating thin pages and partially
duplicated content

Most competitors automatically generate
optimised meta tags

They needed to do the same for
indexable filter pages

They also needed to create pages targeting
specific category search terms

Other than a company page and a
handful of blog posts, there were
no core content pages

So we performed in-depth keyword research
and opportunity analysis to see what
content generates traffic

And we provided a content audit
and strategy to go with it

My ‘Aha!’ Moment felt complete

I was happy, I was pleased, I was ecstatic!

This was the piece of the puzzle
that was missing

Now all that was left was for
the client to implement it!

Four months later, the client confirmed that
they implemented everything

By that point, we had started working on
creative campaigns and outreach

The first thing that caught my attention
was their site went from over
2.5M crawled pages to 20K

Which initially seemed like good news…

Until I realised that their traffic had
declined…

Remember this?
Organic Traffic
2017 X
2016 Y
Change -45.6%

Organic Traffic
2018 X
2017 Y
2016 Z
Change -85.7%

The Findings I’m Rushing To Find In A Panic
To Prove That They Haven’t Implemented
My Recommendations Accurately,
Hence I ...

I went through the original list of
50 recommended actions

At that point, there were 29 that had
not been implemented

I also discovered ten new issues that were
affecting their indexability

Google was choosing to only index
20% of the submitted sitemap

Crawling is one of the most expensive
things that Google do

Crawl budget is precious – it should
not be wasted

Googlebot will choose to visit their site less
if their indexability is not in check

The client had added canonical tags and
felt that was enough

They’re relying on Googlebot to:
1) Crawl the pages

1) Crawl the pages
2) Find the canonical tags

1) Crawl the pages
2) Find the canonical tags
3) Then choose to ignore them

Canonical tags are simply hints for bots

Google might decide to ignore the
tags and pick other pages to index

Canonical tags do not preserve
crawl budget

Almost 80K pages had been indexed
despite not being submitted
via the sitemap

And some pages were included in the
sitemap but not indexed

/job/finance-manager-liverpool

/job/finance-manager-liverpool?index=
57271b7c~4&utm_campaign=job-
detail&utm_source=search-
result&utm_medium=website

Because this page was the one
discovered via internal links

Over 5K similar pages with parameters
were getting indexed

Your main goal is to maximise crawl budget

You cannot use canonical tags as a
sticking plaster to fix that

This implementation was still incomplete

I had to change my way of conveying
this message

I put a stop to the endless stream of emails
and scheduled a face-to-face meeting

We reviewed each and every single
remaining task and discussed them in detail

We re-prioritised tasks and put
estimated completion dates

It was not an easy meeting but
it felt productive

On a personal level, I discovered that I
suffer from Imposter Syndrome

This was my constant state of mind

I was working closely with the CTO
throughout this project

I felt he didn’t trust me or my knowledge

No matter how much I tried to assure him that
he’s in safe hands

The fact of the matter is,
this site is his baby

And I was constantly attacking his baby

So, what would I have done differently?

If I could go back in time, I would realise
what the actual problem was

All technical problems are people problems

The SEO recommendations were solid

Getting it implemented was the hard part

As a tech SEO, the most you can do
is to influence priorities

In this instance, I didn’t manage to
persuade him to implement
the recommendations

I also learned that the way I’ve been
doing SEO audits is plain wrong

I always focused on delivering a set
of comprehensive actions

Instead, maybe I should just deliver
a SINGLE recommendation

And once that’s implemented…

Then, and only then, will I recommend
another

And maybe I shouldn’t recommend
Nice-To-Do’s…

Until there are only Nice-To-Do’s
left to do

Because they are simply a distraction from
the main problem

Over the past year, we created
6 creative campaigns for them

These campaigns generated 261 links and
had 1.6M estimated views

We got coverage on all of these sites and more…

Yet it barely had an impact…

On a daily basis, I was being asked:
“Why aren’t our rankings improving?”

On a daily basis, I was constantly saying…

“The core reason your rankings aren’t
improving is due to incomplete
tech actions…”

“Let’s work together to get these
implemented ASAP as they will
directly impact your traffic and rankings.”

This talk doesn’t have a happy ending…

It isn’t a successful case study…

It isn’t a successful case study…
YET

There’s no upward visibility graph and
page one rankings for me to show off…

There’s no upward visibility graph and
page one rankings for me to show off…
YET

This talk is about real life…

It’s about a long-overdue project where I
learned a lot about working
alongside CTOs

It’s about how I created a framework that
fixes indexability issues that I’m proud of
and I know in my gut *works*

So I wanted to share it with you

Because I can see this applied across
many sites in plenty of industries

And I’d love forYOU to implement it

So I’m going to share my full
methodology with you

Getting the basics right is so fundamental

If you can do nothing else,
just do the tech.

Areej AbuAli
Head of SEO |Verve Search
Slides: slideshare.net/areejabuali
Questions: areej@vervesearch.com
Tweet things @a...

Restructuring Websites to Improve Indexability