When a client with a huge website had a tricky indexation issue, we needed to definitively know which pages were and were not indexed. Although Search Console has always taunted us with how many pages are indexed from your sitemap, they don't tell you which ones specifically. So, we built a tool using an undocumented Google API we discovered which allows us to check thousands of URLs in Google at a time, and released it to the public.

2. Sean Malseed | @seanmalseed | #TechSEOBoost
EXTREME
FULL-CONTACT
GOOGLE APIS
–
How Absentmindedly Watching
Network Requests Led to our Most
Popular SEO Tool

3. Sean Malseed | @seanmalseed | #TechSEOBoost
About me
I’m Sean
Tech Director at Greenlane
Extreme amateur programmer
Hobbies include dark beer and long
stretches of doing nothing in particular
greenlanemarketing.com/team/sean-malseed

4. Sean Malseed | @seanmalseed | #TechSEOBoost
THERE WILL BE FREE SPAGHETTI

5. Sean Malseed | @seanmalseed | #TechSEOBoost

6. Sean Malseed | @seanmalseed | #TechSEOBoost
SPAGHET YOU CAN DEPLOY TO THE CLOUD

7. Sean Malseed | @seanmalseed | #TechSEOBoost
This kills the spaghetti meme.

8. Sean Malseed | @seanmalseed | #TechSEOBoost

9. Sean Malseed | @seanmalseed | #TechSEOBoost
Send a request to a URL, and get back information in a simplified and easy-
to-import format.
Request URL: https://someservice.com/rankapi?keywords=banana,smoothie,fruit
Server gives back:
{"ranks": {"banana":12,"smoothie":24,"fruit":48}}
What an API actually is

10. Sean Malseed | @seanmalseed | #TechSEOBoost
Modern API usage
APIs are intended for two primary groups of people:
• Professional developers building robust applications
• Nosey marketing people who know just enough to be dangerous

11. Sean Malseed | @seanmalseed | #TechSEOBoost
How to find APIs
Two ways to find useful APIs:
• Published APIs, such as Google’s excellent comprehensive API Explorer
• Unpublished APIs, usually discovered by clicking around randomly

12. Sean Malseed | @seanmalseed | #TechSEOBoost
Google, why aren’t you indexing all these pages?
We Had a Problem

13. Sean Malseed | @seanmalseed | #TechSEOBoost
Troubleshooting Indexation, “The Old Way”
YAY
NOT YAY

14. Sean Malseed | @seanmalseed | #TechSEOBoost
Troubleshooting Indexation, “The Old Way”
YAY
NOT YAY

15. Sean Malseed | @seanmalseed | #TechSEOBoost
But what if we have almost literally
a bazillion pages to check?
NOT YAY
AT ALL

16. Sean Malseed | @seanmalseed | #TechSEOBoost
We’re not alone in this problem

17. Sean Malseed | @seanmalseed | #TechSEOBoost
Something like this could work

18. Sean Malseed | @seanmalseed | #TechSEOBoost
Whoops

19. Sean Malseed | @seanmalseed | #TechSEOBoost
Whoops

20. Sean Malseed | @seanmalseed | #TechSEOBoost
GOOGLE WHY
CAN’T YOU JUST
BE COOL ABOUT
THIS

21. Sean Malseed | @seanmalseed | #TechSEOBoost
Everything is garbage, time to
procrastinate

22. Sean Malseed | @seanmalseed | #TechSEOBoost
Watching network requests because ¯_(ツ)_/¯

23. Sean Malseed | @seanmalseed | #TechSEOBoost
You can find a of interesting things under XHR

24. Sean Malseed | @seanmalseed | #TechSEOBoost
Hey, neat!

25. Sean Malseed | @seanmalseed | #TechSEOBoost
This trick works on lots of websites

26. Sean Malseed | @seanmalseed | #TechSEOBoost
Hey, neat!

27. Sean Malseed | @seanmalseed | #TechSEOBoost
And once you have an API of it...
https://bit.ly/suggkw

28. Sean Malseed | @seanmalseed | #TechSEOBoost
“If it pops up, in, or out...
good chance there’s an API
of it.”

29. Sean Malseed | @seanmalseed | #TechSEOBoost
Ok, back to the index thing

30. Sean Malseed | @seanmalseed | #TechSEOBoost
I wonder if there’s something like the suggest API that can help me...

31. Sean Malseed | @seanmalseed | #TechSEOBoost
What the heck is f.txt?

32. Sean Malseed | @seanmalseed | #TechSEOBoost
Oh, that’s an API of the entire SERP. Hey, neat!

33. Sean Malseed | @seanmalseed | #TechSEOBoost
FIRST COURSE OF SPAGHETTI

34. Sean Malseed | @seanmalseed | #TechSEOBoost
Google Apps Scriptscetti with a Rustic World Wide Wine Sauce

35. Sean Malseed | @seanmalseed | #TechSEOBoost
Oh boy! That’s helpful!

36. Sean Malseed | @seanmalseed | #TechSEOBoost
Times out after running for 5 mins. How can I make it run a bazillion URLs?

37. Sean Malseed | @seanmalseed | #TechSEOBoost
But what about...

38. Sean Malseed | @seanmalseed | #TechSEOBoost
But what about...
Requests to Google made from Google Apps Script are
made from an internal Google IP address, not yours!

39. Sean Malseed | @seanmalseed | #TechSEOBoost
Sample this spaghetti yourself!
Google Sheet with full working script
served with garlic bread and a world wide
wine sauce:
https://bit.ly/test-index

40. Sean Malseed | @seanmalseed | #TechSEOBoost
SECOND COURSE OF SPAGHETTI

41. Sean Malseed | @seanmalseed | #TechSEOBoost
How about a daily automated snapshot of a keyword’s top 100 results?

42. Sean Malseed | @seanmalseed | #TechSEOBoost
Sample this spaghetti yourself!
Google Sheet with full working script
served with a fresh pesto sauce.
https://bit.ly/serptracker
Make sure you go Tools → Script Editor, then Edit →
Current Project’s Triggers and set a Time-driven trigger
to run runRankings daily.

43. Sean Malseed | @seanmalseed | #TechSEOBoost
THIRD COURSE OF SPAGHETTI

44. Sean Malseed | @seanmalseed | #TechSEOBoost
“What if we use this API to
make our own API?”

45. Sean Malseed | @seanmalseed | #TechSEOBoost
Generic Inception Meme

46. Sean Malseed | @seanmalseed | #TechSEOBoost
Google Script can run as a web app, and just return JSON-P to beat CORS

47. Sean Malseed | @seanmalseed | #TechSEOBoost
Look ma, no CORS!

48. Sean Malseed | @seanmalseed | #TechSEOBoost
Sample this spaghetti yourself!
Google Apps Script ready to File → Make
a Copy served with butter and a cors-free
roll.
https://bit.ly/rankapi
Make sure you go File → Make a copy, then Publish →
Deploy as web app and set a Execute the app as to Me
and Who has access to the app to Anyone, even
anonymous.

49. Sean Malseed | @seanmalseed | #TechSEOBoost
This one simple trick
works so well, you
could build a whole
back-end out of it

50. Sean Malseed | @seanmalseed | #TechSEOBoost
Links
–
https://bit.ly/suggkw - Google Sheet with a formula to pull the live Google Suggest API
https://bit.ly/test-index - Bulk URL indexation tester sheet
https://bit.ly/serptracker - Daily SERP top 100 search result snapshot sheet
https://bit.ly/rankapi - JSONP keyword rank API in Apps Script that you can integrate anywhere
https://developers.google.com/apis-explorer - Google’s awesome list of official APIs
https://realkeyword.com - Experimental live keyword rank checker using the Apps Script API thing

51. Sean Malseed | @seanmalseed | #TechSEOBoost
Thx!
–
Sean Malseed
@seanmalseed
greenlanemarketing.com