HOW TO PAINLESSLY MIGRATE LARGE PAGES WITH MILLIONS OF PAGES. The methodology and technical aspects of the migration of large parties: The process of technical configuration, tools, examples of traps, and more.

1. URL Platform Migration
for BIG Websites
Michal Magdziarz
DeepCrawl
CEO & Co-founder
How to avoid a disaster

2. Success Influencers
There are two main factors that affect a success of a migrations
• The quality of redirections – short & long term effect
• The quality of the new website – long term effect

3. Success Influencers
Today we will focus on the first element so the traffic immediately after
the migration doesn’t look like this.

4. Table of Contents
• Migration Process
• Technical Server Setup & Apache RewriteRules
• Migration/Redirections Testing
• Practical/Pitfalls

5. Migration Process
Maintaining Traffic, Authority & Quality

6. Migration Process
Migration Process
• Traffic Maintenance
• Authority Maintenance
• Priority URLs list
• Page Relevancy & Quality

7. Migration Process - Traffic Maintenance
domain.com/popular-page
LEGACY PLATFORM
1,410
Visits
200

8. Migration Process - Traffic Maintenance
domain.com/popular-page
NEW PLATFORM
1,410
Visits
404

9. domain.com/popular-page
Migration Process - Traffic Maintenance
NEW PLATFORM
1,410
Visits
domain.com/new-url
301
200

10. Migration Process - Authority Maintenance
domain.com/authority-page
LEGACY PLATFORM
Authority
Link
domain.com/authority-page
NEW PLATFORM
404
200

11. Migration Process - Authority Maintenance
Authority
Link
domain.com/authority-page
NEW PLATFORM
domain.com/new-url
301
200

12. Migration Process - Priority URLs list
URL Visits Backlinks Priority
% Total
Traffic
% Total
Authority
domain.com/path/page-1 2233 201 467 28% 29%
domain.com/path/page-1?parameter=par1 1233 123 259 43% 46%
domain.com/path/page-2 345 56 75 48% 54%
domain.com/path/page-2?parameter=par1 324 43 69 52% 60%
domain.com/path/page-3 303 23 63 55% 64%
domain.com/path/page-3?parameter=par1 282 12 58 59% 65%
domain.com/path/page-4 261 11 53 62% 67%
domain.com/path/page-4?parameter=par1 240 7 49 65% 68%
domain.com/path/page-5 219 6 44 68% 69%
domain.com/path/page-5?parameter=par1 198 6 40 70% 70%
domain.com/path/page-6 177 6 36 73% 71%
domain.com/path/page-6?parameter=par1 156 6 32 75% 71%
domain.com/path/page-7 135 6 28 76% 72%
domain.com/path/page-8 114 6 23 78% 73%
domain.com/path/page-9 93 6 19 79% 74%
domain.com/path/page-10?parameter=par1 72 6 15 80% 75%
domain.com/path/page-11?parameter=par1 51 6 11 80% 76%

13. Migration Process - Priority URLs list
Divide your Test URL list based on priority:
• Top 5000 - covering 70% of traffic
• Top 10K - covering 90% of traffic
• Top 100K - covering 95% of traffic
Based on page templates
• Top 10k products
• Top 10k categories
• Top 10k search results
• etc.
Based on rule type - Sample URLs per each rule

14. Migration Process - Page Relevancy & Quality
NEW PLATFORMLEGACY PLATFORM
151 results 23 results

15. Migration Process - Page Relevancy & Quality
NEW PLATFORMLEGACY PLATFORM

16. Technical Setup
Server setup, Rewrite rules & Types

17. Technical Setup - Apache Rewrite Rules
Apache RewriteRules:
• RewriteRule - Reguly Przekierowan
• RewriteCond - Warunki Przekierowan
• RewriteMap - Mapy Przekierowan
Reference: http://httpd.apache.org/docs/2.0/misc/rewriteguide.html

18. Technical Setup - Test/Work Servers Setup
redirect.domain.com/popular-page
REDIRECTION SERVER
URL
Priority List
404
new-platform.domain.com/new-url
301
200
NEW PLATFORM (TEST SERVER)

19. Technical Setup - Test/Work Servers Setup
new-platform.domain.com/popular-page
NEW PLATFORM (TEST SERVER)
URL
Priority List
new-platform.domain.com/new-url
301
200
NEW PLATFORM (TEST SERVER)
http://new-platform.domain.com

20. Technical Setup - Rewrite Rules
• /category-shoes-id234
• /category-tshirts-id89
• /category-shirts-id15
• etc
• /category/shoes/234/
• /category/tshirts/89/
• /category/shirts/15/
• etc
LEGACY URLS NEW URLS
RewriteRule ^/category-([^-]*)-id([0-9]*) /category/$1/$2/? [L,R=301]
1
1
2
2

21. Technical Setup - Rewrite Rules

22. Technical Setup - Rewrite Cond
• /category/?cat=shoes&catID=234
• /category/?cat=tshirts&catID=89
• /category/?cat=tshirts&catID=15
• etc
• /category/shoes/234/
• /category/tshirts/89/
• /category/shirts/15/
• etc
LEGACY URLS NEW URLS
RewriteCond %{QUERY_STRING} ^cat=([^&$]*).*catID=([0-9]*)(&|$)
RewriteRule ^/category/ /category/%1/%2/? [L,R=301]
1 12 2

23. Technical setup - Rewrite Maps
• /category-shoeaccessories
• /category-tshirtclothes
• /category-shirtclothes
• etc
• /accessories/shoes/
• /clothes/tshirts/
• / clothes /shirts/
• etc
LEGACY URLS NEW URLS
RewriteMap category-map txt:category-map.txt
RewriteCond ${category-map:$1} ^.+$
RewriteRule ^/category-([^/$&?]*) /${lc:%0}? [NC,L,R=301]
1
2
3

24. Technical Setup - Rewrite Maps
• /category-shoeaccessories
• /category-tshirtclothes
• /category-shirtclothes
• etc
• /accessories/shoes/
• /clothes/tshirts/
• / clothes /shirts/
• etc
LEGACY URLS NEW URLS
../category-map.txt
shoeaccessories /accessories/shoes/
tshirtclothes /clothes/tshirts/
shirtclothes /clothes/shirts/

25. Technical Setup - Rewrite Map

26. Technical Setup - [IF] & [ELSEIF]

27. Migration/Redirections Testing
404, Redirects to non-indexable, Broken 200s etc.

28. Testing – 404 Error Pages

29. Testing – Non-indexable

30. Testing – Redirect Chains

31. Testing – Thin Pages

32. Testing – Broken 200 Pages
0

33. Testing – Other potential issues
• Changes to Mobile configuration e.g. Dedicated Mobile
Consolidation to Responsive
• Changes to International hreflang setup
• Changes to Schema tags
• Changes to Titles & Descriptions
• Crawlability & Authority Distribution
• Multidomain Migrations
• Etc.

34. Practical/Pitfalls
404, Redirects to non-indexable, Broken 200s etc.

35. Practical/Pitfalls
Practical/Pitfalls
• Robots.txt – disallow for a URL in the Redirect Chain
• Don't forget to remove your test domain from rewrite rules
• Test the set up in the live ENV
• 301 vs 302 vs canonical
• Performance - one to one vs rule based
• Other: PPC, Social, Missing content

36. Pitfalls - disallow of a URL in a Redirect Chain
domain.com/legacy-url
>> 301 >>
domain.com/redirected-to-url
>> 301 >>
domain.com/redirected-to-disallowed-url
>> 301 >>
domain.com/new-url (200 status)
../robots.txt
User-agent: *
Disallow: *disallowed*

37. Practical - 301 vs 302
• Straight 301s are for the braves
• My preference is to go for 302s for 2 weeks before committing to 301
• This allows for any last minute corrections
• Please see below the result of robots.txt issue and straight 301s below
• 70% drop in traffic in 1st week, the website has never fully recovered

38. Pitfalls - Performance – one-to-one vs. rule based
• A big ecommerce shop received a poorly written one-to-one
RewiteRules
• The RewiteRule file contained 100K rules!!
• This resulted in such poor performance that the RewriteRules were
switched off as they were slowing the site significantly
• The lack of redirections resulted in a massive drop in traffic
• The 100K RewriteRules were rewritten into:
• 17 - RewriteRules
• 10 - RewriteCond
• 4 - RewriteMaps
• The traffic was partially recovered

39. Pitfalls – PPC Landing Pages
domain.com/legacy-ppc-page?utm_campaign=abc&utm_medium=bcd
>> 301 >>
domain.com/new-ppc-page (200 status)
RewriteRule ^/legacy-ppc-page /new-ppc-page? [L,R=301]
RewriteRule ^/legacy-ppc-page /new-ppc-page [L,R=301]
domain.com/legacy-ppc-page?utm_campaign=abc&utm_medium=bcd
>> 301 >>
domain.com/new-ppc-page?utm_campaign=abc&utm_medium=bcd

40. Pitfalls – Don't forget to remove your test domain
Rewrite
Rules
Redirection
Test Server
Live
Server
LIVE
Traffic
Live
Server
Rewrite
Rules
LIVE
Traffic!

41. Thanks for your time
FREE Trial:
https://www.deepcrawl.com/sem-krk