Command Line Automation for Repetitive Tasks

Mike Osolinski
@Fearless_Shultz
Edit
CLI Automation – Using the
Command Line to
automate repetitive tasks
https://www.slideshare.net/MikeOsolinski

@Fearless_Shultz #brightonSEO
What the command line
is and how it helps us to
work more efficiently
Basic examples of how
time and effort can be
saved by using CLI Scripts
Chaining multiple scripts /
modules together to create
automated solutions
What I’m Going to Talk About.

Confidential
Basic Command Line Examples
@fearless_Shultz

Confidential
The Command
Line and how it
helps us work
more efficiently.

A command-line interface or command
language interpreter, allows a user to
issues commands in the form of successive
lines of text

What is the Command Line?
Command Prompt
http://bit.ly/windows-command-prompt

Terminal
http://bit.ly/open-mac-terminal

Bash
http://bit.ly/open-bash-linux

Cygwin
http://bit.ly/open-bash-linux

Extremely powerful (in both good and bad
ways…)

“With great power comes great
responsibility….”
- Uncle Ben

@Fearless_Shultz #brightonSEO@Fearless_Shultz #brightonSEO
Quick and easy to write ad-hoc scripts
that would otherwise take much longer

Take advantage of a huge library of pre-
built scripts.

Chain different scripts and libraries
together to create fully automated
solutions.

Save time on
repetitive tasks.

Confidential
Basic Examples Of
Using the
Command Line to
Save Time.

• Merging Files
• Optimising Images
• Extracting Data
• Running Lighthouse Reports
• Running Screaming Frog Reports
• Renaming Files
• Manipulating CSV Files
* I know, I know, bullet points kill Kittens

Confidential
You Could . . .

Create a new
master file
Open each
individual csv
Select and
copy and
paste the data
Double check
everything has
been copied

Confidential
Or . . .

Confidential
copy *.csv > merged.csv
cat *.csv > merged.csv
Windows
Linux / Mac

Work through all files and
rename them

Confidential
for /f "Tokens=*" %f in ('dir /l/b/a-d') do (rename "%f" "%f")
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
Windows
Linux / Mac

Download
images via FTP
Upload in
batches to an
image
processing
service
Tweak image
settings as
required
Download
optimised images

Confidential
jpegoptim –size 500k *.jpg
wget -i large-image-urls.txt

https://github.com/tjko/jpegoptim

Create a
new SF Log
File project
Import log
file data
Filter out
Googlebot
User Agent
Export the
data to
Excel

Confidential
cat *.log | grep Googlebot >new-file.csv
Select-String -Path "C:a_ex190207.log" -Pattern
'Googlebot'

Install Lighthouse
extension
Open each URL
in a Browser
Select options
and run
performance
test
Once each
report has
finished running,
save as a HTML
file

urls.txt

test.bat

Confidential
test.bat

• This does require some additional modules installing but it is generally very quick and
easy to get set up. Full instructions here https://edit.co.uk/blog/save-time-by-batch-
processing-lighthouse-audit-files-from-the-command-line/

Open
Screaming
Frog
Set
Configuration
Options
Wait For
Crawl to Run
Select Filter
100 KB
Images
Export the
Report

Confidential
cd "C:Program Files (x86)Screaming Frog SEO Spider"
ScreamingFrogSEOSpiderCli.exe ^
--crawl https://edit.co.uk ^
--headless ^
--save-crawl ^
--config “C:image-crawl.seospiderconfig" ^
--output-folder “C:scriptsScreamingFrog" ^
--export-tabs "Images:Over X KB"

Introducing PowerShell…

Introducing PowerShell
Scripting Panel
Command Line
cmdlet Library
PowerShell is a task-based command-line shell and scripting language built on .NET. PowerShell helps
system administrators and power-users rapidly automate tasks that manage operating systems (Linux,
macOS, and Windows) and processes

Introducing PowerShell - cmdlets
A cmdlet (pronounced "command-let") is a lightweight Windows PowerShell script that performs a
single function.
Import-CSV Invoke-RestMethodExport-CSV

Introducing PowerShell – Import-Csv
The Import-CSV cmdlet turns a
CSV file into an object which
has properties which can easily
be referenced with filters
including:
-lt (Less than)
-gt (Greater than
-eq (Equal to)
-like (Like)

Confidential
Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'|
select Address, Status,'Title 1 Length'|
Where Address -like *services*|
where Status -eq 'OK'|
where 'Title 1 Length' -gt 55

Confidential
Introducing PowerShell – Export-Csv
$myData = Import-Csv -Path 'C:BrightonSEOscriptscsv examplesinternal_html.csv'|
select Address, Status,'Title 1 Length'|
Where Address -like *services*|
where Status -eq 'OK'|
where 'Title 1 Length' -gt 55
$myData | Export-CSv 'C:BrightonSEOscriptscsv examplesinternal_html-new.csv'

Introducing PowerShell – Export-Csv

Open redirect
report from SF
in Excel
Create filter to
find any rows
with a redirect
count of more
than one.
Create a new
column to
contain the
redirect
Use a
concatenation
formula to
prepend the
rule to the first
and last URLs
Remove all the
other columns
which aren’t
required and
save.

Confidential
$myObjectsArray = Import-Csv "C:powershell-scriptsredirect_and_canonical_chains.csv" |
Where { $_."Number of Redirects" -gt 1}
$newArray = @()
$myObjectsArray |
ForEach-Object{
$redirectType = "Redirect 301 "
$addressToRedirectFrom = $_.Address -replace
'^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', ''
$addressToRedirectTo = $_.'Final Address'
$rewriteRule = $redirectType + $addressToRedirectFrom + ' ' + $addressToRedirectTo
$newArray += $rewriteRule
}
$newArray | Out-File "C:powershell-scriptstesting-export.csv"

Introducing PowerShell – Invoke-RestMethod
Invoke-RestMethod consumes
REST APIs and parses the result
and where JSON is returned
the data is converted into a
PowerShell object that can be
easily queried.

Confidential
$username = “user-name-here"
$password = “api-key-here"
$uri = "https://gtmetrix.com/api/0.1/test"
$testURL = "https://edit.co.uk"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}"
-f $username,$password)))
$testing = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f
$base64AuthInfo)} `
-Method Post -Uri $uri -Body @{url="https://edit.co.uk"} `
$testID = $testing.test_id
$testURLWithID = $testing.poll_state_url
$testing

Confidential
$fullData = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f
$base64AuthInfo)} `
-Method GET -Uri $uri+$testURLWithID
$testScore = $fullData.resources.pagespeed.Length
$testDoMTime = $fullData.results.dom_content_loaded_time
$testFullyLoaded = $fullData.results.fully_loaded_time
'The test score is:'+ $testScore
'The DOM content loaded time is: ' +$testDoMTime
'The fully loaded time is: ' +$testFullyLoaded

Linking things together

Confidential
Generate Redirect Report and Create
Redirects.

Confidential
--crawl https://edit.co.uk/ ^
--headless ^
--save-crawl ^
--config "C:redirect.seospiderconfig" ^
--output-folder "C:redirects" ^
--save-report "Redirect & Canonical Chains"
chaining-redirects.bat

Confidential
cd C:redirects
C:redirectschaining-redirects.bat
Get-Content C:redirectsredirect_and_canonical_chains.csv |
Select-Object -Skip 1 | Out-File C:redirectsredirects.csv
$myObjectsArray = Import-Csv "C:redirectsredirects.csv" |
Where { $_."Number of Redirects" -gt 1}
$newArray = @()
$myObjectsArray |
ForEach-Object{
$rType = "Redirect 301 "
$rFrom = $_.Address -replace
'^(?:https?://)?(?:[^@/n]+@)?(?:www.)?([^:/n]+)', ‘’
$rTo = $_.'Final Address'
$rewriteRule = $rType + $rFrom + ' ' + $rTo
$newArray += $rewriteRule}
$newArray | Out-File "C:redirectsredirect-rules.csv"

Confidential
Generate Lighthouse Reports.

Confidential
--crawl https://edit.co.uk ^
--headless ^
--save-crawl ^
--config “C:image-crawl.seospiderconfig" ^
--output-folder “C:scriptsScreamingFrog" ^
--export-tabs “Internal:ALL"
Chaining-lighthouse.bat

Confidential
@echo off
cd C:scriptsChainingLighthouselighthouse-reports
for /f "delims=" %%a in
(C:scriptsChainingLighthousetest-urls.txt) DO (
ECHO Line is: %%a
lighthouse --quiet --chrome-flags="--headless" %%a
)
Test.bat

Confidential
cd C:lighthouse
Get-Content C:scriptsScreamingFroginternal_html.csv |
Select-Object -Skip 1 | Out-File C:lighthouseurls-to-test.csv
$myObjectsArray = Import-Csv "C:lighthouseurls-to-test.csv"
C:Lighthousechaining-lighthouse.bat
$newArray = @()
$myObjectsArray |
ForEach-Object{
$newArray += $_.Address
}
$newArray | Out-File "C:lighthousetest-urls.txt" -Encoding ascii
C:Lighthousetest.bat

This is just the tip of the iceberg

Confidential
Useful Resources
For Learning and
Working With CLI
Scripts.

CLI Scripting Resources
Title Link
A beginners guide to batch scripting in
Windows http://steve-jansen.github.io/guides/windows-batch-scripting/
Distilled Beginners guide to command line for
SEO https://www.distilled.net/blog/seo/the-beginners-guide-to-using-the-command-line-for-seo/
SEO Strategies to impress your boss https://elite-strategies.com/impress-seo-boss-linux-command-line-tricks/
Beginners guide to Terminal on Mac https://www.makeuseof.com/tag/beginners-guide-mac-terminal/
Tom Pool CLI Hacks presentation http://bit.ly/tom-pool-command-line-hacks-presentation
Tom Pool CLI Cheat Sheet http://bit.ly/command-line-cheat-sheet
Tom Pool Podcast http://bit.ly/tom-pool-presentation-podcast
64 Resources for learning to code online https://skillcrush.com/2016/03/15/64-online-resources-to-learn-to-code-for-free/
The 100 days of code project https://www.100daysofcode.com/
PowerShell download packages https://github.com/PowerShell/PowerShell
Beginners guide to PowerShell https://blog.netwrix.com/2018/02/21/windows-powershell-scripting-tutorial-for-beginners/
Guide to using Arrays in Poweshell https://4sysops.com/archives/arrays-in-powershell-create-change-read-sort-delete/
PowerShell Foreach-Object https://www.youtube.com/watch?v=BoutUXsBGcU
PowerShell out-File https://stackoverflow.com/questions/19450616/export-csv-exports-length-but-not-name

CLI Scripting Resources
Title Link
CSV kit for PowerShell https://searchenginewatch.com/2019/02/06/technical-seo-2019-statelessness/
Loop through files in directory in powershell
https://stackoverflow.com/questions/18847145/loop-through-files-in-a-directory-using-
powershell
Understanding invoke-rest method https://4sysops.com/archives/understanding-the-invoke-restmethod-powershell-cmdlet
Working With Csvs in PowerShell https://www.petri.com/powershell-import-csv-cmdlet-parse-comma-delimited-csv-text-file
Video example of connecting to REST Method https://www.youtube.com/watch?v=7mEmQgGowMY
PowerShell add-ons https://4sysops.com/archives/my-favorite-windows-powershell-ise-add-ons/
Powershell substrings
https://4sysops.com/archives/strings-in-powershell-replace-compare-concatenate-split-
substring/
Screaming Frog Command Scheduler
https://www.greenlanemarketing.com/blog/schedule-your-screaming-frog-crawls-with-
punctual-frog/
Screaming Frog Command Line
Documentation https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#commandlineoptions
Running Screaming Frog CLI On a Server https://www.databulle.com/blog/seo/screaming-frog-headless.html
GTMetrix API https://gtmetrix.com/api/
Cygwin to run Linux and Mac commands on
Windows https://www.cygwin.com/
Huge searchable DB of REST Apis https://www.programmableweb.com/apis/directory
Regex to strip out domain from URL (For
Redirects) https://stackoverflow.com/questions/4791700/regex-needed-to-strip-out-domain-name
Content and sentiment analysis https://www.pveller.com/content-work-automation-with-text-analytics/

Closing Thoughts.

Repetitive tasks are often PERFECT
candidates for automation

If a tool doesn’t already exist to perform a
task don’t assume you have to do things
the hard way. BUILD IT.

Writing scripts is not as HARD as you might
imagine it to be.

Remember the words of Uncle Ben and BE
CAREFULL!

Thank You.

Command Line Automation for Repetitive Tasks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Command Line Automation for Repetitive Tasks

Similar to Command Line Automation for Repetitive Tasks (20)

Recently uploaded

Recently uploaded (20)

Command Line Automation for Repetitive Tasks