Presented by: Chris Sistrunk, Entergy
Abstract: IT folks have been doing it for years – building labs to test new products before rolling them out – but the concept is still rather revolutionary to most practitioners of SCADA security. Yet the benefits of a lab are many, including training staff and solving real-world problems by replicating and attacking them in the relatively low-risk lab environment.
But how do you pitch this (not inexpensive) idea in a way that gets organizational buy-in? And if your organization is just too small, what are the factors to considering when using a third-party lab? Hear ideas and ask questions of someone who evolved his organization’s capabilities from one small lab to five complete labs.
Gen AI in Business - Global Trends Report 2024.pdf
Why You Need an ICS Lab
1. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Come
see
what’s
cooking
in
my
lab:
Why
you
need
a
lab
and
how
to
get
one
Chris
Sistrunk,
PE
Sr.
Engineer
Entergy
–
Jackson,
MS
2. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Why
do
we
need
a
lab,
Chris?
3. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
What
happens
when
you
use
nmap
on
an
Industrial
Control
System
http://securityreactions.tumblr.com
4. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Why
do
we
need
a
lab?
With
a
lab,
you
can
• Test
relay
and
RTU
seAngs
on
a
replica
of
producDon
systems
• Test
new
firmware
before
issuing
to
field
• Perform
root-‐cause
analysis
– Why
is
this
device
locking
up
once
a
month?
• Try
out
new
equipment
from
a
vendor
5. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Why
do
we
need
a
lab?
Save
Dme
&
money
by
• CreaDng
standard
seAngs
templates
• Find
problems
before
they
are
widespread
(Not
having
to
recall
units
with
firmware
issues)
• Develop
and
test
equipment
pilots
in-‐house
rather
than
hiring
a
company
to
do
it
• Use
lab
equipment
as
emergency
spare
6. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Why
security
tesDng?
• Not
all
SCADA/relay
vendors
do
negaDve
or
security
tesDng
at
their
factories
• Even
if
they
did,
they
can’t
test
equipment
the
EXACT
way
that
you
use
it
• Test
your
own
equipment
before
hackers
or
some
drive-‐by
malware
does
it
for
you
• Use
the
results
to
miDgate
vulnerabiliDes
7. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
What
kinds
of
tesDng?
9. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
What
would
be
your
stuxnet?
• Be
a
hardhat
hacker
• Think
like
an
aacker
who
has
your
prints!
• Build
your
systems
with
layers
of
defense
• If
you
find
a
vulnerability,
let
your
vendor
know
(they
might
even
have
a
patch)
“To
make
things
work
well,
you
must
break
them!”
10. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
How
I
Audit
SCADA
Systems
http://securityreactions.tumblr.com
11. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
OK,
how
do
I
get
a
lab?
12. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
OK,
how
do
I
get
a
lab?
• Ask
your
boss!
Ask
the
CIO!
Ask
Ask
Ask!
• If
you
are
the
boss,
ask
your
best
people
what
they
want
in
their
lab
and
go
buy
it!
• Put
together
a
plan
or
a
business
case!
– Add
it
to
NERC/CIP
compliance
budget
(big
driver)
• Go
get
spare
equipment
and
make
a
rack!
• Start
small
and
add
to
it.
– Mine
started
as
2
relay
racks
in
my
cubicle
15. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Can’t
afford
one,
don’t
have
the
manpower,
don’t
have
the
experDse?
• 3rd
party
tesDng
such
as
Enernex,
Digital
Bond,
Kinectrics,
CimaDon
to
name
a
few
• The
US
Gov’t
has
the
Idaho
NL
NaDonal
SCADA
Test
Bed,
Pacific
NW
NL,
&
Sandia
NL
• Colleges
such
as
Louisiana
Tech,
Mississippi
State,
Jackson
State
have
power,
SCADA,
and
security
equipment
in
their
labs
• Farm
out
the
tesDng
and
work
with
them
to
get
the
results
you
want
&
capitalize
the
test
costs
16. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
To
be
the
best,
you
need
the
best
tools!
18. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Transmission
HQ
Labs
• Transmission
HQ
moved
from
NOLA
to
Jackson
• Business
conDnuity
aTer
Hurricane
Katrina
• Brand
new
building
in
Fall
of
2009
• 5
large
rooms
designated
for
lab
space
– Relay
&
SCADA
Lab
– CommunicaDons
&
Security
Lab
– Real-‐Dme
Power
System
Simulator
Lab
– Mississippi
Grid
Lab
– High
Voltage
Lab
23. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Relay
&
SCADA
Lab
• THE
LAB
OF
MY
DREAMS!
• We
can
replicate
almost
any
substaDon
• Test
new
configuraDons
• Test
problemaDc
field
configuraDons
• Test
new
firmware
&
soTware
• Test
drive
new
equipment
• Train
relay
&
RTU
technicians
and
engineers
25. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
CommunicaDons
&
Security
Lab
• SubstaDon
Hardened
Router
&
Switch
• Radios
of
different
bands
and
technologies
• Six-‐sided
PSP
for
simulaDng
CCA
sites
• Several
field
firewalls
• Wurldtech
Achilles
Fuzzer
– Test
network
robustness
of
devices
– Fuzzing
DNP3,
Modbus,
&
IEC
61850
– Test
new
RTU
&
Relay
firmware
patches
– Will
network
storm
affect
control
outputs?
26. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
CommunicaDons
&
Security
Lab
• Custom
DNP3
Fuzzer
– Created
by
Adam
Crain
to
test
openDNP3
– Closed
source
for
now
– Tests
DNP3
*Client*
and
Server
– Project
Robus
– hp://Automatak.com/robus
– Plan
to
release
as
open
source
next
year
…stay
tuned
29. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Power
Real-‐Time
Simulator
Lab
“Hypersim
is
the
only
real-‐Dme
digital
simulator
with
the
power
to
simulate
and
analyze
very
large-‐scale
power
systems
with
more
than
2000
three-‐phase
buses.”
-‐
hp://www.opal-‐rt.com
• Simulate
different
fault
scenarios
– Will
the
Relay
A,
B,
C
have
a
misoperaDon?
– Will
relay
fault
acDvity
affect
comm
(vice
versa)?
• R&D
&
commissioning
tests
31. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Mississippi
Grid
Lab
• MulDpurpose
type
lab
used
by
Entergy
Mississippi
T&D
Grid
Engineers
• InspecDng/repairing
equipment
• Pre-‐test
new
panels
before
field
installaDon
• Spare
parts
inventory
32. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
High
Voltage
Lab
33. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
High
Voltage
Lab
• The
Hi-‐VARC
(High
Voltage
AC
ResisDve
Current)
test
set
provides
rapid,
automaDc
evaluaDon
of
MOV
arresters
and
polymer
insulators
using
AC
voltages
up
to
132kV.”
hp://www.jmxservices.com
• InspecDon
&
root
cause
of
failed
insulators,
HV
circuit
breaker
components,
etc
34. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Last
but
not
least…
35. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
Go
make
stuff…Go
break
stuff
36. 8th
Security
Summit
Portland,
Oregon
9th
Security
Summit
Denver,
Colorado
A
Few
Thoughts
SCADA
Security
isn’t
easy
• Doing
the
best
we
can
with
what
we
have
SCADA,
Relay,
&
Security
Labs
• Having
a
lab
is
so
valuable
for
tesDng,
troubleshooDng,
breaking
&
fixing
stuff
• Yes
I
have
a
fuzzer
and
I’m
not
afraid
to
use
it
DNP3/IP
Secure
AuthenDcaDon
v5
• Please
tell
your
vendors
you
want
NEED
it