[CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito

@ CODEBLUE 2016 on Thu 20 Oct 2016
DAI SHIMOGAITO
OSAKA DATA RECOVERY ( daillo,inc. )

Who is Dai Shimogaito ?
Dai Shimogaito is a Japanese ,
Data Recovery Engineer – Retrieving data from computer crash
Digital Forensic Investigator – Examining digital evidences
Cyber Security Researcher – On hidden data area in HDD
h t t p s : / / w w w . f a c e b o o k . c o m / d a i . s h i m o g a i t o

Let’s Open it and see the structure !

1. DISK
1 2
3
This circle flat disk like a mirror is
the data recording DISK.
This part holds DATA and Firmware.

3
Read and Write HEAD is located at
the tip of the black rectangle part,
SLIDER.
2. Head Stack Assembly ( HSA, Head )
2

Main Controller and ROM are located.
Rom contains the 1st part of firmware.
Data port
Power port
RAM
3
3. PCB ( Printed Circuit Board )

Firmware is the implemented software for controlling
the movement of DISK and HSA to Read/Write data.
ROM contains the starting part of the firmware.
DISK contains the rest of the firmware.
4. Firmware
4

SA and UA
SA Service Area
Most of the firmware ( SA modules ) is stored
UA User Area
User data such as operating system, pictures,
and document files and directories are saved
Spare sectors are here

SA and SA Module
SA Service Area
Most of the firmware ( SA modules ) is stored
SA Module
Each module has its own function as firmware
such as P-List, G-List, S.M.A.R.T. and ATA-PW.
The number of SA modules differes depending
on the design of the product

1. Power ON
2. Controller reads ROM
3. Disk spins up and Head moves to SA
4. Controller reads SA Modules
5. Ready
What happens during HDD booting

SA ModuleROM SA Module SA ModuleSA Module SA Module SA ModuleSA Module SA Module SA Module
Power ON Ready
SA Module
RAM
Controller

Power ON Ready
RAM
Controller
Power ON Not Ready
RAM
Controller
The cause could be,,,,
1. Head is bad for reading the SA Module
2. Disk area for the SA Module is bad
3. The content of the SA Module is bad
Operating System not found
Impossible
to access
any data

Internal Sector Location Management
Head 0
Head 1
Which Cylinder ( = Track ) ?
Which Head ( = Surface ) ?
Which Sector ?
By CHS, the physical location of a sector inside the HDD can be specified.
PBA ( Physical Block Address ) is assigned to each physical sector.
PBA 0 = CHS( 0 , 0 , 0 )
PBA 1 = CHS( 0 , 0 , 1 )
PBA 2 = CHS( 0 , 0 , 2 )
PBA 3 = CHS( 0 , 0 , 3 )
PBA 4 = CHS( 0 , 0 , 4 )
PBA 5 = CHS( 0 , 0 , 5 )
,
PBA 10 000 000 = CHS( 234 , 1 , 18 )
PBA 10 000 001 = CHS( 234 , 1 , 19 )
PBA 10 000 002 = CHS( 234 , 1 , 20 )
PBA 10 000 003 = CHS( 234 , 1 , 21 )
PBA 10 000 004 = CHS( 234 , 1 , 22 )
PBA 10 000 005 = CHS( 234 , 1 , 23 )
,
Sector is specified by
PBAinstead of LBA
inside HDD
* The values are not actual information. This is an example.

LBA is mapped to PBA
Physical sector location management
in HDD is controlled by
PBA
Logical sector location management
in PC is controlled by
LBA
PBA 0 ↔ LBA 0
PBA 1 ↔ LBA 1
PBA 2 ↔ LBA 2
PBA 3 ↔ LBA 3
PBA 4 ↔ LBA 4
PBA 5 ↔ LBA 5
PBA 6 ↔ LBA 6
PBA 8 ↔ LBA 7
-
-
PBA 640768 ↔ LBA 623001
PBA 640769 ↔ LBA 623002
PBA 640771 ↔ LBA 623003
PBA 640772 ↔ LBA 623004
PBA 640773 ↔ LBA 623005
PBA 640774 ↔ LBA 623006
PBA 640782 ↔ LBA 623007
PBA 640783 ↔ LBA 623008
Firmware

Physical sectors & LBA / ! misunderstanding !
Total number of physical sectors are equal ?
Physical sector to which LBA is not mapped

LBA is mapped to all the physical sectors ?
Physical sector to which LBA is not mapped Physical sector to which LBA is mapped

NO !

Total Number of Physical Sectors differs HDD to HDD
HDD-A
HDD-B
HDD-C
PHASE-01

Primary Defects on Disk ( P-List )
Physical sector to which LBA is not mapped Physical sector to which LBA is mapped Physical sector to which factory has skipped mapping LBA
 List of the location information of is called "P-List"
 P-List ( Primary Defects List )
 P-List is saved in SA as an SA Module
 P-List is unique and essential part of firmware
HDD-A
HDD-B
HDD-C
PHASE-02

At the time of Factory Shipment
Equal Number of LBA are mapped to each HDD
so that the capacity would be the same
HDD-A
HDD-B
HDD-C
PHASE-03

Focus on LBA mapped sectors distribution
Accessible sectors are physically NOT continuous from the 1st LBA to the last LBA
HDD-A
HDD-B
HDD-C
PHASE-04

Total number of LBA are equal to each HDD
Accessible sectors are physically NOT continuous from the 1st LBA to the last LBA
On the contrary, accessible sectors are logically continuous from the 1st LBA to the last LBA
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-04PHASE-05

Let's see how Bad Sectors appear
At the time of Factory Shipment
Mint Condition
HDD-A
HDD-B
HDD-C
PHASE-03

Bad Sectors after Bad Sector Reallocation
Bad sector after bad sector reallocation
 List of the location information of is called “G-List"
 G-List ( Growth Defects List )
 G-List is saved in SA as an SA Module
 G-List is unique and essential part of firmware
 After G-List is cleared, past data may appear.
HDD-A
HDD-B
HDD-C
PHASE-06

Bad Sectors after Bad Sector Reallocation
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-06PHASE-07

Possible to access bad sectors only by E-SE
Q1.
Can we access the LBA
mapped physical sectors ?
Q2.
Can we access the Bad
Sectors, after bad sector
reallocation, to which
LBA is not mapped ?
A1.
YES
A2.
Basically
NO
but Enhanced Secure Erase
can access exceptionally
for trying to erase data
PBA LBAFirmware
PBA Firmware LBA
HDD-BHDD-BPHASE-08PHASE-07

Comparison of 3 data erase methods for HDD
Secure Erase
( ATA Command / Purge )
Enhanced Secure Erase
Data Erase Software
( Overwrite / Clear )
&
One and Only method
which may erase the
Largest data area
Limited to the
LBA mapped area
Limited to the
LBA mapped area
or less
Physical sector to which LBA is mapped Bad sector after bad sector reallocation
Shown only the physical sectors which may be erased ( accessed ) by each method
HDD-B
HDD-B
HDD-B
PHASE-08
PHASE-08
PHASE-07&08

Comparison of 3 data erase methods for HDD
Physical sector to which LBA is mapped Bad sector after bad sector reallocation
Secure Erase
Enhanced Secure Erase
Data Erase Software
( Overwrite / Clear )
&
One and Only method
which may erase the
Largest data area
Limited to the
LBA mapped area
Limited to the
LBA mapped area
or less
All the physically existing sectors are shown
Physical sector to which LBA is not mapped Physical sector to which factory has skipped mapping LBA
HDD-B
HDD-B
HDD-B
PHASE-06
PHASE-06
PHASE-06

Survey of total physical sectors in 3 HDDs
2TB SATA HDD * 3
Same model, Same capacity
( Capacity:3 907 029 168 LBA ）
HDD-A HDD-B HDD-C
Total PBA 3 931 988 368 3 933 712 984 3 933 659 976
Difference from
LBA
24 959 200 26 683 816 26 630 808
Difference in
Bytes
12 779 110 400 13 662 113 792 13 634 973 696
Difference in % 0.635% 0.678% 0.677%
( Total PBA) - ( Total LBA ) = Difference = Surplus Physical Sectors

Survey of total physical sectors in 3 HDDs
Surplus Physical Sectors are inaccessible,
because LBAs are not mapped to them
PBA Firmware LBA
What if there is DATA ?

Enhanced Secure Erase by Ultimate Boot CD

SN and Model are recorded with finish time

Demonstration
Let's connect HDD with a write blocker and view the LBA 0 by binary editor

Firmware defines the appearance of DATA
LBAPBA Firmware
1
LBAPBA Firmware
2
Firmware 1
Physical location of MBR ( LBA 0 ) may even differ depending on the firmware
Firmware 2
LBA is NOT always mapped to the same PBA forever. It's UNSTABLE !
HDD-B
HDD-B
HDD-B
PHASE-08
PHASE-14
PHASE-01

P A R A D A I S
When LBAs are mapped to the surplus physical sectors, they become accessible
despite they used to be inaccessible even by enhanced secure erase.
1. It may remain even after initializing and formatting.
2. It may remain even after OS installing / reinstalling.
3. Malware may preexist but no way to detect.
4. Inaccessible by conventional methods.
5. Any software and data may be stored.
6. There is no restriction.
7. Whatever you want.
8. Free space FOR "SOMEONE" LBAPBA Firmware
?
▼
HDD-BPHASE-10

3 year old HDD may look like these
HDD-A
HDD-B
HDD-C
PHASE-06

PARADAIS
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-06PHASE-09

PARADAIS
Bad sector after bad sector reallocation PARADAIS
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-06PHASE-10

PARADAIS
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-06PHASE-11

PARADAIS
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-06PHASE-12

PARADAIS
HDD-A
HDD-B
HDD-C
HDD-A
HDD-B
HDD-C
PHASE-06PHASE-13

Are these physically acquired disk images ?
Could be, but not always.
HDD-A
HDD-B
HDD-C
PHASE-05

These have been missed by disk imaging tools
HDD-A
HDD-B
HDD-C

PARADAIS Activation
PARADAIS can be activated either by external or internal trigger.
1. External Activation
When a certain ATA command is sent to HDD, PARADAIS may become ready to be activated
for the next power ( boot ) session.
2. Internal ( Self ) Activation
Without any external trigger, it may be activated. Just wait until it gets activated, someday.
This trigger works for offline PCs, therefore the activation may occur even in
air-gapped control systems.
▲
Manipulating /etc/shadow
for login to Debian Linux
as root
▲
/etc/shadow
Without external operation,
unidentified partition appeared
suddenly after reboot

PROBLEMS
Consumers & Users Vendors & Makers
You should be
responsible for the
accident !
I will sue you !
We had never
expected such
incident.
Product Liability

PROBLEMS
Court Judge
Are you sure ?
Law Enforcement
Your honor,
We've examined all the
data area of the HDD.
Physically extracted
image file is a perfect
copy of the HDD.
Digital Forensics

PROBLEMS
Victim Criminal
My data is gone,,,
Hostage for RANSOM
Your data is in your HDD,
but inaccessible for you.
If you pay me ransom,
your data would be back.

PROBLEMS
None of Data Erasure software can erase all sectors.
Data Erasure

PROBLEMS
Cyber Security
What do you wanna embed here ?
What could be embedded here ?

Solutions for PARADAIS activation
1. HDD inspection before use
The more critical the data is, the better it is to inspect the firmware of HDD before use.
Block the activation of PARADAIS even if there is unidentified data there.
To do so, firmware inspection would be useful to eliminate the activating mechanism.
Erase data on the surplus physical sectors.
To do so, first LBA mapping to the surplus physical sectors is required and then erase data.
2. Select reliable distribution channels
Who do you buy HDD from ? Why do you buy HDD from them ?

This research is goin on / Important Notice
Although I have described the mechanism of HDD and PARADAIS, it is unknown if
PARADAIS exists in all HDD products of all the manufacturers.
It could be possible that it exists only in several models that I have verified so far,
because the structure and the mechanism differ depending on the design of each
manufacturer and model.
To make it more precise and clear, it is preferable to explain on each different
product. However, it could affect the product's reliability. So I've been avoiding
mentioning the name of the products and the manufacturers so far.
I would appreciate your understanding.
このPARADAISがどのメーカーのどの製品にどの程度存在しうるのかについてはまだ調査の余地が残されており、Ｈ
ＤＤはメーカごとの設計等によっても構造が異なるため、より具体的かつ正確性を確保するためには、各製品の設計
や仕様に沿った検証が本来ならば必要ですが、当研究の提言内容は、ともすれば特定のメーカや製品の信頼性に影響
を及ぼし兼ねないとの考えにより、積極的に特定のメーカ名や製品名を公表することは今の時点では控えております。
この点につきましてご理解を賜れますようお願い申し上げます。

After a Natural Disaster, HDD can look like this

DR from scratched disk had been impossible
If the surface is partly
damaged, there should be
recoverable data in the areas
which were not damaged.

The 1st step of the research completed with a good result
0.02%
94%
UP !
Newspaper : Nikkei Business Daily,
26th Septempber 2013
This was a joint research with Kansai University
and Osaka Data Recovery ( daillo,inc. )

Survey of 12 DR cases
No. Model Failure State
Difficulty
Level
After Cleaning by DDRH Effect
1 ST2000DM001 Unable to boot / Abrasion Powder B Improvement in serial port output C
2 ST2000DM001 Unable to boot / Abrasion Powder B Improvement in serial port output C
3 WD10EADS-22M2B0
Unable to boot / HSA Replacement /
FW Modification
D Read error partly solved B
4 SV1203N
FW Modification
C Read error solved B
5 ST3000DM001
FW Modification
C Improvement in serial port output C
6 ST2000DM001
FW Modification
B Improvement in serial port output C
7 ST2000DM001 Abrasion Powder a lot A No improvement D
8 ST1000DM003 Bootable E No change in serial port output D
9 ST3000DM001
FW Modification
C Read error partly solved C
10 ST31000528AS Unable to boot / FW Modification C Read error partly solved C
11 ST1000DM003
FW Modification
C Read error partly solved C
12 ST3000DM001 Unable to boot B Became bootable A
Difficulty Level
Disk surface totally turned into abrasion powder A
Disk Scratched Damage B
HSA Replacement and more processes required C
HSA Replacement required D
Minor Failure ( Part replacement not required ) E
Effect
Remarkable improvement A
Significant improvement *1 B
Improved C
No effect D
Became Worse E
This survey report was submitted to Osaka city
because the research and the development of
DDRH were partly funded by Osaka city subsidy
program in March 2016.
*1 More than 1000 read error sectors solved

Survey of 12 DR cases
Remarkable
improvement
8%
Significant
improvement
17%
Improved
58%
No effect
17%
Became
worse
0%
Disk surface totally turned
into abrasion powder
8%
Disk Scratched
Damage
34%
HSA Replacement
and more processes
42%
HSA
Replace
8%
Minor
failure
8%
Difficulty Level of Data Recovery Cleaning Effect by DDRH
Disk surface cleaning worked for approx. 80% of the DR cases.

Ongoing Research
FIRMWARE & PARADAIS
Bad
Lubricant Layer &
Disk Surface Cleaning
Good
Thank you very much for attending this lecture !

[CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to [CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito

Similar to [CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito (20)

More from CODE BLUE

More from CODE BLUE (20)

Recently uploaded

Recently uploaded (20)

[CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito