Unlocking Australia’s Landsat Archive: Lessons learned from Geoscience Australia’s journey towards transforming the way we use satellite data

1. Unlocking the Landsat Archive,
the Australian Geoscience Data
Cube & etc
Adam Lewis, Geoscience
Australia

2. Outline
• Steps toward unlocking GA’s Landsat Archive
• Putting data to use : the Australian Geoscience Data Cube
(AGDC)
• Where the data cube is heading - some future directions
• Discrete Global Gridding System
Business Systems Development NEO Team Brief

3. National Flood Risk Information Portal
Value layer
Delivery,
storage and
analysis layer
Data acquisition
and preparation
layer
Emergency
managementWater
Private
sector
Carbon
accounting
Climate
and
weather
APS 200 and
FOI reform
Emergency
management
tools
Water
toolsCarbon
accounting
tools
Climate and
weather tools
National
framework
datasets:
Authoritative
Base Image and
Landcover of
Australia
Geometric
correction
Image
generation
Observation
corrections
Analysis of
biophysical dynamics
(Green/brown/water/s
oil fraction and
indices)
Generation of
landcover map
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Scene storage
Projected grid storage
Un-projected grid storage
Virtual compute
Cloud compute
Web services
SecurityWMS
WFS
WCS
WCPS
Getting useful information out of data

4. Traditional remote sensing product process is too
slow
GA Wednesday Seminar 30/10/13 - Datacube
Petabyte heirarchical
archive: Millions of
individual scenes
Tape store accessed by
robot.
Orthorectification
calibration, cloud
Masking, atmospheric
correction, mosaicing
Client
requests
product
Identify footprint
of product in
space or time
Search catalog
order scenes
Product packaging
and delivery
Feature extraction,
algorithm application
spectral unmixing

5. National Flood Risk Information Portal
Value layer
Delivery,
storage and
analysis layer
Data acquisition
and preparation
layer Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Historical focus – collecting data
Historical strength
-1979 Australian Landsat
Station
- circa 600,000 Landsat
scenes
- unique archive over Australia
(now largely repatriated to the
USGS)
- raw data, on tape, ~200 tB
John MacDonald &
Warren Serone, 2012

6. National Flood Risk Information Portal
Data acquisition
and preparation
layer
Image
generation
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Automate the production of images
Automated systems ‘Process Management Application’
Baseline: 50 scenes per day, manual (3 fte)
Target: 1000 scenes per day
Actual: 2000+ scenes per day
Driver: International Forest Carbon Initiative

7. National Flood Risk Information Portal
Data acquisition
and preparation
layer
Image
generation
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Deal with the storage problem
Automated systems ‘Process Management Application’
Baseline: 50 scenes per day, manual (3 fte)
Target: 1000 scenes per day
Actual: 2000+ scenes per day
Driver: International Forest Carbon Initiative
Storing processed images - Earth
Observation Data Store
- Baseline: a few hundred scenes
- Jan 2012: ~500,000 records
- Dec 2012: >2,000,000 records
Driver: International Forest Carbon Initiative

8. National Flood Risk Information Portal
Data acquisition
and preparation
layer
Image
generation
Geometric
correction
Observation
corrections
Analysis of
biophysical dynamics
(Green/brown/water/s
oil fraction and
indices)
Generation of
landcover map
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Calibration to produce a measurement in x,y,z,t
Calibration – Surface reflectance
Baseline: empirical methods
Target: physics-based method
Actual: community acceptance,
published method
Driver: Data stewardship
Correlation betw een Landsat 5 and 7 in Band 1
y = 0.9796x
R2
= 0.986
0
1000
2000
3000
4000
5000
6000
7000
0 1000 2000 3000 4000 5000 6000 7000
Landsat 7 (Surface reflectance x 10000)
Landsat5(Surfacereflectancex10000)
Series1
Linear (Series1)

9. National Flood Risk Information Portal
Data acquisition
and preparation
layer
Image
generation
Geometric
correction
Observation
corrections
Analysis of
biophysical dynamics
(Green/brown/water/s
oil fraction and
indices)
Generation of
landcover map
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Quality assessment / filters
Calibration – Pixel quality assessment
Baseline: accepted methods exist
Target: implement an accepted method
Actual: achieved
Driver: Australian Space Research
Program, Unlocking the Landsat Archive

10. National Flood Risk Information Portal

11. National Flood Risk Information Portal
Data acquisition
and preparation
layer
Image
generation
Geometric
correction
Observation
corrections
Analysis of
biophysical dynamics
(Green/brown/water/s
oil fraction and
indices)
Generation of
landcover map
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Algorithms to estimate biophysical parameters
Extracting ‘biophysical primaries’,
i.e,. water (in this case)
Baseline: no accepted methods
Target: published, accepted, available,
rapid, automatic method
Current state: accepted, automated, un-
published, involves commercial
software.
Driver: Emergency management

12. Density of quality-assured observations (15 years)
Lewis and Thankappan, AOMSUC October 2013

13. How do you work with this???
Lewis and Thankappan, AOMSUC October 2013

14. GA Wednesday Seminar 30/10/13 - Datacube
Value layer
Delivery,
storage and
analysis layer
Data acquisition
and preparation
layer
Emergency
managementWater
Private
sector
Carbon
accounting
Climate
and
weather
APS 200 and
FOI reform
Emergency
management
tools
Water
toolsCarbon
accounting
tools
Climate and
weather tools
National
framework
datasets:
Authoritative
Base Image and
Landcover of
Australia
Geometric
correction
Image
generation
Observation
corrections
Analysis of
biophysical dynamics
(Green/brown/water/s
oil fraction and
indices)
Generation of
landcover map
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Scene storage
Projected grid storage
Un-projected grid storage
Virtual compute
Cloud compute
Web services
SecurityWMS
WFS
WCS
WCPS
Capture, analysis and application of Earth obsvns

15. Data organised for HPC - time series observations
GA Wednesday Seminar 30/10/13 - Datacube

16. Data organised for HPC - time series observations
Lewis and Thankappan, AOMSUC October 2013
Tile Count
(Currently approx. 4M tiles)
Landsat Scene Count
(Currently approx. 650k
scenes)

17. GA Wednesday Seminar 30/10/13 - Datacube
Value layer
Delivery,
storage and
analysis layer
Data acquisition
and preparation
layer
Emergency
managementWater
Private
sector
Carbon
accounting
Climate
and
weather
APS 200 and
FOI reform
Emergency
management
tools
Water
toolsCarbon
accounting
tools
Climate and
weather tools
National
framework
datasets:
Authoritative
Base Image and
Landcover of
Australia
Geometric
correction
Image
generation
Observation
corrections
Analysis of
biophysical dynamics
(Green/brown/water/s
oil fraction and
indices)
Generation of
landcover map
Data acquisition
-Public good data (Landsat, MODIS, GMES Sentinels)
-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)
Scene storage
Projected grid storage
Un-projected grid storage
Virtual compute
Cloud compute
Web services
SecurityWMS
WFS
WCS
WCPS
Getting useful information out of data

18. GA Wednesday Seminar 30/10/13 - Datacube
Four-month non-interpolated median NDVI for
entire Murray Darling Basin
• Initial Datacube test area
• 2,112,000,000 pixels (i.e. 2.1 Billion).
• Every observation can be traced back to
its source capture image through
provenance information layers

19. Normalised 15-year surface water count (25m)
Lewis and Thankappan, AOMSUC October 2013

20. Normalised 15-year surface water count (25m)
Datacube Overview - 13/09/2013
Area NE of Lake Eyre showing channel bathymetry and porous dunes

21. Normalised 15-year surface water count (25m)
Lewis and Thankappan, AOMSUC October 2013
Area NE of Lake Eyre showing channel bathymetry and porous dunes

22. Normalised 15-year surface water count (25m)
Lewis and Thankappan, AOMSUC October 2013
Area NE of Lake Eyre showing channel bathymetry and porous dunes

23. Normalised 15-year surface water count (25m)
Datacube Overview - 13/09/2013
Area NE of Lake Eyre showing channel bathymetry and porous dunes

24. Normalised 15-year surface water count (25m)
Datacube Overview - 13/09/2013
Area NE of Lake Eyre showing channel bathymetry and porous dunes

25. Some next steps for the data cube
Steering committee of key stakeholders– GA, NCI, CSIRO
Data reside on the RDSI – NCI node
Exploring the relationship between data and models
More data
• Geology - radiometrics; gravity; ASTER mineral maps
• Topographics – elevation, slope, topographic elements
• Climate surfaces
• Additional EO datasats: Landsat-8, MODIS, Landsat-MSS,
other, future satellites – Sentinel-2; himawari-8/9
• Derived measurements: Fractional cover, Surface Water,
Burnt areas, etc (using nationally accepted algorithms
developed through collaborative efforts)
Business Systems Development NEO Team Brief

26. Discrete Global Grid Systems
A common global grid architecture would allow us to:
• Organise measurements over the globe
• Calculate gradients faithfully
• Compare time-series of globally distributed data
• Make statistically meaningful regional comparisons of global
data
• Compare and combine data from multiple measurements
taken at different resolutions
• Improve operation of numerical models
• Document the precision as well as location of spatial data on
the globe
Towards a Global Discrete Nested Grid

27. Kimerling/Goodchild Criteria for gridding systems
Towards a Global Discrete Nested Grid
Criterion
Criteria in Kimerling et al. (1999)
(Goodchild's Numbers given in
parentheses)
Criteria in Goodchild (1994)
1.Domain is globe
Areal cells constitute a complete tiling of
the globe, exhaustively covering the
globe without overlapping. (3,7)
1. Each area contains one point
2. Equal area
Areal cells have equal areas. This
minimizes the confounding effects of
area variation in analysis, and provides
equal probabilities for sampling designs.
(2)
2. Areas are equal in size
3. Same topology
Areal cells have the same topology
(same number of edges and vertices). (9,
14)
3. Areas exhaustively cover the domain
4. Equal shape
Areal cells have the same shape. ideally
a regular spherical polygon with edges
that are great circles. (4)
4. Areas are equal in shape
5. Compactness Areal cells are compact. (10)
5. Points form a hierarchy preserving
some (undefined) property for m < n
points
6. Straight Edges on Projection
Edges of cells are straight in a projection.
(8)
6. Areas form a hierarchy preserving
some (undefined) property for m < n
areas
7. Perimeter Bisection
The midpoint of an arc connecting two
adjacent cells coincides with the midpoint
of the edge between the two cells.
7. The domain is the globe (sphere,
spheroid)

28. Kimerling/Goodchild Criteria for gridding systems
Towards a Global Discrete Nested Grid
8. Hierarchy
The points and areal cells of the various resolution grids which constitute the grid
system form a hierarchy which displays a high degree of regularity. (5,6)
8. Edges of areas
are straight on
some projection
9. Single point A single areal cell contains only one grid reference point.(1)
9. Areas have the
same number of
edges
10. Maximally centred Grid reference points are maximally central within areal cells. (11)
10. Areas are
compact
11. Equidistant Grid reference points are equidistant from their neighbors. (12)
11. Points are
maximally central
within areas
12. Addressing
Grid reference points and areal cells display regularities and other properties
which allow them to be addressed in an efficient manner.
12. Points are
equidistant
13. Latitude Longitude The grid system has a simple relationship to latitude and longitude.
13. Edges are
areas of equal
length
14. Arbitrary resolution The grid system contains grids of any arbitrary defined spatial resolution. (5,6)
14. Addresses of
points and areas
are regular and
reflect other
properties

29. Progressing global gridding systems
rHEALPix from Landcare Research New Zealand
OGC Special Working Group to be proposed in May
Towards a Global Discrete Nested Grid

30. What about the modelling layer?
Business Systems Development NEO Team Brief

31. Opening up new possibilities
• Establishing the ability to do things that we don’t
yet know about / are not yet possible
“For example, over 40% of the revenue from IBM
last year came from products and services that
were impossible to do just two years ago.”
GA Wednesday Seminar 30/10/13 - Datacube