The Java Advanced Imaging is a powerful Java image processing engine underlines our popular OSGeo open source projects - including GeoTools, GeoServer, GeoNetwork, and GeoNode, and more! Tragically there has been one problem with this, the JAI library is not open source! The library originated at Sun Microsystem as a core component of the Java Runtime Environment, but was not included as part of OpenJDK collaboration. This talk explores: * Capabilities that make JAI attractive for GeoSpatial work * How JAI has been used in our community * The exciting JAI-EXT project by GeoSolutions One of the reasons our community has been so addicted to this library is its power. It explored concepts like parallel processing, and distributed parallel processing in 1999, well ahead of the curve. It is an excellent example of engineering and software design. Importantly we will cover the search for an open source alternative, and are the exciting progress in producing an open source alternative. Come see how our this foundational library is being propelled into an open source future by our community.

1. State of JAI
Java Image Processing for GeoSpatial Community

2. Java Advanced Imaging

3. What is Java Advanced Imaging?
● Image Processing Library for Java Applications
○ a “pure java” solution for image processing
● Tiling computational model
○ Images accessed tile-by-tile
○ Centrally-maintained cache for performance
○ Facilitates multi-threaded processing
● Deferred execution
○ Construct a chain of operations
○ Only compute as needed
● Threaded computation
● Object-oriented extensibility
○ Register your own operations
○ Or override the built-in operations
● Remote imaging
○ Evaluate operator chains on remote hosts
○ Transfer images tile by tile
● Border extension
● Image warping
● Pixel interpolation
● Graphics2D Drawing
● Regions of Interest (ROI)
● Image file handling
○ Using JAI ImageIO project

4. Demo!

5. What else was cool in 1999

6. How JAI has been used for GeoSpatial

7. JAI-EXT
JAI-Ext has been developed by GeoSolutions as an internal effort to replace Oracle JAI
operation implementations (remember deferred binding?).
The result is a strong open source project with an Apache 2.0 License.
Improves on JAI in three different ways:
● Adding more features to existing operations, like support for “no data” and ROI
● Improving performance of some operations
● Developing new operations

8. JAI-EXT Functionality
● Operations implemented supporting nodata
○ Affine*
○ BandMerge
○ Border
○ Crop
○ Lookup
○ Mosaic
○ Null
○ Rescale
○ Scale*
○ Statistic
○ Translate
○ Warp*
○ ZonalStatistics
* Support nearest-neighbor, bilinear, bicubic interpolation
● ConcurrentCache
○ Replacement JAI TileCache
○ Offering better concurrency
○ Uses Guava cache internally
● And more
○ Public roadmap

9. JAI-Tools
Project by Michael Bedward with a number of great ideas:
● Contour
● Image generation
● Kernel stats
● Rangelookup
● Regionalize
● Vectorize
● Zonal Stats
Project is no longer active, functionality migrating as needed to JAI-EXT.

10. Jiffle
Map algebra language by Michael Bedward:
● provides a “raster calculator” language
for dynamic processing of bands
● Integrated as a JAI Operation
Project has migrated to JAI-EXT and it is
included starting version 1.1.0

11. Jiffle Example
nir = src[7];
vir = src[3];
dest = (nir-vir)/(nir+vir);
Sentinel 2 dataset, 13
bands
Jiffle script On the fly NDVI index display

12. GeoTools ImageWorker Helper Class
Utility builder helping to run operations on images
Hides some of the complexity, optimizes repeated operations, handles differences
between JAI and JAI-EXT (e.g., can you talk about no-data, or not?)
ImageWorker worker = new ImageWorker(image);
ImageWorker = worker.scale( percent, percent, 0,0,
bilinear).
RenderedImage result = worker.getRenderedImage();

13. GeoTools GridCoverage
AbstractGridFormat format = GridFormatFinder.findFormat( file );
Hints hints = null;
if (format instanceof GeoTiffFormat) {
hints = new Hints(Hints.FORCE_LONGITUDE_FIRST_AXIS_ORDER, Boolean.TRUE);
}
GridCoverage2DReader gridReader = format.getReader(file, hints);
GridCoverage2D gridCoverage = gridReader.read(null);
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
GridCoverage
The pixels
From pixel coords to real world coords
The “meaning” of the real world coords

14. GeoTools CoverageProcessor
CoverageProcessor processor = CoverageProcessor.getInstance();
final ParameterValueGroup param =
processor.getOperation("CoverageCrop").getParameters();
param.parameter("Source").setValue(gridCoverage);
param.parameter("Envelope").setValue(envelope);
GridCoverage2D result = processor.doOperation(param);
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
CROP

15. GeoTools Operations
Equivalent of ImageWorker, but helps applying operations at the coverage (georeferenced) level
Operations ops = new Operations(null);
double scale = 0.5; // 50%
GridCoverage scaledCoverage = ops.scale(coverage, scale, scale, 0, 0);

18. JAI Licensing and Distribution
Sun Distribution License
● Free to Distribute
● Not open source!
Java Research License
● Access to source code
for research purposes only!
Distributed as “standard extension” to Java
● Additional install into JRE
● An early attempt at Java “modules”
Optional “native” acceleration mediaLib binaries
Java Runtime Environment
Core system classes
- java.lang.*
- java.io.*
- java.awt.*
- ...
Extension
javax.media.j3d.*
Extension
javax.media.jai.*

19. Making Peace with Open Source

20. Two problems with JAI
● The Binary License / Research License is not open source
○ Upset projects like OSGeo Live that would like to be purely open source
○ Blocker for uDig graduating from LocationTech
● The project has been abandoned by Oracle

21. OpenCV
● Open Source Computer Vision Library
○ Roughly comparable to JAI
○ BSD license
○ Memory bound? Does it have ability to work with images larger than memory?
○ C/C++ codebase with bindings for Java
■ Some overhead taking data across JNI divide
■ Intended use is to set up a processing chain and visualize the result
● Features
○ Can use OpenCL to share workload between CPU/GPU

22. ImageJ
● “Image Processing and Analysis in Java”
○ Much better comparison with JAI
○ Open source? Not really it is public-domain which causes problem…
○ Pure-java solution, runs anywhere
○ Memory bound, wants to load images into memory for processing
● Community
○ imagej.net
○ Popular in the scientific and health community
■ Started in the 70s in Pascal
■ Large base of community developed plugins
■ FIJI (Fiji is just ImageJ) offers a distribution of plugins that work
○ Author has now retired from National Institute of Mental health, but continues to develop
■ imagej.nih.gov/ij

23. Apache Commons Imaging
● “a Pure-Java Image Library”
○ Looks to focus on great image format support
○ Open source! Apache License
○ Pure-java solution, runs anywhere
○ Memory bound!
● Community
○ No stable release available because there are lots of outstanding issues
○ Version 0.97 released in 2009 :(

24. Emerging Requirements
● Pure-java image processing
● Extendable
○ Enjoy the ability to create custom operations
○ Opens the door to native operation (example use of OpenCV)
● no artificial limit on number of bands

25. ● Long term goal is to replace JAI? Is it in
position to do so?
● Pros:
○ Large portion of operations are rewritten
○ Key components like tile cache are rewritten
○ Integrates with JDK image support, does not
reinvent the wheel
● Cons:
○ Makes use of JAI Interfaces,
(which are still not open source)
○ Usability issues around JAI Interfaces
(programming style has changed since 1999)
JAI-EXT as an Alternative ?
It is too bad we are really stuck, if only Oracle would
make the interfaces open source we are this close to
being able to do the rest!
Oracle has made the vast majority of Java open
source, but for whatever reasons JAI missed out!

26. ● Updated approach
○ wrapper implementation at OSGeo
■ Delegate to existing JAI / JAI-Ext
○ Migrate GeoTools to RPE Interfaces
○ Provide start-up switch to change between
■ Wrapper
■ Clean room
● Cons:
○ This is an idea that requires development.
■ “I tried to look at the source code, but
apparently there is not much”
○ Will not integrate with Java native raster types
Raster Processing Engine
● Raster processing engine:
○ Modern Java API using Java 8 constructs,
literate programing style, as appropriate
○ Pure Java implementation
○ Ability to stage larger rasters as tiles in
memory and process tiles in parallel
○ Clear image processing operations, allowing
installations to use native libs to accelerate
processing if available
● Initial strategy
○ Provide new “clean room” interfaces
○ Initial implementation
■ Migrate JAI-EXT operations
○ Incubation at LocationTech
■ Benefit from strong IP practices
■ fundraising opportunities

27. Bonn Code Sprint
Raster Processing Engine Scoping Exercise

28. Bonn Code Sprint

29. Bonn Sprint Objectives
● Scope the functionality required for Raster Processing Engine
○ If JAI can not be fully brought into Open Source, what do we have to replace?
● Prototype user facing interfaces
● Quickly morphed into:
○ Wait how does JAI actually work?
○ Wait what are the important interfaces

30. We don’t write docs this way anymore

32. Bonn Sprint: Code Archeology
“Primitive civilizations must have used this to … enumerate possible categorical values?”

33. Relationship with AWT
Image: RenderedImage
● Interface
● Good for final result
● Tiled
● Relative to a larger canvas
○ Offset to (0, 0)
○ .getMinX()
○ .getMinY()
● Tile grid to larger canvas
○ Offset to (0,0)
○ .getTileGridXOffset()
○ .getTileGridYOffset()
● Tiles allow working with larger
than memory rasters.

34. Relationship with AWT
Image: PlanarImage
● Abstracts over
○ Image in memory
○ Image source
○ Operation DAG
● Does not expose TileCache
● Does not expose TileScheduler
● Execution does not happen until
○ .getData(bbox)
○ .getTile(x, y)
● Exposes Direct Acyclic Graph for
deferred Operations
○ .getSources
○ .getSinks
JAI

36. Deferred Execution
Read CropWarpSource
Encode
PNG/JPEG
Tile cache
One tile at a time: handle images larger than memory
Allows also to compute multiple tiles in parallel, one per thread
Adds the basis for distributed computation (given the right tile cache)

37. Deferred Binding
Read OpImageOpImage
Source

38. Deferred Binding
CropOpImageWrapOpImage
Read OpImageOpImage
Source
Encode
PNG/JPEG

39. JAI/JAI-EXT Entanglements
How deep are the roots of this thing?
What are we going to snag when we pull on it?
● jai-ext source:
○ public abstract class WarpOpImage extends javax.media.jai.WarpOpImage
● jai-ext implements concurrent TileCache
● TileScheduler is not concurrent
● Use of TileCache and TileScheduler is of course hidden from jai-ext ops
○ Most operations are driven by RenderedImage.getData
● TileScheduler is also hidden from jai-ext
○ Although … it is not not use Java Executors.
● So it is … possible

40. JAIRPE

41. RPE

42. JAI ParameterBlock Example
FileSeekableStream stream = new FileSeekableStream(args[0]);
RenderedOp image1 = JAI.create(“stream”, stream);
ParameterBlock params = new ParameterBlock();
params.addSource(image1);
params.add(2.0F); // x scale factor
params.add(2.0F); // y scale factor
params.add(0.0F); // x translate
params.add(0.0F); // y translate
params.add(Interpolation.INTERP_BILINEAR);
/* Create an operator to scale image1. */
RenderedOp image2 = JAI.create(“scale”, params, hints);

43. FileSeekableStream stream = new FileSeekableStream(args[0]);
RenderedOp image1 = JAI.create(“stream”, stream);
RenderedOp image2 =
ScaleDescriptor.create(image1,2.0F, 2.0F, 0.0F, 0.0F,Interpolate.INTERP_BILINEAR, hints);
JAI Descriptor Example

44. Raster Processing Engine Example
FileSeekableStream stream = new FileSeekableStream(args[0]);
Operation image1 = ImageRead.stream(fileSeekableStream)
Operation image2 = Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR).create();

45. Interaction with Java SWT Image

46. Audit of GeoTools and GeoServer Use

47. But wait ...
Oracle is using JAI inside their database product!Oracle has just donated Java Enterprise Edition to
the Eclipse Foundation, forming the Jakarta project.
I wonder if Oracle would donate the JAI interfaces to
us? It would save a lot of time.
Answer is perhaps, just one more release?
Huh?
“1.10 Oracle Multimedia Architecture”
Oracle Database Online Documentation 11g Release 1 (11.1)

48. Questions and Discussion

49. Q: Will JAI Work with Java 11?
● Pure java implementation on the CLASSPATH should work
○ We have not tried it with the module system
● The JRE “standard extensions” system was removed in Java 9
○ Replaced by “jigsaw” module system

50. JAI Descriptor Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
RenderedOp image1 =
JAI.create(“stream”, stream);
RenderedOp image2 = ScaleDescriptor.create(
image,
2.0F, 2.0F, 0.0F, 0.0F,
Interpolate.INTERP_BILINEAR,
null);
JAI and RPE Compared
RPE Operation Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
Operation image1 =
ImageRead.stream(fileSeekableStream);
Operation image2 =
Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR)
.create();

51. JAI and RPE Compared
JAI ParameterBlock Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
RenderedOp image1 =
JAI.create(“stream”, stream);
ParameterBlock params = new ParameterBlock();
params.addSource(image1);
params.add(2.0F); // x scale factor
params.add(2.0F); // y scale factor
params.add(0.0F); // x translate
params.add(0.0F); // y translate
params.add(Interpolation.INTERP_BILINEAR);
RenderedOp image2 = JAI.create(“scale”, params);
RPE Operation Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
Operation image1 =
ImageRead.stream(fileSeekableStream)
Operation image2 =
Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR)
.create();

52. References
- Java Advanced Imaging API Home Page (Oracle)
- Introduction to the Java(TM) Advanced Imaging API (SlideShare)