Many companies today extract data from documents and forms through manual data entry, which is slow and expensive, or through simple optical character recognition (OCR) software, which is difficult to customize. Amazon Textract overcomes these challenges by using machine learning to instantly “read” virtually any type of document to accurately extract text and data without the need for any manual effort or custom code. In this session, you learn how to extract data from documents using Amazon Textract. We also demonstrate how you can create smart search indexes and better maintain compliance with document archival rules after the information is captured.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automating document analysis and text
extraction with Amazon Textract
A I M 2 0 2
Randall Hunt
Senior technical evangelist and software engineer
AWS

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Documents are important
Primary tool of record keeping, communicating, collaborating, and transacting
Finance
Insurance
Real estate
Accounting
Tax management
Medical
Legal
Business management
Education
And many more

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
16.3 million US mortgage applications ($2.1 trillion) in
2016
*

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
*
About 240 million W-2 tax forms processed for FY 2018 in
the US

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Need for processing documents

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
How documents are processed today
Optical character recognition
(OCR)
Manual
processing
Rules and
template-based extraction

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Challenges for processing
documents: Manual processing
Expensive Error-prone Time-consuming

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Challenges for processing documents:
Manual processing
Output
1. Exempt is true
2. 28 is true
3. CPP/QPP is true
4. RPC/RRQ is true

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Error-prone Flat bag of wordsSimple documents only
Challenges for processing
documents: OCR

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Output
Extract data quickly & No code or templates to accurately maintain
Challenges for processing
documents: OCR

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Output
Start Date End Date Employer Name Position Held Reason for leaving
1/15/2009 6/30/2013 Any Company Head Baker Family relocated
Challenges for processing
documents: OCR

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Challenges for processing
documents
Output
Full Name Date of Birth Gender
John X Doe 01 01 1971
Male
First Middle Last MM DD YYYY
Female

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Limited by
accuracy of OCR
Significant development and
management overhead
Templates
are brittle
Challenges for processing documents: Rules and template-
based extraction

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Challenges for processing documents: Rules and template-
based extraction
The well-known W-2 US tax form has hundreds of variants each year

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
It looks easy, but …
…not a single corresponding pixel value in common

16. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Amazon Textract features
Text extraction Table extraction Form extraction

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Amazon Textract: Text extraction
Blocks: PAGE, PARAGRAPH, LINE, WORD
is washed by waves, and cooled

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Name Description
Blocks List of blocks identified
from the document
ID Unique ID of the unit
Relationships CHILD
Block type PAGE, PARAGRAPH, LINE, WORD
Pages Contains number of
pages in the document
Amazon Textract text extraction API:
DetectDocumentText
Name Description
Document Blob or Amazon S3 object

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Blocks: PAGE, TABLE, CELL
For each block, you get
• Text
• Confidence score
• Block relationships (e.g., cells within a table)
Amazon Textract: Table extraction

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Name Description
Document Blob or Amazon S3 object
FeatureTypes TABLES
Name Description
Blocks List of blocks identified
from the document
ID Unique ID of the unit
Relationships CHILD
Block type PAGE, TABLE, CELL
Pages Contains number of
pages in the document
Amazon Textract table extraction API:
Analyze document with tables as FeatureTypes parameter

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Blocks: PAGE, KEY_VALUE_SET
For each block of your document
• Form field name (key) and field value (value) association
• Confidence score
• Page number
• Block relationships
Amazon Textract: Form extraction

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Name Description
Document Blob or Amazon S3 object
FeatureTypes FORMS
Name Description
Blocks List of blocks identified
from the document
ID Unique ID of the unit
Relationships KEY, VALUE, CHILD
Block type PAGE, KEY_VALUE_SET
Pages Contains number of
pages in the document
Amazon Textract forms extraction API:
Analyze document with forms as FeatureTypes parameter

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Supports single-page documents
such
as images (e.g.,
mobile capture)
For multi-page documents,
up to 3,000 pages
Amazon Textract: Sync and async

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Output
Extract data quickly &
accurately
No code or templates to
maintain
Amazon Textract: Text extraction
simplified

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Output {
Start Date: 1/15/2009
End Date: 6/30/2013
Employer Name: Any Company
Position Held: Head Baker
Reason for leaving: Family relocated
}
Amazon Textract: Table
extraction simplified

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Output Full Name:
First: John
Middle: X
Last: Doe
Date of Birth:
MM: 01
DD: 01
YYYY: 1971
Gender:
Male: True
Female: False
Amazon Textract: Form
extraction simplified

28. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Text extraction: OCR reimagined
Orientation

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Text extraction: OCR reimagined
Structure variability

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Text extraction: OCR reimagined
Document variability

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Segmentation and rectification
Photometric

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Segmentation and rectification
Geometric

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Table and cell detection
Understand document structure and context to find tables

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Table and cell detection
Understand cells even without explicit boundaries

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Table and cell detection
Variable-sized rows and columns

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Field name (key) and value extraction
Full Name:
First: John
Middle: X
Last: Doe
Date of Birth:
MM: 01
DD: 01
YYYY: 1971
Gender:
Male: True
Female: False
Detect phrases or groups of words
Output

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Inferring key/value association
Detect structures of the same form without templates

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Inferring key/value association
Key/value association

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Beyond OCR: Inferring key/value association
Infer empty values
Full Name:
First: John
Middle: null
Last: Doe
Date of Birth:
MM: 01
DD: 01
YYYY: 1971
Gender:
Male: True
Female: False
Output

41. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Reference architecture: Index and search documents
Input
Uploaded document
images such as tax
forms, credit
applications, or
medical notes
Amazon S3
Uploaded
documents are
stored in data lake
AWS Lambda
A Lambda function is
triggered to initiate
document analysis
using the Amazon
Textract API
Amazon Textract
Automatically
extract text,
including key-value
pairs and tables
Amazon
Elasticsearch
Service
Extracted data and
confidence scores
are indexed to
enable document
search
Output
Perform contextual
search on millions of
documents or
integrate data into
your document
management system

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Reference architecture: Form capture
Input
Customer uses mobile
app to capture a photo of
a W-2 form
Amazon Textract
The Amazon Textract API
is integrated into the end-
user application to
automatically extract text
from the W-2 form and
auto-populate the
form fields
Customer application
Customers experience
real-time capture of tax
information by taking a
photo instead of
performing manual data
entry
Database
User-submitted data
is loaded into a
database for use in tax
preparation

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Reference architecture: Extract for NLP
Quickly turn extracted text and data into actionable insights
Input
Uploaded document
images of medical
notes, explanation of
benefits, and
patient forms
Amazon S3
Uploaded
documents are
stored in S3
NLP
Use natural language
processing to extract
insights from
medical documents
Amazon
Elasticsearch
Service
Easily search
through extracted
data and text
insights
Output
Discover
medical
insights to
improve
patient care
Amazon
Textract
Automatically
extract words,
lines of text, and
tables

45. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Amazon Textract: Launch customers

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T
Amazon Textract: Benefits

48. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Randall Hunt