Proz Virtual Conference Post-editing MT overview

Understanding Post-Editing

Kirti Vashee
kirti.vashee@asiaonline.net

Copyright © 2011, Asia Online Pte Ltd

• Growth in Word Volume for Traditional
Localization Projects
• Faster Turnaround Time Requirements
• Changing Translation Price-Value Expectations
• Increasing Acceptance of MT by Enterprise
Buyers
• New Rapidly Growing Types of Content
– Patents & Scientific Content
– Customer Support & Care Content
– Customer Conversations
– User Generated Content

Human Example Words
Corporate Brochures 2,000
Corporate

Product Brochures 10,000
Products

Software Products 50,000
User Interface

Manuals / Online Help 200,000
User Documentation
Existing Markets $31.4B
New Markets Enterprise Information HR / Training / Reports 500,000

Communications Email / IM 10,000,000

Support / Knowledge Base Call Center / Help Desk 20,000,000+

User Generated Content Blogs / Reviews 50,000,000+
Machine


• Traditional Localization Projects
– Documentation and Localization
• Focused on improving translation productivity
• Same quality deliverable but faster and cheaper

• New MT Enabled Projects
– Patents & Scientific Content
• Huge Volume – Hundreds of millions of words
– Customer Support & Care Content
• Very high value but short-lived
• Technical Support & Knowledge base
– Customer Conversations
• Editing work only, focused on corrections.


Linguistic Target Quality (TEP Level)
Translation The effort and
Quality linguistic work done
to raise RAW MT to
target quality levels is
PEMT
Raw MT Output Quality

Common Misperceptions
• The target quality level is always the same as TEP or other HT standards
• The raw MT output quality is consistent from system to system
• The corrective effort is always the same from language to language
• There is little or no “a priori” control on the MT output quality
• MT error patterns are consistent from segment to segment

Pre-Analysis of Source Error Pattern Correction
Material Linguistic Error Pattern Identification Unknown Word Handling
Profiling and Development of Linguistic
Identification of Key Target Quality Rules
Patterns Expansion of vocabulary
Terminology Development of TL Style &
Standards Expression Data
Development Corrective Feedback Process
Translation Quality
Uneven Raw MT Output Quality Development
Source Cleanup Raw Corrections
Amplification

• MT engines (especially SMT-based ones) get better with feedback
• MT is not exactly the T of the TEP process
• MT engines require upfront investments and analysis for best results
• MT engines differ from language to language (FIGS easier than CJK)
• MT error patterns can vary from segment to segment

How do you pay post-editors
fairly if each engine is different?
Tools Needed:
• Effective Quality metrics
– Automated
– Human
• Confidence scores
– Scores on a 0-100 scale
– Can be mapped to fuzzy TM match
equivalents
• Post Edit Quality Analysis
– After editing is complete or even
while editing is in progress, effort
can be easily measured

MT System Characteristics – Productivity Implications
Quality
Free Online Engines Can be useful in some languages but often lower productivity than
using TM alone and impossible to adapt to specific needs
1,000 to 3,000 Words/ Day per human editor
Average segment quality = 40% - 50% TM Fuzzy Match
Human TEP Process Typically produce 2,500 Words / Day per translator
Low Quality - Moses Less than 5% of these systems can outperform free online MT and
best case productivity may be in the 3,000 Words/Day range
Average Expert These systems can provide 5,000 to 7,000 Words/Day per editor
System Average segment quality = 60% - 75% TM Fuzzy Match
Superior Expert These systems can provide 9,000 to 12,000 Words/Day per editor
Exceptional MT These systems can provide 12,000+ Words/Day per editor

Data Preparation
Data Cleaning Translate
Training

Data
Collections
Diagnostics and
Fine Tuning

Quality
Assurance

Language Pair Foundation Data
Customer Translation Data
Domain Foundation Data
and Linguistic Assets

Your Data Client Asia Online
Bilingual Translation Memories
In domain historical translations in source 1 Identify
and target language. Language Pair

Identify
Target Language Monolingual Data
2 Top Level
Monolingual target language text and URLs of Domain
in-domain websites.
3 Upload Your 4 Process Data
Extra Data (If Available) Data

Bilingual Dictionaries and Glossaries
In domain and client specific glossaries and Receive Tuning
dictionaries. 5 and Test Set
Source Language Non-Translatable Terms File
Source language terms such as product names and
place names that should not be translated.
6 Select Best
7
3000 Segments Train Engine
Source Material To Be Translated
Source material can be analyzed and processed to
further improve quality.
Ready to Translate
Style Guides
Quality
Rules can be added to match client style guide
requirements.
8 Improvement
Copyright © 2011, Asia Online Pte Ltd Plan

LP Source Human Reference Customized Foundation
JA-EN なお, 以下の座標系の定義は Definitions pertaining to the Furthermore, the definition of the Furthermore, the following
以下の通り。 coordinate systems are given coordinate systems are as follows. coordinate system as defined.
below.
JA-EN せん断試験の管理特性を規定 Are the control characteristics of Are the control characteristics of Shear test criterion for defining
し判断基準は明確か shearing test defined to specify shear test defined to specify characteristics of the clear?
criteria for judgement clearly? criteria for judgement clearly?
JA-EN ベントチューブスポット溶接の Is the strength of spot-welds on Is the strength of spot-welds on It is the intensity of the welding
強度は確認しているか vent tubes checked? vent tubes checked? spot vent tubes?
EN-DE An alternate host can start the Alternative Gastgeber können das Alternative Gastgeber können das Stellvertretendes Gastgeber
meeting and act as the host. Meeting starten und als Gastgeber Meeting starten und als Gastgeber beginnen können und so zu tun, als
handeln. handeln. die Tagung des Aufnahmelandes.
EN-DE You can publish a recorded Sie können eine aufgezeichnete Sie können eine aufgezeichnete Sie können eine namentliche
training session that was Schulungssitzung veröffentlichen, schulungssitzung veröffentlichen, Fortbildungsveranstaltung
created with WebEx Recorder. die mit dem WebEx-Rekorder die mit dem WebEx-Rekorder veröffentlichen, mit WebEx
aufgezeichnet wurde. erstellt wurde. Fahrtenschreiber.
EN-DE Once customer approves your Wenn der Kunde Ihre Anforderung Wenn der Kunde Ihre Anforderung Wenn Verbraucher stimmt ihrem
request, the customer can select genehmigt, kann er eine genehmigt, kann der Kunde eine Antrag, der Kunde auswählen
an application to share. Applikation zum Teilen auswählen. Applikation zum Teilen auswählen. können, einen Antrag zu teilen.
EN-ES Remove the steel ball from the Retire la bola de acero de la Retire la bola de acero de la Eliminar la bola de acero de la
main oil gallery before cleaning. canalización de aceite principal canalización de aceite principal limpieza galería antes de petróleo.
antes de limpiar. antes de la limpieza.
EN-ES Continuously with the ignition Continuamente con el encendido Continuamente con el encendido Continuamente con la ignición en
on and the propulsion system conectado y el sistema de en posición on y el sistema de activo y el sistema de propulsión.
active. propulsión activo. propulsión activo.
EN-ES The average response time goal El objetivo del tiempo de respuesta El objetivo del tiempo de La meta media del tiempo de
is assigned a specific time goal. medio se asigna a un objetivo de respuesta medio se asigna a un respuesta se asigna una meta del
tiempo específico. objetivo de tiempo específico. momento específico.

Customization teaches an engine how to translate using YOUR style and vocabulary

A method of distilling a polymerizable vinyl compound selected from the
group consisting of acrolein, methacrolein, acrylic acid, methacrylec acid,
hydroxyethyl acrylate, hydroxyethyl methacrylate, hydroxypropyl acrylate,
hydroxypropyl methacrylate, glycidyl acrylate and glycidyl methacrylate, the
method comprising distilling the polymerizable vinyl compound in the
presence of a polymerization inhibitor using a distillation tower having
perforated trays without downcomers and wherein the temperature of the
inner wall of the tower is maintained at a temperature sufficient to prevent
the condensation of the vapor being distilled, whereby the polymerizable
vinyl compound is distilled without the formation of polymer.

Actual sample of Japanese to English MT output
• Requires a significant terminology database effort
• Special handling for long sentences
• Monolingual target language analysis
• Linguistic parsing

• Training of post-editors – New Skills
– MT Post Editing Is Different to HT Proof Editing
• Different error patterns and different ways to resolve issues
• Some LSPs are creating e-learning courses for post editors
• 3 Kinds of Post Editors
– Professional Bilingual MT Post Editors:
• Often with domain expertise, these editors have
been trained to understand issues with MT
and not only correct the error in the
sentence, but also create learning material
– Early Career Post Editors:
• Editing work only, focused on corrections
– Monolingual Post Editors
• Experts in the domain, but may not be fluently bilingual
• With a mature engine, this approach will often deliver the best, most
natural sounding results


Metrics That Really Count Productivity is the
• Productivity – Words translated per day per Best Quality Measure
Raw MT often has a greater number of
human resource errors than first pass human translation
• Margin – improvement in the profit margin is but:
1. MT errors are easy to see and easy
critical to greater use and adoption to fix
• Consistency – Writing style and terminology (i.e. simple grammar/ word order).
 MT + Human delivers higher quality than a 2. MT provides more accurate and
human only approach consistent terminology
3. Human errors may be fewer, but
harder to see and harder to fix.
MT with more total errors is often
Other “Useful” Quality Indicators
faster to edit and fix than first pass
Automated Metrics (Good indicators, but not absolute) human translations with fewer number
• BLEU (Bilingual Evaluation Understudy) of errors.
• F-Measure (F1 Score or F-Score)
Manual Quality Metrics (Most not designed for MT, more Margin
for HT) Time
• Edit Distance (Does not take into account complexity of edit)
• SAE-J2450 (Industry specific)

Standard TEP Excellent Average Excellent
Moses Expert Expert
Translated Words / 2,500 3,000 6,000 9,000
Day
Hourly Rate $45 $45 $45 $45
Word Rate 15 cents 12 cents 10 cents 7.5 cents

Daily Cost at Hourly Rate $360 $360 $360 $360
Daily Cost at Word Rate $375 $360 $600 $675

500,000 Word Project
Hourly Cost
$72,000.00 $ 60,000.00 $30,000.00 $20,000.00
Word Rate Cost
$75,000.00 $ 60,000.00 $50,000.00 $37,500.00
Man Days 200.00 166.67 83.33 55.56

Translator 1

Translator 2

Translator 3

Translator 4 Human Only
MT + Post Editing

Words Per Day 0 2,000 4,000 6,000 8,000 10,000 12,000

• Productivity improvement results differ by translator. The above data is derived by studying
4 different translators productivity used only human and then with the addition of MT +
human post editing by professionals
• Weaker translators often tend to benefit more from technology
• Customization is key to minimizing translator frustration
• Rapid measurement and assessment of quality is key to profitability

Incremental Improvement Training


1. Customize 2. Measure
Create a new custom engine Measure the quality of the
using foundation data and engine for rating and future
your own language assets improvement comparisons

4. Manage 3. Improve
Manage translation projects Provide corrective feedback
while generating corrective removing potential for
data for quality improvement. translation errors.


Machine Translate Compare and Score
S Original C Translation R Human
Source Candidate Reference

Note: Multiple machine 3 Measurement Tools
C translation candidates can • Human Quality Assessment
be scored at one time to • Automated Quality Metrics
compare against each other. • Sentence Evaluation
E.g. Asia Online, Google, Systran

– Original Source:
S
• The original sentences that are to be translated.
– Human Reference
R
• The gold standard of what a high quality human translation would look like.
– Translation Candidate
C
• This is the translated output from the machine translation system that you are comparing.

• The test set being measured: Different test sets will give very different
scores. Very small test sets can give misleading results.
• How many human reference translations were used: If there is more
than one human reference translation, the resulting BLEU score will be
higher.
• The complexity of the language pair: Spanish is a simpler language in
terms of grammar and structure than Finnish or Chinese.
• The complexity of the domain: A patent has more complex text and
structure than a children’s story book. It is not practical to use two
different test sets and conclude that one translation engine is better than
the other.
• The capitalization of the segments being measured: When comparing
metrics, the most common form of measurement is Case Insensitive.
• The size of the test set: Use 1,000 or more BLIND segments to get good
assessments
• The measurement software: There are many measurement tools for
translation quality. Each may vary slightly with respect to how a score is
calculated
It is clear from the above list of variations that a BLEU score number by itself has no real meaning.

Evaluation Criteria of MT output
Read the MT output first. Then read the source text (ST). Your
understanding is not improved by the reading of the ST because the MT
Excellent (4) output is satisfactory and would not need to be modified (grammatically
correct/proper terminology is used/maybe not stylistically perfect but fulfills
the main objective, i.e. transferring accurately all information.)

Read the MT output first. Then read the source text. Your understanding is
Good (3) not improved by the reading of the ST even though the MT output contains
minor grammatical mistakes .You would not need to refer to the ST to
correct these mistakes.

Read the MT output first. Then read the source text. Your understanding is
Medium (2) improved by the reading of the ST, due to significant errors in the MT output
. You would have to re-read the ST a few times to correct these errors in the
MT output.

Read the MT output first. Then read the source text. Your understanding
Poor (1) only derives from the reading of the ST, as you could not understand the MT
output. It contained serious errors. You could only produce a translation by
dismissing most of the MT output and/or re-translating from scratch.

Human evaluators can develop custom
error taxonomy to help identify key
error pattern problems .


Before Machine Translation
Source text is processed and modified.
Pre-Translation Corrections (PTC)
- A list of terms that adjust the source
text fixing common issues and
making it more suitable for translation.
Non-Translatable Terms (NTT) After Machine Translation
- A list of monolingual terms that are
used to ensure key terms are not Target text is processed and modified.
translated. Post Translation Adjustment (PTA)
Runtime Glossary (GLO) - A list of terms in the target language that
- A list of bilingual terms that are used to modify the translated output. This is very
ensure terminology is translated a useful for normalization of target terms.
specific way.

Each of the above runtime customizations can be applied in 2 forms:
Default: Applied to all jobs.
Job Specific: A different set of customizations can be applied for different clients.

Original Source Corrected Source
PrecisionTMWorkstations Precision™ Workstations
ChinaSingaporeSydney China, Singapore, Sydney
Hyper-VTM Hyper-V™
6TBExternal 6TB External
w/ with
TO Q1 TO QUESTION 1
— <wall/>:<wall/>
(d)'|"(?=[ ](HD|disp|SAS|SATA)) ${1}-inch

• Support for case sensitive and case insensitive matches
• Support for regular expressions


Term
New York Times
PCs Limited
Asia Online Pte Ltd
Fortune 500
John Jacob
Microsoft Office
Cisco Local Director
Man Yee Wai


Original Source Specified Translation
Portugal-Portuguese Portugais (Portugal)
Independent Software Vendor (ISV) éditeurs de logiciels indépendants (ISV)
South Holland Province La Province Hollande-Méridionale
Proof of Concept (POC) engagement mission de validation technique
HBA adaptateur de bus hôte
Fine print Clauses complémentaires
Standup HBA adapter pour adaptateur de bus hôte
HBA standup adapter pour adaptateur de bus hôte


Original Target Adjusted Target
double port 2 port
double-port 2 port
deux port 2 port
deux-port 2 port
I5 i5
e/s E/S
cloud computing Cloud Computing
ompm OMPM


Additional Training Data Runtime Improvements
Each custom engine is a living engine and Fine tuning to specific formats
constantly improves with use. There are many new and style guide requirements
kinds of data sources that can improve an engine’s can be performed at runtime
translation quality. without retraining the engine.
Posted Edited Machine Translations • Pre-Translation Corrections
Post editing of raw MT rapidly improves translation quality. • Non-Translatable Terms
Data Manufacturing • Runtime Glossary
Language Studio™ will analyze edits and other data and • Post-Translation Adjustments
manufacture new data to improve quality.
These features enable:
Bilingual Translation Memories
• Normalization of terms
Additional in domain historical translations in source and
target language that were not included in earlier training. • Control of preferred terminology
• Mapping of complex rules as
Target Language Monolingual Data specified in the style guide
Additional monolingual target language text and URLs of
in-domain websites that were not included in earlier training.

Bilingual Dictionaries and Glossaries
Additional in domain and client specific glossaries and
dictionaries that were not included in earlier training.
Source Language Non-Translatable Terms
Additional source language terms that should not be
translated that were not included in earlier training.

Example of Training Data More initial data provided for
training results in greater vocabulary
and grammatical coverage above the
Sufficient Data Threshold and less
post editing feedback required.
Data Volume

Sufficient Data Threshold
Data Shortfall

Post Edited Feedback and
Generated Data to Fill Gaps

Gaps in Topic Coverage

• Training data can often have gaps in coverage and an excess of data in other areas.
• Gaps in coverage reduce translation quality.
• Gaps can quickly be filled via post editing the machine translated output and submitting
the data back to the system for further learning.
• Many gaps can be filled with monolingual data only.
• Further gaps can be identified and resolved by analyzing the text that is to be translated
for high frequency terms and unknown words
• In some cases incorrect data may be statistically more relevant. Post editing will raise the
relevance of the correct grammar.

The quick brown fox over jumps the lazy dog

The quick brown fox jumps over the lazy dog
Additional corrective data generated

Buddha jumps over the wall
Siemens Wind Power CEO jumps over to Repower
by Language Studio™ Pro

Judge jumps over bench in courtroom melee
Military surveillance bot jumps over 25 foot walls
Robbie Maddison jumps over Tower Bridge on motorbike
Man jumps over Grand Canyon
Cow jumps over Moon
With IE9 in sight, Firefox jumps over 50% market share mark
Long jumper Brian Thomas jumps over a car to raise money
Rally car jumps over a crazy fan!
Kobe jumps over a speeding Aston Martin
A deer jumps over a motorcycle
A woman jogging in a California state park jumps over a 100-foot cliff to get away from attacker
An Afghan Army soldier jumps over a irrigation canal while conducting a foot patrol

Language Studio™ Pro analyzes corrections and generates other examples that include the corrected
phrase to fill gaps in grammatical patterns. Each post edited correction is amplified producing many
other corrective patterns, improving future translations.

1

Original Machine Raw Machine Human
Source File Translate Translations Post Editing

2
1
2
Post Edited
Translations
Incremental Send Raw MT and
Quality Data Analysis and Post Edited Translations
Improvement Manufacturing back to Asia Online Supported File Types


Client Requirements
An existing technology client that has large (100K+ docs) English knowledge
base and technical support document repository and wishes to make this
self-support content multilingual

Translate Subset Edit Subset
Train Initial Engine (~5000 docs)

Unedited documents Repeat
can be retranslated as
multiple times as Required
engine improves

Translated Output Translate Documents Improve Engine

Key Human Feedback
Correct
Mistranslation Targeted Corrections
of Bad Learning
Syntax/Grammar
Terminology Correct
Spelling and
Spelling Terminology
Punctuation
Correct

Initial System
Correct

Correct

Human Feedback can raise the raw
output to previously unseen quality levels

Post Editing Cost 6

Cost Per Word
MT learns from post editing feedback and quality of 5
Post Editing (Human Translation)
translation constantly improves. 4
Cost of post editing progressively reduces as MT quality 3
increases after each engine learning iteration. 2
1 MT Post Editing
1 2 3 4 5 6
Engine Learning Iteration

Post Editing Effort Reduces Over Time Publication Quality Target
The post editing and cleanup effort gets easier as the

Quality
MT engine improves. Post Editing Effort
Initial efforts should focus on error analysis and
correction of a representative sample data set. Raw MT Quality
Each successive project should get easier and more
efficient. 1 2 3 4 5 6

Job Duration and Human Resources
MT with the same number of physical human resources
Job Duration

Human Translation
can reduce the time required to complete the job (job + Human Post Editing
duration) vs. human only.
MT + human post editing reduces overall project MT
duration by multiples of human only approach. + Human Post Editing

Copyright © 2011, Asia Online Pte Ltd Human Resources

Initial System
put into
production

Changes are collected and Trained Internal Experts
added to initial corpus to drive begin initial error analysis
continuous retraining and correction process

All editors and users allowed Experienced editors
to suggest changes which goes also allowed to make
through vetting process changes

Publication Quality Target

Post-editing effort and cost can be managed by
Quality
Post Editing Effort
improving the quality and performance of the
MT engine via corrective linguistic feedback Raw MT Quality

1 2 3 4 5 6

• Hunnect: Eastern European Language Focus
• First Engine
– Customized, without any additional engine feedback
• Domain: IT / Engineering
• Words: 25,000
• Measurements:
– Cost
– Timeframe
– Quality
• Quality of client delivery with machine translation
+ human approach must be the same or better as
a human only TEP approach.

100% 25,000 Words
Translation Editing Proofing
90%

80%
10 Days 3 Days 2 Days

70%

60%
Cost

50%
Translation Post Editing Proofing
46% Time Saving
40% (7 Days)
With PEMT Approach
30% 1 Day 5 Days 2 Days

20%

10%

Time
Copyright © 2011, Asia Online Pte Ltd 38

Margin
Margin 25%

Proofing
Proofing 5% 45% Margin
Editing
TEP Editing 20%

5% Proofing

Translation 30% MT Post Editing
27% Cost Saving Human Translation 50%

20% Machine Translation

Copyright © 2011, Asia Online Pte Ltd 39

• LSP: Sajan
• End Client Profile:
– Large global multinational corporation in the IT domain.
– Has developed its own proprietary MT system that has been developed over
many years.
• Project Goals
– Eliminate the need for full TEP translation and limit it to MT + Post-editing
• Language Pair:
– English -> Simplified Chinese.
– English -> European Spanish.
• Domain: IT
• 2nd Iteration of Customized Engine
– Customized initial engine, followed by an incremental improvement based
on client feedback.
• Data
– Client provided millions of TM phrase pairs for training
– 26% were rejected in cleaning process as unsuitable for SMT training.


• Quality
– Client performed their own metrics
– Asia Online Language Studio™ was 5
BLEU points better than the clients
own MT solution.
– Significant quality improvement after 60% Cost Saving
providing feedback – 65 BLEU score.
– Chinese scored better than first pass
human translation as per end client’s
feedback
• Result
– Client extremely impressed with result 70% Time Saving
especially when compared to the
output of their own MT engine.
– Client has commissioned Sajan to
work with more languages

LRC have uploaded slides and video presentation from the conference:
Slides: http://bit.ly/r6BPkT Video: http://bit.ly/trsyhg

Linguistic Steering
Pattern Identification, Corpus Analysis,
Linguistic Problem Solver, Quality Assessment,
Linguistic Asset Development and Test & Tuning
Set Development

MT-Savvy Translators & Editors
Rapid Error Identification / Correction
Manufacture Corrective Data and Drive Early
Development of MT Engines

Less Skilled Editors to Correct Target Language
Content
Can be Monolingual, Students, Housewives
Monolingual Data Cleanup
N-gram Resolution and Preparation


 Corpus Analysis & Preparation
 Pattern Identification
 Linguistic Structural Analysis
 Linguistic Problem Solving
 Linguistic Production Process Management
 Translation & MT Engine Quality Assessment
 Rapid Quality Assessment
 Effective Use and Development of Automated Measurements
 Steering Guidance to MT Developers
 Rapid Error Detection & Correction
 Open minded translators
 Better translator workbenches and tools
 Skilled monolinguals with subject matter expertise (SME)
 Community Management
 Recruiting different types of editors
 Quality Management


• Better quality MT systems developed by experts working
together with linguists will produce the best ROI
• Low initial investment is not the best way to evaluate an MT
strategy as these cheap systems often produce marginal
benefits
• Careful metric based evaluation is the best way to evaluate
different strategies
• Quality is most likely to be a product of systems developed
in collaboration with experts (MT + Language)
• Long-term defensible competitive advantage comes from
the best systems

Be Wary of Any Instant and Free Solutions


Any LSP not using MT in 5 years time will be
marginalized or be a niche player.

In 5 years time, leading LSPs will be
translating more content in 1 year than in the
previous 5 years combined.

There will be more demand for translators
than ever before, but roles will evolve and
change.

www.kv-emptypages.blogspot.com

Understanding Post-Editing

Kirti Vashee – kirti.vashee@asiaonline.net

Follow on Twitter: @kvashee

Join the Automated Language Translation Group in
LinkedIn


Proz Virtual Conference Post-editing MT overview

Recommended

Recommended

More Related Content

Featured

Featured (20)

Proz Virtual Conference Post-editing MT overview