Neural Machine Translation: a report from the front line
•Download as PPTX, PDF•
1 like•1,575 views
This presentation was given at various events in June 2017 on the current status of Neural Machine Translation development at Iconic. Rule based, statistical, hybrid, neural - at the end of the day it's all machine translation. At Iconic, we've been "doing neural" for over 12 months in various guises but, frequently, we find that our clients don't care what we use once we get the job done. In these slides, we go through a number of case studies involving MT and show how fit for purpose translations were delivered, combining various different approaches to MT.
Recommended
Recommended
More Related Content
What's hot
What's hot (15)
Similar to Neural Machine Translation: a report from the front line
Similar to Neural Machine Translation: a report from the front line (20)
More from Iconic Translation Machines
More from Iconic Translation Machines (10)
Recently uploaded
Recently uploaded (20)
Neural Machine Translation: a report from the front line
- 1. Neural MT Separating the hype from reality a report from the front line TAUS Webinar / Industry Leaders Forum / LocWorld 34
- 2. The Neural Narrative Great, thanks. Let me use it now…
- 3. LevelofDifficulty Korean Japanese French IT Marketing Patent Gisting PEMT Perfect German Use caseIndustryLanguage Use cases for machine translation
- 4. Impact of Neural MT Use cases covered by generic MT Use cases needing custom MTThe Bar
- 5. Impact of Neural MT Use cases covered by generic MT Use cases needing custom MTThe Bar Neural MT has raised the bar
- 7. Of course. We’re a team of MT experts. This is a big part of the value that we bring to the table. We’re not just taking open-source tools off the shelf. We’re innovating, researching, developing new processes. Same for Neural MT. e.g. lexically constrained decoding Neural MT @ Iconic “Do you ‘do’ Neural MT?”
- 8. It’s one of the ways. MT is not a one-size-fits-all technology. What constitutes the best approach depends on the language pair, domain, use case, and various other factors. In some cases, the best approach will be Neural MT, but not yet all the time. Neural MT @ Iconic “Is this the way you do MT now?”
- 9. When it gives the best output! When you’re customising MT, there are so many things you can do – different processors, parameters, ways of combining data and tuning. We try multiple approaches and allow our systems to use the best one. Let’s look at some case studies, but first… Neural MT @ Iconic “When do you use it then?”
- 10. The Iconic Ensemble Architecture™
- 11. Patents Case Study Average length: 7 words Average length: 30 words 3 languages Same data sets Client evaluation Ranking 1. Unusable 2. Poor 3. Adequate 4. Good 5. Excellent Criteria 90% Adequate or above 0% Unusable
- 12. Linguist Review both pass title criteria only Iconic MT passed abstracts Patents Case Study – Chinese to English 31 3938 19 0 5 10 15 20 25 30 35 40 45 Titles Abstracts Iconic MT Iconic Neural MT Outcome Iconic MT deployed in production
- 13. Linguist Review both passed on titles only NMT passed abstracts Patents Case Study – Japanese to English 52 33 0 10 20 30 40 50 60 Titles Abstracts Iconic MT Iconic Neural MT Outcome Iconic MT deployed for titles Neural MT deployed for abstracts 4 5 4 4
- 14. Patents Case Study – Korean to English 40 25 0 5 10 15 20 25 30 35 40 45 Titles Abstracts Iconic MT Iconic Neural MT Linguist Review Iconic MT below criteria Neural MT significantly better Outcome Under review! 2 6 4 3
- 15. Neural MT raises the bar for general purpose MT but the bar still needs to be tested. Customisation Case Study English to French English to Hindi BLEU 1-TER BLEU 1-TER Iconic MT 43.0 (+10.4) 55.2 (+7.7) 46.75 (+12.96) 56.4 (+5.5) GNMT 32.6 47.5 33.79 50.9 Iconic NMT 39.2 50.5 - - 2 languages 1.5M training segments IT content
- 16. The Iconic Ensemble Architecture™ Neural MT is another powerful tool in our arsenal that helps us deliver best-in-class machine translation output
- 17. “I don’t give a damn what you do!”
- 18. MT @ Iconic – what we don’t do MT MT MT MT MT MTMT MTMTMTMTMT MTMTMTMTMTMT The ability to build your own MT engines with Moses, Phrasal, OpenNMT, Nematus, Fairseq. Provide off-the-shelf general industry engines. There are some very adequate solutions for that! MT
- 19. Customised expert-built MT, using the most appropriate tool for the job, MT or otherwise. MT @ Iconic – what we do do! Develop products and solutions that incorporate machine translation – not just access to an API.
- 20. Engage our expert team on an Neural MT project to see if it works for your content Neural MT – Early Adopter Program To date Custom-developments with some of our closest partners Now Inviting early adopters to expand the range of casesEarly Adopter Program
- 21. Thank You! john@iconictranslation.com @johntins / @iconictrans
Editor's Notes
- Direct people to previous talks I’ve given about the impact and history of neural MT: SUMMARY it’s promising, but it’s not a one size fits all taking over approach still looking case by case in short term long term, wait and see, it’s exciting
- Been speaking recently about Neural MT The (commercial) narrative around neural MT is moving even faster than the pace of development! We’ve gone very quickly from explaining what it is, to companies showing initial results, to having fully blown production systems producing the best results ever – and in some cases from companies who never even offered MT before. Wow that’s impressive Our team at Iconic know a little bit about MT but you can call me a cynic if we take the approach of still trying to manage expectations while we all learn a little more about Neural MT. Much of it’s still marketing Still needs to be contextualised This was always the case with MT, and that doesn’t change with Neural MT
- Short-term impact | Long-term prospects
- Ultimately just another type of MT so we’re still have a lot of the same issues, and some new issues Whether that custom MT is neural, or SMT, or hybrid, STILL depends. It’s still to be judged on a case by case basis
- Ok, so let’s talk about our approach neural MT at Iconic, but we’ll get some key question out of the way up front. Questions we're asked frequently as a provider of machine translation
- This is what we've been doing for many years now Fancy way of saying we can apply terminology With Moses, the way ppl do MT, a lot of this was out of the box. Now we have to implement it ourselves so that will separate the wheat from the chaff in terms of software.
- Beam size, phrase length, distortion limit – now training epochs, vocab size, number of hidden layers Before we talk about the HOW we do it (which is less important as I’ll point out) let’s look at what we’ve done so far with some case studies REAL RESULTS FROM THE FIELD
- A lot of ongoing patent work with some of our biggest clients which is an ideal starting point for us to test our production metal in this area TEST NEURAL MT ON REAL USE CASES AND CRITERIA “Baseline” Iconic MT here is
- WHAT: Chinese – pre-ordering system, mature 3 years, auto post-editing OUTCOME: Use Iconic where possible, because it’s quicker to retrain and more control
- WHAT: Japanese – syntax based pre-ordering system, transliteration and script normalisation Interesting, because we were doing A LOT of development before Neural MT but it was hard to make big improvements with constraints on data OUTCOME: Use Iconic where possible, because it’s quicker to retrain and more control in the short term
- Korean project Interesting, because it wasn’t something we had in production before Neural MT – mainly because it was so hard. WHAT: Korean – hierarchical system It’s a live one so the results are actually still with the client – LET’S SEE NEXT WEEK, TAUS AND LOCWORLD! Internal QA on the output suggest the automatic scores are generous to Iconic MT. The Neural output is significantly better
- Still ongoing. Iconic Neural MT engines in building, but this is the baseline we establish! CONCLUSIONS HERE: Customised Iconic MT is better than general neural MT for 2 very different languages Again, even customised NMT not as good So, we will use what we can where’s it’s best. NEURAL MT CUSTOMISATION NOT YET VERY POWERFUL ACROSS THE BOARD. WE’RE STARTING TO GET AN INTUITION WHERE IT WILL HELP AND NOT.
- Where does NEURAL FIT IN? HOWEVER, it’s all well and good looking being the curtain and how we do this but at the end of the day, a particular quote springs to mind….
- This was a quote from a prospective client on a call last month. I was in the midst of explain what we use in which cases, where neural MT fits in, and they interrupted me as said “I don’t give a damn what you use”. I thought for a second as I was disrupted from my flow and thought, he’s right you know. I’m telling people to leave it to us, and that’s what he wants. Why does it matter how the translations are produced? Does it really matter if it’s neural or not – once it’s guaranteed as the best of what we could achieve, including using neural. I don’t think it does to anyone! But because you’ve decided to watch, I’ll give you some insight into what we do and don’t do
- - With SMT/NMT, whatever, this it will get you so far before more expertise is required. This is the case now more than ever with NMT and how experimental it is. - Not trying to provide off the shelf generic engines. Never done that. That’s the realm of Google. You’ll find it’s quite good for the general use case! Good basis for customisation.
- We’re doing what we’ve always done! EDISCOVERY REGULATORY COMPLIANCE WHERE MT IS A PART OF A BROADER SOLUTION
- To close the loop on this story with Iconic and NEURAL MT We’re learning, fast, and the more REAL opportunities there are the better for everyone. Let us know if you have any questions or if it’s something you’d like to explore. THANKS!