Human vs. AI Translation Battle in Korea: We Have a Winner!

By Ray S
February 23, 2017
Is Machine Translation Software Better Than a Human Translator? Recent advances in the field of Machine Translation (MT) have led to calls that Artificial Intelligence might soon compete with human translators in terms of accuracy. In South Korea this week, leading players in the field of MT threw their latest systems into competition with professional […]

Is Machine Translation Software Better Than a Human Translator?

Recent advances in the field of Machine Translation (MT) have led to calls that Artificial Intelligence might soon compete with human translators in terms of accuracy. In South Korea this week, leading players in the field of MT threw their latest systems into competition with professional human translators – in the so-called Human vs. AI Translation Battle.

But has MT really reached a point where a professional translator of the human variety simply can’t compete?

Machine Translation  

The term “Machine Translation” arose from terminology which was in use before the advent of the modern computer age.

Some of the earliest known machine translation was used as a code-breaking tool during World War II and in the immediate post-war era. The earliest machine techniques used purely for translation were based on a similar theory – essentially treating the source document as a message which needed to be “decoded” to form the target document.

The next generation of language technology built upon this. It was a rules-based approach, which broke the source document down into a sort of intermediate non-language which a computer could understand, and then reconstructed that non-language into whatever target language was required.

This sort of approach eventually resulted in the “Babelfish” era of machine translation (somewhat overconfidently named after the translation device – a fish which one slipped inside one’s ear in order to receive instant interpretation – from Douglas Adams’ The Hitchhiker’s Guide to the Galaxy). Users of this early generation of MT translator will remember how inaccurate – and occasionally outright embarrassing – were the translations produced using this technology!

But there was about to be an advance which paved the way for automated translation software which actually worked…

Statistical Methods of Machine Translation 

Using statistical methods in translation software was the technology’s next step forward, and this approach made Google Translate a household name. By around 2007 Google and other large-scale players had started using software which scanned the internet to find bodies of text which had already been translated by humans. These were to be used as reference points for future translations.

The aim was to locate two different bodies of work (called corpora, the plural of corpus – i.e. body) which could be used to “train” Google’s language machines on the likely translations of certain strings or phrases of text. The software could then copy these patterns for use in future projects.

This technology could only come into use because of the huge expansion in the amount of human-translated content which was now available online, and the dramatic increases in computer processor power.

Now Machine Translation actually started to work.

Digital Neural Networks – Translating Like the Human Brain? 

The most modern automated translation engines work on a system known as Digital Neural Networks (DNNs), also sometimes called NMT (Neural Machine Translation). These are theoretically modelled on the human brain, but are in reality simply another type of mathematical model which had been understood for many years.

Now though, by using GPUs (Graphics Processing Units) usually used to handle the latest video games, DNNs have leapt forward in terms of quality – to the point where they’re actually quite good, if not great, in terms of what they can be used to achieve in the field of translation. This latest generation of machine translation engines use whole sentences as the basis for comparison between all of the corpora they can find online.

The same DNN technology is now also being used in handwriting recognition, face recognition, and image classification fields.

Problems With Machine Translation and Automated Translation Software

Translation software “based on the human brain” sounds like we’ve firmly crossed into the territory of science fiction. And from the point of view of early researchers in the field, we almost certainly have!

But there are still problems which limit the quality of translation AI can achieve. These include the difficulties involved in:

1. Programming machines to recognise exceptions to grammar rule
2. Making a machine understand the context of how a word is being used
3. Locating available data on required language pairings

Problems with grammar rule exceptions are fairly self-explanatory, but let’s look at the two latter points in greater detail:

Problems With Context

Issues relating to context remain perhaps the most problematic stumbling block in the way of future generations of automated translation software:

Machines simply don’t understand language in the way that a human does.

There are several classic examples of this. Whether a document which includes mention of a “French teacher” was referring to an educator who was teaching the French language, or an educator who hailed from France, for instance.

The breakthrough in AI “understanding” required for a system to be aware of this sort of context has yet to occur. It might even be the case that a true “strong AI” or “hard AI” (the sort of Artificial Intelligence popularised in most science fiction) – rather than the limited “weak AI” or “soft AI” which we are currently capable of producing – would be required in order to understand it.

Problems With Language Pairings

The second problem with even the latest statistical NMT tools is that they rely on sets of data which have already been translated by a human. Even in the trillions of web pages which currently exist, some language pairings simply don’t appear that often.

With European languages this is less of an issue. The European Union has 24 official languages, and routinely creates sets of data with multiple language versions which NMTs can refer to. For other languages, a pairing of Farsi and Spanish for example, or Urdu and Italian, there is considerably less data for the software to work from.

These were issues which the Human vs. AI Translation Battle in Korea made all the more obvious…

Human vs. AI Translation Battle in Korea: The Winner and the Contest

Getting back to the contest, the battle between human and AI translation took place in South Korea on Tuesday 21st February.

On the AI side, the competitors were three AI translation engines. One provided by Google Inc. (Translate), one from South Korea’s Naver Inc. – the country’s top Internet Service Provider, which supplied their Papago system – and one system supplied by Systran International, whose Korean credentials were boosted by its acquisition by Korea’s CSLi in 2014. All of the systems put forward by the tech giants relied on the very latest NMT technology.

Up against them?

Four professional translators, each with a minimum of 5 years’ experience.

The Contest

The human and AI competitors were presented with 4 articles written in English, and 4 in Korean. Each human translator was assigned 1 English article to translate into Korean, and 1 Korean article to translate into English. Each AI translation engine was required to translate all the articles.

The human translators were allowed 50 minutes to complete the translations, while the AI competitors completed all 8 articles within 10 minutes.

A panel of two independent professional translators – chosen by Korea’s International Interpretation Translation Association and the Sejong Cyber University, both of whom hosted the competition – would judge the results.

So what happened?

The Winner 

With final scores rated out of 30:

  • AI: 10-15
  • Human: 25

The humans have it!

Considering the lack of efficacy of machine translation only a decade ago, this result still represents a significant advancement in computer-based translation technology. And of course the machines were much faster than the humans.

That said, the Asian Absolute team aren’t particularly fearful that we’re going to be replaced with a bank of computer processors just yet.

Not Replacing Translation Memory Just Yet

This primer on translation technology wouldn’t be complete without a quick mention of Translation Memory (TM) software. Widespread adoption of TM tools by professional translators began in the 1990s, and these days few translators would dream of working without a TM.

A TM is a database of previously translated terms and phrases which a human translator can refer to in order to speed up a translation project. Each new segment of translation is added to the database as the translator works, dramatically increasing the efficiency of a project, while leaving vital human qualities like understanding context, grammar exceptions, and cultural preferences safely in the mix.

Many of these tools can now be connected to MT engines, potentially giving a client the best of both worlds.

As the march of the machines continues we at Asian Absolute are optimistic about what the future holds – both for consumers of translation and for the highly-skilled translators with whom AI is striving to catch up:

  • Translations which no-one was ever going to pay for anyway will be done using MT – helping you, for example, to understand that Facebook message somebody sent you in Spanish
  • Translations for which there’s no real need (or budget) for top quality translation will be done by MT and TM, with a helping hand from a professional translator to rephrase the most garbled parts
  • And when publication-quality translation is needed, we will still rely on the human translator, supported by TM, MT, internet resources – even old-fashioned printed dictionaries if that’s what it takes

Translation is a highly skilled profession. We salute the engineers who are building these fantastic MT engines and, equally so, the professional translators who are giving them a run for their money.