Tuesday, August 19, 2008

Evaluating Computer Translation Output


Evaluating Computer Translation Output

Abdullah Hassan, Ph.D.
Sultan Idris University of Education, Malaysia
(President, Malaysian Translators Association)
prof.abdullah@fb.upsi.edu.my

Abstract

The paper begins with a brief account of efforts undertaken in developing computer-aided translation projects in Malaysia. Then, it proceeds to assess a product of such a computer-aided translation research project. It examines the accuracy of the translation of three bank reports, from English (source text) into Malay (target text). Each source text was separated into paragraphs, sentences, and in turn each sentence into meaningful units. Then, their corresponding translations in the target text were identified. The accuracy of each translated unit was evaluated. Five categories of translation errors were identified: i.e. No Translation (NT), Literal Translation (LT), Under Translation (UT), Over Translation (OT), and Wrong Translation (WT). A further analysis was done and factors contributing to such errors were identified, i.e. lexicon (terminology, concepts, etc); grammatical elements such as word order, tense, affixes, etc.; and idiomatic usages, cultural mismatch, etc. Finally, an edited version was produced. It is hoped that these evaluation efforts will produce useful remarks and suggestions for the computer engineers to enhance their programs, and hence produce better translation output that would minimize human editorial efforts. (182 words)

Key words: raw translation; evaluation; correct translation; under translation; over translation; wrong translation; literal translation; correct transation; and no translation.


1. INTRODUCTION

The Computer Aided Translation Project started in Science University of Malaysia in 1980’s, in collaboration with the Universite Joseph Fourier at Grenoble. It has many ups and downs in its development. Other Malaysian universities and agencies; National University, University of Technology, Putra University, Multimedia University, International Islamic University, National Language and Literary Agency were all involved in one way or another in such research activities. For some time, these activities were spear-headed by the Malaysian Institute of Microelectronic System (MIMOS). This particular thrust in research is collectively known as the Language Technology Cluster. It is an on-going research that had scored many successes. But, the computer-aided translation in particular has a long way to go as yet.

It is not the intention of this paper to provide a detail historical account of such research development in Malaysia. It merely wishes to focus on evaluating the accuracy of such translation output produced by a Computer-Aided Translation Project, namely that of MIMOS.

2. THE ANALYSIS

This evaluation project was conducted by the Malaysian Translators Association (MTA) at the request of MIMOS Language Technology Cluster, in 2007. Three annual bank reports (2005) were translated from English into Malay by the computer. And the raw translation produced was handed over to MTA for evaluation.

2.1 Evaluation Team

MTA assembled a team of experienced translators to conduct the evaluation. It comprised professional translators, translation trainers, and editors. They are:

a. Abdullah Hassan (Leader)
b. Atiah Salleh
c. Sa’odah Abdullah
d. Fatimah Zainal
e. Razin Ong Abdullah
f. Wan Aziz Wan Bakar

2.2 Methodology

The team decided to use a simple but effective method of evaluating the translation output, namely that of back translation. In such an activity, the source text was first separated into paragraphs and sentences. Each sentence was then parsed into meaningful units. Similar process was repeated for the target text.

Processed texts were organized into a table of three respective columns for convenience in evaluationg the accuracy of each meaningful unit involved. An example of how a sentence, and parts of a sentence being evaluated in detail, is shown below.






A detailed account of this nature was necessary to be helpful. It is hoped that detailed comments on accuracy of translation and grammatical aspects would in turn assist fellow researchers to improve their translation soft-wares.
2.3 Category of Errors

Accuracy of each translated unit was evaluated by identifying errors made, be it semantic or grammatical. This is done by carefully examining and doing a back transltion for each of the meaningful unit and sentence involved. Five categories of translation errors were identified: i.e. No Translation (NT), Literal Translation (LT), Under Translation (UT), Over Translation (OT), and Wrong Translation (WT).

A further analysis was done and factors contributing to such errors were identified, i.e. lexicon (terminology, concepts, etc); grammatical elements such as word order, tense, affixes, etc.; and idiomatic usage, cultural mismatch, etc.

When no such errors are committed, then the translation is considered Correct Translation (CT).

The above evaluation table, will illustrate these points more clearly. Comments in the third column are for the benefit of soft-ware engineers to enhance their programs.
2.4 Analysis of Translation Errors

Three translated bank reports were analysed, studied and evaluated. As analysis of the reports is lengthy and repetitive, only part of it is discussed in detail here. However, examples of cases used in the paper are drwan from all the three.

In the discussion below, each sentence is analysed and treated separately.


























2.4.1 Analysis of BCB Annual Report 2002

Sentence 1. Commerce Asset-Holding Berhad has a well established presence in the financial services industry by building upon solid foundations of expertise and sound experience as well as a commitment to move with the times.









































Sentence 2. With a strategic vision to compete, we are forging solid alliances and innovating capabilities throughout the Group to create new pillars of growth.


ENGLISH TEXT MALAY TEXT COMMENTS
With a strategic vision to compete,
dengan wawasan strategik supaya lebih berdaya saing,
- dengan is a CT for with
- wawasan strategik is a CT for a strategic vision.
- a needs not be translated.
- supaya lebih berdaya saing is an OT for to compete. It is sufficient to translate the phrase as: untuk bersaing
Edited: Dengan wawasan strategik untuk bersaing
we are forging solid alliances kami diikat pakatan kukuh - kam diikat is a WT for the phrase we are forging
- we are forging is an active construction, and should be translated in an active mode as well, i.e. kami membina
- and not as kami diikat which can be back translated into English as –we were tied.>
- pakatan kukuh is CT for solid alliances ...
Edited: .. kami membina pakatan kukuh ...
And innovating capabilities throughout the Group dan menginovasi keupayaan sepanjang kumpulan itu - dengan menginovasi keupayaan is a CT for and innovating capabilities
- sepanjang is WT for – throughout – the CT would be seluruh Kumpulan itu …
Edited version: dan menginovasi keupayaan dalam seluruh Kumpulan itu.
to create new pillars of growth.
mewujudkan tiang baru bagi pertumbuhan.
- mewujudkan tiang baru bagi petumbuhan – except for tiang - would be a CT for - to create new pillars of growth.

- however tiang – is not a CT for pillars.

- an idomatic translation would be – tunggak.

- to need not be translated.
Edited:-mewujudkan tunggak baru bagi pertumbuhan.
Edited version of the whole sentence:
Dengan wawasan strategik untuk bersaing, kami membina pakatan kukuh dan menginovasi keupayaan dalam seluruh Kumpulan itu (bagi) mewujudkan tunggak baru bagi pertumbuhan.

- preposition – (bagi) – is reguired to modify verbs – forging and inovating.

Sentence 3. As we evolve, we will continue to build upon our traditional strengths to move ahead in an increasingly competitive global environment – and achieve new milestones of excellence as a premier regional Group.

ENGLISH TEXTS MALAY TEXTS COMMENTS
As we evolve,
Seperti kami berkembang, - seperti is WT for preposition as. It is more appropriately translated as – semasa or dalam.

- as we evlove – is a reflexive construction, may be translated into – ber+verb
- can be translated as – Semasa kami berkembang – or – Dalam kami berkembang –
Edited: Dalam kami berkembang …
we will continue to build upon kita akan terus dibina - we is an inclusive first person plural pronoun. It should be translated as kami, not kita -

- This form is not differentiated in English

- akan terus dibina is a passive construction, whereas the English phrase is in active mode. It should be translated in the active mode too, i.e. akan terus membina …

Edited: kami akan terus membina …
Our traditional strengths to move ahead
berdasarkan kekuatan tradisi belayar di hadapan
- berdasarkan kekuatan tradisional (kami) .. is CT. However, first person possessive plural pronoun our is not translated.

- should ber translated as – kami ..

The finite verb form – to need not be translated in Malay.

- to move should be translated as bergerak maju, or – bergerak ke hadapan. CT

- preposition of direction – to should be translated as –ke- to indicate direction, not – di – which indicates location.

Edited: berdasarkan (pada) kekuatan tradisional kami

- preposition pada relate the verb berdasarkan to the adverbial phrase of reason.
in an increasingly competitive global environment –
dalam persekitaran yang global yang semakin kompetitif-

- dalam persekitaran yang global – is an OT. The relative pronoun yang need not be translated.
It is sufficient to translate the phrase as – dalam persekitaran global –
- yang semakin kompetitif is a CT, however, it may be better translated as – yang semakin bersaing -
Edited: - dalam persekitaran global yang semakin kompetitif -
And achieve new milestones of excellence dan mencapai kemajuan baru kecemerlangan bagi - dan mencapai kemajuan baru is an UT, –

- preposition – bagi for – of is WT. It may be translated as – yang -.

- kemajuan baru – is not idiomatic, and should be translated as – tahap kejayaan baru –

– of – in the phrase – of excellence – need not be translated.

Edited: - dan mencapai tahap kejayaan baru -
as a premier regional Group. sebagai kumpulan serantau unggul - sebagai Kumpulan serantau (yang) unggul – is an UT,
- relative pronound – yang – should be used to connect the adjective unggul to the noun Kumpulan –
Edited: - sebagai kumpulan serantau yang unggul.
Edited version:
Dalam kami berkembang, kami akan terus membina berdasarkan kekuatan tradisional kami dalam persekitaran global yang semakin kompetitif dan mencapai tahap kejayaan baru sebagai kumpulan serantau yang unggul.

2.4.2 Analysis of Other Reports

A total of 60 sentences were analysed from the three reports. However inclusion of such lengthy analysis in this paper was felt unnecessary, as they were repetitive in nature. Not even as appendices, as they would consume a lot of space. They are nevertheless an integral part of the paper.

3. FINDINGS OF ANALYSIS

Findings of the analysis are dicussed in three parts.

a. Translation of meaning of words, phrases and sentences.
b. Translation of grammatical elements.
c. Translation of idiomatic expressions and cultural elements.

3.1 Translation of Meaning

Discussions on translation of menaing of words, phrases and sentences are based on the five categories of error identified: i.e. No Translation (NT), Literal Translation (LT), Under Translation (UT), Over Translation (OT), and Wrong Translation (WT).

3.1.1 No Translation (NT)

Some words and phrases were ignored by the computer. There are words and phrases in the source text that are simply reproduced in the target text whilst some are missing, thus producing no translation (NT). Examples are as follows:

a. in short terms  in short terms (NT) – should be dalam jangka masa pendek
b. dampening  dampening (NT) – should be melembapkan
c. balance returns fundings  Balance Returns Fundings (NT) – should be dana pulangan terimbang -

This could mean that the system has provided insufficient equivalences for the computer to draw upon. As such it merely reproduces phrases that possess no equivalences.
d. In the short term, fiscal tightening and monetary restraint …  In the short term, fiskal pengetatan dan kewanan restraint … (NT)
e. The effect of the crisis on the major industrial economies will vary, depending on …  The effect of the crisis on the major industrial ekonomi akan vary, bergantung pada … (NT)
f. While these inflows supported economic growth, …  manakala aliran masuk supported economic growth … (NT)

It goes without saying that, the system need to augement its repertoire of equivalences, or design rules that would enable the system to generate such equivalences.

3.1.2 Literal Translation (LT)

The followings are examples of literal translations produced by the computer. They are merely string of words not capable of translating the original meaning. The gravity of the error is evident when a back translation is done, and such examples are provided in brackets.

a. The Company’s Act, 1965  Syarikat itu Bertindak, 1965 (BT - The Company acted in 1965)
b. Well established presence  kehadiran baik maju (BT - good presence advances)
c. sound experience  pengalaman bunyi (BT - experiencing sound)
d. commitment to move  komitmen belayar (BT - commitment to sail)
e. The People’s Republic of China  orang republik daripada China (BT - the (Chinese) republicans from China)

3.1.3 Under Translation (UT)

An under translation (UT) was produced by the computer when part of the phrases and words present in the source text were not translated accordingly into the target text. The corresponding meanings provided in the target text column, is partial and inadequate. Examples:

a. The dividend, if approved to shareholders will be paid on 30 May 2003  Dividend itu, jika diluluskan kepada pemegang saham (the unit will be paid on 30 May 2003 was not translated into Malay … (UT)
b. in the Finance and Closed-End Fund Category  dalam Kewangan dan closed-end fund category (unit Closed-End Fund Category was not transled into Malay … (UT)
c. .. the impact of export slowdown …  kesan of the export slowdown … (UT)
d. … making the region vulnerable …  membuat the region vulnerable … (UT)
e. … significant ajustments in one currency were almost instantly followed by adjustments …  … penyesuaian dalam satu mata wang ketara almost instantly followed by adjustments ... (UT).

These kinds of under translation can be handled by the system as it can translate other phrases containing similar words adequately.

3.1.4 Over Translation (OT)

Over translation occurred when an extra semantic or grammatical elements are added to the translation. Examples:
a. to compete  supaya lebih berdaya saing (Edited: only bersaing is sufficient - OT)

An OT also resulted from the translation of English grammatical elements that do not require equivalence in Malay. In other words, such grammatical elements though expressed overtly in English, need not be lexicalized in Malay.

a. the  itu/ini (a noun in Malay needs no demonstrative pronoun)
b. a/an  se/satu (a noun in Malay needs no definite or indefinite article)
c. it  nya (dative case may not be indicated in Malay, if it can be marked word order)
d. of  daripada (this is often indicated by word order)
e. was  telah (tense is redundant when time and date have been specified)
f. in  preposition (it may be translated as dalam, depending on contexts.)
g. most  paling, (may indicate superlative case depending on context, or it may be an additional and redundant element of meaning.)
h. were  adalah, (a copula, is often redundant in Malay as it does not need to indicate tense.)
i. by  oleh (this preposition is often mistaken for dengan which back translate to with. Its use would add an extra meaning.

3.1.5 Wrong Translation (WT)

Proper names such as names of companies were translated fully by the computer. Translation would not be deemed necessary for proper names. The soft-ware must be designed to recognize proper names. Examples are as follows:

a. Commerce Asset–Holding Berhad  Berhad Asset–Holding Perdagangan (WT)
b. United States  Bersatu Negeri – (Amerika) Syarikat
c. Commerce Trust Berhad  Berhad Perdagangan Amanah – Perdagangan Amanah Berhad

The computer does not follow spelling conventions in using upper case for the first letter in spelling proper nouns. However, more names of countries (Philippine, Korea), and cities (Hong Kong, Singapore) are readily recognized and spelled correctly with initial capital letters.

3.1.6 Wrong terminology

Other frequent errors committed are unsuitable terms or words used to translate certain concepts and processes. Examples:

a. rate  penarafan (should be kadar)
b. sound experience  pengalaman bunyi (should be pengalaman yang mantap)
c. throughout the group  sepanjang kumpulan itu (should be ke seluruh kumpulan)
d. magnitude  kesan (should be: besarnya...)
e. authority  pihak berkuasa (should be kuasa)
f. clinching the Best Performing Fund  berpeluk best performing fund into (should be memenangi …..)

3.2 Translation of grammatical elements

A further analysis of the target text revealed many factors contributing to translation errors identified. These are errors in the use of lexical items, terminology, concepts words, and grammatical elements such as word order, tense, affixes, etc.

3.2.1 Word order

Frequent grammatical errors resulted from the failure of the computer to sequence words in the target text in proper Malay word order. Examples are as the following:

a. for best total realised return fund manager  untuk terbaik jumlah menyedari fund manager pulangan (Edited: untuk jumlah pulangan pengurusan dana terhasil yang terbaik… )
b. National Annual Corporate Report Awards  nasional anugerah laporan korporat yang tahun (Edited: Anugerah Kebangsaan Tahunan Laporan Korporat)
c. Total shareholder’s funds of the banking system  Jumlah pemegang saham dana daripada itu sistem (Edited: Jumlah dana pemegang saham untuk sistem perbankan)

A more adequate grammtical rules should be written for the computer soft-ware to enable it to recognize and assemble a Malay phrases correctly.

3.2.2 Active and Passive Mode

The failure of the computer to translate passive constructions in English into corresponding passive constructions in Malay resulted in reversing the meaning of such sentences or phrases. The result is, to say the least, wrong. Examples:

a. shares transferred into the depositor’s account  saham memindahkan ke dalam akaun penyimpan (BT: the shares tranfer (…..) into the depositor’s account) (Edited: saham-saham yang dipindahkan ke dalam akaun penyimpan)
b. we will continue to build upon...  kita akan terus dibina berdasarkan ... (BT: we are continuously being built …) (Edited: kami akan terus membina atas ..)
c. we are forging solid alliances  kami diikat pakatan kukuh (BT: we are being tied securely.. ) (Edited: kami membina pakatan kukuh)
d. … we will continue to build upon...  kita akan terus dibina berdasarkan (BT: we are being built … Edited: ... kita akan terus membina atas…)

3.2.3 Definite and Indefinite Article

Definite and indefinite articles are very perplexing in Malay. It is often mistaken and be translated as a demonstative pronoun itu, which means that. Indefinite article a likewise, is often mistaken and be translated as satu which means one, a grammatical category indicating number. In actual fact, this grammatical category need not be translated. Examples:

a. the severity of the effect  itu keterukan daripada itu kesan – BT: that is the severity of the effect … (Edited: keterukan kesan)
b. the Taiwan dollar  itu Taiwan dolar BT: That is the Taiwan Dollar - (Edited: Taiwan dolar)
c. the German mark  itu Jerman mark BT: That is a German mark - (Edited: mark Jerman)
d. a brief period  selepas se tempoh BT: after one period (edited: selepas tempoh)
e. in a move  dalam satu langkah BT: in one step (edited: dalam langkah)

3.2.4 Tense and Aspects

As Malay does not express tense overtly, the computer introduced a redundant adverb of completion in its place. Examples:

a. The United States dollar had strengthened agains the major currencies …  Dolar Amerika Syarikat telah bertambah kukuh berbanding dengan mata wang utama … (telah is an adverb of completion. It is not necessary in Malay. Context would have indicated that the action had been completed.)
b. .. the Board is recommending  Lembaga sedang mencadangkan (The adverb sedang is redundant.)

3.2.5 Polysemy

One difficult task is to assign a particular meaning to an English word that has more than one meanings, or that of an idiomatic expressions, such as:

a. …. The tracks laid were suitable for light trains …  Landasan yang dibina sesuai untuk keretapi berlampu ... (As light has two meanings: (1) not heavy, and (2) light or lamp, resulted in meaning (2) being chosen, giving a back translation as The track was suitable for lighted trains (with lamps). Other examples are:

b. … to create new pillars of growth …  untuk mewujudkan tiang baru bagi pertumbuhan (should be translated as … untuk mewujudkan tunggak baru bagi pertumbuhan.
c. … to transact ordinary buisness …  (untuk berurus niaga perniagaan biasa … should be translated as … bagi menguruskan perniagaan biasa …
d. … experiencing sound growth …  … mengalami bunyi pertumbuhan … BT: experiencing the sound of growing … should be: mengalami pertumbuhan mantap …

3.3. Correct Translation

The following are examples of correct translation produced by the computer. Such translation outputs require no further editing.

a. … in an increasingly competitive global environment  dalam persekitaran global yang semakin kompetitif …
b. … as a premier regional group  sebagai kumpulan serantau (yang) unggul
c. … any other ordinary business  sebarang perniagaan biasa

4. CONCLUSION

The team found that the translation text evaluated has only achieved 30% accuracy.

The source text was full of proper nouns, concepts and processes that have no equivalences in Malay, at least not in the system used. As a result the computer was not able to identify the equivalent corresponding units or words. Many such translations were literal, and there was a general failure to assign meaning according to contexts.

The machine was also unable to ‘arrange’ words in Malay according to Malay syntactical structure.

Of particular importance was the general inability of the computer to identify such grammatical functions as tense, voice, prepositions, conjunctions, and articles; and hence inable to provide accurate translations.

The team is of the opinion that the system should be equipped with a full list of various financial acts, names of financial companies and agencies, names of countries and major cities, states, departments, etc.

The system should also assemble a dictionary of phrasses and idioms. Its entries should include phrases like: pillars of growth; milestone; in respect of; in accordance with; came into force DSB, etc.

A Malay thesaurus should also be incorporated into the system to assisst translators to choose words according their contexts.

The Malay grammatical rules used must be enhanced.

Though there are major advances made in computer-aided translation to date, computers are not likely to replace human translators altogether. The computer should be regarded as a machine to provide translation tools. There are many types of computer-aided translation tools developed to support translators. Online dictionaries, thesaurus, encylopedias, etc could provide valuable assistance to translators. These tools can be put together into a translation workbench, which can be made availabe online. This particular exercise will undoubtedly help improve the currently available tools. Such activities should be best left to the soft-ware engineers.

5. REFERENCES

[1]. Lim Lian Tze, et al. 2008. ’Digitising Dictionaries for Advanced Look-up and Lexical Knowledge Research in Malay’ in Proceedings of the Eleventh International Conference on Translation. Kuala Lumpur: The Malaysian Translators Assoiation.
[2]. Lim Huan Ngee, et al. 2008. ’Adapting an Existing Example-Based Machine Translation (EBMT) System for New Language Pairs based on an Optimized Bilingual Knowledge Bank (BKB)’ in Proceedings of the Eleventh International Conference on Translation. Kuala Lumpur: The Malaysian Translators Assoiation.
[3] Zaharin Yusoff, 1996, ‘A Perception of Computerized Linguistics,’ a Professorial Inaugural Lecture, Pulau Pinang: Science University of Malaysia.

No comments: