Influence Analysis of Smoothing Algorithms in Language Modelling for Indonesian Statistical Machine Translation
DOI:
https://doi.org/10.26418/ijeasr.v1i1.41088Abstract
The diversity of languages makes the need for translation so that communication between individuals of different languages can be appropriately established. The statistical translator engine (SMT) was a translator engine based on a statistical approach to parallel corpus analysis. One crucial part of SMT was language modeling (LM). LM was the calculation of word probability from a corpus-based on n-grams. There was a smoothing algorithm in LM where this algorithm will bring up the probability of a word whose value was zero. This study compares the use of the best smoothing algorithm from each of the three LM according to the standard Moses, namely KenLM, SRILM, and IRSTLM. For SRILM using smoothing algorithm interpolation with Witten-bell and interpolation with Ristad’s natural discounting, for KenLM using interpolation with modified Kneser-ney smoothing algorithm, and for IRSTLM using modified Kneser-ney and Witten-bell algorithm which was referenced based on previously researched. This study uses a corpus of 10,000 sentences. Tests carried out by BLEU and testing by Melayu Sambas linguists. Based on the results of BLEU testing and linguist testing, the best smoothing algorithm was chosen, namely modified Kneser-ney in KenLM LM, where the average results of automated testing, for Indonesian-Melayu Sambas and vice versa were 41. 6925% and 46. 66%. Moreover, for testing linguists, the accuracy of the Indonesian-Melayu Sambas language and vice versa was 77. 3165% and 77. 9095%References
Aan Setyawan. "Bahasa Daerah dalam Perspektif Kebudayaan dan Sosilinguistik: Peran dan Pengaruhnya dalam Pergeseran dan Pemertahanan Bahasa." Semarang, International Seminar Language Maintenance and Shift, 2011
Christopher Manning dan Hinrich Schutze. "Foundations Of Statistical Natural Language Processing." London : The MIT Press Cambridge Massachusetts, 2000
Phillip Koehn. "Statistical Machine Translation." Cambridge University Press. p. 27; 2009
Hansel Tanuwijaya. "Penerjemahan Inggris-Indonesia Menggunakan Mesin Penerjemah Statistik Dengan Word Reordering dan Phrase Reordering." Jakarta, Jurnal ilmu Komputer dan Informasi Vol 2 No 1, 2009
Herry Sujaini, Kuspriyanto, Arry Akhmad A., Ayu Purwarianti. "Pengaruh Part-Of-Speech pada Mesin Penerjemah bahasa Inggris – Indonesia Berbasis Factored Translation Model." Yogyakarta, Seminar Nasional Aplikasi Teknologi Informasi 2012
Ibnu Hadi. "Uji Akurasi Mesin Penerjemah Statistik (MPS) Bahasa Indonesia ke Bahasa Melayu Sambas dan Mesin Penerjemah Statistik (MPS) Bahasa Melayu Sambas ke Bahasa Indonesia." Pontianak, Jurnal Sistem dan Teknologi Informasi (JUSTIN) Vol. 2 No. 3, 2014
Chen Stanley and Joshua Goodman. "An Empirical Study of Smoothing Techniques for Language Modeling." Harvard Computer Science Group Technical Report TR-10-98; 1998
Tri Apriyani, Herry Sujaini, Novi Safriadi. "Pengaruh Kuantitas Korpus Terhadap Akurasi Mesin Penerjemah Statistik Bahasa Bugis Wajo ke Bahasa Indonesia." Pontianak, Jurnal Sistem dan Teknologi Informasi (JUSTIN) Vol. 1, No. 1, 2016.
Tony McEnery dan Andrew Wilson. "Corpus Linguistics." Edinburgh:
Edinburgh University Press, 1996
Candra Triawati. "Metode Pembobotan Statistical Concept Based untuk Clustering dan Kategorisasi Dokumen Berbahasa Indonesia". Jakarta: IT Universitas TELKOM, 2009.
Linda Marlinda dan Harsih Rianto. "Pembelajaran Bahasa Indonesia Berbasis Web Menggunakan Metode Maximum Marginal Relevance". Seminar Nasional Sistem Informasi Indonesia 2013.
Phillip Koehn et al. "Moses: Open Source Toolkit for Statistical Machine Translation." Prague, Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 2007
Franz Josef Och and Hermann Ney. "A Systematic Comparison of Various Statistical Alignment Models." Computational Linguistics, 29(1):19-51; 2003
Andreas Stolcke, Jing Zhen, Wen Wang, Victor Abrash. "SRILM at Sixteen: Update and Outlook." Big Island, Proc. IEEE Automatic Speech Recognition and Understanding Workshop, 2011
Marcello Federico, Nicola Bertoldi, Mauro Cettolo. "IRSTLM: an Open Source Toolkit for Handling Large Scale Language Models." Povo, Interspeech 2008 incorporating SST08; 2008
Kenneth Heafield. "KenLM: faster and smaller language model queries." In Proc. WMT, pages 187–197. ACL. 2011
Kishore Papineni, Salim Ruokos, Todd Ward, Wei-Jing Zhu. "BLEU: A Method For Automatic Evaluation of Machine Translation." USA: IBM TJ Watson Research Center, 2002
Robert Grossman, Giovanni Seni, John Elder, Nitin Argawal, Huan Liu. "Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions." Morgan & Claypool Publisher, 2010
Herry Sujaini. " An Approach to Improving Corpus Quality for Indonesian-English Statistical Machine Translation". International Journal of Engineering Research & Technology (IJERT) , 4(2). 2015.
Berlin Chen. "Introduction to SRILM Toolkit". Taiwan, Departement of Computer Science & Information Engineering National Taiwan Normal University, 2008
Nicola Bertoldi. "A Tutorial On the IRSTLM Library". Italy, FBK Irst, 2008
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Phillip Koehn. "Scalable Modified Kneser-Ney Language Model Estimation." In Proc. ACL, pages 690–696. ACL. 2013
Martin Sundermeyer; Raft Schlu ̈lter, Hermann Ney. "On the Estimation of Discount Parameters for Language Model Smoothing." Florence, INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association, 2011
Junfei Guo, Juan Liu, Qi Han, Andreas Maletti. "A Tunable Language Model for Statistical Machine Translation." Vancouver, Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol 1: MT Researchers; 2014
Wenyang Zhang. "Comparing the Effect of Smoothing and N-gram Order: Finding the Best Way to Combine the Smoothing and Order of N-gram." Melbourne, Florida Institute of Technology, 2015
Yosep Jarob R, Herry Sujaini, Novi Safriadi. "Uji Akurasi Penerjemahan Bahasa Indonesia – Dayak Taman dengan Penandaan Kata Dasar dan Imbuhan." Pontianak, Jurnal Edukasi dan Penelitian Informatika (JEPIN) Vol. 2 No. 2, 2016
Marcello Federico. "IRSTLM Toolkit." Trento, FBK-Irst, 2010
Eric Ristad. "A Natural Law of Succesion." Research Report CS-TR-495-95; 1995
