Predicting Machine Translation Comprehension with a Neural Network

Comprehension of natural language translations is dependent upon several factors including textual variables (grammatical, spelling


INTRODUCTION
Machine translation often provides fast and cheap transliteration that is particularly useful for situations in which human interpreters are not available or the foreign source text might not be important enough to pay for professional translation services [8].This automated translation is often performed with Web programs such as Google Translate that are becoming increasingly accurate and provide support for more and more languages, but performance still suffers with many language pairs [7,16].
Even if machine translation accuracy is low with incorrect word choices, poor grammar, and other errors, some meaning can often be found and these translations can still be useful to the reader [11]. In addition, some people are better able to understand poor translations than others, with wide-ranging comprehension rates for equivalent texts [1,12]. However, very little is known about why some readers have better comprehension than others. With more research, we might be able to predict which target audiences are more likely to benefit from machine translation services. For example, those with more motivation to understand might be able to grasp meanings more readily than other readers.
Prior research has shown that many variables might affect reading comprehension, including topic knowledge, critical thinking skills and language fluency (e.g., [10,13,14]). One study [3] using 80 undergraduate students found a significant but weak and negative correlation between results from their performance on a Cloze comprehension test [17] and understanding of some, but not all, machine-translated text (R = -0.225 , p = 0.045). In another study [15], 96 undergraduate college students read poor machine translations and were asked to write in English what they thought was meant, using correct grammar and word choices. Results showed a significant, positive correlation between grammatical ability and comprehension of some of the text (R=.207, p=.043) and a significant, but negative, correlation between their self-assessed foreign language fluency and comprehension of other parts of the text (R= -0.210, p = 0.040).
In this paper, we conduct a new study to assess which variables are most important for human understanding. An artificial neural network and multilinear analysis are applied in an attempt to yield a more predictive model. The study then concludes with a summary, limitations, and directions for further research.

SURVEY Subjects and Task Description
The purpose of this study is to predict how well a reader might comprehend a passage of text that could be translated by machine.We recruited 121 undergraduate business students from a university in the southern United States to assess possible factors that might impact comprehension, including language fluency, motivation, and subject knowledge. Motivation was measured by a 7-item (1=disagree, 7=agree), self-assessed Likert scale with two statements: "I was interested in this task." (INT) and "I was motivated to understand the sentences." (MOT). Subject knowledge was evaluated based upon the statement "I know the subject matter of these sentences." (SUBJ), and language fluency was assessed based upon the statements "I know English very well." (ENG), "I have a large English vocabulary." (VOC), "I rarely make grammatical mistakes." (GRAM), and "I know a non-English language well." (LANG). In addition, the students took the short Cloze test shown below as an additional means of assessing their language fluency (Source: http://www.testyourenglish.net/english-online/cloze-reading/cloze1.html). Correct answers are shown in bold.
Can we see (1) ___ the earth is a globe? Yes, we can, when we watch a ship that sails out to sea. If we watch closely, we see that the ship begins (2) ___ . The bottom of the ship disappears first, and then the ship seems to sink lower and lower, (3) ___ we can see only the top of the ship, and then we see nothing at all. What is hiding the ship from us? It is the earth. Stick a pin most of the way into an orange, and (4) ___ turn the orange away from you. You will see the pin disappear, (5) ___ a ship does on the earth.  Table 1 shows a summary of the variables. The students considered themselves to be fluent in English, as they selfreported knowing their native language well, making few grammatical errors, and having a good vocabulary. In addition, the 79.3% overall score on the Cloze test was well above the 58% threshold for an "independent reading" level [5]. However, few students reported knowing a foreign language well. The students were also motivated and interested. There D e c e m b e r 0 8 , 2 0 1 5 were no significant differences between the two groups of students on these measures.There was a significant difference between the easy-text and difficult-text groups in terms of scores on the reading comprehension tests, however, because the latter contained several errors (F = 14.40, p < 0.001).

Summary Results
A Pearson correlation analysis (Table 2) shows that only their self-assessed grammatical knowledge was significantly correlated with text comprehension. However, the association was very weak. Interest and motivation were almost significantly correlated with text comprehension, but the relationships were weak as well. Thus, the results could not find strong determinants of reading comprehension; this result is consistent with the prior studies discussed earlier.

NEURAL NETWORKS
Artificial neural networks are mathematical models of the biological processes that occur in brains when learning new material and have been proven to be more accurate than competing forecasting tools such as multi-linear regression, autocorrelation, and logistic regression [9]. While the models do not provide predictive significance values, they do not require any statistical assumptions. Therefore, when using a neural network, no tests need to be conducted for the standard assumptions required by other parametric statistical techniques.
Neural networks are trained by exposing the network to individual examples of the data to be used for predictions or classifications. The process is repeated until the neural network recognizes underlying patterns between inputs (independent variables) and outputs (dependent variables).
In this study, only one significant, weak correlation could be found to forecast reading comprehension, but a combination of several factors might add predictive power. First, as a basis for comparison against traditional statistical techniques, we developed a multilinear regression model using SPSS. A random sample of 100 observations was used for development of the model, giving the results shown in Table 3 (F = 3  Next, a neural network was developed using NeuroForecaster, a commercial product from NIBS, Inc. [6]. The network used the genetic algorithm. Training was conducted until the 100-observation in-sample mean absolute percentage error MAPE was reduced down to approximately 11.72%, as shown in Figure 1. The forecast of the 21-item hold-out sample is shown in Figure 2, with a MAPE of 12.2%. Further, the correlation between the actual and forecasted values was relatively strong and significant (R = 0.489, p = 0.025).

CONCLUSION Summary
Predicting reading comprehension is a complex problem with many interrelated, uncertain variables, which has made it difficult for previous studies to find strong predictive models using traditional statistical techniques. In this study, using data arising from self-reported assessments of language fluency, motivation, and topic knowledge, as well as results from a Cloze test, we have demonstrated that an artificial neural network can be used to make significantly better forecasts of who might be able to comprehend machine-translated passages of text.

Limitations
The study has several limitations. First, only one short reading comprehension test and one Cloze test were administered. This limited approach could yield an incomplete measure of the subjects' true understanding of the translated text. Future studies will need to examine these phenomena in a wider variety of topics and languages. Second, there was no variation of the reading difficulty level of the text or the number of errors in the passage, which could be additional sources of predictive power in more complicated experimental situations.Third, perhaps the college students' reading comprehension abilities were too similar to adequately explore the full range of effects arising from the independant variables. Finally, other unknown factors might be involved in the highly complicated mental process of comprehending machine translations.

Future Research
As mentioned above, a more thorough study with more comprehensive Cloze and reading comprehension tests might be necessary. It would also be beneficial to conduct additional studies that incorporate greater variation in subjects' abilities and levels of motivation. D e c e m b e r 0 8 , 2 0 1 5 D e c e m b e r 0 8 , 2 0 1 5