Skip to main navigation menu Skip to main content Skip to site footer


Número 0

New voices

Methodological considerations for emotion research in MTPE: A multimethod design using eye-tracking and self-report questionnaires

Published 2026-10-20

  • María Inmaculada Vicente López

María Inmaculada Vicente López
University of Murcia

Abstract

This paper explores the methodological challenges of researching cognitive and emotional dimensions in machine translation post-editing (MTPE) through a multimethod design. The study combined psychometric, physiological, and behavioral data to examine differences in translators’ anxiety, emotional arousal, cognitive effort, and visual attention during an English-to-Spanish MTPE task involving statistical machine translation (SMT) and neural machine translation (NMT). Emotional arousal was measured via pupil dilation, anxiety through the State-Trait Anxiety Inventory, and cognitive effort via fixation durations. Visual attention was analyzed through fixation counts on the source text, machine translation (MT) output, and Internet browser. The influence of MT technology, anxiety, and attitude on visual attention distribution was explored using data from both professional and student translators working in ecologically valid conditions. This study reflects critically on the methodological challenges encountered, including issues related to construct operationalization, participant variability, and ecological validity. At the same time, it demonstrates the value of a multimethod research design—combining eye-tracking and self-report measures—for investigating the interplay between cognition and emotion in MTPE. Finally, it argues that the rise of new MT technologies, including generative AI, poses an emerging methodological challenge that future Cognitive Translation and Interpreting Studies (CTIS) should address.

Keywords

Eye-tracking, machine translation post-editing, cognitive effort, emotional arousal, anxiety, attitude toward MT, visual attention

PDF

References

  1. Aziz, Wilker, Sheila Castilho, and Lucia Specia. 2012. “{PET}: A Tool for Post-Editing and Assessing Machine Translation.” In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), 3982–87. Istanbul: European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2012/pdf/985_Paper.pdf.
  2. Baayen, Rolf Harald. 2008. Analyzing Linguistic Data: A Practical Introduction to Statistics Using R. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511801686.
  3. Balling, Laura Winther. 2008. “A Brief Introduction to Regression Designs and Mixed-Effects Modelling by a Recent Convert.” Copenhagen Studies in Language 36: 175–92. https://doi.org/10.1075/ts.21013.sil.
  4. Balling, Laura Winther, and Kristian Tangsgaard Hvelplund. 2015. “Design and Statistics in Quantitative Translation (Process) Research.” Translation Spaces 4 (1): 170–187. https://doi.org/10.1075/ts.4.1.08bal.
  5. Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using Lme4.” Journal of Statistical Software 67 (1): 1–48. https://doi.org/10.18637/jss.v067.i01.
  6. Bolaños Medina, Alicia. 2014. “La investigación de las diferencias individuales en traductología cognitiva” TRANS. Revista de Traductología 18: 199–214. https://doi.org/10.24310/TRANS.2014.v0i18.3253.
  7. Bundgaard, Kristine. 2017. “Translator Attitudes towards Translator–Computer Interaction: Findings from a Workplace Study.” Hermes 56: 125–44. https://doi.org/10.7146/hjlcb.v0i56.97228.
  8. Carl, Michael, and Matthias Buch-Kromann. 2010. “Correlating Translation Product and Translation Process Data of Professional and Student Translators.” In EAMT 2010—14th Annual Conference of the European Association for Machine Translation. Saint Raphaël: European Association for Machine Translation.
  9. Chiang, Yung Nan. 2010. “Foreign Language Anxiety and Student Interpreters’ Learning Outcomes: Implications for the Theory and Measurement of Interpretation Learning Anxiety.” Meta 55 (3): 589–601. https://doi.org/10.7202/045079AR.
  10. Cifuentes-Férez, Paula, Ana María Rojo López, and Laura Espín López. 2024. “Self-Efficacy as a Protective Factor When Translating under Time Pressure.” Vigo International Journal of Applied Linguistics, no. 21. https://doi.org/10.35869/vial.v0i21.4215.
  11. Cooper, Alan. 2004. The Inmates Are Running the Asylum: Why Hi-Tech Products Drive Us Crazy and How to Restore the Sanity. Indianapolis: Sams Publishing.
  12. Daems, Joke. 2016. A Translation Robot for Each Translator? A Comparative Study of Manual Translation and Post-Editing of Machine Translations: Process, Quality and Translator Attitude. PhD Thesis, Ghent University.
  13. De Almeida, Giselle. 2013. Translating the Post-Editor: An Investigation of Post-Editing Changes and Correlations with Professional Experience across Two Romance Languages. PhD Thesis, Dublin City University.
  14. Dillon, Sarah, and Janet Fraser. 2006. “Translators and TM: An Investigation of Translators’ Perceptions of Translation Memory Adoption.” Machine Translation 20 (2): 67–79. https://doi.org/10.1007/s10590-006-9004-8.
  15. Gieshoff, Anne Catherine, Caroline Lehr, and Andrea Hunziker Heeb. 2021. “Stress, Cognitive, Emotional and Ergonomic Demands in Interpreting and Translation.” Cognitive Linguistic Studies 8 (2): 404–39. https://doi.org/10.1075/cogls.00084.gie.
  16. Guerberof Arenas, Ana. 2013. “What Do Professional Translators Think about Post-Editing.” JoSTrans: The Journal of Specialised Translation 19: 75–95.
  17. Holmqvist, Kenneth, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost van de Weijer. 2011. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press.
  18. Hvelplund, Kristian Tangsgaard. 2017. “Translators’ Use of Digital Resources during Translation.” Hermes 56: 71–87. https://doi.org/10.7146/hjlcb.v0i56.97205.
  19. —. 2016. “Cognitive Efficiency in Translation.” In Reembedding Translation Process Research, edited by Ricardo Muñoz Martín, 149–70. Amsterdam and Philadelphia: John Benjamins Publishing Company. https://doi.org/10.1075/btl.128.08hve.
  20. —. 2014. “Eye Tracking and the Translation Process: Reflections on the Analysis and Interpretation of Eye-Tracking Data.” In MonTI. Monografías de Traducción e Interpretación, edited by Ricardo Muñoz Martín, 201–23. Alicante: Publicaciones de la Universidad de Alicante. https://doi.org/10.6035/monti.2014.ne1.6.
  21. —. 2011. Allocation of Cognitive Resources in Translation: An Eye-Tracking and Key-Logging Study. PhD Thesis, Copenhagen Business School.
  22. Jääskeläinen, Riitta. 2010. “Are All Professionals Experts?” In Translation and Cognition, edited by Gregory M. Shreve and Erik Angelone, 213–27. Amsterdam and Philadelphia: John Benjamins Publishing Company. https://doi.org/10.1075/ata.xv.12jaa.
  23. Jankowiak, Katarzyna, and Paweł Korpal. 2018. “On Modality Effects in Bilingual Emotional Language Processing: Evidence from Galvanic Skin Response.” Journal of Psycholinguistic Research 47 (3): 663–77. https://doi.org/10.1007/s10936-017-9552-5.
  24. Kendall, Maurice G. 1938. “A New Measure of Rank Correlation.” Biometrika 30 (1–2): 81–93.
  25. Koglin, Arlene. 2015. “An Empirical Investigation of Cognitive Effort Required to Post-Edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation & Interpreting 7 (1): 126–141.
  26. Koponen, Maarit, Wilker Aziz, Luciana Ramos, and Lucia Specia. 2012. “Post-Editing Time as a Measure of Cognitive Effort.” In Proceedings of the AMTA 2012 Workshop on Post-Editing Technology and Practice, 11–20. http://amta2012.amtaweb.org/AMTA2012Files/html/13/13_paper.pdf
  27. Korpal, Paweł. 2017. Linguistic and Psychological Indicators of Stress in Simultaneous Interpreting. Poznań: Adam Mickiewicz University Press.
  28. Kuznetsova, Alexandra, Per B. Brockhoff, and Rune H. B. Christensen. 2017. “LmerTest Package: Tests in Linear Mixed Effects Models.” Journal of Statistical Software 82 (13): 1–26. https://doi.org/10.18637/jss.v082.i13.
  29. Lavie, Alon, and Michael J. Denkowski. 2009. “The Meteor Metric for Automatic Evaluation of Machine Translation.” Machine Translation 23 (2): 105–15. https://doi.org/10.1007/s10590-009-9059-4.
  30. Lehr, Caroline, and Kristian Tangsgaard Hvelplund. 2020. “Emotional Experts: Influences of Emotion on the Allocation of Cognitive Resources during Translation.” In Multilingual Mediated Communication and Cognition, edited by Ricardo Muñoz Martín and Sandra L. Halverson, 44–68. Abingdon: Routledge.
  31. Mann, H. B., and D. R. Whitney. 1947. “On a Test of Whether One of Two Random Variables Is Stochastically Larger than the Other.” The Annals of Mathematical Statistics 18 (1): 50–60. https://doi.org/10.1214/aoms/1177730491.
  32. Massardo, Isabella, Jaap van der Meer, Sharon O’Brien, Fred Hollowood, Nora Aranberri, and Katrin Drescher. 2016. MT Postediting Guidelines. TAUS Signature Editions.
  33. Mellinger, Christopher D., and Thomas A. Hanson. 2020. “Methodological Considerations for Survey Research: Validity, Reliability, and Quantitative Analysis.” Linguistica Antverpiensia, New Series – Themes in Translation Studies 19: 172–90. https://doi.org/10.52034/lanstts.v19i0.549.
  34. Mellinger, Christopher, and Thomas Hanson. 2017. Quantitative Research Methods in Translation and Interpreting Studies. London and New York: Routledge. https://doi.org/10.4324/9781315647845.
  35. Moorkens, Joss. 2018. “Eye Tracking as a Measure of Cognitive Effort for Post-Editing of Machine Translation.” In Eye Tracking and Multidisciplinary Studies on Translation, edited by Callum Walker and Federico M. Federici, 55–70. Amsterdam and Philadelphia: John Benjamins Publishing Company. https://doi.org/10.1075/btl.143.
  36. Moorkens, Joss, Sharon O’Brien, Igor da Silva, Norma Fonseca, and Fabio Alves. 2015. “Correlations of Perceived Post-Editing Effort with Measurements of Actual Effort.” Machine Translation 29 (3): 267–84. https://doi.org/10.1007/s10590-015-9175-2.
  37. Moser-Mercer, Barbara. 2005. “Remote Interpreting: Issues of Multi-Sensory Integration in a Multilingual Task.” Meta 50 (2): 727–38. https://doi.org/10.7202/011014ar.
  38. Muñoz Martín, Ricardo, and Ana María Rojo López. 2025. “Introduction.” In Research Methods in Cognitive Translation and Interpreting Studies, 1–20. Amsterdam: John Benjamins Publishing Company. https://doi.org/10.1075/rmal.10.intro.
  39. Nimdzi. 2025. The 2025 Nimdzi 100: The Size and State of the Language Services Industry. MultiLingual Media LLC.
  40. Nitzke, Jean. 2019. Problem Solving Activities in Post-Editing and Translation from Scratch: A Multi-Method Study. New Empirical Perspectives on Translation and Interpreting. Berlin: Language Science Press. https://doi.org/10.5281/zenodo.2546446.
  41. Nunes Vieira, Lucas. 2018. “Automation Anxiety and Translators.” Translation Studies (13) 1: 1–21. https://doi.org/10.1080/14781700.2018.1543613.
  42. —. 2015. Cognitive Effort in Post-Editing of Machine Translation: Evidence from Eye Movements, Subjective Ratings, and Think-Aloud Protocols. PhD Thesis, Newcastle University.
  43. O’Brien, Sharon. 2009. “Eye Tracking in Translation-Process Research: Methodological Challenges and Solutions.” In Methodology, Technology and Innovation in Translation Process Research: A Tribute to Arnt Lykke Jakobsen, edited by Inger M. Mees, Susanne Göpferich, and Fabio Alves, 15–36. Frederiksberg: Samfundslitteratur. https://doi.org/10.1075/target.24.1.13tir.
  44. —. 2007. “Eye-tracking and Translation Memory Matches.” Perspectives: Studies in Translatology 14 (3): 185–205. https://doi.org/10.1080/09076760708669037.
  45. O’Brien, Sharon, and Maureen Ehrensberger-Dow. 2020. “MT Literacy-A Cognitive View.” Translation, Cognition and Behavior 3 (2): 145–64. https://doi.org/10.1075/tcb.00038.obr.
  46. Pinazo Calatayud, Daniel, and Amparo Jiménez Ivars. 2002. “Fear of public speaking, state anxiety and performance: The case of interpreter trainees.” Ansiedad y estrés 8 (1): 37–47. https://www.ansiedadyestres.es/sites/default/files/rev/ucm/2002/anyes2002a4.pdf.
  47. Pym, Anthony. 2013. “Translation Skill-Sets in a Machine-Translation Age.” Meta 58 (3): 487–503. https://doi.org/10.7202/1025047ar.
  48. R Core Team. 2023. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. https://www.R-project.org/.
  49. Ragni, Valentina, and Lucas Nunes Vieira. 2022. “What Has Changed with Neural Machine Translation? A Critical Review of Human Factors.” Perspectives 30 (1): 137–58. https://doi.org/10.1080/0907676X.2021.1889005.
  50. Rojo López, Ana María, and Paula Cifuentes Férez. 2021. “Introduction: Experimenting with Emotions: Insights into Empirical Emotion Research in Cognitive Translation Studies.” Onomázein Revista de Lingüística Filología y Traducción, no. 8, 1–15. https://doi.org/10.7764/onomazein.ne8.01.
  51. Rojo López, Ana María, Paula Cifuentes Férez, and Laura Espín López. 2021a. “The Influence of Time Pressure on Translation Trainees’ Performance: Testing the Relationship between Self-Esteem, Salivary Cortisol and Subjective Stress Response.” PLoS ONE 16 (9): e0257727. https://doi.org/10.1371/journal.pone.0257727.
  52. Rojo López, Ana María, Ana Isabel Foulquié-Rubio, Laura Espín López, and Francisco Martínez Sánchez. 2021b. “Analysis of Speech Rhythm and Heart Rate as Indicators of Stress on Student Interpreters.” Perspectives 29 (4): 591–607. https://doi.org/10.1080/0907676X.2021.1900305.
  53. Rojo López, Ana María, and Ricardo Muñoz Martín. 2022. “Translation Process Research.” In The Routledge Handbook of Translation and Methodology, 1st ed., 356–72. London: Routledge. https://doi.org/10.4324/9781315158945-26.
  54. Rojo López, Ana María, and Marina Ramos Caro. 2022. “Introduction: Probing the Process: Towards More Integrative Research Methods.” The International Journal of Translation and Interpreting Research 14 (2): 1–7. https://doi.org/10.12807/ti.114202.2022.a01.
  55. —. 2018. “The Role of Expertise in Emotion Regulation.” In Innovation and Expansion in Translation Process Research, edited by Isabel Lacruz and Rita Jääskeläinen, 105–29. Amsterdam and Philadelphia: John Benjamins Publishing Company. https://doi.org/10.1075/ata.18.06roj.
  56. Rösener, Christoph. 2016. “Eye Tracking and Beyond: The Dos and Don’ts of Creating a Contemporary Usability Lab.” In Eyetracking and Applied Linguistics, edited by Silvia Hansen-Schirra and Sambor Grucza, 143–63. Berlin: Language Science Press. https://doi.org/10.17169/langsci.b108.297.
  57. Rossi, Caroline, and Jean Pierre Chevrot. 2019. “Uses and Perceptions of Machine Translation at the European Commission.” The Journal of Specialised Translation, no. 31: 177–200. https://shs.hal.science/halshs-01893120v2.
  58. Saldanha, Gabriela, and Sharon O’Brien. 2013. Research Methodologies in Translation Studies. London: Routledge.
  59. Seisdedos Cubero, Nicolás. 1997. STAI: Cuestionario de Ansiedad Estado-Rasgo: Manual. Adaptación española del cuestionario y redacción del manual realizadas por Nicolás Seisdedos. Madrid: TEA Ediciones.
  60. Sjørup, Annette Camille. 2013. Cognitive Effort in Metaphor Translation: An Eye-Tracking and Key-Logging Study. PhD Thesis, Copenhagen Business School.
  61. Spielberger, C. D., R. L. Gorsuch, R. Lushene, P. R. Vagg, and G. A. Jacobs. 1983. Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press. https://psycnet.apa.org/doiLanding?doi=10.1037%2Ft06496-000.
  62. Stasimioti, Maria, and Vilelmini Sosoni. 2021. “Investigating Post-Editing: A Mixed-Methods Study with Experienced and Novice Translators in the English-Greek Language Pair.” In Translation, Interpreting, Cognition: The Way out of the Box, edited by Tra&Co Group, 79–104. Berlin: Language Science Press. https://doi.org/10.5281/zenodo.4545037.
  63. Stasimioti, Maria, Vilelmini Sosoni, Despoina Mouratidis, and Katia Kermanidis. 2020. “Machine Translation Quality: A Comparative Evaluation of SMT, NMT and Tailored-NMT Outputs.” In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT 2020), Lisboa, Portugal, 441–50.
  64. Tatsumi, Midori, and Johann Roturier. 2010. “Source Text Characteristics and Technical and Temporal Post-Editing Effort: What Is Their Relationship?” In Proceedings of the Second Joint EM+/CNGL Workshop: Bringing MT to the User – Research on Integrating MT in the Translation Industry (JEC 2010), 43–51. http://www.mt-archive.info/JEC-2010-Tatsumi.pdf.
  65. Teixeira, Carlos S. C. 2014. The Impact of Metadata on Translator Performance: How Translators Work with Translation Memories and Machine Translation. PhD Thesis, Universitat Rovira i Virgili. https://doi.org/10.13140/RG.2.1.2190.2887.
  66. Tobii Technology. 2010. Product Description for Tobii T & X Series Eye Trackers. Stockholm: Tobii Technology. https://www.tobiipro.com/siteassets/tobii-pro/product-descriptions/tobii-pro-tx-product-description.pdf.
  67. Vanroy, Bram, Arda Tezcan, and Lieve Macken. 2023. “MATEO: MAchine Translation Evaluation Online.” In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 499-500, https://aclanthology.org/2023.eamt-1.52.
  68. Vicente López, María Inmaculada. 2024. Aspectos cognitivos y emocionales de la posedición de traducción automática: Un estudio de movimientos oculares. PhD Thesis, Universidad de Murcia.
  69. Weng, Yu, and Binghan Zheng. 2020. “A Multi-Methodological Approach to Studying Time-Pressure in Written Translation: Manipulation and Measurement.” Linguistica Antverpiensia, New Series – Themes in Translation Studies 19:218–36. https://doi.org/10.52034/lanstts.v19i0.548.
  70. Weng, Yu, Binghan Zheng, and Yanping Dong. 2022. “Time Pressure in Translation.” Target: International Journal of Translation Studies 34 (4): 601–26. https://doi.org/10.1075/target.20148.wen.