Evaluation of Transfer Learning Techniques for Low-Resource Natural Language Processing
Keywords:
Transfer Learning, Low-Resource NLP, Fine-Tuning, Language Models, Cross-lingual Adaptation, Domain AdaptationAbstract
Natural Language Processing (NLP) has seen significant advancements due to the application of deep learning techniques and large-scale pre-trained language models. However, these models often perform suboptimally when applied to low-resource languages or domains with limited labeled data. This paper explores various transfer learning approaches aimed at addressing the low-resource NLP problem, evaluating their effectiveness, architecture designs, and performance trade-offs. Two diagrams illustrate the transfer learning process and a comparative model architecture. Two tables summarize benchmark results and dataset availability. Our analysis reveals that with carefully adapted transfer techniques, significant performance can still be achieved even under low-resource constraints
References
1. Artetxe, M., Labaka, G., & Agirre, E. (2018). Unsupervised statistical machine translation. arXiv preprint arXiv:1809.01272.
2. Chen, X., et al. (2018). Adversarial multilingual training for low-resource speech recognition. Interspeech 2018.
3. Conneau, A., et al. (2018). LASER: Language-Agnostic SEntence Representations. arXiv preprint arXiv:1812.10464.
4. Devalla, S. (2024). From principles to practice: Continuous verification of the CIA triad in distributed microservice system. Journal of Recent Trends in Computer Science and Engineering (JRTCSE), 12(2), 82–97. Retrieved from https://jrtcse.com/index.php/home/article/view/JRTCSE.2024.2.8
5. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL.
6. Devalla, S. (2024). Operationalizing Helm chart security: A topology-aware framework for enterprise Kubernetes environments. International Journal of Science and Research (IJSR), 13(6), 1967–1974. https://dx.doi.org/10.21275/SR24628103829
7. Gururangan, S., et al. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020.
8. Howard, J., & Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. ACL 2018.
9. Devalla, S. (2024). From promise to production: Virtual threads in Java 21 and their impact on enterprise-scale microservice. International Journal of Science and Research (IJSR), 13(1), 1865–1871. https://dx.doi.org/10.21275/SR24128103553
10. Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. NeurIPS 2019.
11. Pires, T., Schlinger, E., & Garrette, D. (2019). How multilingual is Multilingual BERT? ACL 2019.
12. Devalla, S. (2023). Cross-platform resilience: A security-aware framework for migrating enterprise workloads from PCF to OpenShift. International Journal of Information Technology and Management Information Systems (IJITMIS), 14(2), 129–152. https://doi.org/10.34218/IJITMIS_14_02_013
13. Schuster, S., et al. (2019). Cross-lingual transfer learning for multilingual task-oriented dialog. NAACL 2019.
14. Devalla, S. (2023). Adaptive predictive monitoring in enterprise serverless deployments: Enabling early fault detection across multi-cloud environments. International Journal of Computer Applications (IJCA), 4(1), 54–72. https://doi.org/10.34218/IJCA_04_01_007
15. Sennrich, R., Haddow, B., & Birch, A. (2016). Improving neural machine translation models with monolingual data. ACL 2016.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Guzel Yakhina Sorokin (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
