Scopus

Fine-Tuning Tiny Language Models for Legal Question Answering: A Comparative Study of Gemma 2, Qwen 2.5 and Llama 3.2

Năm XB 2026 Tạp chí / Hội thảo International Conference on Advances in Information and Communication Technology (ICTA2025) Vol 2, 336–345 Đơn vị CNTT DOI / Link https://doi.org/10.1007/978-3-032-18159-6_36 ↗

Tác giả

Huu-Khanh Nguyen ; Huu-Cong Nguyen ; Van-Viet Nguyen ; Thi Minh-Hue Luong ; Kim-Son Nguyen ; Dinh-Cuong Do ; The-Vinh Nguyen ^✉

Tóm tắt

The domain of legal question answering presents significant challenges for natural language processing due to the complexity and nuance of legal texts. While large language models have shown promise, the efficacy of the latest generation of “tiny” language models remains underexplored. This paper presents a comparative study of three prominent open-weight TLMs - Google’s Gemma 2 2B, Alibaba’s Qwen 2.5 1.5B, and Meta’s Llama 3.2 1B - fine-tuned for legal QA on the CaseHOLD dataset. Employing parameter-efficient fine-tuning (LoRA, QLoRA) on a limited subset of 10,000 training examples to simulate resource-constrained conditions, we evaluate their performance. Our results demonstrate that fine-tuning yields substantial improvements over base models, with Gemma 2 2B achieving a top Micro F1 score of 74.47, followed by Qwen 2.5 at 72.17 and Llama 3.2 at 71.17. Notably, Gemma 2’s performance …

Tài liệu tham khảo

[1] Martinez-Gil, J.: A survey on legal question answering systems. Comput. Sci. Rev. 48, 100552 (2023). https://doi.org/10.1016/j.cosrev.2023.100552

[2] Bourcier, D., Clergue, G.: From a rule-based conception to dynamic patterns. analyzing the self-organization of legal systems. Artif. Intell. Law 7(2–3), 211–225 (1999). https://doi.org/10.1023/A:1008388719330

[3] Pal, K., Campbell, J.A.: An application of rule-based and case-based reasoning within a single legal knowledge-based system. SIGMIS Database 28(4), 48–63 (1997). https://doi.org/10.1145/277339.277344

[4] Timmer, I., Rietveld, R.: Rule-Based Systems for Decision Support and Decision-Making in Dutch Legal Practice. A Brief Overview of Applications and Implications: Droit et société, vol. 103, no. 3, pp. 517–534 (2019). https://doi.org/10.3917/drs1.103.0517

[5] Vaswani, A., et al.: Attention Is All You Need (2023). arXiv: arXiv:1706.03762. https://doi.org/10.48550/arXiv.1706.03762

[6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North, Minneapolis, Minnesota: Association for Computational Linguistics, pp. 4171–4186 (2019). https://doi.org/10.18653/v1/N19-1423

[7] Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopoulos, I.: LEGAL-BERT: the muppets straight out of law school. In: Findings of the Association for Computational Linguistics: EMNLP 2020, Online: Association for Computational Linguistics, pp. 2898–2904 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.261

[8] Xiao, C., Hu, X., Liu, Z., Tu, C., Sun, M.: Lawformer: a pre-trained language model for Chinese legal long documents. AI Open 2, 79–84 (2021). https://doi.org/10.1016/j.aiopen.2021.06.003

[9] Kent State University et al., “GiusBERTo: Italy’s AI-Based Judicial Transformation: A Teaching Case,” CAIS, vol. 53, no. 1, pp. 751–766 (2023). https://doi.org/10.17705/1CAIS.05331

[10] Licari, D., Comandè, G.: ITALIAN-LEGAL-BERT models for improving natural language processing tasks in the Italian legal domain. Comput. Law Secur. Rev. 52, 105908 (2024). https://doi.org/10.1016/j.clsr.2023.105908

[11] Bogdanović, M., Kocić, J., Stoimenov, L.: SRBerta—a transformer language model for serbian cyrillic legal texts. Information 15(2), 74 (2024). https://doi.org/10.3390/info15020074

[12] Zheng, L., Guha, N., Anderson, B.R., Henderson, P., Ho, D.E.: When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, São Paulo Brazil: ACM, pp. 159–168 (2021). https://doi.org/10.1145/3462757.3466088

[13] Chalkidis, I., et al.: LexGLUE: a benchmark dataset for legal language understanding in English. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4310–4330. Dublin, Ireland: Association for Computational Linguistics (2022). https://doi.org/10.18653/v1/2022.acl-long.297

[14] Abdin, M., et al.: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024). arXiv. https://doi.org/10.48550/ARXIV.2404.14219

[15] Huu-Khanh, N., Viet, N.V., Nguyen, T.V., Nguyen, H.C.: Phi-3 meets law: fine-tuning mini language models for legal document understanding. Res. Dev. Appl. Inf. Commun. Technol. 3, 2024 (2024). https://doi.org/10.32913/mic-ict-research-vn.v2024.n3.1281

[16] Team, G., et al.: Gemma 2: Improving Open Language Models at a Practical Size. (2024). arXiv: arXiv:2408.00118. https://doi.org/10.48550/arXiv.2408.00118

[17] Qwen et al.: Qwen2.5 Technical Report (2025). arXiv: arXiv:2412.15115. https://doi.org/10.48550/arXiv.2412.15115

[18] Grattafiori, A., et al.: The Llama 3 Herd of Models (2024). arXiv: arXiv:2407.21783. https://doi.org/10.48550/arXiv.2407.21783

[19] Hu, E.J., et al.: LoRA: Low-Rank Adaptation of Large Language Models (2021). arXiv. https://doi.org/10.48550/ARXIV.2106.09685

[20] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLoRA: efficient finetuning of quantized LLMs. In: Proceedings of the 37th International Conference on Neural Information Processing Systems, Curran Associates Inc. (2023). https://doi.org/10.5555/3666122.3666563

← Quay lại danh sách bài báo