Scopus

Fine-Tuning Mini Language Models for Legal Multiple-Choice Question Answering: A Comparative Study of Phi-3.5, Qwen 2.5 and Llama 3.2

Năm XB 2026 Tạp chí / Hội thảo The 14th Conference on Information Technology and its Applications (CITA 2025) volume 1581, 671–682 DOI / Link https://doi.org/10.1007/978-3-032-00972-2_49 ↗

Tác giả

Vinh T. Nguyen

Tóm tắt

In this study, we explore the mini language models applications in legal domain, specifically Phi-3.5 Mini, Qwen 2.5 3B and Llama 3.2 3B, for legal multiple-choice question answering. We fine-tuned these models on CaseHOLD dataset to adapt them to the structural and semantic nuances of legal language and reasoning. The results show that fine-tuning improves performance of these models significantly with Phi-3. 5 Mini achieved a Micro F1 score of 76.93%, exceeding previous bests for the field of miniaturised models. Also, Qwen 2.5 3B and Llama 3.2 3B scored similarly competitive scores of 74.27% and 75.40%, respectively, reinforcing their viability as resource-efficient options compared to larger models. Mini language models offer competitive performance with specialize models like Legal-BERT, Caselaw-BERT, while operating on a lower computational resources and ability of natural language understanding. The results from this study illuminate the potential of mini language models as a way to increase access to state-of-art legal natural language processing tools and proposes directions for additional future work to continue exploring their versatility across various legal task and datasets.

Tài liệu tham khảo

[1] Abdin M, Aneja J, Awadalla H, Awadallah A, Awan AA, Bach N, Bahree A, Bakhtiari A, Bao J, Behl H et al (2024) Phi-3 technical report: a highly capable language model locally on your phone. https://doi.org/10.48550/arXiv.2404.14219

[2] Bourcier D, Clergue G (1999) From a rule-based conception to dynamic patterns. Analyzing the self-organization of legal systems. Artif Intell Law 7:211–225. https://doi.org/10.1023/A:1008388719330

[3] Chalkidis I, Fergadiotis M, Malakasiotis P, Aletras N, Androutsopoulos I (2020) LEGAL-BERT: the Muppets straight out of law school. In: Cohn T, He Y, Liu Y (eds) Findings of the association for computational linguistics: EMNLP 2020. Association for Computational Linguistics, pp 2898–2904. https://doi.org/10.18653/v1/2020.findings-emnlp.261

[4] Chalkidis I, Jana A, Hartung D, Bommarito M, Androutsopoulos I, Katz D, Aletras N (2022) Lexglue: a benchmark dataset for legal language understanding in English. In: Proceedings of the 60th annual meeting of the association for computational linguistics (vol 1: long papers). Association for Computational Linguistics, , pp 4310–4330. https://doi.org/10.18653/v1/2022.acl-long.297

[5] Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L (2024) Qlora: efficient finetuning of quantized LLMS. In: Proceedings of the 37th international conference on neural information processing systems. NIPS ’23, Curran Associates Inc., Red Hook, NY, USA. https://dl.acm.org/doi/10.5555/3666122.3666563

[6] Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Poceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, vol 1 (long and short papers). Association for Computational Linguistics, pp 4171–4186. https://doi.org/10.18653/V1/N19-1423

[7] Douka S, Abdine H, Vazirgiannis M, El Hamdani R, Restrepo Amariles D (2021) JuriBERT: a masked-language model adaptation for French legal text. In: Aletras N, Androutsopoulos I, Barrett L, Goanta C, Preotiuc-Pietro D (eds) Proceedings of the natural legal language processing workshop 2021. Association for Computational Linguistics, Punta Cana, Dominican Republic, pp 95–101. https://doi.org/10.18653/v1/2021.nllp-1.9

[8] Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A, et al (2024) The llama 3 herd of models. https://arxiv.org/abs/2407.21783

[9] Han X, Zhang Z, Ding N, Gu Y, Liu X, Huo Y, Qiu J, Yao Y, Zhang A, Zhang L et al (2021) Pre-trained models: past, present and future. AI Open 2:225–250. https://doi.org/10.1016/j.aiopen.2021.08.002

[10] Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2021) Lora: low-rank adaptation of large language models (6). https://doi.org/10.48550/arXiv.2106.09685

[11] Kejriwal M (2022) AI in industry today. In: Artificial intelligence for industries of the future: beyond Facebook, Amazon, Microsoft and Google. Springer, pp 47–73. https://doi.org/10.1007/978-3-031-19039-1

[12] Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692. http://arxiv.org/abs/1907.11692

[13] Masala M, Iacob RCA, Uban AS, Cidota M, Velicu H, Rebedea T, Popescu M (2021) jurBERT: a Romanian BERT model for legal judgement prediction. In: Aletras N, Androutsopoulos I, Barrett L, Goanta C, Preotiuc-Pietro D (eds) Proceedings of the natural legal language processing workshop 2021. Association for Computational Linguistics, Punta Cana, Dominican Republic, pp. 86–94.https://doi.org/10.18653/v1/2021.nllp-1.8

[14] Nguyen HK, Van VN, Nguyen VT, Nguyen CH (2024) Phi-3 meets law: fine-tuning mini language models for legal document understanding. Res Dev Appl In Commun Technolo v2024.n3. https://doi.org/10.32913/mic-ict-research-vn.v2024.n3.1281, https://ictmag.ictvietnam.vn/cntt-tt/article/view/1281/597

[15] Pal K, Campbell JA (1997) An application of rule-based and case-based reasoning within a single legal knowledge-based system. ACM SIGMIS Database: DATABASE Adv Inf Syst 28:48–63. https://doi.org/10.1145/277339.277344

[16] Timmer I, Rietveld R (2019) Rule-based systems for decision support and decision-making in Dutch legal practice. A brief overview of applications and implications. Droit et Societe 103:517–534. https://doi.org/10.3917/DRS1.103.0517

[17] Wang X, Zhang X, Hoo V, Shao Z, Zhang X (2024) Legalreasoner: a multi-stage framework for legal judgment prediction via large language models and knowledge integration. IEEE Access 12:166843–166854. https://doi.org/10.1109/ACCESS.2024.3496666

[18] Xiao C, Hu X, Liu Z, Tu C, Sun M (2021) Lawformer: a pre-trained language model for Chinese legal long documents. AI Open 2:79–84. https://doi.org/10.1016/j.aiopen.2021.06.003

[19] Yang A, Yang B, Zhang B, Hui B, Zheng B, Yu B, Li C, Liu D, Huang F, Wei H et al (2025) Qwen2.5 technical report. https://arxiv.org/abs/2412.15115

[20] Zheng L, Guha N, Anderson BR, Henderson P, Ho DE (2021) When does pretraining help?: assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In: Proceedings of the eighteenth international conference on artificial intelligence and law. ACM, pp. 159–168. https://doi.org/10.1145/3462757.3466088

← Quay lại danh sách bài báo