NL2Vis Transformed: From Linguistic Abstraction to Visual Specification in the Generative AI Era
Tác giả
Tóm tắt
Tài liệu tham khảo
[1] Yang W, Liu M, Wang Z, Liu S. Foundation models meet visualizations: challenges and opportunities. Comput Visual Media. 2024;10(3):399–424. https://doi.org/10.1007/s41095-023-0393-x.
[2] Wu A, Wang Y, Shu X, Moritz D, Cui W, Zhang H, et al. Ai4vis: Survey on artificial intelligence approaches for data visualization. IEEE Trans Visual Comput Graphics. 2021;28(12):5049–70. https://doi.org/10.1109/TVCG.2021.3099002.
[3] Ye Y, Hao J, Hou Y, Wang Z, Xiao S, Luo Y, et al. Generative Ai for visualization: state of the art and future directions. Visual Informatics. 2024. https://doi.org/10.1016/j.visinf.2024.04.003.
[4] Wu Y, Wan Y, Zhang H, Sui Y, Wei W, Zhao W, et al. Automated data visualization from natural language via large language models: an exploratory study. Proc ACM Manag Data. 2024;2(3):1–28.
[5] Zhang W, Wang Y, Song Y, Wei VJ, Tian Y, Qi Y, et al. Natural language interfaces for tabular data querying and visualization: A survey. IEEE Trans Knowl Data Eng. 2024. https://doi.org/10.1109/TKDE.2024.3400824.
[6] c Y, Tang N, Li G, Tang J, Chai C, Qin X. Natural language to visualization by neural machine translation. IEEE Trans Visual Computer Graphics 2021;28(1):217–226. https://doi.org/10.1109/TVCG.2021.3114848
[7] Maddigan P, Susnjak T. Chat2vis: Generating data visualizations via natural language using chatgpt, codex and gpt-3 large language models. Ieee Access. 2023;11:45181–93. https://doi.org/10.1109/ACCESS.2023.3274199.
[8] Lundgard A, Satyanarayan A. Accessible visualization via natural language descriptions: A four-level model of semantic content. IEEE Trans Visual Comput Graphics. 2021;28(1):1073–83. https://doi.org/10.1109/TVCG.2021.3114770.
[9] Zhao Y, Zhang Y, Zhang Y, Zhao X, Wang J, Shao Z, et al. Leva: Using large language models to enhance visual analytics. IEEE Trans Visual Comput Graphics. 2024. https://doi.org/10.1109/TVCG.2024.3368060.
[10] Xu Z, Wall E. Exploring the capability of llms in performing low-level visual analytic tasks on svg data visualizations. In: 2024 IEEE Visualization and Visual Analytics (VIS), St. Pete Beach, FL, USA, pp. 2024:126–130. IEEE.
[11] Xie L, Zheng C, Xia H, Qu H, Zhu-Tian C. Waitgpt: Monitoring and steering conversational llm agent in data analysis with on-the-fly code visualization. In: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. UIST ’24. Association for Computing Machinery, New York, NY, USA 2024. https://doi.org/10.1145/3654777.3676374.
[12] Tang J, Luo Y, Ouzzani M, Li G, Chen H. Sevi: Speech-to-visualization through neural machine translation. In: Proceedings of the 2022 International Conference on Management of Data, 2022;2353–2356. https://doi.org/10.1145/3514221.3520150
[13] Srinivasan A, Stasko J. Orko: Facilitating multimodal interaction for visual exploration and analysis of networks. IEEE Trans Visual Comput Graphics. 2017;24(1):511–21. https://doi.org/10.1109/TVCG.2017.2745219.
[14] Shen L, Shen E, Luo Y, Yang X, Hu X, Zhang X, et al. Towards natural language interfaces for data visualization: A survey. IEEE Trans Visual Comput Graphics. 2022;29(6):3121–44. https://doi.org/10.1109/TVCG.2022.3148007.
[15] Wang Q, Chen Z, Wang Y, Qu H. A survey on ml4vis: Applying machine learning advances to data visualization. IEEE Trans Visual Comput Graphics. 2021;28(12):5134–53. https://doi.org/10.1109/TVCG.2021.3106142.
[16] Wan Z, Song Y, Li S, Zhang CJ, Wong RC-W. Datavist5: A pre-trained language model for jointly understanding text and data visualization. In: 2025 IEEE 41st International Conference on Data Engineering (ICDE), Hong Kong, Hong Kong, pp. 1704–1717 2025. IEEE.
[17] Arya S, Kaji AH, Boermeester MA. Prisma reporting guidelines for meta-analyses and systematic reviews. JAMA Surg. 2021;156(8):789–90.
[18] Nguyen T-V, Phung T-N, Cuong D-D. A bibliometric and thematic analysis of systematic reviews of artificial intelligence in education. In: International Conference on Advances in Information and Communication Technology, pp. 337–351 2024. https://doi.org/10.1007/978-3-031-50818-9_37. Springer
[19] Nguyen VT, Nguyen CT, Yooc S-C, Jung K. Unveiling augmented reality applications: exploring influential factors through comprehensive review. SN Comput Sci. 2023;4(5):694. https://doi.org/10.1007/s42979-023-02175-4.
[20] Nguyen VT, Nguyen CT. A systematic review of structural equation modeling in augmented reality applications. Indon J Electr Eng Comput Sci 328–338 2022. https://doi.org/10.11591/ijeecs.v28.i1.pp328-338
[21] Quandt M, Freitag M. A systematic review of user acceptance in industrial augmented reality. Front Educ. 2021;6(498):10.
[22] Petersen K, Gerken JM. On the road to interactive llm-based systematic mapping studies. Inf Softw Technol. 2025;178:107611. https://doi.org/10.1016/j.infsof.2024.107611.
[23] Gao T, Dontcheva M, Adar E, Liu Z, Karahalios KG. Datatone: Managing ambiguity in natural language interfaces for data visualization. In: Proceedings of the 28th Annual Acm Symposium on User Interface Software & Technology, pp. 489–500 2015. https://doi.org/10.1145/2807442.2807478
[24] Setlur V, Battersby SE, Tory M, Gossweiler R, Chang AX. Eviza: A natural language interface for visual analysis. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 365–377 2016. https://doi.org/10.1145/2984511.2984588
[25] Hoque E, Setlur V, Tory M, Dykeman I. Applying pragmatics principles for interaction with visual analytics. IEEE Trans Visual Comput Graphics. 2017;24(1):309–18. https://doi.org/10.1109/TVCG.2017.2744684.
[26] Narechania A, Srinivasan A, Stasko J. Nl4dv: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Trans Visual Comput Graphics. 2020;27(2):369–79. https://doi.org/10.1109/TVCG.2020.3030378.
[27] Rashid MM, Jahan HK, Huzzat A, Rahul RA, Zakir TB. Meem F, Mukta MSH, Shatabda S. Text2chart: A multi-staged chart generator from natural language text. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 3–16 2022. https://doi.org/10.1007/978-3-031-05936-0_1. Springer
[28] Chen Q, Pailoor S, Barnaby C, Criswell A, Wang C, Durrett G, et al. Type-directed synthesis of visualizations from natural language queries. Proc ACM Program Lang. 2022;6(OOPSLA2):532–59. https://doi.org/10.1145/3563307.
[29] Liu C, Han Y, Jiang R, Yuan X. Advisor: Automatic visualization answer for natural-language question on tabular data. In: 2021 IEEE 14th Pacific Visualization Symposium (PacificVis), Tianjin, China, pp. 11–20 2021. IEEE
[30] Dibia V, Demiralp Ç. Data2vis: Automatic generation of data visualizations using sequence-to-sequence recurrent neural networks. IEEE Comput Graphics Appl. 2019;39(5):33–46. https://doi.org/10.1109/MCG.2019.2924636.
[31] Luo Y, Qin X, Tang N, Li G. Deepeye: Towards automatic data visualization. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 101–112 (2018). IEEE
[32] Song Y, Zhao X, Wong RC-W, Jiang D. Rgvisnet: A hybrid retrieval-generation neural framework towards automatic data visualization generation. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1646–1655 2022. https://doi.org/10.1145/3534678.3539330
[33] Li S, Chen X, Song Y, Song Y, Zhang C. Prompt4vis: Prompting large language models with example mining and schema filtering for tabular data visualization. arXiv preprint arXiv:2402.07909 2024.
[34] Wang T, He J, Xiong C. Ragviz: Diagnose and visualize retrieval-augmented generation. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 320–327. Association for Computational Linguistics, Miami, Florida, USA 2024.
[35] Ma X, Zhuang S, Koopman B, Zuccon G, Chen W, Lin J. Visa: Retrieval augmented generation with visual source attribution. arXiv preprint arXiv:2412.14457 2024.
[36] Reimers N, Gurevych I. Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992 2019.
[37] Chen J, Xiao S, Zhang P, Luo K, Lian D, Liu Z. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In: Ku L-W, Martins A, Srikumar V (eds) Findings of the Association for Computational Linguistics: ACL 2024, pp. 2318–2335. Association for Computational Linguistics, Bangkok, Thailand 2024. https://aclanthology.org/2024.findings-acl.137/
[38] Johnson J, Douze M, Jégou H. Billion-scale similarity search with gpus. IEEE Trans Big Data. 2019;7(3):535–47. https://doi.org/10.1109/TBDATA.2019.2921572.
[39] Malkov YA, Yashunin DA. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Trans Pattern Anal Mach Intell. 2018;42(4):824–36. https://doi.org/10.1109/TPAMI.2018.2889473.
[40] Luong-Thi-Minh H, Nguyen-The V, Xuan TQ. Vizagent: Towards an intelligent and versatile data visualization framework powered by large language models. In: International Conference on Advances in Information and Communication Technology, pp. 89–97 2024. Springer.
[41] Dibangoye J, Buffet O. Learning to act in decentralized partially observable mdps. In: International Conference on Machine Learning, pp. 1233–1242 2018. PMLR.
[42] Goswami K, Mathur P, Rossi R, Dernoncourt F. Plotgen: Multi-agent llm-based scientific data visualization via multimodal feedback. arXiv preprint arXiv:2502.00988 2025.
[43] Ouyang G, Chen J, Nie Z, Gui Y, Wan Y, Zhang H, Chen D. nvagent: Automated data visualization from natural language via collaborative agent workflow. arXiv preprint arXiv:2502.05036 2025.
[44] Rahman MWU, Nevarez R, Mim LT, Hariri S. Multi-agent actor-critic generative ai for query resolution and analysis. IEEE Trans Artif Intell. 2025. https://doi.org/10.1109/TAI.2025.3544173.
[45] Yang Z, Zhou Z, Wang S, Cong X, Han X, Yan Y, Liu Z, Tan Z, Liu P, Yu D, Liu Z, Shi X, Sun M. Matplotagent: Method and evaluation for llm-based agentic scientific data visualization 2024. arXiv:2402.11453 [cs.CL]
[46] Wang C, Lee B, Drucker S, Marshall D, Gao J. Data formulator 2: Iterative creation of data visualizations, with ai transforming data along the way. 2025. arXiv:2408.16119 [cs.HC]
[47] Ram GKS, Muthumanikandan V. Visistant: A conversational chatbot for natural language to visualizations with gemini large language models. IEEE Access. 2024;12:138547–63. https://doi.org/10.1109/ACCESS.2024.3465541.
[48] Zhao Y, Wang J, Xiang L, Zhang X, Guo Z, Turkay C, Zhang Y, Chen S. Lightva: Lightweight visual analytics with llm agent-based task planning and execution. IEEE Trans Vis Comput Gr, 2024:1–13. https://doi.org/10.1109/TVCG.2024.3496112
[49] Luo Y, Tang J, Li G. nvbench: A large-scale synthesized dataset for cross-domain natural language to visualization task. arXiv preprint arXiv:2112.12926 (2021)
[50] Yu T, Zhang R, Yang K, Yasunaga M, Wang D, Li Z, Ma J, Li I, Yao Q, Roman S et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 2018.
[51] Chen N, Zhang Y, Xu J, Ren K, Yang Y. Viseval: A benchmark for data visualization in the era of large language models. IEEE Trans Visual Comput Gr. 2024. https://doi.org/10.1109/TVCG.2024.3456320.
[52] Srinivasan A, Nyapathy N, Lee B, Drucker SM, Stasko J. Collecting and characterizing natural language utterances for specifying data visualizations. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021;1–10.
[53] Song Y, Zhao X, Wong RC-W. Marrying dialogue systems with data visualization: Interactive data visualization generation from natural language conversations. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024;2733–2744.
[54] Ge Y, Wei VJ, Song Y, Zhang JC, Wong RC-W. Automatic data visualization generation from chinese natural language questions. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 1889–1898 2024.
[55] Hu K, Bakker MA, Li S, Kraska T, Hidalgo C. Vizml: A machine learning approach to visualization recommendation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 2019.
[56] Hu K, Gaikwad S, Hulsebos M, Bakker MA, Zgraggen E, Hidalgo C, Kraska T, Li G, Satyanarayan A, Demiralp Ç. Viznet: Towards a large-scale visualization learning and benchmarking repository. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 2019.
[57] Kantharaj S, Leong RT, Lin X, Masry A, Thakkar M, Hoque E, Joty S. Chart-to-text: A large-scale benchmark for chart summarization. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4005–4023 2022.
[58] Wu Y, Yan L, Shen L, Wang Y, Tang N, Luo Y. Chartinsights: Evaluating multimodal large language models for low-level chart question answering. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12174–12200 2024.
[59] Methani N, Ganguly P, Khapra MM, Kumar P. Plotqa: Reasoning over scientific plots. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527–1536 2020.
[60] Masry A, Do XL, Tan JQ, Joty S, Hoque E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279 2022.
[61] Kahou SE, Michalski V, Atkinson A, Kádár Á, Trischler A, Bengio Y. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300 2017.
[62] Ko H-K, Jeon H, Park G, Kim DH, Kim NW, Kim J, Seo J. Natural language dataset generation framework for visualizations powered by large language models. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1–22 2024.
[63] Sah S, Mitra R, Narechania A, Endert A, Stasko J, Dou W. Generating analytic specifications for data visualization from natural language queries using large language models. arXiv preprint arXiv:2408.13391 2024.
[64] Tian Y, Cui W, Deng D, Yi X, Yang Y, Zhang H, et al. Chartgpt: Leveraging llms to generate charts from abstract natural language. IEEE Trans Visual Comput Graphics. 2024. https://doi.org/10.1109/TVCG.2024.3368621.
[65] Shi S, Ren T, Zhu G, Feng G, Hu J. Closing the feedback loop in text2vis: Refining visualization with vision-language models. In: Proceedings of the 33rd ACM International Conference on Multimedia, pp. 9053–9061 2025.
[66] Luo Y, Tang N, Li G, Chai C, Li W, Qin X. Synthesizing natural language to visualization (nl2vis) benchmarks from nl2sql benchmarks. In: Proceedings of the 2021 International Conference on Management of Data, pp. 1235–1247 2021.
[67] Li G, Wang X, Aodeng G, Zheng S, Zhang Y, Ou C, Wang S, Liu CH. Visualization generation with large language models: An evaluation. arXiv preprint arXiv:2401.11255 2024.
[68] Liu S-C, Wang S, Lin W, Hsiung C-W, Hsieh Y-C, Cheng Y-P, Luo S-H, Chang T, Zhang J. Jarvix: A llm no code platform for tabular data analysis and optimization. arXiv preprint arXiv:2312.02213 2023.
[69] Mallick T, Yildiz O, Lenz D, Peterka T. Chatvis: Automating scientific visualization with a large language model. In: SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, Atlanta, GA, USA, pp. 49–55 2024. IEEE
[70] Mitra R, Narechania A, Endert A, Stasko J. Facilitating conversational interaction in natural language interfaces for visualization. In: 2022 IEEE Visualization and Visual Analytics (VIS), pp. 6–10 (2022). https://doi.org/10.1109/VIS54862.2022.00010. IEEE.
[71] Hong M-H, Crisan A. Conversational ai threads for visualizing multidimensional datasets. arXiv preprint arXiv:2311.05590 2023.
[72] Bromley D, Setlur V. Dash: A bimodal data exploration tool for interactive text and visualizations. In: 2024 IEEE Visualization and Visual Analytics (VIS), pp. 256–260 2024. IEEE.
[73] Chen Y, Li R, Mac A, Xie T, Yu T, Wu E. Nl2interface: Interactive visualization interface generation from natural language queries. arXiv preprint arXiv:2209.08834 2022.
[74] Zhan Y, Cui L, Weng H, Wang G, Tian Y, Liu B, Yang Y, Yin X, Xie J, Sun Y. Towards database-free text-to-sql evaluation: A graph-based metric for functional correctness. In: Proceedings of the 31st International Conference on Computational Linguistics, pp. 4586–4610 2025.
[75] Sharma M, Gogineni AK, Ramakrishnan N. Neural methods for data-to-text generation. ACM Trans Intell Syst Technol. 2024;15(5):1–46. https://doi.org/10.1145/3660639.
[76] Kim H. Interactive systems for data visualization for multiple user contexts. PhD thesis, Northwestern University 2024.
[77] Zhou K, Liu Z, Chen R, Li L, Choi S-H, Hu X. Table2graph: Transforming tabular data to unified weighted graph. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pp. 2420–2426 2022. https://doi.org/10.24963/ijcai.2022/336
[78] Wang Y, Hou Z, Shen L, Wu T, Wang J, Huang H, et al. Towards natural language-based visualization authoring. IEEE Trans Visual Comput Gr. 2022;29(1):1222–32. https://doi.org/10.1109/TVCG.2022.3209357.
[79] Wang C, Lee B, Drucker S, Marshall D, Gao J. Data formulator 2: Iteratively creating rich visualizations with ai. arXiv preprint arXiv:2408.16119 2024.
[80] Dibia V. LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. In: Bollegala D, Huang R, Ritter A (eds) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 113–126. Association for Computational Linguistics, Toronto, Canada 2023. https://aclanthology.org/2023.acl-demo.11/
[81] Feng Y, Wang X, Pan B, Wong KK, Ren Y, Liu S, et al. Xnli: Explaining and diagnosing nli-based visual data analysis. IEEE Trans Vis Comput Gr. 2023. https://doi.org/10.1109/TVCG.2023.3240003.
[82] Young S, Gašić M, Thomson B, Williams JD. Pomdp-based statistical spoken dialog systems: a review. Proc IEEE. 2013;101(5):1160–79. https://doi.org/10.1109/JPROC.2012.2225812.
[83] Galimzyanov T, Titov S, Golubev Y, Bogomolov E. Drawing pandas: A benchmark for llms in generating plotting code. arXiv preprint arXiv:2412.02764 2024.
[84] Zou R, Tang Y, Chen J, Lu S, Lu Y, Yang Y, Ye C. Gistvis: Automatic generation of word-scale visualizations from data-rich documents. arXiv preprint arXiv:2502.03784 2025.
[85] Liu C, Yu J, Guo Y, Zhuang J, Luo Y, Yuan X. Breathing new life into existing visualizations: A natural language-driven manipulation framework. arXiv preprint arXiv:2404.06039 2024.
[86] Auger T, Saroyan E. Overview of the openai apis. In: Generative AI for Web Development: Building Web Applications Powered by OpenAI APIs and Next. Js, pp. 87–116. Springer, Berlin, 2024. https://doi.org/10.1007/979-8-8688-0885-2_6
[87] Nguyen C, Carrion D, Badawy MK. Comparative performance of anthropic claude and openai gpt models in basic radiological imaging tasks. J Med Imaging Radiat Oncol. 2025. https://doi.org/10.1111/1754-9485.13858.
[88] Bako HK, Bhutani A, Liu X, Cobbina KA, Liu Z. Evaluating the semantic profiling abilities of llms for natural language utterances in data visualization. In: 2024 IEEE Visualization and Visual Analytics (VIS), pp. 261–265 2024. https://doi.org/10.1109/VIS55277.2024.00060. IEEE.
[89] Zhang Z, Ma W, Vosoughi S. Is gpt-4v (ision) all you need for automating academic data visualization? exploring vision-language models’ capability in reproducing academic charts. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8271–8288 2024.
[90] Beasley C, Abouzied A. Pipe (line) dreams: Fully automated end-to-end analysis and visualization. In: Proceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics, New York, NY, USA, pp. 1–7 2024.