CallNavi, A challenge and empirical study on LLM function calling and routing

Yewei Song; Xunzhu Tang; Cedric Lothritz; Saad Ezzini; Jacques Klein; Tegawendé Bissyande; Andrey Boytsov; Ulrick Ble; Anne Goujon

doi:10.1145/3756681.3756975

CallNavi, A challenge and empirical study on LLM function calling and routing

Yewei Song^*
, Xunzhu Tang
, Cedric Lothritz
, Saad Ezzini
, Jacques Klein
, Tegawendé Bissyande
, Andrey Boytsov
, Ulrick Ble
, Anne Goujon

^*Corresponding author for this work

Department of Information and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.

Original language	English
Title of host publication	Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025
Editors	Muhammad Ali Babar, Ayse Tosun, Stefan Wagner, Viktoria Stray
Publisher	Association for Computing Machinery, Inc
Pages	114-125
Number of pages	12
ISBN (Electronic)	9798400713859
DOIs	https://doi.org/10.1145/3756681.3756975
State	Published - 24 Dec 2025
Event	29th International Conference on Evaluation and Assessment of Software Engineering, EASE 2025 - Istanbul, Turkey Duration: 17 Jun 2025 → 20 Jun 2025

Publication series

Name	Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025

Conference

Conference	29th International Conference on Evaluation and Assessment of Software Engineering, EASE 2025
Country/Territory	Turkey
City	Istanbul
Period	17/06/25 → 20/06/25

Bibliographical note

Publisher Copyright:
© 2025 Copyright held by the owner/author(s).

Keywords

Benchmark
Chatbot
Function Calling
Large Language Models

ASJC Scopus subject areas

Software

Access to Document

10.1145/3756681.3756975

Cite this

Song, Y., Tang, X., Lothritz, C., Ezzini, S., Klein, J., Bissyande, T., Boytsov, A., Ble, U., & Goujon, A. (2025). CallNavi, A challenge and empirical study on LLM function calling and routing. In M. A. Babar, A. Tosun, S. Wagner, & V. Stray (Eds.), Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025 (pp. 114-125). (Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025). Association for Computing Machinery, Inc. https://doi.org/10.1145/3756681.3756975

Song, Yewei ; Tang, Xunzhu ; Lothritz, Cedric et al. / CallNavi, A challenge and empirical study on LLM function calling and routing. Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025. editor / Muhammad Ali Babar ; Ayse Tosun ; Stefan Wagner ; Viktoria Stray. Association for Computing Machinery, Inc, 2025. pp. 114-125 (Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025).

@inproceedings{aa92f46293f24bb7bdfeecfd0d9c1974,

title = "CallNavi, A challenge and empirical study on LLM function calling and routing",

abstract = "API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.",

keywords = "Benchmark, Chatbot, Function Calling, Large Language Models",

author = "Yewei Song and Xunzhu Tang and Cedric Lothritz and Saad Ezzini and Jacques Klein and Tegawend{\'e} Bissyande and Andrey Boytsov and Ulrick Ble and Anne Goujon",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 29th International Conference on Evaluation and Assessment of Software Engineering, EASE 2025 ; Conference date: 17-06-2025 Through 20-06-2025",

year = "2025",

month = dec,

day = "24",

doi = "10.1145/3756681.3756975",

language = "English",

series = "Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025",

publisher = "Association for Computing Machinery, Inc",

pages = "114--125",

editor = "Babar, \{Muhammad Ali\} and Ayse Tosun and Stefan Wagner and Viktoria Stray",

booktitle = "Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025",

}

Song, Y, Tang, X, Lothritz, C, Ezzini, S, Klein, J, Bissyande, T, Boytsov, A, Ble, U & Goujon, A 2025, CallNavi, A challenge and empirical study on LLM function calling and routing. in MA Babar, A Tosun, S Wagner & V Stray (eds), Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025. Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025, Association for Computing Machinery, Inc, pp. 114-125, 29th International Conference on Evaluation and Assessment of Software Engineering, EASE 2025, Istanbul, Turkey, 17/06/25. https://doi.org/10.1145/3756681.3756975

CallNavi, A challenge and empirical study on LLM function calling and routing. / Song, Yewei; Tang, Xunzhu; Lothritz, Cedric et al.
Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025. ed. / Muhammad Ali Babar; Ayse Tosun; Stefan Wagner; Viktoria Stray. Association for Computing Machinery, Inc, 2025. p. 114-125 (Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - CallNavi, A challenge and empirical study on LLM function calling and routing

AU - Song, Yewei

AU - Tang, Xunzhu

AU - Lothritz, Cedric

AU - Ezzini, Saad

AU - Klein, Jacques

AU - Bissyande, Tegawendé

AU - Boytsov, Andrey

AU - Ble, Ulrick

AU - Goujon, Anne

PY - 2025/12/24

Y1 - 2025/12/24

N2 - API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.

AB - API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.

KW - Benchmark

KW - Chatbot

KW - Function Calling

KW - Large Language Models

UR - https://www.scopus.com/pages/publications/105027162498

U2 - 10.1145/3756681.3756975

DO - 10.1145/3756681.3756975

M3 - Conference contribution

AN - SCOPUS:105027162498

T3 - Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025

SP - 114

EP - 125

BT - Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025

A2 - Babar, Muhammad Ali

A2 - Tosun, Ayse

A2 - Wagner, Stefan

A2 - Stray, Viktoria

PB - Association for Computing Machinery, Inc

T2 - 29th International Conference on Evaluation and Assessment of Software Engineering, EASE 2025

Y2 - 17 June 2025 through 20 June 2025

ER -

Song Y, Tang X, Lothritz C, Ezzini S, Klein J, Bissyande T et al. CallNavi, A challenge and empirical study on LLM function calling and routing. In Babar MA, Tosun A, Wagner S, Stray V, editors, Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025. Association for Computing Machinery, Inc. 2025. p. 114-125. (Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , EASE, 2025 edition, EASE 2025). doi: 10.1145/3756681.3756975

CallNavi, A challenge and empirical study on LLM function calling and routing

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this