How to make an API call in the middle of an OpenAI conversation

While AI integrations become more and more desirable features, the need to perform additional functionality, for example an API call, in the middle of the conversations is the biggest concern for developers. Even though it is possible to provide AI assistants with clear instructions, they sometimes struggle to follow them. The most problematic are if/else conditions.

For demonstration purposes, I've created a console application that will simulate a chat with OpenAI via completion API. So, everything is put inside the background Worker class. We will use Azure.AI.OpenAI package to interact with the API.

Step 0. The basics

Completions API doesn't store the state, so we need to pass the chat history with every call. Chat history should consist of ChatMessage type.
There are several chat ChatMessage types:

• SystemChatMessage (system) - use to provide system instructions that will be considered by OpenAI but won't be visible to the user;
• UserChatMessage (user) - contains messages from the user;
• AssistantChatMessage (assistant) - contains messages from OpenAI;
• ToolChatMessage (tool) - use to provide result of the tool (in our case function) call.

ChatTool - is a concept of additional chat functionality, in our case, a function. So, every function is a ChatTool.

Step 1. Starting a conversation

First of all, we need to have system instructions to describe OpenAI's role in this conversation. Here, we can also tell OpenAI that we want to execute some function after specific user actions.

In this example, I want it to be an assistant in a bookstore, and when the user picks up a book - execute a function that will call our API to get book information, check if there are any deals, and tell the user if there are any.

So, when the user says, "I want to buy this book", we will make a call to our bookstore API, check if there are any, and send SystemChatMessage with new instructions. Doing the deal check on our backend will also help to avoid OpenAI misbehavior with if/else conditions.

Inside such function calls, we can do whatever we need: call an API, do some calculations, send emails, etc.

In the diagram, I show the part of the simplified flow from the user message that indicates a desire to buy a book to the new instruction for OpenAI.

Let's now move on to the implementation.

We will store message history in the messagesHistory property of type List<ChatMessage>.
ExecuteAsync is an entry point for our background worker. So, we are waiting for the user to type anything, then save the message and continue our conversation.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    var systemInstructions = 
        $"""
            You are an assistant in a bookstore. Ask user about their interests and genres they like 
            or what they would like to read about and based on the answer suggest 5 books.
            After suggestion ask if user wants to buy any of the books. If user wants to buy a book,
            call {FindBookFunctionName} with the book name and author.
        """;

    _messagesHistory.Add(new SystemChatMessage(systemInstructions));

    Console.WriteLine("Assistant: Hello! How can I help?");

    while (!stoppingToken.IsCancellationRequested)
    {
        Console.Write("User: ");

        var question = Console.ReadLine() ?? throw new InvalidOperationException();
        _messagesHistory.Add(new UserChatMessage(question));

        await ContinueConversation();
    }
}

Calling a function without any context would be useless. We need to have information about the book that user has chosen: name and author. For this, we need to define which fields we are expecting to receive from OpenAI. It can be done as the following:

private static List<ChatTool> GetFunctionsDefinitions()
{
    return
    [
        ChatTool.CreateFunctionTool(
            functionName: FindBookFunctionName,
            functionDescription: "Returns a list of books suggested to a user.",
            functionParameters: BinaryData.FromString(JsonSerializer.Serialize(new
                {
                    type = "object",
                    properties = new
                    {
                        name = new { type = "string", description = "Book name" },
                        author = new { type = "string", description = "Book author" },
                    },
                    additionalProperties = false,
                    required = new[] { "name", "author" }
                }
        ))),
    ];
}

Step 2. Continue Conversation

To continue conversation, we will always send message history to the OpenAI API including functions definition if there are any.

private async Task<ChatCompletion> SendMessages(ChatMessage[] messages)
{
    var chatClient = openAiClient.GetChatClient("gpt-4o-mini");

    ChatCompletionOptions options = new();
    foreach (var tool in GetFunctionsDefinitions())
    {
        options.Tools.Add(tool);
    }

    return await chatClient.CompleteChatAsync(messages, options);
}

The response will always be a ChatCompletion object with FinishReason which could be one of the following: Stop, ToolCalls, Length, ContentFilter and FunctionCall.
We are interested in the first two. For simplification, others are omitted in this example.

• Stop indicates that there is an ordinary response from OpenAI with some text. We need to store it as an AssistantChatMessage.
• ToolCalls means that OpenAI identified that some Tool is required to be executed. It is our responsibility to execute it. The text from the response should be also stored in AssistantChatMessage. And, in addition, each tool call result should be saved as ToolChatMessage. After ToolChatMessage and SystemChatMessage are sent, we need to ContinueConversation, so OpenAI can react to new prompts.

In this example, we will execute tool calls in ExecuteToolCall method.

private async Task ContinueConversation()
{
    var chatCompletion = await SendMessages(_messagesHistory.ToArray());

    switch (chatCompletion.FinishReason)
    {
        case ChatFinishReason.Stop:
            _messagesHistory.Add(new AssistantChatMessage(chatCompletion.Content[0].Text));
            Console.WriteLine($"{chatCompletion.Role}: {chatCompletion.Content[0].Text}");
            break;
        case ChatFinishReason.ToolCalls:
        {
            _messagesHistory.Add(new AssistantChatMessage(chatCompletion.ToolCalls));

            foreach (var toolCall in chatCompletion.ToolCalls)
            {
                var response = await ExecuteToolCall(toolCall);
                _messagesHistory.Add(new ToolChatMessage(toolCall.Id, JsonSerializer.Serialize(response)));

                if (response.SystemMessage is not null)
                {
                    _messagesHistory.Add(new SystemChatMessage(response.SystemMessage));
                }
            }

            await ContinueConversation();
            break;
        }
        default:
            throw new ArgumentOutOfRangeException();
    }

Step 3. Execute Tool Call

In ExecuteToolCall we will define how to execute each function that we may receive. In this case, it's only one, but could be any amounts.

private async Task<FunctionResponse> ExecuteToolCall(ChatToolCall toolCall)
{
    var response = toolCall.FunctionName switch
    {
        FindBookFunctionName => await GetBookDetails(toolCall),
        //other functions calls
        _ => throw new NotImplementedException()
    };

    return response;
}

Our goal here is to call our bookstore backend API to get information about the book that user wants to buy: is it available and does it have any deals that user should be notified about.

Step 4. Make an API call

GetBookDetails is the method where we can perform anything we need to extend conversation logic. It is the actual implementation of the FindBookFunctionName.

Since toolCall.FunctionArguments is binary, we can convert it into string and deserialize it to the BookRecommendation DTO.
Then, it's time to call bookstore API to get book information. After that, depending on the Deal type, add a system message.

private async Task<FunctionResponse> GetBookDetails(ChatToolCall toolCall)
{
    var bookRecommendation = JsonSerializer.Deserialize<BookRecommendation>(toolCall.FunctionArguments.ToString());
    if (bookRecommendation is null)
    {
        throw new Exception("Cannot parse the book recommendation.");
    }

    var bookDetail = await bookApiClient.GetBooks(bookRecommendation); 

    string? systemMessage = null;

    if (bookDetail.Deals.Contains(Deal.OnePlusOne))
    {
        systemMessage = "If you bye one more book, you get another one for free!";
    }
    if (bookDetail.Deals.Contains(Deal.ThirtyPercentOff))
    {
        systemMessage += "Lucky time! This book is 30% off! To apply the discount, use the code: LUCKY30";
    }
    if (bookDetail.Deals.Contains(Deal.FreeShippingIfOverFifty))
    {
        systemMessage += "If you buy books for more than 50$, you get free shipping!";
    }

    return new FunctionResponse(JsonSerializer.Serialize(bookDetail), systemMessage);
}

public record BookRecommendation(string Name, string Author);

public record BookDetail(string Name, string Author, decimal Price, List<Deal> Deals);

public enum Deal
{
    OnePlusOne,
    ThirtyPercentOff,
    FreeShippingIfOverFifty
}

public record FunctionResponse(string Content, string? SystemMessage = null);

As you can see in the second step, the system message will be sent to the OpenAI to be considered.

Let's see how the conversation will look like.

For the selected book, the deal is 1+1=2, meaning buy one more book - get another one for free. OpenAI understands the user's requests and, according to the system instructions, kindly informs the user about this possibility and even asks if he wants to use it.

I think that's a very accurate example of how businesses can benefit from OpenAI integration, especially when it is enhanced by custom functions calls.

Have you already tried OpenAI or other integrations on your projects? Do you like how it performs?

Comments

[Igor Igor]

Very helpful topic, thank you!
We are using quite similar approach and the main things we have figured out are:

• gpt-4o model performs much more better than gpt-4o-mini without visible response time differences;
• additional System messages during conversation performs much more better than 1 huge instruction in the beginning of the conversation;
• unfortunately from time to time OpenAI going crazy. Is possible that some flow is working good 40 times, but generates weird things on the 41th run.

[Julia Shevchenko]

Thanks for sharing!
It is my main concern that response may be unpredictable. Let's hope gpt-4.5 will be more stable.