Jupyter AI & .NET Aspire: Building an LLM-Enabled Jupyter Environment

In this post, we will cover installing and configuring Jupyter AI with Jupyter while driving configuration from .NET Aspire. The approach documented here provides out of the box solution using Jupyter AI without having to manually set it up.

What we will cover?

• What is Jupyter AI?
• Adding Jupyter AI to Jupyter image
• Adding Microsoft.dotnet-interactive Jupyter kernel to add c# support to Jupyter Notebooks
• .NET Aspire Configuration to specify code and embedding models / model providers
• Running the custom image using .NET Aspire
• And a quick Python demo

Jupyter AI

Jupyter AI is an extension that adds generative AI support to JupyterLab. With this extension it is possible to use the chat interface to ask about our code in Jupyter Notebooks, the source files and documentation accessible. It can also be used to generate code and inject to a cell.

To get Jupyter AI working the following steps are necessary:

• Install and activate the extension.
• Configure the embedding and language model (model name, endpoint, api keys if needed)
• Access the extension and interact with it

Installing Jupyter AI and .NET Interactive kernel using a custom Dockerfile

In this section we will cover the Dockerfile used as well as the configuration via custom entry point script.

Creating the Dockerfile

This part is straightforward. We start with an appropriate Jupyter base image and then:

• Install .NET 9
• Install Python dependencies using requirements.txt
• Install Microsoft.dotnet-interactive (so we can install .Net Interactive Kernel)
• Copy our entry point file (run.sh)
• Call entry point as a non root user

Yes with this minimal Dockerfile we have JupyterLabs server, Jupyter AI and even .Net Interactive Kernel allowing us using C#, F# and even PowerShell in nor notebooks.

FROM jupyter/base-notebook:ubuntu-22.04
ENV PYTHONDONTWRITEBYTECODE=1

USER root
RUN apt-get update && apt-get install software-properties-common cmake build-essential  libc6  -y

RUN add-apt-repository ppa:dotnet/backports \
    && apt-get update \
    && apt-get install -y dotnet-sdk-9.0  libgl1-mesa-dev  libglib2.0-0 \
    && apt-get clean && rm -rf /var/cache/apt/archives /var/lib/apt/lists/*

USER ${NB_UID}

RUN dotnet tool install -g Microsoft.dotnet-interactive
ENV PATH="${PATH}:/home/jovyan/.dotnet/tools"

COPY ./requirements.txt /home/jovyan/requirements.txt
RUN pip install --no-cache-dir -r /home/jovyan/requirements.txt
RUN dotnet interactive jupyter install

USER root
COPY ./run.sh /home/jovyan/run.sh
RUN chmod +x /home/jovyan/run.sh

USER ${NB_UID}

ENTRYPOINT ["/home/jovyan/run.sh"]

Configuring Jupyter AI via entry point script

This is achieved by our run.sh file as following:

We know that .NET Aspire injects the connection strings and additional config as environment variables. So we will utilise this:

• Pass '' as token so we do not have auth for local Jupyter server.
• We know that Jupyter AI extension is configured at startup by passing --AiExtension.* arguments to Jupyter lab command.
  • We inject relevant --AiExtension.* arguments as passed from our Aspire host using environment variables.
  • Pass EMBEDDING_MODEL and CODE_MODEL as language and embedding models using relevant arguments.
  • Optionally set Embedding and Language model urls (if not using OpenAI).
  • Inject the API Keys (required ig using OpenAI)
• Execute the built entry command.

#!/bin/bash
CODEMODELURL=$(echo "${ConnectionStrings__codemodel}" | cut -d'=' -f2 | cut -d';' -f1)
EMBEDDINGMODELURL=$(echo "${ConnectionStrings__embeddingmodel}" | cut -d'=' -f2 | cut -d';' -f1)

# Base command
CMD="jupyter lab --NotebookApp.token=''"
echo "code model: $CODEMODELURL"
echo "embedding model: $EMBEDDINGMODELURL"

# Add embedding model
CMD="$CMD --AiExtension.default_embeddings_model=$EMBEDDING_MODEL"
# Add code model
CMD="$CMD --AiExtension.default_language_model=$CODE_MODEL"
CMD="$CMD --AiExtension.default_api_keys='{\"OPENAI_API_KEY\":\"${OPENAI_API_KEY}\", \"HUGGINGFACEHUB_API_TOKEN\":\"$HUGGINGFACEHUB_API_TOKEN\"}'"
CMD="$CMD --AiExtension.default_max_chat_history=12"
#,

# Add embedding model URL if specified
if [ ! -z "$EMBEDDINGMODELURL" ]; then
    CMD="$CMD --AiExtension.model_parameters $EMBEDDING_MODEL='{\"base_url\":\"$EMBEDDINGMODELURL\"}'"
fi

# Add code model URL if specified
if [ ! -z "$CODEMODELURL" ]; then
    CMD="$CMD --AiExtension.model_parameters $CODE_MODEL='{\"base_url\":\"$CODEMODELURL\"}'"
fi

# Execute the command
eval "$CMD"

The entry command above will provide us the following when we run out Aspire host:

.NET Aspire configuration and execution

In this section, we will cover the structure of launchSettings.json and the Aspire code putting it all together.

Configuration

Provided example has 3 profiles as following:

• http-ollama-host : Using Ollama running on host with ollama:qwen2.5-coder:32b as code model and ollama:nomic-embed-text as embedding model.
• http-ollama-local : ollama:qwen2.5-coder:14b as code model and ollama:nomic-embed-text as embedding model. and
• http-openai : openai-chat:chatgpt-4o-latest as code model and openai:text-embedding-3-large as embedding model.

As an example, the following is how the settings is configured within launchsettings.json:

    "http-ollama-host": {
      ... 
      "environmentVariables": {
        ... 
        "CODE_MODEL": "ollama:qwen2.5-coder:32b",
        "CODE_MODEL_PROVIDER": "OllamaHost",
        "EMBEDDING_MODEL": "ollama:nomic-embed-text",
        "EMBEDDING_MODEL_PROVIDER": "OllamaHost",
        "EXTERNAL_OLLAMA_CONNECTION_STRING": "Endpoint=http://host.docker.internal:11434;"
      }
    }

Aspire Code

There is not much new here. We are spinning up a Jupyter container using the Dockerfile and optionally spinning up Ollama if the configuration requires so. For more reference the source file is available here

The Dockerfile is run as a container as below:

var jupyter = builder
    .AddDockerfile(Constants.ConnectionStringNames.JupyterService, "./Jupyter")
    .WithBuildArg("PORT", applicationPorts[Constants.ConnectionStringNames.JupyterService])
    .WithBindMount("./Jupyter/Notebooks/", "/home/jovyan/work")
    .WithHttpEndpoint(targetPort: applicationPorts[Constants.ConnectionStringNames.JupyterService], env: "PORT", name:"http")
    .WithLifetime(ContainerLifetime.Session)
    .WithOtlpExporter()
    .WithEnvironment("OTEL_SERVICE_NAME", "jupyterdemo")
    .WithEnvironment("OTEL_EXPORTER_OTLP_INSECURE", "true")
    .WithEnvironment("PYTHONUNBUFFERED", "0")
    .WithEnvironment("CODE_MODEL", chatConfiguration.CodeModel)
    .WithEnvironment("EMBEDDING_MODEL", chatConfiguration.EmbeddingModel);

Python Demo

For the demo the following use case is considered:

"Using the coding assistant, write code to extract SIFT features from two images and match them using approximate nearest neighbour approach (ANN). Then guide the assistant to implement RANSAC using Homography to improve the matches and eliminate false positives."

The initial prompt was straightforward and got a working code without much effort. However this approach is also full of false positives as seen below:

Initial Matches

From prompt two, we start asking improving matches by RANSAC and things start going wrong. however after a number of prompt and /fix commands, we actually get working code without human interaction

Improved matches

The conversation can be seen by inspecting the notebook snapshot:
src/AspireJupyterAI.AppHost/Jupyter/Notebooks/py-qwen-2-5-coder-32b-ollama-host.ipynb

C# Demo

Same use case was tried with a C# notebook however it took GPT-4o to come up with the solution. Given that OpenCV support in .Net is a bit of a niche, it is not surprising the models are not as effective. In addition as we are using .NET interactive in Jupyter, we are also in a niche territory there.

Here is an example notebook with c# (the prompts that led to the final code are missing unfortunately. csharp-openai-gpt-40-latest.ipynb

To get this working on an ARM laptop, the Dockerfile is also more complicated as we actually build OpenCv and OpenCVSharp as a stage in pour Docker file then copy native libraries and bindings to our final stage. Modified Dockerfile to support OpenCvSharp on ARM

The changes to this file are adopted from one of my older posts - Docker multi-architecture, .NET 6.0 and OpenCVSharp

Links

• Repository
• .NET Interactive
• Jupyter AI Post
• Jupyter AI RTD
• Jupyter AI GitHub