Coding solutions with AI

We talk a lot on Dev about how we are worried that AI will take our jobs, that it will write code instead of us. I guess we all have a position on that and this article isn't about this subject at all.

What I want to write about is the new multi-tool in my toolbox that I seem to reach for more and more frequently as it continually succeeds in solving my customers' problems. You may have guess that the tool is GPT4-Turbo.

The problem I list below was solved on a train ride of about 90 minutes, including all coding, debugging and running.

The world changed

I've found myself selecting problems to solve that would previously have been left low on the list because of complexity, or that would have been a mostly human task where the resources committed would have been way too costly for the benefit delivered.

Solving problems with AI

If we are going to use AI to solve a problem we have to break down the challenge using the thought processes we use on a daily basis as programmers. We need to decompose problems into multiple smaller units and then solve those.

We can choose to select AI as the solution for a sub problem when that problem is not a clear formula or data manipulation, the most obvious place for this is when there is natural language involved. Natural language is a complex problem, the same meaning can be represented in many different ways - there are lots of other similar classes of problem with scientific measurements, financial data etc, but language is the one I personally run into the most.

A simple example

I've already solved 5 major problems for my solution using the processes I'll talk about here. I'd bore you to tears if I explained some of the complicated ones and I'd need to give you way too much data to explain the issue.

My business

I'm the CTO of a business that aims to ensure buildings are safe for their residents or occupants. For more than 30 years SFG20 has created the industry standard for maintenance regimes that seek to ensure our clients do the activities that prevent harm to individuals and ensure equipment has the longest possible life. Many of these tasks are legal requirements and in recent years the law has changed dramatically after a residential tower block fire in London claimed 72 lives.

SFG20 is used by the UK government, hospitals, care homes, residential property developers and hundreds of private businesses to ensure that they meet legal requirements and maximise the health and safety of building occupants.

SFG20 content is typically used in Computer Aided Facilities Management solutions that actually instruct people to action the tasks at the appropriate time and then ensure that their completion is tracked.

The problem

I have thousands of maintenance schedules that contain tens of thousands of actions to take. These actions are authored by engineers and are ratified to ensure compliance with the law and best practice. The actions are written in free text and comprise a title and a list of steps to perform.

Many CAFM vendors want to allow their users to modify the frequency of some actions or to disable them because they are not relevant to a specific piece of equipment.

Recently a few schedules were rewritten for clarity which worked fine on paper, but meant task steps were grouped into time periods - this messed up the ability to identify specific sub tasks.

I need to make a definitive list of schedules where task titles are now solely time periods and find if there is a previous version where they were not organised that way.

The solution

First lets break down my task into steps.

1. For each maintenance schedule collect the task titles.
2. Identify if one or more of the task titles relate only to a time period: "6 monthly checks" is a problem, but "engine oil three monthly check" is not.
3. If there was a problem, loop over previous versions running steps 1 and 2 until we find a version without a problem.
4. Output a list of schedules where there is a problem and indicate if there is a previous version without an issue so that the Technical Authors can address the challenge.

Using AI

If we were to try to analyse the schedules by hand this would take forever.

If we were to look for tasks that NLP said had a time period: the list is still huge and would take weeks of work for the authors.

AI is good at this kind of task, acting like a person to read the title and identify if the task is solely a time period.

Building and debugging the solution

My approach to the prompt engineering for this task is to iterate.

1. First I ask ChatGPT to tell me what it would make as a prompt for my problem. Meta I know. This is the first debugging stage, if the prompt GPT comes up with seems to miss the point then I fix it here.

2. Next I construct the framework code and use some helper functions I've made to get the output I want, which for most of my cases is some kind of JSON.

3. The final iterative step is to run a bunch of test cases through the AI part and examine the result. If it doesn't do what I want then I give these as examples to GPT and ask it what prompt I should use to avoid these - back to step 1.

The solution

Here's the JS for the code that check a specific schedule. The resulting prompt here was based on two goes around my iteration cycle.

Running this resulted in 64 schedules to be looked at further, 6 of which had been changed in the previous 3 months - which are the ones we were receiving complaints about.

export async function checkScheduleTasks(schedule) {
    if (!schedule.tasks.filter((f) => !f.isHeader && !!f.title).length) return false
    const result = await chatJSON(
        [
            system(
                `
       You will be provided with a list of task names from a maintenance schedule. Your task is to examine the provided task names and determine if any of them represent only a time period, without being part of a more specific task description.

  Criteria for true:
  - A task name that consists solely of a time period description (e.g., "6 monthly", "Annual", "Bi-weekly").

  Criteria for false:
  - A task name that includes a time period as part of a broader task description (e.g., "Electrical tests - 6 monthly", "Annual safety inspection").

  Respond with {"response": true} if any task name meets the criteria for true. Otherwise, respond with {"response": false}.

    `
            ),
            jsonOutput(),
            `
       Here are the task names:

${schedule.tasks
    .filter((f) => !f.isHeader)
    .map((c) => c.title)
    .join("\n")}

    `,
        ],
        0.05
    )
    return result.response
}

Utilities

You can see from the above code I have utility functions to make system instructions and a default instruction for JSON. I'll list them here in case they are of some help.

import OpenAI from "openai"

export let openai
try {
    openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
} catch (e) {
    console.warn(`!!!!!! ERROR in OpenAI`)
    console.error(e)
}

export async function chat(messages, temperature = 0.15, model = "gpt-4-turbo-preview") {
    messages = messages.filter(Boolean).map((m) => (typeof m === "string" ? { role: "user", content: m } : m))
    let retry = false
    let response
    do {
        retry = false
        try {
            response = await openai.chat.completions.create({
                messages,
                model,
                temperature,
            })
        } catch (e) {
            if (e.status === 429) {
                await new Promise((resolve) => setTimeout(resolve, 250))
                retry = true
            }
        }
    } while (retry)
    return response.choices[0].message.content?.value ?? response.choices[0].message.content
}

export async function chatJSON(...params) {
    const text = await chat(...params)
    try {
        return JSON.parse(text)
    } catch (e) {
        console.log(text)
        console.error(e)
    }
}

export async function chatStream(messages, temperature = 0.15, model = "gpt-4-turbo-preview") {
    messages = messages.filter(Boolean).map((m) => (typeof m === "string" ? { role: "user", content: m } : m))
    return openai.chat.completions.create({
        messages,
        model,
        temperature,
        stream: true,
    })
}

export function cleanLines(message) {
    if (Array.isArray(message)) return message.map(cleanLines)
    return message
        .split("\n")
        .map((s) => s.trim())
        .join("\n")
}

export function system(message) {
    return {
        role: "system",
        content: cleanLines(message),
    }
}

export function jsonOutput() {
    return system(`
            Your output will be JSON, not wrapped in \`\`\`json.  Validate the JSON parses correctly before returning, adjust as necessary
        `)
}