Launching AI Prototyping Projects

Define your AI Prototyping Goals

In the first part of this series we discussed what AI Prototyping Projects are and why you would launch one. In this article we'll move along in that process and discuss launching AI prototyping projects in a way that sets them up for success.

In order to start your project, you’ll need to have a good idea of what your team needs to build. This could range anywhere from a paragraph description of a simple system to more complex definitions such as wireframes and component user stories the team can work on.

A simple project statement might look like this:

We’re launching a half-week spike to investigate the viability of using Azure Document Intelligence to automate extracting these 10 pieces of information from documents in 12 different formats from various vendors so that we can reliably automate data extraction and better handle high-volume spikes of incoming documents. This project will involve an AI specialist on staff as well as a team lead responsible for the area of concern.

This goal statement makes the following things clear:

• The problem type (document data extraction)
• The technology or technologies we’re considering using (Azure Document Intelligence)
• Who is working on the project
• How long the project will take
• Characteristics of the inputs to the system (documents from different sources that contain common pieces of information)
• Concerns about reliably, accuracy, and scalability of the resulting system

A more complex project might have an overall goal statement like this, but also a number of different user stories made available in pre-prioritized order.

As you build a system and interact with it, you and your team will have additional ideas or find you may need to adjust the project direction to fit the realities of the interactions you’re seeing. This is normal and acceptable, but you want to start the project with a goal of what you’re building.

Before we move on, I do want to state that if your changes in-flight are substantial, they may call for pausing your prototyping project and launching a new project with a modified goal.

Defining Usage Scenarios

With your goal in hand, you now need to outline the main ways you envision your system being used.

For a conversational AI system, I’d recommend documenting things you think a user might ask the system - and possibly how you would want the system to respond.

Here’s a sample set of requests you might include for an automated chat system related to eCommerce:

What are the best electric shaving products for men designed for the head and neck area available below $60?*

What is the typical return policy like for an item I don’t like?

Is the kilt hose in navy blue that I added to my wishlist back in stock?

When is my board game expected to arrive?

Each of these questions is specific to the business, and in some cases, specific to the user using the application. Each request also shows how the system is designed to be used and represents a discrete task the application might need to support (or respond to stating that it cannot do that yet).

By documenting your sample requests to your system, you have a good way of explaining what your system was designed to do, a good way of testing your system after making changes, and the beginnings of some training data for optimizing your system’s behavior.

In fact, if you were to expand your sample questions to also include sample answers, you could include these in your prompt to your conversational AI system as few-shot inference examples as a way of teaching your system how to structure its responses.

Of course, with data-specific queries about products in stock or past user orders, you’re likely looking at a retrieval augmentation generation (RAG) or AI orchestration system, but these are details about how a solution is implemented, not how we expect the system to be used.

Picking your Team and Timeline

With your goal and sample usage scenarios defined, it’s important to figure out who on your team should be included in the project.

The more people you have on the project, the less any one individual can focus on executing their tasks and getting things done, so add people to your project carefully.

I recommend putting highly-skilled people with specific specializations in terms of skills or domain knowledge on the project. For best results, I recommend having people with complementary specializations to one another, such as an AI specialist, DevSecOps specialist, and a front end specialist on a small team together.

You’ll want your team members to be able to attack specific tasks independently while occasionally coordinating with each other to make sure their efforts are lining up well.

For particularly complex problem areas you may want to have several people with the same or related skillsets so they can talk through complex obstacles or approaches with each other. In these scenarios it can be helpful to partition work areas in advance so individuals can work independently without impacting each other’s efforts.

As far as a timeline goes, you should ask your team to think about the things they know are easily achievable versus the things that will take some additional time, research, and trial and error. I recommend your team plans out your first day or day and a half on the project in terms of what they’ll be working on. Based on your comfort for how much they’ll get done in that time and how much work is left, you can adjust the end date of the project.

I’d strongly recommend keeping your projects contained to a few days to a week and treating them almost like an in-office hackathon because the longer your project goes, the slower your overall pace will be. This is because innovation projects tend to have a lot of early progress as people work through their plans and tackle known problems and expand into new areas, but the high-intensity pace starts to wear at the team’s overall productivity as the days go on.

By keeping projects constricted in terms of time, you keep the team focused on the core goal and are more responsible with their overall energy level - as well as with external commitments from their normal work that you need to keep at bay during a prototyping project.

Defining Data Needs

One critical factor to consider when launching an AI prototyping project is the quality of your data.

If you are using data to train an AI system for predictions or even simply as a resource to search, that data needs to be reliable or your resulting AI system will internalize the inaccuracies of your data.

Depending on what you’re trying to achieve, inaccurate data could manifest as simply as incorrect recommendations or facts spouted by the AI system. However, in more sensitive scenarios, such as training machine learning models, incorrect data could significantly bias your model towards inaccurate predictions.

These predictions may only show up with certain other combinations of data and may be hard to detect as a result. For example, if a data collection anomaly resulted in incorrect data points for the state of Ohio for a two month period of time, your system may wind up being inaccurate for residents of Ohio during certain months. Because it’s unlikely you routinely test data in your systems from every state for every month of the year, these inaccuracies could very well reach production if you don’t find them in your data earlier.

For this reason, you should be familiar with the data your team will likely want to use for the AI prototyping project and its limitations. Additionally, you should reserve time to analyze your dataset for anomalies and additional data cleaning opportunities as data reliability can make or break a project.

Defining AI Project Deployment and Operation Parameters

One final aspect of AI projects to consider is how the AI system will be deployed if the prototype succeeds and is promoted to a full project that then goes on to deploy to production.

While it may seem counterintuitive to think about deployment of a prototyping project designed for exploring viability, one of the concerns for a viable AI project is the ability to deploy and maintain such a system.

For example, if your system relies around data, you should have a plan for how new data gets into your system and how you evaluate that system to ensure you’re not breaking any existing capabilities by adding new context.

Even if your system doesn’t rely on changing data, you will want to patch it from time to time, move it to a different model, add capabilities, or even just style it differently or tweak its personality.

Each of these operations involves deploying or updating the AI system and, in turn, evaluating the effectiveness of the system has not significantly diminished after the change.

In the short term, this likely means a manual testing plan, however more professional and automated solutions would involve automated tests that ensure your system acts properly and its responses continue to meet your organizational standards.