Connecting AI Agent with UI - State Management🧩

Image Generation In Next.js application.

From your UI based on your prompt message the copilot agent going to generate the image in your desired location.🤨

Thinking how? If so,let's dive in :)

Hello World! 😀
How are you all doing?Hoping everything is fine!😉So,Today let's dive in with my new stuff which i'm excited to share and if ur intrested u can try along.!👌
As the title of my post describes about ** AI Agent ** and copilotkit.

What are these..?Suddenly Y I ended up connecting these things? Yes,I got the same question when I was doing,But this is fascinating experience where, letting ai agent connect with us and do stuff which we specify with my little document knowledge itself made me more intrested in diving more deeper.

I love this evolution, where there is continues growth!Unlike getting stuck in building and recovering things from bugs and bugs and bugs..😓 through the entire day,which is literally a hopeless experience .And this exciting experience journey.I'm thankful to see such a drastic change in me aswell.Hahahhaa..its quite a story right? I know !😅

In my view CopilotKit works more efficient and smooth than tools like assistant-ui and open-canvas.

What we will walk through here is ,assuming that you are connected to to your nextjs applicaiton with copilot kit if not it's a simple step go through the documentation you will find the selfhosting part or copilot cloud, based on your preference go through it.And set things up like choosing amoung:

• CopilotPopup
• CopilotSidebar
• CopilotChat
• Headless UI I'm working with CopilotkitSidebar.

Unsplash Account - it's free
Now to generate images based on your prompts, create an account in unsplash, and create your new application and access api key token to fetch the images through url, which hepls to generate images based on the url encoding, which will be handled through the state management.

ConnectData.tsx

• Create a new component in your next.js application.

Here we use 2 hooks, useCopilotReadable(), useCopilotAction() from @copilotkit/react-core library.

useCopilotReadable
This hook helps to read the state from the UI.
Example:

useCopilotReadable({
    description: "User-provided prompt when Copilot cannot generate an image",
    value: imagePrompt,
  });

as the default library provides the hook with different parameters described.

useCopilotAction
This hook helps to write state to the UI.

• define the component which you want to display

for example:

export default function ConnectData() {

  return(
    <div>
      <h2>AI Image Generator</h2>
      <div>
        <input
          type="text"
          placeholder="Describe an image..."/>
        <button>Save Prompt</button>
      </div>
     <div>
            <p>Prompt:</p>
            <img src={} alt="Generated"/>
     </div>
        ))}
      </div>
    </div>
}

• Now, to handle states from UI ,I declared 3 useState() hooks.

//for image output
 const [images, setImages] = useState<ImageObject[]>([]);
//for prompt input
  const [imagePrompt, setImagePrompt] = useState<string>("");
//for highlight input
  const [needsUserInput, setNeedsUserInput] = useState(false);

• update the UI state handlers and the display in the return component with preferred tailwindCSS.

return (
    <div className="p-6">
      <h2 className="text-2xl font-semibold">AI Image Generator</h2>

      {/* ✅ Input box is ALWAYS visible, but highlights when needed */}
      <div
        className={`flex flex-col items-center bg-white p-6 rounded-lg shadow-lg w-96 border 
        ${needsUserInput ? "border-red-500 shadow-red-300" : "border-gray-300"}
        transition-all duration-300`}
      >
        <input
          type="text"
          placeholder="Describe an image..."
          value={imagePrompt}
          onChange={(e) => setImagePrompt(e.target.value)}
          className={`w-full p-3 border rounded-md focus:outline-none text-gray-700 shadow-sm
          ${needsUserInput ? "border-red-500 focus:ring-2 focus:ring-red-500" : "border-gray-300 focus:ring-blue-500"}
          transition-all duration-300`}
        />

        <button
          onClick={() => setNeedsUserInput(false)}
          className="mt-4 px-6 py-2 bg-blue-600 text-white font-semibold rounded-md shadow-md hover:bg-blue-700 transition-all duration-200"
        >
          Save Prompt
        </button>
      </div>

      {/* ✅ Display generated images */}
      <div className="mt-5 flex flex-wrap gap-4">
        {images.map((img) => (
          <div key={img.id} className="border p-3 rounded-lg shadow-md">
            <p className="font-semibold">Prompt: {img.prompt}</p>
            <img src={img.url} alt="Generated" className="w-48 rounded-md" />
          </div>
        ))}
      </div>
    </div>
  );

• reading input from the ui by copilotkit

  // ✅ Log state changes when CopilotKit requires input
  useCopilotReadable({
    description: "User-provided prompt when Copilot cannot generate an image",
    value: imagePrompt,
  });

• fetching images from the unsplash

 // ✅ Function to fetch an image based on the user's prompt
  const fetchImageByPrompt = async (prompt: string): Promise<ImageObject | null> => {
    try {
      const response = await fetch(`https://source.unsplash.com/featured/?${encodeURIComponent(prompt)}`);
      return { id: crypto.randomUUID(), url: response.url, prompt };
    } catch (error: unknown) {
        if(error)
        {
            console.log(error);
        }else{
            console.log("unknown error occured");
        }

      return null;
    }
  };

• returning user specific output

// ✅ Copilot Action: Generates image, keeps input field intact
  useCopilotAction({
    name: "generateImage",
    description: "Generate an image based on a user's description",
    parameters: [],
    handler: async () => {
      if (!imagePrompt.trim()) {
        setNeedsUserInput(true);
        console.log("CopilotKit: Needs user input for prompt.");
        return "I need a description to generate an image. Please enter one.";
      }

      const newImage = await fetchImageByPrompt(imagePrompt);

      if (!newImage) {
        setNeedsUserInput(true);
        console.log("CopilotKit: Image generation failed, requesting new input.");
        return "I couldn't generate an image. Please provide a new prompt.";
      }

      setNeedsUserInput(false);
      setImagePrompt(""); // Reset the prompt after successful generation
      setImages((prevImages) => [...prevImages, newImage]);
      console.log("CopilotKit: Image generated successfully.");
      return `Generated an image for: "${imagePrompt}"`;
    },
  });

from the above steps if you add this component to your main page it looks like this.

And if you check your console log and you can test it's state management , starting from reading input ,fetching images, generating results everything can be kept track.

• If not attempted the prompt and asked copilotkit to generate image it triggers the ui component in highlight state, which uses
  our boolean highlight input hook.
  

• If entered proper prompt, the state is updated when we click on the save prompt that triggers the copilotAction to respond to the prompt when user types generate image in the chat,which results in creating the result in our specified location.
  

• U can either check the console.

Doesn't this interaction with AI Agent represents Developer mode UI-Agent Interaction.😄I did enjoyed writing this.What about you.Share your expreriences, queries or anything else in comment section.
I'd love to reach out.
Happy developing.Bye bye!