Cloudflare AI Hackathon: MedicalAI - Revolutionizing Healthcare with AI

#cloudflarechallenge #devchallenge #ai #webdev

This is a submission for the Cloudflare AI Challenge.

What We Built

In the fast-paced world of healthcare, access to comprehensive patient data is paramount for informed decision-making. However, with data scattered across various sources such as text, images, and audio recordings (conversation between doctor and patient), extracting meaningful insights can be a daunting task for doctors and healthcare professionals.

That's where our innovative solution comes in. We're excited to introduce our hackathon project aimed at empowering doctors with a powerful tool to streamline data aggregation and analysis: Medical AI.

This is a team submission.
Team Member: SenthilBalaji

Demo

Demo App Link: AI Medical Advisor

Our Code

Client Repo

Worker repo

Journey

We have created a solution which will extract text from other data sources like text (observations from previous consultation), image (pic of the disease/affected area), audio (Discussion between patient and doctor or similar). And create a summarized patient data which will help the doctors to get clear idea about the patient current condition and history.

Key Features

Process any data: This feature allows to provide from different data source, it makes user easy to feed the data.
Diagnose : Generate a clear summary from all the available text extracted and provide a clear context and precise idea.
Ask me anything: This feature not only extract the data, but also vectorise it for interacting with it later. So medical professionals can raise specific questions about a scenario.

Services used from Cloudflare:

Workers AI
Workers & Pages
Vectorize
R2

Models used:

@cf/qwen/qwen1.5-14b-chat-awq | LLM - For analysing and diagnosing patient data.
@cf/baai/bge-base-en-v1.5 | text-to-embedding - For text embedding.
@cf/openai/whisper | audio-to-text - For text extraction from audio.
@cf/unum/uform-gen2-qwen-500m | image-to-text - For text extraction from image.
@cf/facebook/bart-large-cnn | summarization - For text summarization

We've developed a versatile client page that can handle various data types like images, text, and audio. Using advanced Cloudflare AI models, we extract text from images and audio recordings from AI models. Simultaneously, we embed this extracted text for vectorization, which is crucial for running prompts against the large data. By merging all extracted data, we utilize the summarization AI model for getting a quick idea, on top of that we are using **@cf/qwen/qwen1.5-14b-chat-awq** AI model to generate a clear and concise diagnose, providing a comprehensive overview of the scattered data. Additionally, the vectorization process enables our "Ask Me Anything" feature, allowing users to instantly query specific information from the provided data.

Challenges Overcome:

Not able to extract relevant data from Summarization model, gives inappropriate details - @cf/facebook/bart-large-cnn.
Not able to extract text from image uploaded using @cf/unum/uform-gen2-qwen-500m model.
An alternate model for summary we used @cf/qwen/qwen1.5-14b-chat-awq model, which has max-length of content for only 4096.
Some response from summary model and LLM is returning response in chinese ⁠⁠- Upgrading to pro version for workers had payment issues

Multiple Models and/or Triple Task Types

This Project utilized totally Five Models as mentioned above.
And also we covered, Four task: