Speech to Musical Notation with AssemblyAI

This is a submission for the AssemblyAI Challenge : Sophisticated Speech-to-Text.

What I Built

I built Speech-to-Note, an innovative web application that combines speech recognition and musical note detection. The application allows users to record audio (either speech or singing) and processes it in two ways:

1. Converts spoken words into text using AssemblyAI's Speech-to-Text API
2. Analyzes the audio to detect musical notes, including their pitch, octave, and duration

The application features a modern, responsive UI built with React and TailwindCSS, and a robust backend powered by FastAPI. It's particularly useful for musicians, music teachers, and anyone interested in analyzing the musical properties of their voice or instruments.

Demo

Landing

Audio Processing

Result

Journey

AssemblyAI's Universal-2 Speech-to-Text Model was integrated into the application through their Python SDK. The implementation can be found in the upload_audio endpoint of our FastAPI backend:

1. When a user records audio, it's sent to our backend as a WAV file
2. The audio file is processed in parallel:
   • Sent to AssemblyAI's API for transcription
   • Analyzed locally using librosa for musical note detection
3. The transcribed text and detected musical notes are returned to the frontend

The AssemblyAI integration was straightforward thanks to their well-documented SDK:

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file_path)
transcribed_text = transcript.text

What makes this implementation sophisticated is the dual-processing approach:

1. Using AssemblyAI's advanced speech recognition for accurate text transcription
2. Complementing it with custom pitch detection algorithms to extract musical information
3. Providing a synchronized playback experience where users can hear the detected notes while seeing the transcribed text

This creates a unique tool that bridges the gap between spoken word and musical notation, making it valuable for various musical applications, from education to composition.

The application qualifies for additional prompts as it implements:

• Real-time audio processing
• Custom pitch detection algorithms
• Interactive audio playback
• Modern, responsive UI with TailwindCSS
• Full-stack implementation with React and FastAPI

The project demonstrates how AssemblyAI's technology can be combined with custom audio processing to create innovative applications that go beyond simple speech-to-text conversion.