Mastering Query Answering with RAG: Overcoming Key Challenges in Large-Scale Meeting Data

In the digital age of information overload, extracting actionable insights from large datasets is more crucial than ever. Recently, I embarked on a journey to leverage Retrieval-Augmented Generation (RAG) to address a major challenge — delivering precise answers from a vast collection of meeting notes. This blog explores the obstacles, solutions, and achievements that turned my RAG-based query-answering system into a robust tool for extracting insights from unstructured meeting data.

Problem Statement: Challenges in Query Answering with RAG
One of the primary challenges was building a system capable of processing complex, intent-specific queries within a massive repository of meeting notes. Traditional RAG query-answering models frequently returned irrelevant or incomplete information, failing to capture user intent. The unstructured nature of meeting data combined with diverse query types necessitated a more refined solution.

Initial Approach: Laying the Foundation for Effective Query Answering
I started with a foundational RAG model designed to combine retrieval and response generation. Two initial techniques used were:

1. Chunking: Breaking large documents into smaller segments by sentence boundaries improved retrieval by narrowing the search scope.

2. Embedding and Vector Storage: After chunking, each segment was embedded and stored in a vector database, enabling efficient searches.

However, this setup had limitations. The initial chunking approach often led to the retrieval of irrelevant information, and generated answers lacked precision and alignment with the intent of each query.

Challenges in Large-Scale RAG Query Answering

• Handling Complex Queries: Certain complex questions required a deeper semantic understanding beyond basic semantic search.
• Contextual Mismatches: Retrieved chunks were often contextually similar but not precise enough to satisfy the query’s requirements.
• Retrieval Precision Limitations: Retrieving a small set of documents (e.g., five to ten) often resulted in limited results that lacked relevance.

These challenges underscored the need for a more advanced approach to improve accuracy in RAG query answering.

Advanced RAG Techniques for Enhanced Query Accuracy (Solution)
To address these issues, I applied several advanced methodologies, iteratively refining the system:
Semantic Chunking
Unlike traditional chunking, Semantic Chunking prioritizes meaning within each segment, enhancing relevance by aligning retrieved information with the query’s intent.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings
from langchain.schema import Document

# Initialize OpenAI Embeddings with API key
openai_api_key = ""
embedder = OpenAIEmbeddings(openai_api_key=openai_api_key)
text_splitter = SemanticChunker(embedder)

def prepare_docs_for_indexing(videos):
    all_docs = []

    for video in videos:
        video_id = video.get('video_id')
        title = video.get('video_name')
        transcript_info = video.get('details', {}).get('transcript_info', {})
        summary = video.get('details', {}).get('summary')
        created_at = transcript_info.get('created_at')  # Getting the created_at timestamp

        # Get the full transcription text
        transcription_text = transcript_info.get('transcription_text', '')

        # Create documents using semantic chunking
        docs = text_splitter.create_documents([transcription_text])

        for doc in docs:
            # Add metadata to each document
            doc.metadata = {
                "created_at": created_at,
                "title": title,
                "video_id": video_id,
                "summary": summary
            }
            all_docs.append(doc)

    return all_docs


docs = prepare_docs_for_indexing(videos)

# Output the created documents
for doc in docs:
    print("____________")
    print(doc.page_content)

Maximum Margin Retrieval
This method improved retrieval precision by differentiating between relevant and irrelevant data, ensuring that only the best-matched data chunks were retrieved.

Lambda Scoring
Using Lambda Scoring, I could rank results based on relevance, prioritizing responses that aligned more closely with query intent for better answer quality.

from langchain_community.vectorstores import OpenSearchVectorSearch
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

docsearch = OpenSearchVectorSearch.from_documents(
    docs, embeddings, opensearch_url="http://localhost:9200"
)

query = "your query"
docs = docsearch.max_marginal_relevance_search(query, k=2, fetch_k=10, lambda_param=0.25)

Multi-Query and RAG Fusion
For complex questions, the system generates multiple sub-queries. RAG Fusion then integrates diverse answers into a single, cohesive response, enhancing response quality and reducing error.

def generate_multi_queries(question: str):
    # Template to generate multiple queries
    template = """You are an AI language model assistant. Your task is to generate five 
    different versions of the given user question to retrieve relevant documents from a vector 
    database. By generating multiple perspectives on the user question, your goal is to help
    the user overcome some of the limitations of the distance-based similarity search. 
    Provide these alternative questions separated by newlines. Original question: {question}"""

    # Creating a prompt template for query generation
    prompt_perspectives = ChatPromptTemplate.from_template(template)

    # Generate the queries using ChatOpenAI and output parser
    generate_queries = (
        prompt_perspectives 
        | ChatOpenAI(temperature=0, openai_api_key=openai_api_key) 
        | StrOutputParser() 
        | (lambda x: x.split("\n"))
    )

    # Invoke the chain to generate queries
    multi_queries = generate_queries.invoke({"question": question})

    return multi_queries

def reciprocal_rank_fusion(results: list[list], k=60):
    """Applies Reciprocal Rank Fusion (RRF) to fuse ranked document lists."""
    fused_scores = {}
    for docs in results:
        for rank, doc in enumerate(docs):
            doc_str = dumps(doc)  # Convert to a serializable format
            if doc_str not in fused_scores:
                fused_scores[doc_str] = 0
            fused_scores[doc_str] += 1 / (rank + k)  # RRF formula

    # Sort documents by the fused score
    reranked_results = [
        (loads(doc), score)
        for doc, score in sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)
    ]
    return reranked_result

Enhanced Indexing and Optimized Vector Search
Improving the indexing mechanism and refining vector search parameters made retrieval faster and more accurate, especially for large datasets.

Results: Key Achievements in RAG Query Answering
Implementing these techniques led to significant improvements:

• Increased Retrieval Precision: Techniques like Semantic Chunking and Maximum Margin Retrieval refined data retrieval, ensuring that only the most relevant chunks were returned.
• Enhanced Relevance: Lambda Scoring effectively prioritized pertinent results, closely aligning responses with query intent.
• Improved Handling of Complex Queries: Multi-Query generation and RAG Fusion enabled the system to manage intricate questions, delivering comprehensive answers.
• Greater System Robustness: These refinements elevated the system from a basic model to a sophisticated, reliable query-answering tool for large-scale, unstructured meeting data.

Key Takeaways and Lessons Learned
Through this journey, I identified several core insights:

1. Adaptability is Key: Effective solutions rarely emerge on the first attempt; iterative improvement and flexibility are essential.
2. Layered Methodologies Improve Robustness: Integrating multiple approaches — Semantic Chunking, Maximum Margin Retrieval, Lambda Scoring — created a stronger, more effective system.
3. Thorough Query Handling: Multi-Query generation and RAG Fusion highlighted the importance of addressing questions from multiple perspectives.
4. Focusing on Semantics: Emphasizing meaning within data rather than structure alone improved retrieval accuracy significantly.

Conclusion: Future Prospects for RAG-Based Systems
Enhancing RAG models with advanced techniques transformed a simple retrieval system into a powerful tool for answering complex, nuanced queries. Looking forward, I aim to incorporate real-time learning capabilities, allowing the system to dynamically adapt to new data. This experience deepened my technical skills and highlighted the importance of flexibility, semantic focus, and iterative improvement in data retrieval systems.

Final Thoughts: A Guide for Implementing Advanced RAG Systems
By sharing my experience in overcoming RAG challenges, I hope to offer a guide for implementing similar solutions. Strategic techniques, combined with iterative refinement, not only resolved immediate issues but also laid a strong foundation for future advancements in query-answering systems.