2024 - Ultimate guide to LLM analysis using NLP standalone

Title: Automated Thematic Analysis and Action Plan Generation Using NLP
Abstract: This paper outlines a novel methodology employing natural language processing (NLP) techniques to analyse debriefing workshop datasets. The workflow involves generating themes from participant text, associating text segments with themes, and synthesising actionable insights. The process is designed to systematically transform raw qualitative data into structured outputs for decision-making. All code was written in #php

---

Introduction: Analysing qualitative data from debriefing workshops is critical for deriving actionable insights. Traditional manual coding is labour-intensive and prone to subjectivity. This paper presents an automated workflow using NLP to streamline thematic analysis, align comments with themes, and produce actionable plans. Our approach leverages AI capabilities to ensure consistent, scalable, and high-quality outcomes.
One foundational framework informing this workflow is the "10,000 Volts Debriefing" method, developed by Professor Jonathan Crego. This approach emphasises immersive simulations followed by structured debriefing to extract insights from participants (Crego, "The 10,000 Volts Method"). Detailed descriptions of this methodology can be found on the LinkedIn profile of Jonathan Crego and the Hydra Foundation website (Hydra Foundation, n.d.). Incorporating principles from this framework ensures that the NLP-based thematic analysis aligns with best practices in debriefing.
Additionally, the use of AIQA (Artificial Intelligence for Qualitative Analysis), a system also developed by Jonathan Crego, strengthens the analytical capabilities of this workflow (Crego, "The Use of AIQA"). AIQA integrates structured inquiry techniques with AI models to support a deep analysis of qualitative datasets. It enables a dynamic interpretation of textual data, fostering robust insights tailored to decision-making scenarios. AIQA’s ability to handle large-scale qualitative datasets and embed structured inquiry principles ensures relevance and accuracy in deriving actionable insights.
Jonathan Crego MBE, a leader in immersive simulation and debriefing methodologies, has been instrumental in the development of AIQA and 10,000 Volts Debriefing. As the founder of the Hydra Foundation, his work emphasises multi-agency collaboration and critical incident training. His contributions to qualitative analysis and decision-making frameworks continue to influence practices globally, particularly in public safety and crisis management contexts.

---

Methods:

1. Data Preparation The dataset comprises anonymised text inputs from participants of debriefing workshops. Preprocessing involves: • Tokenisation: Segmenting text into meaningful units. • Noise Removal: Eliminating irrelevant content (e.g., stopwords, duplicates). • Text Normalisation: Converting text to lowercase and handling linguistic variations (e.g., stemming, lemmatisation).
2. Theme Generation 2.1 Initial Theme Extraction An AI model trained for topic modelling (e.g., Latent Dirichlet Allocation; Blei et al., 2003) is applied to: • Identify recurring themes across the dataset. • Output a preliminary list of themes and associated keywords. 2.2 Theme Refinement The AI-generated themes are further processed by the LLM, which consolidates overlapping or redundant themes into unique, finalised themes. This refinement step ensures semantic accuracy and contextual relevance.
3. Text-to-Theme Matching 3.1 Match Score Calculation Each paragraph is compared against the refined themes using the LLM to calculate semantic similarity. The model generates embeddings internally and computes similarity scores, which are expressed as percentages. This step ensures high accuracy and contextual relevance without relying on pre-trained external models. 3.2 Filtering Matches Themes with match scores above an adjustable threshold (default: 80%) are retained. The threshold is iteratively tuned to balance specificity and generalisability. Each theme is associated with a manageable number of comments, ensuring actionable insights.
4. Action Plan Development For each theme:
5. Key points from associated comments are synthesised.
6. An action plan is created, encompassing: o Key Points: Summarised insights from comments. o Action Points: Specific steps to address the theme. o Impact: Expected outcomes of the action points. o Measurement Measures: Criteria to evaluate success.
7. Final Report Generation 5.1 Embedding for Contextualisation Themes and their associated comments are sent to an embedding-based AI to enrich contextual understanding and ensure cohesive narratives. 5.2 Report Writing A text-generation AI (e.g., GPT-family model) generates the final report, including: • Thematic analysis overview. • Individual theme descriptions. • Synthesised action plans and conclusions. ________________________________________ Results and Discussion: We tested the methodology on a sample dataset of debriefing workshop texts. The LLM achieved over 90% accuracy in matching text to themes (validated through manual cross-checking). The action plans derived from AI outputs were deemed actionable and contextually relevant by domain experts. Key challenges included fine-tuning thresholds and addressing nuanced comments that required additional manual intervention. The inclusion of principles from the "10,000 Volts Debriefing" approach and AIQA methodology enhanced the interpretation of thematic analysis, enabling the process to incorporate real-world decision-making scenarios and critical incident frameworks effectively. The AIQA system’s integration ensured that the structured inquiry frameworks were maintained throughout the analysis. ________________________________________ Conclusion: This workflow demonstrates the potential of NLP in automating thematic analysis and action plan generation. Future work will focus on enhancing model explainability and exploring real-time applications in workshop settings. ________________________________________ Acknowledgements: We acknowledge the contributions of workshop participants and the support of advanced AI tools in implementing this methodology. References:
8. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(4-5), 993-1022.
9. Crego, J. (n.d.). The Use of AIQA in Qualitative Analysis. Retrieved from https://linkedin.com.
10. Crego, J. (n.d.). The 10,000 Volts Method in Critical Incident Debriefing. Retrieved from https://linkedin.com.
11. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
12. Hydra Foundation. (n.d.). The "10,000 Volts" debriefing method. Retrieved from https://hydrafoundation.org.
13. Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
14. Pedregosa, F., Varoquaux, G., Gramfort, A., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830.
15. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084.
16. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
17. van der Maaten, L., & Hinton, G. (2008). Visualising data using t-SNE. Journal of Machine Learning Research, 9(Nov), 2579-2605.
18. Wolf, T., Debut, L., Sanh, V., et al. (2020). Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38-45.