Development and Evaluation of a Hybrid Retrieval-Augmented Conversational Agent for Naturopathy-Based Health Guidance

Abstract

Objectives: To develop and evaluate, to the best of our knowledge, one of the first domain-specific hybrid Retrieval-Augmented Generation (RAG) frameworks for naturopathy-based health guidance, addressing the lack of reliable, structured conversational systems in this domain.
Methods: A knowledge base of 729 semantically structured chunks was derived from 42 disease chapters of The Encyclopedia of Natural Medicine (Murray and Pizzorno, ISBN: 978-1-4516-6300-6). Hybrid retrieval combined ChromaDB semantic search (nomic-embed-text, 768-dim, cosine similarity, top-6) with BM25 keyword retrieval (Rank-BM25, k1=1.5, b=0.75, top-4). A cross-encoder reranker (cross-encoder/ms-marcoMiniLM-L-6-v2) refined the merged candidate pool before context was passed to a locally deployed Llama 3.2 3B-instruct (q4_K_M) language model via Ollama. A two-layer disease detection mechanism combined a manually curated alias dictionary (10 aliases) with SentenceTransformer-based cosine similarity (all-MiniLML6-v2, threshold=0.40). The system was evaluated on 30 disease-specific queries using Precision@3, Hit@3, and Mean Reciprocal Rank (MRR).
Findings: Over 30 queries, the proposed system achieved Precision@3 of 0.644, Hit@3 of 0.767, and MRR of 0.767, compared to the baseline which recorded 0.322, 0.433, and 0.433 respectively — improvements of 100%, 77%, and 77%. All improvements were statistically significant (Wilcoxon signed-rank test: Precision@3 W=10.5, p=0.000766; Hit@3 W=6.5, p=0.001946; MRR W=6.5, p=0.001946). Mean retrieval latency was 142.3 ms; LLM generation latency averaged 3,280 ms on warm inference.
Novelty: To the best of our knowledge, this is among the first hybrid RAG systems specifically designed for naturopathy knowledge retrieval, integrating section-aware metadata filtering, dual-mode retrieval, neural reranking, and two-layer semantic disease detection — all deployed locally without cloud API dependency.