Development of a Virtual Assistant Based on LLMs for the Knowledge Domain in Biomedical Metrology

Abstract

Accurate measurements are essential for effective diagnosis and treatment in the healthcare sector. However, there is limited training in biomedical metrology for Biomedical Engineers, which may hinder their performance. This study evaluated the knowledge of large language models (LLMs) in biomedical metrology to develop a specialized virtual assistant that supports these professionals. The effectiveness of the LLMs was assessed based on the accuracy and coherence of their responses using the CBET Certification exam and metrics such as Rouge-L, F1 score, and cosine similarity. The Llama 3.2-3B Mini model, optimized with retrieval- augmented generation (RAG), showed an increase in the F1 score from 0.402 to 0.526, a Rouge-L score of 0.497, and a cosine similarity of 0.657, demonstrating its ability to generate relevant and accurate responses. The developed virtual assistant represents a promising tool for improving the training and performance of biomedical engineers, ensuring access to precise and reliable information, thereby strengthening safety in the healthcare sector. Our source code is publicly available at https://github.com/yamilet2662/assistant_biome.