Multimodal Fusion of Image and Recipe Features for Nutrient Estimation of Complementary Foods - Purwati, Nani; Isnanto, R Rizal; Irene Kartasurya, Martha (International Journal of Online and Biomedical Engineering (iJOE), 2026)
The assessment of the nutritional content of complementary foods for infants aged 6–24 months is still largely done manually, which is time-consuming and prone to errors. This study proposes an automated nutritional content prediction model based on a multimodal approach that integrates food images and recipe texts. The experiment was conducted using the ComFoodID25 dataset, which consists of 2,783 images of complementary foods, complete with information on ingredients, processing methods, and ten types of nutrients. Visual features were extracted using pre-trained ResNet50, while text features were obtained using IndoBERT, and then both modalities were combined through a multilayer perceptron (MLP) architecture. The evaluation results showed that the multimodal model produced low error values for most nutrients, with an MAE value below 1 for the majority of nutrients and an overall PMAE value of 2.55%. Additionally, the high coefficient of determination values indicates a strong correlation between the predicted and reference values. These findings suggest that the proposed multimodal approach is effective and reliable for automatically estimating the nutritional content of complementary foods and has the potential to support artificial intelligence-based complementary food monitoring and recommendation systems.