FitCoach: A Multimodal AI-Powered Personal Fitness Coaching System using Human Pose Estimation, Nutrition Intelligence, and Large Language Models

FitCoach: A Multimodal AI-Powered Personal Fitness Coaching System using Human Pose Estimation, Nutrition Intelligence, and Large Language Models

Abhinav Kumar

Atria Institute of Technology

Aamir Hamza

Atria Institute of Technology

Aditi Jha

Atria Institute of Technology

Ayan Abdul

Atria Institute of Technology

C E Chandana

Assistant Prof.,

Atria Institute of Technology

Abstract – Maintaining a consistent and effective fitness routine requires more than simply following predefined exercise plans, as users often struggle with incorrect exercise form, inadequate personalization, nutritional decision-making, and the lack of continuous feedback. Recent advances in computer vision, deep learning, pose estimation, and artificial intelligence have enabled automated systems for exercise recognition and form assessment. However, many existing fitness applications address these capabilities independently and provide limited integration between exercise monitoring, nutritional analysis, and personalized coaching. This paper presents FitCoach, an AI-assisted fitness coaching system designed to provide an integrated and personalized fitness experience using computer vision and intelligent decision-making. The proposed system utilizes human pose estimation to identify body landmarks and analyze exercise movements, enabling real-time assessment of exercise form and feedback. In addition, the system incorporates food recognition and nutritional information to support dietary monitoring, while user-specific information is considered to generate more personalized fitness recommendations. The system is designed to combine multimodal inputs, including visual exercise information, user activity, and nutritional data, into a unified coaching framework. By integrating exercise-form analysis, dietary awareness, and personalized guidance, FitCoach aims to overcome limitations of isolated fitness technologies and provide users with accessible, adaptive, and intelligent fitness support. The proposed framework demonstrates how computer vision and AI-based

personalization can be combined to develop a more comprehensive digital fitness coaching solution.

Index Terms

human pose estimation, fitness, nutrition analysis, large language models, multimodal AI

1. INTRODUCTION

   Regular physical activity and appropriate nutrition are important components of maintaining physical fitness and overall health. However, individuals following fitness routines often face difficulties in maintaining correct exercise form, selecting appropriate exercises, monitoring their dietary intake, and receiving continuous personalized guidance. The increasing availability of artificial intelligence (AI) and computer vision technologies has created opportunities to address these challenges through intelligent digital fitness systems. Computer vision-based human pose estimation has become an important foundation for analyzing human movement. DeepPose demonstrated the application of deep convolutional neural networks to human pose estimation, establishing an important direction for learning body-joint representations directly from visual data [2]. Subsequently, Part Affinity Fields enabled real-time multi-person two-dimensional pose estimation by associating body parts with corresponding individuals [3]. Further work on simple baseline architectures demonstrated that relatively straightforward deep-learning architectures can provide effective solutions for human pose estimation

   and tracking [4]. These developments have made pose-based analysis increasingly suitable for applications involving human movement and physical activity. Real-time perception frameworks have further increased the practicality of computer-vision-based fitness applications. MediaPipe introduced a framework for building perception pipelines capable of processing different types of visual information efficiently [6]. Object-detection approaches such as YOLO also established the feasibility of performing visual detection in real-time, making them relevant to applications where rapid analysis is required [8]. More recent implementations such as YOLOv8 provide a modern object-detection framework that can be incorporated into computer-vision applications [10]. The application of these technologies specifically to fitness has also received research attention. Pose-based fitness recognition has been investigated using convolutional neural networks and human keypoint detection to identify exercise-related movements [16]. Deep-learning-based approaches have additionally been explored for real-time correction of exercise form, demonstrating the relevance of automated visual analysis for fitness training [17]. These studies indicate that body-pose information can provide a foundation for evaluating exercise movements without requiring specialized motion-capture equipment. However, exercise analysis represents only one aspect of an effective fitness-coaching system. Nutrition is another important component of fitness management, and computer vision has been explored for recognizing food items from images [18]. Object-detection-based approaches have also been investigated for food detection and dietary monitoring, demonstrating the potential of visual recognition systems to support nutrition-related applications [19]. These developments create an opportunity to connect exercise monitoring with dietary awareness within a single fitness platform. Beyond exercise and nutrition analysis, effective fitness support also requires consideration of user motivation, engagement, and behavioral change. Self-determination theory provides a framework for understanding factors associated with motivation and goal-directed behavior [23], while the stages-of-change model provides a perspective on how individuals progress through behavioral change [24]. The Behaviour Change Wheel further provides a structured approach for designing interventions intended to influence health-related behaviors [26]. Research on digital interventions has also examined the effectiveness of technology-based approaches for fitness and health-related activities [28]. Personalization is therefore an important consideration when designing intelligent fitness systems. Research has specifically

   investigated deep learning for personalized fitness coaching, indicating the growing role of AI in adapting fitness-related support to individual users [45]. Mobile technology has also been studied as a means of delivering personalized fitness coaching and influencing user engagement and behavioral outcomes [53]. Interactive personalized fitness interventions have further explored the relationship between digital engagement and behavior change [54].The development of multimodal machine-learning techniques provides another important direction for intelligent fitness applications. Multimodal machine learning focuses on combining information from multiple sources or modalities to support more robust intelligent systems [30]. In the context of fitness, this provides a foundation for integrating information such as visual exercise analysis, user-specific information, and nutritional data rather than treating each component as an isolated application.Despite these developments, existing research commonly focuses on individual aspects of digital fitness, such as pose estimation, exercise-form recognition, food recognition, behavioral intervention, or personalized coaching. The integration of these capabilities into a unified fitness-coaching framework remains an important area for development. This motivates the design of FitCoach, an AI-assisted fitness coaching system that combines computer-vision-based exercise analysis with nutritional awareness and personalized fitness guidance. FitCoach is designed to use visual information to analyze exercise movements and provide feedback on exercise form while also incorporating food-related information to support dietary monitoring. The system further considers user-specific information to provide more personalized fitness recommendations. By bringing exercise analysis, nutrition awareness, and personalized coaching together, FitCoach aims to provide a more comprehensive digital fitness experience than systems that address these components independently. The objective of this work is therefore to develop and evaluate an integrated AI-based fitness-coaching framework that can assist users with exercise monitoring, dietary awareness, and personalized guidance. The proposed approach builds upon established research in human pose estimation, real-time computer vision, food recognition, behavioral science, and personalized digital fitness interventions while combining these areas into a unified application-oriented framework.

2. RELATED WORK

   Research in AI-assisted fitness has developed across several interconnected areas, including human pose estimation, exercise-form analysis, food recognition, personalized coaching, behavioral change, and multimodal machine

   learning. These areas provide the foundation for the development of integrated intelligent fitness systems such as FitCoach.

   1. Human Pose Estimation and Computer Vision Human pose estimation is a fundamental component of computer-vision-based exercise analysis because it enables the identification of body joints and their spatial relationships during movement. Toshev and Szegedy introduced DeepPose, demonstrating the use of deep convolutional neural networks for human pose estimation [2]. Cao et al. proposed Part Affinity Fields for real-time multi-person 2D pose estimation, providing an approach for associating detected body parts with individual people [3]. Xiao et al. further investigated simple baseline architectures for human pose estimation and tracking, demonstrating the effectiveness of deep-learning-based approaches for this task [4]. MediaPipe provides a framework for constructing perception pipelines that can process visual information efficiently and is particularly relevant to real-time applications [6]. Object-detection approaches such as YOLO established the possibility of performing detection at high speed, which is valuable for applications requiring immediate visual feedback [8]. The YOLOv8 framework further represents a modern object-detection approach that can be incorporated into computer-vision applications [10]. These developments establish the technical foundation required for analyzing human movement through visual input. FitCoach builds upon this area by applying pose and computer-vision techniques to fitness-related movement analysis.

   2. Exercise Recognition and Form Correction

      The application of pose estimation to fitness has been explored specifically for recognizing exercise movements. Wang et al. investigated exercise-related pose recognition using convolutional neural networks and human keypoint detection, demonstrating the applicability of pose-based approaches to fitness activities [16]. Kwon et al. investigated deep-learning-based real-time exercise-form correction, demonstrating the potential of automated visual analysis for providing feedback during fitness training [17]. Although these approaches provide important foundations for automated exercise analysis, FitCoach aims to incorporate exercise-form assessment into a broader coaching framework. Instead of treating exercise recognition as an isolated computer-vision task, the proposed system combines movement analysis with user-specific fitness guidance and nutritional awareness.

   3. Food Recognition and Dietary Monitoring

      Computer vision has also been applied to nutrition-related applications. Kawano and Yanai investigated food image recognition using deep convolutional networks, establishing the applicability of deep-learning methods to visual food analysis [18]. He et al. explored YOLO-based food detection for dietary monitoring, demonstrating the potential of object detection for identifying food items in nutrition-related applications [19].These studies provide the

      basis for incorporating visual food analysis into FitCoach. The proposed system extends this concept by considering dietary information as part of a broader fitness-coaching workflow rather than treating food recognition as an independent application.

   4. Personalization and Behavioural Change

      Fitness applications require more than technical recognition of exercises or food items because sustained participation is influenced by motivation and behavioral factors. Deci and Ryan's self-determination theory provides a theoretical foundation for understanding motivation and goal-directed behavior [23]. Prochaska and DiClemente proposed a stages-of-change model describing behavioral change as a progression through different stages [24]. Michie et al. introduced the Behaviour Change Wheel as a structured framework for designing interventions intended to influence behavior [26]. Digital fitness interventions have also been investigated as technology-based approaches for supporting health and fitness-related behavior [28]. These studies highlight the importance of designing fitness systems that consider user engagement and behavioral factors in addition to technical functionality.

   5. Personalized Fitness Coaching

      Recent research has increasingly explored the use of artificial intelligence and mobile technologies for personalized fitness support. Suárez-Cuenca et al. examined deep learning approaches for personalized fitness coaching and highlighted the growing application of AI in this area [45]. Ormsbee et al. investigated personalized fitness coaching through mobile technology and its relationship with user engagement and behavioral outcomes [53]. Vandelanotte et al. examined interactive engagement with web-based personalized fitness interventions in the context of behavior change [54]. These studies demonstrate the relevance of personalization and user engagement in digital fitness systems. FitCoach incorporates this perspective by considering individual user information when providing fitness-related recommendations rather than relying solely on generic exercise guidance.

   6. Multimodal Fitness Systems

      Multimodal machine learning provides a broader framework for combining information from multiple sources within an intelligent system. Baltruaitis et al. presented a survey and taxonomy of multimodal machine learning, describing approaches for integrating information from different modalities [30]. This concept is particularly relevant to fitness applications where visual exercise information, dietary information, and user-specific data can contribute to a more comprehensive understanding of an individual's fitness context. FitCoach applies this integrated perspective by combining exercise analysis, nutritional awareness, and personalized guidance within a unified fitness-coaching framework. Therefore, the proposed system builds upon existing research across computer vision, exercise analysis, nutrition recognition, behavioral

      science, personalization, and multimodal AI rather than relying on a single technological approach.

   7. Summary of Existing Research

   The reviewed literature demonstrates significant progress in individual components of AI-assisted fitness. Human pose estimation and real-time computer vision provide mechanisms for analyzing physical movements [2], [3], [4], [6]. Exercise-specific research demonstrates the applicability of these techniques to exrcise recognition and form correction [16], [17]. Food recognition and object detection provide approaches for visual dietary monitoring [18], [19].

   Research in behavioral change and personalized digital coaching emphasizes the importance of motivation, engagement, and individualized guidance [23], [24], [26],

   [28], [45], [53], [54]. Multimodal machine learning provides a foundation for integrating information from different sources [30]. However, these research areas are frequently investigated as separate components. FitCoach addresses this opportunity by bringing exercise-form analysis, dietary awareness, and personalized fitness guidance together within a single AI-assisted framework. This integration forms the basis for the proposed system and motivates the methodology presented in the following sections.

3. RESEARCH GAP AND PROBLEM DEFINITION

   1. Research Gap

      The existing literature demonstrates substantial progress in individual components of AI-assisted fitness systems. Human pose estimation has enabled the identification and tracking of body movements, while real-time computer-vision frameworks have made visual analysis increasingly practical for interactive applications [2], [3], [4], [6]. Research has also explored the application of these techniques to exercise recognition and automated form correction [16], [17]. Similarly, computer-vision-based food recognition and detection have provided approaches for identifying food items and supporting dietary monitoring [18], [19]. Research in digital fitness interventions and personalized coaching has further emphasized the importance of user engagement, behavioral change, and individualized guidance [23], [24], [26], [28], [45], [53], [54]. Despite these developments, the reviewed approaches primarily address individual aspects of fitness management. Exercise analysis, food recognition, behavioral support, and personalized coaching are often treated as separate components rather than as interconnected parts of a single fitness-coaching system. This creates an opportunity for an integrated framework capable of considering multiple aspects of a user's fitness journey. Another limitation is the separation between visual exercise analysis and personalized decision-making. Pose estimation can provide information about body movement, while nutritional recognition can provide information about food, but these outputs need to be interpreted in the context of an individual user's goals and requirements. Multimodal

      Machine learning provides a foundation for combining information from different sources, but its application to an integrated fitness-coaching workflow remains an area that can be further explored [30]. Therefore, the primary research gap addressed by this work is the lack of an integrated AI-assisted fitness framework that combines exercise-form analysis, nutritional awareness, and personalized fitness guidance within a unified system.

   2. Problem Definition

      Many individuals following fitness routines depend on generic exercise instructions and manually track aspects of their diet and progress. Such approaches may provide limited real-time feedback regarding exercise execution and may not adequately adapt recommendations to individual users.

      The problem addressed in this research is therefore defined as follows:

      To design and develop an AI-assisted fitness coaching system capable of analyzing exercise movements, supporting dietary awareness, and generating personalized fitness guidance through the integration of computer vision, machine learning, and user-specific information.

      The proposed system, FitCoach, is designed to address this problem by integrating multiple fitness-related functionalities within a unified framework. The system uses visual analysis for exercise-related movement assessment, incorporates food-related information for dietary monitoring, and considers user-specific information when providing fitness recommendations.

      The proposed approach is motivated by the need to move from isolated AI-based fitness functions toward a more comprehensive coaching system. By combining multiple sources of fitness-related information, FitCoach aims to provide users with a more personalized and interactive approach to digital fitness management.

   3. Objectives

      The major objectives of the proposed FitCoach system are:

      1. To develop a computer-vision-based mechanism for analyzing exercise movements and supporting exercise-form assessment.

      2. To incorporate food recognition and dietary information into the fitness-coaching workflow.

      3. To provide personalized fitness recommendations based on user-specific information and fitness requirements.

      4. To integrate exercise, nutrition, and personalized guidance within a unified AI-assisted framework.

      5. To develop a system that can provide interactive fitness support without requiring specialized fitness-monitoring equipment.

      6. To evaluate the effectiveness of the proposed system using appropriate performance and user-oriented evaluation measures.

4. THE FITCOACH SYSTEM WE ARE PROPOSING

   1. What FitCoach Is All About

      FitCoach represents a comprehensive, AI-driven fitness coaching platform designed to optimize health and wellness routines. The system employs an integrated approach, leveraging multimodal artificial intelligence and computer vision to synthesize data regarding exercise performance, dietary habits, and user-specific physiological profiles into personalized coaching strategies. Unlike conventional fitness applications that address exercise and nutrition in isolation, FitCoach provides a unified framework where these components are interconnected, ensuring that guidance remains holistic rather than fragmented.

      Fig. 1. Overview of the FitCoach AI platform and its integrated functional modules.

   2. How The FitCoach System Works

      The FitCoach architecture is comprised of five core functional modules: a user data acquisition interface, a computer-vision-based exercise analysis engine, an image-processing nutritional recognition unit, a centralized user profile management system, and an AI-driven personalized recommendation generator. The data acquisition interface serves as the primary gateway for users to input multidimensional variables, including exercise logs, dietary entries, and specific wellness objectives. This telemetry is then processed by specialized engines to ensure that the resulting coaching interventions are both evidence-based and tailored to the individual.

      Fig 2. System architecture of the proposed FitCoach AI application.

   3. The Exercise Analysis Part Of FitCoach and Progress Tracking Module

      The exercise analysis module is central to the platforms biomechanical assessment capabilities. It utilizes computer vision algorithms to perform real-time skeletal landmark identification and kinematic analysis. By processing input streams to evaluate exercise form, the system identifies anatomical keypoints and calculates joint angles to ensure execution conforms to biomechanically sound patterns. This immediate feedback loop facilitates rep counting and form validation, minimizing injury risk while maximizing physiological recruitment.

      Fig. 3. FitCoach progress tracking interface for monitoring workout activity and exercise performance.

   4. The Nutrition Analysis Part Of FitCoach

      The nutrition analysis module facilitates dietary monitoring through advanced object-detection algorithms and convolutional neural networks. By analyzing visual data from food imagery, the system lassifies food items and estimates caloric and macronutrient density. This automated inference of food content, combined with nutritional database lookups, provides users with evidence-based dietary insights that are systematically aligned with their broader fitness objectives and metabolic requirements.

      Fig. 4. FitCoach Nutrition Tracker.

   5. The Personalization Part Of FitCoach

      The personalization module serves as the intelligence core, synthesizing inputs from the exercise, nutrition, and profile management components. Utilizing advanced algorithms, it tailors coaching recommendations to individual user needs, ensuring that advice on fitness progression, recovery, and dietary adjustments remains context-aware and goal-oriented. By applying multimodal AI integration, the system achieves a level of personalization that exceeds the individual contributions of any single data modality.

      Fig. 5. AI Coach interface providing personalized fitness guidance and recommendations.

   6. How FitCoach Gives People Feedback

      The systems feedback mechanism aggregates multidimensional data to generate comprehensive wellness insights and actionable intelligence. By delivering structured guidance on exercise execution, nutritional planning, and recovery states, the feedback loop optimizes long-term health outcomes. This synthesis of recovery signalssuch as sleep quality and perceived fatigueallows FitCoach to provide highly specific instructions tailored to the user's current physiological state, fostering sustained engagement and progress.

      Deep-learning-based food recognition and object-detection approaches have previously been investigated for food identification and dietary monitoring [18], [19].

      E. AI-Based Personalization and Coaching

      The processed user, exercise, and nutrition information can be used to provide personalized fitness guidance. The AI Coach interface shown in Fig. 5 provides an interactive mechanism through which users can receive fitness-related guidance and recommendations.

      The integration of information from multiple sources follows the principles of multimodal machine learning [30]. Personalized fitness coaching and interactive digital

   7. What FitCoach Is About

      In summary, FitCoach represents a paradigm shift from isolated fitness tools to an integrated ecosystem. By harmonizing computer vision, machine learning, and individualized user profiles, the system delivers a cohesive coaching experience grounded in academic research and technical precision.

5. METHODOLOGY

1. Methodology Overview

   The FitCoach framework utilizes a modular methodology that seamlessly integrates user inputs, exercise tracking, nutritional analysis, longitudinal progress monitoring, and AI-based coaching. The overall application and its major functional modules are presented in Fig. 1, while the underlying interaction between the frontend, FastAPI backend, AI modules, database, and output components is illustrated in Fig. 2.

2. User Data and Fitness Input

   The system begins by collecting user-specific information and fitness-related inputs. User goals, preferences, and activity information are used to support personalized fitness guidance. The application provides centralized access to its major modules through the dashboard shown in Fig. 1.

   Personalized digital fitness coaching has been explored as an approach for adapting fitness support to individual users [45], [53], [54].

3. Exercise and Progress Monitoring

   FitCoach records workout-related information and presents it through the progress monitoring module. The progress interface shown in Fig. 3 provides information such as workout count, workout duration, current streak, weight, activity, and personal records.

   Computer-vision-based pose estimation and exercise analysis provide the foundation for automated fitness movement assessment [2], [3], [4], [16], [17]. The collected exercise and activity information can subsequently contribute to the user's overall fitness profile.

4. Nutrition Analysis

The nutrition module allows users to provide food-related information and obtain corresponding nutritional insights. Fig. 4 presents the FitCoach Nutrition Tracker, which provides a visual interface for tracking meals and displaying information associated with the detected food.

The progress part shows workout and activity details. These details include how workouts were run, how long they

interventions have also been investigated for improving the relevance and engagement of fitness support [28], [45], [53], [54].

were, how many in a row and personal bests. This is shown in Fig. 3. The Nutrition Tracker shows meal details. This is shown in Fig. 4. The AI Coach gives help with fitness. This is shown in Fig. 5.