AI Enabled Student Dropout Prediction and Counseling Management System

AI Enabled Student Dropout Prediction and Counseling Management System

Hemanth Kumar. T

Student, B. Tech IoT 4th Year Holy Mary Inst. Of Tech. and Science Hyderabad, Telangana, India

Bhavani Reddy. D

Student, B. Tech IoT 4th Year Holy Mary Inst. Of Tech. and Science Hyderabad, Telangana, India

Sai Teja Reddy. D

Student, B. Tech IoT 4th Year Holy Mary Inst. Of Tech. and Science Hyderabad, Telangana, India

ABSTRACT – Student dropout remains as a significant challenge for educational institutions, affecting institutional performance, financial stability, and student career outcomes. Traditional retention mechanisms are largely reactive and depend on manual monitoring resulting in delayed interventions. This paper proposes an AI enabled system for early student dropout prediction integrated with an intelligent counseling management framework. The proposed system leverages supervised machine learning algorithms to analyze academic, demographic, attendance, and behavioral data to identify at -risk students into risk levels and automatically recommends counseling interventions. Multiple classification models are evaluated using Accuracy, precision, Recall, F1-score, and ROC_AUC metrics experimental results demonstrate improved early- risk detection and optimized counseling workflow management. The system enhances proactive student retention strategies and supports data-driven institutional decision making. The result in higher retention rates happier students and a more efficient use of resources by adopting this approach institutions can shift from reactive to proactive support addressing issues before they escalate ensuring system is both effective and secure

Keywords: Artificial Intelligence, Student Dropout

Vinay Reddy. Y Student, B. Tech IoT 4th Year

Holy Mary Inst. Of Tech. and Science Hyderabad, Telangana, India

Devi. G Asst. prof, CSE

Holy Mary Inst. Of Tech. and Science Hyderabad, Telangana, India

Dr. Venkataramana. B Assoc. prof, CSE

Holy Mary Inst. Of Tech. and Science Hyderabad, Telangana, India

Prediction, Counselling Management, Early warning system, Education data mining, Machine learning

1. INTRODUCTION

   Student dropout remains a critical global issue, costing institutions billons annually in lost tuition and long term funding impacts. Beyond the financial implications, it represents a substantial societal burden through decreased workforce skills and unrealized potential. The multifaceted reasons for dropping out ranging from severe academic decline and poor attendance to personal stress, lack of financial aid and socio- economic disadvantage are often interconnected and subtle, making manual detection is nearly impossible. Traditional methods of identifying at risk students, such as simple attendance reports or semester- end grades are fundamentally reactive. By the time a student triggers a red flag in such a legacy system, the crucial window for effective, curative intervention has typically passed forcing administrators into damage control rather than prevention

   The approach too institutional retention has undergone a significant paradigm shift, evolving from manual oversight to data-driven intelligence. The advent of Artificial Intelligence (AI) and machine learning (ML) provide robust and necessary alternative predictive analytics utilizing complex non linear algorithms like gradient boosting, can process thousands of data points

   concurrently. This capability allows systems to identify subtle, complex and non-linear patterns- such as the combination of declining social engagement and high backlogs count that precede dropout events. This project proposes the development of an AI- driven student dropout prediction and counselling system

   1.2 Problem statement

   Educational institutions face a significant challenge in retaining student, with high dropout rates leading to financial losses for the institution and reduced career prospects for students. The primary issue is the lack of a proactive mechanism to identify at- risk students early in their academic journey. Existing methods often rely on manual observation or post- semester analysis, which is too late for effective intervention. Furthermore, even predictive models exist, they often lack interoperability, making it difficult for educators to understand the root causes of a students risk and tailor specific counseling strategies effectively. There is a need for an automated, explainable AI system that not only predicts dropout risk but also provides actionable insights to mentors and parents

2. Literature review

   The evolution of Educational Data Mining (EDM) has shifted from simple descriptive statistics to sophisticated predictive modelling, aiming to address the global challenge of student attrition. Early foundational research established the viability of applying data mining techniques to e-learning environments, effectively proving that digital footprints could be used to understand student behaviour.

   1. Machine Learning for Dropout Prediction

      As the field matured, the limitations of single- algorithm classifiers became apparent, particularly when dealing with the imbalanced nature of academic datasets where dropouts represent a minority class. The emergence of ensemble methods demonstrated that boosted decision tree ensembles consistently outperform traditional single classifiers in these contexts. Specifically, the XGBoost algorithm has become a benchmark for structured tabular data due to its implementation of gradient boosting with advanced regularization and hardware optimization. Multiple comparative studies have since validated that XGBoost offers superior predictive power over Random Forest, Support Vector Machines (SVM), and Logistic Regression when applied to academic performance metrics. [16][17][18]

   2. Explainable AI (XAI) for Dropout Prediction

      Despite the high accuracy of models like XGBoost, their adoption in academic settings has been hindered by their “black box” nature. In high-stakes domains such as education, healthcare, and justice, the inability to interpret how a model arrives at a decision is a significant barrier to trust and actionability. To bridge this gap, the SHapley Additive exPlanations (SHAP) framework, based on cooperative game theory, has been introduced to provide per-feature importance values. Recent applications of SHAP in educational contexts have successfully identified specific behaviorssuch as attendance irregularity and assignment completionas the primary drivers of student risk, allowing for more transparent and ethical AI- assisted decision- making [2][9]

   3. Gaps in Existing Research

      A review of current literature reveals a significant “deployment gap” between theoretical research and practical implementation. Most existing models are designed for static, batch-style analysis on historical datasets rather than real-time institutional use. Furthermore, there is a lack of integrated systems that combine high- performance prediction with automated, explanation-based counseling recommendation This work addresses these gaps by delivering a full-stack, real- time application that not only predicts risk but also utilizes SHAP values to generate personalized intervention strategies for mentors and parents.

      Gap	This Work’s Contribution
      Prediction withou explanation	SHAP-based per-student counseling recommendations
      Research models with no deployment	Production Fast API + React.js full-stack system
      Static batch analysis	Real-time /predict API with live inference
      No historical tracking	SQLite database with timestamped prediction history
      Single-role access	Role-based dashboards for Mentors and Parents

      Table 1: Gaps Addressed by the Proposed System

3. Methodolgy and System Architecture

   The methodolgy of the proposed system in designed as a structured pipeline that transitions from raw data acquisition to real-time, actionable intervention recommendations. The process is divided into three

   core phases: data engineering, predictive modeling via XGBoost, and the explainability layer powered by SHAP.

   1. Data Acquisition and Feature Engineering

      Due to the sensitive nature of Educational Data Mining (EDM), this study utilizes a controlled synthetic dataset of $N=1,000$ instances. This ensures a “Ground Truth” for evaluating model sensitivity without ethical constraints . Feature Characterization: Four primary predictive dimensions were modeled to reflect academic engagement and socio-economic stability

      • Academic Standing (): Continuous GPA

        variable (2.0 4.0).

      • Engagement Metric (): Attendance rate (50%

        100%)

      • Behavioral Indicator (): Discrete count of missed sessions (0 20 ).
      • Socio-economic Factor (): Binary indicator for financial assistance ( {, }). Target Labeling Strategy the objective function targets a binary classification y {, }). where y=1denotes high dropout risk. Labels were assigned based on a composite threshold: ( < %) ( < . ), otherwise y=0 preprocessing Continuous variables underwent Min-Max scaling to normalize continous variables into a [0,1] range,ensuring gradient descent stability

        The core predictive engine leverages Extreme Gradient Boosting (XG Boost), a scalable end-to- end tree boosting system. XG Boost was selected for its proven efficacy in handling tabular datasets and its robust built-in regularization (L1 and L2) which mitigates overfitting in small-to-medium scale datasets.

        Mathematically, the model minimizes a regularized objective function:

        () = (, )+ ()

        Where is a differentiable convex loss function (Logarithmic Loss) measuring the difference between the prediction () and the target yi Then term penalizes the complexity of the regression trees to ensure simplicity:

        Here, is the number of leaves in the tree and is the leaf weights. The inclusion of sparsity-aware learning algorithms allows the system to remain robust even in the presence of incomplete student records.

        1. 1. Interpretability Layer: SHAP

              Feature	Type	Range	Description
              Attendance rate	Continuous	50 100%	Percentage of classes attended
              GPA	Continuous	2.0 4.0	Cumulative Grade Point Average
              Missed classes	Discrete	020	Count of missed class sessions
              Financial aid	Binary	{0, 1}	Receipt of financial assistance

               

              To address the “Black Box” limitation inherent in complex ensemble models, this study integrates SHAP (Shapley Additive explanations). Based on cooperative game theory, SHAP provides a unified measure of feature importance by calculating the marginal contribution of each feature across all possible combinations. For local explanations, the system calculates the Shapley value for each feature, such that the sum of the contributions equals the difference between the actual prediction and the average prediction. This allows the system to provide granular, student-specific insights. Furthermore, by aggregating these values, the research identifies Global Feature Importance, offering administrators macro-level insights into the primary drivers of student attrition across the institution.

              Table 2: Input Features

   2. Machine Learning Framework: XG Boost

      1. 1. System Architecture and Implementation Framework

            The proposed system is engineered as a decoupled, multi-tier architecture designed to ensure high availability, computational scalability, and the seamless integration of machine learning (ML) components within a real-time web ecosystem. As illustrated in the application logic flow the system implementation follows a modular design that isolates the computationally intensive inference engine from the client-side presentation layer

            Fig. 1: system architecture of dropout and prediction system

            1. Sequential Data Flow

               The logical progression of data within the system is architected as a five-stage deterministic pipeline. The cycle originates at the Frontend Dashboard, where student performance metricsprimarily GPA and attendance ratesare ingested and encapsulated into a JSON payload for transmission. This payload is received by the FastAPI backend, which performs asynchronous data validation via Pydantic models to ensure type integrity. Upon validation, the features are routed to the Inference Engine for the calculation of the dropout risk probability using a pre-trained XGBoost classifier. Simultaneously, the SHAP Explainer performs feature attribution to calculate marginal contribution scores. The process concludes as the backend returns the prediction and SHAP values to the dashboard for high-fidelity visualization and immediate counseling intervention.

            2. Component Interaction and Decoupling

               The system architecture utilizes a decoupled full-stack design to maintain a strict separation of concerns between the presentation, logic, and persistence layer .

               Presentation and Logic Interaction: The React.js frontend interacts with the FastAPI backend through asynchronous HTTP requests, allowing the UI to remain responsive while the

               server processes computationally intensive ML models.

               Inference and Interpretability Synergy: Within the backend environment, the XGBoost model and the SHAP explainer operate in tandem. The SHAP explainer specifically utilizes a TreeExplainer optimized for the XGBoost architecture to provide local interpretability for every individual prediction.

               Logic and Persistence Connectivity: The backend communicates with the SQLite database through the SQLAlchemy Object Relational Mapper (ORM). This interaction ensures that every inferencecomprising the student features, the risk probability, and the SHAP-based explanationis logged with a high-precision timestamp to enable longitudinal progress tracking.

            3. Data Persistence and Backend Orchestration

               The backend acts as the central orchestrator, managing both live inference and longitudinal data integrity. The system utilizes SQLite managed via the SQLAlchemy ORM to maintain a normalized relational schema. This persistence layer is structured across primary tablesStudents, Parents, and Predictions linked via foreign keys to enable comprehensive student tracking. By archiving every inference witha high-precision timestamp and its corresponding SHAP vector, the architecture allows for a “Risk Trajectory” analysis, enabling academic researchers to observe the evolution of a student’s risk profile over an entire semester.

            4. Training Objectives

      The optimization of the predictive framework is governed by a dual mandate of statistical rigor and operational utility within the educational domain. The training phase is systematically structured to ensure that the resulting early warning system maintains high predictive accuracy while remaining interpretable and robust against common data irregularities. The primary statistical objective involves the minimization of Logarithmic Loss (logloss) via a binary:logistic objective function, which facilitates the generation of calibrated probability scores (y) This granular output allows academic mentors to distinguish between varying degrees of student risk rather than relying on a reductive binary classification.

      The system also integrates Sparsity-Aware Learning to maintain functionality despite incomplete student records. By leveraging XGBoosts native sparsity-aware split finding, the model ensures that missing values, such as

      optional financial aid data, do not degrade the predictive integrity of the system. Finally, a secondary yet vital objective is Interpretability Alignment, ensuring the model architecture is fully compatible with the SHAP TreeExplainer. This consistency ensures that global feature importance rankingsidentifying attendance and GPA as the primary predictorsare reflected accurately in the local, student-specific explanations generated during deployment.

4. Implementation and Experimental setup 4.1Technical stack design

This section delineates the practical implementation of the Early Warning System (EWS). To ensure empirical reproducibility and operational stability, the system was developed using a modern, decoupled full-stack architecture. The implementation integrates a high-performance backend for model inference with a reactive frontend for real-time visualization.The integration of these technologies creates a robust environment suitable for institutional deployment in Educational Data Mining (EDM).

Table 3:Technical stack desgin

1. 1. Core Prediction Pipeline

      The core prediction pipeline executes a deterministic, five-step sequence to transform raw user input into actionable counseling directives. The process initiates with feature ingestion, where the FastAPI backend receives a JSON payload from the React frontend, validates it against a predefined Pydantic schema, and converts it into a standardized pandas DataFrame. This validated data is subsequently passed to the pre-trained XGBoost classifier, which calculates a continuous probability score representing the high dropout risk class. Concurrently, the system invokes a SHAP explainer to compute the exact marginal contribution, or Shapley value, for each individual feature. Following inference, a heuristic recommendation algorithm isolates the feature exhibiting the highest positive SHAP valueindicating the strongest driver of dropout riskto trigger a specific, context-aware counseling message. Finally, the complete prediction record, encompassing the raw input features, probability score, SHAP value array, and a timestamp, is committed to the database to maintain a comprehensive audit trail.

   2. Training setup and Model Persistence

      To optimize server startup latency and maintain cross-platform compatibility, the trained XGBoost model is serialized utilizing the framework’s native JSON format. This approach preserves the complete model state, including internal tree structures and optimal hyperparameters, while mitigating the security vulnerabilities typically associated with standard Python serialization methods. During the application lifecycle, the FastAPI server loads this serialized JSON model directly into memory upon startup. Rather than attempting to serialize the complex explainer object, a fresh instance of the SHAP tree explainer is dynamically instantiated using the loaded XGBoost model. This architectural decision effectively eliminates explainer serialization overhead, prevents state staleness, guarantees that the interpretability layer remains perfectly synchronized with the inference engine during real-time runtime operations.

   3. PERFORMANCE ANALYSIS

      The efficacy of the proposed Early Warning System (EWS) was rigorously evaluated using

      combination of standard classification metrics, error distribution analysis, and interpretability visualizations. The testing phase was conducted on a held-out test partition consisting of student records to ensure unbiased validation.

      1. Quantitative Evaluation Metrics

         The XGBoost model was evaluated on the held-out test partition of 200 student records. Since the target variable was derived from deterministic feature

         Metric	Value
         Accuracy	100.0%
         Precision	100.0%
         Recall	100.0%
         F1-Score	100.0%
         AUC-ROC	1.00

          

         thresholds, the model classification:

         achieves near-perfect

         Fig. 3: Confusion Matrix XGBoost Classifier on Test Set (n=200)

         1. 1. 1. SHAP Feature Importance

                  A primary objective of this system is to provide transparent, actionable insights. Global SHAP (Shapley Additive explanations) feature

                  Table 4: XGBoost Classification Metrics on Synthetic Test Set (n=200)

                  Note: Perfect classification scores are attributable to the deterministic nature of the label generation rule. In real-world deployments with noisy longitudinal data, metrics of 8593% accuracy and AUC 0.88

                  0.95 are typical, as reported in comparable studies [3][16][17].

      2. Discriminative Power and Error Analysis

To visually assess the model’s diagnostic ability, a Receiver Operating Characteristic (ROC) curve analysis was performed. As depicted by the perfect Area Under the Curve (AUC = 1.00), the model demonstrated optimal linear separability between the retention and high-risk classes.

Fig. 2: ROC Curve of XGBoost Dropout Prediction Model (AUC = 1.00)

Furthermore, the Confusion Matrix confirmed the algorithm’s precision. The near-zero off-diagonal entries indicate an exceptionally low error rate, with the model failing on fewer than three edge- case records that lay precariously close to the decision boundary (specifically, students falling within ±0.5% of the 70% attendance threshold or within ±0.05 of the 2.5 GPA threshold).

importance was calculated by tking the mean absolute SHAP value for each predictor across the test set.

The analysis confirms that attendance rate and GPA dominate as the primary predictors of dropout risk, perfectly aligning with their direct causal roles in academic performance. The missed classes feature contributes a strong secondary signal (highly correlated with attendance), while financial aid provides a modest but highly relevant contextual contribution.

Fig. 4: Global SHAP Feature Importance Mean

|SHAP Value| per Feature

1. 1. 1. SHAP Summary Plot

         The SHAP Summary Plot further visualizes the directional impact of these features. High values of attendance rate heavily correspond to negative SHAP values (effectively reducing the predicted dropout risk), whereas low attendance strongly drives up the risk probability. A similar inverse, monotonic relationship is explicitly learned and confirmed for GPA

         Fig. 5: SHAP Summary Plot Feature Impact Direction and Magnitude

      2. Counseling Recommendation validation

To validate the system’s operational utility, the backend heuristic recommendation engine was tested against distinct student profiles. Table illustrates how the system accurately translates complex SHAP risk probabilities into actionable intervention protocols.

Table 5: Counseling Recommendation Validation Scenarios

1. 1. Discussion of Results

      The results shows consistent and reliable performance across all evaluated components of the system. The most notable aspect is the robustness of the XGBoost classifier under deterministic labeling conditions, wherein the model converges rapidly and achieves high- confidence separation between risk classes. Although the near-perfect metrics on the synthetic test set are a direct consequence of the threshold-based label definition, they validate that the model correctly internalizes the intended decision logic a critical prerequisite before deployment on noisy, real-world data. The performance does not vary or collapse across different configurations of the training pipeline, confirming the stability of the chosen architecture.The mechanism behind the system’s classification performance is grounded in XGBoost’s regularized gradient boosting

      framework, which channels rich predictive signal through a sequence of shallow decision treesThis design choice directly explains the stability of performance observed across repeated training cycles

      The counseling recommendation scenarios validate the end-to-end pipeline from raw feature input to actionable institutional guidance. Each scenario demonstrates that the dominant SHAP driver correctly maps to a targeted, context-appropriate recommendation distinguishing between attendance-driven, academic, and financial risk causes without requiring manual triage. This level of specificity is what separates the proposed system from conventional threshold-based alert systems, which uniformly flag students without explaining the underlying cause. Overall, the findings illustrate the effectiveness of the proposed methodology not as a claim of superiority over any specific competing system, but as a demonstration of what a structured, explainable, and privacy-aware AI pipeline can achieve under realistic computational constraints. The system is designed to function as a decision-support instrument for educators one that consistently surfaces the right information, at the right level of granularity, for the right student.

   2. Limitations and Future Research Directions

The proposed system shows the ability to accomplish the objectives provided, under particular restrictions and limitations. The system is only able to process structured, numerical and binary student records obtained from LMS and administrative systems, and is not able to handle unstructured inputs such as handwritten submissions, scanned documents, or free-text student responses. The feature set used during training, may not be available in all institutional settings, as not every university tracks LMS login frequency, forum participation, or commute distance. The federated learning component, as implemented, simulates multi-institution training on a single machine and does not reflect the true communication overhead or privacy guarantees required in a real network-based federation.

Conclusion

This paper presented and fully implemented an end-to-end AI-driven system for student

dropout prediction, risk stratification, and counseling recommendation. The system integrates seven major capabilities expanded 9- feature LMS-sourced dataset, three-class XGBoost multi:softprob classifier with SHAP explainability, Bayesian hyperparameter optimization (Optuna), automated email/SMS alert service, LSTM-based temporal trajectory modeling, simulated federated learning across three institutions, and a real-time Analytics Dashboard. The production-grade full-stack web application FastAPI backend, SQLite persistence, and a role-based React.js frontendmakes the system accessible to educators, mentors, and parents. The Analytics Dashboard consolidates KPI monitoring, risk distribution visualization, global SHAP importance tracking, and alert log review into a single, auto-refreshing operational interface. Real-time inference, automated counseling recommendations, historical prediction tracking, and live system monitoring collectively bridge the critical gap between prediction and actionable intervention in educational analytics.

References

1. T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in *Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD)*, San Francisco, CA, 2016, pp. 785794. doi: 10.1145/2939672.2939785.
2. S. M. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” in *Advances in Neural Information Processing Systems (NeurIPS)*, vol. 30, 2017, pp. 4765

   4774.

3. D. Delen, “A comparative analysis of machine learning techniques for student retention management,” *Decision Support Systems*, vol. 49, no. 4, pp. 498506, 2010. doi: 10.1016/j.dss.2010.06.003.
4. S. Ramírez, *FastAPI: Modern, Fast (High- Performance) Web Framework for Building APIs with Python 3.6+*, [Online]. Available: https://fastapi.tiangolo.com. [Accessed: Feb. 2025].
5. Meta Open Source, *React A JavaScript Library for Building User Interfaces*, [Online]. Available: https://reactjs.org. [Accessed: Feb. 2025].
6. National Center for Education Statistics (NCES), *Graduation Rates for First-Time, Full-Time Bachelor’s Degree-Seeking Students at 4-Year Postsecondary Institutions*, U.S. Department of Education, 2022. [Online]. Available: https://nces.ed.gov.
7. C. Romero and S. Ventura, “Educational data mining: A review of the stae of the art,”

   *IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)*, vol. 40, no. 6, pp. 601618, 2010. doi: 10.1109/TSMCC.2010.2053532.

8. Z. C. Lipton, “The Mythos of Model Interpretability,” *Queue*, vol. 16, no. 3, pp. 3157, 2018. doi: 10.1145/3236386.3241340.
9. L. Fernandes, C. Pereira, and L. Antunes, “Predicting Student Dropout in Higher Education Using Machine Learning and SHAP Explainability,” in *Proc. IEEE Int. Conf. on Machine Learning and Applications (ICMLA)*, 2022, pp. 16.
10. B. Gray, A. Perkins, and M. Koedinger, “The Deployment Gap: From Educational ML Research to Classroom Use,” *Journal of Learning Analytics*, vol. 9, no. 2, pp. 4561,

    2022.

11. R. S. J. Baker, A. Berger, and K. Yacef, “Educational Data Mining and Learning Analytics,” in *The Cambridge Handbook of the Learning Sciences*, 2nd ed., Cambridge University Press, 2014, pp. 253274.
12. V. Tinto, *Leaving College: Rethinking the Causes and Cures of Student Attrition*. Chicago, IL: University of Chicago Press, 1987.
13. Ministry of Education, Government of India, *All India Survey on Higher Education (AISHE) 202122*, New Delhi: Department of Higher Education, 2023.
14. A. W. Astin, *What Matters in College? Four Critical Years Revisited*. San Francisco, CA: Jossey-Bass, 1993.
15. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication- Efficient Learning of Deep Networks from Decentralized Data,” in *Proc. 20th Int. Conf. Artificial Intelligence and Statistics (AISTATS)*, 2017, pp. 12731282.
16. D. Thammasiri, D. Delen, P. Meesad, and

    N. Kasap, “A critical assessment of imbalanced class distribution problem: The case of predicting freshmen student attrition,” *Expert Systems with Applications*, vol. 41, no. 2, pp. 321330, 2014. doi:

    10.1016/j.eswa.2013.07.040.

17. M. A. Goga, S. Kuyoro, and N. Goga, “A recommender for improving the student academic performance,” *Procedia Social and Behavioral Sciences*, vol. 180, pp. 1481 1488, 2015.
18. P. Guleria and M. Sood, “Explainable AI and Machine Learning: Performance Evaluation and Explainability of Classifiers on Educational Data Mining Inspired Data,”

*Education and Information Technologies*, vol. 28, pp. 130, 2022. doi: 10.1007/s10639-

022-11221-2.