Predicting Agriculture Yield Using Machine Learning

Predicting Agriculture Yield Using Machine Learning

Asst Prof. Madhura M Dept. of CSEData Science ACS College of Engineering Bengaluru, India jayanthmadhura@gmail.com

Girish M

Dept. of CSEData Science ACS College of Engineering Bengaluru, India girishgowda62003@gmail.com

Amit

Dept. of CSEData Science ACS College of Engineering Bengaluru, India amitrasure1@gmail.com

Amogh Basavaraj Mathapati Dept. of CSEData Science ACS College of Engineering Bengaluru, India amogh975@gmail.com

Dhanush Krishna S Dept. of CSEData Science ACS College of Engineering

Bengaluru, India dhanush23nov2004@gmail.com

AbstractAgriculture forms a critical pillar of Indias eco- nomic framework, supporting approximately 58 percent of the rural workforce while remaining highly vulnerable to climatic fluctuations, irregular rainfall patterns, and diverse regional farming practices. Traditional yield estimation approaches rely heavily on historical records and labor-intensive field sur- veys, rendering them slow, inconsistent, and inadequate for responding to rapidly evolving environmental conditions. This research presents a machine learning-driven crop yield prediction framework that delivers precise, district-level forecasts utilizing publicly accessible agricultural datasets. The proposed system employs a Random Forest Regression model trained on multiple features including geographical identifiers, crop variety, cultiva- tion season, farming area, harvest year, and annual precipita- tion. A comprehensive preprocessing pipeline addresses missing value imputation, categorical encoding, outlier elimination via interquartile range analysis, and unit standardization. Model per- formance is rigorously evaluated using coefficient of determina- tion (R2), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE), demonstrating competitive accuracy across di- verse crop varieties and geographical regions. An interactive Streamlit-based web interface enables farmers, researchers, and policymakers to obtain real-time yield predictions and visualize historical production trends through intuitive analytics. The developed system promotes evidence-based agricultural decision- making, optimized resource allocation, and sustainable farming strategies.

Index Termscrop yield prediction, machine learning, random forest regression, agriculture analytics, Streamlit, data prepro- cessing, feature engineering

1. Introduction

   Indian agriculture sustains approximately 58 percent of the rural population and contributes substantially to the national gross domestic product. However, the sector remains highly susceptible to erratic monsoon patterns, progressive soil de- terioration, pest invasions, and significant regional variations in cultivation practices [1]. Accurate crop yield forecasting is therefore essential for proactive resource management, strate- gic government procurement planning, and long-term food security.

   Conventional estimation methodologies encompass statis- tical modeling, satellite-based remote sensing, crop growth simulation tools, and sensor-driven monitoring systems. While these approaches provide preliminary analytical insights, they exhibit several inherent limitations. Statistical models fail to capture non-linear interactions between climate variables and crop productivity. Remote sensing outputs suffer degradation from atmospheric interference and cloud cover. Crop simula- tion models require extensive calibration data, while sensor- based infrastructure demands prohibitive investment beyond the reach of smallholder farms in rural India [3]. Furthermore, most governmental forecasting platforms generate predictions at state or national aggregation levels, limiting their practical utility for localized agricultural planning.

   Recent advances in machine learning present a compelling alternative. Machine learning algorithms, particularly ensem- ble methods such as Random Forest, can effectively learn complex non-linear relationships between environmental and agronomic variables without requiring explicit mechanistic model formulation. This paper proposes a district-level crop yield prediction system that integrates Random Forest Regres- sion with a robust preprocessing pipeline and an accessible Streamlit-based user interface.

   The primary contributions of this work are threefold:

   • A reproducible machine learning pipeline encompassing data cleaning, categorical encoding, outlier removal, and feature engineering procedures.

   • A trained Random Forest model demonstrating strong generalization capability across diverse crops, districts, and seasonal conditions.

   • An interactive web application enabling non-technical stakeholders to obtain instantaneous yield forecasts and comprehensive visual analytics.

2. Literature Review

   Sharma et al. [1] conducted a comprehensive evaluation of Decision Tree, Random Forest, XGBoost, Convolutional Neu-

   ral Networks (CNN), and Long Short-Term Memory (LSTM) networks using governmental agricultural datasets spanning 1997 to 2020. Their experimental framework employed a 70:30 train-test partition with four-fold cross-validation. Ran- dom Forest achieved the highest accuracy at 98.96 percent, while CNN models demonstrated the lowest loss values. LSTM networks exhibited suboptimal performance due to challenges in capturing temporal agricultural patterns, high- lighting the complexity of time-series modeling in agricultural contexts.

   Rao et al. [2] assessed K-Nearest Neighbours, Decision Tree, and Random Forest algorithms for crop recommendation using soil nutrient profiles and climatic attributes from pub- licly available datasets. Random Forest surpassed competing algorithms, achieving 99.32 percent accuracy with an 80:20 data partition, reinforcing its suitability for agricultural classi- fication and regression applications.

   Gandhi et al. [3] applied Support Vector Machines (SVM) to predict rice yield across 27 districts of Maharashtra, catego- rizing productivity into low, medium, and high classifications. The SVM model achieved 78.76 percent accuracy, underper- forming compared to Naive Bayes and Multilayer Perceptron alternatives. This highlights the limitations of linear-kernel SVMs when applied to non-linear agricultural datasets with complex feature interactions.

   Khan and Noor [4] investigated machine learning techniques for irrigation runoff volume prediction, finding Decision Trees superior to Multiple Linear Regression and Artificial Neu- ral Networks in modeling non-linear hydrological relation- ships. This finding corroborates the effectiveness of tree-based methodologies for agricultural applications characterized by intricate environmental dependencies.

   Kerimbayev et al. [6] demonstrated dashboard-driven adap- tive analytics for educational settings. Their principles of in- teractive visualization and real-time feedback mechanisms are directly transferable to agricultural decision-support interfaces, informing the user experience design of this research.

   The reviewed literature consistently identifies two critical gaps. First, most existing systems optimize exclusively for model accuracy while neglecting deployment considerations and practical usability. Second, many approaches depend on specialized sensors or satellite data that remain unavail- able across all agricultural regions, particularly in resource- constrained rural areas. The proposed system addresses both limitations through a lightweight, browser-accessible deploy- ment utilizing solely publicly available tabular datasets, thereby ensuring broad aplicability and accessibility.

3. Methodology

   1. Dataset Acquisition and Description

      Historical crop production records were obtained from pub- licly accessible governmental agricultural repositories main- tained by the Indian Ministry of Agriculture. Each record contains seven primary attributes: state identifier, district iden- tifier, crop name, cultivation season (Kharif, Rabi, Whole Year), cultivated area in hectares, crop year, annual rainfall

      in millimeters, and crop yield in quintals per hectare. The dataset spans multiple years and diverse geographical regions, enabling the model to learn long-term productivity trends and seasonal patterns. Data partitioning followed a 70 percent training, 15 percent validation, and 15 percent testing distri- bution to ensure robust model evaluation.

   2. Preprocessing Pipeline

      Raw agricultural data exhibit missing values, unit inconsis- tencies, and categorical attributes incompatible with numeri- cal machine learning algorithms. The preprocessing pipeline addresses these challenges through five sequential stages, as illustrated in Fig. 1:

      1. Missing value imputation: Rainfall gaps are filled using a hierarchical strategy. The system first attempts district-level average imputation, followed by state-level average, with a fallback constant of 1000 mm for cases where neither average is available.

      2. Outlier removal: Records with yield values outside the interquartile range boundaries are systematically discarded. Bounds are calculated as Q1 1.5 × IQR

         (lower bound) and Q3 +1.5×IQR (upper bound), where IQR = Q3 Q1 represents the interquartile range. This statistical approach removes extreme values that could

         bias model training.

      3. Label encoding: Categorical fields including state, dis- trict, crop, and season are transformed to integer codes using pre-fitted LabelEncoder objects from scikit- learn. These encoders are persisted via Joblib to ensure consistent encoding during inference.

      4. Unit conversion: Area measurements are standardized by converting from acres to hectares using the conver- sion factor of 0.4047. Production values are converted from tonnes to kilograms through multiplication by 1000, ensuring consistent units across all calculations.

      5. Feature engineering: Derived attributes such as average seasonal rainfall and region-wise yield trends are com- puted and appended to the feature matrix, enriching the input representation with domain-informed features.

         Fig. 1. Data preprocessing pipeline showing sequential transformation stages from raw agricultural data to model-ready features.

   3. Random Forest Regression Model

      Random Forest constructs an ensemble of T decision trees, each trained on a bootstrap sample of the training data with random feature subsets selected at each node split. This bag-

      Root Mean Square Error (RMSE) penalizes larger errors more heavily than MAE, providing insight into prediction outliers.

      i i

      n (y y)2

      ging approach reduces variance and improves generalization. The final prediction is computed as the arithmetic mean of

      R2 = 1 i=1

      n

      i=1

      (yi y¯)2

      (8)

      individual tree predictions:

      y =

      1 T

      T

      t=1

      ft(x) (1)

      The coefficient of determination (R2) quantifies the propor- tion of variance in yield explained by the model, with values approaching 1 indicating superior predictive performance.

      F. System Architecture

      where ft(x) represents the prediction of the t-th decision tree for input feature vector x.

      During tree construction, node splits minimize the mean squared impurity:

      n

      The system follows a modular three-layer architecture, as depicted in Fig. 2:

      • Data Layer: Manages persistent storage of the CSV- based agricultural dataset, serialized Random Forest model, and label encoders. All artifacts are stored via

      MSEnode

      = 1 (y 2

      n i y¯)

      i=1

      (2)

      Joblib for efficient loading during inference.

      • Machine Learning Layer: Encapsulates preprocessing logic, feature encoding mechanisms, and the prediction

      where n is the number of samples in the node, yi represents individual yield values, and y¯ is the mean yield within the node.

      Hyperparameters were tuned empirically through cross- validation with the following optimal configuration: number of estimators T = 200, maximum tree depth of 20, minimum samples per split of 5, minimum samples per leaf of 2, and random state of 42 for reproducibility. This configuration balances model complexity with computational efficiency.

   4. Yield Computation and Unit Conversion

      Post-prediction, the raw model output in quintals per acre undergoes conversion to actionable agricultural metrics:

      Yieldkg/ha = Yieldq/acre × 247.1 (3)

      engine that wraps scikit-learn inference. This layer en- sures consistent data transformation and model execution.

      • Presentation Layer: Implements a Streamlit web appli- cation providing interactive input forms, real-time pre- diction cards, historical trend visualizations, and district- wise comparison charts. This layer democratizes access to advanced ML capabilities through an intuitive interface.

      q/ha

      Yield = Yieldkg/ha

      100

      (4)

      Productionkg = Yieldkg/ha × Areaha (5)

      These conversions enable farmers and policymakers to interpret predictions in familiar units aligned with regional agricultural practices.

   5. Evaluation Metrics

      Model quality is rigorously assessed using three comple- mentary regression metrics:

      n

      Fig. 2. Three-layer system architecture illustrating data flow from user input through machine learning processing to visualization output.

4. Implementation

   The system is implemented entirely in Python 3.8+, lever- aging industry-standard libraries for scientific computing and machine learning. Core dependencies include Pandas 1.x and NumPy for data manipulation, scikit-learn for model training and encoding, Matplotlib and Seaborn for visualization, and Streamlit for the web interface. The trained model and en-

   MAE = 1 |y y| (6) coders are serialized with Joblib and loaded once at application

   n i i

   i=1

   startup to minimize inference latency.

   Mean Absolute Error (MAE) provides a linear measure of average prediction error magnitude.

   ,

   n

   u 1

   The end-to-end prediction workflow executes the following sequence:

   1. User submits agricultural parameters via the Streamlit interface.

   RMSE = ,

   n

   i=1

   (yi yi)2 (7) 2) Input validator checks range constraints and categorical

   vocabulary membership.

   TABLE I

   Key Implementation Modules

   Module Responsibility

   Data Acquisition Load raw CSV; expose typed DataFrames Data Preparation Impute, de-duplicate, normalize formats Feature Engineering Encode categoricals; derive aggregates Model Building Train Random Forest; tune hyperparameters

   Model Evaluation Compute R2, MAE, RMSE; plot residuals

   Prediction Engine Validate inputs; run inference; convert units User Interface Streamlit forms, charts, metric cards Model Persistence Joblib serialization and fast reload

5. Preprocessor and encoder manager transform inputs into a seven-element feature vector x R7.

6. Prediction engine invokes model.predict(x) to ob- tain raw yield estimate y in quintals per acre.

7. Post-processor converts y to kilograms per hectare, quintals per hectare, and total production using Equa- tions (3)(5).

8. Report generator retrieves historical statistics and con- structs visualization data.

9. Streamlit renders interactive metric cards, trend charts, and district comparisons.

This research presented a comprehensive district-level crop yield prediction system that combines a Random Forest Re- gression engine with a structured preprocessing pipeline and an interactive Streamlit-based interface. The system accurately forecasts crop yield from readily available agricultural param- etersstate, district, crop type, season, area, year, and rain- fallwithout requiring sensor hardware or satellite imagery.

References

1. P. Sharma, P. Dadheech, N. Aneja, and S. Aneja, Predicting agriculture yields based on machine learning using regression and deep learning, IEEE Access, vol. 11, pp. 117, 2023.

2. M. S. Rao, A. Singh, N. V. S. Reddy, and D. U. Acharya, Crop prediction using machine learning, J. Phys.: Conf. Ser., vol. 2161, no. 1, p. 012042, IOP Publishing, 2022.

3. N. Gandhi, O. Petkar, L. J. Armstrong, and A. K. Tripathy, Rice crop yield prediction in India using support vector machines, in Proc. 13th Int. Joint Conf. Computer Science and Software Engineering (JCSSE),

   pp. 15, IEEE, 2016.

4. M. Khan and S. Noor, Irrigation runoff volume prediction using

   machine learning lgorithms, Eur. Int. J. Sci. Technol., vol. 8, no. 4,

   pp. 4451, 2019.

5. S. Noor and M. Khan, Performance analysis of regressionmachine learning algorithms for prediction of runoff time, Agrotechnology, vol. 8, no. 2, pp. 15, 2019.

6. N. Kerimbayev, V. Jotsov, A. Akramova, and N. Nurym, Application of student-centred mobile learning models in higher education, Int. J. Mob. Learn. Organ., vol. 17, no. 12, pp. 5069, 2023.

7. N. Ibrahim, M. Hanum, and Z. Abu Bakar, Studentindustry matching for internship placement, in Proc. Int. Conf. Computational Intelligence,

   pp. 123135, 2020.

8. B. Sreya, N. V. S. Reddy, and G. Akshitha, College placement mon- itoring system using cloud computing, Int. J. Eng. Res. Technol., vol. 10, no. 5, pp. 234239, 2021.

9. D. S. Prakash, K. M. Dhanashree, and R. S. P. Murthy, Integrated web-based platform for enhanced management and analytics, Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 8, pp. 456462, 2021.

10. S. Agarwal and S. Tarar, A hybrid approach for crop yield prediction using machine learning and deep learning algorithms, School of ICT, Gautam Buddha University, Greater Noida, India, Tech. Rep., 2023.