LightGBM based classification for Pneumonia Detection

LightGBM based classification for Pneumonia Detection

Abhishek Kumar Mishra, Devendra Singh, Harpreet Singh Chawla Department of Computer Science & Engineering, IFTM University, Moradabad

Abstract:

Pneumonia remains a leading cause of morbidity and mortality worldwide, particularly affecting vulnerable populations such as children and the elderly. Early detection is crucial for effective treatment, yet traditional methods often require significant expertise and are subject to variability. In this article, we proposed an innovative method for pneumonia detection by integrating an attention-enhanced InceptionResNetV2 model with a LightGBM classifier. The inclusion of a channel attention mechanism into the InceptionResNetV2 architecture significantly improved the model's ability to extract discriminative features, enabling focus on the most informative channels within images. The results shows that the proposed model outperforms existing models, achieving an accuracy of 92.3%.

Keyword: Pneumonia, Accuracy, classification, machine learning

1. Introduction

   Pneumonia remains a leading cause of morbidity and mortality worldwide, especially among vulnerable populations such as children and the elderly. According to World Health Organization (2022), pneumonia accounts for approximately 14% of all deaths of children under five, underscoring the critical need for early and accurate diagnosis. Timely diagnosis is crucial for effective treatment and management, as delays can lead to severe complications or even death. While chest radiography is the commonly used imaging modality for diagnosing pneumonia, interpreting chest X-rays requires significant expertise and is subject to inter-observer variability (Wootton & Feldman, 2014).

   Advancements in AI, particularly deep learning (DL) techniques, have transformed medical imaging, offering new avenues for enhancing pneumonia detection from chest X-rays. DL have exhibited exceptional proficiency in the domain of image classification, achieving high accuracy rates in detecting pneumonia. For instance, studies have shown that CNNs can outperform Support Vector Machines (SVM), in terms of diagnostic accuracy (Saraswati et al., 2024). Moreover, the integration of in which models that have undergone pre-training on extensive datasets are subsequently refined, has further improved the performance of pneumonia detection systems, allowing for effective utilization of limited medical imaging data (Yoon & Kang, 2024).

   Attention mechanisms have been integrated into CNNs to enhance model performance by allowing networks to focus on the most informative parts of an image (Vaswani et al., 2017). The incorporation of attention mechanisms into DL architectures has also emerged as a promising strategy to enhance the interpretability and accuracy of pneumonia detection models. Attention mechanisms facilitate models in concentrating on the most pertinent features of an image, thereby improving classification performance. Recent research has highlighted the effectiveness of attention-based models, such as the Quaternion Channel-Spatial Attention (QCSA) network, which achieved significant accuracy in pneumonia detection by emphasizing critical areas within chest X-ray images (Singh et al., 2023). Furthermore, hybrid approaches that combine CNNs with Long Short-Term Memory (LSTM) networks or other optimization algorithms, have shown potential in improving diagnostic outcomes (Alalwan et al., 2024; Sanida et al., 2023).

2. Literature Review

   The detection of pneumonia from chest X-ray images using DL techniques, particularly with the integration of attention mechanisms, has garnered significant attention in recent years. This surge in interest is primarily due to the increasing prevalence of pneumonia and the critical need for accurate diagnostic tools that can assist healthcare professionals in making timely decisions.

   One notable approach is the use of attention mechanisms within deep learning models, which allows for the selective focus on relevant features within the X-ray images. Singh et al. proposed a QCSA network that proficiently integrates spatial and channel attention mechanisms within a quaternion residual framework. Their findings revealed a classification accuracy of 95.2% in the identification of pneumonia from chest X-ray images, showcasing the potential of attention mechanisms to improve model performance by focusing on relevant image features (Singh et al., 2023). This aligns with the findings of An et al., developed a DCNN that integrates EfficientNetB0 and DenseNet121,

   enhanced by attention mechanisms, achieving an accuracy of 94.5% (An et al., 2024).

   In addition to attention mechanisms, hybrid approaches that combine different deep learning architectures have shown promise. Mishra's comparative study of various deep learning methods for pneumonia classification reported an overall accuracy of 95% across multiple CNN models, indicating that hybrid architectures can leverage the strengths of individual models to enhance diagnostic performance (Mishra et al., 2023). Similarly, Ali et al., demonstrated that the EfficientNetV2L model achieved an accuracy of 94.02% when trained on a dataset of 5,856 chest X-ray images, further validating the efficacy of hybrid models in pneumonia detection (Ali et al., 2024).

   Transfer learning has also played a crucial role in advancing pneumonia detection methodologies. By utilizing pre- trained models, researchers can effectively address the challenges posed by limited datasets. Yoon's study employed VGG16 as a pre-trained model and achieved an accuracy of 92% in classifying chest X-ray images into categories of COVID-19 pneumonia, non-COVID-19 pneumonia, and healthy cases (Yoon & Kang, 2024). This approach not only enhances model performance but also reduces the time and resources required for training DL models from scratch. Similarly, Puspita's research demonstrated the successful application of transfer learning techniques, achieving an accuracy of 91% in pneumonia prediction (Puspita & Rahayu, 2024).

3. Proposed Methodology:

   The proposed methodology integrates DL and ML techniques to enhance lung cancer detection using X-Ray images. Initially, InceptionResNet-V2 enhanced with a channel attention mechanism is employed for feature extraction. Subsequently, these extracted features are classified using LightGBM, a gradient boosting framework. This hybrid approach aims to improve detection accuracy while addressing challenges such as computational efficiency and class imbalance.

   Dataset Description:

   The Chest X-Ray (Kermany et al., 2018) dataset utilized in this study comprises a total of 5,863 images, "Normal" with 1,341 images and "Pneumonia" with 4,522 images.

   Preprocessing

   Preprocessing is a crucial step to standardize the X-Ray images before feature extraction. Initially, all images are resized to 224 × 224 pixels to conform to the input requirements of InceptionResNet-V2. Intensity normalization is applied to scale pixel values to the range [0, 1].

   Feature Extraction with InceptionResNet-V2

   InceptionResNet-V2 (Szegedy et al., 2017) is a CNNs that combines the strengths of the Inception module with the residual connections of ResNet, yielding a powerful mechanism for deep feature extraction. The Inception blocks, integrated with residual connections, capture multi-scale features by applying filters of varying sizes in parallel, enabling the network to learn deeper representations while preserving gradient flow, which is essential in very deep networks. The architecture effectively balances computational efficiency and representational power, making it highly suitable for complex image classification tasks, particularly in the field of medical image analysis.

   Classification using LightGBM

   LightGBM (Ke et al., 2017) is an efficient and fast gradient boosting framework based on decision tree algorithms. It is designed to handle large-scale datasets and provides robust performance even with sparse features. The fundamental concept underlying LightGBM involves the construction of an ensemble comprising weak learners (decision trees), wherein each successive tree addresses the inaccuracies produced by its predecessors.

   The objective function is

   N M

   j

   (y, y) = f(yi , yi) + ;\ w

   (1)

   i=1 j=1

   f(yi, yi) is the loss function, wj represents the weights of the weak learners, and ;\ is a regularization parameter. The feature vectors extracted from InceptionResNet-V2 are fed into the LightGBM classifier. The

   Hyperparameter tuning is performed using grid search to identify the optimal settings for the LightGBM classifier. Key hyperparameters include the learning rate (7), number of leaves (l), and number of boosting rounds (b).

4. Results and Analysis

   1. Experimental setup

      The experiments were conducted on a high-performance laptop with GeForce RTX 4060 GPU with 8 GB VRAM, an Intel Core i7 processor, and 32 GB of RAM. The operating system used was Ubuntu 22.04 LTS. The deep learning models were implemented using Python 3.8 and PyTorch.

      The models underwent training utilizing the Adam optimizers with a learning rate of 1 × 10-4 and a batch size of

      32. To reduce the risk of overfitting, early stopping was assertively implemented, based on the validation loss metrics. For classification, several tree-based ensemble methods were evaluated, including Random Forest (Breiman, 2001), XGBoost (Chen & Guestrin, 2016), CatBoost (Prokhorenkova et al., 2018), LightGBM (Ke et al., 2017), and SVM (Cortes & Vapnik, 1995). Hyperparameters for each classifier were optimized using grid search to achieve the best performance based on validation metrics.

   2. Results

      Several deep learning architectures were evaluated to determine the most effective model for feature extraction from chest X-Ray images. The models tested included Vision Transformer (ViT) (Dosovitskiy et al., 2021), InceptionNet (Szegedy et al., 2016), ResNet50 (He et al., 2016), MobileNetV2 (Howard et al., 2019), EfficientNetB0 (Tan & Le, 2019), VGG19 (Simonyan & Zisserman, 2015), and InceptionResNet-V2 (Szegedy et al., 2017).

      Table 1: Performance Comparison of Deep Learning Models for Feature Extraction

      Algorithm	Acc	Pre	Rec	F1
      InceptionNet	91.5	90.83	90.61	90.72
      ResNet50	90.8	90.07	89.20	89.63
      MobileNetV2	89.7	88.93	88.41	88.66
      EfficientNet	91.8	91.29	90.37	90.83
      VGG19	90.5	89.91	89.08	89.49
      InceptionResNet-V2	92.3	91.97	91.21	91.59

      Acc: Accuracy, Pre: Precision, Rec: Recall, F1: F1-score

5. Conclusion

In conclusion, this research introduced an innovative method for pneumonia detection utilizing chest X-ray images by combining an attention-enhanced InceptionResNetV2 model with a LightGBM classifier. The integration of the channel attention mechanism into the InceptionResNetV2 architecture have significantly improved the model's capability to extract discriminative features, enabling it to focus on the most informative channels within the images. When these features were classified using LightGBM, the model achieved superior performance compared to other classifiers, with an F1-score of 96.27% and an AUC of 0.98.

References

Alalwan, N., Taloba, A. I., Abozeid, A., Alzahrani, A. I., & Al-Bayatti, A. H. (2024). A Hybrid Classification and Identification of Pneumonia Using African Buffalo Optimization and CNN from Chest X-Ray Images. Computer Modeling in Engineering & Sciences, 138(3), 24972517. https://doi.org/10.32604/cmes.2023.029910

Ali, M., Shahroz, M., Akram, U., Mushtaq, M. F., Altamiranda, S. C., Obregon, S. A., Díez, I. D. L. T., & Ashraf, I. (2024). Pneumonia Detection Using Chest Radiographs With Novel EfficientNetV2L Model. IEEE Access, 12, 3469134707. https://doi.org/10.1109/ACCESS.2024.3372588

An, Q., Chen, W., & Shao, W. (2024). A Deep Convolutional Neural Network for Pneumonia Detection in X-ray Images with Attention Ensemble. Diagnostics, 14(4), 390. https://doi.org/10.3390/diagnostics14040390

Breiman, L. (2001). Random forests. Random Forests, 1122. Machine Learning, 45(45), 532. https://link.springer.com/article/10.1023/A:1010933404324

Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 13-17-Augu, 785794. https://doi.org/10.1145/2939672.2939785

Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273297.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. 9th International Conference on Learning Representations, {ICLR} 2021, Virtual Event, Austria, May 3-7, 2021. https://openreview.net/forum?id=YicbFdNTTy

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770778. https://doi.org/10.1109/CVPR.2016.90

Hidayati, S. C., Tasyanita, J., Fatichah, C., & Anistyasari, Y. (2023). Understanding Human Face Shape via Inception- ResNet Neural Network Architecture. 2023 International Conference on Advanced Mechatronics, Intelligent Manufacture and Industrial Automation (ICAMIMIA), 631636. https://doi.org/10.1109/ICAMIMIA60881.2023.10427897

Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., Adam, H., & Le, Q. (2019). Searching for MobileNetV3. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 13141324. https://doi.org/10.1109/ICCV.2019.00140

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: a highly efficient gradient boosting decision tree. Proceedings of the 31st International Conference on Neural Information Processing Systems, 31493157.

Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu,

X., Yan, F., Dong, J., Prasadha, M. K., Pei, J., Ting, M. Y. L., Zhu, J., Li, C., Hewett, S., Dong, J., Ziyar, I., Zhang, K. (2018). Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell, 172(5), 1122-1131.e9. https://doi.org/10.1016/j.cell.2018.02.010

Mishra, V. K., Mishra, M., Sheetlani, J., Kumar, A., Pachouri, P., Nagapraveena, T., Puttamallaiah, A., Sravya, M. N., & Parijatha, K. (2023). The The Classification andSegmentation of Pneumonia using Deep Learning Algorithms: A Comparative Study. International Journal of Experimental Research and Review, 36, 7688. https://doi.org/10.52756/ijerr.2023.v36.007

Mohapatra, S. K., Abebe, M., Mekuanint, L., Prasad, S., Bala, P. K., & Dhala, S. K. (2023). Pneumonia and tuberculosis detection with chest x-ray images and medical records using deep learning techniques. Review of Computer Engineering Research, 10(4), 136149. https://doi.org/10.18488/76.v10i4.3533

Neshat, M., Ahmed, M., Askari, H., Thilakaratne, M., & Mirjalili, S. (2024). Hybrid Inception Architecture with Residual Connection: Fine-tuned Inception-ResNet Deep Learning Model for Lung Inflammation Diagnosis from Chest Radiographs. Procedia Computer Science, 235, 18411850. https://doi.org/10.1016/j.procs.2024.04.175

Peng, C., Liu, Y., Yuan, X., & Chen, Q. (2022). Research of image recognition method based on enhanced inception- ResNet-V2. Multimedia Tools and Applications, 81(24), 3434534365. https://doi.org/10.1007/s11042-022- 12387-0

Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). Catboost: Unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 2018-Decem, 66386648.

Puspita, R., & Rahayu, C. (2024). Pneumonia prediction on chest x-ray images using deep learning approach. IAES International Journal of Artificial Intelligence (IJ-AI), 13(1), 467. https://doi.org/10.11591/ijai.v13.i1.pp467- 474

Sanida, T., Tabakis, I.-M., Sanida, M. V., Sideris, A., & Dasygenis, M. (2023). A Robust Hybrid Deep Convolutional Neural Network for COVID-19 Disease Identification from Chest X-ray Images. Information, 14(6), 310. https://doi.org/10.3390/info14060310

Saraswati, N. W. S., Dewi, D. A. P. R., & Pirozmand, P. (2024). Comparative Analysis of SVM and CNN for Pneumonia Detection in Chest X-Ray. Lontar Komputer : Jurnal Ilmiah Teknologi Informasi, 15(1), 38. https://doi.org/10.24843/LKJITI.2024.v15.i01.p04

Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations (ICLR 2015), 114. https://arxiv.org/abs/1409.1556

Singh, S., Kumar, M., Kumar, A., Verma, B. K., & Shitharth, S. (2023). Pneumonia detection with QCSA network on chest X-ray. Scientific Reports, 13(1), 9025. https://doi.org/10.1038/s41598-023-35922-x

Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. A. (2017). Inception-v4, inception-ResNet and the impact of residual connections on learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 42784284.

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 28182826. https://doi.org/10.1109/CVPR.2016.308