Probability Neural Network Classification Model of Brain Tissue Pathologies Using Signal Intensity Gradient Technique

DOI : 10.17577/IJERTV3IS20719

Download Full-Text PDF Cite this Publication

Text Only Version

Probability Neural Network Classification Model of Brain Tissue Pathologies Using Signal Intensity Gradient Technique

S. S. Shanbhag1, G. R. Udupi1, K. M. Patil2, K. Ranganatp

1Electronics and Communication Engineering, Gogte Institute of Technology, Belgaum, India

2Retired, Indian Institute of Technology (Madras), Presently at Belgaum, India

3RAGAVS, Diagnostics and Research Center Pvt. Ltd., Bangalore, India

Abstract –Automated classification in medical images is motivated by the need of high accuracy when dealing with a human life. Conventional operator-assisted classification techniques are not viable for large amounts of medical data and are generally non-reproducible. In the proposed work, we have made an effort to realize an automated classification system to derive useful information about the brain pathologies, namely, cerebral infarction, Intracerebral Haemorrhage (ICH) and brain tumors (glioma and meningioma), by using prior information about the signal intensity characteristics on Diffusion Weighted-Magnetic Resonance (DW-MR) images. The automated system is designed using Probability Neural Network (PNN) architecture that works on the quantified signal intensity variations on DW- MR images, via Signal Intensity Gradient (SIG) parameter. The brain pathology PNN models were able to accurately (100%) categorize ICH subjects into their respective stages, and presented an overall efficiency of 86.67% in classifying the infarct subjects. Also the model presented an overall efficiency of 92.86% in differentiating the subjects with glioma and meningioma. The PNN models could aid in better understanding the degree of brain pathology in the absence of an expert, directly in terms of the brain image signal intensity parameters, and additionally aid in identifying the extent of damage caused to the brain tissues.

Keywords-Diffusion weighted images, Glioma, Infarction, Intracerebral haemorrhage, Magnetic resonance imaging, Meningioma, Signal intensity, Signal intensity gradient

  1. INTRODUCTION

    Neurological diseases are common important disorders resulting in various degrees of disability and loss of productive life [1]. Early and accurate diagnosis of the brain pathologies plays a vital role in the implementation of successful treatment and thereby improving the quality of life. In recent times, when neurological disorders are gradually increasing, we are also witnessing remarkable advances in brain imaging, which is becoming an increasingly essential tool in clinical research and care [1]. Conventional imaging techniques of monitoring and diagnosing brain pathologies rely on detecting the presence of particular features, by a human observer. Such human

    assisted diagnostic techniques are not viable for large amounts of medical data and are generally non-reproducible [2]. Thereby several techniques for automated diagnostic systems have been developed in recent years, which could improve the results of humans, so that false negative cases are kept at a very low rate [2]. In this domain the applications of neural networks to feature classification have been extensively studied in the past many years. Various kinds of neural network architectures including MultiLayer Perceptron (MLP) neural network, Radial Basis Function (RBF) neural network, Self-Organizing Map (SOM) neural network and Probability Neural Network (PNN) have been proposed [3]. However, because of ease of training and a sound statistical foundation in Bayesian estimation theory, PNN has become an effective tool for solving many classification problems [4-6]. PNN often learns more quickly than many neural network models, and has had success on a variety of applications [7].

    Diffusion Weighted – Magnetic Resonance Imaging (DW-MRI) is one of the most rapidly evolving imaging techniques, both for very basic clinical needs and for advanced diagnosis and treatment planning [8]. It is an extremely sensitive tool to monitor brain water diffusion changes with high temporal and spatial resolution, and therefore allows for more detailed insight into the pathophysiology of the brain pathologies [9]. Diffusion Weighted – Magnetic Resonance (DW-MR) images present variation in the signal intensity characteristics relative to the amount of restriction to water molecules, post the occurrence of the brain pathologies [10]. Therefore, it is very essential in understanding the characteristic features of the signal intensity variations on DW-MR images post brain pathologies, as it may allow for the assessment of the brain affected areas and thereby allow for the diagnosis and proper planning of the clinical treatment. In the present study, PNN architecture is employed to develop a classification model based on the quantified signal intensity variations on DW-MR images, derived from the subjects with brain pathologies, using Signal Intensity Gradient (SIG) parameter. Studies are performed on a set of DW-MR images of the human brain (in the axial plane as shown in

    Fig.1) to understand the characteristics of the signal intensity variations, for the subjects with brain pathologies, namely, cerebral infarction, Intracerebral Haemorrhage (ICH) and brain tumor (glioma and meningioma); the details of the study given in our earlier published works [11, 12]. An effort is made to grade the signal intensity variations on DW-MR images using SIG values, for the subjects with these brain pathologies, compared to the corresponding SIG values obtained from the respective contralateral normal hemispheres. The quantified signal intensity variations using SIG values are employed in developing brain pathology PNN models, to automatically classify the stages of cerebral infarction [13], categorize the stages of ICH [14], and differentiate between the types of brain tumors, namely, glioma and meningioma.

    Fig. 1 DW-MR image showing region-of-interest on pathology (glioma) side and on contralateral side (axial slice 13)

    The PNN models developed in the present work could positively be employed to derive valuable information about the particular brain tissue pathology, which may assist the medical personnel in the speedy diagnosis and execution of treatment. Further, the proposed computer based technique could simplify the estimation process and provide information essential for the further management and therapeutic decisions, even in the absence of a medical expert.

  2. CLINICAL DATA AND DIFFUSION WEIGHTED IMAGING

    1. Subjects

      The clinical data in the present study is obtained from

      RAGAVS Diagnostic and Research Center, Bangalore, India, and Vikram Hospital, Bangalore, India. The ethics approval is obtained from the committee of clinical research at the RAGAVS Diagnostic and Research Center, and Vikram Hospital, to carry out the investigations on the clinical data provided. Applying the clinical inclusion criteria, we identified 98 cases with clinically definite cerebral infarctions with known symptom onset, which were employed in training and testing the PNN model. This population consisted of 64 male and 34 female subjects, ranging in age from 25 to 95 years (mean, 62.87 years). DW-MRI examinations were performed on 24 subjects within 24 hours of symptom onset; 15 subjects between days 1 and 4; 38 subjects between days 5 and 9; 12 subjects

      between days 10 and 14; and 09 subjects after day 15.

      Among all the ICH subjects admitted, we retrospectively selected 42 cases that showed isolated ICH without the

      presence of underlying tumor or infarction on initial radiologic and follow-up examinations, which were employed in training and testing the PNN model.This population consisted of 29 male and 13 female subjects, ranging in age from 28 to 85 years (mean, 58.62 years). DW-MRI examinations were performed on 09 subjects within 24 hours of symptom onset; 05 subjects between days 1 and 7; 15 subjects between days 7 and 14; and 13 subjects after 14 days.

      Findings of DW-MRI examinations performed on 22 cases (15 male, 07 female; mean age, 56.55 years; age range, 39 83 years) with clinically proved glioma and 12 cases (04 male, 08 female; mean age, 56.50 years; age range, 26 81 years) with clinically proved meningioma, were also retrospectively selected, and employed in training and testing the PNN model.

    2. Imaging parameters

    All the subjects underwent clinical MR imaging with 1.5 Tesla symphony maestro class MR scanning system from Siemens. DW-MRI was performed by using a multisection, single-shot, spin-echo, echo-planar pulse sequence with following parameters: Repetition Time [TR] = 3200 ms, Echo Time [TE] = 94 ms, acquisition matrix = 128 x 128, Field of View [FOV] = 230 mm x 230 mm, and diffusion gradient value of b = 1000 s/m2 along 19 axial sections, 5 mm thick section and intersection gap of 1.5 mm.

  3. METHODOLOGY

    1. Signal intensity gradient

      In biological tissues, the presence of various tissue components restrict the Brownian motion in certain directions, and this may render the water diffusion have different values when measured in different directions. Thereby, the molecular mobility may not be the same in all directions, resulting in a spatial direction-related diffusion property termed as diffusion anisotropy [15]. In order to take care of the orientation dependent contrast (diffusion anisotropy), the SIG value in the present work is evaluated using the compass detection technique, that allows extracting explicit information about the change in intensity in any direction.

      We have employed Robinson compass operator comprising of 8 kernels, as shown in Fig. 2, to calculate the SIG value. Robinson compass operator is chosen as it is easier to implement compared to the other compass detectors, because it relies only on coefficients of 0, 1 and 2, and is symmetrical about their directional axis (axis with zeros). The set of 8 kernels applied in Robinsons compass operator are produced by taking one kernel and rotating its coefficients circularly. These kernels are convolved with the input image to obtain the SIGs along the corresponding directions. GS, GSE, GE, GNE, GN, GNW, GW, and GSW, represents gradients along South, South-East, East, North- East, North, North-West, West and South-West directions respectively. The magnitude of the maximum change in the SIG is the maximum value gained from applying all the 8 kernels to the pixel neighbourhood [16, 17]. For each pixel, the local gradient magnitude is estimated with the maximum

      response of all the masks at this pixel location, as given in Eq. 1 [16].

      SIGmax = max{GS, GSE, GE, GNE, GN, GNW, GW, GSW} (1)

      Fig. 2 Robinson kernels and their orientation

      The SIG value evaluated in the area of brain pathology is compared to the corresponding SIG value on the contralateral normal hemisphere of the same subject to obtain the relative signal intensity SIG (RSIG) value for the subject, as given in Eq. 2.

      SIG value on p-athology side

      RSIG correspond ing SIG value on contralateral side (2)

      correspond ing SIG value on contralateral side

      Fig. 3 PNN network architecture [19]

      (Note: The symbols and notations are adopted as used by MATLAB Neural Network Toolbox. Dimensions of arrays are marked under their names)

      The PNN has three layers: the Input Layer, Radial Basis Layer and the Competitive Layer. It is assumed that there are Q input vector/target vector pairs, with each target vector having K elements. Each input vector is associated with one of K classes. The dimension of the input vector, denoted as p, is R x 1. In radial basis layer, the vector distances between input vector and the weight vector made of each row of weight matrix, W, are calculated. The weights of the radial basis layer, IW1, 1, are set to the transpose of the matrix formed from the Q training pairs, P'. When an input is presented, the ||dist|| box produces a vector whose elements indicate how close the input is to the vectors of the training set. These elements are multiplied, element by element, by the bias and sent to the radial basis (radbas) transfer function. The transfer function for a radial basis neuron is given by Eq. 3 and the plot of the radial basis transfer function [19] is shown in Fig. 4.

      The RSIG value is quantified across different subjects diagnosed with the brain pathology, and is employed in differentiating pathology tissues from healthy tissues, and further used in the categorization of the different stages of the pathology (cerebral infarction/ICH) and also in differentiating between the different pathology types (glioma and meningioma). The details of the calculation of SIG and RSIG values are elaborated in our earlier studies [11, 12].

    2. Probability neural network

      All PNNs predominantly classifiers, are a special form of RBF network that can map any input pattern to a number of classifications. A function is radial basis if its output

      radbas (n) en2

      Fig. 4 Radial basis function

      (3)

      depends on the distance of the input from a given stored vector. The RBF network [18] is a special class of multilayer feedforward networks wherein each unit in the hidden layer employs a RBF, such as a Gaussian kernel, as the activation function. The RBF is centered at the point specified by the weight vector associated with the unit. Both the positions and the widths of these kernels must be learned from training patterns. Each output unit implements a linear combination of these RBFs. We have chosen a basic Matlab PNN for its simple structure and training manner. The network architecture of a PNN employed in the present work is shown in Fig. 3.

      The biases are all set to 0.8326/spread, where, spread is the spread value of RBF. As spread becomes larger, the designed network takes into account several nearby design vectors. An input vector close to a training vector is represented by a number close to 1 in the output vector a1. If an input is close to several training vectors of a single class, it is represented by several elements of a1 that are close to 1 [19].

      The weights of the competitive layer, LW2, 1, are set to the matrix of target vectors. Each vector has a 1 only in the row associated with that particular class of input, and 0s elsewhere. The competitive layer sums these contributions for each class of inputs to produce as its net output a vector

      of probabilities. Finally, compete transfer function picks the maximum of these probabilities, and produces a 1 for that class and a 0 for the other classes. Thus, the network classifies the input vector into a specific K class because that class has the maximum probability of being correct [19, 20].

    3. Quantitative brain pathology models using SIG

    Brain pathology models for the subjects with cerebral infarction, ICH and brain tumors (glioma and meningioma) with SIG parameter are developed using PNN architecture as shown in Fig. 3. The PNN is implemented by using MATLAB version 7.7. The data set is divided into two separate data sets the training data set and the testing data set. The training data set is used to train the network, whereas the testing data set is used to verify the accuracy and the effectiveness of the trained network, for the classification process. Spread is an important parameter for PNN, and the accuracy of the classification process may vary with different spread values. The task in designing a PNN thereby lies in selecting spread values for the particular classification problem. To facilitate choose the spread value in the present work we have made use of the

    leave one out method of cross-validation. Here cross- valdation is carried out using a single observation from the training sample (N) as the validation data, and the remaining observations (N-1) as the training data. The value of spread is adjusted so that the chosen data for validation from the training set is appropriately classified. The process is repeated, such that each observation in the training sample is used once as the validation data [21]. By doing so we choose the optimal spread value, giving maximum accuracy for the given classification problem. The brain pathology models using PNN architecture developed in the present work could help in identifying the stage of cerebral infarction, stage of ICH, and differentiating between the tumor types, quantitatively, relative to the SIG parameter.

  4. RESULTS AND DISCUSSIONS

    1. Cerebral infarction classification model using SIG

      A block schematic diagram of the cerebral infarction classification model using SIG parameter developed using PNN architecture (Fig. 3) is shown in Fig. 5.

      Fig. 5 Block schematic diagram of cerebral infarction classification model using SIG

      Studies are performed on cerebral infarct subjects to understand the signal intensity variation distribution on DW-MR images, using SIG parameter. The RSIG value is given as the input variable, and the output of the PNN block is the resulting stage of infarction. An attempt is made to associate the RSIG parameter evaluated from the infarct subjects with the different stages of infarction, by designing a PNN and training it with adequate number of input-output patterns. The developed PNN model could help in

      classifying the stages of infarction, quantitatively, using SIG measurements.

      The PNN model is tested with infarct subjects in different stages of infarction. A total number of 65 data sets (split into different stages of infarction as shown in Table 1) are used for training the PNN. Similarly, a total number of

      33 data sets (split into different stages of infarction as shown in Table 1) are used for testing the PNN. The spread value of the RBF is used as a smoothing factor and classifier accuracy is examined with different values of spread. The optimal spread value for the classification of cerebral infarct subjects using SIG parameter is found to be 0.5.

      The performance of the classification model is evaluated as the percentage of the total number of patterns in the testing data set that are correctly classified. The number of data sets used for training, testing, and the percentage of correct classification obtained from the PNN model, shown in Fig. 5, is tabulated in Table 1. It is observed from Table 1 that the cerebral infarct classification model using SIG

      Table 1 Number of data sets used for training, testing, and the number of correct classification obtained for tested data using cerebral infarct

      classification model with SIG

      Cerebral infarction stage (days)

      Number of data sets used for training

      Number of data sets used for testing

      Percentage of correct classification

      Stage 1

      (<1)

      16

      8

      75.00

      Stage 2

      (1 4)

      9

      6

      83.33

      Stage 3

      (5 9)

      26

      12

      75.00

      Stage 4

      (10 14)

      8

      4

      100.00

      Stage 5

      (>15)

      6

      3

      100.00

      parameter is able to classify clearly (100%) the infarct subjects in stage 4 and 5 only. The performance of the model for stage 1, 2, and 3 could be improved by training with more number of corresponding input patterns.

    2. ICH classification model using SIG

      A block schematic diagram of the ICH classification model using SIG parameter developed using PNN architecture (Fig. 3) is shown in Fig. 6.

      Fig. 6 Block schematic diagram of ICH classification model using SIG

      Studies are performed on ICH subjects to understand the signal intensity variation distribution on DW-MR images,

      using SIG parameter. The RSIG value is given as the input variable, and the output of the PNN block is the resulting stage of ICH. An attempt is made to associate the RSIG parameter evaluated from the ICH subjects with the different stages of ICH, by designing a PNN and training with adequate number of input-output patterns. The developed PNN model could help in classifying the stage of ICH, quantitatively, using SIG measurements.

      The PNN model is tested with ICH subjects in different stages of ICH. A total number of 28 data sets (split into different stages of ICH as shown in Table 2) are used for training the PNN. Similarly, a total number of 14 data sets (split into different stages of ICH as shown in Table 2) are used for testing the PNN. The spread value of the RBF is used as a smoothing factor and classifier accuracy is examined with different values of spread. The optimal spread value for the classification of ICH subjects using SIG parameter is found to be 0.7.

      The performance of the classification model is evaluated as the percentage of the total number of patterns in the testing data set that are correctly classified. The number of data sets used for training, testing, and the percentage of correct classification obtained from the PNN model, shown in Fig. 6, are tabulated in Table 2. It is observed from Table 2 that the ICH classification model is able to classify clearly (100%) the ICH subjects in all the stages of ICH.

      Table 2 Number of data sets used for training, testing, and the number of correct classification obtained for tested data using ICH classification

      model with SIG

      ICH stage (days)

      Number of data sets used for

      training

      Number of data sets used for

      testing

      Percentage of correct classification

      Hyperacute (< 1)

      6

      3

      100

      Acute (1 7)

      3

      2

      100

      Late subacute

      (7 14)

      10

      5

      100

      Chronic

      (>14)

      9

      4

      100

    3. Brain tumor classification model using SIG

    A block schematic diagram of the brain tumor classification model using SIG parameter developed using PNN architecture (Fig. 3) is shown in Fig. 7. Studies are performed on subjects with glioma and meningioma to understand the signal intensity variation distribution on DW-MR images, using SIG parameter.

    Fig. 7 Block schematic diagram of brain tumor classification model using SIG

    The RSIG value is given as the input variable, and the output of the PNN block is the resulting type of tumor, i.e.

    either glioma or meningioma. An attempt is made to associate the RSIG parameter evaluated from the tumor subjects, with the appropriate tumor type, by designing a PNN and training with adequate number of input-output patterns. The developed PNN model could help in the differential diagnosis of brain tumors namely glioma and meningioma, quantitatively, using SIG measurements.

    The PNN model is tested for the subjects with gliomas and meningiomas. A total number of 15 data sets with glioma and 08 data sets with meningioma are used for training the PNN. Similarly, a total number of 07 data sets with glioma and 04 data sets with meningioma are used for testing the PNN. The spread value of the RBF is used as a smoothing factor and classifier accuracy is examined with different values of spread. The optimal spread value for the classification of brain tumor subjects using SIG parameter is found to be 0.7.

    The performance of the classification model is evaluated as thepercentage of the total number of patterns in the testing data set that are correctly classified. The number of data sets used for training, testing, and the percentage of correct classification obtained from the PNN model, shown in Fig. 7, is tabulated in Table 3.

    Table 3 Number of data sets used for training, testing, and the number of correct classification obtained for tested data using brain tumor SIG

    classification model

    Tumor type

    Number of data sets used for

    training

    Number of data sets used for

    testing

    Percentage of correct classification

    Glioma

    15

    7

    85.71

    Meningioma

    8

    4

    100

    It is observed from Table 3 that the brain tumor classification model is able to clearly (100%) differentiate the subjects with meningiomas. The performance of the model could be improved by training with more number of corresponding input patterns for subjects with glioma.

  5. CONCLUSIONS

Quantitative brain pathology models using PNN architecture are developed for the classification of the different stages of cerebral infarction and ICH, and differentiation between the two tumor types, namely, glioma and meningioma. Separate PNN models are developed for each type of pathology using RSIG as the input parameter. The PNN model developed for the classification of cerebral infarct stages is able to classify the infarct cases into their respective stages with an overall efficiency of 86.67%. The PNN model developed for the classification of ICH stages is able to classify the ICH cases accurately (100%) into their respective stages. Also, the PNN model developed for the differential diagnosis of glioma and meningioma is able to differentiate between the two tumor types with an overall

efficiency of 92.86 %. The models can be additionally trained with sufficient number of data sets of signal intensity measurements to further enhance their performance. The PNN models provide helpful information that could positively assist the medical personnel in the speedy diagnosis and execution of treatment of the respective brain pathologies. Thereby, we suggest that the adoption of the proposed method in the clinical diagnosis and treatment of these brain pathologies could be instructive in devising corrective methods and providing proper treatment, to protect the subjects from additional damage to their brain tissue.

ACKNOWLEDGEMENTS

The authors would like to thank Dr. Sudhir Pai (Sr. Consultant, Neurosurgeon, Vikram Hospital, Bangalore, India) for his valuable assistance in data collection and Dr. Deepak Y. S. (Consultant Radiologist, Bangalore, India) for his helpful and constructive suggestions during the planning and development of this research work.

REFERENCES

  1. M. Gourie-Devi, Organization of Neurology Services in India: Unmet Needs and the Way Forward, Neurology India, January- March 2008, 56(1), pp. 4-12.

  2. M. V. Kumar and S. Kulkarni, Tumors Classification Using PNN Methods, International Journal of Soft Computing and Engineering, November 2012, 2(5), pp. 266-268.

  3. K. Z. Mao, K. C. Tan, W. Ser, Probabilistic Neural Network Structure Determination for Pattern Classification, IEEE Transactions on Neural Networks, July 2000, 11(4), pp. 1009-1016.

  4. D. F. Specht, Probabilistic Neural Networks, Neural Networks, 1990, 3, pp. 109-118.

  5. C. Kramer, B. Mckay, J. Belina, Probabilistic Neural Network Array Architecture for ECG Classification, Proceedings of Annual International Conference of IEEE Engineering in Medicine and Biology Society, 1995, 17, pp. 807-808.

  6. M. T. Musavi, K. H. Chan, D. M. Hummels, K. Kalantri, On the Generalization Ability of Neural Network Classifier, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994, 16(6), pp. 659-663.

  7. I. M. M. El Emary and S. Ramakrishnan, On the Application of Various Probabilistic Neural Networks in Solving Different Pattern Classification Problems, World Applied Sciences Journal, 2008, 4 (6), pp. 772-780.

  8. R. Rajeshkannan, S. Moorthy, K. P. Sreekumar, R. Rupa, N. K. Prabhu, Clinical Applications of Diffusion Weighted MR Imaging: A Review, Indian Journal of Radiology and Imaging, 2006, 16(4), pp. 705-710.

  9. L. C. Maas and P. Mukherjee, Diffusion MRI: Overview and Clinical Applications in Neuroradiology, Applied Radiology, November 2005, 34(11), pp. 44-60.

  10. D. Le Bihan, Looking into the Functional Architecture of the Brain with Diffusion MRI, Nature Reviews. Neuroscience, June 2003, 4(6), pp. 469-480.

  11. S. S. Shanbhag, G. R. Udupi, K. M. Patil, K. Ranganath, Quantitative Analysis of Diffusion Weighted MR Images of Intracerebral Haemorrhage by Signal Intensity Gradient Technique, International Journal of Medical Imaging, 2013, 1(1), pp. 12-18.

  12. S. S. Shanbhag, G. R. Udupi, K. M. Patil, K. Ranganath, Quantitative Analysis of Brain Diffusion Weighted MR Images for Infarct Using Signal Intensity Gradient Technique, International Journal of Biotechnology, 2013, 111, pp. 215-223.

  13. J. M. Shen, X. W. Xia, W. G. Kang, J. J. Yuan, L. Sheng, The Use of MRI Apparent Diffusion Coefficient (ADC) in Monitoring the Development of Brain Infarction, BMC Medical Imaging, 2011, 11(2).

  14. W. G. Bradley Jr., MR Appearance of Hemorrhage in the Brain, Radiology, October 1993, 189(1), pp. 15-26.

  15. J. V. Hajnal, M. Doran, A. S. Hall, A. G. Collins, A. Oatridge, J. M. Pennock, I. R. Young, G. M. Bydder, MR Imaging of Anisotropically Restricted Diffusion of Water in the Nervous System: Technical, Anatomic, and Pathologic Considerations, Journal of Computer Assisted Tomography, January 1991,15(1), pp. 1-18.

  16. K. C. Sudeep, J. A. Majumdar, Novel Architecture for Real Time Implementation of Edge Detectors on FPGA, International Journal of Computer Science Issues, 2011, 8(1), pp. 193-202.

  17. G. S. Robinson, Edge Detection by Compass Gradient Masks, Computer Graphics and Image Processing, 6(5), October 1977, pp. 492-501.

  18. S. Haykin, Neural Networks: A Comprehensive Foundation, 1994, MacMillan College Publishing Co., New York.

  19. MATLAB Users Guide, The Math Works Inc., Natick, MA 01760-2098, 1998.

  20. P. D. Wasserman, Advanced Methods in Neural Computing, 1993, Van Nostrand Reinhold, New York, pp. 35-55.

  21. A. M. Molinaro, R. Simon, R. M. Pfeiffer, Prediction Error Estimation: a Comparison of Resampling Methods, Bioinformatics, 2005, 21(15), pp. 3301-3307.

Leave a Reply