- Open Access
- Authors : Saif Mohammed Saif Al Salti , Vinu Sherimon , Remya Revi K
- Paper ID : IJERTV11IS090100
- Volume & Issue : Volume 11, Issue 09 (September 2022)
- Published (First Online): 01-10-2022
- ISSN (Online) : 2278-0181
- Publisher Name : IJERT
- License: This work is licensed under a Creative Commons Attribution 4.0 International License
An Artificial Intelligence Lie Detection Model using ResNet-18 Network
Saif Mohammed Saif Al Salti, Alazhar Saif Musallam Al Sharji, Sherimon P.C
Arab Open University Muscat
Sultanate of Oman
University of Technology and Applied Sciences Muscat
Sultanate of Oman
Remya Revi K.
Saintgits College of Engineering Kottayam, Kerala
AbstractAn Artificial Intelligence (AI) Lie Detection system is developed using the deep learning technique which detects the expression of fear from face of a liar. The proposed lie detection model uses the transfer learning approach in the deep learning technique to classify the facial expressions into fear or neutral. A pre-trained lightweight deep Convolutional Neural Network architecture called ResNet-18 is used in this work. Publicly available benchmarked Facial Expression Recognition-2013 (FER-2013) dataset is used for the experimental analysis. The model performed well with a classification accuracy of 82.85%.
KeywordsLie detection; artificial intelligence; deep learning; transfer learning; micro-expression
Lie detection is significant process in criminal investigations, security verifications in airports and in many other situations. Psychophysiological or Polygraph is one of the widely used lie detection method . Other methods like Voice Stress Analysis (VSA), functional Magnetic Resonance Imaging (fMRI), Analysis of non-verbal behavior, etc. are yet to be validated. Many have questioned process of the polygraph lie testing method due to the lack of its scientific backing. Also, people are discovering several ways to cheat the polygraph test by consuming antiperspirant, sedatives, etc.
Analysis of facial micro-expressions is another significant lie detection method, where the investigators can detect lies by observing emotional leakage from the facial expressions of liars . When an analyst interrogates the people and observe the body and facial expression, it takes a lot of time for the observation and conclusion, and people may only be able to analyze one or two observations at a time. So, in this research, we develop an intelligent system that utilizes human facial micro-expressions to detect lies. Studies shows that the
emotional leakage in the form of fear can be observed from human faces when they are lying . This expression of fear arises in the face due to the tensions inside a person while lying. Hence in this work, we develop an Artificial Intelligence (AI) Lie Detection model that identifies the expression of fear from face of a liar. The proposed lie detection model uses the transfer learning approach in the deep learning technique to classify the facial expressions into fear or neutral. Here, we use a lightweight pre-trained Convolutional Neural Network (CNN) called ResNet-18 architecture to develop the model. The experiments are conducted using Facial Expression Recognition-2013 (FER- 2013) dataset .
The upcoming sections discuss about the related works, proposed method, experimental result analysis, and conclusion.
The following are the brief discussion of works related to research on the lie detection using AI and machine learning techniques.
Owayjan et al. developed a LabVIEW computer vision software based on a mathematical algorithm to analyze the facial micro-expressions to detect lies . Singh et al. developed an image processing method to detect lies based on eye blink rates. They used the Haar cascade algorithm to locate eye region from the face . Soumya Barathi designed a lie detection system by analyzing facial micro-expression, body language, and speech. She used Principal Component Analysis, Speeded Up Robust Features, and k-means algorithm in the lie detection system .
Fig. 1. Outline of the proposed method
Shen et al. analyzed the facial expression of fear to detect lies from videos. They used Random Forest, k-nearest neighbors, and bagging classification algorithms . Diaz et al. designed a machine learning model to detect lies from facial expressions. They extracted handcrafted features from the facial images as well as extracted deep features from the facial images. . Monaro et al. developed lie detection model by analyzing facial micro-expressions. Features are extracted using Open Face, and classifiers used are Support Vector Machine and Long Short-Term Memory network. In addition, they designed a 3-dimensional CNN architecture to extract features directly from the images and to classify the facial expressions .
The proposed lie detection system is a two-class classification problem, to classify the facial expressions into fear or neutral. The proposed method uses a deep neural network architecture for the classification, and outline of the method is displayed in Fig. 1.
The deep learning techniques which use CNNs give remarkable results in the computer vision tasks [11-13]. A CNN is designed to mimic the human visual system and basically used in image recognition tasks . To obtain accurate and reliable classification results, a CNN should be trained on huge quantity of labelled image data. But, in most real-life circumstances, there is a shortage in the availability of the number of labelled data, and hence, training the network from scratch is a difficult procedure. So, to overcome this problem, we can utilize a pre-trained CNN using the transfer learning approach [15-18]. The lie detection is also a small sample size classification problem, and hence, we use the transfer learning approach to develop the classification model. In this work, we utilize a lightweight pre-trained CNN architecture called ResNet-18, which is already trained on a large number of images from the ImageNet dataset. The specifications of ResNet-18 are: 11.7 million learnable parameters, depth is 44, size is 18 MB, input size is 224 x 224 x 3 . The layers of this pre-trained network learned the various basic features as well as discriminative features . Hence, in this research work, we fine-tune the ResNet-18 to classify the facial expressions into fear or neutral in the lie detection problem. To fine-tune the network using the transfer learning, as a first step we remove the Fully Connected (FC) layer of the ResNet-18, and is replaced with a new FC layer
with two classes for the lie detection problem. Then, we train the reorganized network using the dataset for the two-class classification.
RESULT AND DISCUSSION
A Graphic Processing Unit (GPU) based system having NVIDIA GTX 1060 card, 6.0 GB RAM is used to perform the experiments. The experiments and analysis are done on MATLAB platform with help of Deep Learning Toolbox. The next subsections briefs about the dataset used, evaluation metrics, training options, and experiments results and analysis.
The fear and neutral facial expression images from the Facial Expression Recognition-2013 (FER-2013) dataset are used for the experimental analysis. A total number of 10,000 images are used, i.e., 5,000 images each in fear and neutral classes. The images are in png format with a size of 48 x 48.
The performance evaluation metrics used for assessing the model are Accuracy, Precision, Recall, and F1-score are, and are calculated based on (1), (2), (3) and (4), respectively. They are calculated from the confusion matrix of the model using True Negative (TN), True Positive (TP), False Positive (FP), and False Negative (FN).
The options used to train the network is provided in Table
From the dataset, 60% are used for training the model, 20% are used for validating the model, and 20% are used for testing the model. The size of the images is resized to 224 x 224 x 3 resolution based on the input size of the ResNet-18 network.
TABLE I. TRAINING OPTIONS
No. of epochs
TABLE II. PERFORMANCE EVALUATION
Fig. 2 Confusion matrix of the model
Experimental Results and Analysis
The ResNet-18 network is fine-tuned on the dataset for classifying into the two categories using the above-mentioned training options. The confusion matrix obtained while testing the model is shown in Fig. 2. The values of the performance metrics are evaluated from the confusion matrix, and is given in Table 2. The analysis shows that, Accuracy, Precision, Recall, and F1-score achieved by the model are 82.85%, 0.781, 0.863, and 0.819, respectively. This experimental analysis shows that the proposed deep learning lie detection method achieved a decent performance.
In this work, a deep learning model is developed to detect lies by analyzing human facial expression. We utilized the emotional leakage in the form of fear to detect lies. An intelligent lie detection model using ResNet-18 deep neural network is developed. The model is evaluated using the (FER-2013) dataset. The model exhibited a good performance with a detection accuracy of 82.85%.
The research leading to these results has received funding from the Research Council (TRC) of the Sultanate of Oman under the Block Funding Program BFP/URG/ICT/21/152.
REFERENCES J. Winkler-Galicki, J. Bartkowiak-Wieczorek, D. Synowiec, R. Dbrowska, and E. Mdry, Polygraph analyses: technical and practical background, J. Med. Sci., vol. 91, no. 1, pp. 3946, 2022, doi: 10.20883/medical.e590.  R. Tamuli, S. Samantara, S. Sarkar, and S. Adhikari, Lie Detection Using Facial Micro-Expressions, Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol., vol. 4, no. 11, pp. 2530, 2020.  S. Jia, S. Wang, C. Hu, P. J. Webster, and X. Li, Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods, Front. Psychol., vol. 11, pp. 112, 2021, doi: 10.3389/fpsyg.2020.580287.  X. Shen, G. Fan, C. Niu, and Z. Chen, Catching a Liar Through Facial Expression of Fear, Front. Psychol., vol. 12, pp. 110, 2021, doi: 10.3389/fpsyg.2021.675097.  M. SAMBARE, FER-2013 Learn facial expressions from an image. https://www.kaggle.com/datasets/msambare/fer2013 (accessed Jul. 02, 2022).  M. Owayjan, A. Kashour, N. Al Haddad, M. Fadel, and G. Al Souki, The design and development of a lie detection system using facial micro-expressions, in 2nd International Conference on Advances in Computational Tools for Engineering Applications, ACTEA 2012, 2012, pp. 3338. doi: 10.1109/ICTEA.2012.6462897.  B. Singh, P. Rajiv, and M. Chandra, Lie detection using image processing, in Proceedings of the 2nd International Conference on Advanced Computing and Communication Systems (ICACCS
-2015), 2015, pp. 15. doi: 10.1109/ICACCS.2015.7324092. Soumya Barathi. C, Lie Detection based on Facial Micro Expression, Body Language and Speech Analysis, Int. J. Eng. Res. Technol., vol. 5, no. 02, pp. 337343, 2016, doi: 10.17577/ijertv5is020336.  N. Rodriguez-Diaz, D. Aspandi, F. M. Sukno, and X. Binefa, Machine learning-based lie detector applied to a novel annotated game dataset, Futur. Internet, vol. 14, no. 1, pp. 118, 2022, doi: 10.3390/fi14010002.  M. Monaro, S. Maldera, C. Scarpazza, G. Sartori, and N. Navarin, Detecting deception through facial expressions in a dataset of videotaped interviews: A comparison between human judges and machine learning models, Comput. Human Behav., vol. 127, pp. 110, 2022, doi: 10.1016/j.chb.2021.107063.  N. Sharma, V. Jain, and A. Mishra, An Analysis of Convolutional Neural Networks for Image Classification, in Procedia Computer Science, 2018, vol. 132, pp. 377384. doi: 10.1016/j.procs.2018.05.198.  F. Sultana, A. Sufian, and P. Dutta, Advancements in image classification using convolutional neural network, in 2018 Fourth International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN), 2018, pp. 122129. doi: 10.1109/ICRCICN.2018.8718718.  L. Chen, S. Li, Q. Bai, J. Yang, S. Jiang, and Y. Miao, Review of image classification algorithms based on convolutional neural networks, Remote Sens., vol. 13, no. 22, pp. 151, 2021, doi: 10.3390/rs13224712.  A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Commun. ACM |, vol. 60, no. 6, pp. 8490, 2017.  A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, CNN features off-the-shelf: An astounding baseline for recognition, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2014, pp. 512519. doi: 10.1109/CVPRW.2014.131.  J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, How transferable are features in deep neural networks?, in Advances in Neural Information Processing Systems, 2014, pp. 33203328.  N. Becherer, J. Pecarina, S. Nykl, and K. Hopkinson, Improving optimization of convolutional neural networks through parameter fine-tuning, Neural Comput. Appl., vol. 31, no. 8, pp. 3469 3479, 2019, doi: 10.1007/s00521-017-3285-0.  R. N. Dsouza, P. Y. Huang, and F. C. Yeh, Structural Analysis and Optimization of Convolutional Neural Networks with a Small
Sample Size, Sci. Rep., vol. 10, no. 1, pp. 113, 2020, doi: 10.1038/s41598-020-57866-2. K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, pp. 770778. doi: 10.1109/CVPR.2016.90.  M. Oquab, L. Bottou, I. Laptev, and J. Sivic, Learning and transferring mid-level image representations using convolutional neural networks, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014, pp. 17171724. doi: 10.1109/CVPR.2014.222.