Volume 17 Number 6
December 2020
Article Contents
Dilantha Haputhanthri, Gunavaran Brihadiswaran, Sahan Gunathilaka, Dulani Meedeniya, Sampath Jayarathna, Mark Jaime, Christopher Harshaw. Integration of facial thermography in eeg-based classification of asd. International Journal of Automation and Computing, vol. 17, no. 6, pp.837-854, 2020. https://doi.org/10.1007/s11633-020-1231-6
Citation: Dilantha Haputhanthri, Gunavaran Brihadiswaran, Sahan Gunathilaka, Dulani Meedeniya, Sampath Jayarathna, Mark Jaime, Christopher Harshaw. Integration of facial thermography in eeg-based classification of asd. International Journal of Automation and Computing, vol. 17, no. 6, pp.837-854, 2020. https://doi.org/10.1007/s11633-020-1231-6

Integration of Facial Thermography in EEG-based Classification of ASD

Author Biography:
  • Dilantha Haputhanthri is an undergraduate at Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. His research interests include bioinformatics, machine learning, data analytics and algorithm development. E-mail: dilantha.15@cse.mrt.ac.lk ORCID iD: 0000-0003-4319-9876

    Gunavaran Brihadiswaran is an undergraduate at Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. His research interests include emerging memory technologies, computer architecture, data structures and algorithms, and computational biology. E-mail: gunavaran.15@cse.mrt.ac.lk ORCID iD: 0000-0002-4086-3607

    Sahan Gunathilaka is an undergraduate in the Department of Computer Science and Engineering at the University of Moratuwa, Sri Lanka. His research interests include bioinformatics, machine learning, and image processing. E-mail: sahanruwanga.15@cse.mrt.ac.lk ORCID iD: 0000-0002-9876-5508

    Dulani Meedeniya received the Ph. D. degree in computer science from University of St Andrews, UK. She is a senior lecturer in Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. She is a Fellow of HEA(UK), MIET, MIEEE, and a Charted Engineer at EC (UK). Her research interests indude software modeling and design, workflow tool support for bioinformatics, data visualization and recommender systems. E-mail: dulanim@cse.mrt.ac.lk (Corresponding author) ORCID iD: 0000-0002-4520-3819

    Sampath Jayarathna received the Ph. D. degree in computer science from the Texas A&M University College Station, USA in 2016. He is an assistant professor of computer science at Old Dominion University, USA, where he is associated with Web Science and Digital Libraries (WS-DL) research group. He is a member of ACM, IEEE, and Sigma XI. His research interests include machine learning, information retrieval, data science, eye tracking, and brain-computer interfacing. E-mail: sampath@cs.odu.edu ORCID iD: 0000-0002-4879-7309

    Mark Jaime received the Ph. D. degree in psychology from Florida International University, USA in 2008, where he was trained in developmental psychobiology. He then completed two post-doctoral fellowships specializing in the neurocognitive development of autism spectrum disorder (ASD) and childhood intersensory processing in Department of Psychology, University of Miami, USA and Department of Psychology and Neuroscience at Dalhousie University, Canada. He is an associate professor of psychology at the Columbus Center of Indiana University-Purdue University, USA. He is a full member of the International Society for Autism Research. His research interests include characterizing psychophysiological mechanisms of social information processing in ASD and typical populations. He also engages in interdisciplinary collaborations focused on the development of neurocognitive markers of social impairment. E-mail: mjaime@iu.edu ORCID iD: 0000-0002-4166-3308

    Christopher Harshaw received the Ph. D. degree in developmental Science from Florida International University, USA. He then completed a postdoc at Indiana University, USA with a focus on Behavioral Neuroscience and Developmental Psychobiology. He is an assistant professor at Department of Psychology, University of New Orleans, USA. He directs the Mechanisms Underlying Sociality (MUS) Laboratory, which is currently focused on how deficits in temperature regulation may relate to differences in social cognitive abilities in Autism Spectrum Disorders.His research interest is gaining a better understanding of how bodily mechanisms influence cognition and behavior. E-mail: charshaw@uno.edu ORCID iD: 0000-0002-1400-7356

  • Received: 2019-10-31
  • Accepted: 2020-03-20
  • Published Online: 2020-05-11
  • Autism spectrum disorder (ASD) is a neurodevelopmental disorder affecting social, communicative, and repetitive behavior. The phenotypic heterogeneity of ASD makes timely and accurate diagnosis challenging, requiring highly trained clinical practitioners. The development of automated approaches to ASD classification, based on integrated psychophysiological measures, may one day help expedite the diagnostic process. This paper provides a novel contribution for classifing ASD using both thermographic and EEG data. The methodology used in this study extracts a variety of feature sets and evaluates the possibility of using several learning models. Mean, standard deviation, and entropy values of the EEG signals and mean temperature values of regions of interest (ROIs) in facial thermographic images were extracted as features. Feature selection is performed to filter less informative features based on correlation. The classification process utilizes Naïve Bayes, random forest, logistic regression, and multi-layer perceptron algorithms. The integration of EEG and thermographic features have achieved an accuracy of 94% with both logistic regression and multi-layer perceptron classifiers. The results have shown that the classification accuracies of most of the learning models have increased after integrating facial thermographic data with EEG.
  • 加载中
  • [1] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th ed., London, UK: American Psychiatric Association, 2013.
    [2] G. Dawson, S. Rogers, J. Munson, M. Smith, J. Winter, J. Greenson, A. Donaldson, J. Varley.  Randomized, controlled trial of an intervention for toddlers with autism: The Early Start Denver Model[J]. Pediatrics, 2010, 125(1): e17-e23. doi: 10.1542/peds.2009-0958
    [3] G. Dawson, E. J. H. Jones, K. Merkle, K. Venema, R. Lowy, S. Faja, D. Kamara, M. Murias, J. Greenson, J. Winter, M. Smith, S. J. Rogers, S. J. Webb.  Early behavioral intervention is associated with normalized brain activity in young children with autism[J]. Journal of the American Academy of Child & Adolescent Psychiatry, 2012, 51(11): 1150-1159. doi: 10.1016/j.jaac.2012.08.018
    [4] G. Brihadiswaran, D. Haputhanthri, S. Gunathilaka, D. Meedeniya, S. Jayarathna.  EEG-based processing and classification methodologies for autism spectrum disorder: A review[J]. Journal of Computer Science, 2019, 15(8): 1161-1183. doi: 10.3844/jcssp.2019.1161.1183
    [5] D. Haputhanthri, G. Brihadiswaran, S. Gunathilaka, D. Meedeniya, Y. Jayawardena, S. Jayarathna, M. Jaime. An EEG based channel optimized classification approach for autism spectrum disorder. In Proceedings of Moratuwa Engineering Research Conference, IEEE, Moratuwa, Sri Lanka, pp. 123–128, 2019. DOI: 10.1109/MERCon.2019.8818814.
    [6] S. D. Hill, E. A. Wagner, J. G. Shedlarski Jr, S. P. Sears.  Diurnal cortisol and temperature variation of normal and autistic children[J]. Developmental Psychobiology, 1977, 10(6): 579-583. doi: 10.1002/dev.420100612
    [7] R. M. J. Cotterill.  Fever in autistics[J]. Nature, 1985, 313(6002): 426-. doi: 10.1038/313426c0
    [8] L. K. Curran, C. J. Newschaffer, L. C. Lee, S. O. Crawford, M. V. Johnston, A. W. Zimmerman.  Behaviors associated with fever in children with autism spectrum disorders[J]. Pediatrics, 2007, 120(6): e1386-e1392. doi: 10.1542/peds.2007-0360
    [9] C. Harshaw, M. S. Blumberg, J. R. Alberts. Thermoregulation, energetics, and behavior. In APA Handbook of Comparative Psychology: Basic Concepts, Methods, Neural Substrate, and Behavior, J. Call, G. M. Burghardt, I. M. Pepperberg, C. T. Snowdon, T. Zentall, Washington, USA: American Psychological Association, pp. 931–952, 2017. DOI: 10.1037/0000011-045.
    [10] T. Deboer, I. Tobler.  Temperature dependence of EEG frequencies during natural hypothermia[J]. Brain Research, 1995, 670(1): 153-156. doi: 10.1016/0006-8993(94)01299-W
    [11] T. Deboer.  Brain temperature dependent changes in the electroencephalogram power spectrum of humans and animals[J]. Journal of Sleep Research, 1998, 7(4): 254-262. doi: 10.1046/j.1365-2869.1998.00125.x
    [12] E. Salazar-López, E. Domínguez, V. Juárez Ramos, J. de la Fuente, A. Meins, O. Iborra, G. Gálvez, M. A. Rodríguez-Artacho, E. Gómez-Milán.  The mental and subjective skin: Emotion, empathy, feelings and thermography[J]. Consciousness and Cognition, 2015, 34(): 149-162. doi: 10.1016/j.concog.2015.04.003
    [13] A. Truzzi, V. P. Senese, P. Setoh, C. Ripoli, G. Esposito.  In utero testosterone exposure influences physiological responses to dyadic interactions in neurotypical adults[J]. Acta Neuropsychiatrica, 2016, 28(5): 304-309. doi: 10.1017/neu.2016.15
    [14] K. Gotham, S. Risi, A. Pickles, C. Lord.  The autism diagnostic observation schedule: Revised algorithms for improved diagnostic validity[J]. Journal of Autism and Developmental Disorders, 2007, 37(4): 613-627. doi: 10.1007/s10803-006-0280-1
    [15] K. Gotham, A. Pickles, C. Lord.  Standardizing ADOS scores for a measure of severity in autism spectrum disorders[J]. Journal of Autism and Developmental Disorders, 2009, 39(5): 693-705. doi: 10.1007/s10803-008-0674-3
    [16] D. S. Mandell, R. F. Ittenbach, S. E. Levy, J. A. Pinto-Martin.  Disparities in diagnoses received prior to a diagnosis of autism spectrum disorder[J]. Journal of Autism and Developmental Disorders, 2007, 37(9): 1795-1802. doi: 10.1007/s10803-006-0314-8
    [17] BioHealth Group. Bio-health Informatics, [Online], Available: https://sites.google.com/cse.mrt.ac.lk/biohealth, September 06, 2019.
    [18] D. A. Meedeniya, I. D. Rubasinghe. A review of supportive computational approaches for neurological disorder identification. Interdisciplinary Approaches to Altering Neurodevelopmental Disorders, T. Wadhera, B. R. Ambedkar, D. Kakkar, Eds., Hershey, USA: IGI Gloabal, pp. 271–302, 2020.
    [19] C. C. Chernecky, B. J. Berger. Laboratory Tests and Diagnostic Procedures, 6th ed. Saint Louis, USA: Elsevier, 2013.
    [20] S. Noachtar, J. Rémi.  The role of EEG in epilepsy: a critical review[J]. Epilepsy & Behavior, 2009, 15(1): 22-33. doi: 10.1016/j.yebeh.2009.02.035
    [21] S. Leske, S. S. Dalal.  Reducing power line noise in EEG and MEG data via spectrum interpolation[J]. Neuroimage, 2019, 189(): 763-776. doi: 10.1016/j.neuroimage.2019.01.026
    [22] S. Deyashi, D. Banerjee, B. Chakraborty, D. Ghosh, J. Debnath. Application of CBR on Viral Fever Detection System (VFDS). In Proceedings of the 9th IEEE International Conference on Industrial Informatics, IEEE, Caparica, Portugal, pp. 660–665, 2011. DOI: 10.1109/INDIN.2011.6035037.
    [23] F. G. Filip, C. B. Zamfirescu, C. Ciurea. Computer-Supported Collaborative Decision-Making, Cham, USA: Springer, pp. 1−69, 2017.
    [24] K. Kawamoto, C. A. Houlihan, E. A. Balas, D. F. Lobach.  Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success[J]. BMJ, 2005, 330(7494): 765-. doi: 10.1136/bmj.38398.500764.8F
    [25] S. De Silva, S. Dayarathna, G. Ariyarathne, D. Meedeniya, S. Jayarathna, A. M. P. Michalek, Gavindya Jayawardena. A rule-based system for ADHD identification using eye movement data. In Proceedings of 2019 Moratuwa Engineering Research Conference, IEEE, Moratuwa, Sri Lanka, pp. 538–543, 2019. DOI: 10.1109/MERCon.2019.8818865.
    [26] D. Rubasinghe, D. A. Meedeniya. Automated neuroscience decision support framework. Deep Learning Techniques for Biomedical and Health Informatics, B. Agarwal, V. E. Balas, L. C. Jain, R. C. Poonia, Manisha, Eds., Academic Press, pp. 305−326, 2020. DOI: 10.1016/B978-0-12-819061-6.00013-6.
    [27] I. Rish. An empirical study of the naive Bayes classifier. In Proceedings of IJCAI Workshop on Empirical Methods in Artificial Intelligence, Morgan Kaufmann, Seattle, USA, pp. 41−46, 2001.
    [28] S. Thapaliya, S. Jayarathna, M. Jaime. EvalUATING THE EEG and eye movements for autism spectrum disorder. In Proceedings of IEEE International Conference on Big Data, IEEE, Seattle, WA, USA, pp. 2328−2336, 2018. DOI: 10.1109/BigData.2018.8622501.
    [29] E. Grossi, C. Olivieri, M. Buscema.  Diagnosis of autism through EEG processed by advanced computational algorithms[J]. Computer methods and programs in biomedicine, 2017, 142(C): 73-79. doi: 10.1016/j.cmpb.2017.02.002
    [30] S. Dreiseitl, L. Ohno-Machado.  Logistic regression and artificial neural network classification models: A methodology review[J]. Journal of Biomedical Informatics, 2002, 35(5–6): 352-359. doi: 10.1016/S1532-0464(03)00034-0
    [31] A. Liaw, M. Wiener.  Classification and regression by randomForest[J]. R News, 2002, 2(3): 18-22.
    [32] W. J. Bosl, H. Tager-Flusberg, C. A. Nelson.  EEG analytics for early detection of autism spectrum disorder: A data-driven approach[J]. Scientific Reports, 2018, 8(1): 6828-. doi: 10.1038/s41598-018-24318-x
    [33] T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, Q. L. Liao.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review[J]. International Journal of Automation and Computing, 2017, 14(5): 503-519. doi: 10.1007/s11633-017-1054-2
    [34] Z. J. Yao, J. Bi, Y. X. Chen.  Applying deep learning to individual and community health monitoring data: A survey[J]. International Journal of Automation and Computing, 2018, 15(6): 643-655. doi: 10.1007/s11633-018-1136-9
    [35] M. Ahmadlou, H. Adeli, A. Adeli.  Fuzzy Synchronization Likelihood-wavelet methodology for diagnosis of autism spectrum disorder[J]. Journal of Neuroscience Methods, 2012, 211(2): 203-209. doi: 10.1016/j.jneumeth.2012.08.020
    [36] R. Djemal, K. AlSharabi, S. Ibrahim, A. Alsuwailem.  EEG-based computer aided diagnosis of autism spectrum disorder using wavelet, entropy, and ANN[J]. BioMed Research International, 2017, 2017(): 9816591-. doi: 10.1155/2017/9816591
    [37] S. Afrakhteh, M. R. Mosavi, M. Khishe, A. Ayatollahi.  Accurate classification of EEG signals using neural networks trained by hybrid population-physic-based algorithm[J]. International Journal of Automation and Computing, 2020, 17(1): 108-122. doi: 10.1007/s11633-018-1158-3
    [38] S. Siuly, V. Bajaj, A. Sengur, Y. C. Zhang.  An advanced analysis system for identifying alcoholic brain state through EEG signals[J]. International Journal of Automation and Computing, 2019, 16(6): 737-747. doi: 10.1007/s11633-019-1178-7
    [39] S. De Silva, S. Dayarathna, G. Ariyarathne, D. Meedeniya, S. Jayarathna.  A survey of attention deficit hyperactivity disorder identification using psychophysiological data[J]. International Journal of Online and Biomedical Engineering, 2019, 15(13): 61-76. doi: 10.3991/ijoe.v15i13.10744
    [40] W. Bosl, A. Tierney, H. Tager-Flusberg, C. Nelson.  EEG complexity as a biomarker for autism spectrum disorder risk[J]. BMC Medicine, 2011, 9(1): 18-. doi: 10.1186/1741-7015-9-18
    [41] S. Jayarathna, Y. Jayawardana, M. Jaime, S. Thapaliya. Electroencephalogram (EEG) for delineating objective measure of autism spectrum disorder. Computational Models for Biomedical Reasoning and Problem Solving, C. H. Chen, S. C. S. Cheung, Eds., Hershey, PA, USA: IGI Global, pp. 34-65, 2019.
    [42] M. Ahmadlou, H. Adeli, A. Adeli.  Fractality and a wavelet-chaos-neural network methodology for EEG-based diagnosis of autistic spectrum disorder[J]. Journal of Clinical Neurophysiology, 2010, 27(5): 328-333. doi: 10.1097/WNP.0b013e3181f40dc8
    [43] M. Ahmadlou, H. Adeli, A. Adeli.  Improved visibility graph fractality with application for the diagnosis of Autism Spectrum Disorder[J]. Physica A: Statistical Mechanics and its Applications, 2012, 391(20): 4720-4726. doi: 10.1016/j.physa.2012.04.025
    [44] V. Hus, C. Lord.  The autism diagnostic observation schedule, module 4: Revised algorithm and standardized severity scores[J]. Journal of Autism and Developmental Disorders, 2014, 44(8): 1996-2012. doi: 10.1007/s10803-014-2080-3
    [45] A. McCrimmon, K. Rostad.  Test review: Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) manual (Part II): Toddler module[J]. Journal of Psychoeducational Assessment, 2014, 32(1): 88-92. doi: 10.1177/0734282913490916
    [46] LiveAmp (32 channels)[Apparatus]. (2018). Gilching, Germany: Brain Products GmbH. [Online], Available: https://www.brainproducts.com, September 25, 2019.
    [47] L. Leuchs. Choosing your reference – and why it matters. Brain Products Press Release, 2019. [Online], Available: https://pressrelease.brainproducts.com/referencing/#21, September 20, 2019.
    [48] R. Kher.  Signal processing techniques for removing noise from ECG signals[J]. Journal of Biomedical Engineering and Research, 2019, 1(1): 1-9.
    [49] R. Milo, P. Jorgensen, U. Moran, G. Weber, M. Springer.  BioNumbers-the database of key numbers in molecular and cell biology[J]. Nucleic Acids Research, 2009, 38(Suppl 1): D750-D753. doi: 10.1093/nar/gkp889
    [50] S. Sanei, J. A. Chambers. EEG Signal Processing, United Kingdom: John Wiley & Sons, 2013.
    [51] M. A. Hall. Correlation-based Feature Selection for Machine Learning, Ph. D. dissertation, University of Waikato, Hamilton, USA, 1999.
    [52] E. Y. K. Ng, G. J. L. Kawb, W. M. Chang.  Analysis of IR thermal imager for mass blind fever screening[J]. Microvascular Research, 2004, 68(2): 104-109. doi: 10.1016/j.mvr.2004.05.003
    [53] A. C. Marinescu, S. Sharples, A. C. Ritchie, T. Sánchez López, M. McDowell, H. P. Morvan.  Physiological parameter response to variation of mental workload[J]. Human Factors: The Journal of the Human Factors and Ergonomics Society, 2018, 60(1): 31-56. doi: 10.1177/0018720817733101
    [54] C. K. L. Or, V. Duffy.  Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement[J]. Occupational Ergonomics, 2007, 7(2): 83-94.
    [55] D. Shearn, E. Bergman, K. Hill, A. Abel, L. Hinds.  Facial coloration and temperature responses in blushing[J]. Psychophysiology, 1990, 27(6): 687-693. doi: 10.1111/j.1469-8986.1990.tb03194.x
    [56] S. Bejerot, J. M. Eriksson, E. Mörtberg.  Social anxiety in adult autism spectrum disorder[J]. Psychiatry Research, 2014, 220(1–2): 705-707. doi: 10.1016/j.psychres.2014.08.030
    [57] G. Bradski. The OpenCV library. Dr. Dobb′s Journal of Software Tools, vol. 25, no. 11, pp. 120-126, 2000.
    [58] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I. H. Witten.  The WEKA data mining software: An update[J]. ACM SIGKDD Explorations Newsletter, 2009, 11(): 10-18. doi: 10.1145/1656274.1656278
    [59] M. Saltan, S. Terzi.  Modeling deflection basin using artificial neural networks with cross-validation technique in back calculating flexible pavement layer moduli[J]. Advances in Engineering Software, 2008, 39(7): 588-592. doi: 10.1016/j.advengsoft.2007.06.002
    [60] G. Ariyarathne, S. De Silva, S. Dayarathna, D. Meedeniya, S. Jayarathna. ADHD identification using convolutional neural network with seed-based approach for fMRI data. In Proceedings of the 9th International Conference on Software and Computer Applications, ACM, Malaysia, pp. 31–35, 2020.
    [61] S. De Silva, S. Dayarathna, G. Ariyarathne, D. Meedeniya, S. Jayarathna.  fMRI feature extraction model for ADHD classification using convolutional neural network[J]. International Journal of E-Health and Medical Communications, 2020, 12(1): 6-.
    [62] I. Rubasinghe, D. Meedeniya.  Ultrasound nerve segmentation using deep probabilistic programming[J]. Journal of ICT Research and Applications, 2019, 13(3): 241-256. doi: 10.5614/itbj.ict.res.appl.2019.13.3.5
  • 加载中
  • [1] Senuri De Silva, Sanuwani Dayarathna, Gangani Ariyarathne, Dulani Meedeniya, Sampath Jayarathna, Anne M. P. Michalek. 

    Computational Decision Support System for ADHD Identification

    . International Journal of Automation and Computing, 2021, 18(2): 233-255.  doi: 10.1007/s11633-020-1252-1
    [2] Ahmad Alqwadri, Mohammad Azzeh, Fadi Almasalha. Application of Machine Learning for Online Reputation Systems . International Journal of Automation and Computing, 2021, 18(): 1-11.  doi: 10.1007/s11633-020-1275-7
    [3] Soni Lanka Karri, Liyanage Chandratilak De Silva, Daphne Teck Ching Lai, Shiaw Yin Yong. Identification and Classification of Driving Behaviour at Signalized Intersections Using Support Vector Machine . International Journal of Automation and Computing, 2021, 18(): 1-12.  doi: 10.1007/s11633-021-1295-y
    [4] Diclehan Karakaya, Oguzhan Ulucan, Mehmet Turkan. Electronic Nose and Its Applications: A Survey . International Journal of Automation and Computing, 2020, 17(2): 179-209.  doi: 10.1007/s11633-019-1212-9
    [5] Maryam Aljanabi, Mohammad Shkoukani, Mohammad Hijjawi. Ground-level Ozone Prediction Using Machine Learning Techniques: A Case Study in Amman, Jordan . International Journal of Automation and Computing, 2020, 17(5): 667-677.  doi: 10.1007/s11633-020-1233-4
    [6] Ziheng Chen, Hongshik Ahn. Item Response Theory Based Ensemble in Machine Learning . International Journal of Automation and Computing, 2020, 17(5): 621-636.  doi: 10.1007/s11633-020-1239-y
    [7] N. R. Nayak, P. K. Dash, R. Bisoi. A Hybrid Time Frequency Response and Fuzzy Decision Tree for Non-stationary Signal Analysis and Pattern Recognition . International Journal of Automation and Computing, 2019, 16(3): 398-412.  doi: 10.1007/s11633-018-1113-3
    [8] Siuly Siuly, Varun Bajaj, Abdulkadir Sengur, Yanchun Zhang. An Advanced Analysis System for Identifying Alcoholic Brain State Through EEG Signals . International Journal of Automation and Computing, 2019, 16(6): 737-747.  doi: 10.1007/s11633-019-1178-7
    [9] Danko Nikolić. Why Deep Neural Nets Cannot Ever Match Biological Intelligence and What To Do About It? . International Journal of Automation and Computing, 2017, 14(5): 532-541.  doi: 10.1007/s11633-017-1093-8
    [10] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, Qianli Liao. Why and When Can Deep-but Not Shallow-networks Avoid the Curse of Dimensionality:A Review . International Journal of Automation and Computing, 2017, 14(5): 503-519.  doi: 10.1007/s11633-017-1054-2
    [11] Xue-Fang Li,  Jian-Xin Xu. Lifted System Framework for Learning Control with Different Trial Lengths . International Journal of Automation and Computing, 2015, 12(3): 273-280.  doi: 10.1007/s11633-015-0882-1
    [12] Pavla Bromová,  Petr Škoda,  Jaroslav Vážný. Classification of Spectra of Emission Line Stars Using Machine Learning Techniques . International Journal of Automation and Computing, 2014, 11(3): 265-273.  doi: 10.1007/s11633-014-0789-2
    [13] Li-Jie Zhao, Tian-You Chai, De-Cheng Yuan. Selective Ensemble Extreme Learning Machine Modeling of Effluent Quality in Wastewater Treatment Plants . International Journal of Automation and Computing, 2012, 9(6): 627-633 .  doi: 10.1007/s11633-012-0688-3
    [14] Xu-Hui Bu,  Zhong-Sheng Hou. Stability of Iterative Learning Control with Data Dropouts via Asynchronous Dynamical System . International Journal of Automation and Computing, 2011, 8(1): 29-36.  doi: 10.1007/s11633-010-0551-3
    [15] Shu-Xi Liu, Ming-Yu Wang, Yu-Guang Chen, Shan Li. A Novel Fuzzy Direct Torque Control System for Three-level Inverter-fed Induction Machine . International Journal of Automation and Computing, 2010, 7(1): 78-85.  doi: 10.1007/s11633-010-0078-7
    [16] Tian-Guo Jin, Feng-Yang Bi. A Computer-aided Design System for Framed-mould in Autoclave Processing . International Journal of Automation and Computing, 2010, 7(3): 261-270.  doi: 10.1007/s11633-010-0501-0
    [17] Dieter D. Genske, Dongbin Huang, Ariane Ruff. An Assessment Tool for Land Reuse with Artificial Intelligence Method . International Journal of Automation and Computing, 2010, 7(1): 1-8.  doi: 10.1007/s11633-010-0001-2
    [18] Computational Intelligence and Games:Challenges and Opportunities . International Journal of Automation and Computing, 2008, 5(1): 45-57.  doi: 10.1007/s11633-008-0045-8
    [19] Alma Lilia Garcia-Almanza,  Edward P. K. Tsang. Evolving Decision Rules to Predict Investment Opportunities . International Journal of Automation and Computing, 2008, 5(1): 22-31.  doi: 10.1007/s11633-008-0022-2
    [20] L. Meng,  Q. H. Wu. Fast Training of Support Vector Machines Using Error-Center-Based Optimization . International Journal of Automation and Computing, 2005, 2(1): 6-12.  doi: 10.1007/s11633-005-0006-4
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Figures (13)  / Tables (4)


Abstract Views (400) PDF downloads (38) Citations (0)

Integration of Facial Thermography in EEG-based Classification of ASD

Abstract: Autism spectrum disorder (ASD) is a neurodevelopmental disorder affecting social, communicative, and repetitive behavior. The phenotypic heterogeneity of ASD makes timely and accurate diagnosis challenging, requiring highly trained clinical practitioners. The development of automated approaches to ASD classification, based on integrated psychophysiological measures, may one day help expedite the diagnostic process. This paper provides a novel contribution for classifing ASD using both thermographic and EEG data. The methodology used in this study extracts a variety of feature sets and evaluates the possibility of using several learning models. Mean, standard deviation, and entropy values of the EEG signals and mean temperature values of regions of interest (ROIs) in facial thermographic images were extracted as features. Feature selection is performed to filter less informative features based on correlation. The classification process utilizes Naïve Bayes, random forest, logistic regression, and multi-layer perceptron algorithms. The integration of EEG and thermographic features have achieved an accuracy of 94% with both logistic regression and multi-layer perceptron classifiers. The results have shown that the classification accuracies of most of the learning models have increased after integrating facial thermographic data with EEG.

Dilantha Haputhanthri, Gunavaran Brihadiswaran, Sahan Gunathilaka, Dulani Meedeniya, Sampath Jayarathna, Mark Jaime, Christopher Harshaw. Integration of facial thermography in eeg-based classification of asd. International Journal of Automation and Computing, vol. 17, no. 6, pp.837-854, 2020. https://doi.org/10.1007/s11633-020-1231-6
Citation: Dilantha Haputhanthri, Gunavaran Brihadiswaran, Sahan Gunathilaka, Dulani Meedeniya, Sampath Jayarathna, Mark Jaime, Christopher Harshaw. Integration of facial thermography in eeg-based classification of asd. International Journal of Automation and Computing, vol. 17, no. 6, pp.837-854, 2020. https://doi.org/10.1007/s11633-020-1231-6
    • Autism spectrum disorder (ASD) is a complex neurodevelopmental condition characterized by deficits in social, communicative, and repetitive behavior[1]. ASD is a heterogeneous condition wherein the symptoms and their severity are unique to each individual. Currently, there are no medical tests for ASD.

      Studies reveal that early diagnosis has a positive impact on the child's response to treatment[2, 3]. However, ASD symptomatology makes early detection complex, requiring a comprehensive evaluation of the child, including input from multi-disciplinary experts. This process can take a considerable amount of time, which results in a delayed intervention. In order to overcome these shortcomings, several behavior-independent, data-driven classification approaches have been proposed over the years which utilize machine learning in the statistical analysis of physiological measures such as electroencephalogram (EEG), and functional magnetic resonance imaging (fMRI) for the classification of ASD[4]. Such approaches have the potential to reduce the time and cost of ASD diagnosis and serve as a support system in the diagnostic process.

      Neuroimaging techniques such as EEG and fMRI rely heavily on brain dynamics. This study proposes a novel approach by integrating physiological measures, facial thermographic data with EEG, to improve the classification of ASD as an extension of the tool ASDGenus that we developed in our previous study[5].

      Although several ASD classification approaches based on EEG data have been proposed in related studies, none of them has attempted to incorporate thermographic data. Moreover, literature provides strong evidence for the existence of thermoregulatory abnormalities in ASD and the influence of temperature on EEG activity. Thus, the primary motivation for the integration of thermographic data with EEG is based on evidence of thermoregulatory abnormalities in ASD[6], including anecdotal reports of the “fever effect”, in which children with ASD sometimes show improvements in social and communicative abilities during fever[7, 8]. Such anecdotal reports have received relatively little attention from researchers to date but nevertheless warrant attention, given that temperature impacts many parameters of neuronal functioning[9], including EEG activity[10, 11]. Interestingly, two previous studies employing infrared thermography of the face found that temperature change in specific facial regions correlated with social-cognitive abilities in non-ASD samples[12, 13]. Thus, infrared thermography during social interaction may enhance the feature sets used for the classification of ASD.

      Accordingly, the data had to be processed in multiple phases before reaching the classification phase as we used raw EEG signals and facial thermography. Multiple techniques were employed to remove the artifacts from the raw EEG signal during the pre-processing phase. In the feature extraction phase, we calculated mean and standard deviation which are the statistical attributes of a signal and the entropy values[4]. For the facial thermography, after identifying regions of interest (ROIs) of the face to represent the skin surface temperature, the mean temperature of each ROI was calculated. Finally, the extracted features are used to derive the feature sets for the classification. The proposed model can be used as a decision support system (DSS) for ASD identification.

    • Current clinical diagnostic criteria require the presence or absence of specific behaviors in the child. This analysis involves direct observation of the child and the collection of information from the parents. ASD diagnosis is a multi-step process. In most cases, parents and/or teachers are the ones who initially raise concerns about a child's social development. These children are then screened during well-baby check-ups to ensure that there are no delays in their development. If screening reveals delays, then children undergo a comprehensive evaluation by specialists. Instruments such as Autism Diagnostic Observation Schedule, 2nd edition (ADOS-2) and the Autism Diagnostic Interview-Revised (ADI-R) are used to assess the severity of ASD[14, 15]. Such comprehensive evaluations can take a prolonged period of time. Additional factors such as disparities in accurate diagnosis among minority groups and overlapping symptomatology of ASD and ADHD can make timely diagnosis even more challenging[16]. Thus, there is a need for a novel approach that can drive costs down and expedite the diagnostic process for children and families impacted by ASD.

    • Healthcare challenges are becoming more diverse and complex, making them almost impossible to be solved by medical practitioners alone. However, unlike in the past decades, we have access to an enormous amount of medical data and modern technology. Bio-health informatics is an interdisciplinary field that utilizes this advantage by borrowing expert knowledge from the fields of biology, mathematics and computer science with the goal of analyzing and understanding biological data to address critical healthcare problems[17]. Research in bio-health informatics involves, but is not limited to, mathematical modeling and simulation, developing tools to support data analysis, and designing decision support systems to assist clinical diagnosis[18]. This field of research has made impactful contributions to neurodevelopmental disorder diagnosis, exhibiting a promising future.

    • EEG is a technique of recording the spontaneous electrical activity of the brain during a given time period, measuring the voltage fluctuation of the ionic current within the neurons using multiple electrodes placed on the scalp[4]. Abnormalities in these recordings can be observed and used to diagnose epilepsy, depth of anesthesia, coma, encephalopathies, and brain death. Before magnetic resonance imaging (MRI) and computed tomography (CT), EEG was employed as a first-line diagnosis technique for tumors, strokes, and other brain disorders[19]. Significantly lower cost hardware, availability, safety, and non-invasiveness are some of the advantages of EEG compared to the other techniques that are used to study the brain. At the same time, poor signal-to-noise ratio, low spatial resolution, and the execution time for a test are considered to be the main disadvantages[20].

      Poor signal-to-noise ratio is one of the major challenges related to EEG data processing, resulting in the need for a considerable amount of experience to clinically interpret the EEG readings. Raw EEG data is usually contaminated with different types of artifacts, mainly categorized as biological and environmental. Environmental artifacts are mainly caused by sources outside the body, such as 50–60 Hz AC line artifact (based on the standard frequency of the country) due to the poor grounding of the electrodes and other electromagnetic interferences to the electrodes[21]. Biological artifacts are mainly caused by eye blinks, heart beating (cardiac) and muscle movements of the participant.

      Thermography is a method that can be used to detect patterns of heat and blood flow by detecting infrared radiation emitted from the skin. This method involves an infrared thermographic camera to detect surface radiated heat that can be captured and visualized as a “thermogram”. Thermography is a useful tool for studying variations in skin surface temperature. Fever screening using infrared thermal detection systems[22] can be considered an existing application of thermography in medical diagnosis.

    • Decision making includes three key components: intelligence, design, and choice. However, several challenges are associated with system engineering such as cognitive constraints, cost issues, temporal constraints, communication or collaboration limits, and low trust issues in decision making[23]. In order to overcome such hurdles, computers are integrated with systems to support decision making that is collectively called DSSs. Such systems have been used widely in the healthcare domain, especially in the field of neuroscience, because of their ability to analyse a large amount of data efficiently and evaluate infinite possibilities. DSSs compare the patient profile with an existing knowledge base and generate patient-specific recommendations. Studies have shown that DSSs have made a significant impact in reducing serious medical errors, and improving services and practice[24-26]. ASDGenus, a DSS to classify ASD with the minimum number of EEG channels proposed in our previous work[5], demonstrates the usefulness of automated systems in classifying neurodevelopmental disorders.

    • Deep learning, which integrates machine learning with artificial intelligence, has revolutionized learning and problem-solving in the medical field[26]. It provides us with the required tools and know-how to identify a wider range of patterns and to build complex models from existing data. The machine learning based applications mainly have three phases: 1) data pre-processing, 2) feature extraction, and 3) classification[4]. In the pre-processing phase, noise filtering and data transformation are performed. After pre-processing the data, different features are extracted, and less informative features are removed. Finally, in the classification phase, a model is trained, and the results are evaluated. Even though the interpretability of the results produced by such models remains questionable, the extent to which they expand the boundaries of learning and decision making outweighs the existing drawbacks.

      The Naïve Bayes classifier is considered as the gold standard by the machine learning researchers since it is used as a benchmark against which the models are compared. It is based on Bayes' theorem and considered naïve because of the naïve assumption of the class conditional independence[27]. The Naïve Bayes algorithm has been able to produce reasonable and satisfactory results, even though the assumption does not hold in many real-world problems. The Naïve Bayes classifier requires relatively smaller amount of training data, and because of its scalability, simplicity, easy implementation, and speed, many classification problems are addressed using it, including ASD classification[28, 29].

      Logistic regression originated in the field of statistics during the 19th century. The machine learning community has extensively explored and employed logistic regression frequently for binary classification problems, including classifying ASD and no-ASD[30]. Logistic regression models are trained while reducing the loss, and for the prediction, the output is a probability where a threshold is set in order to get the output class. Logistic regression is considered to be a simple algorithm which requires comparatively low computational power. The studies[28, 29] have used logistic regression to diagnose ASD.

      Random forest is an ensemble algorithm which generates the overall result by combining the results of multiple models built using the input data[31]. It develops a collection of decision trees from randomized subsets of the training data, and during classification, the result is generated by combining the results of each decision tree. The objective of building several models is to increase the quality of the result by reducing the noise and other biases. Although building many decision trees can delay the classification, studies have obtained reasonable results using random forest for ASD classification[29, 32].

      Neural networks are built imitating the human nervous system. A typical neural network consists of an input layer, one or more hidden layers and an output layer which can perform classification, clustering, prediction, etc. The weighted sums of the inputs are fed into each node of the first hidden layer, and the output of each node in the layer is decided by the activation function. If x is the input vector, w is the weight vector and f is the activation function, then the output of a node is defined by f(wx). The weighted sums of the outputs of the nodes in the first hidden layer become the inputs to the next hidden layer (if one exists). The final output is generated by the output layer. Despite the neural network's ability to build complex relationships and to avoid issues such as the curse of dimensionality[33], their “black box” nature hinders the interpretability of the system. However, neural networks are widely used for EEG based health applications[34] including ASD classification[28, 35, 36].

    • Neuroimaging is a term for a set of methodologies and techniques, which could image the structure and functionalities of the nervous system. Some of the popular neuroimaging techniques are fMRI, positron emission tomography, magnetoencephalography (MEG) and electroencephalography (EEG)[18, 37]. It is challenging to use these techniques in creating the knowledge base for computational learning models, because of the inherited noise which affects the signal. Hence, neuroimaging data need to be pre-processed before utilizing them to extract features and train the learning model[38]. However, the chances of losing valuable information along with the noise during the process are high.

      Different techniques such as neuroimaging correction, noise removal, normalization, smoothing, and image registration are used in pre-processing the electrophysiological data[4, 18, 39]. Machine learning based ASD classification approaches employ many methods to extract features from EEG data, entropy functions such as multiscale entropy (MSE) and Shannon's entropy, discrete wavelet transform, fast Fourier transform (FFT) and statistical methods such as mean, standard deviation, etc.[4]

    • Recent related works on computational models for ASD classification have made significant strides[18]. For example, Bosl et al.[32] used a data-driven approach for ASD detection. They collected 64 channels or 128 channels of EEG data from 188 participants. EEG was recorded while the experimenter blew bubbles. From this sample, 89 were low-risk controls (LRC), three of which were diagnosed with ASD. In the high risks for ASD (HRA) group, there were 99 subjects among which 32 were diagnosed with ASD. Nine features namely sample entropy, detrended fluctuation analysis, entropy derived from recurrence plot, max line length, mean line length, recurrence rate, determinism, laminarity, and trapping time were extracted for each channel. Then the less informative features were removed using a recursive feature elimination algorithm. Using support vector machine (SVM), this approach managed to distinguish ASD subjects from LRC subjects with 100% accuracy. The HRA subjects were also classified with high accuracy but classifying subjects who are placed close to the decision boundary were found to be challenging. One significant contribution of this study was generating severity scores in the range of 1 to 10 with a high resemblance to the actual ADOS-2 scores. Unlike most other studies, the availability of data from many subjects increases the reliability and statistical significance of the results. One of the important goals of similar research is diagnosing ASD as early as possible. However, the datasets used by most of the other studies were collected from older subjects and hence might not reflect the brain functionality of younger children. Further, the participants were between 3 and 36 months of age, which is a unique feature of this study with regard to the above issue.

      In a previous study, Bosl et al[40]. demonstrated that modified multiscale entropy (mMSE) could be used as a biomarker for ASD classification. Their dataset consisted of 64 channel EEG data from 79 subjects between 6 and 24 months of age including 46 HRA and 33 controls. The data were collected multiple times from the same subjects at different ages and they were considered independent data. In total, the dataset consists of 143 independent sets of data. Thus, similar to their work in [32], the results have relatively higher statistical power due to sample size. The data were collected while the experimenters were blowing bubbles to draw the infants′ attention. SVM, K-nearest neighbors (K-NN) and Naïve Bayes classifiers have used to distinguish the ASD subjects from typically developing ones. Two unique features of this study were: 1) girls and boys were classified separately and as a unified group and 2) children at different ages of 6, 9, 12, 18 and 24 months were classified separately. When boys and girls were classified separately, SVM produced the best accuracy of 100% for the boys at the age group of 9 months and 80% accuracy for the girls at the age group of 6 months. When boys and girls were classified together, k-NN produced an accuracy of 90% for the age groups of 9 and 18 months.

      In [41], several experiments were carried out to analyze the functional connectivity and temporal relationship between ASD and brain activity for ASD classification and severity score prediction. EEG data was recorded from 17 subjects between 5 and 17 years of age among which eight were diagnosed with ASD. Two sets of training data were generated to capture short-term and long-term trends in the temporal relationship. The EEG signals were decomposed into five frequency bands (delta, theta, alpha, beta, and gamma) and five corresponding feature sets were created. Each feature set consisted of the amplitude and power for each electrode. For the first experiment, the features of all the channels were used. Out of the 43 classifiers that were used, accuracies of 98.06% and 98.00% were produced by JRip and random forest, respectively, when evaluating the short-term dependencies. For the second experiment, only the selected channels of F7, F8, T7, T8, TP9, TP10, P7, P8, C3, and C4 were used. The random forest classifier produced 97.04% accuracy. The ADOS-2 scores were also predicted where bagging yielded the highest correlation coefficient of 0.9079 with a root mean squared error of 2.93. To evaluate the long-term dependencies, a convolutional neural network (CNN) was used and achieved accuracy of over 90%. This is one of the few works which successfully generated a severity score similar to the ADOS score to measure the severity of ASD in addition to classifying them as ASD versus no-ASD. Evaluating 43 classifiers for classification is another unique feature of Jayarathna et al. study. However, the smaller dataset does impact the statistical power of the results.

      Thapaliya et al.[28] have proposed an ASD classification methodology that combines EEG and eye-tracking data collected from 34 subjects while they watched video clips that engendered joint attention. The EEG data were recorded from 128 channels. This was the first attempt to combine two different datasets (EEG and eye-tracking) for the classification of ASD. Multiple sets of features combining eye-tracking data with various EEG features (e.g. mean and standard deviation, discrete fast Fourier transform, and entropy) were used to train the learning models. Feature selection was performed using principal component analysis (PCA) and sequential feature selection. Several classifiers including SVM, logistic regression, deep neural network (DNN), and Gaussian Naïve Bayes were used for classification. A resulting 100% accuracy was achieved by the logistic regression algorithm while using EEG standard deviation and eye-tracking data without PCA as the features. Although these results are promising, acquisition of 128 channels of EEG data from toddlers (as an early screening method for ASD) can impose significant challenges in real-world practice.

      Grossi et al.[29] have proposed a multi-scale ranked organizing map (MS-ROM) coupled with an implicit function as squashing time (I-FAST) algorithm for the classification of ASD. One distinct feature of this integrated algorithm is that it does not require any preliminary pre-processing. Resting-state EEG data were collected from 25 participants (15 ASD and 10 typically developing) while they were closing and opening their eyes. The I-FAST algorithm consists of three phases: squashing phase, noise elimination phase, and classification phase. In the squashing phase, features were extracted using MS-ROM and multiscale entropy (MSE). In the noise elimination phase, irrelevant features were filtered using the TWIST (training with input selection and testing) algorithm. Finally, in the classification phase, seven learning algorithms (sine net neural networks, logistic regression, sequential minimal optimization (SMO), K-NN, K-contractive map, Naïve Bayes, and random forest) were used as classifiers. One hundred percent accuracy was obtained consistently using a training-testing approach. However, the leave-one-out approach produced only 92.8% accuracy while using random forest.

      Ahmadlou et al. have proposed three different techniques that are based on fractality[42], improved visibility graph fractality[43] and fuzzy synchronization likelihood[35]. In the fractality based approach, they have introduced the idea of using fractal dimensions as features. Their dataset contained 19-channel EEG data from 17 subjects, 9 ASD and 8 typically developing children. Statistically significant fractal dimensions were chosen using analysis of variance (ANOVA) and classification was performed using radial basis function neural network. This approach has achieved a 90% average accuracy with a variance of 0.15%. In the improved visibility graph fractality based approach[43], the effectiveness of the power of scale-freeness of visibility graphs (PSVG) and improved PSVG for ASD classification was evaluated. For this study, the same dataset used in [42] was used. Using ANOVA for feature selection and enhanced probabilistic neural network (EPNN) for classification, improved power of scale-freeness of visibility graphs (PSVG) and PSVG produced average accuracies of 95.5% and 84.2% with variances 1.7% and 1.8% respectively. For the fuzzy synchronization likelihood-based approach[35], a dataset that contains 18 subjects, 9 ASD and 9 typically developing children, was used. In this study, ANOVA and EPNN were used for feature selection and classification. Similar to their previous study[43], this approach also produced an average accuracy of 95.5% with a variance of 1.2%. All three studies[35, 42, 43] utilized 19-channel EEG recordings which increase feasibility with younger samples. Further, data pre-processing applications consisted of bandpass filters and discrete wavelet transforms which are simple and easy to implement and interpret.

      All of the related works reviewed in this section have shown accuracies greater than or equal to 90%, as given in Table 1. Here, SN denotes sine net neural network, SMO states sequential minimal optimization, K-CM denotes K-contractive map and ASR stands for artifact subspace reconstruction. The percentages of the dataset used for training and testing in the holdout method are provided.

      Description of the related studyPre-processing techniquesLearning modelEvaluation technique
      A data driven approach to classifying ASD[32]Bandpass fier, wavelet transformK-NN, random forest, SVMLeave-one-out, cross-validation
      Classifying ASD based on EEG
      and eye movement data[28]
      Makato′s pre-processing pipeline,
      visual inspection
      Naïve Bayes, logistic regression, SVM, DNN10×2 cross-validation, holdout method (80%, 20%)
      Classifying ASD using MS-ROM/
      I-FAST algorithm[29]
      I-FAST algorithmSN, logistic regression, SMO, K-NN, K-CM, Naïve Bayes, random forestLeave-one-out cross-validation, holdout method (68%, 32%)
      An EEG based objective measure
      of ASD[41]
      Baseline drift & AC noise removal, ASR, channel rejection, interpolation
      & re-referencing, wavelet transform,
      Naïve Bayes, logistic regression, SVM, DNN, AdaBoostM1, random forest, linear regression, bagging, JRip, REP tree10-fold cross-validation
      ASD diagnosis using fuzzy synchronization likelihood[35]Bandpass filter, wavelet transformEnhanced probabilistic neural networkHoldout method (78%, 22%)
      Visibility graph fractality for ASD diagnosis[43]Low-pass filter, wavelet transformEnhanced probabilistic neural networkHoldout method (80%, 20%)
      Classifying ASD using channel optimized method[5]Voltage thresholding, visual inspection, wavelet transformLogistic regression, SVM, Naïve Bayes, random forest10-fold cross-validation
      EEG complexity as a biomarker
      for ASD[40]
      Bandpass filterSVM, k-NN, Naïve Bayes10-fold cross-validation
      ASD diagnosis using fractal dimensions[42]Low-pass filter, wavelet transformRadial basis function neural
      Holdout method (80%, 20%)

      Table 1.  Summary of related studies on ASD identification using EEG data.

      Most of the studies have used bandpass filters and wavelet transformation for preprocessing the EEG signal, especially to remove noise and to decompose the signal into frequency bands. Machine learning algorithms such as SVM, logistic regression, random forest and Naïve Bayes have been used often in these classification studies. Different types of neural networks have also been widely used as classifiers because of the neural network′s ability to learn complex relationships better than other classification algorithms. However, one major limitation in almost all these prior studies is the small dataset size. The difficulty of finding available subjects with ASD (who can tolerate EEG) certainly imposes limitations to sample size and therefore reduces the statistical power. As a result, the risk of overfitting is high when using neural networks that are trained on small datasets.

      Another limitation is that many machine learning approaches for ASD classification heavily rely on brain dynamics (via EEG, fMRI). This approach is inherently problematic because the human brain is a stochastic, context-dependent system. Given the enormous complexity of the human neural activity, there is likely no sample large enough that can model an exact activity pattern that is specific to all the dynamic brains of individuals with ASD. Thus, building a feature set that integrates other levels of human physiology (thermal homeostasis, heart rate, movement, etc.) including brain activity may be a beneficial strategy towards developing algorithms that achieve higher levels of sensitivity and specificity.

      Currently, no studies have integrated thermal features with EEG features for ASD auto-classification. Here we combine facial thermography with EEG to evaluate how the integration of features from different physiological systems might reduce the risk of overfitting.

    • The proposed approach is designed with three main modules, namely pre-processing, feature extraction, and classification. Fig. 1 shows the abstract view of the proposed system addressed in this paper. The workflow of the proposed approach is shown in Fig. 2. The description of the process is explained in Sections 3.3 and 3.4.

      Figure 1.  Abstract view of ASDGenus system

      Figure 2.  Workflow diagram of ASDGenus

      This approach has considered both facial thermograms and raw EEG data as inputs. The output of the approach is the set of accuracies for the used learning models for each feature set with the objective to evaluate the effect with the addition of thermographic features. The raw EEG data of a subject contains 32 sequences of voltage values for the 32 channels, sampled at 250 Hz. This data is stored in a 2D array with 32 subarrays representing the 32 channels. The pre-processing pipeline contains four main steps explicitly for EEG noise removal that outputs cleaned EEG data.

      The feature extraction module extracts entropy and statistical features such as mean and standard deviation from the cleaned EEG data. Then the extracted EEG features are directed through a correlation-based feature selection process to filter out the insignificant features. At the same time, thermographic data processing initiates in the feature extraction module, where the region selection of the thermograms and the mean temperature calculation of the selected regions is conducted. The mean temperature values of each ROI are taken as the thermographic features of a subject. The selected EEG features (FS1, FS2, FS3) and the thermographic features are then concatenated to build feature sets for the classification module. The feature selection process is described in Section 3.3.2.

      In the classification module, four learning algorithms are employed, namely Naïve Bayes, random forest, logistic regression, and multi-layer perceptron neural network (MLP) to evidently show the effect of thermographic features. The classification module yields the accuracy for each model which then gets validated to obtain the results. These results show the impact of the addition of thermographic features comparatively, evidencing the feasibility of using skin surface temperature, in classifying ASD using EEG data.

    • This study considers both EEG and thermograms for the identification of ASD. Data acquisition was carried at Indiana University-Purdue University, Columbus, USA. Both EEG and thermal data were recorded during the administration of the ADOS-2[44]. ADOS-2 is a standardized assessment of communication, social interaction, play, imagination, and stereotyped behaviors[45]. The ADOS-2 was used as a social interaction platform to maintain the social context consistent for each participant throughout the data acquisition process.

      The EEG data were acquired using a 32-channel LiveAmp wireless EEG system[46] with active electrodes and a digital sampling rate of 250 Hz (Brain products, GmbH). All the channels were uninterruptedly recorded for each participant throughout the administration of ADOS-2, using the frontal central zero (FCz) electrode as the reference. The wireless system allowed for the head movement of the participants, and the active electrodes allowed for recordings at higher impedances thereby facilitating the speed of the application. Fig. 3 is a plot of the 32 channels of a raw EEG sample.

      Figure 3.  Raw EEG sample

      High-resolution infrared thermograms of the participants' faces were captured using high-resolution infrared cameras (ICI, Inc.). The infrared instruments used for thermography measure the radiated thermal energy of the participant's face to generate a temperature of each pixel remotely, based on the algorithm that maps the pixel value of the thermograms to the corresponding temperature. These generated temperature values of each thermogram were stored in a separate comma-seperated values (CSV) file which was used for the mean temperature calculation. Fig. 4 shows a facial thermogram sample used for this study.

      Figure 4.  Sample facial thermogram

      As stated in Table 2, the overall dataset of this study contains data of 17 participants (10 male and 7 female) between the ages of 5 and 17 years. The target class (ASD diagnosis) was obtained by separating the participants with respect to prior ASD diagnosis disclosed by parents upon enrolment in the study.

      Participant IDSexAgeADOS-2Diagnosis

      Table 2.  Participant details

    • The raw EEG data were acquired using the FCz channel (electrode) as the reference channel. The amplitudes of the EEG data for each channel depends on the reference used in the data acquisition. The channels that are located closer to the reference are likely to have similar voltage readings, resulting in low voltage differences, while the channels that are located distantly to the reference are likely to have more substantial voltage differences, subsequently showing larger amplitudes in EEG data.

      In order to avert this bias, we used the average reference for re-referencing the channels. The average referencing was achieved by generating an average over all the channels as given in (1) and subtracting the resultant signal from each channel as given in (2). Here, n represents the number of channels and Uorg[i] denotes the voltage of channel i, which against the original reference, FCz channel. After the re-referencing, the amplitudes of each channel were reduced, having an unbiased contribution to the new reference[47]. Fig. 5 shows the plotting of FP1 channel (a) before (raw signal) and (b) after re-referencing raw signal to average reference.

      Figure 5.  Average re-referencing

      $ {\rm{Average}}\;{\rm{reference:}}\;\overline {{u}} = \frac{{\sum\nolimits_{{{i}} = 1}^{{n}} {{{{U}}_{{\rm{org }}}}} [{{i}}]}}{{{n}}}. $


      Define u[i] as voltage of channel i, which against the average reference:

      $ {{u[i] = }}{{{U}}_{{\rm{org}}}}{{[i] - }}\overline {{u}}. $

    • Baseline drift is the effect where the base axis of the signal appears to drift up and down from the straight x-axis, resulting in a drift of the whole signal from the baseline. The main reason for the baseline drift can be attributed to improper channels caused by the electrode-scalp impedance and head movements of the participant[48]. The baseline drift frequencies are usually low and can be removed using a 1 Hz high-pass filter[41]. Fig. 6 shows the EEG signal after removing the baseline drift.

      Figure 6.  EEG signal after baseline drift removal

    • The electromagnetic interference caused by the AC power transmission line is one of the prevalent artifacts that affect the quality of biomedical signals like EEG. The power line frequency depends on the standard of the country where the data acquisition is carried out and, in this research, it is 60 Hz, which is the standard of North America. We used a 60 Hz notch filter which is a commonly used method to remove this artifact[21]. Fig. 7 shows the EEG signal (a) before and (b) after removing the AC line noise on the FP1 channel.

      Figure 7.  AC powerline noise removal

    • As illustrated in Fig. 8 (a), eye blinks in an EEG signal have a unique pattern of irregular voltage values which limit to a small-time range. In order to remove the eye-blink artifact without affecting the attributes of the signal, we have discarded the time intervals of the eye blinks using a custom algorithm, as the discarded time interval can be considered insignificant[5]. The algorithm iterates the EEG signal, detecting voltage values that exceed the threshold (200–500 µV), which is set uniquely to each signal after visual inspection. Then a time interval of 0.4 ms is discarded around the detected voltage values in order to remove the whole eye blink since the duration of an eye blink is 0.4 ms[49]. Fig. 8 (b) shows the EEG signal after the eye blink removal.

      Figure 8.  Eye-blink removal

    • In this research, we used statistical and entropy functions for the EEG feature extraction. Statistical values comprise the mean and standard deviation. Mean and standard deviation are the most widely used statistical features in recent studies of ASD classification using EEG[4, 5, 32]. Standard deviation was calculated using (3), where n is the signal length and $ \overline {{x}} $ and xi represent the signal mean and the amplitude respectively, at a given point i.

      $ {\rm{ Standard}}\;{\rm{deviation }} = \sqrt {\frac{1}{{{{n - 1}}}}\sum\limits_{{{i = 1}}}^{{n}} {{{\left( {{{{x}}_{{i}}}{{ - \bar x}}} \right)}^2}} } . $


      The entropy value quantifies the amount of regularity and the unpredictability of fluctuations of a signal over time, thus, it considers as the primary attribute to explore the abnormalities of the EEG signals as it can be used to measure the level of the chaos of the signal[50]. The entropy is calculated using (4), where pi is the probability of the i-th amplitude value.

      $ {\rm{Shannon}}'{\rm{s}}\;{\rm{entropy = - }}\sum\limits_{{{k = 0}}}^{{n}} {{{{p}}_{{i}}}} {\log _{\rm{2}}}{{{p}}_{{i}}} . $


      The statistical features and the entropy values were calculated for each channel and created two feature vectors for each participant.

    • The statistical feature vector contains 64 features (mean and standard deviation for 32 channels) and the entropy feature vector contains 32 features. A third feature vector was created concatenating the statistical and entropy feature vectors which contain 96 features. In order to reduce the dimensionality and to improve the classification, we employed a correlation-based feature selection (CFS) technique on EEG features. The technique evaluates all the subsets of the features, preferring the subsets that are highly correlated with the target class (ASD diagnosis) while having low intercorrelation using three different measures of relatedness: minimum description length (MDL), symmetrical uncertainty, and relief[51].

      Next, the CFS technique was used to select the 3 feature sets, from statistical features, entropy features and a combination of both, to be used for the classification. The best subset is yielded as the selected feature set as follows.

      FS1 was obtained from the 64 statistical features. CFS was used to select the channels P3, O1, C4 from the mean features and the channel C4 from the standard deviation features. Thus, FS1 contains 3 channels.

      FS2 was obtained from the 32 entropy features. CFS was used to select the channels C3, P3, FC6 from the entropy features. Thus, FS2 contains 3 channels.

      FS3 was obtained from the combination of 64 statistical features and 32 entropy features. CFS was used to select the channels P3, O1, C4 from the mean features, channel C4 from the standard deviation features, and channels P3, FC6 from the entropy features. Thus, FS3 contains 4 channels.

      The feature selection process outputs three feature sets for the three feature vectors as shown in Fig. 2: selected statistical features (FS1), selected entropy features (FS2) and selected features of the concatenated feature set (FS3).

    • One of the novelties of this research is exploring the ability of processing thermographic data for feature extraction and ASD classification. Initially, several ROIs of the face that represents the internal temperature of the participant are identified. Ng et al. [52] showed a high correlation between internal body temperature and the surface temperature of the eye region. They concluded that the temperature of the eye region is the site that correlates most highly with internal body temperature. The thin skin in the eye region makes it a preferred area to approximate the core body temperature. Therefore, eye region was focused more, and six sub-regions were selected in the eye region; eye left/right (EL/ER), inner canthus area left/right (ECL/ECR) and the supraorbital area left/right (ESL/ESR).

      Apart from the eye region, we have chosen the nose (N), left cheek (CL), and right cheek (CR). Marinescu et al.[53] have identified the nose and the forehead as ideal sites for skin temperature measurements. Or and Duffy[54] have found out that there is a strong correlation between nasal temperature and mental workload which made the nose a potential region. Shearn et al.[55] has concluded that the cheek temperature responses are significantly high during blushing and in [56], they have stated that physiological signs like blushing are rarely observed in ASD which led us to select cheeks as regions of interest for this research.

      Therefore, the nine ROIs: eye (EL/ER), inner canthus (ECL/ECR), supraorbital area (ESL/ESR), nose (N) and cheeks (CL/CR), were selected based on the related studies that have addressed the temperature effect of eye region[52], nose[53, 54] and cheeks[55]. The selected ROIs were carefully marked on each thermogram manually, as the face of each participant was located (magnification, placement) uniquely. Then, the average temperatures of each marked ROI of the face were calculated as the facial thermographic features of each participant. Fig. 9 illustrates the selected regions of the face.

      Figure 9.  Selected regions of the face

      Fig. 10 plots the temperature values obtained for each region. According to the plot, the highest and the lowest values were recorded from the left inner canthus and the nose regions, respectively. The most extensive distribution of the temperature was recorded in the nose region. At the same time, obtaining higher temperatures for the eye region was a precise observation.

      Figure 10.  Temperature distribution of each region

    • Our approach is designed with several classification models as we consider two different sets of EEG features, combined with thermographic features and multiple learning algorithms, as shown in Fig. 11.

      Figure 11.  Classification process of ASDGenus

      There are mainly three EEG feature sets: FS1, FS2, and FS3. Each of these feature sets is combined with thermographic features resulting in a total of six feature sets. Thermographic features are only used as an addition to the EEG data since the objective was to compare the accuracy values for each EEG feature set before and after the addition of the thermographic features. We employed four learning algorithms, trained separately for each feature set, resulting in a total of 24 models. The performances of the 12 models trained using EEG feature sets were compared with the 12 models trained using EEG feature sets concatenated with thermographic features.

      In order to further evidence the effect of the thermographic features, we used multiple types of learning algorithms: a generative linear classifier (Naïve Bayes), a decision tree classifier (random forest) and two discriminative linear classifiers (logistic regression and MLP). Naïve Bayes is considered to be the standard in machine learning applications[27], therefore we have used Naïve Bayes to obtain the benchmark performance for the research. Then, logistic regression was used because of its simplicity and the low computational resources consumption[30]. We used a multinomial logistic regression model with a ridge estimator in order to improve the parameter estimates and to reduce the error furthermore. Considering the recent studies that are focused on ASD classification, reasonable results were produced using random forest[5, 29, 32]. Therefore, a random forest classifier was also used to obtain the results. Finally, we used a multi-layer perceptron neural network where sigmoid activation was used in all the nodes with a learning rate of 0.3. We dynamically changed the number of hidden layers for each model according to the number of features in the feature set used to achieve high performances.

    • The main modules of the proposed approach can be identified as pre-processing, feature extraction, and classification. Cartool software by Denis Brunet, and python were used for the implementation of data pre-processing. In the thermographic feature extraction section, the region selection was carried out using image processing software, and the mean temperature calculation was implemented using Python and OpenCV library[57]. The calculation of the statistical and entropy values for the EEG feature extraction was implemented using python, because of the wide availability of third-party libraries. For the feature selection and the classification modules, the WEKA[58] software package was used, due to its comprehensive collection of data pre-processing and modeling techniques. Fig. 12 shows the system tool stack used for the implementation of the ASDGenus system.

      Figure 12.  System tool stack diagram of ASDGenus

    • For this research, we have employed the leave-one-out cross-validation method. Cross-validation is a useful evaluation technique when it comes to handling datasets with a limited number of data samples[59]. In k-fold cross-validation, the dataset is divided into k partitions of approximately equal sizes. The evaluation performs iteratively, for k iterations. During each iteration, one partition is used for testing while the rest ones are used for training. The overall accuracy is calculated by averaging all the accuracies, obtained for each iteration. This accuracy is used as the primary evaluation performance metric of the learning model. Table 3 shows the comparison of the accuracies for EEG feature sets (FS1, FS2, and FS3), before and after the addition of thermographic features.

      FS1FS1 + thermoFS2FS2 + thermoFS3FS3 + thermo
      Naïve Bayes70%64%64%82%70%70%
      Logistic regression70%94%76%94%76%82%
      Random forest76%76%76%88%88%88%

      Table 3.  Accuracies for EEG feature sets before and after the addition of thermographic features in ASDGenus

      Considering FS1, which is the statistical features of EEG data, the highest accuracy was achieved using logistic regression for the feature set obtained after the addition of thermographic features. Both logistic regression and MLP models were improved with the addition of the thermographic features. The performance of the Naïve Bayes model was reduced from 70 to 64, and the random forest model's performance remained unchanged.

      For FS2, which is the entropy features of EEG data, the accuracies from all the models were improved with the addition of the thermographic features, achieving the highest accuracy of 94%, using logistic regression for the feature set obtained after the addition of thermographic features.

      For FS3, the accuracies of both logistic regression and MLP models are improved. The accuracy from the MLP model achieved are highest accuracy of 94% with the addition of thermographic features, while the performances of Naïve Bayes and random forest models remained unchanged.

      Thus, according to the classification accuracy values, all the three feature sets FS1, FS2 and FS3 have shown the highest accuracy of 94% with thermographic data, with different classifiers and the different number of selected channels. That is, the highest classification accuracy can be obtained by either FS1 or FS2 or FS3 with thermographic data.

      Fig. 13 summarizes the performances of the models. The performance of eight models was improved, while the accuracy of only one combination was decreased and the accuracy of three combinations remained unchanged.

      Figure 13.  Accuracy comparison for FS1, FS2, and FS3

      For all three feature sets, the performances of the logistic regression and MLP models were improved, and the concept of confounding is a possible reason for this. In that case, it is possible to assume that the relationship between the EEG features and the diagnosis class improves with the presence of the thermographic features, which performs as confounding features. At the same time, the behavior of the Naïve Bayes model with FS1 can be justified by the assumption made by the Naïve Bayes algorithms, that the features are independent of each other. Here, the EEG features and the thermographic features may be dependent on each other, being the cause for the enhancement of the classification. This assumption can rule out any sort of dependencies between the EEG and thermographic features, resulting in reduced performance.

    • The current study is an exploration of the ability to improve EEG-based classification. We employed raw EEG data, and infrared facial thermograms, which capture the skin surface temperature as the input data and a simple pre-processing and feature extraction methodology for classification. The primary objective of this research is to introduce analysis and a comparison of combining EEG data with facial thermographic data for ASD classification. We have used four learning algorithms with three EEG feature sets and illustrated the association between EEG and thermographic data in ASD classification and the potential to use the combination of data to improve the classification process of ASDGenus prototype.

      Because of the originality of the concept of improving ASD classification using non-invasive quantification of skin surface temperature, no prior related research has been conducted. Thus, there is a limitation of comparing the results of the proposed methodology with existing studies in the context of emphasizing the impact of the thermographic features. Nonetheless, for the completeness of the study, a comparison with the classification results of related ASD classification approaches is shown in Table 4 indicating the diverse settings such as dataset, features and classifiers. The accuracy field represents the highest accuracy of each related study and the classifier field shows the learning algorithm/s used to achieve the highest accuracy, with the considered number of subjects.

      Description of the related studyDatasetSubjectsFeaturesNo. of EEG channels used for classificationClassifierAccuracy
      ASD classification[32]EEG (64/128 channels)188Multiscale entropy, recurrence quantitative analysis, detrended fluctuation analysis19SVM100%
      ASD classification[28]EEG (128 channels), eye movement data34Shannon entropy, statistical methods128Logistic regression, Naïve Bayes100%
      ASD classification using MS-ROM/I-FAST algorithm[29]EEG (19 channels)35Multiscale entropy19Random forest100%
      An objective measure of ASD[41]EEG (32 channels)17Shannon entropy, statistical methods, fast Fourier transform19JRip98.06%
      ASD diagnosis[35]EEG (19 channels)18Fuzzy synchronization likelihood19Enhanced probabilistic neural network95.5%
      Visibility graph fractality for ASD diagnosis[43]EEG (19 channels)17Power of scale-freeness of the visibility graph19Enhanced probabilistic neural network95.5%
      ASD classification with channel optimization[5]EEG (32 channels)17Statistical methods5Random forest93.3%
      Biomarker for ASD[40]EEG (64 channels)143Modified multiscale entropy64k-NN, Naïve Bayes90%
      ASD diagnosis using fractal dimensions[42]EEG (19 channels)17Higuchi′s/Katz′s fractal dimension19Radial basis function neural network90%
      Proposed ASDGenus DSSEEG (32 channels), thermographic data17Shannon entropy, statistical methods3Logistic regression94%

      Table 4.  Accuracy comparison of related work with ASDGenus

      Here, the number of channels used for the classification, can be taken as the comparison factor. The proposed study needs only 3 channels to obtain the highest classification accuracy, which is the significant contribution of the ASDGenus DSS. The optimal number of channels can be obtained from either FS1 or FS2. Specifically, the feature set FS1 consists of the three channels P3, O1, C4 and FS2 contains the channels C3, P3, FC6 as described in Section 3.3.2, and has given the highest classification accuracy of 94% with the incorporation of thermographic features using the logistic regression classifier, as shown in Table 3. In contrast, as shown in Table 4, all the other related studies have used more channels for the classification process.

      Another significant observation of the related studies is the diversity in extracted features and the classifiers which were employed in acquiring the highest accuracies. Nevertheless, the features used in this research, statistical methods and entropy, were commonly utilized in the related works as well. Different kinds of neural networks, logistic regression, and Naïve Bayes classifiers have often given the highest accuracy values.

      Considering the datasets used in the studies, precise observation is the usage of a different number of channels. This causes a significant change in the amount of data employed for the research. Another limitation in most of the studies including the proposed approach, except for [32] and [40], is the low number of subjects in the dataset because of the difficulty in getting participants for the data acquisition. The low number of subjects in the dataset may not fully capture the heterogeneity of the population. At the same time, having a larger dataset would reduce the impact of noise, which is a major problem in EEG data. Large datasets also enable the utilization of modern learning approaches such as deep learning, which can enhance the performance of the learning process with large datasets.

      For instance, we have applied deep learning classifiers in an experimental setup for the combination of facial thermography and EEG data. However, high accuracies were not obtained as the size of the dataset is not sufficient for deep learning. Therefore, the proposed methodology can be extended in the future with large datasets to increase the generalizability and the performance of this classification approach.

      In addition, although the eye blink artifact was detected and removed with an algorithm in the proposed solution, muscle and cardiac artifacts were not identified. The requirement of expert knowledge for manual region selection of the thermograms is an existing limitation of the proposed ASDGenus DSS.

      Moreover, the proposed DSS can be extended with further research to support different neurological disorders that address comorbidity. The processing of several psychophysiological data types such as EEG[5], thermal images, fMRI[25, 60, 61] and eye-movement data[26], with different feature extractions, classification and analysis methods can be integrated into a generic DSS framework. For instance, the classification process can be performed using deep probabilistic programming that uses deep learning with probabilistic modeling to perform computations efficiently and flexibly[62].

      The quality attributes such as accuracy, reliability and user-friendliness can be incorporated in future computational solutions. Thus, a generic framework with a combination of support tools is beneficial for researchers and practitioners in clinical practice.

    • ASD is a complex neurodevelopmental disorder, affecting social, communicative, and repetitive behavior. Although early intervention can improve the behavior of persons with ASD, the unknown etiology of the disorder and absence of diagnostic medical tests challenge the early diagnosis of ASD. Designing an objective measure to classify ASD, based on acquired physiological data could enhance the diagnostic process. Even though the relationship between ASD and temperature regulation remains inconclusive, past observations and studies have presented evidence for the potential existence of such a relationship. In this study, we have taken the first step towards evaluating differences in skin surface temperature as a potential biomarker for ASD classification, which is an original contribution of this study.

      This paper proposes a prototype named ASDGenus, a tool to support the clinical diagnosis of ASD. It consists of three modules for pre-processing, feature extraction and classification. The pre-processing module filters baseline drift, AC powerline and eye blink noises from the raw EEG signal. We extracted statistical (mean and standard deviation) and entropy values as the features and insignificant features were removed in the feature extraction and feature selection modules. Finally, four different machine learning algorithms were used for classification and models were evaluated using leave-one-out cross-validation. We were able to indicate a noteworthy improvement of the ASD classification with the addition of thermographic features, with eight of 12 models showing improvement and only one deterioration. The highest performing models for each learning algorithm: Naïve Bayes, logistic regression, MLP, and random forest, achieved the accuracies of 82%, 94%, 94%, and 88%, respectively, with the addition of thermographic features. With the assumption that the data in the medical domain give better results by minimizing the false negative values, for the combination of EEG with thermographic data, the logistic regression method gives the best results for the selected statistical features and the random forest model gives the best results for the selected entropy features. The high accuracies confirm the existence of a correlation between skin surface temperature measurements and ASD. The promising results illustrate that thermographic features may be used with EEG as a potential biomarker for ASD classification. In addition, further research associating ASD and differences in temperature regulation may unveil new possibilities regarding the etiology and treatment of ASD.

    • This work was supported by Old Dominion University, Norfolk, Virginia and University of Moratuwa, Sri Lanka.

Reference (62)



    DownLoad:  Full-Size Img  PowerPoint