Volume 17 Number 2
March 2020
Article Contents
Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang and Anil K. Jain. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. International Journal of Automation and Computing, vol. 17, no. 2, pp. 151-178, 2020. doi: 10.1007/s11633-019-1211-x
Cite as: Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang and Anil K. Jain. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. International Journal of Automation and Computing, vol. 17, no. 2, pp. 151-178, 2020.

# Adversarial Attacks and Defenses in Images, Graphs and Text: A Review

Author Biography:
• Han Xu is a second year Ph. D. student of computer science in DSE Lab, Michigan State University, USA. He is under supervision by Dr. Ji-Liang Tang. His research interests include deep learning safety and robustness, especially studying the problems related to adversarial examples.E-mail: xuhan1@msu.edu (Corresponding author) ORCID iD: 0000-0002-4016-6748

Yao Ma received the B. Sc. degree in applied mathematics at Zhejiang University, China in 2015, the M. Sc. degree in statistics, probabilities and operation research at Eindhoven University of Technology, the Netherlands in 2016. He is now a Ph. D. degree candidate of Department of Computer Science and Engineering, Michigan State University, USA. His Ph. D. advisor is Dr. Jiliang Tang. His research interests include graph neural networks and their related safety issues. E-mail: mayao4@msu.edu

Hao-Chen Liu is currently a Ph. D. student at the Department of Computer Science and Engineering at Michigan State University, under the supervision of Dr. Jiliang Tang. He is a member of Data Science and Engineering (DSE) Lab. His research interests include natural language processing problems, especially in the robustness, fairness of dialogue systems. E-mail: liuhaoc@msu.edu

Debayan Deb is a Ph. D. degree candidate in the Biometrics Lab, Michigan State University, USA under the supervision of Dr. Anil K. Jain. Before joining the Biometrics Lab of MSU, he graduated from Michigan State University with a Bachelor Degree of Computer Science and Engineering. His research interests include face recognition and computer vision tasks. E-mail: debdebay@msu.edu

Hui Liu is a research associate at Michigan State University. Before joining MSU, she received her Ph. D. degree of Electrical Engineering in Southern Methodist University, USA under the supervision by Dr. Dinesh Rajen. Her research interests include signal processing, wireless communication, and deep learning related topics. E-mail: liuhui7@msu.edu

Ji-Liang Tang is an assistant professor in the computer science and engineering department at Michigan State University since Fall 2016. Before that, he was a research scientist in Yahoo Research and received his Ph. D. degree from Arizona State University in 2015. He was the recipients of 2019 NSF Career Award, the 2015 KDD Best Dissertation runner up and 6 Best Paper Awards (or runner-ups) including WSDM 2018 and KDD 2016. He serves as conference organizers (e.g., KDD, WSDM and SDM) and journal editors (e.g., TKDD). He has published his research in highly ranked journals and top conference proceedings, which received thousands of citations and extensive media coverage.His research interests include social computing, data mining and machine learning and their applications in education. E-mail: tangjili@msu.edu

Anil K. Jain (Ph. D., 1973, Ohio State University; B. Tech., IIT Kanpur) is a University Distinguished Professor at Michigan State University where he conducts research in pattern recognition, machine learning, computer vision, and biometrics recognition. He was a member of the United States Defense Science Board and Forensics Science Standards Board. His prizes include Guggenheim, Humboldt, Fulbright, and King-Sun Fu Prize. For advancing pattern recognition, Jain was awarded Doctor Honoris Causa by Universidad Autónoma de Madrid. He was Editor-in-Chief of the IEEE Transactions on Pattern Analysis and Machine Intelligence and is a Fellow of ACM, IEEE, AAAS, and SPIE. Jain has been assigned 8 U.S. and Korean patents and is active in technology transfer for which he was elected to the National Academy of Inventors. Jain is a member of the U.S. National Academy of Engineering (NAE), foreign member of the Indian National Academy of Engineering (INAE), a member of The World Academy of Science (TWAS) and a foreign member of the Chinese Academy of Sciences (CAS). His research interests include pattern recognition, machine learning, computer vision, and biometrics recognition.E-mail: jain@egr.msu.edu

• Received: 2019-10-13
• Accepted: 2019-11-11
• Published Online: 2020-03-27
• Deep neural networks (DNN) have achieved unprecedented success in numerous machine learning tasks in various domains. However, the existence of adversarial examples raises our concerns in adopting deep learning to safety-critical applications. As a result, we have witnessed increasing interests in studying attack and defense mechanisms for DNN models on different data types, such as images, graphs and text. Thus, it is necessary to provide a systematic and comprehensive overview of the main threats of attacks and the success of corresponding countermeasures. In this survey, we review the state of the art algorithms for generating adversarial examples and the countermeasures against adversarial examples, for three most popular data types, including images, graphs and text.
• 1Note that the softmax function at a temperature $T$ means: $softmax (x, T)_i = \dfrac{{\rm e}^{\frac{x_i}{T}}}{\displaystyle\sum_{j} {\rm e}^{\frac{x_j}{T}}}$, where $i = 0, 2, \cdots , K-1$.
•  [1] A. Krizhevsky, I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Curran Associates Inc., Lake Tahoe, USA, pp. 1097–1105, 2012. [2] K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 770–778, 2016. DOI: 10.1109/CVPR.2016.90. [3] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, B. Kingsbury.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups[J]. IEEE Signal Processing MagazineIEEE Signal Processing Magazine, 2012, 29(6): 82-97. doi: 10.1109/MSP.2012.2205597 [4] S. Hochreiter, J. Schmidhuber.  Long short-term memory[J]. Neural ComputationNeural Computation, 1997, 9(8): 1735-1780. doi: 10.1162/neco.1997.9.8.1735 [5] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, D. Hassabis.  Mastering the game of go with deep neural networks and tree search[J]. NatureNature, 2016, 529(7587): 484-489. doi: 10.1038/nature16961 [6] D. Cireşan, U. Meier, J. Masci, J. Schmidhuber.  Multi-column deep neural network for traffic sign classification[J]. Neural NetworksNeural Networks, 2012, 32(): 333-338. doi: 10.1016/j.neunet.2012.02.023 [7] T. N. Kipf, M. Welling. Semi-supervised classification with graph convolutional networks. ArXiv: 1609.02907, 2016. [8] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus. Intriguing properties of neural networks. ArXiv: 1312.6199, 2013. [9] I. J. Goodfellow, J. Shlens, C. Szegedy. Explaining and harnessing adversarial examples. ArXiv: 1412.6572, 2014. [10] D. Zügner, A. Akbarnejad, S. Günnemann. Adversarial attacks on neural networks for graph data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, London, UK, pp. 2847–2856, 2018. DOI: 10.1145/3219819.3220078. [11] J. Ebrahimi, A. Y. Rao, D. Lowd, D. J. Dou. HotFlip: White-box adversarial examples for text classification. ArXiv: 1712.06751, 2017. [12] N. Papernot, P. McDaniel, X. Wu, S. Jha, A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of IEEE Symposium on Security and Privacy, IEEE, San Jose, USA, pp. 582–597, 2016. DOI: 10.1109/SP.2016.41. [13] A. Athalye, N. Carlini, D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. ArXiv: 1802.00420, 2018. [14] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv: 1706.06083, 2017. [15] A. Kurakin, I. Goodfellow, S. Bengio. Adversarial examples in the physical world. ArXiv: 1607.02533, 2016. [16] N. Carlini, D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, ACM, Dallas, USA, pp. 3–14, 2017. DOI: 10.1145/3128572.3140444. [17] W. L. Xu, D. Evans, Y. J. Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. ArXiv: 1704.01155, 2017. [18] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, A. Madry. Adversarial examples are not bugs, they are features. ArXiv: 1905.02175, 2019. [19] B. Biggio, B. Nelson, P. Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, Omnipress, Edinburgh, UK, 2012. [20] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. W. Xiao, A. Prakash, T. Kohno, D. Song. Robust physical-world attacks on deep learning models. ArXiv: 1707.08945, 2017. [21] F. Tramer, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel. Ensemble adversarial training: Attacks and defenses. ArXiv: 1705.07204, 2017. [22] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, F. Roli. Evasion attacks against machine learning at test time. In Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, Prague, Czech Republic, pp. 387–402, 2013. DOI: 10.1007/978-3-642-40994-3_25. [23] M. Barreno, B. Nelson, A. D. Joseph, J. D. Tygar.  The security of machine learning[J]. Machine LearningMachine Learning, 2010, 81(2): 121-148. doi: 10.1007/s10994-010-5188-5 [24] N. Dalvi, P. Domingos, Mausam, S. Sanghai, D. Verma. Adversarial classification. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, Seattle, USA, pp. 99–108, 2004. DOI: 10.1145/1014052.1014066. [25] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, A. Madry. Robustness may be at odds with accuracy. ArXiv: 1805.12152, 2018. [26] D. Su, H. Zhang, H. G. Chen, J. F. Yi, P. Y. Chen, Y. P. Gao. Is robustness the cost of accuracy? – A comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 644–661, 2018. DOI: 10.1007/978-3-030-01258-8_39. [27] D. Stutz, M. Hein, B. Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the 32nd IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Piscataway, USA, pp. 6976–6987, 2019. [28] H. Y. Zhang, Y. D. Yu, J. T. Jiao, E. P. Xing, L. El Ghaoui, M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. ArXiv: 1901.08573, 2019. [29] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, F. F. Li. Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Miami, USA, pp. 248–255, 2009. DOI: 10.1109/CVPR.2009.5206848. [30] D. C. Liu, J. Nocedal.  On the limited memory BFGS method for large scale optimization[J]. Mathematical ProgrammingMathematical Programming, 1989, 45(1–3): 503-528. doi: 10.1007/BF01589116 [31] A. Kurakin, I. Goodfellow, S. Bengio. Adversarial machine learning at scale. ArXiv: 1611.01236, 2016. [32] S. M. Moosavi-Dezfooli, A. Fawzi, P. Frossard. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 2574–2582, 2016. DOI: 10.1109/CVPR.2016.282. [33] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, A. Swami. The limitations of deep learning in adversarial settings. In Proceedings of IEEE European Symposium on Security and Privacy, IEEE, Saarbrucken, Germany, pp. 372−387, 2016. DOI: 10.1109/EuroSP.2016.36. [34] N. Carlini, D. Wagner. Towards evaluating the robustness of neural networks. In Proceedings of IEEE Symposium on Security and Privacy, IEEE, San Jose, USA, pp. 39–57, 2017. DOI: 10.1109/SP.2017.49. [35] N. Carlini, G. Katz, C. Barrett, D. L Dill. Provably minimally-distorted adversarial examples. ArXiv: 1709.10207, 2017. [36] G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Proceedings of the 29th International Conference on Computer Aided Verification, Springer, Heidelberg, Germany, pp. 97–117, 2017. DOI: 10.1007/978-3-319-63387-9_5. [37] V. Tjeng, K. Xiao, R. Tedrake. Evaluating robustness of neural networks with mixed integer programming. ArXiv: 1711.07356, 2017. [38] K. Y. Xiao, V. Tjeng, N. M. Shafiullah, A. Madry. Training for faster adversarial robustness verification via inducing ReLU stability. ArXiv: 1809.03008, 2018. [39] J. W. Su, D. V. Vargas, K. Sakurai.  One pixel attack for fooling deep neural networks[J]. IEEE Transactions on Evolutionary ComputationIEEE Transactions on Evolutionary Computation, 2019, 23(5): 828-841. doi: 10.1109/TEVC.2019.2890858 [40] P. Y. Chen, Y. Sharma, H. Zhang, J. F. Yi, C. J. Hsieh. EAD: Elastic-net attacks to deep neural networks via adversarial examples. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018. [41] Y. Sharma, P. Y. Chen. Attacking the madry defense model with L1-based adversarial examples. ArXiv: 1710.10733, 2017. [42] S. M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, P. Frossard. Universal adversarial perturbations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 86–94, 2017. DOI: 10.1109/CVPR.2017.17. [43] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. A. Ma, Z. H. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, F. F. Li.  ImageNet large scale visual recognition challenge[J]. International Journal of Computer VisionInternational Journal of Computer Vision, 2015, 115(3): 211-252. doi: 10.1007/s11263-015-0816-y [44] C. W. Xiao, J. Y. Zhu, B. Li, W. He, M. Y. Liu, D. Song. Spatially transformed adversarial examples. ArXiv: 1801.02612, 2018. [45] Y. Song, R. Shu, N. Kushman, S. Ermon. Constructing unrestricted adversarial examples with generative models. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montréal, Canada, pp. 8312–8323, 2018. [46] A. Odena, C. Olah, J. Shlens. Conditional image synthesis with auxiliary classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 2642–2651, 2017. [47] A. Athalye, L. Engstrom, A. Ilyas, K. Kwok. Synthesizing robust adversarial examples. ArXiv: 1707.07397, 2017. [48] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, A. Swami. Practical black-box attacks against machine learning. In Proceedings of ACM on Asia Conference on Computer and Communications Security, ACM, Abu Dhabi, United Arab Emirates, pp. 506–519, 2017. DOI: 10.1145/3052973.3053009. [49] Y. P. Dong, F. Z. Liao, T. Y. Pang, H. Su, J. Zhu, X. L. Hu, J. G. Li. Boosting adversarial attacks with momentum. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 9185–9193, 2018. DOI: 10.1109/CVPR.2018.00957. [50] P. Y. Chen, H. Zhang, Y. Sharma, J. F Yi, C. J. Hsieh. ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, ACM, Dallas, USA, pp. 15–26, 2017. DOI: 10.1145/3128572.3140448. [51] A. Ilyas, L. Engstrom, A. Athalye, J. Lin. Black-box adversarial attacks with limited queries and information. ArXiv: 1804.08598, 2018. [52] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, J. Schmidhuber. Natural evolution strategies.  Natural evolution strategies[J]. Journal of Machine Learning ResearchJournal of Machine Learning Research, 2014, 15(1): 949-980. [53] M. Alzantot, Y. Sharma, S. Chakraborty, M. Srivastava. Genattack: Practical black-box attacks with gradient-free optimization. ArXiv: 1805.11090, 2018. [54] C. W. Xiao, B. Li, J. Y. Zhu, W. He, M. Y. Liu, D. Song. Generating adversarial examples with adversarial networks. ArXiv: 1801.02610, 2018. [55] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 2672–2680, 2014. [56] D. Deb, J. B. Zhang, A. K. Jain. Advfaces: Adversarial face synthesis. ArXiv: 1908.05008, 2019. [57] G. Cauwenberghs, T. Poggio. Incremental and decremental support vector machine learning. In Proceedings of the 13th International Conference on Neural Information Processing Systems, MIT Press, Denver, USA, pp. 388–394, 2000. [58] P. W. Koh, P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 1885–1894, 2017. [59] A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, T. Goldstein. Poison frogs! Targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montréal, Canada, pp. 6103–6113, 2018. [60] G. Hinton, O. Vinyals, J. Dean. Distilling the knowledge in a neural network. ArXiv: 1503.02531, 2015. [61] J. Buckman, A. Roy, C. Raffel, I. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018. [62] C. Guo, M. Rana, M. Cisse, L. van der Maaten. Countering adversarial images using input transformations. ArXiv: 1711.00117, 2017. [63] V. K. Ha, J. C. Ren, X. Y. Xu, S. Zhao, G. Xie, V. M. Vargas. Deep learning based single image super-resolution: A survey. In Proceedings of the 9th International Conference on Brain Inspired Cognitive Systems, Springer, Xi′an, China, pp. 106–119, 2018. DOI: 10.1007/978-3-030-00563-4_11. [64] G. S. Dhillon, K. Azizzadenesheli, Z. C. Lipton, J. Bernstein, J. Kossaifi, A. Khanna, A. Anandkumar. Stochastic activation pruning for robust adversarial defense. ArXiv: 1803.01442, 2018. [65] C. H. Xie, J. Y. Wang, Z. S. Zhang, Z. Ren, A. Yuille. Mitigating adversarial effects through randomization. ArXiv: 1711.01991, 2017. [66] Y. Song, T. Kim, S. Nowozin, S. Ermon, N. Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. ArXiv: 1710.10766, 2017. [67] P. Samangouei, M. Kabkab, R. Chellappa. Defense-GAN: Protecting classifiers against adversarial attacks using generative models. ArXiv: 1805.06605, 2018. [68] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, K. Kavukcuoglu. Conditional image generation with PixelCNN decoders. In Proceedings of the 30th Conference on Neural Information Processing Systems, Curran Associates Inc., Barcelona, Spain, pp. 4790–4798, 2016. [69] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, N. Usunier. Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 854–863, 2017. [70] T. Miyato, S. I. Maeda, M. Koyama, K. Nakae, S. Ishii. Distributional smoothing with virtual adversarial training. ArXiv: 1507.00677, 2015. [71] S. X. Gu, L. Rigazio. Towards deep neural network architectures robust to adversarial examples. ArXiv: 1412.5068, 2014. [72] S. Rifai, P. Vincent, X. Muller, X. Glorot, Y. Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, Omnipress, Bellevue, USA, pp. 833–840, 2011. [73] S. Ioffe, C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv: 1502.03167, 2015. [74] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, T. Goldstein. Adversarial training for free! ArXiv: 1904.12843, 2019. [75] D. H. Zhang, T. Y. Zhang, Y. P. Lu, Z. X. Zhu, B. Dong. You only propagate once: Accelerating adversarial training via maximal principle. ArXiv: 1905.00877, 2019. [76] L. S. Pontryagin. Mathematical Theory of Optimal Processes, London, UK: Routledge, 2018. [77] A. Raghunathan, J. Steinhardt, P. Liang. Certified defenses against adversarial examples. ArXiv: 1801.09344, 2018. [78] E. Wong, J. Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. ArXiv: 1711.00851, 2017. [79] M. Hein, M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, USA, pp. 2266–2276, 2017. [80] L. Vandenberghe, S. Boyd. Semidefinite programming.  Semidefinite programming[J]. SIAM ReviewSIAM Review, 1996, 38(1): 49-95. doi: 10.1137/1038003 [81] A. Raghunathan, J. Steinhardt, P. S. Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montréal, Canada, pp. 10877–10887, 2018. [82] E. Wong, F. Schmidt, J. H. Metzen, J. Z. Kolter. Scaling provable adversarial defenses. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montréal, Canada, pp. 8400–8409, 2018. [83] A. Sinha, H. Namkoong, J. Duchi. Certifying some distributional robustness with principled adversarial training. ArXiv: 1710.10571, 2017. [84] K. Grosse, P. Manoharan, N. Papernot, M. Backes, P. McDaniel. On the (statistical) detection of adversarial examples. ArXiv: 1702.06280, 2017. [85] Z. T. Gong, W. L. Wang, W. S. Ku. Adversarial and clean data are not twins. ArXiv: 1704.04960, 2017. [86] J. H. Metzen, T. Genewein, V. Fischer, B. Bischoff. On detecting adversarial perturbations. ArXiv: 1702.04267, 2017. [87] D. Hendrycks, K. Gimpel. Early methods for detecting adversarial images. ArXiv: 1608.00530, 2016. [88] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, A. Smola. A kernel two-sample test.  A kernel two-sample test[J]. Journal of Machine Learning ResearchJournal of Machine Learning Research, 2012, 13(): 723-773. [89] R. Feinman, R. R. Curtin, S. Shintre, A. B. Gardner. Detecting adversarial samples from artifacts. ArXiv: 1703.00410, 2017. [90] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning ResearchJournal of Machine Learning Research, 2014, 15(1): 1929-1958. [91] Y. Sharma, P. Y. Chen. Bypassing feature squeezing by increasing adversary strength. ArXiv: 1803.09868, 2018. [92] A. Fawzi, S. M. Moosavi-Dezfooli, P. Frossard. Robustness of classifiers: From adversarial to random noise. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 1632–1640, 2016. [93] S. M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, P. Frossard, S. Soatto. Analysis of universal adversarial perturbations. ArXiv: 1705.09554, 2017. [94] A. Fawzi, O. Fawzi, P. Frossard.  Analysis of classifiers′ robustness to adversarial perturbations[J]. Machine LearningMachine Learning, 2018, 107(3): 481-508. doi: 10.1007/s10994-017-5663-3 [95] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, T. Goldstein. Are adversarial examples inevitable? ArXiv: 1809.02104, 2018. [96] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, A. Madry. Adversarially robust generalization requires more data. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Montréal, Canada, pp. 5014–5026, 2018. [97] H. J. Dai, H. Li, T. Tian, X. Huang, L. Wang, J. Zhu, L. Song. Adversarial attack on graph structured data. ArXiv: 1806.02371, 2018. [98] Y. Ma, S. H. Wang, T. Derr, L. F. Wu, J. L. Tang. Attacking graph convolutional networks via rewiring. ArXiv: 1906.03750, 2019. [99] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller. Playing Atari with deep reinforcement learning. ArXiv: 1312.5602, 2013. [100] D. Züugner, S. Günnemann. Adversarial attacks on graph neural networks via meta learning. ArXiv: 1902.08412, 2019. [101] C. Finn, P. Abbeel, S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, JMLR.org, Sydney, Australia, pp. 1126–1135, 2017. [102] A. Bojchevski, S. Günnemann. Adversarial attacks on node embeddings via graph poisoning. ArXiv: 1809.01093, 2018. [103] B. Perozzi, R. Al-Rfou, S. Skiena. DeepWalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, USA, pp. 701–710, 2014. DOI: 10.1145/2623330.2623732. [104] F. L. Feng, X. N. He, J. Tang, T. S. Chua. Graph adversarial training: Dynamically regularizing based on graph structure. ArXiv: 1902.08226, 2019. [105] K. D. Xu, H. G. Chen, S. J. Liu, P. Y. Chen, T. W. Weng, M. Y. Hong, X. Lin. Topology attack and defense for graph neural networks: An optimization perspective. ArXiv: 1906.04214, 2019. [106] N. Carlini, D. Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In Proceedings of IEEE Security and Privacy Workshops, IEEE, San Francisco, USA, pp. 1–7, 2018. DOI: 10.1109/SPW.2018.00009. [107] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, A. Y. Ng. Deep speech: Scaling up end-to-end speech recognition. ArXiv: 1412.5567, 2014. [108] T. Miyato, A. M. Dai, I. Goodfellow. Adversarial training methods for semi-supervised text classification. ArXiv: 1605.07725, 2016. [109] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, Curran Associates Inc., Lake Tahoe, USA, pp. 3111–3119, 2013. [110] B. Liang, H. C. Li, M. Q. Su, P. Bian, X. R. Li, W. C. Shi. Deep text classification can be fooled. ArXiv: 1704.08006, 2017. [111] J. Gao, J. Lanchantin, M. L. Soffa, Y. J. Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In Proceedings of IEEE Security and Privacy Workshops, IEEE, San Francisco, USA, pp. 50–56, 2018. DOI: 10.1109/SPW.2018.00016. [112] J. F. Li, S. L. Ji, T. Y. Du, B. Li, T. Wang. Textbugger: Generating adversarial text against real-world applications. ArXiv: 1812.05271, 2018. [113] S. Samanta, S. Mehta. Towards crafting text adversarial samples. ArXiv: 1707.02812, 2017. [114] M. Iyyer, J. Wieting, K. Gimpel, L. Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. ArXiv: 1804.06059, 2018. [115] Q. Lei, L. F. Wu, P. Y. Chen, A. G. Dimakis, I. S. Dhillon, M. Witbrock. Discrete attacks and submodular optimization with applications to text classification. ArXiv: 1812.00151, 2018. [116] R. Jia, P. Liang. Adversarial examples for evaluating reading comprehension systems. ArXiv: 1707.07328, 2017. [117] Y. Belinkov, Y. Bisk. Synthetic and natural noise both break neural machine translation. ArXiv: 1711.02173, 2017. [118] M. H. Cheng, J. F. Yi, H. Zhang, P. Y. Chen, C. J. Hsieh. Seq2Sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. ArXiv: 1803.01128, 2018. [119] T. Niu, M. Bansal. Adversarial over-sensitivity and over-stability strategies for dialogue models. ArXiv: 1809.02079, 2018. [120] T. X. He, J. Glass. Detecting egregious responses in neural sequence-to-sequence models. ArXiv: 1809.04113, 2018. [121] H. C. Liu, T. Derr, Z. T. Liu, J. L Tang. Say what I want: Towards the dark side of neural dialogue models. ArXiv: 1909.06044, 2019. [122] M. Sharif, S. Bhagavatula, L. Bauer, M. K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, ACM, Vienna, Austria, pp. 1528–1540, 2016. DOI: 10.1145/2976749.2978392. [123] O. M. Parkhi, A. Vedaldi, A. Zisserman. Deep face recognition. Machine Learning 2015. [124] C. H. Xie, J. Y. Wang, Z. S. Zhang, Y. Y. Zhou, L. X. Xie, A. Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 1378–1387, 2017. DOI: 10.1109/ICCV.2017.153. [125] J. H. Metzen, M. C. Kumar, T. Brox, V. Fischer. Universal adversarial perturbations against semantic image segmentation. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 2774–2783, 2017. DOI: 10.1109/ICCV.2017.300. [126] S. S. Li, A. Neupane, S. Paul, C. Y. Song, S. V. Krishnamurthy, A. K. R. Chowdhury, A. Swami. Adversarial perturbations against real-time video classification systems. ArXiv: 1807.00458, 2018. [127] J. Kos, I. Fischer, D. Song. Adversarial examples for generative models. In Proceedings of IEEE Security and Privacy Workshops, IEEE, San Francisco, USA, pp. 36–42, 2018. DOI: 10.1109/SPW.2018.00014. [128] D. P. Kingma, M. Welling. Auto-encoding variational Bayes. ArXiv: 1312.6114, 2013. [129] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, O. Winther. Autoencoding beyond pixels using a learned similarity metric. ArXiv: 1512.09300, 2015. [130] K. Grosse, N. Papernot, P. Manoharan, M. Backes, P. McDaniel. Adversarial perturbations against deep neural networks for malware classification. ArXiv: 1606.04435, 2016. [131] D. Arp, M. Spreitzenbarth, H. Gascon, K. Rieck. DREBIN: Effective and explainable detection of android malware in your pocket. In Proceedings of Symposium Network Distributed System Security, Internet Society, San Diego, USA, 2014. [132] W. W. Hu, Y. Tan. Generating adversarial malware examples for black-box attacks based on GAN. ArXiv: 1702.05983, 2017. [133] H. S. Anderson, J. Woodbridge, B. Filar. DeepDGA: Adversarially-tuned domain generation and detection. In Proceedings of ACM Workshop on Artificial Intelligence and Security, ACM, Vienna, Austria, pp. 13–21, 2016. DOI: 10.1145/2996758.2996767. [134] T. Chugh, A. K. Jain. Fingerprint presentation attack detection: Generalization and efficiency. ArXiv: 1812.11574, 2018. [135] T. Chugh, K. Cao, A. K. Jain.  Fingerprint spoof buster: Use of minutiae-centered patches[J]. IEEE Transactions on Information Forensics and SecurityIEEE Transactions on Information Forensics and Security, 2018, 13(9): 2190-2202. doi: 10.1109/TIFS.2018.2812193 [136] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, P. Abbeel. Adversarial attacks on neural network policies. ArXiv: 1702.02284, 2017. [137] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, P. Abbeel. Trust region policy optimization. In Proceedings of the 31st International Conference on Machine Learning, JMLR, Lille, France, pp. 1889–1897, 2015. [138] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International conference on Machine Learning, PMLR, New York, USA, pp. 1928–1937, 2016.
•  [1] Ai-Hua Zheng, Zi-Han Chen, Cheng-Long Li, Jin Tang, Bin Luo. Learning Deep RGBT Representations for Robust Person Re-identification . International Journal of Automation and Computing, 2021, 18(): 1-14.  doi: 10.1007/s11633-020-1262-z [2] Punyanuch Borwarnginn, Worapan Kusakunniran, Sarattha Karnjanapreechakorn, Kittikhun Thongkanchorn. Knowing Your Dog Breed: Identifying a Dog Breed with Deep Learning . International Journal of Automation and Computing, 2021, 18(1): 45-54.  doi: 10.1007/s11633-020-1261-0 [3] Kittinun Aukkapinyo, Suchakree Sawangwong, Parintorn Pooyoi, Worapan Kusakunniran. Localization and Classification of Rice-grain Images Using Region Proposals-based Convolutional Neural Network . International Journal of Automation and Computing, 2020, 17(2): 233-246.  doi: 10.1007/s11633-019-1207-6 [4] Chang-Hao Zhu, Jie Zhang. Developing Soft Sensors for Polymer Melt Index in an Industrial Polymerization Process Using Deep Belief Networks . International Journal of Automation and Computing, 2020, 17(1): 44-54.  doi: 10.1007/s11633-019-1203-x [5] Fu-Qiang Liu, Zong-Yi Wang. Automatic “Ground Truth” Annotation and Industrial Workpiece Dataset Generation for Deep Learning . International Journal of Automation and Computing, 2020, 17(4): 539-550.  doi: 10.1007/s11633-020-1221-8 [6] Bin Hu, Jiacun Wang. Deep Learning Based Hand Gesture Recognition and UAV Flight Controls . International Journal of Automation and Computing, 2020, 17(1): 17-29.  doi: 10.1007/s11633-019-1194-7 [7] Viet Khanh Ha, Jin-Chang Ren, Xin-Ying Xu, Sophia Zhao, Gang Xie, Valentin Masero, Amir Hussain. Deep Learning Based Single Image Super-resolution: A Survey . International Journal of Automation and Computing, 2019, 16(4): 413-426.  doi: 10.1007/s11633-019-1183-x [8] Zhen-Jie Yao, Jie Bi, Yi-Xin Chen. Applying Deep Learning to Individual and Community Health Monitoring Data: A Survey . International Journal of Automation and Computing, 2018, 15(6): 643-655.  doi: 10.1007/s11633-018-1136-9 [9] Ting Zhang, Ri-Zhen Qin, Qiu-Lei Dong, Wei Gao, Hua-Rong Xu, Zhan-Yi Hu. Physiognomy: Personality Traits Prediction by Learning . International Journal of Automation and Computing, 2017, 14(4): 386-395.  doi: 10.1007/s11633-017-1085-8 [10] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, Qianli Liao. Why and When Can Deep-but Not Shallow-networks Avoid the Curse of Dimensionality:A Review . International Journal of Automation and Computing, 2017, 14(5): 503-519.  doi: 10.1007/s11633-017-1054-2 [11] Bo Zhao, Jiashi Feng, Xiao Wu, Shuicheng Yan. A Survey on Deep Learning-based Fine-grained Object Classification and Semantic Segmentation . International Journal of Automation and Computing, 2017, 14(2): 119-135.  doi: 10.1007/s11633-017-1053-3 [12] Guo-Bing Zhou, Jianxin Wu, Chen-Lin Zhang, Zhi-Hua Zhou. Minimal Gated Unit for Recurrent Neural Networks . International Journal of Automation and Computing, 2016, 13(3): 226-234.  doi: 10.1007/s11633-016-1006-2 [13] Rong-Min Cao,  Zhong-Sheng Hou,  Hui-Xing Zhou. Data-driven Nonparametric Model Adaptive Precision Control for Linear Servo Systems . International Journal of Automation and Computing, 2014, 11(5): 517-526.  doi: 10.1007/s11633-014-0834-1 [14] Bao-Chang Xu,  Xin-Le Liu. Identification Algorithm Based on the Approximate Least Absolute Deviation Criteria . International Journal of Automation and Computing, 2012, 9(5): 501-505.  doi: 10.1007/s11633-012-0673-x [15] Xu-Hui Bu,  Zhong-Sheng Hou. Stability of Iterative Learning Control with Data Dropouts via Asynchronous Dynamical System . International Journal of Automation and Computing, 2011, 8(1): 29-36.  doi: 10.1007/s11633-010-0551-3 [16] Han Xue,  Xun Li,  Hong-Xu Ma. Random Fuzzy Chance-constrained Programming Based on Adaptive Chaos Quantum Honey Bee Algorithm and Robustness Analysis . International Journal of Automation and Computing, 2010, 7(1): 115-122.  doi: 10.1007/s11633-010-0115-6 [17] Qing-Zheng Gao,  Xue-Jun Xie. Robustness Analysis of Discrete-time Indirect Model Reference Adaptive Control with Normalized Adaptive Laws . International Journal of Automation and Computing, 2010, 7(3): 381-388.  doi: 10.1007/s11633-010-0518-4 [18] Aymeric De Cabrol, Thibault Garcia, Patrick Bonnin, Maryline Chetto. A Concept of Dynamically Reconfigurable Real-time Vision System for Autonomous Mobile Robotics . International Journal of Automation and Computing, 2008, 5(2): 174-184.  doi: 10.1007/s11633-008-0174-0 [19] Chun-Yan Gao, Guang-Ren Duan, Xiang-Yu Meng. Robust H∞ Filter Design for 2D Discrete Systems in Roesser Model . International Journal of Automation and Computing, 2008, 5(4): 413-418.  doi: 10.1007/s11633-008-0413-4 [20] Bibhrajit Halder,  Nilanjan Sarkar. Robust Nonlinear Analytic Redundancy for Fault Detection and Isolation in Mobile Robot . International Journal of Automation and Computing, 2007, 4(2): 177-182.  doi: 10.1007/s11633-007-0177-2
###### 通讯作者: 陈斌, bchen63@163.com
• 1.

沈阳化工大学材料科学与工程学院 沈阳 110142

Figures (15)  / Tables (2)

## Metrics

Abstract Views (1092) PDF downloads (123) Citations (0)

## Adversarial Attacks and Defenses in Images, Graphs and Text: A Review

###### 1. Department of Computer Science and Engineering, Michigan State University, Michigan 48823, USA

Abstract: Deep neural networks (DNN) have achieved unprecedented success in numerous machine learning tasks in various domains. However, the existence of adversarial examples raises our concerns in adopting deep learning to safety-critical applications. As a result, we have witnessed increasing interests in studying attack and defense mechanisms for DNN models on different data types, such as images, graphs and text. Thus, it is necessary to provide a systematic and comprehensive overview of the main threats of attacks and the success of corresponding countermeasures. In this survey, we review the state of the art algorithms for generating adversarial examples and the countermeasures against adversarial examples, for three most popular data types, including images, graphs and text.

1Note that the softmax function at a temperature $T$ means: $softmax (x, T)_i = \dfrac{{\rm e}^{\frac{x_i}{T}}}{\displaystyle\sum_{j} {\rm e}^{\frac{x_j}{T}}}$, where $i = 0, 2, \cdots , K-1$.
Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang and Anil K. Jain. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. International Journal of Automation and Computing, vol. 17, no. 2, pp. 151-178, 2020. doi: 10.1007/s11633-019-1211-x
 Citation: Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang and Anil K. Jain. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. International Journal of Automation and Computing, vol. 17, no. 2, pp. 151-178, 2020.
Reference (138)

### Catalog

/

DownLoad:  Full-Size Img  PowerPoint