[FL]Adversarial Machine Learning (1)

Reading Alleviates Anxiety [Simba的阅读障碍治疗计划:#1]

Reading Notes for (NIST AI 100-2e2023)[https://csrc.nist.gov/pubs/ai/100/2/e2023/final\].

Adversarial Machine Learning

A Taxonomy and Terminology of Attacks and Mitigations

Section 1: Introduction

  • There are two broad classes of AI systems, based on their capabilities: Predictive AI (PredAI) and Generative AI (GenAI).

  • However, despite the signifcant progress that AI and machine learning (ML) have made in a number of different application domains, these technologies are also vulnerable to attacks that can cause spectacular failures with dire consequences.

  • Unlike cryptography, there are no information-theoretic security proofs for the widely used machine learning algorithms. Moreover, information-theoretic impossibility results have started to appear in the literature [102, 116] that set limits on the effectiveness of widely-used mitigation techniques. As a result, many of the advances in developing mitigations against different classes of attacks tend to be empirical and limited in nature.

Section 2: Pridictive AI Taxonomy

  • The attacker's objectives are shown as disjointed circles with the attacker's goal at the center of each circle: Availability breakdown , Integrity violations , and Privacy compromise.
  • Machine learning involves a TRAINING STAGE, in which a model is learned, and a DEPLOYMENT STAGE, in which the model is deployed on new, unlabeled data samples to generate predictions.

  • Adversarial machine learning literature predominantly considers adversarial attacks against AI systems that could occur at either the training stage or the ML deployment stage. During the ML training stage, the attacker might control part of the training data, their labels, the model parameters, or the code of ML algorithms, resulting in different types of poisoning attacks. During the ML deployment stage, the ML model is already trained, and the adversary could mount evasion attacks to create integrity violations and change the ML model's predictions, as well as privacy attacks to infer sensitive information about the training data or the ML model.

  • Training-time attacks. Poisoning Attack. Data poisoning attacks are applicable to all learning paradigms, while model poisoning attacks are most prevalent in federated learning, where clients send local model updates to the aggregating server, and in supply-chain attacks where malicious code may be added to the model by suppliers of model technology.

  • Deployment-time attack. Adversarial Example.

  • Attacker Goals and Objectives. Availability breakdown, Integrity violations, and Privacy compromise.

  • Privacy Compromise. Attackers might be interested in learning information about the training data (resulting in DATA PRIVACY attacks) or about the ML model (resulting in MODEL PRIVACY attacks). The attacker could have different objectives for compromising the privacy of training data, such as DATA RECONSTRUCTION (inferring content or features of training data), MEMBERSHIP-INFERENCE ATTACKS (inferring the presence of data in the training set), data EXTRACTION (ability to extract training data from generative models), and PROPERTY INFERENCE (inferring properties about the training data distribution). MODEL EXTRACTION is a model privacy attack in which attackers aim to extract information about the model.

  • Attacker Capabilities. Training Data Control. Model Control. Testing Data Control. Label Limit. Source Code Control. Query Access.

  • Attacker Knowledge. White-box attacks. These assume that the attacker operates with full knowledge about the ML system, including the training data, model architecture, and model hyper-parameters. Black-box attacks. These attacks assume minimal knowledge about the ML system. An adversary might get query access to the model, but they have no other information about how the model is trained. Gray-box attacks. There are a range of gray-box attacks that capture adversarial knowledge between black-box and white-box attacks. Suciu et al. introduced a framework to classify gray-box attacks. An attacker might know the model architecture but not its parameters, or the attacker might know the model and its parameters but not the training data.

  • Data Modality: Image. Text. Audio. Video. Cybersecurity. Tabular Data.

  • Recently, the use of ML models trained on multimodal data has gained traction, particularly the combination of image and text data modalities. Several papers have shown that multimodal models may provide some resilience against attacks, but other papers show that multimodal models themselves could be vulnerable to attacks mounted on all modalities at the same time.

  • An interesting open challenge is to test and characterize the resilience of a variety of multimodal ML against evasion, poisoning, and privacy attacks.

  • Evasion Attacks and Mitigations. Methods for creating adversarial examples in black-box settings include zeroth-order optimization, discrete optimization, and Bayesian optimization, as well as transferability, which involves the white-box generation of adversarial examples on a different model architecture before transferring them to the target model.

  • The most promising directions for mitigating the critical threat of evasion attacks are adversarial training (iteratively generating and inserting adversarial examples with their correct labels at training time); certifed techniques, such as randomized smoothing (evaluating ML predic-

    tion under noise); and formal verifcation techniques [112, 154] (applying formal method techniques to verify the model's output). Nevertheless, these methods come with different limitations, such as decreased accuracy for adversarial training and randomized smoothing, and computational complexity for formal methods. There is an inherent trade-off between robustness and accuracy [297, 302, 343]. Similarly, there are trade-offs between a model's robustness and fairness guarantees.

相关推荐
rocksun4 分钟前
GraphRAG vs. RAG:差异详解
人工智能
一块plus12 分钟前
什么是去中心化 AI?区块链驱动智能的初学者指南
人工智能·后端·算法
张德锋13 分钟前
Pytorch实现mnist手写数字识别
机器学习
txwtech20 分钟前
第10.4篇 使用预训练的目标检测网络
人工智能·计算机视觉·目标跟踪
羊小猪~~32 分钟前
【NLP入门系列四】评论文本分类入门案例
人工智能·自然语言处理·分类
roman_日积跬步-终至千里36 分钟前
【学习线路】机器学习线路概述与内容关键点说明
人工智能·学习·机器学习
静心问道41 分钟前
APE:大语言模型具有人类水平的提示工程能力
人工智能·算法·语言模型·大模型
zskj_zhyl44 分钟前
科技有温度:七彩喜智慧康养平台,为银发生活织就“数字守护网”
人工智能·科技·生活
嘉讯科技HIS系统1 小时前
嘉讯科技:医疗信息化、数字化、智能化三者之间的关系和区别
大数据·数据库·人工智能·科技·智慧医疗
音视频牛哥1 小时前
计算机视觉的新浪潮:扩散模型(Diffusion Models)技术剖析与应用前景
人工智能·计算机视觉·ai·音视频·实时音视频·扩散模型