[隐私计算研习社]
本文是由白小鱼博主整理的ICCV 2023会议中，与联邦学习相关的论文合集及摘要翻译。
MAS: Towards Resource-Efficient Federated Multiple-Task Learning
Authors: Weiming Zhuang; Yonggang Wen; Lingjuan Lyu; Shuai Zhang
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url:(openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) is an emerging distributed machine learning method that empowers in-situ model training on decentralized edge devices. However, multiple simultaneous FL tasks could overload resource-constrained devices. In this work, we propose the first FL system to effectively coordinate and train multiple simultaneous FL tasks. We first formalize the problem of training simultaneous FL tasks. Then, we present our new approach, MAS (Merge and Split), to optimize the performance of training multiple simultaneous FL tasks. MAS starts by merging FL tasks into an all-in-one FL task with a multi-task architecture. After training for a few rounds, MAS splits the all-in-one FL task into two or more FL tasks by using the affinities among tasks measured during the all-in-one training. It then continues training each split of FL tasks based on model parameters from the all-in-one training. Extensive experiments demonstrate that MAS outperforms other methods while reducing training time by 2x and reducing energy consumption by 40%. We hope this work will inspire the community to further study and optimize training simultaneous FL tasks.
abstractTranslation: 联邦学习（FL）是一种新兴的分布式机器学习方法，可以在分散的边缘设备上进行原位模型训练。然而，多个同步 FL 任务可能会使资源受限的设备过载。在这项工作中，我们提出了第一个有效协调和训练多个同步 FL 任务的 FL 系统。我们首先将同步 FL 任务的训练问题形式化。然后，我们提出了新方法 MAS（合并和拆分），以优化训练多个同步 FL 任务的性能。MAS 首先将 FL 任务合并为具有多任务架构的一体化 FL 任务。经过几轮训练后，MAS 利用在一体式训练期间测量的任务之间的亲和力，将一体式 FL 任务拆分为两个或多个 FL 任务。然后，它根据一体化训练中的模型参数继续训练 FL 任务的每个部分。大量实验表明，MAS 的性能优于其他方法，同时将训练时间缩短 2 倍，能耗降低 40%。我们希望这项工作能够激励社区进一步研究和优化同步 FL 任务的训练。
Notes:
Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster ConvergenceAuthors: Yuhao Zhou; Mingjia Shi; Yuanxi Li; Yanan Sun; Qing Ye; Jiancheng Lv
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Reducing communication overhead in federated learning (FL) is challenging but crucial for large-scale distributed privacy-preserving machine learning. While methods utilizing sparsification or other techniques can largely reduce the communication overhead, the convergence rate is also greatly compromised. In this paper, we propose a novel method named Single-Step Synthetic Features Compressor (3SFC) to achieve communication-efficient FL by directly constructing a tiny synthetic dataset containing synthetic features based on raw gradients. Therefore, 3SFC can achieve an extremely low compression rate when the constructed synthetic dataset contains only one data sample. Additionally, the compressing phase of 3SFC utilizes a similarity-based objective function so that it can be optimized with just one step, considerably improving its performance and robustness. To minimize the compressing error, error feedback (EF) is also incorporated into 3SFC. Experiments on multiple datasets and models suggest that 3SFC has significantly better convergence rates compared to competing methods with lower compression rates (i.e., up to 0.02%). Furthermore, ablation studies and visualizations show that 3SFC can carry more information than competing methods for every communication round, further validating its effectiveness.
abstractTranslation: 减少联邦学习（FL）中的通信开销具有挑战性，但对于大规模分布式隐私保护机器学习至关重要。虽然利用稀疏化或其他技术的方法可以大大减少通信开销，但收敛速度也受到很大影响。在本文中，我们提出了一种名为单步合成特征压缩器（3SFC）的新方法，通过直接构建包含基于原始梯度的合成特征的微小合成数据集来实现高效通信的 FL。因此，当构建的合成数据集仅包含一个数据样本时，3SFC可以实现极低的压缩率。此外，3SFC的压缩阶段采用了基于相似性的目标函数，只需一步即可优化，大大提高了其性能和鲁棒性。为了最大限度地减少压缩误差，3SFC 中还集成了误差反馈 (EF)。对多个数据集和模型的实验表明，与压缩率较低（即高达 0.02%）的竞争方法相比，3SFC 具有明显更好的收敛率。此外，消融研究和可视化表明，3SFC 在每一轮通信中都可以比竞争方法携带更多信息，进一步验证了其有效性。
TARGET: Federated Class-Continual Learning via Exemplar-Free DistillationAuthors: Jie Zhang; Chen Chen; Weiming Zhuang; Lingjuan Lyu
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: This paper focuses on an under-explored yet important problem: Federated Class-Continual Learning (FCCL), where new classes are dynamically added in federated learning. Existing FCCL works suffer from various limitations, such as requiring additional datasets or storing the private data from previous tasks. In response, we first demonstrate that non-IID data exacerbates catastrophic forgetting issue in FL. Then we propose a novel method called TARGET (federatTed clAss-continual leaRninG via Exemplar-free disTillation), which alleviates catastrophic forgetting in FCCL while preserving client data privacy. Our proposed method leverages the previously trained global model to transfer knowledge of old tasks to the current task at the model level. Moreover, a generator is trained to produce synthetic data to simulate the global distribution of data on each client at the data level. Compared to previous FCCL methods, TARGET does not require any additional datasets or storing real data from previous tasks, which makes it ideal for data-sensitive scenarios.
abstractTranslation: 本文重点关注一个尚未充分探索但重要的问题：联邦类持续学习（FCCL），其中新类在联邦学习中动态添加。现有的 FCCL 工作受到各种限制，例如需要额外的数据集或存储以前任务的私有数据。作为回应，我们首先证明非独立同分布数据会加剧 FL 中的灾难性遗忘问题。然后，我们提出了一种称为 TARGET（联邦分类-通过无样本蒸馏进行持续学习）的新方法，该方法可以减轻 FCCL 中的灾难性遗忘，同时保护客户数据隐私。我们提出的方法利用先前训练的全局模型将旧任务的知识转移到模型级别的当前任务。此外，生成器经过训练可以生成合成数据，以在数据级别模拟每个客户端上数据的全局分布。与以前的 FCCL 方法相比，TARGET 不需要任何额外的数据集或存储以前任务的真实数据，这使得它非常适合数据敏感的场景。
GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning
Authors: Jianqing Zhang; Yang Hua; Hao Wang; Tao Song; Zhengui Xue; Ruhui Ma; Jian Cao; Haibing Guan
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated Learning (FL) is popular for its privacy-preserving and collaborative learning capabilities. Recently, personalized FL (pFL) has received attention for its ability to address statistical heterogeneity and achieve personalization in FL. However, from the perspective of feature extraction, most existing pFL methods only focus on extracting global or personalized feature information during local training, which fails to meet the collaborative learning and personalization goals of pFL. To address this, we propose a new pFL method, named GPFL, to simultaneously learn global and personalized feature information on each client. We conduct extensive experiments on six datasets in three statistically heterogeneous settings and show the superiority of GPFL over ten state-of-the-art methods regarding effectiveness, scalability, fairness, stability, and privacy. Besides, GPFL mitigates overfitting and outperforms the baselines by up to 8.99% in accuracy.
abstractTranslation: 联邦学习（FL）因其隐私保护和协作学习功能而广受欢迎。最近，个性化 FL (pFL) 因其解决统计异质性和实现 FL 个性化的能力而受到关注。然而，从特征提取的角度来看，现有的pFL方法大多只注重在局部训练过程中提取全局或个性化特征信息，无法满足pFL的协作学习和个性化目标。为了解决这个问题，我们提出了一种新的 pFL 方法，称为 GPFL，以同时学习每个客户端的全局和个性化特征信息。我们在三种统计异构设置中对六个数据集进行了广泛的实验，并证明了 GPFL 在有效性、可扩展性、公平性、稳定性和隐私性方面优于十种最先进的方法。此外，GPFL 可以缓解过度拟合，并且准确率比基线高出高达 8.99%。
Notes:
PDF CODE [SUPP (openaccess.thecvf.com/content/ICC...)
Generative Gradient Inversion via Over-Parameterized Networks in Federated LearningAuthors: Chi Zhang; Zhang Xiaoman; Ekanut Sotthiwat; Yanyu Xu; Ping Liu; Liangli Zhen; Yong Liu
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning has gained recognitions as a secure approach for safeguarding local private data in collaborative learning. But the advent of gradient inversion research has posed significant challenges to this premise by enabling a third-party to recover groundtruth images via gradients. While prior research has predominantly focused on low-resolution images and small batch sizes, this study highlights the feasibility of reconstructing complex images with high resolutions and large batch sizes. The success of the proposed method is contingent on constructing an over-parameterized convolutional network, so that images are generated before fitting to the gradient matching requirement. Practical experiments demonstrate that the proposed algorithm achieves high-fidelity image recovery, surpassing state-of-the-art competitors that commonly fail in more intricate scenarios. Consequently, our study shows that local participants in a federated learning system are vulnerable to potential data leakage issues. Source code is available at github.com/czhang024/C....
abstractTranslation: 联邦学习作为一种在协作学习中保护本地私有数据的安全方法已获得认可。但梯度反演研究的出现对这一前提提出了重大挑战，它使第三方能够通过梯度恢复真实图像。虽然之前的研究主要集中在低分辨率图像和小批量大小，但这项研究强调了重建具有高分辨率和大批量大小的复杂图像的可行性。该方法的成功取决于构建超参数化卷积网络，以便在拟合梯度匹配要求之前生成图像。实际实验表明，所提出的算法实现了高保真图像恢复，超越了在更复杂的场景中通常失败的最先进的竞争对手。因此，我们的研究表明，联邦学习系统中的本地参与者很容易受到潜在数据泄露问题的影响。源代码可在 [github.com/czhang024/C...] 获取。
Notes:CODE[SUPP(openaccess.thecvf.com/content/ICC...)
Global Balanced Experts for Federated Long-Tailed Learning
Authors: Yaopei Zeng; Lei Liu; Li Liu; Li Shen; Shaoguo Liu; Baoyuan Wu
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url:(openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) is a prevalent distributed machine learning approach that enables collaborative training of a global model across multiple devices without sharing local data. However, the presence of long-tailed data can negatively deteriorate the model's performance in real-world FL applications. Moreover, existing re-balance strategies are less effective for the federated long-tailed issue when directly utilizing local label distribution as the class prior at the clients' side. To this end, we propose a novel Global Balanced Multi-Expert (GBME) framework to optimize a balanced global objective, which does not require additional information beyond the standard FL pipeline. In particular, a proxy is derived from the accumulated gradients uploaded by the clients after local training, and is shared by all clients as the class prior for re-balance training. Such a proxy can also guide the client grouping to train a multi-expert model, where the knowledge from different clients can be aggregated via the ensemble of different experts corresponding to different client groups. To further strengthen the privacy-preserving ability, we present a GBME-p algorithm with a theoretical guarantee to prevent privacy leakage from the proxy. Extensive experiments on long-tailed decentralized datasets demonstrate the effectiveness of GBME and GBME-p, both of which show superior performance to state-of-the-art methods.
abstractTranslation: 联邦学习 (FL) 是一种流行的分布式机器学习方法，可以跨多个设备协作训练全局模型，而无需共享本地数据。然而，长尾数据的存在可能会降低模型在现实 FL 应用中的性能。此外，当直接利用本地标签分布作为客户端的优先级时，现有的重新平衡策略对于联邦长尾问题的效果较差。为此，我们提出了一种新颖的全球平衡多专家（GBME）框架来优化平衡的全球目标，该框架不需要标准 FL 管道之外的其他信息。特别是，代理是根据本地训练后客户端上传的累积梯度得出的，并由所有客户端共享作为重新平衡训练的先验类。这样的代理还可以指导客户分组来训练多专家模型，其中可以通过对应于不同客户组的不同专家的集合来聚合来自不同客户的知识。为了进一步增强隐私保护能力，我们提出了一种GBME-p算法，该算法具有防止代理泄露隐私的理论保证。对长尾分散数据集的大量实验证明了 GBME 和 GBME-p 的有效性，两者都显示出优于最先进方法的性能。
Efficient Model Personalization in Federated Learning via Client-Specific Prompt GenerationAuthors: Fu-En Yang; Chien-Yi Wang; Yu-Chiang Frank Wang
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url:(openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) emerges as a decentralized learning framework which trains models from multiple distributed clients without sharing their data to preserve privacy. Recently, large-scale pre-trained models (e.g., Vision Transformer) have shown a strong capability of deriving robust representations. However, the data heterogeneity among clients, the limited computation resources, and the communication bandwidth restrict the deployment of large-scale models in FL frameworks. To leverage robust representations from large-scale models while enabling efficient model personalization for heterogeneous clients, we propose a novel personalized FL framework of client-specific Prompt Generation (pFedPG), which learns to deploy a personalized prompt generator at the server for producing client-specific visual prompts that efficiently adapts frozen backbones to local data distributions. Our proposed framework jointly optimizes the stages of personalized prompt adaptation locally and personalized prompt generation globally. The former aims to train visual prompts that adapt foundation models to each client, while the latter observes local optimization directions to generate personalized prompts for all clients. Through extensive experiments on benchmark datasets, we show that our pFedPG is favorable against state-of-the-art personalized FL methods under various types of data heterogeneity, allowing computation and communication efficient model personalization.
abstractTranslation: 联邦学习 (FL) 作为一种去中心化学习框架而出现，它可以训练来自多个分布式客户端的模型，而无需共享数据以保护隐私。最近，大规模预训练模型（例如 Vision Transformer）已经表现出强大的推导鲁棒表示的能力。然而，客户端之间的数据异构性、有限的计算资源和通信带宽限制了 FL 框架中大规模模型的部署。为了利用大规模模型的鲁棒表示，同时为异构客户端实现高效的模型个性化，我们提出了一种新颖的客户端特定提示生成（pFedPG）的个性化 FL 框架，它学习在服务器上部署个性化提示生成器以生成客户端特定的视觉提示可以有效地使冻结的主干适应本地数据分布。我们提出的框架联邦优化了本地个性化提示适应和全局个性化提示生成的阶段。前者旨在训练视觉提示，使基础模型适应每个客户，而后者观察局部优化方向，为所有客户生成个性化提示。通过对基准数据集的大量实验，我们表明我们的 pFedPG 在各种类型的数据异构性下优于最先进的个性化 FL 方法，从而实现计算和通信高效的模型个性化。
FedPD: Federated Open Set Recognition with Parameter Disentanglement
Authors: Chen Yang; Meilu Zhu; Yifan Liu; Yixuan Yuan
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url:(openaccess.thecvf.com/content/ICC...Abstract "https://openaccess.thecvf.com/content/ICCV2023/html/Yang_FedPD_Federated_Open_Set_Recognition_with_Parameter_Disentanglement_ICCV_2023_paper.html)Abstract"): Existing federated learning (FL) approaches are deployed under the unrealistic closed-set setting, with both training and testing classes belong to the same set, which makes the global model fail to identify the unseen classes as `unknown'. To this end, we aim to study a novel problem of federated open-set recognition (FedOSR), which learns an open-set recognition (OSR) model under federated paradigm such that it classifies seen classes while at the same time detects unknown classes. In this work, we propose a parameter disentanglement guided federated open-set recognition (FedPD) algorithm to address two core challenges of FedOSR: cross-client inter-set interference between learning closed-set and open-set knowledge and cross-client intra-set inconsistency by data heterogeneity. The proposed FedPD framework mainly leverages two modules, i.e., local parameter disentanglement (LPD) and global divide-and-conquer aggregation (GDCA), to first disentangle client OSR model into different subnetworks, then align the corresponding parts cross clients for matched model aggregation. Specifically, on the client side, LPD decouples an OSR model into a closed-set subnetwork and an open-set subnetwork by the task-related importance, thus preventing inter-set interference. On the server side, GDCA first partitions the two subnetworks into specific and shared parts, and subsequently aligns the corresponding parts through optimal transport to eliminate parameter misalignment. Extensive experiments on various datasets demonstrate the superior performance of our proposed method.abstractTranslation: 现有的联邦学习（FL）方法部署在不切实际的封闭集设置下，训练类和测试类属于同一集合，这使得全局模型无法将未见过的类识别为"未知"。为此，我们的目标是研究联邦开放集识别（FedOSR）的新问题，该问题在联邦范式下学习开放集识别（OSR）模型，以便对可见类进行分类，同时检测未知类。在这项工作中，我们提出了一种参数解纠缠引导的联邦开放集识别（FedPD）算法来解决 FedOSR 的两个核心挑战：学习闭集和开放集知识之间的跨客户端集间干扰以及跨客户端内部识别数据异构性造成的不一致。所提出的FedPD框架主要利用两个模块，即局部参数解缠（LPD）和全局分而治之聚合（GDCA），首先将客户端OSR模型解缠到不同的子网络中，然后跨客户端对齐相应部分以进行匹配的模型聚合。具体来说，在客户端，LPD根据任务相关重要性将OSR模型解耦为闭集子网和开集子网，从而防止集间干扰。在服务器端，GDCA首先将两个子网划分为特定部分和共享部分，然后通过最佳传输对齐相应部分，以消除参数错位。对各种数据集的广泛实验证明了我们提出的方法的优越性能。Notes:
CODE
Personalized Semantics Excitation for Federated Image Classification
Authors: Haifeng Xia; Kai Li; Zhengming Ding
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning casts a light on the collaboration of distributed local clients with privacy protected to attain a more generic global model. However, significant distribution shift in input/label space across different clients makes it challenging to well generalize to all clients, which motivates personalized federated learning (PFL). Existing PFL methods typically customize the local model by fine-tuning with limited local supervision and the global model regularizer, which secures local specificity but risks ruining the global discriminative knowledge. In this paper, we propose a novel Personalized Semantics Excitation (PSE) mechanism to breakthrough this limitation by exciting and fusing personalized semantics from the global model during local model customization. Specifically, PSE explores channel-wise gradient differentiation across global and local models to identify important low-level semantics mostly from convolutional layers which are embedded into the client-specific training. In addition, PSE deploys the collaboration of global and local models to enrich high-level feature representations and facilitate the robustness of client classifier through a cross-model attention module. Extensive experiments and analysis on various image classification benchmarks demonstrate the effectiveness and advantage of our method over the state-of-the-art PFL methods.
abstractTranslation: 联邦学习揭示了分布式本地客户端在隐私保护下的协作，以获得更通用的全球模型。然而，不同客户端之间输入/标签空间的显着分布变化使得很好地推广到所有客户端变得具有挑战性，这激发了个性化联邦学习（PFL）。现有的 PFL 方法通常通过有限的局部监督和全局模型正则化器进行微调来定制局部模型，这保证了局部特异性，但有破坏全局判别性知识的风险。在本文中，我们提出了一种新颖的个性化语义激励（PSE）机制，通过在局部模型定制过程中激励和融合来自全局模型的个性化语义来突破这一限制。具体来说，PSE 探索全局和局部模型中的通道梯度差异，以从嵌入到特定于客户端的训练中的卷积层中识别重要的低级语义。此外，PSE 部署全局和局部模型的协作来丰富高级特征表示，并通过跨模型注意模块促进客户端分类器的鲁棒性。对各种图像分类基准的大量实验和分析证明了我们的方法相对于最先进的 PFL 方法的有效性和优势。
Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive CollaborationAuthors: Xinghao Wu; Xuefeng Liu; Jianwei Niu; Guogang Zhu; Shaojie Tang
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Personalized federated learning (PFL) reduces the impact of non-independent and identically distributed (non-IID) data among clients by allowing each client to train a personalized model when collaborating with others. A key question in PFL is to decide which parameters of a client should be localized or shared with others. In current mainstream approaches, all layers that are sensitive to non-IID data (such as classifier layers) are generally personalized. The reasoning behind this approach is understandable, as localizing parameters that are easily influenced by non-IID data can prevent potential negative effects of collaboration. However, we believe that this approach is too conservative for collaboration. For example, for a certain client, even if its parameters are easily influenced by non-IID data, it can still benefit by sharing these parameters with clients having similar data distribution. This observation emphasizes the importance of considering not only the sensitivity to non-IID data but also the similarity of data distribution when determining which parameters should be localized in PFL. This paper introduces a novel guideline for client collaboration in PFL. Unlike existing approaches that prohibit all collaboration of sensitive parameters, our guideline allows clients to share more parameters with others, leading to improved model performance. Additionally, we propose a new PFL method named FedCAC, which employs a quantitative metric to evaluate each parameter's sensitivity to non-IID data and carefully selects collaborators based on this evaluation. Experimental results demonstrate that FedCAC enables clients to share more parameters with others, resulting in superior performance compared to state-of-the-art methods, particularly in scenarios where clients have diverse distributions.
abstractTranslation: 个性化联邦学习 (PFL) 允许每个客户端在与其他客户端协作时训练个性化模型，从而减少客户端之间非独立且同分布 (non-IID) 数据的影响。PFL 中的一个关键问题是决定客户端的哪些参数应该本地化或与其他人共享。在当前的主流方法中，所有对非独立同分布数据敏感的层（例如分类器层）通常都是个性化的。这种方法背后的原因是可以理解的，因为本地化容易受非独立同分布数据影响的参数可以防止协作的潜在负面影响。然而，我们认为这种方法对于协作来说过于保守。例如，对于某个客户端，即使其参数很容易受到非独立同分布数据的影响，它仍然可以通过与具有相似数据分布的客户端共享这些参数而受益。这一观察结果强调了在确定 PFL 中应定位哪些参数时不仅要考虑对非 IID 数据的敏感性，还要考虑数据分布的相似性的重要性。本文介绍了 PFL 中客户协作的新颖指南。与禁止所有敏感参数协作的现有方法不同，我们的指南允许客户与其他人共享更多参数，从而提高模型性能。此外，我们提出了一种名为 FedCAC 的新 PFL 方法，该方法采用定量指标来评估每个参数对非独立同分布数据的敏感性，并根据此评估仔细选择合作者。实验结果表明，FedCAC 使客户端能够与其他人共享更多参数，从而与最先进的方法相比具有更优越的性能，特别是在客户端具有不同分布的情况下。
Enhancing Privacy Preservation in Federated Learning via Learning Rate PerturbationAuthors: Guangnian Wan; Haitao Du; Xuejing Yuan; Jun Yang; Meiling Chen; Jie Xu
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) is a privacy-enhanced distributed machine learning framework, in which multiple clients collaboratively train a global model by exchanging their model updates without sharing local private data. However, the adversary can use gradient inversion attacks to reveal the clients' privacy from the shared model updates. Previous attacks assume the adversary can infer the local learning rate of each client, while we observe that: (1) using the uniformly distributed random local learning rates does not incur much accuracy loss of the global model, and (2) personalizing local learning rates can mitigate the drift issue which is caused by non-IID (identically and independently distributed) data. Moreover, we theoretically derive a convergence guarantee to FedAvg with uniformly perturbed local learning rates. Therefore, by perturbing the learning rate of each client with random noise, we propose a learning rate perturbation (LRP) defense against gradient inversion attacks. Specifically, for classification tasks, we adapt LPR to ada-LPR by personalizing the expectation of each local learning rate. The experiments show that our defenses can well enhance privacy preservation against existing gradient inversion attacks, and LRP outperforms 5 baseline defenses against a state-of-the-art gradient inversion attack. In addition, our defenses only incur minor accuracy reductions (less than 0.5%) of the global model. So they are effective in real applications.
abstractTranslation: 联邦学习 (FL) 是一种隐私增强的分布式机器学习框架，其中多个客户端通过交换模型更新来协作训练全局模型，而无需共享本地私有数据。然而，攻击者可以使用梯度反转攻击来从共享模型更新中泄露客户的隐私。以前的攻击假设对手可以推断每个客户端的局部学习率，而我们观察到：（1）使用均匀分布的随机局部学习率不会导致全局模型的准确性损失太多，以及（2）个性化局部学习率可以减轻由非 IID（相同且独立分布）数据引起的漂移问题。此外，我们从理论上推导了具有均匀扰动的局部学习率的 FedAvg 的收敛保证。因此，通过用随机噪声扰动每个客户端的学习率，我们提出了一种针对梯度反转攻击的学习率扰动（LRP）防御。具体来说，对于分类任务，我们通过个性化每个局部学习率的期望来将 LPR 调整为 ada-LPR。实验表明，我们的防御措施可以很好地增强针对现有梯度反转攻击的隐私保护，并且 LRP 优于针对最先进的梯度反转攻击的 5 种基线防御措施。此外，我们的防御仅导致全局模型的准确性略有下降（小于 0.5%）。所以它们在实际应用中是有效的。
Notes:
SUPP
When Do Curricula Work in Federated Learning?Authors: Saeed Vahidian; Sreevatsank Kadaveru; Woonjoon Baek; Weijia Wang; Vyacheslav Kungurtsev; Chen Chen; Mubarak Shah; Bill Lin
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: An oft-cited open problem of federated learning is the existence of data heterogeneity among clients. One path- way to understanding the drastic accuracy drop in feder- ated learning is by scrutinizing the behavior of the clients' deep models on data with different levels of "difficulty", which has been left unaddressed. In this paper, we investi- gate a different and rarely studied dimension of FL: ordered learning. Specifically, we aim to investigate how ordered learning principles can contribute to alleviating the hetero- geneity effects in FL. We present theoretical analysis and conduct extensive empirical studies on the efficacy of or- derings spanning three kinds of learning: curriculum, anti- curriculum, and random curriculum. We find that curricu- lum learning largely alleviates non-IIDness. Interestingly, the more disparate the data distributions across clients the more they benefit from ordered learning. We provide analysis explaining this phenomenon, specifically indicating how curriculum training appears to make the objective land- scape progressively less convex, suggesting fast converging iterations at the beginning of the training procedure. We derive quantitative results of convergence for both convex and nonconvex objectives by modeling the curriculum train- ing on federated devices as local SGD with locally biased stochastic gradients. Also, inspired by ordered learning, we propose a novel client selection technique that benefits from the real-world disparity in the clients. Our proposed approach to client selection has a synergic effect when applied together with ordered learning in FL.
abstractTranslation: 联邦学习经常被提及的一个开放性问题是客户端之间存在数据异构性。理解联邦学习中准确率急剧下降的一种途径是仔细检查客户深度模型在不同"难度"级别的数据上的行为，而这一问题尚未得到解决。在本文中，我们研究了 FL 的一个不同且很少研究的维度：有序学习。具体来说，我们的目标是研究有序学习原则如何有助于减轻 FL 中的异质性影响。我们对跨越三种学习的秩序的功效进行了理论分析并进行了广泛的实证研究：课程、反课程和随机课程。我们发现课程学习在很大程度上缓解了非独立同分布现象。有趣的是，客户之间的数据分布越不同，他们从有序学习中受益就越多。我们提供了解释这种现象的分析，特别指出了课程培训如何使目标景观逐渐变得不那么凸，表明在培训过程开始时进行快速收敛迭代。我们通过将联邦设备上的课程训练建模为具有局部偏置随机梯度的局部 SGD，得出凸和非凸目标收敛的定量结果。此外，受有序学习的启发，我们提出了一种新颖的客户选择技术，该技术受益于客户的现实世界差异。我们提出的客户选择方法与 FL 中的有序学习一起应用时会产生协同效应。
Communication-Efficient Vertical Federated Learning with Limited Overlapping SamplesAuthors: Jingwei Sun; Ziyue Xu; Dong Yang; Vishwesh Nath; Wenqi Li; Can Zhao; Daguang Xu; Yiran Chen; Holger R. Roth
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url:(openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning is a popular collaborative learning approach that enables clients to train a global model without sharing their local data. Vertical federated learning (VFL) deals with scenarios in which the data on clients have different feature spaces but share some overlapping samples. Existing VFL approaches suffer from high communication costs and cannot deal efficiently with limited overlapping samples commonly seen in the real world. We propose a practical vertical federated learning (VFL) framework called one-shot VFL that can solve the communication bottleneck and the problem of limited overlapping samples simultaneously based on semi-supervised learning. We also propose few-shot VFL to improve the accuracy further with just one more communication round between the server and the clients. In our proposed framework, the clients only need to communicate with the server once or only a few times. We evaluate the proposed VFL framework on both image and tabular datasets. Our methods can improve the accuracy by more than 46.5% and reduce the communication cost by more than 330xcompared with state-of-the-art VFL methods when evaluated on CIFAR-10. Our code is publicly available.
abstractTranslation: 联邦学习是一种流行的协作学习方法，使客户能够在不共享本地数据的情况下训练全局模型。纵向联邦学习（VFL）处理客户端数据具有不同特征空间但共享一些重叠样本的场景。现有的 VFL 方法面临着高昂的通信成本，并且无法有效地处理现实世界中常见的有限重叠样本。我们提出了一种实用的纵向联邦学习（VFL）框架，称为一次性VFL，它可以基于半监督学习同时解决通信瓶颈和有限重叠样本的问题。我们还提出了少样本 VFL，只需在服务器和客户端之间多进行一轮通信即可进一步提高准确性。在我们提出的框架中，客户端只需要与服务器通信一次或几次。我们在图像和表格数据集上评估了所提出的 VFL 框架。在 CIFAR-10 上进行评估时，与最先进的 VFL 方法相比，我们的方法可以将准确性提高 46.5% 以上，并将通信成本降低 330 倍以上。我们的代码是公开的。
FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning
Authors: Guangyu Sun; Matias Mendieta; Jun Luo; Shandong Wu; Chen Chen
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Personalized Federated Learning (PFL) represents a promising solution for decentralized learning in heterogeneous data environments. Partial model personalization has been proposed to improve the efficiency of PFL by selectively updating local model parameters instead of aggregating all of them. However, previous work on partial model personalization has mainly focused on Convolutional Neural Networks (CNNs), leaving a gap in understanding how it can be applied to other popular models such as Vision Transformers (ViTs). In this work, we investigate where and how to partially personalize a ViT model. Specifically, we empirically evaluate the sensitivity to data distribution of each type of layer. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix, which leverages plugins to transfer information from the aggregated model to the local client as a personalization. Finally, we evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness in improving the model's performance compared to several advanced PFL methods.
abstractTranslation: 个性化联邦学习（PFL）代表了异构数据环境中去中心化学习的一种有前景的解决方案。部分模型个性化被提出来通过选择性地更新局部模型参数而不是聚合所有参数来提高 PFL 的效率。然而，之前关于部分模型个性化的工作主要集中在卷积神经网络（CNN）上，在理解如何将其应用于视觉变压器（ViT）等其他流行模型方面存在差距。在这项工作中，我们研究了在何处以及如何部分个性化 ViT 模型。具体来说，我们凭经验评估每种类型层对数据分布的敏感性。基于自注意力层和分类头是 ViT 最敏感部分的见解，我们提出了一种名为 FedPerfix 的新颖方法，它利用插件将信息从聚合模型传输到本地客户端作为个性化。最后，我们在 CIFAR-100、OrganAMNIST 和 Office-Home 数据集上评估了所提出的方法，并证明了与几种先进的 PFL 方法相比，其在提高模型性能方面的有效性。
Notes:
L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation LearningAuthors: Yasar Abbas Ur Rehman; Yan Gao; Pedro Porto Buarque de Gusmao; Mina Alibeigi; Jiajun Shen; Nicholas D. Lane
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: The ubiquity of camera-enabled devices has led to large amounts of unlabeled image data being produced at the edge. The integration of self-supervised learning (SSL) and federated learning (FL) into one coherent system can potentially offer data privacy guarantees while also advancing the quality and robustness of the learned visual representations without needing to move data around. However, client bias and divergence during FL aggregation caused by data heterogeneity limits the performance of learned visual representations on downstream tasks. In this paper, we propose a new aggregation strategy termed Layer-wise Divergence Aware Weight Aggregation (L-DAWA) to mitigate the influence of client bias and divergence during FL aggregation. The proposed method aggregates weights at the layer-level according to the measure of angular divergence between the clients' model and the global model. Extensive experiments with cross-silo and cross-device settings on CIFAR-10/100 and Tiny ImageNet datasets demonstrate that our methods are effective and obtain new SOTA performance on both contrastive and non-contrastive SSL approaches.
abstractTranslation: 摄像头设备的普及导致边缘产生大量未标记的图像数据。将自监督学习 (SSL) 和联邦学习 (FL) 集成到一个连贯的系统中可以潜在地提供数据隐私保证，同时还可以提高所学习的视觉表示的质量和鲁棒性，而无需移动数据。然而，数据异质性导致的 FL 聚合过程中的客户端偏差和分歧限制了下游任务中学习到的视觉表示的性能。在本文中，我们提出了一种新的聚合策略，称为分层发散感知权重聚合（L-DAWA），以减轻 FL 聚合期间客户端偏差和发散的影响。所提出的方法根据客户端模型和全局模型之间的角度散度的度量在层级别聚合权重。在 CIFAR-10/100 和 Tiny ImageNet 数据集上进行的跨孤岛和跨设备设置的广泛实验表明，我们的方法是有效的，并且在对比和非对比 SSL 方法上都获得了新的 SOTA 性能。
PGFed: Personalize Each Client's Global Objective for Federated LearningAuthors: Jun Luo; Matias Mendieta; Chen Chen; Shandong Wu
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Personalized federated learning has received an upsurge of attention due to the mediocre performance of conventional federated learning (FL) over heterogeneous data. Unlike conventional FL which trains a single global consensus model, personalized FL allows different models for different clients. However, existing personalized FL algorithms only implicitly transfer the collaborative knowledge across the federation by embedding the knowledge into the aggregated model or regularization. We observed that this implicit knowledge transfer fails to maximize the potential of each client's empirical risk toward other clients. Based on our observation, in this work, we propose Personalized Global Federated Learning (PGFed), a novel personalized FL framework that enables each client to personalize its own global objective by explicitly and adaptively aggregating the empirical risks of itself and other clients. To avoid massive (O(N^2)) communication overhead and potential privacy leakage while achieving this, each client's risk is estimated through a first-order approximation for other clients' adaptive risk aggregation. On top of PGFed, we develop a momentum upgrade, dubbed PGFedMo, to more efficiently utilize clients' empirical risks. Our extensive experiments on four datasets under different federated settings show consistent improvements of PGFed over previous state-of-the-art methods. The code is publicly available at (github.com/ljaiverson/...).
abstractTranslation: 由于传统联邦学习（FL）在异构数据上的表现平平，个性化联邦学习受到了广泛的关注。与训练单一全局共识模型的传统 FL 不同，个性化 FL 允许为不同的客户使用不同的模型。然而，现有的个性化 FL 算法仅通过将知识嵌入到聚合模型或正则化中来隐式地跨联邦传输协作知识。我们观察到，这种隐性知识转移未能最大限度地发挥每个客户对其他客户的经验风险的潜力。根据我们的观察，在这项工作中，我们提出了个性化全球联邦学习（PGFed），这是一种新颖的个性化 FL 框架，使每个客户能够通过明确且自适应地聚合自身和其他客户的经验风险来个性化自己的全球目标。为了在实现这一目标的同时避免大量 (O(N^2)) 通信开销和潜在的隐私泄露，每个客户端的风险是通过其他客户端自适应风险聚合的一阶近似来估计的。在 PGFed 之上，我们开发了动力升级，称为 PGFedMo，以更有效地利用客户的经验风险。我们在不同联邦设置下对四个数据集进行的广泛实验表明，PGFed 相对于以前最先进的方法具有一致的改进。该代码可在 (github.com/ljaiverson/...) 上公开获取。
No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier
Authors: Zexi Li; Xinyi Shang; Rui He; Tao Lin; Chao Wu
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Data heterogeneity is an inherent challenge that hinders the performance of federated learning (FL). Recent studies have identified the biased classifiers of local models as the key bottleneck. Previous attempts have used classifier calibration after FL training, but this approach falls short in improving the poor feature representations caused by training-time classifier biases. Resolving the classifier bias dilemma in FL requires a full understanding of the mechanisms behind the classifier. Recent advances in neural collapse have shown that the classifiers and feature prototypes under perfect training scenarios collapse into an optimal structure called simplex equiangular tight frame (ETF). Building on this neural collapse insight, we propose a solution to the FL's classifier bias problem by utilizing a synthetic and fixed ETF classifier during training. The optimal classifier structure enables all clients to learn unified and optimal feature representations even under extremely heterogeneous data. We devise several effective modules to better adapt the ETF structure in FL, achieving both high generalization and personalization. Extensive experiments demonstrate that our method achieves state-of-the-art performances on CIFAR-10, CIFAR-100, and Tiny-ImageNet. The code is available at (github.com/ZexiLee/ICC...).
abstractTranslation: 数据异构性是阻碍联邦学习（FL）性能的固有挑战。最近的研究发现局部模型的有偏差的分类器是关键瓶颈。以前的尝试在 FL 训练后使用分类器校准，但这种方法在改善由训练时分类器偏差引起的不良特征表示方面存在不足。解决 FL 中的分类器偏差困境需要充分理解分类器背后的机制。神经崩溃方面的最新进展表明，完美训练场景下的分类器和特征原型会崩溃为称为单纯形等角紧框架（ETF）的最佳结构。基于这种神经崩溃的见解，我们提出了一种解决 FL 分类器偏差问题的方法，即在训练期间利用合成的固定 ETF 分类器。最优的分类器结构使所有客户端即使在极其异构的数据下也能学习统一且最优的特征表示。我们设计了几个有效的模块来更好地适应 FL 中的 ETF 结构，实现高度通用化和个性化。大量实验表明，我们的方法在 CIFAR-10、CIFAR-100 和 Tiny-ImageNet 上实现了最先进的性能。代码可在 (github.com/ZexiLee/ICC...) 获取。
Notes:
ProtoFL: Unsupervised Federated Learning via Prototypical DistillationAuthors: Hansol Kim; Youngjun Kwak; Minyoung Jung; Jinho Shin; Youngsung Kim; Changick Kim
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) is a promising approach for enhancing data privacy preservation, particularly for authentication systems. However, limited round communications, scarce representation, and scalability pose significant challenges to its deployment, hindering its full potential. In this paper, we propose 'ProtoFL', Prototypical Representation Distillation based unsupervised Federated Learning to enhance the representation power of a global model and reduce round communication costs. Additionally, we introduce a local one-class classifier based on normalizing flows to improve performance with limited data. Our study represents the first investigation of using FL to improve one-class classification performance. We conduct extensive experiments on five widely used benchmarks, namely MNIST, CIFAR-10, CIFAR-100, ImageNet-30, and Keystroke-Dynamics, to demonstrate the superior performance of our proposed framework over previous methods in the literature.
abstractTranslation: 联邦学习（FL）是增强数据隐私保护的一种有前途的方法，特别是对于身份验证系统。然而，有限的回合通信、稀缺的代表性和可扩展性对其部署构成了重大挑战，阻碍了其充分发挥潜力。在本文中，我们提出了"ProtoFL"，即基于原型表示蒸馏的无监督联邦学习，以增强全局模型的表示能力并降低回合通信成本。此外，我们引入了基于标准化流的本地一类分类器，以提高有限数据的性能。我们的研究代表了首次使用 FL 来提高单类分类性能的研究。我们对五个广泛使用的基准（即 MNIST、CIFAR-10、CIFAR-100、ImageNet-30 和 Key Strike-Dynamics）进行了广泛的实验，以证明我们提出的框架比文献中以前的方法具有优越的性能。
Notes:
PDF
Multi-Metrics Adaptively Identifies Backdoors in Federated LearningAuthors: Siquan Huang; Yijiang Li; Chong Chen; Leyu Shi; Ying Gao
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: The decentralized and privacy-preserving nature of federated learning (FL) makes it vulnerable to backdoor attacks aiming to manipulate the behavior of the resulting model on specific adversary-chosen inputs. However, most existing defenses based on statistical differences take effect only against specific attacks, especially when the malicious gradients are similar to benign ones or the data are highly non-independent and identically distributed (non-IID). In this paper, we revisit the distance-based defense methods and discover that i) Euclidean distance becomes meaningless in high dimensions and ii) malicious gradients with diverse characteristics cannot be identified by a single metric. To this end, we present a simple yet effective defense strategy with multi-metrics and dynamic weighting to identify backdoors adaptively. Furthermore, our novel defense has no reliance on predefined assumptions over attack settings or data distributions and little impact on benign performance. To evaluate the effectiveness of our approach, we conduct comprehensive experiments on different datasets under various attack settings, where our method achieves the best defensive performance. For instance, we achieve the lowest backdoor accuracy of 3.06% under the difficult Edge-case PGD, showing significant superiority over previous defenses. The results also demonstrate that our method can be well-adapted to a wide range of non-IID degrees without sacrificing the benign performance.
abstractTranslation: 联邦学习（FL）的去中心化和隐私保护性质使其容易受到后门攻击，这些后门攻击旨在操纵最终模型在特定对手选择的输入上的行为。然而，大多数基于统计差异的现有防御仅对特定攻击有效，特别是当恶意梯度与良性梯度相似或数据高度非独立且同分布（非IID）时。在本文中，我们重新审视基于距离的防御方法，发现i）欧几里德距离在高维中变得毫无意义，ii）具有不同特征的恶意梯度无法通过单一指标来识别。为此，我们提出了一种简单而有效的防御策略，具有多指标和动态权重来自适应地识别后门。此外，我们的新颖防御不依赖于攻击设置或数据分布的预定义假设，并且对良性性能影响很小。为了评估我们的方法的有效性，我们在不同的攻击设置下对不同的数据集进行了全面的实验，其中我们的方法实现了最佳的防御性能。例如，我们在困难的 Edge-case PGD 下实现了 3.06% 的最低后门准确率，与之前的防御相比显示出显着的优越性。结果还表明，我们的方法可以很好地适应各种非独立同分布程度，而不会牺牲良性性能。
Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to BeatAuthors: Erdong Hu; Yuxin Tang; Anastasios Kyrillidis; Chris Jermaine
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: We carefully evaluate a number of algorithms for learning in a federated environment, and test their utility for a variety of image classification tasks. We consider many issues that have not been adequately considered before: whether learning over data sets that do not have diverse sets of images affects the results; whether to use a pre-trained feature extraction "backbone"; how to evaluate learner performance (we argue that classification accuracy is not enough), among others. Overall, across a wide variety of settings, we find that vertically decomposing a neural network seems to give the best results, and outperforms more standard reconciliation-used methods.
abstractTranslation: 我们仔细评估了多种在联邦环境中学习的算法，并测试它们对各种图像分类任务的实用性。我们考虑了许多以前没有充分考虑的问题：对没有不同图像集的数据集进行学习是否会影响结果；是否使用预先训练的特征提取"骨干"；如何评估学习者的表现（我们认为分类准确性还不够）等等。总的来说，在各种设置中，我们发现垂直分解神经网络似乎能给出最佳结果，并且优于更标准的协调方法。
Towards Attack-tolerant Federated Learning via Critical Parameter AnalysisAuthors: Sungwon Han; Sungwon Park; Fangzhao Wu; Sundong Kim; Bin Zhu; Xing Xie; Meeyoung ChaConference : Proceedings of the IEEE/CVF International Conference on Computer VisionUrl: (openaccess.thecvf.com/content/ICC...Abstract "https://openaccess.thecvf.com/content/ICCV2023/html/Han_Towards_Attack-tolerant_Federated_Learning_via_Critical_Parameter_Analysis_ICCV_2023_paper.html)Abstract"): Federated learning is used to train a shared model in a decentralized way without clients sharing private data with each other. Federated learning systems are susceptible to poisoning attacks when malicious clients send false updates to the central server. Existing defense strategies are ineffective under non-IID data settings. This paper proposes a new defense strategy, FedCPA (Federated learning with Critical Parameter Analysis). Our attack-tolerant aggregation method is based on the observation that benign local models have similar sets of top-k and bottom-k critical parameters, whereas poisoned local models do not. Experiments with different attack scenarios on multiple datasets demonstrate that our model outperforms existing defense strategies in defending against poisoning attacks.abstractTranslation: 联邦学习用于以去中心化的方式训练共享模型，而无需客户端彼此共享私有数据。当恶意客户端向中央服务器发送虚假更新时，联邦学习系统很容易受到中毒攻击。现有的防御策略在非独立同分布数据设置下无效。本文提出了一种新的防御策略，FedCPA（带有关键参数分析的联邦学习）。我们的耐攻击聚合方法基于这样的观察：良性局部模型具有相似的 top-k 和 Bottom-k 关键参数集，而中毒的局部模型则不然。在多个数据集上进行不同攻击场景的实验表明，我们的模型在防御中毒攻击方面优于现有的防御策略。Notes:
PDF CODESUPP
FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge DistillationAuthors: Jingwen Guo; Hong Liu; Shitong Sun; Tianyu Guo; Min Zhang; Chenyang Si
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Existing skeleton-based action recognition methods typically follow a centralized learning paradigm, which can pose privacy concerns when exposing human-related videos. Federated Learning (FL) has attracted much attention due to its outstanding advantages in privacy-preserving. However, directly applying FL approaches to skeleton videos suffers from unstable training. In this paper, we investigate and discover that the heterogeneous human topology graph structure is the crucial factor hindering training stability. To address this issue, we pioneer a novel Federated Skeleton-based Action Recognition (FSAR) paradigm, which enables the construction of a globally generalized model without accessing local sensitive data. Specifically, we introduce an Adaptive Topology Structure (ATS), separating generalization and personalization by learning a domain-invariant topology shared across clients and a domain-specific topology decoupled from global model aggregation. Furthermore, we explore Multi-grain Knowledge Distillation (MKD) to mitigate the discrepancy between clients and the server caused by distinct updating patterns through aligning shallow block-wise motion features. Extensive experiments on multiple datasets demonstrate that FSAR outperforms state-of-the-art FL-based methods while inherently protecting privacy for skeleton-based action recognition.
abstractTranslation: 现有的基于骨架的动作识别方法通常遵循集中式学习范式，这在暴露与人类相关的视频时可能会带来隐私问题。联邦学习（FL）因其在隐私保护方面的突出优势而备受关注。然而，直接将 FL 方法应用于骨架视频会遇到训练不稳定的问题。在本文中，我们研究并发现异构的人体拓扑图结构是阻碍训练稳定性的关键因素。为了解决这个问题，我们开创了一种新颖的基于联邦骨架的动作识别（FSAR）范例，该范例能够在不访问本地敏感数据的情况下构建全局通用模型。具体来说，我们引入了自适应拓扑结构（ATS），通过学习跨客户端共享的域不变拓扑和与全局模型聚合解耦的特定于域的拓扑来分离泛化和个性化。此外，我们探索多粒度知识蒸馏（MKD），通过对齐浅块运动特征来减轻由不同更新模式引起的客户端和服务器之间的差异。对多个数据集的大量实验表明，FSAR 优于最先进的基于 FL 的方法，同时本质上保护了基于骨架的动作识别的隐私。
zPROBE: Zero Peek Robustness Checks for Federated LearningAuthors: Zahra Ghodsi; Mojan Javaheripi; Nojan Sheybani; Xinqiao Zhang; Ke Huang; Farinaz Koushanfar
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Privacy-preserving federated learning allows multiple users to jointly train a model with coordination of a central server. The server only learns the final aggregation result, thereby preventing leakage of the users' (private) training data from the individual model updates. However, keeping the individual updates private allows malicious users to degrade the model accuracy without being detected, also known as Byzantine attacks. Best existing defenses against Byzantine workers rely on robust rank-based statistics, e.g., setting robust bounds via the median of updates, to find malicious updates. However, implementing privacy-preserving rank-based statistics, especially median-based, is nontrivial and unscalable in the secure domain, as it requires sorting of all individual updates. We establish the first private robustness check that uses high break point rank-based statistics on aggregated model updates. By exploiting randomized clustering, we significantly improve the scalability of our defense without compromising privacy. We leverage the derived statistical bounds in zero-knowledge proofs to detect and remove malicious updates without revealing the private user updates. Our novel framework, zPROBE, enables Byzantine resilient and secure federated learning. We show the effectiveness of zPROBE on several computer vision benchmarks. Empirical evaluations demonstrate that zPROBE provides a low overhead solution to defend against state-of-the-art Byzantine attacks while preserving privacy.
abstractTranslation: 保护隐私的联邦学习允许多个用户在中央服务器的协调下共同训练模型。服务器仅了解最终的聚合结果，从而防止用户（私人）训练数据从各个模型更新中泄漏。然而，保持单个更新的私密性会导致恶意用户在不被发现的情况下降低模型的准确性，也称为拜占庭攻击。针对拜占庭工人的现有最佳防御依赖于强大的基于排名的统计数据，例如，通过更新的中位数设置强大的界限，以发现恶意更新。然而，在安全领域中，实现基于隐私保护的基于排名的统计，尤其是基于中值的统计，是非常重要且不可扩展的，因为它需要对所有单独的更新进行排序。我们建立了第一个私有稳健性检查，该检查对聚合模型更新使用基于高断点排名的统计数据。通过利用随机聚类，我们在不损害隐私的情况下显着提高了防御的可扩展性。我们利用零知识证明中派生的统计界限来检测和删除恶意更新，而不会泄露私人用户更新。我们的新颖框架 zPROBE 可实现拜占庭弹性且安全的联邦学习。我们在多个计算机视觉基准测试中展示了 zPROBE 的有效性。实证评估表明，zPROBE 提供了一种低开销的解决方案，可以防御最先进的拜占庭攻击，同时保护隐私。
Towards Instance-adaptive Inference for Federated LearningAuthors: Chun-Mei Feng; Kai Yu; Nian Liu; Xinxing Xu; Salman Khan; Wangmeng Zuo
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. However, the performance of the global model is often hampered by non-i.i.d. distribution among the clients, requiring extensive efforts to mitigate inter-client data heterogeneity. Going beyond inter-client data heterogeneity, we note that intra-client heterogeneity can also be observed on complex real-world data and seriously deteriorate FL performance. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Instead of huge instance-adaptive models, we resort to a parameter-efficient fine-tuning method, i.e., scale and shift deep features (SSF), upon a pre-trained model. Specifically, we first train an SSF pool for each client, and aggregate these SSF pools on the server side, thus still maintaining a low communication cost. To enable instance-adaptive inference, for a given instance, we dynamically find the best-matched SSF subsets from the pool and aggregate them to generate an adaptive SSF specified for the instance, thereby reducing the intra-client as well as the inter-client heterogeneity. Extensive experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
abstractTranslation: 联邦学习 (FL) 是一种分布式学习范例，使多个客户端能够通过聚合本地训练来学习强大的全局模型。然而，全局模型的性能常常受到非独立同分布的阻碍。客户端之间的分布，需要付出大量努力来减轻客户端间的数据异构性。除了客户端间数据异构性之外，我们注意到在复杂的现实世界数据上也可以观察到客户端内的异构性，并严重恶化 FL 性能。在本文中，我们提出了一种新颖的 FL 算法，即 FedIns，通过在 FL 框架中启用实例自适应推理来处理客户端内数据异构性。我们没有使用巨大的实例自适应模型，而是采用参数有效的微调方法，即在预训练模型上缩放和移动深度特征（SSF）。具体来说，我们首先为每个客户端训练一个SSF池，并在服务器端聚合这些SSF池，从而仍然保持较低的通信成本。为了实现实例自适应推理，对于给定的实例，我们从池中动态查找最匹配的 SSF 子集，并将它们聚合以生成为该实例指定的自适应 SSF，从而减少客户端内以及客户端间的干扰异质性。大量的实验表明，我们的 FedIn 优于最先进的 FL 算法，例如，与最好的方法相比，在 Tiny-ImageNet 上的通信成本低于 15%，提高了 6.64%。
Robust Heterogeneous Federated Learning under Data CorruptionAuthors: Xiuwen Fang; Mang Ye; Xiyuan Yang
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Model heterogeneous federated learning is a realistic and challenging problem. However, due to the limitations of data collection, storage, and transmission conditions, as well as the existence of free-rider participants, the clients may suffer from data corruption. This paper starts the first attempt to investigate the problem of data corruption in the model heterogeneous federated learning framework. We design a novel method named Augmented Heterogeneous Federated Learning (AugHFL), which consists of two stages: 1) In the local update stage, a corruption-robust data augmentation strategy is adopted to minimize the adverse effects of local corruption while enabling the models to learn rich local knowledge. 2) In the collaborative update stage, we design a robust re-weighted communication approach, which implements communication between heterogeneous models while mitigating corrupted knowledge transfer from others. Extensive experiments demonstrate the effectiveness of our method in coping with various corruption patterns in the model heterogeneous federated learning setting.
abstractTranslation: 建模异构联邦学习是一个现实且具有挑战性的问题。然而，由于数据采集、存储和传输条件的限制，以及搭便车参与者的存在，客户端可能会遭受数据损坏。本文首次尝试研究模型异构联邦学习框架中的数据损坏问题。我们设计了一种名为增强异构联邦学习（AugHFL）的新颖方法，该方法由两个阶段组成：1）在本地更新阶段，采用抗损坏数据增强策略来最小化本地损坏的不利影响，同时使模型能够学习丰富的当地知识。 2）在协作更新阶段，我们设计了一种鲁棒的重新加权通信方法，该方法实现异构模型之间的通信，同时减少来自其他模型的损坏的知识传输。大量的实验证明了我们的方法在应对模型异构联邦学习设置中的各种腐败模式方面的有效性。
Reducing Training Time in Cross-Silo Federated Learning Using Multigraph Topology
Authors: Tuong Do; Binh X. Nguyen; Vuong Pham; Toan Tran; Erman Tjiputra; Quang D. Tran; Anh Nguyen
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning is an active research topic since it enables several participants to jointly train a model without sharing local data. Currently, cross-silo federated learning is a popular training setting that utilizes a few hundred reliable data silos with high-speed access links to training a model. While this approach has been widely applied in real-world scenarios, designing a robust topology to reduce the training time remains an open problem. In this paper, we present a new multigraph topology for cross-silo federated learning. We first construct the multigraph using the overlay graph. We then parse this multigraph into different simple graphs with isolated nodes. The existence of isolated nodes allows us to perform model aggregation without waiting for other nodes, hence effectively reducing the training time. Intensive experiments on three public datasets show that our proposed method significantly reduces the training time compared with recent state-of-the-art topologies while maintaining the accuracy of the learned model.
abstractTranslation: 联邦学习是一个活跃的研究课题，因为它使多个参与者能够在不共享本地数据的情况下联邦训练模型。目前，跨孤岛联邦学习是一种流行的训练设置，它利用数百个具有高速访问链接的可靠数据孤岛来训练模型。虽然这种方法已广泛应用于现实场景中，但设计稳健的拓扑以减少训练时间仍然是一个悬而未决的问题。在本文中，我们提出了一种用于跨孤岛联邦学习的新多图拓扑。我们首先使用叠加图构建多重图。然后，我们将这个多重图解析为具有孤立节点的不同简单图。孤立节点的存在使得我们无需等待其他节点即可进行模型聚合，从而有效减少了训练时间。对三个公共数据集的密集实验表明，与最近最先进的拓扑相比，我们提出的方法显着减少了训练时间，同时保持了学习模型的准确性。
Notes:
Local or Global: Selective Knowledge Assimilation for Federated Learning with Limited LabelsAuthors: Yae Jee Cho; Gauri Joshi; Dimitrios Dimitriadis
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Many existing FL methods assume clients with fully-labeled data, while in realistic settings, clients have limited labels due to the expensive and laborious process of labeling. Limited labeled local data of the clients often leads to their local model having poor generalization abilities to their larger unlabeled local data, such as having class-distribution mismatch with the unlabeled data. As a result, clients may instead look to benefit from the global model trained across clients to leverage their unlabeled data, but this also becomes difficult due to data heterogeneity across clients. In our work, we propose FedLabel where clients selectively choose the local or global model to pseudo-label their unlabeled data depending on which is more of an expert of the data. We further utilize both the local and global models' knowledge via global-local consistency regularization which minimizes the divergence between the two models' outputs when they have identical pseudo-labels for the unlabeled data. Unlike other semi-supervised FL baselines, our method does not require additional experts other than the local or global model, nor require additional parameters to be communicated. We also do not assume any server-labeled data or fully labeled clients. For both cross-device and cross-silo settings, we show that FedLabel outperforms other semi-supervised FL baselines by 8-24%, and even outperforms standard fully supervised FL baselines (100% labeled data) with only 5-20% of labeled data.
abstractTranslation: 许多现有的 FL 方法假设客户拥有完全标记的数据，而在现实环境中，由于标记过程昂贵且费力，客户的标签有限。客户端有限的标记本地数据通常会导致其本地模型对其较大的未标记本地数据的泛化能力较差，例如与未标记数据的类分布不匹配。因此，客户可能会寻求从跨客户训练的全局模型中受益，以利用其未标记的数据，但由于客户之间的数据异构性，这也变得很困难。在我们的工作中，我们提出了 FedLabel，客户可以选择性地选择本地或全局模型来伪标记其未标记的数据，具体取决于哪个模型更擅长数据。我们通过全局-局部一致性正则化进一步利用局部和全局模型的知识，当两个模型对于未标记数据具有相同的伪标签时，可以最小化两个模型输出之间的差异。与其他半监督 FL 基线不同，我们的方法除了本地或全局模型之外不需要额外的专家，也不需要传达额外的参数。我们也不假设任何服务器标记的数据或完全标记的客户端。对于跨设备和跨筒仓设置，我们表明 FedLabel 的性能优于其他半监督 FL 基线 8-24%，甚至优于标准完全监督 FL 基线（100% 标记数据），仅 5-20% 的标记数据数据。
Workie-Talkie: Accelerating Federated Learning by Overlapping Computing and Communications via Contrastive Regularization
Authors: Rui Chen; Qiyu Wan; Pavana Prakash; Lan Zhang; Xu Yuan; Yanmin Gong; Xin Fu; Miao Pan
Conference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...)
Abstract: Federated learning (FL) over mobile devices is a promising distributed learning paradigm for various mobile applications. However, practical deployment of FL over mobile devices is very challenging because (i) conventional FL incurs huge training latency for mobile devices due to interleaved local computing and communications of model updates, (ii) there are heterogeneous training data across mobile devices, and (iii) mobile devices have hardware heterogeneity in terms of computing and communication capabilities. To address aforementioned challenges, in this paper, we propose a novel "workie-talkie" FL scheme, which can accelerate FL's training by overlapping local computing and wireless communications via contrastive regularization (FedCR). FedCR can reduce FL's training latency and almost eliminate straggler issues since it buries/embeds the time consumption of communications into that of local training. To resolve the issue of model staleness and data heterogeneity co-existing, we introduce class-wise contrastive regularization to correct the local training in FedCR. Besides, we jointly exploit contrastive regularization and subnetworks to further extend our FedCR approach to accommodate edge devices with hardware heterogeneity. We deploy FedCR in our FL testbed and conduct extensive experiments. The results show that FedCR outperforms its status quo FL approaches on various datasets and models.
abstractTranslation: 移动设备上的联邦学习（FL）是一种有前景的分布式学习范式，适用于各种移动应用程序。然而，在移动设备上实际部署 FL 非常具有挑战性，因为（i）由于交错的本地计算和模型更新通信，传统的 FL 会给移动设备带来巨大的训练延迟，（ii）跨移动设备存在异构训练数据，并且(ii) 移动设备在计算和通信能力方面具有硬件异构性。为了解决上述挑战，在本文中，我们提出了一种新颖的"workie-talkie"FL方案，该方案可以通过对比正则化（FedCR）重叠本地计算和无线通信来加速FL的训练。 FedCR 可以减少 FL 的训练延迟并几乎消除掉队问题，因为它将通信的时间消耗埋入/嵌入到本地训练的时间消耗中。为了解决模型陈旧和数据异构性共存的问题，我们引入了类对比正则化来纠正 FedCR 中的本地训练。此外，我们共同利用对比正则化和子网络来进一步扩展我们的 FedCR 方法，以适应具有硬件异构性的边缘设备。我们在 FL 测试台中部署 FedCR 并进行广泛的实验。结果表明，FedCR 在各种数据集和模型上均优于现有的 FL 方法。
Notes:
SUPP
FRAug: Tackling Federated Learning with Non-IID Features via Representation AugmentationAuthors: Haokun Chen; Ahmed Frikha; Denis Krompass; Jindong Gu; Volker TrespConference : Proceedings of the IEEE/CVF International Conference on Computer VisionUrl: (openaccess.thecvf.com/content/ICC...Abstract "https://openaccess.thecvf.com/content/ICCV2023/html/Chen_FRAug_Tackling_Federated_Learning_with_Non-IID_Features_via_Representation_Augmentation_ICCV_2023_paper.html)Abstract"): Federated Learning (FL) is a decentralized machine learning paradigm, in which multiple clients collaboratively train neural networks without centralizing their local data, and hence preserve data privacy. However, real-world FL applications usually encounter challenges arising from distribution shifts across the local datasets of individual clients. These shifts may drift the global model aggregation or result in convergence to deflected local optimum. While existing efforts have addressed distribution shifts in the label space, an equally important challenge remains relatively unexplored. This challenge involves situations where the local data of different clients indicate identical label distributions but exhibit divergent feature distributions. This issue can significantly impact the global model performance in the FL framework. In this work, we propose Federated Representation Augmentation (FRAug) to resolve this practical and challenging problem. FRAug optimizes a shared embedding generator to capture client consensus. Its output synthetic embeddings are transformed into client-specific by a locally optimized RTNet to augment the training space of each client. Our empirical evaluation on three public benchmarks and a real-world medical dataset demonstrates the effectiveness of the proposed method, which substantially outperforms the current state-of-the-art FL methods for feature distribution shifts, including PartialFed and FedBN.abstractTranslation: 联邦学习（FL）是一种去中心化的机器学习范例，其中多个客户端协作训练神经网络，而不集中其本地数据，从而保护数据隐私。然而，现实世界的 FL 应用程序通常会遇到因各个客户端本地数据集的分布变化而产生的挑战。这些变化可能会导致全局模型聚合发生漂移，或者导致收敛到偏离局部最优值。虽然现有的努力已经解决了标签空间的分布变化，但同样重要的挑战仍然相对未得到探索。这一挑战涉及不同客户端的本地数据指示相同的标签分布但表现出不同的特征分布的情况。此问题可能会严重影响 FL 框架中的全局模型性能。在这项工作中，我们提出联邦表示增强（FRAug）来解决这个实际且具有挑战性的问题。 FRAug 优化共享嵌入生成器以获取客户共识。其输出合成嵌入通过本地优化的 RTNet 转换为特定于客户端的嵌入，以扩大每个客户端的训练空间。我们对三个公共基准和真实世界医学数据集的实证评估证明了所提出方法的有效性，该方法在特征分布变化方面大大优于当前最先进的 FL 方法，包括 PartialFed 和 FedBN。Notes:PDFSUPP
Knowledge-Aware Federated Active Learning with Non-IID DataAuthors: Yu-Tong Cao; Ye Shi; Baosheng Yu; Jingya Wang; Dacheng TaoConference : Proceedings of the IEEE/CVF International Conference on Computer Vision
Url: (openaccess.thecvf.com/content/ICC...Abstract "https://openaccess.thecvf.com/content/ICCV2023/html/Cao_Knowledge-Aware_Federated_Active_Learning_with_Non-IID_Data_ICCV_2023_paper.html)Abstract"): Federated learning enables multiple decentralized clients to learn collaboratively without sharing local data. However, the expensive annotation cost on local clients remains an obstacle in utilizing local data. In this paper, we propose a federated active learning paradigm to efficiently learn a global model with a limited annotation budget while protecting data privacy in a decentralized learning manner. The main challenge faced by federated active learning is the mismatch between the active sampling goal of the global model on the server and that of the asynchronous local clients. This becomes even more significant when data is distributed non-IID across local clients. To address the aforementioned challenge, we propose Knowledge-Aware Federated Active Learning (KAFAL), which consists of Knowledge-Specialized Active Sampling (KSAS) and Knowledge-Compensatory Federated Update (KCFU). Specifically, KSAS is a novel active sampling method tailored for the federated active learning problem, aiming to deal with the mismatch challenge by sampling actively based on the discrepancies between local and global models. KSAS intensifies specialized knowledge in local clients, ensuring the sampled data is informative for both the local clients and the global model. Meanwhile, KCFU deals with the client heterogeneity caused by limited data and non-IID data distributions by compensating for each client's ability in weak classes with the assistance of the global model. Extensive experiments and analyses are conducted to show the superiority of KAFAL over recent state-of-the-art active learning methods. Code is available at (github.com/ycao5602/KA....abstractTranslation "https://github.com/ycao5602/KAFAL).abstractTranslation"): 联邦学习使多个分散的客户端能够协作学习，而无需共享本地数据。然而，本地客户端昂贵的注释成本仍然是利用本地数据的障碍。在本文中，我们提出了一种联邦主动学习范式，以有限的注释预算有效地学习全局模型，同时以去中心化学习的方式保护数据隐私。联邦主动学习面临的主要挑战是服务器上全局模型的主动采样目标与异步本地客户端的主动采样目标之间的不匹配。当数据以非 IID 方式分布在本地客户端时，这一点变得更加重要。为了应对上述挑战，我们提出了知识感知联邦主动学习（KAFAL），它由知识专用主动采样（KSAS）和知识补偿联邦更新（KCFU）组成。具体来说，KSAS是一种针对联邦主动学习问题量身定制的新型主动采样方法，旨在通过基于局部模型和全局模型之间的差异主动采样来应对不匹配的挑战。 KSAS 强化了本地客户的专业知识，确保采样数据为本地客户和全球模型提供信息。同时，KCFU通过在全局模型的帮助下补偿每个客户在弱类中的能力来处理有限数据和非独立同分布数据分布引起的客户异质性。大量的实验和分析表明 KAFAL 相对于最近最先进的主动学习方法的优越性。代码可在(github.com/ycao5602/KA...) 获取。
作者: 白小鱼（上海交通大学计算机系博士生）