We consider federated learning (FL) with multiple wireless edge servers having their own local coverage. We focus on speeding up training in this increasingly practical setup. Our key idea is to utilize the clients located in the overlapping coverage areas among adjacent edge servers (ESs); in the model-downloading stage, the clients in the overlapping areas receive multiple models from different ESs, take the average of the received models, and then update the averaged model with their local data. These clients send their updated model to multiple ESs by broadcasting, which acts as bridges for sharing the trained models between servers. Even when some ESs are given biased datasets within their coverage regions, their training processes can be assisted by adjacent servers through the clients in their overlapping regions. As a result, the proposed scheme does not require costly communications with the central cloud server (located at the higher tier of edge servers) for model synchronization, significantly reducing the overall training time compared to the conventional cloud-based FL systems. Extensive experimental results show remarkable performance gains of our scheme compared to existing methods. Our design targets latency-sensitive applications where edge-based FL is essential, e.g., when a number of connected cars/drones must cooperate (via FL) to quickly adapt to dynamically changing environments.
Federated learning (FL) operates based on model exchanges between the server and the clients, and suffers from significant communication as well as client-side computation burden. While emerging split learning (SL) solutions can reduce the clientside computation burden by splitting the model architecture, SL-based ideas still require significant time delay and communication burden for transmitting the forward activations and backward gradients at every global round. In this paper, we propose a new direction to FL/SL based on updating the client/server-side models in parallel, via local-loss-based training specifically geared to split learning. The parallel training of split models substantially shortens latency while obviating server-to-clients communication. We provide latency analysis that leads to optimal model cut as well as general guidelines for splitting the model. We also provide a theoretical analysis for guaranteeing convergence of our method. Extensive experimental results indicate that our scheme has significant communication and latency advantages over existing FL and SL ideas.
Abstract: Federated learning (FL) is an actively studied training protocol for distributed artificial intelligence (AI). One of the challenges for the implementation is a communication bottleneck in the uplink communication from devices to FL server. To address the issue, many researches have been studied on the improvement of communication efficiency. In particular, analog transmission for the wireless implementation provides a new opportunity allowing whole bandwidth to be fully reused at each device. However, it is still necessary to compress the parameters to the allocated communication bandwidth despite the communsication efficiency in analog FL. In this paper, we introduce the count-sketch (CS) algorithm as a compression scheme in analog FL to overcome the limited channel resources. We develop a more communication-efficient FL system by applying CS algorithm to the wireless implementation of FL. Numerical experiments show that the proposed scheme outperforms other bench mark schemes, CA-DSGD and state-of-theart digital schemes. Furthermore, we have observed that the proposed scheme is considerably robust against transmission power and channel resources.
Abstract: Variational particle-based Bayesian learning methods have the advantage of not being limited by the bias affecting more conventional parametric techniques. This paper proposes to leverage the flexibility of non-parametric Bayesian approximate inference to develop a novel Bayesian federated unlearning method, referred to as Forget-Stein Variational Gradient Descent (Forget-SVGD). Forget-SVGD builds on SVGD – a particle-based approximate Bayesian inference scheme using gradient-based deterministic updates – and on its distributed (federated) extension known as Distributed SVGD (DSVGD). Upon the completion of federated learning, as one or more participating agents request for their data to be “forgotten”, Forget-SVGD carries out local SVGD updates at the agents whose data need to be “unlearned”, which are interleaved with communication rounds with a parameter server. The proposed method is validated via performance comparisons with non-parametric schemes that train from scratch by excluding data to be forgotten, as well as with existing parametric Bayesian unlearning methods.
Abstract: Federated Bayesian learning offers a principled framework for the definition of collaborative training algorithms that are able to quantify epistemic uncertainty and to produce trustworthy decisions. Upon the completion of collaborative training, an agent may decide to exercise her legal “right to be forgotten”, which calls for her contribution to the jointly trained model to be deleted and discarded. This paper studies federated learning and unlearning in a decentralized network within a Bayesian framework. It specifically develops federated variational inference (VI) solutions based on the decentralized solution of local free energy minimization problems within exponentialfamily models and on local gossip-driven communication. The proposed protocols are demonstrated to yield efficient unlearning mechanisms.
Abstract: Deep learning (DL)-based channel estimations for multiple-input-multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) systems have shown remarkable
performance at the cost of huge sample size and complexity. While such complexity can be offloaded onto offline training phase for stationary channels, this becomes problematic when non-stationary channels are arisen. In this letter, we resolve this issue by proposing meta-learning-aided online training that only requires small sample size with reduced complexity. Numerical results under 3GPP channel models verify that proposed meta learning approach outperforms not only conventional DL-based estimators but also conventional model-based estimators, e.g., least squares and linear minimum mean square error estimators, especially in the small sample size/low complexity regime.
Abstract: This paper considers wireless communication system consisted of one transmitter, one legitimate receiver, and one eavesdropper. The transmitter transmits perturbation-added signal (i.e. adversarial example) with a certain modulation type, while the legitimate receiver and the eavesdropper adopt deep neural networks (DNN)-based classifier to recognize the modulation type of the received signal. Compared to the fact that the general goal of adversarial examples being a misclassification of all available classifiers, our objective is to design an adversarial example that lets the legitimate receiver classify accurately while the eavesdropper misclassifies. To this end, we propose two design approaches of the adversarial examples: (i) A priori; (ii) A posteriori, i.e. before and after learning steps of the receiver, respectively. Numerical results show that both approaches are effective for securing the communication link.
Authors : Sangseok Yun, Jae-Mo Kang, Jeongseok Ha, Sangho Lee, Dong-Woo Ryu, Jihoe Kwon and Il-Min Kim
Journal : IEEE Transactions on Geoscience and Remote Sensing Letters (published: March 2021)
본 논문은 딥 러닝 기반의 효율적인 지상 진동 모니터링 시스템에 관한 연구이다. 본 연구에서는 데이터 수집 및 분류의 실제 문제를 효과적으로 처리하기 위해 인공 지능 (AI) 기술을 채택하였다. 특히, 심층 Q-네트워크 기반 강화 학습을 채택하여 새로운 에너지 효율적인 데이터 수집 체계를 개발하였다. 또한 지상 진동 분류를 위한 향상된 공동 순환 신경망 (RNN) 및 합성 곱 신경망 (CNN) 접근 방식을 제안하였다. 제안한 방식의 성능은 실제 지반 진동 데이터를 사용하여 평가된다. 실험 결과 제안된 분류 체계는 분류 정확도 측면에서 기존의 CNN 방식보다 13 % 이상 우수한 성능을 보였다. 또한 제안한 에너지 관리 계획은 동일한 전력 할당을 사용하는 유사한 방식보다 지반 진동 모니터링 시스템의 정확도를 7.6 % 향상시킬 수 있음을 보여준다.
Fig. 1 Structure of the proposed ground vibration classification scheme.
Fig. 2. Performance of the proposed deep reinforcement learning-based energy management scheme.
Fig. 3. Performances of the proposed ground vibration classification scheme and the end-to-end ground vibration monitoring system.
Author: Ji Hyung Jung, Hye Won Chung, Ji Oon Lee
Conference and Year: ICML 2021
Keywords: Signal detection, Spiked Rectangular Model
데이터사이언스 분야에서 가장 중요한 문제 중 하나는 주어진 데이터로부터 신호를 감지, 복원하는 기법 개발이다. 본 논문에서는 신호 플러스 노이즈 유형의 데이터 모델에서 신호를 감지, 복원하는 문제를 연구한다. 이와 같은 모델에서 신호의 강도가 노이즈의 강도보다 상당히 강하면 신호를 안정적으로 감지, 복원 할 수 있지만, 신호 대 노이즈 비율이 특성 임계값 보다 낮을 경우 신호를 정확히 복원 할 수 없다. 본 논문에서는 우선 노이즈가 정규 분포를 따르지 않을 경우, 행렬의 각 엔트리를 사전 변환하는 기법을 통해 행렬의 주성분 분석법으로부터 기존보다 더 작은 신호 대 노이즈 비율에서도 신호를 복원 할 수 있음을 증명한다. 또한 신호를 완벽히 복원할 수 없는 영역에서도 신호 존재 여부를 가장 높은 정확도로 감지할 수 있는 이진 가설 테스트 알고리즘을 설계한다.
Figure1: We compare the reconstruction performance of the proposed PCA (top lines) and the standard PCA (bottom lines) for two FashionMNIST images, with the number of measurements N = [3136, 1568, 784, 588, 392] where the data dimension is M = 784. The left most column displays the original images for comparison.
Figure2: The histograms of the test statistic under null hypothesis H0 and alternative hypothesis H1, respectively, for the Gaussian noise with SNR ω = 0.35 and ω = 0.45. It can be shown that the difference of the means of the test statistic under H0 and H1 is larger for ω = 0.45.
Author: Doyeon Kim and Hye Won Chung
Conference and Year: IEEE International Symposium on Information Theory (ISIT), 2021.
우리는 d형 작업자–작업 전문화 모델에 따라 크라우스소싱에서 라벨링 된 문제를 고려한다. 여기서, 각 작업자와 작업은 유한한 유형 집합 중 하나의 특정 유형과 연관되어 있으며 작업자는 일치하지 않는 유형의 작업보다 일치된 유형의 작업에 대해 더 신뢰할 수 있는 답변을 제공한다. 우리는 작업자 군집화, 작업자 기술 추정 및 가중치 다수 투표를 이용하여 이진 작업 레이블 (주어진 복구 정확도까지)을 복구하는 추론 알고리즘을 제안한다. 제안된 추론 알고리즘은 작업자/작업 유형에 대한 어떠한 정보도 요구하지 않으며, 가장 잘 알려진 성능(작업당 최소 질문 수)으로 목표 복구 정확도를 달성한다.
