[보관함:] AI in Communication division

 Title: Communication in Multi-Agent Reinforcement Learning: Intention Sharing

Authors: Woojun Kim, Jongeui Park and Youngchul Sung

To be presented at International Conference on Learning Representation (ICLR) 2021

 

Communication is one of the core components for learning coordinated behavior in multi-agent systems. In this work, W. Kim et al. proposed a new communication scheme named Intention Sharing (IS) for multi-agent reinforcement learning in order to enhance the coordination among agents. In the proposed scheme, each agent generates an imagined trajectory by modeling the environment dynamics and other agents’ actions. The imagined trajectory is a simulated future trajectory of each agent based on the learned model of the environment dynamics and other agents and represents each agent’s future action plan. Each agent compresses this imagined trajectory capturing its future action plan to generate its intention message for communication by applying an attention mechanism to learn the relative importance of the components in the imagined trajectory based on the received message from other agents. Numeral results show that the proposed IS scheme significantly outperforms other communication schemes in multi-agent reinforcement learning.

 

Fig. 1. The overall structure of the proposed IS scheme from the perspective of Agent i

Fig. 2 Performance: : MADDPG (blue), DIAL (green), TarMAC (red), Comm-OA (purple), ATOC (cyan) and the proposed IS method (black).  (PP: Predator-and-Prey, CN: Cooperative Navigation, TJ: Traffic Junction)

Fig. 3. Imagined trajectories and attention weights of each agent on PP (N=3): 1st row – agent1 (red), 2nd row – agent2 (green), and 3rd row – agent3 (blue). Black squares, circle inside the times icon, and other circles denote the prey, current position, and estimated future positions, respectively. The brightness of the circle is proportional to the attention weight.

Title: Population-Guided Parallel Policy Search for Reinforcement Learning

Authors: Whiyoung Jung, Giseung Park and Youngchul Sung

Presented at International Conference on Learning Representation (ICLR) 2020

In this work, a new population-guided parallel learning scheme is proposed to enhance the performance of off-policy reinforcement learning (RL). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The key point is that the information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search region by the multiple learners. The guidance by the previous best policy and the enlarged range enable faster and better policy search. Monotone improvement of the expected cumulative return by the proposed scheme is proved theoretically. Working algorithms are constructed by applying the proposed scheme to the twin delayed deep deterministic (TD3) policy gradient algorithm. Numerical results show that the constructed algorithm outperforms most of the current state-of-the-art RL algorithms, and the gain is significant in the case of sparse reward environment.

 

Fig. 1. The overall proposed structure (P3S):

Fig. 2. The conceptual search coverage in the policy space by parallel learners:

Fig. 3. Performance of different parallel learning methods on MuJoCo environments (up), on delayed MuJoCo environments (down)

Fig. 4. Benefits of P3S (a) Performance and beta (1 seed) with d_min = 0.05, (b) Distance measures with d_min = 0.05, and (c) Comparison with different d_min = 0.02, 0.05.

Title: Machine-Learning-Based Read Reference Voltage Estimation for NAND Flash Memory Systems Without Knowledge of Retention Time

Authors: Hyemin Choe, Jeongju Jee, Seung-Chan Lim, Sung Min Joe, Il Han Park, and Hyuncheol Park

Journal: IEEE Access (published: September 2020)

To achieve a low error rate of NAND flash memory, reliable reference voltages should be updated based on the accurate knowledge of program/erase (P/E) cycles and retention time, because those severely distort the threshold voltage distribution of memory cell. Due to the sensitivity to the temperature, however, a flash memory controller is unable to acquire the exact knowledge of retention time, meaning that it is challenging to estimate accurate read reference voltages in practice.

In addition, it is difficult to characterize the relation between the channel impairments and the optimal read reference voltages in general. Therefore, we propose a machine-learning-based read reference voltage estimation framework for NAND flash memory without the knowledge of retention time.

In the off-line training phase, to define the input features of the proposed framework, we derive alternative information of unknown retention time, which are obtained by sensing and decoding the data in one wordline. For the on-line estimation phase, we propose three estimation schemes: 1) k-nearest neighbors (k-NN)- based, 2) nearest-centroid (NC)-based, and 3) polynomial regression (PR)-based estimations. By applying these estimation schemes, an unlabeled input feature is simply mapped into a pre-assigned class label, namely label read reference voltages, via the on-line estimation phase.

Based on the simulation and analysis, we have verified that the proposed framework can achieve high-reliable and low-latency performances in NAND flash memory systems without the knowledge of retention time.

Figure 1. Flow charts of read reference voltage estimation schemes. (a) k-NN based (b) NC-based (c) PR-based estimations.

Title: Downlink Extrapolation for FDD Multiple Antenna Systems Through Neural Network Using Extracted Uplink Path Gains

Authors: Hyuckjin Choi, Junil Choi

Journal: IEEE Access (published: April 2020)

 

In frequency division duplexing (FDD) communication system, base stations (BSs) should have the downlink (DL) channel state information (CSI) that cannot be obtained at BSs. The conventional FDD communication systems deploy the DL training and feedback where the mobile station (MS) estimates the DL CSI and delivers the CSI to the BS. It becomes infeasible as the number of antennas at the BS increases in a high mobility scenario. When a MS moves at high speed, the channel changes rapidly, which results in a short coherence time.

Without the uplink (UL) feedback, the BS might obtain the DL CSI, which is called the DL extrapolation. Even in the FDD communication system, the UL and DL channel have the reciprocity, which has been proved through previous related works. Using the relation between the UL and DL channel, the UL CSI can be mapped to the DL CSI through the neural network (NN). Prior studies have developed the DL extrapolation algorithm with full dimensional UL and DL channels. However, the complexity of NN training becomes severe as the channel dimension grows.

We proposed the algorithm to simplify the NN input and output for the DL extrapolation. It has been proved through many measurements that the UL and DL channels share same channel path delays and directions in FDD communication systems. The proposed method first extracts the common channel parameters from the UL and DL channel, then trains the NN with the frequency-dependent path gains such that the size of input and output of the NN decreases. The proposed technique outperforms the conventional NN-based DL extrapolation schemes through plenty of simulations.

Figure 1. Flow charts of (a) CH-learning and (b) PG-learning.

Figure 2. NN structures used for numerical studies. (a) MLP for the CH-learning and (b) CNN for the PG-learning.

Title: Massive MIMO Channel Prediction: Kalman Filtering Vs. Machine Learning

Authors: Hwanjin Kim, Sucheol Kim, Hyeongtaek Lee, Junil Choi.

Journal: IEEE Transactions on Communications (published: January 2021)

 

Accurate channel state information (CSI) at the base stations (BSs) is crucial to fully exploit massive multiple-input multiple-output (MIMO) systems. The CSI at the BS can be outdated in the time-varying channel due to the mobility of user equipment (UE). The best way to solve the outdated CSI problem is to predict channels based on the prior CSI. In this paper, we develop the vector Kalman filter (VKF)-based predictor and the machine learning (ML)-based predictor using the spatial channel model (SCM), which is the realistic channels adopted in the 3GPP standard. First, we develop the VKF-based predictor using the autoregressive (AR) parameters from the SCM data based on the Yule-Walker equations. Then, we develop the ML-based channel predictor exploiting the linear minimum mean-square error (LMMSE)-based noise pre-processed data. Numerical results show that both channel predictors have significant gain over the outdated channel with regard to the prediction accuracy and data rate.

Figure 1. Multi-layer perceptron (MLP) structure with LMMSE pre-processing.