[보관함:] AI in Wave division

Imitation Learning with Bayesian Exploration (IL-BE) for Signal Integrity (SI) of PAM-4 based High-speed Serial Link: PCIe 6.0 (DesignCon 2022)

 

Authors: Jihun Kim, Minsu Kim, Hyunwook Park, Jiwon Yoon, Seonguk Choi, Joonsang Park, Haeyeon Kim, Keeyoung Son, Seongguk Kim, Daehwan Lho, Keunwoo Kim, Jinwook Song, Kyungsuk Kim, Jongkyu Park and Joungho Kim

 

 

Abstract: This paper proposes a novel imitation learning with Bayesian exploration (IL-BE) method to optimize via parameters of any given channel parameters for signal integrity (SI) of PAM-4 based high-speed serial link on PCIe 6.0. PCIe 6.0. is a crucial interconnect link for highspeed communication of processors, and PAM-4 signaling is a major component of PCIe 6.0 that can double bandwidth. However, the design space of PAM-4 based PCIe 6.0 is extremely complex. Moreover, because PAM-4 signaling reduces eye-margin 1/3 compared with NRZ signaling, it is more sensitive to optimize. Bayesian optimization (BO) is a candidate method because it shows powerful searching abilities on black-box continuous optimization space. However, BO has a significant limitation to apply via optimization for any given channel parameters because BO needs massive iterations to solve each problem (i.e., no adaptation to new tasks). Deep reinforcement learning is a promising method that deep neural network (DNN) agents learn to capture meta-features among problems interacting with the real-world environment. Therefore, learned DNN agents can adapt to a new problem by optimization with small iterations. However, DNN agents must learn through massive iterative trial and error; it is extremely complex to train DNN. We blend the benefit of the BO and DRL method. Firstly, we collect high-quality expert data using BO rather than relying on poor exploration of the initial DNN agent. Then we use the collected high-quality data to train DNN agents by using an imitation learning scheme. For verification, we target one pair differential PCIe 6.0 (64Gbps) interconnection of SSD board and three-layer transition; a task is formulated to optimize via given channel parameters. The statistical simulation method is used to evaluate SI performances, including PAM-4 eye-diagram. Proposed IL-BE shows 100× faster training speed than conventional DRL method, 16× reduced iterations for the via parameter optimization than BO having state-of-the-art SI performances.

 

Title: PAM-4 based PCIe 6.0 Channel Design Optimization Method using Bayesian Optimization
(EPEPS 2021)

 

Authors: JihunKim, Hyunwook Park, Minsu Kim, Seonguk Choi, Keeyoung Son, Joonsang Park, Haeyeon Kim, Jinwook Song, Youngmin Ku, Jonggyu Park and Joungho Kim.

 

 

Abstract: This paper, for the first time, proposes a pulse amplitude modulation-4 (PAM-4) based peripheral component interconnect express (PCIe) 6.0 channel design optimization method using Bayesian Optimization (BO). The proposed method provides a sub-optimal channel design with PAM-4 signaling that maximizes target function considering signal integrity (SI). We formulate the target function of BO as a linear combination of the channel insertion loss (IL) and crosstalk (FEXT, NEXT) considering characteristics of PAM-4 signaling. To consider the trade-off between insertion loss and crosstalk in PAM-4 signaling, we obtain reasonable coefficients for formulating target function via ablation study. For verification, an eye diagram simulation with PAM-4 signaling is conducted.  We compare the channel performance of the proposed method and random search method (RS). Also, the proposed method is compared with the only IL- considered BO method to verify the impact of crosstalk in PAM-4 signaling. As a result, the channel optimized by the proposed method only obtains eye height and eye width of PAM-4 eye diagram, and the PAM-4 eyes of other comparison methods are closed.

 

Title:

 

Deep Reinforcement Learning-based Channel-flexible Equalization Scheme: An Application to High Bandwidth Memory (DesignCon2022)

 

Authors:

 

Seonguk Choi, Minsu Kim, Hyunwook Park, Haeyeon Rachel Kim, Joonsang Park, Jihun Kim, Keeyoung Son, Seongguk Kim, Keunwoo Kim, Daehwan Lho, Jiwon Yoon, Jinwook Song, Kyungsuk Kim, Jonggyu Park and Joungho Kim.

 

 

Abstract:

In this paper, we propose a channel-flexible hybrid equalizer (HYEQ) design methodology with re-usability based on deep reinforcement learning (DRL). Proposed method suggests the optimized HYEQ design for arbitrary channel dimension. HYEQ is comprised of a continuous time linear equalizer (CTLE) for high-frequency boosting and passive equalizer (PEQ) for low frequency attenuation, and our task is to co-optimize both of them. Our model plays a role as a solver to optimize the design of equalizers, while considering all signal integrity issues such as high frequency attenuation, crosstalk and so on.

 Our method utilizes recursive neural network commonly employed in natural language processing (NLP), in order to design HYEQ based on constructive DRL. Thus, each parameter of the equalizer is designed sequentially, reflecting other parameters. In this process, the design space of machine learning (ML) is determined by applying domain knowledge of equalizer, and thus even precise optimization is conducted. Furthermore, fast inference is conducted by trained neural network for any channel dimension. We validate that the proposed method outperforms conventional optimization algorithms such as random search (RS) and genetic algorithm (GA) in 3-coupled channel system of next generation high-bandwidth memory (HBM).

 

Title:

 

Sequential Policy Network-based Optimal Passive Equalizer Design for an Arbitrary Channel of High Bandwidth Memory using Advantage Actor Critic (EPEPS 2021)

 

Authors:

 

Seonguk Choi, Minsu Kim, Hyunwook Park, Keeyoung Son, Seongguk Kim, Jihun Kim, Joonsang Park, Haeyeon Kim, Taein Shin, Keunwoo Kim and Joungho Kim.

 

Abstract:

In this paper, we proposed a sequential policy network-based passive equalizer (PEQ) design method for an arbitrary channel of high bandwidth memory (HBM) using advantage actor critic (A2C) algorithm, considering signal integrity (SI) for the first time. PEQ design must consider the circuit parameters and placement for improving the performance. However, optimizing PEQ is complicated because various design parameters are coupled. Conventional optimization methods such as genetic algorithm (GA) repeat the optimization process for the changed conditions. In contrast, the proposed method suggests the improved solution based on the trained sequential policy network with flexibility for unseen conditions. For verification, we conducted electromagnetic (EM) simulation with optimized PEQs by GA, random search (RS) and the proposed method. Experimental results demonstrate that the proposed method outperformed the GA and RS by 4.4 \% and 6.4 \% respectively in terms of the eye-height.

 

Title:

 

Deep Reinforcement Learning-based Channel-flexible Equalization Scheme: An Application to High Bandwidth Memory (DesignCon2022)

 

Authors:

 

Seonguk Choi, Minsu Kim, Hyunwook Park, Haeyeon Rachel Kim, Joonsang Park, Jihun Kim, Keeyoung Son, Seongguk Kim, Keunwoo Kim, Daehwan Lho, Jiwon Yoon, Jinwook Song, Kyungsuk Kim, Jonggyu Park and Joungho Kim.

 

 

Abstract:

In this paper, we propose a channel-flexible hybrid equalizer (HYEQ) design methodology with re-usability based on deep reinforcement learning (DRL). Proposed method suggests the optimized HYEQ design for arbitrary channel dimension. HYEQ is comprised of a continuous time linear equalizer (CTLE) for high-frequency boosting and passive equalizer (PEQ) for low frequency attenuation, and our task is to co-optimize both of them. Our model plays a role as a solver to optimize the design of equalizers, while considering all signal integrity issues such as high frequency attenuation, crosstalk and so on.

 Our method utilizes recursive neural network commonly employed in natural language processing (NLP), in order to design HYEQ based on constructive DRL. Thus, each parameter of the equalizer is designed sequentially, reflecting other parameters. In this process, the design space of machine learning (ML) is determined by applying domain knowledge of equalizer, and thus even precise optimization is conducted. Furthermore, fast inference is conducted by trained neural network for any channel dimension. We validate that the proposed method outperforms conventional optimization algorithms such as random search (RS) and genetic algorithm (GA) in 3-coupled channel system of next generation high-bandwidth memory (HBM).

 

Title:

 

Sequential Policy Network-based Optimal Passive Equalizer Design for an Arbitrary Channel of High Bandwidth Memory using Advantage Actor Critic (EPEPS 2021)

 

Authors:

 

Seonguk Choi, Minsu Kim, Hyunwook Park, Keeyoung Son, Seongguk Kim, Jihun Kim, Joonsang Park, Haeyeon Kim, Taein Shin, Keunwoo Kim and Joungho Kim.

 

Abstract:

In this paper, we proposed a sequential policy network-based passive equalizer (PEQ) design method for an arbitrary channel of high bandwidth memory (HBM) using advantage actor critic (A2C) algorithm, considering signal integrity (SI) for the first time. PEQ design must consider the circuit parameters and placement for improving the performance. However, optimizing PEQ is complicated because various design parameters are coupled. Conventional optimization methods such as genetic algorithm (GA) repeat the optimization process for the changed conditions. In contrast, the proposed method suggests the improved solution based on the trained sequential policy network with flexibility for unseen conditions. For verification, we conducted electromagnetic (EM) simulation with optimized PEQs by GA, random search (RS) and the proposed method. Experimental results demonstrate that the proposed method outperformed the GA and RS by 4.4 \% and 6.4 \% respectively in terms of the eye-height.

 

Imitate Expert Policy and Learn Beyond: A Practical PDN Optimizer by Imitation Learning (DesignCon 2022, nominated for best paper award & Early career best paper award finalist)

 

Authors: Haeyeon Kim, Minsu Kim, Seonguk Choi, Jihun Kim, Joonsang Park, Keeyoung Son, Hyunwook Park, Subin Kim and Joungho Kim

 

 

Abstract: This paper proposes a practical and reusable decoupling capacitor (decap) placement solver using the attention model-based imitation learning (AM-IL). The proposed AM-IL framework imitates an expert policy by using pre-collected guiding datasets and trains a policy that outperforms the performance beyond the existing machine learning methods. The trained policy has reusability in terms of PDN with different probing port and keep-out regions; the constructed policy itself becomes the decap placement solver. In this paper, genetic algorithm is taken as an expert policy to verify how the proposed method generates a solver that learns beyond the level of the expert policy. The expert policy for imitation learning can be substituted by any algorithm or conventional tool, which means this is a fast and effective approach to improve existing methods. Moreover, by taking the existing data from the industry as guiding data or human experts as an expert policy, the proposed method can construct a reusable decap placement solver that is data-efficient, practical and guarantees a promising performance. This paper presents verification of AM-IL in comparison to two neural combinatorial optimization networks-based deep reinforcement learning methods, AM-RL and Ptr-RL. As a result, AM-IL achieved a performance score of 11.72, while AM-RL achieved 10.74 and Ptr-RL achieved 9.76. Unlike meta-heuristic methods such as genetic algorithm that require numerous iterations to find a near-optimal solution, the proposed AM-IL generates a near-optimal solution to any given problem by a single trial.

 

Title: Deep Reinforcement Learning Framework for Optimal Decoupling Capacitor Placement on General PDN with an Arbitrary Probing Port (EPEPS 2021)

 

Authors: Haeyeon Kim, Hyunwook Park, Minsu Kim, Seonguk Choi, Jihun Kim, Joonsang Park, Seongguk Kim, Subin Kim and Joungho Kim.

 

 

Abstract: This paper proposes a deep reinforcement learning (DRL) framework that learns a reusable policy to find the optimal placement of decoupling capacitors (decaps) on power distribution network (PDN) with an arbitrary probing port. The proposed DRL framework trains a policy parameterized by pointer network, which is a sequence-to-sequence neural network, based on REINFORCE algorithm. The policy finds the positional combination of a pre-defined number of decaps that best suppresses self-impedance of a given probing port on PDN with randomly assigned keep-out regions. Verification was done by allocating 20 decaps on ten randomly generated test sets with an arbitrary probing port and randomly selected keep-out regions. Performance of the policy generated by the proposed DRL framework was evaluated based on the magnitude of probing port self-impedance suppression followed by decap placement over 434 frequencies between 100MHz and 20GHz. The policy generated by the proposed framework achieves greater impedance suppression with fewer samples in comparison to random search heuristic method.

 

Conference: NeurIPS 2021

 

 

Title:

 

Learning Collaborative Policies to Solve NP-hard Routing Problems

 

Authors:

 

Minsu Kim, Jinkyoo Park and Joungho Kim.

Abstract:

Recently, deep reinforcement learning (DRL) frameworks have shown potential for solving NP-hard routing problems such as the traveling salesman problem (TSP) without problem-specific expert knowledge. Although DRL can be used to solve complex problems, DRL frameworks still struggle to compete with state-of-the-art heuristics showing a substantial performance gap. This paper proposes a novel hierarchical problem-solving strategy, termed learning collaborative policies (LCP), which can effectively find the near-optimum solution using two iterative DRL policies: the seeder and reviser. The seeder generates as diversified candidate solutions as possible (seeds) while being dedicated to exploring over the full combinatorial action space (i.e., sequence of assignment action). To this end, we train the seeder’s policy using a simple yet effective entropy regularization reward to encourage the seeder to find diverse solutions. On the other hand, the reviser modifies each candidate solution generated by the seeder; it partitions the full trajectory into sub-tours and simultaneously revises each sub-tour to minimize its traveling distance. Thus, the reviser is trained to improve the candidate solution’s quality, focusing on the reduced solution space (which is beneficial for exploitation). Extensive experiments demonstrate that the proposed two-policies collaboration scheme improves over single-policy DRL framework on various NP-hard routing problems, including TSP, prize collecting TSP (PCTSP), and capacitated vehicle routing problem (CVRP).