Archives: AI in Computer division

EE Professor Dongsu Han’s Research Team Develops Technology to Accelerate AI Model Training in Distributed Environments Using Consumer-Grade GPUs

<(from left) Professor Dongsu Han, Dr. Hwijoon Iim, Ph.D. Candidate Juncheol Ye>

 

Professor Dongsu Han’s research team of the KAIST Department of Electrical Engineering has developed a groundbreaking technology that accelerates AI model training in distributed environments with limited network bandwidth using consumer-grade GPUs.

 

Training the latest AI models typically requires expensive infrastructure, such as high-performance GPUs costing tens of millions in won and high-speed dedicated networks.

As a result, most researchers in academia and small to medium-sized enterprises have to rely on cheaper, consumer-grade GPUs for model training.

However, they face difficulties in efficient model training due to network bandwidth limitations.

 

<Figure 1. Problems in Conventional Low-Cost Distributed Deep Learning Environments>

 

To address these issues, Professor Han’s team developed a distributed learning framework called StellaTrain.

StellaTrain accelerates model training on low-cost GPUs by integrating a pipeline that utilizes both CPUs and GPUs. It dynamically adjusts batch sizes and compression rates according to the network environment, enabling fast model training in multi-cluster and multi-node environments without the need for high-speed dedicated networks.

 

StellaTrain adopts a strategy that offloads gradient compression and optimization processes to the CPU to maximize GPU utilization by optimizing the learning pipeline. The team developed and applied a new sparse optimization technique and cache-aware gradient compression technology that work efficiently on CPUs.

 

This implementation creates a seamless learning pipeline where CPU tasks overlap with GPU computations. Furthermore, dynamic optimization technology adjusts batch sizes and compression rates in real-time according to network conditions, achieving high GPU utilization even in limited network environments.

 

<Figure 2. Overview of the StellaTrain Learning Pipeline>

 

Through these innovations, StellaTrain significantly improves the speed of distributed model training in low-cost multi-cloud environments, achieving up to 104 times performance improvement compared to the existing PyTorch DDP.

 

Professor Han’s research team has paved the way for efficient AI model training without the need for expensive data center-grade GPUs and high-speed networks. This breakthrough is expected to greatly aid AI research and development in resource-constrained environments, such as academia and small to medium-sized enterprises.

 

Professor Han emphasized, “KAIST is demonstrating leadership in the AI systems field in South Korea.” He added, “We will continue active research to implement large-scale language model (LLM) training, previously considered the domain of major IT companies, in more affordable computing environments. We hope this research will serve as a critical stepping stone toward that goal.”

 

The research team included Dr. Hwijoon Iim and Ph.D. candidate Juncheol Ye from KAIST, as well as Professor Sangeetha Abdu Jyothi from UC Irvine. The findings were presented at ACM SIGCOMM 2024, the premier international conference in the field of computer networking, held from August 4 to 8 in Sydney, Australia (Paper title: Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs). 

 

Meanwhile, Professor Han’s team has also made continuous research advancements in the AI systems field, presenting a framework called ES-MoE, which accelerates Mixture of Experts (MoE) model training, at ICML 2024 in Vienna, Austria.

 

By overcoming GPU memory limitations, they significantly enhanced the scalability and efficiency of large-scale MoE model training, enabling fine-tuning of a 15-billion parameter language model using only four GPUs. This achievement opens up the possibility of effectively training large-scale AI models with limited computing resources.

 

<Figure 3. Overview of the ES-MoE Framework>

 

 

 

 

 

Professor Sung-Ju Lee’s research team develops a smarthphone AI system that diagnoses mental health based on user’s voice and text input

 

 

CAMEL research team has been successively selected for 2023 Samsung Future Technology Development Program

 

 

The CAMEL research team from our department, led by Professor Myoungsoo Jung, has been chosen to participate in the Samsung Future Technology Development Program.

 

This recognition comes with support for their study titled, “Software-Hardware Co-Design for Dynamic Acceleration in Sparsity-aware Hyperscale AI.”

 

Large AI models, such as mixture of experts (MoE), autoencoders, and multimodal learning, grouped under the umbrella of Hyperscale AI, have gained traction due to the success of expansive model-driven applications, including ChatGPT.

Based on the insight that computational traits of these models often shift during training, the research team has suggested acceleration strategies.

These encompass software technologies, unique algorithms, and hardware accelerator layouts.

A key discovery by the team was the inability of existing training systems to account for variations in data sparsity and computational dynamics between model layers. This oversight obstructs adaptive acceleration. 

 

To address this, the CAMEL team introduced a dynamic acceleration method that can detect shifts in computational traits and adapt computation techniques in real-time.

The findings from this research could benefit not only Hyperscale AI but also the larger domain of deep learning and the burgeoning services sector.

The team’s goals include producing tangible hardware models and offering open-source software platforms.

 

Samsung Electronics, since 2013, has initiated the ‘Future Technology Development Program’, investing KRW 1.5 trillion to stimulate technological innovation pivotal for future societal progress.

For a decade, they have backed initiatives in foundational science, innovative materials, and ICT, particularly favoring ventures that are high-risk but offer significant returns.

The CAMEL team has been collaborating with Samsung since 2021 on a project focusing on accelerating Graph Neural Networks (GNNs). We extend our hearty congratulations to them as they embark on this fresh exploration into the realm of Hyperscale AI.

 

Our department’s Professor Myounsoo Jung’s research team has developed the world’s first AI semiconductor for search engines based on CXL 3.0.

 

Approximate nearest neighbor search (ANNS) is widely used in commercial services such as image search, database, and recommendation systems.

However, in production-level ANNS, there is a challenge of requiring a large amount of memory due to the extensive dataset.

To address this memory pressure issue, modern ANNS techniques leverage lossy compression methods or employ persistent storage for their memory expansion.

However, these approaches often suffer from low accuracy and performance.

 

The research team proposed expanding memory capacity via compute express link (CXL), which is PCIe based open-industry interconnect technology that allows the underlying working memory to be highly scalable and composable at a low cost.

Furthermore, the use of a CXL switch enables connecting multiple memory expanders to a single port, providing greater scalability. However, memory expansion through CXL has the disadvantage of increased memory access time compared to local memory.

 

The research team has developed an AI semiconductor, ‘CXL-ANNS‘, which leverages CXL switch and memory expanders to accommodate high memory pressure that comes from extensive datasets without losing accuracy or performance.

Additionally, by using near data processing and data partitioning based on locality, the performance of CXL-ANNS is improved.

They also compared prototyped CXL-ANNS with the existing solutions for ANNS. Compared to previous research, CXL-ANNS shows 111 times higher performance. Particularly, 92 times higher performance can be achieved compared to Microsoft’s solution that is used in commercial service.

 

This research, along with the paper titled “CXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation for Billion-Scale Approximate Nearest Neighbor Search”, will be presented in July at ‘USENIX Annual Technical Conference, ATC, 2023’.

 

 

 

 

The research was supported by Panmnesia (http://panmnesia.com). More information on this paper can be found at CAMELab website (http://camelab.org).

 

[News Link]

The Korea Economic Daily: https://www.hankyung.com/it/article/202305259204i

The Herald Business: http://news.heraldcorp.com/view.php?ud=20230525000225

ChosunBiz: https://biz.chosun.com/science-chosun/technology/2023/05/25/4UW5LPX3WVARVIS3QBBICPINFM/

etnews: https://www.etnews.com/20230525000092

[Prof. David Hyunchul Shim]

 

KAIST PhD candidate Yuji Roh from the School of Electrical Engineering (advisor: Prof. Steven Euijong Whang) was selected as a recipient of the 2022 Microsoft Research PhD Fellowship. 

[Yuji Roh]

 

The list of recipients: https://www.microsoft.com/en-us/research/academic-program/phd-fellowship/2022-recipients/

Interview (Asia): https://www.youtube.com/watch?v=qwq3R1XU8UE

 

 

[Prof. Minsoo Rhu]

 

 

[Prof. Myoungsoo Jung, Miryeong Kwon, Seungjun Lee, and Hyunkyu Cho from left]

 

Our department’s Professor Myoungsoo Jung’s research team has developed the world’s first Predictable Latency Mode (PLM) SSD based hardware and software co-designed framework for Log-Structured Merge Key-Value Stores (LSM KV store).

 

The research team has developed the ‘hardware and software co-designed framework for LSM KV store, Vigil-KV’ that eliminates long-tail latency by utilizing the Predictable Latency Mode (PLM) interface, which provides constant read latency, to the actual datacenter-scale SSD. Vigil-KV outpoerforms 3.19x faster tail latency and 34% faster average latency compared to the existing LSM KV store.

 

LSM KV store, a kind of database, is used to manage various application data, and it must process the user requests within the requirement time in order not to degrade the user experience. To this end, Vigil-KV enables a predictable latency mode (PLM) interface on an actual datacenter-scale NVMe SSD (PLM SSD), which guarantees constant read latency in deterministic mode related to read service without performing SSD’s internal tasks.

 

Specifically, Vigil-KV hardware makes the deterministic mode SSDs exist in the system to remove SSD’s internal tasks by configuring PLM SSD RAID. In addition, Vigil-KV software prevents the deterministic mode from being released by LSM KV store’s internal tasks, scheduling LSM KV store operations (e.x., compaction/flush operations) and client requests.

 

Among the proposed research results, especially noteworthy is that Vigil-KV is the first work that implements the PLM interface in a real SSD and makes the read latency of LSM KV store deterministic in a hardware-software co-design manner. They prototype Vigil-KV hardware on a 1.92TB datacenter-scale NVMe SSD while implementing Vigil-KV software using Linux 4.19.91 and RocksDB 6.23.0.

 

The KAIST Ph.D. Candidates (Miryeong Kwon, Seungjun Lee, and Hyunkyu Choi) participate in this research, and the paper (Vigil-KV: Hardware-Software Co-Design to Integrate Strong Latency Determinism into Log-Structured Merge Key-Value Stores) was reported in July, 11th at ‘USENIX Annual Technical Conference, ATC, 2022’. In addition, they has won the Best Paper Award from Samsung for this paper (Vigil-KV) with Professor Jae-Hyeok Choi’s research team.

 

The Best Paper Award from Samsung recognizes master’s and doctorate students that participated in research grant projects and published papers related to the project among papers adopted by foreign journals/conferences since September 21st. This year’s awards consisted of grand award (2 people), excellence award (1 person), and encouragement award (2 people).

The research was supported by Samsung. More information on this paper can be found at http://camelab.org.