SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning

Achref Rebai; Mubarak Adetunji Ojewale; Anees Ullah; Marco Canini; Suhaib A. Fahmy

doi:10.1145/3672198.3673801

SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning

Achref Rebai
, Mubarak Adetunji Ojewale
, Anees Ullah
, Marco Canini
, Suhaib A. Fahmy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

To alleviate the communication bottleneck of distributed deep learning training, several data compression algorithms have been proposed. However, these algorithms introduce computational overhead and resource allocation concerns on CPUs and GPUs. In this paper, we introduce SqueezeNIC, an FPGA-based Network Interface Card (NIC) that offloads communication compression from CPUs/GPUs, bridging a high bandwidth intra-node network with a high bandwidth inter-node network. It enables better overlap of gradient communication and computation to further reduce training time per iteration in distributed training. Our evaluations shows that SqueezeNIC achieves line rate compression and can speed up training by up to a factor of 1.21×, compared to baseline approaches.

Original language	English
Title of host publication	NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing
Publisher	Association for Computing Machinery, Inc
Pages	61-68
Number of pages	8
ISBN (Electronic)	9798400707131
DOIs	https://doi.org/10.1145/3672198.3673801
State	Published - 4 Aug 2024
Externally published	Yes
Event	1st Workshop on Networks for AI Computing, NAIC 2024, at ACM SIGCOMM 2024 - Sydney, Australia Duration: 4 Aug 2024 → 8 Aug 2024

Publication series

Name	NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing

Conference

Conference	1st Workshop on Networks for AI Computing, NAIC 2024, at ACM SIGCOMM 2024
Country/Territory	Australia
City	Sydney
Period	4/08/24 → 8/08/24

Bibliographical note

Publisher Copyright:
© 2024 Owner/Author.

Keywords

Distributed Training
FPGA
In-Network Compression

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Computer Science Applications
Information Systems
Signal Processing

Access to Document

10.1145/3672198.3673801

Cite this

Rebai, A., Ojewale, M. A., Ullah, A., Canini, M., & Fahmy, S. A. (2024). SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning. In NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing (pp. 61-68). (NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/3672198.3673801

@inproceedings{cc540e54ecb3435188da4ce02e79b2da,

title = "SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning",

abstract = "To alleviate the communication bottleneck of distributed deep learning training, several data compression algorithms have been proposed. However, these algorithms introduce computational overhead and resource allocation concerns on CPUs and GPUs. In this paper, we introduce SqueezeNIC, an FPGA-based Network Interface Card (NIC) that offloads communication compression from CPUs/GPUs, bridging a high bandwidth intra-node network with a high bandwidth inter-node network. It enables better overlap of gradient communication and computation to further reduce training time per iteration in distributed training. Our evaluations shows that SqueezeNIC achieves line rate compression and can speed up training by up to a factor of 1.21×, compared to baseline approaches.",

keywords = "Distributed Training, FPGA, In-Network Compression",

author = "Achref Rebai and Ojewale, \{Mubarak Adetunji\} and Anees Ullah and Marco Canini and Fahmy, \{Suhaib A.\}",

note = "Publisher Copyright: {\textcopyright} 2024 Owner/Author.; 1st Workshop on Networks for AI Computing, NAIC 2024, at ACM SIGCOMM 2024 ; Conference date: 04-08-2024 Through 08-08-2024",

year = "2024",

month = aug,

day = "4",

doi = "10.1145/3672198.3673801",

language = "English",

series = "NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing",

publisher = "Association for Computing Machinery, Inc",

pages = "61--68",

booktitle = "NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing",

}

Rebai, A, Ojewale, MA, Ullah, A, Canini, M & Fahmy, SA 2024, SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning. in NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing. NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing, Association for Computing Machinery, Inc, pp. 61-68, 1st Workshop on Networks for AI Computing, NAIC 2024, at ACM SIGCOMM 2024, Sydney, Australia, 4/08/24. https://doi.org/10.1145/3672198.3673801

SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning. / Rebai, Achref; Ojewale, Mubarak Adetunji; Ullah, Anees et al.
NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing. Association for Computing Machinery, Inc, 2024. p. 61-68 (NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SqueezeNIC

T2 - 1st Workshop on Networks for AI Computing, NAIC 2024, at ACM SIGCOMM 2024

AU - Rebai, Achref

AU - Ojewale, Mubarak Adetunji

AU - Ullah, Anees

AU - Canini, Marco

AU - Fahmy, Suhaib A.

PY - 2024/8/4

Y1 - 2024/8/4

N2 - To alleviate the communication bottleneck of distributed deep learning training, several data compression algorithms have been proposed. However, these algorithms introduce computational overhead and resource allocation concerns on CPUs and GPUs. In this paper, we introduce SqueezeNIC, an FPGA-based Network Interface Card (NIC) that offloads communication compression from CPUs/GPUs, bridging a high bandwidth intra-node network with a high bandwidth inter-node network. It enables better overlap of gradient communication and computation to further reduce training time per iteration in distributed training. Our evaluations shows that SqueezeNIC achieves line rate compression and can speed up training by up to a factor of 1.21×, compared to baseline approaches.

AB - To alleviate the communication bottleneck of distributed deep learning training, several data compression algorithms have been proposed. However, these algorithms introduce computational overhead and resource allocation concerns on CPUs and GPUs. In this paper, we introduce SqueezeNIC, an FPGA-based Network Interface Card (NIC) that offloads communication compression from CPUs/GPUs, bridging a high bandwidth intra-node network with a high bandwidth inter-node network. It enables better overlap of gradient communication and computation to further reduce training time per iteration in distributed training. Our evaluations shows that SqueezeNIC achieves line rate compression and can speed up training by up to a factor of 1.21×, compared to baseline approaches.

KW - Distributed Training

KW - FPGA

KW - In-Network Compression

UR - https://www.scopus.com/pages/publications/85202436614

U2 - 10.1145/3672198.3673801

DO - 10.1145/3672198.3673801

M3 - Conference contribution

AN - SCOPUS:85202436614

T3 - NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing

SP - 61

EP - 68

BT - NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing

PB - Association for Computing Machinery, Inc

Y2 - 4 August 2024 through 8 August 2024

ER -

Rebai A, Ojewale MA, Ullah A, Canini M, Fahmy SA. SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning. In NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing. Association for Computing Machinery, Inc. 2024. p. 61-68. (NAIC 2024 - Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing). doi: 10.1145/3672198.3673801

SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this