Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning

Anis Elgabli; Jihong Park; Amrit S. Bedi; Mehdi Bennis; Vaneet Aggarwal

doi:10.1109/ICASSP40776.2020.9054491

Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning

Anis Elgabli
, Jihong Park
, Amrit S. Bedi
, Mehdi Bennis
, Vaneet Aggarwal

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

15 Scopus citations

Abstract

In this paper, we propose a communication-efficient decentralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). Every worker in Q-GADMM communicates only with two neighbors, and updates its model via the group alternating direct method of multiplier (GADMM), thereby ensuring fast convergence while reducing the number of communication rounds. Furthermore, each worker quantizes its model updates before transmissions, thereby decreasing the communication payload sizes. We prove that Q-GADMM converges to the optimal solution for convex loss functions, and numerically show that Q-GADMM yields 7x less communication cost while achieving almost the same accuracy and convergence speed compared to GADMM without quantization.

Original language	English
Title of host publication	2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	8876-8880
Number of pages	5
ISBN (Electronic)	9781509066315
DOIs	https://doi.org/10.1109/ICASSP40776.2020.9054491
State	Published - May 2020
Externally published	Yes

Publication series

Name	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume	2020-May
ISSN (Print)	1520-6149

Bibliographical note

Publisher Copyright:
© 2020 IEEE

Keywords

ADMM
Communication-efficient decentralized machine learning
GADMM
Quantization

ASJC Scopus subject areas

Software
Signal Processing
Electrical and Electronic Engineering

Access to Document

10.1109/ICASSP40776.2020.9054491

Cite this

Elgabli, A., Park, J., Bedi, A. S., Bennis, M., & Aggarwal, V. (2020). Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning. In 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings (pp. 8876-8880). Article 9054491 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2020-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP40776.2020.9054491

Elgabli, Anis ; Park, Jihong ; Bedi, Amrit S. et al. / Q-GADMM : Quantized group ADMM for communication efficient decentralized machine learning. 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 8876-8880 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

@inproceedings{fbb3914fb6dd4f639f03bcf14c9fc290,

title = "Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning",

abstract = "In this paper, we propose a communication-efficient decentralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). Every worker in Q-GADMM communicates only with two neighbors, and updates its model via the group alternating direct method of multiplier (GADMM), thereby ensuring fast convergence while reducing the number of communication rounds. Furthermore, each worker quantizes its model updates before transmissions, thereby decreasing the communication payload sizes. We prove that Q-GADMM converges to the optimal solution for convex loss functions, and numerically show that Q-GADMM yields 7x less communication cost while achieving almost the same accuracy and convergence speed compared to GADMM without quantization.",

keywords = "ADMM, Communication-efficient decentralized machine learning, GADMM, Quantization",

author = "Anis Elgabli and Jihong Park and Bedi, \{Amrit S.\} and Mehdi Bennis and Vaneet Aggarwal",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE",

year = "2020",

month = may,

doi = "10.1109/ICASSP40776.2020.9054491",

language = "English",

series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "8876--8880",

booktitle = "2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings",

address = "United States",

}

Elgabli, A, Park, J, Bedi, AS, Bennis, M & Aggarwal, V 2020, Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning. in 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings., 9054491, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2020-May, Institute of Electrical and Electronics Engineers Inc., pp. 8876-8880. https://doi.org/10.1109/ICASSP40776.2020.9054491

Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning. / Elgabli, Anis; Park, Jihong; Bedi, Amrit S. et al.
2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. p. 8876-8880 9054491 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2020-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Q-GADMM

T2 - Quantized group ADMM for communication efficient decentralized machine learning

AU - Elgabli, Anis

AU - Park, Jihong

AU - Bedi, Amrit S.

AU - Bennis, Mehdi

AU - Aggarwal, Vaneet

PY - 2020/5

Y1 - 2020/5

N2 - In this paper, we propose a communication-efficient decentralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). Every worker in Q-GADMM communicates only with two neighbors, and updates its model via the group alternating direct method of multiplier (GADMM), thereby ensuring fast convergence while reducing the number of communication rounds. Furthermore, each worker quantizes its model updates before transmissions, thereby decreasing the communication payload sizes. We prove that Q-GADMM converges to the optimal solution for convex loss functions, and numerically show that Q-GADMM yields 7x less communication cost while achieving almost the same accuracy and convergence speed compared to GADMM without quantization.

AB - In this paper, we propose a communication-efficient decentralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). Every worker in Q-GADMM communicates only with two neighbors, and updates its model via the group alternating direct method of multiplier (GADMM), thereby ensuring fast convergence while reducing the number of communication rounds. Furthermore, each worker quantizes its model updates before transmissions, thereby decreasing the communication payload sizes. We prove that Q-GADMM converges to the optimal solution for convex loss functions, and numerically show that Q-GADMM yields 7x less communication cost while achieving almost the same accuracy and convergence speed compared to GADMM without quantization.

KW - ADMM

KW - Communication-efficient decentralized machine learning

KW - GADMM

KW - Quantization

UR - https://www.scopus.com/pages/publications/85091189676

U2 - 10.1109/ICASSP40776.2020.9054491

DO - 10.1109/ICASSP40776.2020.9054491

M3 - Conference contribution

AN - SCOPUS:85091189676

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 8876

EP - 8880

BT - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Elgabli A, Park J, Bedi AS, Bennis M, Aggarwal V. Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning. In 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2020. p. 8876-8880. 9054491. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). doi: 10.1109/ICASSP40776.2020.9054491

Q-GADMM: Quantized group ADMM for communication efficient decentralized machine learning

Abstract

Publication series

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this