GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning
When the data is distributed across multiple servers, lowering the communication cost between the servers (or workers) while solving the distributed learning problem is an important problem and is the focus of this paper. In particular, we propose a fast, and communication-efficient decentralized fr...
Main Authors: | , , , , |
---|---|
Format: | Article in Journal/Newspaper |
Language: | unknown |
Published: |
arXiv
2019
|
Subjects: | |
Online Access: | https://dx.doi.org/10.48550/arxiv.1909.00047 https://arxiv.org/abs/1909.00047 |
id |
ftdatacite:10.48550/arxiv.1909.00047 |
---|---|
record_format |
openpolar |
spelling |
ftdatacite:10.48550/arxiv.1909.00047 2023-05-15T16:02:02+02:00 GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning Elgabli, Anis Park, Jihong Bedi, Amrit S. Bennis, Mehdi Aggarwal, Vaneet 2019 https://dx.doi.org/10.48550/arxiv.1909.00047 https://arxiv.org/abs/1909.00047 unknown arXiv arXiv.org perpetual, non-exclusive license http://arxiv.org/licenses/nonexclusive-distrib/1.0/ Machine Learning cs.LG Distributed, Parallel, and Cluster Computing cs.DC Information Theory cs.IT Networking and Internet Architecture cs.NI Machine Learning stat.ML FOS Computer and information sciences Article CreativeWork article Preprint 2019 ftdatacite https://doi.org/10.48550/arxiv.1909.00047 2022-03-10T16:41:39Z When the data is distributed across multiple servers, lowering the communication cost between the servers (or workers) while solving the distributed learning problem is an important problem and is the focus of this paper. In particular, we propose a fast, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm, Group Alternating Direction Method of Multipliers (GADMM) is based on the Alternating Direction Method of Multipliers (ADMM) framework. The key novelty in GADMM is that it solves the problem in a decentralized topology where at most half of the workers are competing for the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges to the optimal solution for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets. Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove its convergence under the time-varying network topology of the workers. Article in Journal/Newspaper DML DataCite Metadata Store (German National Library of Science and Technology) |
institution |
Open Polar |
collection |
DataCite Metadata Store (German National Library of Science and Technology) |
op_collection_id |
ftdatacite |
language |
unknown |
topic |
Machine Learning cs.LG Distributed, Parallel, and Cluster Computing cs.DC Information Theory cs.IT Networking and Internet Architecture cs.NI Machine Learning stat.ML FOS Computer and information sciences |
spellingShingle |
Machine Learning cs.LG Distributed, Parallel, and Cluster Computing cs.DC Information Theory cs.IT Networking and Internet Architecture cs.NI Machine Learning stat.ML FOS Computer and information sciences Elgabli, Anis Park, Jihong Bedi, Amrit S. Bennis, Mehdi Aggarwal, Vaneet GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning |
topic_facet |
Machine Learning cs.LG Distributed, Parallel, and Cluster Computing cs.DC Information Theory cs.IT Networking and Internet Architecture cs.NI Machine Learning stat.ML FOS Computer and information sciences |
description |
When the data is distributed across multiple servers, lowering the communication cost between the servers (or workers) while solving the distributed learning problem is an important problem and is the focus of this paper. In particular, we propose a fast, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm, Group Alternating Direction Method of Multipliers (GADMM) is based on the Alternating Direction Method of Multipliers (ADMM) framework. The key novelty in GADMM is that it solves the problem in a decentralized topology where at most half of the workers are competing for the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges to the optimal solution for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets. Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove its convergence under the time-varying network topology of the workers. |
format |
Article in Journal/Newspaper |
author |
Elgabli, Anis Park, Jihong Bedi, Amrit S. Bennis, Mehdi Aggarwal, Vaneet |
author_facet |
Elgabli, Anis Park, Jihong Bedi, Amrit S. Bennis, Mehdi Aggarwal, Vaneet |
author_sort |
Elgabli, Anis |
title |
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning |
title_short |
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning |
title_full |
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning |
title_fullStr |
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning |
title_full_unstemmed |
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning |
title_sort |
gadmm: fast and communication efficient framework for distributed machine learning |
publisher |
arXiv |
publishDate |
2019 |
url |
https://dx.doi.org/10.48550/arxiv.1909.00047 https://arxiv.org/abs/1909.00047 |
genre |
DML |
genre_facet |
DML |
op_rights |
arXiv.org perpetual, non-exclusive license http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
op_doi |
https://doi.org/10.48550/arxiv.1909.00047 |
_version_ |
1766397674708271104 |