Utility–Privacy Trade-Off in Distributed Machine Learning Systems

In distributed machine learning (DML), though clients’ data are not directly transmitted to the server for model training, attackers can obtain the sensitive information of clients by analyzing the local gradient parameters uploaded by clients. For this case, we use the differential privacy (DP) mec...

Full description

Bibliographic Details
Published in:Entropy
Main Authors: Xia Zeng, Chuanchuan Yang, Bin Dai
Format: Article in Journal/Newspaper
Language:English
Published: MDPI AG 2022
Subjects:
Q
DML
Online Access:https://doi.org/10.3390/e24091299
https://doaj.org/article/891ae7052f0d4de5a69bf460644127dc
Description
Summary:In distributed machine learning (DML), though clients’ data are not directly transmitted to the server for model training, attackers can obtain the sensitive information of clients by analyzing the local gradient parameters uploaded by clients. For this case, we use the differential privacy (DP) mechanism to protect the clients’ local parameters. In this paper, from an information-theoretic point of view, we study the utility–privacy trade-off in DML with the help of the DP mechanism. Specifically, three cases including independent clients’ local parameters with independent DP noise, dependent clients’ local parameters with independent/dependent DP noise are considered. Mutual information and conditional mutual information are used to characterize utility and privacy, respectively. First, we show the relationship between utility and privacy for the three cases. Then, we show the optimal noise variance that achieves the maximal utility under a certain level of privacy. Finally, the results of this paper are further illustrated by numerical results.