Utility–Privacy Trade-Off in Distributed Machine Learning Systems

• Published in: Entropy
• Main Authors: Xia Zeng, Chuanchuan Yang, Bin Dai
• Format: Article in Journal/Newspaper
• Language: English
• Published: MDPI AG 2022
• Subjects: differential privacy; distributed machine learning; mutual information; Gaussian noise; trade-off; Science; Q; Astrophysics; QB460-466; Physics; QC1-999; DML
• Online Access: https://doi.org/10.3390/e24091299; https://doaj.org/article/891ae7052f0d4de5a69bf460644127dc

In distributed machine learning (DML), though clients’ data are not directly transmitted to the server for model training, attackers can obtain the sensitive information of clients by analyzing the local gradient parameters uploaded by clients. For this case, we use the differential privacy (DP) mechanism to protect the clients’ local parameters. In this paper, from an information-theoretic point of view, we study the utility–privacy trade-off in DML with the help of the DP mechanism. Specifically, three cases including independent clients’ local parameters with independent DP noise, dependent clients’ local parameters with independent/dependent DP noise are considered. Mutual information and conditional mutual information are used to characterize utility and privacy, respectively. First, we show the relationship between utility and privacy for the three cases. Then, we show the optimal noise variance that achieves the maximal utility under a certain level of privacy. Finally, the results of this paper are further illustrated by numerical results.