[2409.17538] On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy

PDF display of the paper entitled on the implicit relationship between low adaptation and differential privacy, by Saber Malekmohammadi and 1 other authors

PDF HTML (experimental) view

a summary:An important approach to the treatment of natural language includes training for models on a large scale on public domain data followed by adapting them to specific tasks or fields. As the models grow in size, the polishing of all its parameters becomes increasingly impractical. To address this, some methods have been suggested to adapt low -ranking tasks to language models, for example, Lora and Flora. These methods maintain the preparatory weights that have been pre -trained and include low -ranking decomposition preservants in some layers of transformers, which are called transformers. This approach significantly reduces the number of training parameters required for the assignment tasks compared to controlling all full parameters. In this work, we look at the low -ranking adaptation of the data privacy lens. Theoretically, the low -ranking adaptation used in Lora and Flora leads to injection of some random noise into the gradients of the impartials that excite the transformer parameters. We determine the injected noise contrast and make it clear that the more classification of the smaller adaptation, the greater the contrast of noise. By creating a SESEEN type binding at the total contrast distance between the injected noise distribution and Gaous distribution with the same contrast, we explain that low-ranking adaptation dynamics approach those specially differential of transformers. Finally, using Johnson-Lindenstrauss Lemma, we make it clear that when the graduation is increased with the graduation, the adaptation is very low soon from the performance of the DPSGD algorithm with a fixed noise scale for adjusting transformers. Our theoretical results have suggested and approved through our experimental results, we explain that low -ranking adaptation, as well as reducing mathematical complications and implicitly provides protection from privacy on precise control data, without stimulating the complexity of the high space of DPSGD.