Principal Singular Values and Singular Vectors Adaptation of Large Language Models

View the PDF file from the paper entitled Pissa: the main individual values ​​and single single air conditioning of the large language models, by Fanxu Ming and 2 other authors

PDF HTML (experimental) view

a summary:To the PEFT linguistic models (PEFT) effectively, the lower adjustment method (Lora) is almost changed the form of the form $ \ Delta w \ in \ mathbb {r}^{M \ Times N} $ through a two -product product $ A \ in \ mathb {r} \ mathbb {r}^{r \ Times N} $, where $ R \ l \ min (M, N) is $, $ A $ with Gaussian noise, $ B $ B with zeros. Lora freezes the original form $ W and update the “Noise & Zero” adapter, which may lead to slow rapprochement. To overcome this restriction, we offer the main individual values ​​and single single adaptation (Pissa). Pissa shares the same architecture as Lora, but it is to prepare the transformer matrix $ A $ A and $ B with the main components of the original matrix $ w, and puts the remaining ingredients in the remaining matrix $ w^{RES} \ in \ mathbb compared to Lora, Pissa updates the main ingredients with freezing the “remaining” parts, which allows rapprochement The fastest and performance of Mohsen. Pissa and Lora’s comparative experiments reveal across 12 different models, ranging from 184 meters to 70b, including 5 NLG and 8 NLU tasks, that Pissa is constantly outperforming Lora under identical experimental settings. On the GSM8K Standard, Mistral-7B is set with a 72.86 % Pissa, exceeding 67.7 % of Lora by 5.16 %. Due to the same architecture, Pissa is also compatible with quantitative measurement to reduce memory requirements for control. Compared to QLora, QPissa displays smaller quantity errors in the initial stages. Llama-3-70B polishing on GSM8K, QPissa gets 86.05 %, and QLora performance exceeds 81.73 %. Take advantage of the fast SVD technology, Pissa can be prepared in just a few seconds, which provides a small cost to move from Lora to Pissa. A code available in this URL https.