Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers

The paper presents Vertical LoRA (VLoRA), a new approach that interprets Transformers as dense Expectation-Maximization algorithms and introduces a model design that substantially reduces parameters while maintaining performance, by learning layer increments and applying LoRA decomposition recursively.

Abstract

In this paper, we show how Transformers can be interpreted as dense Expectation-Maximization algorithms performed on Bayesian Nets. Based on the above interpretation, we propose a new model design paradigm, namely Vertical LoRA (VLoRA), which reduces the parameter count dramatically while preserving performance. In VLoRA, a model consists of layers, each of which recursively learns an increment based on the previous layer. We then apply LoRA decomposition to the increments. VLoRA works on the base model, which is orthogonal to LoRA, meaning they can be used together. We do experiments on various tasks and models. The results show that 1) with VLoRA, the Transformer model parameter count can be reduced dramatically and 2) the performance of the original model is preserved. The source code is available at \url{this https URL}

Method

The authors used a novel methodology called Vertical LoRA (VLoRA) to interpret and design Transformer models. This methodology is based on the concept of viewing Transformer layers as iterations of an Expectation-Maximization (EM) algorithm, where each layer learns an increment based on the previous layer. The increments are then subjected to LoRA (Low-Rank Adaptation) decomposition, which involves factorizing the weight matrices into low-rank components to reduce the parameter count. This approach is distinct from traditional LoRA, which adapts each layer independently, and it allows for a more efficient use of parameters while preserving the model's performance.

Main Finding

The authors discovered that by applying the Vertical LoRA (VLoRA) methodology to Transformer models, they could significantly reduce the number of parameters in the models without compromising their performance. This was achieved by interpreting the Transformer layers as iterations of an Expectation-Maximization (EM) algorithm and then using LoRA decomposition to factorize the weight increments between layers. Their experiments showed that VLoRA not only reduced the parameter count but also improved the models' resistance to overfitting, potentially even outperforming the original models in some cases.

Conclusion

The conclusion of the paper is that the Vertical LoRA (VLoRA) methodology is an effective approach for designing Transformer models that are more parameter-efficient. By interpreting Transformer layers as EM algorithm iterations and applying LoRA decomposition to the weight increments, VLoRA reduces the parameter count dramatically while preserving the performance of the original models. The experiments conducted in the paper demonstrate that VLoRA models are less prone to overfitting and can achieve comparable or even better performance than the original models.

Keywords

Low-rank Adaptation, Transformer, EM Algorithm