publications

2024

1. ArXiv

   A Primer on the Inner Workings of Transformer-based Language Models

   Ferrando, Javier, Sarti, Gabriele, Bisazza, Arianna, and R. Costa-jussà, Marta

   In ArXiv 2024

   Abs PDF

   The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive overview of the known internal mechanisms implemented by these models, uncovering connections across popular approaches and active research directions in this area.

2. EMNLP Findings

   On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Task

   Ferrando, Javier, and R. Costa-jussà, Marta

   In Findings of the Association for Computational Linguistics: EMNLP 2024

   Abs PDF

   Several algorithms implemented by language models have recently been successfully reversed-engineered. However, these findings have been concentrated on specific tasks and models, leaving it unclear how universal circuits are across different settings. In this paper, we study the circuits implemented by Gemma 2B for solving the subject-verb agreement task across two different languages, English and Spanish. We discover that both circuits are highly consistent, being mainly driven by a particular attention head writing a ‘subject number’ signal to the last residual stream, which is read by a small set of neurons in the final MLPs. Notably, this subject number signal is represented as a direction in the residual stream space, and is language-independent. We demonstrate that this direction has a causal effect on the model predictions, effectively flipping the Spanish predicted verb number by intervening with the direction found in English. Finally, we present evidence of similar behavior in other models within the Gemma 1 and Gemma 2 families.

3. EMNLP

   Information Flow Routes: Automatically Interpreting Language Models at Scale

   Ferrando, Javier, and Voita, Elena

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing 2024

   Abs PDF

   Information flows by routes inside the network via mechanisms implemented in the model. These routes can be represented as graphs where nodes correspond to token representations and edges to operations inside the network. We automatically build these graphs in a top-down manner, for each prediction leaving only the most important nodes and edges. In contrast to the existing workflows relying on activation patching, we do this through attribution: this allows us to efficiently uncover existing circuits with just a single forward pass. Additionally, the applicability of our method is far beyond patching: we do not need a human to carefully design prediction templates, and we can extract information flow routes for any prediction (not just the ones among the allowed templates). As a result, we can talk about model behavior in general, for specific types of predictions, or different domains. We experiment with Llama 2 and show that the role of some attention heads is overall important, e.g. previous token heads and subword merging heads. Next, we find similarities in Llama 2 behavior when handling tokens of the same part of speech. Finally, we show that some model components can be specialized on domains such as coding or multilingual texts.

4. ACL Demo

   LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models

   Tufnov, Igor, Hambardzumyan, Karen,  Ferrando, Javier, and Voita, Elena

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics 2024

   Abs PDF

   We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previously existing tools that focus on isolated parts of the decision-making process, our framework is designed to make the entire prediction process transparent, and allows tracing back model behavior from the top-layer representation to very fine-grained parts of the model. Specifically, it (1) shows the important part of the whole input-to-output information flow, (2) allows attributing any changes done by a model block to individual attention heads and feed-forward neurons, (3) allows interpreting the functions of those heads or neurons. A crucial part of this pipeline is showing the importance of specific model components at each step. As a result, we are able to look at the roles of model components only in cases where they are important for a prediction. Since knowing which components should be inspected is key for analyzing large models where the number of these components is extremely high, we believe our tool will greatly support the interpretability community both in research settings and in practical applications.

5. ACL Findings

   Neurons in Large Language Models: Dead, N-gram, Positional

   Voita, Elena,  Ferrando, Javier, and Nalmpantis, Christoforos

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics 2024

   Abs PDF

   We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to 66b parameters and rely only on whether an FFN neuron is activated or not. First, we find that the early part of the network is sparse and represents many discrete features. Here, many neurons (more than 70% in some layers of the 66b model) are "dead", i.e. they never activate on a large collection of diverse data. At the same time, many of the alive neurons are reserved for discrete features and act as token and n-gram detectors. Interestingly, their corresponding FFN updates not only promote next token candidates as could be expected, but also explicitly focus on removing the information about triggering them tokens, i.e., current input. To the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream. With scale, models become more sparse in a sense that they have more dead neurons and token detectors. Finally, some neurons are positional: them being activated or not depends largely (or solely) on position and less so (or not at all) on textual data. We find that smaller models have sets of neurons acting as position range indicators while larger models operate in a less explicit manner.

2023

1. WMT

   Automating Behavioral Testing in Machine Translation

   Ferrando, Javier, Sperber, Matthias, Setiawan, Hendra, Telaar, Dominic, and Hasan, Saša

   In Eight Conference on Machine Translation (WMT 2023) 2023

   Abs PDF

   Behavioral testing in NLP allows fine-grained evaluation of systems by examining their linguistic capabilities through the analysis of input-output behavior. Unfortunately, existing work on behavioral testing in Machine Translation (MT) is currently restricted to largely handcrafted tests covering a limited range of capabilities and languages. To address this limitation, we propose to use Large Language Models (LLMs) to generate a diverse set of source sentences tailored to test the behavior of MT models in a range of situations. We can then verify whether the MT model exhibits the expected behavior through matching candidate sets that are also generated using LLMs. Our approach aims to make behavioral testing of MT systems practical while requiring only minimal human effort. In our experiments, we apply our proposed evaluation framework to assess multiple available MT systems, revealing that while in general pass-rates follow the trends observable from traditional accuracy-based metrics, our method was able to uncover several important differences and potential bugs that go unnoticed when relying only on accuracy.

2. ACL

   Explaining How Transformers Use Context to Build Predictions

   Ferrando, Javier, Gállego, Gerard I., Tsiamas, Ioannis, and Costa-jussà, Marta R.

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics 2023

   Abs PDF

   Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.

2022

1. EMNLP

   Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer

   Ferrando, Javier, Gállego, Gerard I., Alastruey, Belen, Escolano, Carlos, and Costa-jussà, Marta R.

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing 2022

   Abs PDF

   In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has focused solely on source sentence tokens attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks complete input token attributions. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour.

2. EMNLP

   Measuring the Mixing of Contextual Information in the Transformer

   Ferrando, Javier, Gállego, Gerard I., and Costa-jussà, Marta R.

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing 2022

   Abs PDF

   The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block – multi-head attention, residual connection, and layer normalization – and define a metric to measure token-to-token interactions within each layer, considering the characteristics of the representation space. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides faithful explanations and outperforms similar aggregation methods.

3. ACL SRW

   On the Locality of Attention in Direct Speech Translation

   Alastruey*, Belen,  Ferrando*, Javier, Gállego, Gerard I., and Costa-jussá, Marta R.

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop May 2022

   Abs PDF

   Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards.

4. AAAI

   Interpreting Gender Bias in Neural Machine Translation: Architecture Matters

   Costa-jussà, Marta R., Escolano, Carlos, Basta, Christine,  Ferrando, Javier, Batlle, Roser, and Kharitonova, Ksenia

   In Proceedings of AAAI May 2022

   Abs PDF

   Multilingual neural machine translation architectures mainly differ in the number of sharing modules and parameters applied among languages. In this paper, and from an algorithmic perspective, we explore whether the chosen architecture, when trained with the same data, influences the level of gender bias. Experiments conducted in three language pairs show that language-specific encoder-decoders exhibit less bias than the shared architecture. We propose two methods for interpreting and studying gender bias in machine translation based on source embeddings and attention. Our analysis shows that, in the language-specific case, the embeddings encode more gender information, and their attention is more diverted. Both behaviors help in mitigating gender bias.

2021

1. EMNLP Findings

   Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions

   Ferrando, Javier, and Costa-jussà, Marta R.

   In Findings of the Association for Computational Linguistics: EMNLP 2021 Nov 2021

   Abs PDF

   This work proposes an extensive analysis of the Transformer architecture in the Neural Machine Translation (NMT) setting. Focusing on the encoder-decoder attention mechanism, we prove that attention weights systematically make alignment errors by relying mainly on uninformative tokens from the source sequence. However, we observe that NMT models assign attention to these tokens to regulate the contribution in the prediction of the two contexts, the source and the prefix of the target sequence. We provide evidence about the influence of wrong alignments on the model behavior, demonstrating that the encoder-decoder attention mechanism is well suited as an interpretability method for NMT. Finally, based on our analysis, we propose methods that largely reduce the word alignment error rate compared to standard induced alignments from attention weights.

2020

1. ICCS

   Improving Accuracy and Speeding Up Document Image Classification Through Parallel Systems

   Ferrando, Javier, Domínguez, Juan Luis, Torres, Jordi, García, Raúl, García, David, Garrido, Daniel, Cortada, Jordi, and Valero, Mateo

   In Computational Science – ICCS 2020 Nov 2020

   Abs PDF

   This paper presents a study showing the benefits of the EfficientNet models compared with heavier Convolutional Neural Networks (CNNs) in the Document Classification task, essential problem in the digitalization process of institutions. We show in the RVL-CDIP dataset that we can improve previous results with a much lighter model and present its transfer learning capabilities on a smaller in-domain dataset such as Tobacco3482. Moreover, we present an ensemble pipeline which is able to boost solely image input by combining image model predictions with the ones generated by BERT model on extracted text by OCR. We also show that the batch size can be effectively increased without hindering its accuracy so that the training process can be sped up by parallelizing throughout multiple GPUs, decreasing the computational time needed. Lastly, we expose the training performance differences between PyTorch and Tensorflow Deep Learning frameworks.