publications

2024

1. 

   Foundations of Machine Learning Interpretability

   Gianluigi Lopardo

   Université Côte d’Azur, 2024

   Abs Bib HTML PDF

   The rising use of complex Machine Learning (ML) models, especially in critical applications, has highlighted the urgent need for interpretability methods. Despite the variety of solutions proposed to explain automated algorithmic decisions, understanding their decision-making process remains a challenge. This manuscript investigates the interpretability of ML models, using mathematical analysis and empirical evaluation to compare existing methods and propose novel solutions.Our main focus is on post-hoc interpretability methods, which provide insights into the decision-making process of ML models post-training, independent of specific model architectures. We delve into Natural Language Processing (NLP), exploring techniques for explaining text models.We address a key challenge: interpretability methods can yield varied explanations even for simple models. This highlights a critical issue: the absence of a robust theoretical foundation for these methods. To address this issue, we use a rigorous theoretical framework to formally analyze existing interpretability techniques, assessing their behavior and limitations.Building on this, we propose a novel explainer to provide a more faithful and robust approach to interpreting text data models. We also engage with the debate on the effectiveness of attention weights as explanatory tools within powerful transformer architectures.Through this analysis, we expose the strengths and limitations of existing interpretability methods and pave the way for more reliable, theoretically grounded approaches. This will lead to a deeper understanding of how complex models make decisions, fostering trust and responsible deployment in critical ML applications.

   @phdthesis{lopardo2024foundation,
     title = {{Foundations of Machine Learning Interpretability}},
     author = {Lopardo, Gianluigi},
     year = {2024},
     school = {Université Côte d'Azur},
   }

2. 

   Attention Meets Post-hoc Interpretability: A Mathematical Perspective

   Gianluigi Lopardo, Frederic Precioso, and Damien Garreau

   In International Conference on Machine Learning (ICML), 2024

   Abs Bib HTML PDF Code

   Attention-based architectures, in particular transformers, are at the heart of a technological revolution. Interestingly, in addition to helping obtain state-of-the-art results on a wide range of applications, the attention mechanism intrinsically provides meaningful insights on the internal behavior of the model. Can these insights be used as explanations? Debate rages on. In this paper, we mathematically study a simple attention-based architecture and pinpoint the differences between post-hoc and attention-based explanations. We show that they provide quite different results, and that, despite their limitations, post-hoc methods are capable of capturing more useful insights than merely examining the attention weights.

   @inproceedings{lopardo2024attention,
     title = {{Attention Meets Post-hoc Interpretability: A Mathematical Perspective}},
     author = {Lopardo, Gianluigi and Precioso, Frederic and Garreau, Damien},
     year = {2024},
     booktitle = {{International Conference on Machine Learning (ICML)}},
     organization = {PMLR},
   }

2023

1. 

   A Sea of Words: An In-Depth Analysis of Anchors for Text Data

   Gianluigi Lopardo, Frederic Precioso, and Damien Garreau

   In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023

   Abs Bib HTML PDF Code

   Anchors (Ribeiro et al., 2018) is a post-hoc, rule-based interpretability method. For text data, it proposes to explain a decision by highlighting a small set of words (an anchor) such that the model to explain has similar outputs when they are present in a document. In this paper, we present the first theoretical analysis of Anchors, considering that the search for the best anchor is exhaustive. After formalizing the algorithm for text classification, we present explicit results on different classes of models when the vectorization step is TF-IDF, and words are replaced by a fixed out-of-dictionary token when removed. Our inquiry covers models such as elementary if-then rules and linear classifiers. We then leverage this analysis to gain insights on the behavior of Anchors for any differentiable classifiers. For neural networks, we empirically show that the words corresponding to the highest partial derivatives of the model with respect to the input, reweighted by the inverse document frequencies, are selected by Anchors.

   @inproceedings{lopardo2022anchors,
     title = {{A Sea of Words: An In-Depth Analysis of Anchors for Text Data}},
     author = {Lopardo, Gianluigi and Precioso, Frederic and Garreau, Damien},
     year = {2023},
     booktitle = {{International Conference on Artificial Intelligence and Statistics (AISTATS)}},
   }

2. 

   Faithful and Robust Local Interpretability for Textual Predictions

   Gianluigi Lopardo, Frederic Precioso, and Damien Garreau

   arXiv preprint arXiv:2311.01605, 2023

   Abs Bib HTML PDF Code

   Interpretability is essential for machine learning models to be trusted and deployed in critical domains. However, existing methods for interpreting text models are often complex, lack mathematical foundations, and their performance is not guaranteed. In this paper, we propose FRED (Faithful and Robust Explainer for textual Documents), a novel method for interpreting predictions over text. FRED offers three key insights to explain a model prediction: (1) it identifies the minimal set of words in a document whose removal has the strongest influence on the prediction, (2) it assigns an importance score to each token, reflecting its influence on the model’s output, and (3) it provides counterfactual explanations by generating examples similar to the original document, but leading to a different prediction. We establish the reliability of FRED through formal definitions and theoretical analyses on interpretable classifiers. Additionally, our empirical evaluation against state-of-the-art methods demonstrates the effectiveness of FRED in providing insights into text models.

   @article{lopardo2023fred,
     title = {{Faithful and Robust Local Interpretability for Textual Predictions}},
     author = {Lopardo, Gianluigi and Precioso, Frederic and Garreau, Damien},
     year = {2023},
     journal = {arXiv preprint arXiv:2311.01605},
   }

3. 

   Understanding Post-hoc Explainers: The Case of Anchors

   Gianluigi Lopardo, Frederic Precioso, and Damien Garreau

   54es Journées de Statistique, 2023

   Abs Bib HTML PDF

   In many scenarios, the interpretability of machine learning models is a highly required but difficult task. To explain the individual predictions of such models, local model-agnostic approaches have been proposed. However, the process generating the explanations can be, for a user, as mysterious as the prediction to be explained. Furthermore, interpretability methods frequently lack theoretical guarantees, and their behavior on simple models is frequently unknown. While it is difficult, if not impossible, to ensure that an explainer behaves as expected on a cutting-edge model, we can at least ensure that everything works on simple, already interpretable models. In this paper, we present a theoretical analysis of Anchors (Ribeiro et al., 2018): a popular rule-based interpretability method that highlights a small set of words to explain a text classifier’s decision. After formalizing its algorithm and providing useful insights, we demonstrate mathematically that Anchors produces meaningful results when used with linear text classifiers on top of a TF-IDF vectorization. We believe that our analysis framework can aid in the development of new explainability methods based on solid theoretical foundations.

   @article{lopardo2023understanding,
     title = {Understanding Post-hoc Explainers: The Case of Anchors},
     author = {Lopardo, Gianluigi and Precioso, Frederic and Garreau, Damien},
     journal = {54es Journées de Statistique},
     year = {2023},
   }

2022

1. 

   Comparing Feature Importance and Rule Extraction for Interpretability on Text Data

   Gianluigi Lopardo, and Damien Garreau

   In 2-nd Workshop on Explainable and Ethical AI-26TH International Conference on Pattern Recognition (XAIE @ ICPR), 2022

   Abs Bib HTML PDF Code

   Complex machine learning algorithms are used more and more often in critical tasks involving text data, leading to the development of interpretability methods. Among local methods, two families have emerged: those computing importance scores for each feature and those extracting simple logical rules. In this paper we show that using different methods can lead to unexpectedly different explanations, even when applied to simple models for which we would expect qualitative coincidence. To quantify this effect, we propose a new approach to compare explanations produced by different methods.

   @inproceedings{lopardo2022comparing,
     title = {{Comparing Feature Importance and Rule Extraction for Interpretability on Text Data}},
     author = {Lopardo, Gianluigi and Garreau, Damien},
     year = {2022},
     booktitle = {{2-nd Workshop on Explainable and Ethical AI-26TH International Conference on Pattern Recognition (XAIE @ ICPR)}},
   }

2. 

   SMACE: A New Method for the Interpretability of Composite Decision Systems

   Gianluigi Lopardo, Damien Garreau, Frederic Precioso, and 1 more author

   In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD), 2022

   Abs Bib HTML PDF Code

   Interpretability is a pressing issue for decision systems. Many post hoc methods have been proposed to explain the predictions of a single machine learning model. However, business processes and decision systems are rarely centered around a unique model. These systems combine multiple models that produce key predictions, and then apply rules to generate the final decision. To explain such decisions, we propose the Semi-Model-Agnostic Contextual Explainer (SMACE), a new interpretability method that combines a geometric approach for decision rules with existing interpretability methods for machine learning models to generate an intuitive feature ranking tailored to the end user. We show that established model-agnostic approaches produce poor results on tabular data in this setting, in particular giving the same importance to several features, whereas SMACE can rank them in a meaningful way.

   @inproceedings{lopardo2022smace,
     title = {{SMACE: A New Method for the Interpretability of Composite Decision Systems}},
     author = {Lopardo, Gianluigi and Garreau, Damien and Precioso, Frederic and Ottosson, Greger},
     year = {2022},
     booktitle = {Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD)},
     pages = {325--339},
     organization = {Springer},
   }

2021

1. 

   Explainable AI for business decision-making

   Gianluigi Lopardo

   Politecnico di Torino, 2021

   Abs Bib HTML PDF Code

   Machine Learning is increasingly being leveraged in business processes to make automated decisions. Nevertheless, a decision is rarely made by a standalone machine learning model, but is rather the result of an orchestration of predictive models, each predicting key quantities for the problem at hand, which are then combined through decision rules to produce the final decision. For example, a mobile phone company aiming to reduce customer churn would use machine learning to predict churn risk and rank potential retention offers, and then apply eligibility rules and other policies to decide whether a retention offer is worth proposing to a certain customer and, if so, which one. Applying decision rules on top of machine learning-based predictions or classifications is typically performed by companies to deliver better conformance, adaptability, and transparency. Interpretability is a pressing question in these situations. In the example above, it is fundamental for the sales representative to know, even roughly, why a decision was made. While the field of interpretable machine learning is full of open challenges in itself, when trying to explain a decision that relies on both business rules and multiple machine learning models, a number of additional challenges arise. First, the business rules surrounding the models represent non-linearities that cause problems for attribution-based interpretability methods like LIME and SHAP. Second, the already transparent business rules represent knowledge that unless exploited will causeproblems for sampling-based explanation methods. Third, machine learning models with overlapping features will produce conflicting explanation weights. As a result, applying current methods to these real-world decision systems produce unreliable and brittle explanations. In this configuration, there is knowledge that we can exploit to make our explanations processaware. We know which variables are involved in the decision policy and we know its rules. It is worth to exploit this information instead of treating the whole system as a black-box and being completely model-agnostic. In this thesis, we present SMACE - Semi-Model-Agnostic Contextual Explainer, a new interpretability method that combines a geometric approach (for business rules) with existing interpretability solutions (for machine learning models) to generate feature importance based explanations. Specifically, SMACE provides two levels of explanation, for the different users involved in the decision-making process. The first level, which is useful for the business user, must provide a ranking of importance for all the variables used, whether they are input attributes or values calculated in-house. This is useful, for example, to the sales representative, who has access to and knowledge of company policies. By interpreting the process, the business user can explain, modify, override or validate the specific decision. The second level is necessary for the end customer. She does not have access to the internal policy rules, nor to the way in which decision-making processes are managed. It therefore requires explanations based solely on information that she is aware of, i.e., input features, such as her personal details or service usage values. We show that while LIME and SHAP produce poor results when applied to such a decision system, SMACE provides intuitive feature ranking, tailored to business needs.

   @mastersthesis{lopardo2021xai,
     title = {{Explainable AI for business decision-making}},
     author = {Lopardo, Gianluigi},
     year = {2021},
     school = {Politecnico di Torino},
   }