Suppose you are tasked to train a machine learning model. You start with a neural network on more or less the raw data. You are not satisfied with the performance, so you start from scratch. But instead of going with your first approach, you try some feature engineering, use some feature filtering and train an xgboost model. With some disappointment you note that the performance of the model is very similar to your first neural network approach. But you are also curious. How did you end up with such similar performances for different modeling approaches? Feature sets are overlapping but not the same and a glance at feature importances and feature effects shows you that the models make decisions for different reasons.
How can both models be correct at the same time? As if there were multiple parallel universes with different events, but where the result (the model performance) is the same.
A set of models which all perform roughly equally well is called a Rashomon set. The idea was introduced by the paper All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously by Aaron Fisher, Cynthia Rudin and Francesca Dominici and yes, that’s the title of the paper and not the abstract.
Does the existence of a Rashomon set pose a problem? Is a Rashomon set a bazaar from which practitioners can choose the models they like? The one that will pass the audit? The one that sits well with the persons prior believes? The one that confirms the researchers hypothesis? The one a customer cannot legally attack? Rashomon Sets give some freedom to the practitioner that might be exploited in a harmful way.
When you look at the feature importances and feature effects of a machine learning model, you might realize that the model is flawed. Maybe you used a feature that you should not have used. Or there was, by accident, a little bit of target leakage in one of the features. Or you realize that a model that uses features A and B might be targetable by adversarial attacks.
We usually have a lot of implicit knowledge about the prediction task, but we often don’t know how to encode this knowledge in our model. A Rashomon set gives you the option to choose a model that is more robust. But you might only know it when you see it.
In reality, of course, models in a Rashomon sets might not differ too wildly. There might be some ambiguity when it comes to correlated features, where you can choose between A and B. I have not seen too crazy things to happen, like that a feature effect is reversed.
- Predictive Multiplicity in Classification by Charles T. Marx, Flavio du Pin Calmon and Berk Ustun
- A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning by Lesia Semenova and Cynthia Rudin