Mechanistic interpretability, making things up?

Researchers put a dead salmon in an fMRI scanner, showed it pictures of humans, and asked it to guess their emotions. The scan showed brain activity. In a dead fish. Now, the recent study proved that the exact same thing is happening in artificial intelligence. They call it the "Dead Salmons of AI Interpretability" When we use tools like feature attribution, SHAP values, or mechanistic interpretability to see "inside the black box," these methods often generate plausible-looking explanations from completely meaningless computations. You can completely randomize a model's weights and the "saliency map" explanation still looks perfectly valid. You can nudge the data slightly and the entire narrative of why the ai made a decision completely rewrites itself. the researchers call this a "crisis of non-identifiability." the math we use to explain ai behavior is so overdetermined that it is making up stories instead of measuring ground truth. this is a massive reality check for ai alignment and safety.. we are trusting frontier models with critical decisions based on "explanations" that are statistically equivalent to a dead fish having social emotions.. the fix? the authors argue we have to stop treating interpretability like a visualization tool and start treating it as a rigorous statistical inference problem. if your explanation survives a random model check, a label shuffle, and an intervention test.. then maybe you actually understand the model. if not, you're just looking at a dead salmon.

— Superman (@thesupermannx) October 10, 2026