‘A beautiful puzzle’: Looking inside AI models and trying to understand what we see

Thomas Fel, a research fellow at the Kempner Institute, wants to help solve what he describes as one of the most fascinating puzzles of our generation — understanding the inner workings of AI models, and uncovering the mathematical principles that underlie their abilities.

In particular, Fel works on large vision models, advanced AI systems trained on massive amounts of visual data that can interpret images and video and, in many cases, generate new visual content. The models themselves are often described as “black boxes” because, while they can make accurate predictions or decisions, their internal reasoning remains a mystery.

For Fel, however, the idea of a black box doesn’t quite capture the actual nature of the puzzle.

“I think of it more as a glass box,” says Fel. “We can see inside, all the weights and numbers and geometry. We can see it, we just don’t know what it means yet.”

Fel’s research is part of the growing AI subfield of interpretability, which seeks to develop principled, mathematical accounts of how complex AI models encode and represent information internally and arrive at their predictions.

Most of today’s AI vision models are built using neural networks, layered computational systems that form complex representations by learning from large amounts of data. So, when Fel and his colleagues “look inside” a vision model, they’re really trying to understand how its neural network has learned to represent and combine visual features.

“It is a beautiful puzzle,” says Fel.

As he prepares to depart the Kempner for a new role at Goodfire AI, a research company dedicated to interpretability, Fel spoke with us about his last two years of work probing the mathematical structure of large vision models, what he has learned about this puzzle, and where he believes the most important pieces still lie hidden.

The following interview has been edited and condensed.