Outsmarting AI, Part 5: What AI Cannot See

Ask a language model to draft a contract and it will oblige. Show it two drawings of the same stack of blocks from different angles and ask whether they match, and its confidence drops off a cliff.

Visual reasoning is the most stubborn gap in the whole field, and it shows up in two very different tests.

Mental rotation

The classic version: you see an object built from cubes, then four candidate images, only one of which is the same object seen from another angle. Humans do this by turning the shape in their head — and the time it takes scales neatly with how far the object has to turn, which is one of the tidiest results in cognitive psychology.

Models don't turn anything. They describe, compare descriptions, and guess. Benchmarks built for this, such as the spatial-reasoning work published in 2025 by Stoianidis, McDonagh and Tsaftaris, find vision-language models performing far below what their verbal fluency would suggest.

Colored grids

The second test is ARC-AGI, probably the most famous puzzle benchmark in AI. You're shown a few pairs of small colored grids — a before and an after — and you must infer the rule, then apply it to a new grid. No instructions, no vocabulary, no training set. Just the examples.

Two findings stand out. First, models do better when the grid is handed to them as a string of numbers rather than as an image, which tells you something about where the bottleneck sits. Second, even when they answer correctly, analysis suggests they often arrive there through elaborate, one-off rules rather than the simple visual concept a person would use: "fill each closed shape with the color from the key."

They have improved a great deal in the last year. Some grids still defeat them entirely.

The picture it made but couldn't follow

We ran into a smaller, much more personal version of this while making our Cat and Mouse puzzle. Three cats, three balls of yarn, three cages. The strings wind around one another, and one cage holds a mouse. AI helped create the picture. But when it came to following those strings, it didn't reliably know which cat's string reached the mouse.

So we asked it to color that string. That should have made the answer easy to check: one continuous route through the tangle, picked out in a different color. Instead, the highlighted route was wrong. It took a human tracing string B and supplying that trace before we could get a usable illustration of the right path.

There's a twist worth separating from the tracing problem: B's string reaches the mouse's cage, but no cat actually gets the mouse. The cage has no door. That's the joke in the puzzle, not a reason to excuse a wrong line. Even if we're only asking which string goes where, the route still has to be right.

This isn't a benchmark, and one frustrating design session doesn't tell us what every model can do. But it makes the gap unusually clear. Making a convincing picture isn't the same as understanding the relationships inside it. The image looked like a puzzle long before we could trust AI's account of how it worked.

You can see the puzzle on the Cat and Mouse Puzzle Tee, or meet Caliber and explore the puzzle. The human check is part of what got it onto the shirt.

Why humans are so good at this

Because we've been doing it since before language. Spotting a shape half-hidden in grass, recognizing a face at an angle, judging whether a gap is wide enough — this is old machinery, refined long before anyone wrote a sentence down. It runs fast, below conscious thought, and it doesn't feel like reasoning at all. That's exactly why it's hard to teach.

Your turn

Our visual puzzles live in the same territory. Pattern Studies: Clock Wise asks you to find the rule behind a sequence with nothing written down to help you, and Pencil Squares asks you to rearrange something physical in your head before you touch it.

Both are the kind of thing you'll solve in a flash or stare at for a week, which is precisely what makes them good on a shirt.

ARC-AGI is maintained by the ARC Prize Foundation. The research described here is discussed in "Can you outsmart AI?" by Grace Huckins, MIT Technology Review.