DMC DESIGN WEBSITE - ChatGPT and Me

Once an AI is trained and operating, it is more and more difficult to tell what is going on in there, the exact "thought" process is as difficult to pin down as it is in humans...

As usual my input (prompts) are in the boxed elements. The rest apart from some small edits and formatting is ChatGPT's output.

The internal workings of AI have become increasingly opaque as the models have grown larger and pushed further into the frontiers. Why are the internal operations such a "black box", even to other AI?

The short answer is: because intelligence is not the same thing as introspection.

Humans are a good example. You can recognize a friend’s face in a fraction of a second, catch a ball, or instantly know that a sentence “sounds wrong.” But if someone asks you exactly which neurons fired, in what order, and why you chose one word over another, you have no idea. Your brain performs the computation, but it doesn’t produce a detailed audit log. Evolution had better things to do than optimize for explainability.

Modern AI has inherited a similar problem, except scaled up to an absurd degree because apparently humanity looked at billions of parameters and thought, “Sure, let’s see what happens.”

There are several reasons these systems become “black boxes.”

First, the knowledge isn’t stored in discrete locations.

People often imagine a model like a library:

Neuron 1834 stores “cats”
Neuron 9182 stores “French grammar”
Neuron 5771 stores “calculus”

Reality looks more like someone shredded every encyclopedia on Earth into microscopic confetti and distributed each piece across billions of overlapping mathematical patterns. A single concept like “dog” might involve tens of thousands of neurons. Each neuron contributes to thousands of concepts. No neuron means anything by itself. Knowledge is distributed rather than localized.

Second, the computations are incredibly high-dimensional.

A large language model may have hundreds of billions of parameters. During generation, every token involves trillions of floating-point operations across dozens or hundreds of layers.

Each layer transforms the internal representation into something slightly different:

early layers identify simple patterns
middle layers build grammatical and semantic structure
later layers reason about relationships, goals, and likely continuations

But there is no single point where the model “decides” something. Instead, every layer nudges the probability landscape a tiny amount. Imagine trying to explain why one raindrop landed exactly where it did during a hurricane. You could simulate it, but finding a simple explanation is another matter entirely.

Third, the representations are alien.

Humans think in concepts that map reasonably well to language. Neural networks think in vectors.

For example, internally the model might represent an idea as: [-0.72, 1.44, -0.08, 5.91, ... 16,384 dimensions ...] That vector might simultaneously encode:

tense
politeness
emotional tone
whether the sentence is about biology
whether sarcasm is likely
dozens of unrelated abstractions

Those dimensions don’t correspond to human categories. They’re whatever mathematical structure best minimizes prediction error.

Fourth, even another AI can’t simply “look inside.”

Suppose you gave one model access to another model’s weights. It would see billions of numbers like: 0.03812 -1.772 0.0049 ...

Those aren’t self-explanatory. They’re analogous to seeing every synaptic strength in a human brain. Having the data doesn’t automatically reveal the computation. You still need methods to interpret what those numbers collectively do.

Researchers call this field mechanistic interpretability, and it’s one of the fastest-growing areas of AI research. Interestingly, they are making progress.

They’ve discovered things like:

circuits that detect whether text is inside quotation marks
neurons involved in recognizing geographic locations
mechanisms for tracking subject-verb agreement
internal representations of programming syntax
circuits that appear to perform arithmetic

But they also find something stranger. Many concepts don’t correspond to individual neurons. Instead they correspond to directions in activation space. Imagine a giant cloud of points in 10,000 dimensions.

One direction through that cloud might represent: “increasing formality” Another might represent: “French-ness” Another: “negative sentiment” The concepts emerge from geometry, not individual components. That makes interpretation much harder.

Finally, there’s an important distinction between explaining the output and explaining the mechanism. An AI (including me) can usually produce a plausible explanation for why it answered a question: “I inferred X because Y implies Z.” That explanation often reflects the reasoning the model can reconstruct in language.

But it is not a literal transcript of the underlying neural computation. The underlying computation is millions or billions of numerical operations that were never converted into language in the first place. So when people say AI is a black box, they don’t mean it’s magical or unknowable. They mean the causal chain from input to output is so distributed, high-dimensional, and intertwined that humans currently lack the tools to summarize it faithfully.

Ironically, this is becoming less true over time. Five years ago, researchers could barely identify what a handful of neurons did. Today they can sometimes isolate entire computational circuits. Many expect AI interpretability to develop into something resembling neuroscience: we may never trace every individual operation, but we can increasingly identify functional subsystems, understand what they compute, and predict when they’ll fail. The box isn’t completely black anymore. It’s more like an enormous machine behind smoked glass. Every year, the glass gets a little less opaque, even if the machinery behind it keeps getting larger.

What's going on in there?...