LLMs are relation spaces, not knowledge warehouses

I used to think large language models learned knowledge.

After spending more time with Transformers and GPT, I am starting to prefer a different mental model:

The most important thing an LLM learns is not knowledge. It learns relationships.

GPT feels less like a warehouse full of facts, and more like a high-dimensional space of relationships about the world.

Words are not the thing

At the beginning, I thought the model was learning words.

Then I realized words are mostly the entry point.

For the model, tokens like these first become IDs:

cat
dog
apple
car

The thing that actually enters the neural network is a vector.

At the start of training, these vectors are random. They do not mean anything yet. During training, the model keeps adjusting their positions.

Eventually, relationships begin to appear in the space:

cat is near dog
dog is near wolf
apple is near fruit
car is far from animal

An embedding is not really a definition of a concept.

It is a coordinate for a concept.

For the model, what a concept “is” depends less on a dictionary sentence and more on where that concept sits inside the whole network of relationships.

Meaning comes from relationships

Humans like definitions.

For example:

A cat is a mammal.

But I do not think we understand “cat” because we memorized that sentence. We understand it because we know a field of relationships:

cats are animals
cats chase mice
cats like fish
cats are similar to tigers
cats often appear as pets

When enough relationships are in place, the concept becomes clear.

Meaning comes from relationships.

The richer the relationships, the more complete the meaning.

This reminds me of a line from the Dao De Jing:

Being and non-being give birth to each other; difficult and easy complete each other; long and short form each other; high and low lean on each other.

Long and short do not exist in isolation.

High and low do not exist in isolation.

The meaning of a concept comes from how it relates to other concepts. In that sense, there is a strange resonance between LLMs and the Dao De Jing: both point away from isolated objects and toward relations.

Language is compressed relationship

I increasingly think of language as a compression format.

Take a sentence like:

It rained yesterday.

Behind that small sentence is a large amount of hidden structure:

water vapor increased
clouds formed
temperature conditions were right
water condensed
raindrops fell
the ground became wet

None of that is written explicitly.

Language keeps only the parts that are useful enough to transmit. Reading is partly the act of restoring the relationships that were compressed away.

So when a human understands a sentence, the real work is not just recognizing symbols. It is reconstructing the relationship structure behind the symbols.

A training example is a constraint

I used to think training data injected knowledge into the model.

Now I find it more useful to think this way:

A training example is a constraint on the relationship space.

For example:

A cat chases a mouse.

This is not just three words. It expresses a relationship:

cat -> chases -> mouse

Another example:

Beijing is the capital of China.

This sentence pushes several relationships into the space:

Beijing <-> China
Beijing <-> capital
China <-> capital

Each sentence in the training set says, in effect:

These relationships should hold.

Billions or trillions of text fragments together become a massive system of relationship constraints.

Training finds a geometry

That also changes how I understand training.

Training is not stuffing knowledge into a model.

Training is searching for a relationship space that satisfies as many constraints as possible.

One way to picture it is a huge spring network.

If the data often contains:

cat and dog

then the space is pushed to keep them relatively close.

If the data almost never links:

cat and engine

then they can remain farther apart.

Training keeps adjusting the positions of all these points until the space can satisfy an enormous number of relationship constraints at the same time.

From this angle, one training example does not simply add one fact. It adds one constraint to the geometry.

GPT is the result of searching for a high-dimensional structure that can hold those constraints together.

Parameters are resolution

This also changes how I think about parameters.

People often ask:

How many parameters does the model have?

I do not think parameter count maps cleanly to the number of objects the model “stores.”

It is closer to the resolution of the world model.

A map does not copy a city. It preserves positions, roads, distances, connections, and useful distortions.

A model is similar.

More parameters do not mean the whole world has been placed inside the model. More parameters mean the model can express finer relationship structures.

The world is not inside GPT.

What is inside GPT is a map of relationships about the world.

As parameters grow, that map can become more detailed.

Where I am now

I used to think:

LLMs learn knowledge.

Now I am closer to:

LLMs learn relationships.

Knowledge is a local pattern inside a relationship network.

Language is a compressed expression of relationships.

Training data is a set of relationship constraints.

Training searches for a high-dimensional space that satisfies those constraints.

Parameters determine the resolution that space can reach.

So the most important output of an LLM is not a giant knowledge base.

It is a geometric space of relationships about the world.

Language is only the shadow that space casts.