Sitemap
Medium Logo
Welcome Offer
Access to everything. Now 30% off.
Upgrade now

I Stopped Watching Tutorials and Built a Transformer

5 min read5 days ago

So, in my last blog, I started building my own language model.

Got the dataset, messed around with tokenization, nearly murdered my laptop’s RAM, and somehow made it to batching.

Now came the interesting part.

Actually building the model.

I thought I’d watch Andrej Karpathy explain the Transformer, understand everything, and then write the code.

Yeah, about that.

The Tutorial Trap

I started watching Karpathy’s GPT video, and man, there was a lot going on.

One minute we were dealing with embeddings. Next came matrix multiplication, attention heads, masking, normalization, residual connections…

And somewhere in between, my brain stopped processing information.
Then I discovered his makemore series.
Four more videos. Almost eight hours of content.
I genuinely considered watching all of them before touching the code.
But then I realized I was getting dangerously close to tutorial hell.
I’d already spent enough time learning about building an LLM.

It was time to build the damn thing.

B, T, C. What the Hell?

The first thing I kept encountering was this:

[B, T, C]

Everywhere.

Eventually, I understood:

  • B = Batch size
  • T = Number of tokens in a sequence
  • C = Embedding dimension

My batching pipeline was already producing [B, T].
But those were just token IDs. Integers.
A neural network needs useful vector representations, not just arbitrary identifiers.

So I added:

self.token_embedding = nn.Embedding(vocab_size, d_model)
self.position_embedding = nn.Embedding(context_length, d_model)

And suddenly:

Token IDs           [B, T]
↓
Token Embeddings [B, T, C]
+
Position Embeddings [T, C]
↓
[B, T, C]

BTC finally made sense.
One mystery down.
Unfortunately, that was the easy part.

Query, Key, Value. Especially Value.

Next came self-attention.

I’d heard about attention countless times. I’d worked with LLMs, used them in applications, and knew Transformers relied on it.

But knowing that attention exists and understanding what it actually does are two very different things.

I started with three linear layers.

self.query = nn.Linear(d_model, head_size, bias=False)
self.key = nn.Linear(d_model, head_size, bias=False)
self.value = nn.Linear(d_model, head_size, bias=False)

Simple enough. Except I didn’t really understand what was happening.

Query — what a token is looking for.
Key — what a token can be matched against.
And Value?

I genuinely couldn’t wrap my head around Value.

If Query and Key are already comparing tokens, what exactly is Value doing?
The explanation that finally helped was thinking of Query and Key as a routing system.

Query asks where to look.
Key helps determine which positions are relevant.
Value is the information that actually gets collected.

That distinction was important.

Because the attention formula stopped looking like random matrix multiplication.

Q @ Kᵀ
↓
Attention scores
↓
Scale
↓
Mask
↓
Softmax
↓
Attention weights @ V
↓
Output

Q and K decide how much attention each position receives.
V supplies the information being combined.
And one more detail clicked: Q @ Kᵀ produces a [B, T, T] matrix.
That’s token-to-token attention.
Not [B, T, C].

A matrix describing how positions relate to other positions.

Okay. Now we were getting somewhere.

The Model Must Not Cheat

I was building a decoder-only Transformer.

Meaning each token can use what came before it, but not what comes after it. Without a causal mask, the model could look at future tokens during training. And that rather defeats next-token prediction.

So I created a lower-triangular mask.

torch.tril(
torch.ones(context_length, context_length, dtype=torch.bool)
)

Conceptually:

Token 1  Token 2  Token 3  Token 4
Token 1    ✓        ✗        ✗        ✗
Token 2 ✓ ✓ ✗ ✗
Token 3 ✓ ✓ ✓ ✗
Token 4 ✓ ✓ ✓ ✓

Future positions get attention scores of negative infinity.

Softmax turns their probabilities into zero.

The First Attention Head Actually Worked

After enough debugging, I finally had the complete flow:

scores = query @ key.transpose(-2, -1)
scaled_scores = scores / (self.head_size ** 0.5)
masked_scores = scaled_scores.masked_fill(
~mask,
float("-inf")
)
attention_weights = torch.softmax(masked_scores, dim=-1)
output = attention_weights @ value

But I didn’t want to declare victory just because Python stopped throwing errors.

So I checked the output dimensions.

Write on Medium

Checked for non-finite values.

Checked that changing future tokens couldn’t affect earlier outputs.

Checked gradients for Query, Key, and Value.

And finally:

Output shape: torch.Size([2, 4, 2])
All output values are finite
Future token cannot affect earlier tokens
query.weight: gradient OK
key.weight: gradient OK
value.weight: gradient OK
Single-head attention passed all checks

That was satisfying.

Not because I’d written some extraordinary algorithm.

But because, for the first time, I could follow the whole operation and explain why each part existed.

I’d built causal self-attention.

And I understood what it was doing.

Apparently We Need Four of Those

Single-head attention was done.

Then came multi-head attention.

The idea was to give the model multiple sets of learned Q, K, and V projections so it could learn different attention patterns.

For Axiom:

d_model = 256
n_heads = 4
head_size = 64

Four attention heads, each producing a 64-dimensional output.

Concatenate them, and we’re back to 256 dimensions.

There was also a final learnable output projection.

At this point, the concepts were getting easier.

Not because the code had become magically simple, but because the tensor shapes had started making sense.

I could finally look at:

[B, T, 256] → [B, T, 64]

The Rest of the Transformer

After attention, I built the MLP.

Linear: 256 → 1024
↓
GELU
↓
Linear: 1024 → 256

Then came LayerNorm and residual connections.

The Transformer block used pre-normalization:

x = x + self.attention(self.ln1(x))
x = x + self.mlp(self.ln2(x))

Attention lets tokens exchange information.

The MLP transforms each position’s representation.

LayerNorm helps stabilize the computations.

Residual connections let the original representation flow around those transformations.

Put them together, and you have a Transformer block.

Stack four of those.

Add final LayerNorm.

Add a language-model head that maps the 256-dimensional representation into 8,000 vocabulary scores.

And suddenly, I had this:

Token IDs
↓
Token + Position Embeddings
↓
Transformer Block 1
↓
Transformer Block 2
↓
Transformer Block 3
↓
Transformer Block 4
↓
Final LayerNorm
↓
Language Model Head
↓
Vocabulary Logits

My own decoder-only Transformer.

I checked the output shapes, gradients, and the full forward pass.

Everything connected.

What Actually Changed?

I started this project thinking I needed to understand every mathematical detail before writing code.

But that wasn’t working.

What worked was building one component at a time, getting confused, understanding what was wrong, and moving forward.

And I learned something else.

Using an LLM to help build an LLM is easy.

Getting an LLM to explain something until you understand it, then implementing it yourself, is a completely different exercise.

That was the point of Axiom.

Not to write the most impressive Transformer implementation.

But to stop treating these architectures like black boxes.

By the end of this, the model architecture was finished.

I had embeddings, attention, Transformer blocks, and vocabulary logits.

There was just one tiny problem.

The model hadn’t learned anything yet.

It had millions of randomly initialized parameters and absolutely no idea how to write a story.

So naturally, my next step was to train the thing on a laptop GPU with 4GB VRAM.

What could possibly go wrong?