60,000 Training Steps, a Loss Plateau, and a Simple Fix
So, in the previous blog, I finally finished building my Transformer.
Four attention heads. Four Transformer blocks. Token embeddings, positional embeddings, LayerNorm, residual connections, MLP, the whole thing.
Around 7.55 million parameters.
Everything was connected. The tensor dimensions matched. Gradients were flowing.
I had built a language model.
Well, technically.
Because at that point, it was just a bunch of randomly initialized weights.
It couldn’t write a story. It couldn’t predict anything useful.
Basically, it was stupid.
And now I had to make it slightly less stupid.
Training an LLM on a Potato
My laptop has an NVIDIA GTX 1650 with 4GB VRAM.
Four.
Gigabytes.
Now, Axiom is a tiny model compared to today’s LLMs, so this wasn’t some impossible task.
But memory was still something I had to keep in mind.
I had already managed to exhaust my system RAM and swap during tokenization earlier in the project. Didn’t particularly want a repeat of that experience.
So I kept the model small.
vocab_size = 8000
context_length = 128
d_model = 256
n_heads = 4
n_layers = 4
batch_size = 8The training pipeline was straightforward.
TinyStories
↓
Custom BPE Tokenizer
↓
Token IDs
↓
Random Batches
↓
Axiom
↓
Predictions
↓
Cross-Entropy Loss
↓
Backpropagation
↓
AdamW
↓
RepeatI’d already spent enough time understanding forward passes.
Now I had to make the backward pass actually teach the model something.
Making the Model Learn
The training objective was next-token prediction.
Say we have:
Input: [10, 20, 30, 40]
Target: [20, 30, 40, 50]For every position, the model predicts what token comes next.
Cross-entropy measures how wrong those predictions are.
Backpropagation calculates how the parameters contributed to that error.
AdamW updates them.
And we do this again. And again.And again.
The code isn’t the particularly interesting part here.
What mattered was finally understanding how all the components I’d built came together during learning.
Query, Key, and Value weren’t hand-designed rules.
The attention projections were learnable.
The embedding table was learnable.
The MLP weights were learnable.
All the parameters I’d initialized while building the Transformer were now being updated based on prediction errors.
That was pretty cool.
First, the Loss Started Dropping
I started training Axiom using AdamW and a learning rate of 3e-4.
Initially, the loss went down.
Which was good.
At least something was happening.
But watching a number decrease in the terminal doesn’t tell you everything.
I wanted to know whether the model had actually learned enough to produce text.
So I also implemented an autoregressive generator.
The model predicts a token, we sample it, append it to the sequence, and feed the updated sequence back into the model.
Repeat until generation stops.
I added temperature sampling and top-k sampling so generation wasn’t just picking the most likely token every single time.
And eventually…
The thing started writing stories.
One of the generated outputs included:
The little girl was so happy and thanked the fairy for her help. She flew away.
That was my model.
My little Transformer, trained on my laptop, producing an actual sentence that made sense.
Was it intelligent? No.
Was it writing groundbreaking literature? Absolutely not.
Did it sometimes repeat itself, restart stories, or produce weird characters?
Of course.
But it had learned enough structure from TinyStories to generate recognizable story-like text.
That’s a very different feeling from calling a model through an API.
I knew exactly what components were producing those tokens because I’d written them.
50,000 Steps Later…
I let Axiom train through a 50,000-step run, including an overnight session.
And by that point, something became obvious.
The loss wasn’t improving much anymore.
Training and validation loss were hovering around 2.20, with fluctuations.
Now, to be clear, a fluctuating loss isn’t automatically a problem. Especially when validation is estimated using sampled batches.
But the broader trend had slowed.
The model was learning, but not making the kind of progress I was hoping for.
The first reaction is obvious.
Do I need a bigger model?
More layers?
More data?
Have I messed something up in the architecture?
But instead of changing everything, we looked at the training configuration.
And there was one thing I’d kept constant.
The learning rate.
The Learning Rate Problem
I’d been training with:
learning_rate = 3e-4For the entire 50,000-step run.
Here’s the thing.
A learning rate controls the size of the optimizer’s parameter updates.
A larger learning rate can help make faster progress early in training.
But later, when the model is already in a better region of the loss landscape, smaller updates may help it settle into a better solution.
That doesn’t mean a constant learning rate can never work.
It just means my current training run had reached a point where reducing it was worth trying.
So instead of starting over, I resumed the checkpoint and lowered it.
Steps 0–50,000
Learning rate: 3e-4
Validation loss: ~2.20Steps 50,000–55,000
Learning rate: 1e-4
Validation loss: ~2.10Steps 55,000–60,000
Learning rate: 3e-5
Validation loss: ~2.09
And the loss improved.
Not by some ridiculous amount.
But enough to show that the model still had room to improve under a different optimization setting.
That was an important lesson.
Sometimes the next improvement isn’t another layer of attention. It’s changing how you train the model you already have.
The Numbers Weren’t Always Consistent
Another thing I ran into was validation loss fluctuation.
At different points, I saw readings around 2.0175, 2.02, and 2.09.
And naturally, I wondered why I couldn’t just use the lowest number.
But my validation procedure used sampled batches.
Different samples can produce different results.
So taking the single lowest reading would make the model look better than a typical evaluation supports.
That’s why I settled on reporting an approximate validation loss of 2.09 for Axiom v1.
Not a standardized benchmark.
Not a carefully controlled comparison against other models.
Just an honest measurement from my training setup.
And that distinction matters, especially if I’m going to use this model for research experiments.
Because if I want to compare a future optimization against Axiom v1, I need to know what my baseline actually means.
The Final 60,000-Step Model
Eventually, I reached 60,000 training steps.
I now had a model that could generate TinyStories-style text.
A custom tokenizer.
A complete Transformer implementation.
A training pipeline with checkpointing and resumption.
And a working inference system.
But there was one more thing I wanted to do.
Make the model usable outside my training environment.
I’d been working with PyTorch checkpoints during development, but I wanted a cleaner way to package the final trained model.
So I exported the weights to Safetensors, saved the architecture configuration, and bundled the tokenizer.
model.safetensors
config.json
tinystories_tokenizer.modelI also made sure the model could be loaded for inference independently of the training checkpoint.
Then I published the result on Hugging Face.
That was it.
My first language model, built from scratch, trained, and publicly available.
So, Was It Worth It?
Absolutely.
But not because I now have a 7.55-million-parameter model that can write tiny stories.
There are countless existing models that can do much more.
I built Axiom because I wanted to understand how these things work internally.
And now I can open my own code and follow a token through the entire process.
From an integer ID.
To an embedding vector.
Through Query, Key, and Value projections.
Through attention heads and Transformer blocks.
To vocabulary logits.
To loss, gradients, parameter updates, and eventually generated text.
I also learned a few things the hard way.
A working implementation isn’t necessarily a correct one.
A decreasing training loss doesn’t automatically mean good generalization.
One validation reading isn’t the whole story.
And throwing more training steps at a plateau isn’t always the smartest solution.
The biggest difference is that I now have a baseline I can actually experiment with.
What’s Next?
I’m not planning to spend the next few weeks building another identical Transformer just with more parameters.
That wasn’t the goal.
I wanted to understand the fundamentals well enough to start exploring research papers and implementing ideas myself.
Now I can.
I’m particularly interested in inference optimization, attention mechanisms, and getting more out of limited hardware.
The first thing I want to explore is KV caching.
Right now, my autoregressive generator recomputes attention over the available context every time it generates a new token.
That works, but it’s wasteful.
KV caching would let me reuse previously computed Keys and Values rather than recomputing them at every generation step.
Same model weights.
Different inference implementation.
And an actual performance comparison to run.
But that’s for another blog.
For now, Axiom v1 is done.
60,000 steps.
4GB VRAM.
A lot of confused moments, debugging, and learning.
And one tiny language model that can finally write its own stories.
Not bad for a project that started with me trying to understand what [B, T, C] meant.
Model: Axiom on Hugging Face
Built with PyTorch, TinyStories, Neovim, and an unreasonable amount of curiosity.