I've been working on a project called Leo / PSCLS.
The goal isn’t to build yet another Transformer with a different name.
I’ve been trying to explore a different question:
«What if we built an AI architecture around persistent neural state, sparse connectivity, recurrent processing, and memory — instead of making attention and large dense parameter matrices the core building blocks?»
Leo is still very early and nowhere near a fluent language model. But we’ve reached a point where I think the architecture itself is worth talking about.
What is Leo?
Leo’s basic representation is raw UTF-8 bytes.
There is:
- No BPE tokenizer
- No fixed word vocabulary
- No token embeddings as the fundamental representation
- No giant dense parameter matrix as the core representation
The current model has roughly:
- 32,768 neurons
- 1,572,864 fixed sparse synapses
- 524,288 persistent context slots
- 32-dimensional context embeddings
- Maximum context order of 8
The model works directly on bytes.
The idea is that higher-level structure can emerge from learning, instead of being baked in through a predefined token system.
How is Leo different from a Transformer?
A Transformer usually takes tokenized input, turns tokens into embeddings, runs self-attention and dense layers, and predicts the next token.
Leo is built differently.
The core computation is based on:
- Sparse recurrent neurons
- Fixed sparse synaptic connectivity
- Persistent context
- Eligibility traces
- Homeostasis
- Learned neural dynamics
- Next-byte prediction
The key difference isn’t just “sparse vs dense.”
It’s the role of persistent state.
A Transformer is mostly a function over a fixed context:
«“Given this context, compute the next output.”»
Leo is designed more like an evolving system:
«“Process incoming experience, update internal state, and let that state shape future predictions.”»
Right now, Leo is still a trained system, not an autonomous self-learning agent. Online or self-directed learning is a future direction — not something I’m claiming it already does.
How is Leo different from attention?
This is probably the most important distinction.
Attention is not the same thing as persistent memory.
In a Transformer, attention dynamically recomputes relationships across tokens in the current context.
It’s basically asking:
«“What parts of this context matter right now?”»
Leo doesn’t rely on attention as its core mechanism.
Instead, it keeps a persistent internal state that evolves over time. Information can influence future computation through:
- Recurrent neural activity
- Persistent context slots
- Sparse synaptic connections
- Eligibility traces
- Homeostatic regulation
So instead of repeatedly re-scoring relationships across a sequence, Leo is trying to maintain a continuously evolving internal representation as bytes flow through it.
That’s one of the reasons I think of it as more brain-inspired than Transformer-like.
Why call it brain-inspired?
I’m not claiming Leo is a brain simulation.
The brain is vastly more complex.
The inspiration comes from a few broad principles:
Sparse activity
The brain doesn’t activate everything at once.
Leo uses sparse connectivity and sparse activation patterns.
Persistent state
The brain doesn’t reset after every word.
Your understanding carries forward continuously.
Leo maintains persistent recurrent/context state.
Plasticity
Biological systems adapt through experience.
Leo has learning mechanisms that modify its parameters during training.
Homeostasis
Brains regulate activity levels instead of letting everything drift freely.
Leo includes similar stabilizing mechanisms.
Distributed memory
Human memory isn’t a lookup table of sentences.
It’s distributed across activity and connections.
Leo uses recurrent state and sparse structure instead of explicit token memory.
Again: this is inspired by biology, not an attempt to replicate it.
How does Leo learn?
At a high level, imagine feeding it:
"The cat sat on the mat."
The UTF-8 bytes stream in one by one.
Each byte activates a sparse subset of neurons.
That activity flows through the recurrent system and updates internal state.
Learning signals (like eligibility traces) track which parts of the network were involved.
Then the system updates its parameters based on those dynamics.
So instead of:
«“Tokenize everything and train a huge dense model”»
It’s more like:
«“Let a sparse recurrent system process raw bytes and learn from its evolving internal activity.”»
Right now, Leo does not decide on its own what to learn from. That’s still fully controlled by the training setup.
How does Leo generate text?
Generation is also byte-by-byte.
Say the prompt is:
"Once upon a time"
Leo processes those UTF-8 bytes and builds an internal state.
Then it predicts the next byte.
That byte gets appended.
The state updates.
Then it predicts the next byte again.
And so on.
So the loop is:
bytes → neural state → next-byte prediction → updated state → repeat
There is no token vocabulary like:
- “Once”
- “upon”
- “ing”
Everything stays at the byte level.
The hope is that structure emerges from learning patterns over time, rather than being imposed through tokenization.
The important question: does it actually learn?
This was the part I cared about most.
We spent a lot of time optimizing the system.
The original version ran at about:
"~375 bytes/sec"
The current GPU version reaches about:
"~2,345 bytes/sec"
So roughly a 6× speedup.
But speed doesn’t matter if nothing is actually learned.
So we stopped optimizing and ran a controlled experiment.
Experiment setup
- 3,000 TinyStories
- 3 passes
- 9,000 total presentations
- 90 GPU workers
- ~113 minutes total
We compared a trained checkpoint against a frozen baseline on held-out data.
Results
Held-out BpB
2.67848 → 2.64052
Held-out accuracy
52.3737% → 53.6187%
Neural-only BpB
4.12877 → 4.10943
Context gain
1.45029 → 1.46891
Repetition rate
32.166% → 28.466%
All five metrics improved.
So at this scale, we do see that training produces measurable gains on unseen data.
That’s the result I care about most.
Not:
«“This is AGI”»
Not:
«“This beats Transformers”»
It doesn’t.
The more modest takeaway is:
«This unusual architecture can be trained, and training improves performance in a measurable way.»
It’s still far from fluent
This is important.
If I prompt:
"Once upon a time..."
I might get things like:
- “to the store”
- “said that”
- “they went”
- “with her”
But also:
- broken grammar
- malformed words
- repetition
- weak long-range structure
- messy endings
So:
53.6% next-byte accuracy is not fluent English.
It’s still very early.
Why not just scale it up?
That’s one of the next questions.
We don’t yet know if 32K neurons is a real bottleneck.
It’s still improving with more training.
So instead of immediately jumping to 64K or 128K, I want to understand:
- how performance scales with data
- how it scales with capacity
- where it actually saturates
Basically, I want to build a scaling curve for Leo itself.
If it saturates early, that tells us something important.
If it keeps improving, that’s even more interesting.
The bigger question
Transformers have shown what happens when you scale:
- parameters
- data
- compute
Leo is exploring a different direction:
- persistent neural state
- sparse recurrence
- context memory
- eligibility traces
- homeostasis
- byte-level representation
Maybe it doesn’t scale well.
Maybe it scales differently.
Maybe it needs different hardware.
Maybe structure emerges in unexpected ways at larger sizes.
I don’t know yet — that’s the point.
For now, Leo is not AGI.
It’s not a Transformer replacement.
It’s not even a strong language model yet.
It’s an experiment in a different kind of learning system.
The question I’m trying to answer is:
«Can useful intelligence emerge from persistent neural dynamics, memory, and sparse recurrent computation — instead of primarily scaling dense attention-based models?»
We’ve shown it can learn under controlled training.
Now I want to see how far it can go.
[link] [comments]