I've been experimenting for a while with two different ideas for non-Transformer language models, and I'm now considering combining them into one architecture.
The first is NORD, a recurrent/spiking architecture I've been developing around token-time dynamics, persistent state, sparse processing, and SNN-style temporal computation.
The second is RHEA (Reactive Hypergraph Event Architecture), which I'm currently prototyping at ~1B parameters.
The basic idea behind RHEA is that instead of pushing every token through a fixed stack of layers, the model maintains a set of latent events and dynamically chooses which internal computations should happen next.
The scheduler, which I call ARES, estimates whether a candidate reaction is worth executing.
Conceptually:
events / latent facts
v
candidate reactions
v
ARES
"what is worth
computing next?"
/ | \
v v v
R3 R17 R81
\ | /
v
new events
A reaction can combine existing events and create a new latent event:
event A + event B
reaction
v
event C
The interesting part is that I think NORD and RHEA may fit together surprisingly well.
My current idea is:
input tokens
v
NORD sensory / temporal SNN
spike/events
v
RHEA event fabric
v
ARES
decides what should fire
/ | \
v v v
reaction reaction reaction
| | |
NORD NORD NORD
SNN SNN SNN
microcircuit microcircuit
\ | /
v
new events
memory / queries
v
output
The rough division of responsibility would be:
NORD = temporal dynamics
- recurrent state
- LIF/spiking dynamics
- persistent memory
- event triggering
- local temporal computation
RHEA = cognitive/event structure
- latent facts/events
- dynamic interaction graph
- creation of derived events
- multi-step computation
ARES = executive scheduler
- estimates reaction utility
- accounts for compute cost
- decides which reactions actually execute
- allows computation depth to vary with the problem
One thing I'm particularly interested in is making the reaction operators themselves small hybrid SNN microcircuits.
Instead of:
A + B -> dense MLP -> C
something closer to:
A + B
|
v
spiking microcircuit
t0: spike
t1: spike
t2: spike spike
|
v
latent event C
I would NOT make the whole model purely spiking.
My current thinking is to keep latent representations and the language head dense/BF16, while using spiking dynamics for temporal state, memory, event triggering and some reaction computation.
Something like:
token embeddings -> dense
latent event vectors -> dense
ARES utility model -> dense
temporal state -> SNN/recurrent
persistent memory -> SNN/recurrent
reaction dynamics -> hybrid SNN
LM head -> dense
Another part I find interesting is persistent memory.
A RHEA event could write into a slow NORD memory state:
RHEA event
v
NORD persistent memory
... hundreds/thousands of tokens ...
v
memory activity crosses a threshold
v
new recall event
v
RHEA
So memory would not necessarily be passive storage. It could actively generate events when relevant internal states become excited.
I'm also considering a form of path crystallization.
If the system repeatedly performs something like:
reaction A
->
reaction F
->
reaction K
->
reaction B
the repeated sequence could eventually be distilled into a faster macro-reaction or learned skill.
In the hybrid version, this could potentially include recurring spike/reaction patterns as well.
So the architecture would operate across several timescales:
FAST
NORD spike / recurrent dynamics
MEDIUM
RHEA reaction chains and reasoning
SLOW
persistent memory + crystallized skills
The overall principle I'm exploring is basically:
«computation should follow information, rather than information always following a fixed computation graph.»
A simple input might activate very little of the system.
A difficult input could trigger more events, more reactions and deeper computation.
Importantly, I'm not claiming this is better than Transformers.
There are some obvious problems I expect:
- irregular computation is unfriendly to GPUs
- sparse/discrete routing is difficult to train
- skipped reactions create a credit-assignment problem
- SNN dynamics could make an already difficult optimization problem even less stable
- dynamic event memory can accumulate garbage
- batching event-driven computation efficiently is non-trivial
- it's possible that the extra architectural complexity simply won't outperform a well-optimized Transformer/MoE
For skipped-reaction credit I'm currently experimenting with a counterfactual mechanism where near-threshold reactions get a cheap preview, so the scheduler can estimate whether skipping them was a mistake.
The current RHEA prototype is already being trained independently; the NORD/RHEA hybrid described here is still a design direction rather than a finished model.
What I'm most interested in hearing from people here:
- Does this decomposition make sense?
- What do you think would fail first?
- Are there papers/projects that are especially close to this?
- Would you keep the SNN component limited to memory/temporal state, or also use it inside the reaction operators?
- Is dynamic computation at this granularity likely to lose too much hardware efficiency to be worthwhile?
I'd especially appreciate criticism from people working on SNNs, recurrent models, MoE/routing, adaptive computation, or non-Transformer architectures.
[link] [comments]