Two papers out, on a new paradigm of temporal computation! Our first work funded by
@ARIA_research. We're super proud of this, and there's much more coming:
Neural networks, specifically their weights, have become the most useful functional abstraction from the brain. As powerful function approximators they seeded the current era of AI. But there are many more useful abstractions. The brain does so much more than learn weights.
At its core, the brain exploits the structure of the physical world to perform computation. It aligns itself to reality - to time and space.
@achterbrain and I have long been working on how to embed neural networks in space and link this to hardware. But there's another dimension: time itself. We think leveraging both is a big part of the puzzle as to why human learning is so efficient.
If this is true, how can we use space and time in neural networks? What would this even mean? We pitched to
@BramhavarSuraj at
@ARIA_research one possible way. Several years ago, I stumbled upon theoretical work that made it concrete. Time delays - the physical fact that signals take time to travel - can store memory (and other things, including increasing the number of computable functions). Even in feedforward networks. The delay here isn't overhead as would be traditionally thought but a feature that can be exploited.
TL/DR - our main findings show that you can do computation in neural networks with time, without (much) need for weights. And it's remarkably efficient. We also show it's possible to co-design hardware (we open-source a chip design) with novel architectures that exploit time to maintain long contexts.
In our first paper, we train neural networks to learn delays and weights. The result: state-of-the-art performance on all the temporally complex neuromorphic benchmarks we tested. Crucially, once you're encoding time directly, weights become almost irrelevant. We compress them to 1.58 bits, just positive, negative, or absent (ternary) weights. That's it. Model sizes drop to double-digit kilobytes!
This works because we're finally encoding information the way the task needs it. Time and space. Which just so happens to be... everything in the physical world. Robotics. Embodied systems. Physical intelligence.
In the second paper, we turn to memory. Intelligent systems don't just compute - they hold onto information over long context windows. And doing this efficiently in hardware underlying computation is a hard, unsolved problem.
To solve this, we built a dual memory pathway architecture: fast spiking dynamics plus a compact state-space memory module that evolves much more slowly. Inspired by how the mammalian cortex separates fast somatic spiking from slower dendritic integration. Each layer maintains a tiny amount of working memory - just ~5% of hidden width - that summarises recent activity and feeds back into the network. There are many directions to scale this further. We co-designed the algorithms and hardware together from the ground up. The result: >4× throughput and >5× energy efficiency, beating Intel's Loihi2 and other leading neuromorphic platforms like DenRAM and ReckOn.
We're open-sourcing the chip design so people can build on this.
Incredibly proud of the team:
@pengfeisun17,
@achterbrain,
@neuralreckoning, Zhe Su and
@giacomoi
One of our key takeaways: the current AI paradigm is a narrow slice of what's possible. Scaling homogeneous systems only gets you so far. Biological intelligence is deeply heterogeneous: different timescales, different substrates, different specialisations, all co-evolved together. We think the next frontier of scaling means embracing that heterogeneity. Algorithms and hardware aren't separate problems. They need to co-evolve together.
Can't wait to share what we're cooking next.