Projects / Learning without backprop
Alternatives to backpropagation
Hebbian learning, predictive coding, forward-forward, PEPITA, reservoirs and evolutionary methods, benchmarked against backprop on sample efficiency and wall-clock time.
A collection of learning rules that avoid backpropagation, or only partly rely on it, all tested on MNIST under matched architectures.
Implemented
- Hebbian: FastHebb, nonlinear PCA/ICA variants, soft winner-take-all, convolutional Hebbian layers
- Predictive coding: bidirectional, feedforward, and incremental (iPC)
- Generalized principal subspace projection, based on multi-compartment pyramidal neurons
- Forward-only: forward-forward with layer collaboration, and PEPITA
- Reservoir computing on Erdős–Rényi and scale-free graphs
- Evolutionary: CMA-ES and a Kronecker-product genetic algorithm with a codon genotype
- Side quests: graph hierarchy measures, Laplacian renormalization, an empowerment agent on Atari
The benchmark
Networks are parameter-matched (reservoir sizes are computed to match synapse counts), then compared on accuracy per sample and per second of training. Predictive coding’s slow relaxation phase is part of the result.