Projects / Generation & pattern formation
Neural audio codec experiments
A SoundStream/EnCodec-style VQ-VAE with original additions: wavelet upsampling layers, self-organizing codebooks, and an energy-transformer bottleneck that outperformed RVQ.
- Wavelet layers (original): upsampling layers that implicitly learn a wavelet decomposition, since sound is made of waves
- Causal multiresolution convolutions
- Self-organizing-map codebooks, via quantization-maps
- An energy-transformer bottleneck, via energy_transformer
- Discriminator changes that prevent collapse, and training one discriminator at a time, weighted by difficulty