By Ghost of Divinity — Jan 4, 2022
Review of paper by Christian J. Steinmetz and Joshua D. Reiss, Queen Mary University of London, 2021.
Christian J. Steinmetz and Joshua D. Reiss have released an exciting paper titled 'Steerable discovery of neural audio effects' (along with accompanying code and examples) that enables users to easily emulate and modify audio effects through the use of deep learning methods.
Given an unprocessed audio recording (no audio effect applied) and a processed version of that audio recording (audio effect applied), this deep learning algorithm emulates the audio effect in the processed recording with surprising accuracy while providing some inaccuracies that users may find idiosyncratically appealing. The paper is notable in that it also enables users to modify the network activations associated with the model-determined audio effect and download audio files with those applied modifications. The authors hope to eventually create the same result using just the processed input recording.
This technology opens up a new paradigm for sound and audio plugin design. Sound designers will love the ability to easily apply these modifiable effect mappings to new sound recordings, taking advantage of compelling sound effect palettes that are very difficult to emulate with preexisting technologies. Audio plugin designers might eventually be able to incorporate unique "Neural FX" parameters into their audio effect plugins, allowing musicians and audio engineers access to a new range of audio effects for their work.
To get an idea of what this might sound like for your own sound examples, you can input them into this Hugging Face app with several pre-trained models to try:
Steerable Nafx - a Hugging Face Space by akhaliq
Discover amazing ML apps made by the community
The Google Colab notebook accompanying this paper is very straightforward to use and demonstrates the full breadth of sonic possibilities that users can discover with their own sound examples:
Utilizing the power of neural networks appears to be an incredibly promising approach in music and audio technology research. ‘Steerable discovery of neural audio effects’ clearly demonstrates this. The ability to quickly create new audio effects just from inputting two audio files will open up a new world of sound design possibilities that users will love to explore in greater depth. It will be very interesting to see how this research area develops over the coming months and years.