What is style transfer for audio?

Style transfer for audio is the process of transferring the characteristics of the style of one audio signal (for example, timbre, rhythm, acoustic atmosphere) to another audio signal, while preserving the content or melodic structure of the second.

Unlike the classic style transfer for images, where the visual style is transferred, the task in audio is more complex due to the temporal nature of the signal and the complex structure of sound.

Applications:

• Transferring the style of music performance (for example, playing a melody in jazz or rock style).
• Changing the timbre of a voice while maintaining speech.
• Creating new sound effects.

Technically, this is achieved using neural network models that extract and separate representations of content and style of audio, for example, using spectrograms and deep learning methods (autoencoders, GANs, transformers).

Example: a model takes a spectrogram of a melody and a spectrogram of audio with the desired style, then generates a new spectrogram that combines the content of the first and the style of the second, and then converts it back into an audio signal.