Riffusion-Melodiff is simple, but interesting idea, (that I have not seen anywhere else) how to create cover versions from songs.
Riffusion-Melodiff is built on a top of
Riffusion
model, which is fine-tuned Stable Diffusion model to generate Mel Spectrograms. (Spectrogram is kind of
visual representation of music by dividing waveforms into frequencies.) Riffusion-Melodiff does not contain new model, there was no new training, nor fine-tunig.
It uses the same model as Riffusion only in a different way.
Riffusion-Melodiff uses Img2Img pipeline from Diffusers library to modify images of Mel Spectrograms to produce new versions of music. Just upload your audio in wav format (if you have audio in a different format, transfer it first to wav by online converter). Then you may use Img2img pipeline from the Diffusers library with your prompt, seed and strength. Stregth parameter decides, how much will modified audio relate to initial audio and how much it will relate to the prompt. When strength is too low the spectrogram is too similar with original one and we do not receive new modification. When strength is too high, then spectrogram is too close to the new promopt, which may cause loss of melody and/or tempo from the base image. Good values of strength are usually about 0,4-0,5.
Good modifications are possible for proper prompt, seed and strength values. Those modifications will keep the tempo and melody from the initial audio, but they will change eg. instrument, playing that melody. Also with this pipeline longer than 5s music modifications are possible. If you cut your audio into 5s pieces and use the same prompt, seed and strength for each modification, generated samples will be somewhat consistent. So if you concatenate them together, you will have longer audio modified.
Quality of the generated music is not amazing, (mediocre, I would say) and it needs a bit of prompt and seed engineering. But it shows one way, how to make cover versions of music in the future.
Colab notebook is included, where you can find step by step, how to do it. Melodiff_v1.
Examples of music generated by modifying the underlying song:
Amazing Grace, originally played by flute, modified to be played by violin
Bella Cao, originally played by violin, modified to be played by saxophone
Iko iko, originally played by accordion, modified to be played by saxophone
When the Saints, originally played by violin, modified to be sang by vocals
Examples of longer music samples:
Iko iko, originally played by accordion, modified to be played by saxophone
Iko iko, originally played by accordion, modified to be played by violin
When the Saints, originally played by piano, modified to be played by flute
Im using standard (not paid) Google Colab Gpu configuration for inference. Im using default values for number of inference steps (23) from the underlying
pipelines. With this setup it takes about 8s to produce 5s long modified sample. For start it is ok, I would say.