Text-to-music generation technology is progressing rapidly, creating new opportunities for musical composition and editing. However, existing music editing methods often fail to preserve the source music’s temporal structure, including melody and rhythm, when altering particular attributes like instrument, genre, and mood. To address this challenge, this paper conducts an in-depth probing analysis on attention maps within AudioLDM 2, a diffusion-based model commonly used as the backbone for existing music editing methods. We reveal a key finding: cross-attention maps encompass details regarding distinct musical characteristics, and interventions on these maps frequently result in ineffective modifications. In contrast, self-attention maps are essential for preserving the temporal structure of the source music during its conversion into the target music. Building upon this understanding, we present Melodia, a training-free technique that selectively manipulates self-attention maps in particular layers during the denoising process and leverages an attention repository to store source music information, achieving accurate modification of musical characteristics while preserving the original structure without requiring textual descriptions of the source music. Additionally, we propose two novel metrics to better evaluate music editing methods. Both objective and subjective experiments demonstrate that our approach achieves superior results in terms of textual adherence and structural integrity across various datasets. This research enhances comprehension of internal mechanisms within music generation models and provides improved control for music creation.
Accordion to TromboneFlute to Piano(From Figure 6)Trumpet to OrganGuitar to TrumpetMan to TrumpetClarinet to Guitar ＆ Jazz to MetalViolin to PianoGuitar to ViolinDrum to BassBlues to MetalRock to ClassicalFolk to ReggaeJazz to MetalClassical to CountryJazz to ClassicalClarinet to Guitar ＆ Jazz to MetalRock to JazzJazz to HiphopRock to ClassicalTense to Peaceful(From Figure 6)Happy to SadMysterious to Epic Movie SoundtrackEmotional to HappyAblation studies demonstrating the distinct roles of Self-Attention (SA) and Cross-Attention (CA) and justifying the core mechanism of Melodia.
Drum to Violin(From Figure 2)
The original audio with a distinct rhythmic structure.
Preserves the source's temporal structure (rhythm) while changing the instrument to violin.
Blues to Classical(From Figure
2)The original audio in a 'Blues' style.
Effectively preserves the melodic contour of the blues track while shifting the style to classical.
Clarinet to TromboneThe original clarinet recording.
The optimal result: structure is preserved, and timbre is successfully changed to trombone.