Low-Dimensional Audio Inpainting with Mixed Generative-Predictive Model

In this work we present LDAI, a mixed generative-predictive DNN framework aimed at repairing long-duration corrupted audio fragments in musical sequences. The proposed pipeline incorporates a convolutional encoder-decoder structure for waveform mapping and a low-level conditional WGAN for latent inpainting. Leveraging some recent insights of generative network modeling, we managed to reduce the effort required by the task, by transferring the inpainting process to a low dimensional level, thus ensuring robust and fast training convergence, and richer musical note generation. When dealing with gaps up to 1024 ms, we observed better performance than the baseline method of our previous study, namely bin2bin-MIDI, which began to struggle in repairing gaps exceeding 750 ms. The results, evaluated through perceptual metrics and estimated subjective scores, indicated improvements as follows: +0.89 points in PESQ, +3.1% in STOI, +4.1% in PLCMOS, and +0.87% in DNSMOS. Additionally, the Fréchet Audio Distance, aimed at assessing generative tasks, exhibited a remarkable 40% decrease.