HpRNet : Incorporating Residual Noise Modeling for Violin in a Variational Parametric Synthesizer

This accompanying notebook contains audio examples and illustrations supporting our paper

1) The Parametric Representation

The above figure summarizes our parametric representation.

Here is an upper octave note transformed through this representation

import IPython.display as ipd

# print('\033[1m \033[4m 1) Lower Octave Sa')
# ipd.display(ipd.Audio('./audio/70-SaLp_L_So_Sm.wav'))
# print('1.a) Harmonic Portion h(t)')
# ipd.display(ipd.Audio('./audio/70_L_SaLp_Sm_So_reconH.wav'))
# print('1.b) Residual Portion r(t)')
# ipd.display(ipd.Audio('./audio/70_L_SaLp_Sm_So_reconR.wav'))
# print('1.c) Reconstruction h(t) + r(t)')
# ipd.display(ipd.Audio('./audio/70_L_SaLp_Sm_So_recon.wav'))

# print('\033[1m \033[4m 2) Middle Octave Ri1')
# ipd.display(ipd.Audio('./audio/04-Ri1_M_So_Sm.wav'))
# print('3.a) Harmonic Portion h(t)')
# ipd.display(ipd.Audio('./audio/04_M_Ri1_Sm_So_reconH.wav'))
# print('3.b) Residual Portion r(t)')
# ipd.display(ipd.Audio('./audio/04_M_Ri1_Sm_So_reconR.wav'))
# print('3.c) Reconstruction h(t) + r(t)')
# ipd.display(ipd.Audio('./audio/04_M_Ri1_Sm_So_recon.wav'))

print('\033[1m \033[4m 1) Upper Octave Ma2')
ipd.display(ipd.Audio('./audio/49-Ma2_U_So_Sm.wav'))
print('1.a) Harmonic Portion h(t)')
ipd.display(ipd.Audio('./audio/49_U_Ma2_Sm_So_reconH.wav'))
print('1.b) Residual Portion r(t)')
ipd.display(ipd.Audio('./audio/49_U_Ma2_Sm_So_reconR.wav'))
print('1.c) Reconstruction h(t) + r(t)')
ipd.display(ipd.Audio('./audio/49_U_Ma2_Sm_So_recon.wav'))

  1) Upper Octave Ma2

1.a) Harmonic Portion h(t)

1.b) Residual Portion r(t)

1.c) Reconstruction h(t) + r(t)

2) Network Reconstructions

We employ the following 3 networks:

1. Independently modeling the Harmonic and Residual components (INet)
   

2. One network concatenatively modeling the Harmonic and Residual (ConcatNet)

3. Modeling the sum and difference of the harmonic and residual jointly (JNet)
   

In each of the above, the H and R correpsond to the framewise Cepstral Vectors for the harmonic and residual respectively, and f is the fundamental frequency. The networks run on each frame in the input note, and overlap-add is performed to obtain the network reconstructed audio.

We now present examples of notes in the higher octave (where the residual is perceptually important) and their respective network reconstructions below:

print('\033[1m \033[4m Input Note: Upper Octave Sa')
ipd.display(ipd.Audio('./audio/37-Sa_U_So_Sm.wav'))
print('Independent Network Reconstruction - INet')
ipd.display(ipd.Audio('./audio/INet_37_U_Sa_Sm_So_recon_HpR.wav'))
print('Concatenative Network Reconstruction - ConcatNet')
ipd.display(ipd.Audio('./audio/ConcatNet_37_U_Sa_Sm_So_recon_HpR.wav'))
print('Joint Network Reconstruction - JNet')
ipd.display(ipd.Audio('./audio/JNet_37_U_Sa_Sm_So_recon_HpR.wav'))

  Input Note: Upper Octave Sa

Independent Network Reconstruction - INet

Concatenative Network Reconstruction - ConcatNet

Joint Network Reconstruction - JNet

From the MSE of the reconstructed notes, we saw that ConcatNet had the highest MSE for the Residual reconstruction, you can hear that hear as well, the ConcatNet reconstruction sounds more grainer than the other two.

We also reconstruct another note:

print('\033[1m \033[4m Input Note: Upper Octave Ri2')
ipd.display(ipd.Audio('./audio/41-Ri2_U_So_Sm.wav'))
print('Independent Network Reconstruction - INet')
ipd.display(ipd.Audio('./audio/INet_41_U_Ri2_Sm_So_recon_HpR.wav'))
print('Concatenative Network Reconstruction - ConcatNet')
ipd.display(ipd.Audio('./audio/ConcatNet_41_U_Ri2_Sm_So_recon_HpR.wav'))
print('Joint Network Reconstruction - JNet')
ipd.display(ipd.Audio('./audio/JNet_41_U_Ri2_Sm_So_recon_HpR.wav'))

  Input Note: Upper Octave Ri2

Independent Network Reconstruction - INet

Concatenative Network Reconstruction - ConcatNet

Joint Network Reconstruction - JNet

%%HTML
<script>
  function code_toggle() {
    if (code_shown){
      $('div.input').hide('500');
      $('#toggleButton').val('Show Code')
    } else {
      $('div.input').show('500');
      $('#toggleButton').val('Hide Code')
    }
    code_shown = !code_shown
  }

  $( document ).ready(function(){
    code_shown=false;
    $('div.input').hide()
  });
</script>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Cells"></form>