Scattering matrices representing the transformations between modal bases in rectangular waveguide