System and method for determining the generalized scattering matrix of a waveguide device