Modal analysis of nonlinear propagation in dielectric slab waveguide