Influence of modal coupling on the nonlinear dynamics of Augusti’s model