Calculation of optical forces and polarization modulation in metasurface membranes by hybrid RCWA scattering-transfer matrix method