A novel optimization design method for the multiphase pump impeller is proposed through combining the quasi-3D hydraulic design(Q3DHD), the boundary vortex flux(BVF) diagnosis, and the genetic algorithm(GA). The BVF diagnosis based on the Q3DHD is used to evaluate the objection function. Numerical simulations and hydraulic performance tests are carried out to compare the impeller designed only by the Q3DHD method and that optimized by the presented method. The comparisons of both the flow fields simulated under the same condition show that(1) the pressure distribution in the optimized impeller is more reasonable and the gas-liquid separation is more efficiently inhibited,(2) the scales of the gas pocket and the vortex decrease remarkably for the optimized impeller,(3) the unevenness of the BVF distributions near the shroud of the original impeller is effectively eliminated in the optimized impeller. The experimental results show that the differential pressure and the maximum efficiency of the optimized impeller are increased by 4% and 2.5%, respectively. Overall, the study indicates that the optimization design method proposed in this paper is feasible.