Gazebo should be able to handle that swarm size, but there are number of factors to consider.
1.The number and type of sensor on each UAV.
2. The complexity of the environment.
3. What control algorithms are used, and where they are executed.
4. Do you need to simulate aerodynamics, or can you get away with kinematic movement of the objects?
I recommend setting up a simulation with your desired parameters, and see what type of performance you achieve. If the performance is not satisfactory, then you can consider alternative strategies such as reducing environment complexity or running control algorithms on a separate PC(s).