Bottlenecking in real time factor

Hello all,

I am working on a project where I am evolving an UAV controller with a simple genetic algorithm. I am using gazebo, ROS and SITL. I already managed to evolve the controller in a few hundred simulations but this takes some hours.

Therefore, my objective is to run the simulation faster. I already tried to set the update_rate to 0, but my real time factor keeps between 0.6 and 1 (1.05 on tops). I found that when I erase the drone of the world, the real time factor goes to 30-40. I am using the Iris drone model from ardupilot.
I am a little bit confused about this because my CPU usage is 20-30% during the runs, with gzserver having the most CPU % usage.
I already search in many potst and questions but I cant find anything that works. Is there anything that you suggest me to do?

Thank you!

Does the drone have any rendering sensors (cameras, lidars)?

Hello!

No lidars, and I tested with and without cameras, no differences.

I am running these models, ardupilot_gazebo/models at master · SwiftGust/ardupilot_gazebo · GitHub, and got some differences:

  • I tested one world with the iris_with_standoffs, (from what I saw, doesnt have the ardupilot plugin, “libArduPilotPlugin.so”) got RTF of 6 (very good);
  • Tested another world with the iris_with_standoffs_demo (with ardupilot plugin), got RTF of 0.6 (same that I was having before).

I guess it is something related to this ardupilot plugin in specific, not sure what.

The ArduPilot plugin is iterating and lock stepping with the ArduPilot flight control software and waiting for commands to come through and sending updates back to the ArduPilot.

I haven’t looked at this in a while but you probably want to make sure the ArduPilot is configured and has enough resources to run faster than real-time too.

Following up from a discussion on ArduPilot Discord:

@Alex is using SwiftGust’s fork of the ardupliot_gazebo plugin configured to run with Gazebo11. This version doesn’t enforce lock-step of the simulation, so that shouldn’t be blocking in this case (but could be a bottleneck in the version that supports Gazebo Garden - but I think it’s more likely to be the simulation update step).

The ArduPilot parameter that lets SITL run faster than RT is: SIM_SPEEDUP which can be set in MAVProxy with:

> param set SIM_SPEEDUP 10

In Gazebo Garden the Iris quadcopter example will run at RTF ~ 2.5x with max time step set to 0.001. If the max time step is increased to 0.01 the RTF is ~ 100x but the model is unstable (the rotor joints oscillate badly even at rest). This suggests the physics engine update is the bottleneck (4 revolute joints, 8 lift-drag plugins submitting forces and torques) rather than the flight controller or UDP message layer - at least for new version of the plugin.

Edit

With a time step = 0.002 the RTF ~ 400x. The copter will fly auto missions however the simulation is on the edge of stability - so not sure that it’s a good configuration for testing new controllers but it does confirm that the bottleneck is the simulation physics update.

Hey, and thank you all for the replies.
As @srmainwaring said, I am using the SwiftGust’s ardupilot_gazebo plugin. I set the SIM_SPEEDUP in ardupilot parameter to be very high, but the simulation still runs at 0.6. When I delete the ardupilot plugin from the sdf file, the same world and drone run at 6 RTF.