I would love to have an event camera plugin for Gazebo. In event cameras, each pixel operates independently and fires an event once a pre-set threshold in change of brightness was exceeded. This implies that there is no concept of conventional frames anymore; absolute brightness information is not available. For more info, please see Wikipedia or this amazing survey.
There have been some attempts to simulate event cameras, including rpg_esim and v2e (video2events) [apologies, I’m only allowed two links …]. These tools typically take a video as input, use some super slow-motion approach to temporally sample more finely, and then simulate the events.
I think it would be amazing to instead have a plugin for Gazebo that simulates event cameras directly without the intermediate step of using a video.
Unfortunately, I am not well versed in Gazebo / writing plugins for Gazebo. However, I am willing to learn, have good knowledge of ROS (as one of the RoboStack maintainers) and potentially could get a student to work on it, if I get some pointers on where to start / what would be needed for this project. I’d also be more than happy to collaborate with someone who has similar interests.
Render a frame each simulation step (which is usually 250 Hz or 1000 Hz) and do diffs between these frames. This would be possibly the best approximation of event cameras. But super-slow (real-time factor probably well below 1%).
Start your work off of LogicalCameraSensor. It maintains a frustum and checks for collision between models and this frustum. So if each pixel of your camera created such frustum, you would have something like a binary event camera. Unfortunately, the current logical camera only checks whether the centerpoint of each model is inside the frustum, which is probably not what you want.
Start your work off of PerformerDetector. It maintains an axis-aligned box and reports whenever this box intersects with some performer. Performers are special kinds of models “tagged” as performers. If you’d change the AABB to a frustum, you might get something interesting.
I’m not sure if cases 2 and 3 would actually result in faster rendering than the image-based approach. But they’re a way to try…
Apologies for my late reply! I really appreciate these starting points, that’s great. I have written up a student project to investigate this, hopefully someone will pick it up .
Regarding the first way to start, I actually realised that this is what the Neurorobotics platform people do, see Bitbucket - however the update rate is set to 60 Hz to keep it computationally feasible (so probably not a great approximation)
Hi @Tobias-Fischer, have you done any progress on an event-based camera simulation plugin?
I am studying the Neurorobotics DVS simulation, this seams to be the closest implementation. Nevertheless, I would like to have a gazebo plugin using Fortress or Garden and without ROS. I will start working on that if you haven’t created the plugin already.
Hi @sebastienmascha - unfortunately I haven’t found a student yet to pick this project up. If you make any progress in this direction, please let me know, I would be very interested.
I’ve forked this bitbucket repository locally and will be putting some effort into this over the next few weeks. I will be happy to hear about any other progress that has been made offline from this thread and chip in any progress that I make along the way.
Apologies - I missed your response here. Have you made any progress in this direction? I just had an interesting student contacting me. It would be great to join forces.
I’m making a container to unify development and plan to test video this afternoon (I’m in UTC-7 time zone). I ported this work to my Codeberg Repo and am in the very early stages of integration. Feel free to check it out!
.