GSoC 2021: Machine Learning Extension to Ignition Gazebo

Project: Machine Learning Extension to Ignition Gazebo
Organization: Open Source Robotics Foundation
Mentors: Ashton Larkin
Student: Amr Elsersy
Linkedin - Github

Working for Open Robotics as a Google Summer of Code 2021 Intern was an amazing experience that was full of learning and commitment in such a great working environment. It is my second time at Open Robotics as I was a GSoC intern last year and worked on Plotting Tool Project.

I would like to thank my mentor @adlarkin for guiding and helping me throughout the project.

Project Overview

The project is about developing a computer vision datasets generation feature for Ignition simulator, so that users can generate datasets to train machine learning / deep learning models for computer vision applications, The project includes developing sensors for semantic & instance segmentation datasets and 2D / 3D object detection datasets.

Segmentation Camera Sensor

Segmentation sensor provides semantic and panoptic / instance segmentation images from the simulation world data, And publishes them via ign-transport msgs. It provides both colored map (for visualization) and labels map (for ML training usage).

Semantic Segmentation

In the semantic image, objects with the same label have the same color in the colored map and the same label id in the labels map

Panoptic / Instance Segmentation

In the panoptic image, each pixel has 3 values, one for label id and two for instance count of that label. So each object has a unique color in the colored map and a unique 3 values in the labels map.

Bounding Box Camera Sensor

Bounding box sensor provides 2D & 3D bounding boxes for object detection datasets and publishes them via ign-transport msgs.

2D Boxes

There are 2 types of 2D bounding boxes, a visible box which is a box around the visible part of the
object, and a full box, which is a box around the whole object even if it has an invisible part.

Visible Box (datasets ex: COCO & PASCAL VOC)

Full Box (datasets ex: KITTI & nuScenes)

3D Boxes

Oriented 3D bounding boxes in camera coordinates

SDF Models Annotation

Users should annotate the models (give them a label) in the SDF world, so that the sensors can see them, as unlabeled objects will be considered as a background.

Example of the SDF format people will use to label a model from fuel models:

      <plugin filename="ignition-gazebo-label-system" name="ignition::gazebo::systems::Label">

Models annotation / labeling is explained briefly in the tutorials.

Dataset Generation

To generate dataset samples from these sensor, we add the <save> tag to the <camera> tag of the sensor, and it will save the dataset samples in the given path.



There are tutorials to explain briefly how to use these sensors, how to annotate the models and how to generate the datasets.

Pull Requests

Used Technologies

  • C++
  • OGRE rendering engine
  • gtest for unit / integration testing

Future Work

  • APIs to generate samples automatically by positioning objects and sensors in random positions
  • Depth map datasets generation


The Machine learning extension to Ignition provides users with segmentation and object detection datasets from the simulation for computer vision applications in robotics.

Amr Elsersy


Hello AmrElsery, @adlarkin, @chapulina et all. Fantastic work. It is really good to see the direction of sensor capabilities development here. Any plans to add point cloud ground truth data on the roadmap?


Hi cbuscaron, yeah I am thinking about adding it to the 3D dataset, so that the sample of the dataset becomes (Image + Point cloud + 3D boxes + some calibration matrices to map lidar-camera coordinates)

Tutorial link to the segmentation sensor is currently borken :slight_smile: .

Here’s the live tutorial: Ignition Sensors: Segmentation Camera in Ignition Gazebo