GSoC 2021: Machine Learning Extensions to Ignition Gazebo

Projects
7

Hey Amr,

Thanks for expressing interest in GSoC at Open Robotics this year. I’m glad to hear that last year’s program was a positive experience for you! I am going to mention a list of things that I believe would be useful for preparing/familiarizing yourself with the tools/concepts that may be relevant to the Machine Learning Extensions project (I also hope that this list will be helpful to other candidates who are interested in learning more about this project):

  1. Familiarity with ign-rendering, since we will probably need to produce a semantic image (colored image) with a mapping of pixel to label type (take a look at point #5 below for more information/resources regarding semantic segmentation).
    • Going through the ign-rendering tutorials would be useful. You can work through the examples as well if you’d like to gain greater understanding of how the code works.
    • Most of the rendering work will be done with the Ogre rendering engine (specifically, version 2.1). So, it would also be good to go through some Ogre tutorials, and also take a look at how Ogre2 is used in ign-rendering.
    • Taking a look at the Ogre2SelectionBuffer may provide some ideas for how to produce a semantic image with colored pixels for each label.
  2. Familiarity with ign-gazebo. At a minimum, I’d recommend going through the following tutorials (if you’d like to go through more, go for it!):
    • Terminology
    • Create System Plugins
    • Rendering Plugins
    • GUI Configuration
    • Server Configuration
  3. Familiarity with ign-sensors - in particular, how the current camera sensors work there.
  4. Knowledge of data-related concepts for machine learning:
    • Training data vs validation data vs test data
    • Features and labels
  5. Knowledge of other machine learning concepts:
    • semantic segmentation (the Cityscapes Dataset is a good example of this), including object recognition vs scene understanding
    • bounding boxes
  6. Knowledge of how data is formatted for common/popular machine learning libraries (for example: PyTorch, Tensorflow, Scikit-Learn)
  7. Familiarity with ign-gui and Qt for building a GUI in Ignition Gazebo that customizes how data is generated (there may not be time for this)

I hope this is helpful. If you have not officially applied to the Open Robotics GSoC program yet, please do so - applications are now open, so we will be reviewing them over the next few weeks.

2 Likes