GSoC 2022: Potential project ideas

[Thodoris1999]

Hello everyone!

I’ve been using ROS 2 and gazebo pretty much every day for the past two years for Aristurtle’s autonomous racecar and in light of GSoC 2022 I think this is a very nice opportunity to give something back by contributing code in a project this summer.

While working with ROS, I have picked up several ideas for improvements/extensions along with the current project proposals, but I can submit only a finite amount of applications that I can invest effort on. For that reason, I am starting this thread to receive some feedback for my proposals, as they have probably already been discussed either openly or privately. It’s nice after all to work on something that benefits the wider community rather than something that I just find neat.

This is by no means a comprehensive analysis of each project, just a case study to potentially spark some conversation on the importance/priority/viability of them. Perhaps I will discuss anything about OSRF’s projects in another thread. Please feel free to point out any short-sightedness in my proposals.

Convenient bag processing tool
Bags are probably the most central file format in ROS. However, there is no GUI tooling (to my knowledge) for quickly and easily manipulating them. For us it was common to want to perform edits on our bags, whether it was for reducing size by removing a topic with high throughput or transforming a topic so that its data matched a subscriber with a non-matching interface in CI (for example one that uses different coordinate conventions), without creating re-publishers. If we really wanted to alter a rosbag, we had to do it using the undocumented python/C++ API. What would be helpful in this situation is a tool that supports:

• The ability to remove topics from bag
• Cutting rosbags from and to specified timestamps.
• The ability to transform topic messages with python scripts (similar to plotjuggler)
• QoS conversions
• Exporting surgery instructions in simple file format
• Run surgery instructions with command line tool, e.g. ros2 bag_surgery -i data.bag -f conf.surgery -o new_data.bag (this is very useful for batch-patching many bags)
• Provide this functionality in a GUI

ros2 bag convert supports part of that functionality in CLI. The GUI part could be implemented on the existing rqt_bag port for ROS2. This project is similar to facontidavide’s rosbag_editor for ROS1, but with more functionality and for ROS2.

Diagnostics
Diagnostics collection and management is something that almost every robotics project has to worry about, yet to me it’s something that is general enough that can be abstracted to the point that many projects can use common packages, similarly to how the diagnostics stack+robot_monitor worked in ROS1. Currently there exists a port of the diagnostics packages in ROS2 but it is not released yet. A summer project could look into what needs to be refined in order to integrate the diagnostics packages to release the stack in the desktop variant.

Documentation
With a project as large as ROS, it is hard to say no to more documentation. The main things I would like to see more documentation/examples/demos would be on

• Life cycle
• Time concepts
• XML launch
• Executors
• Composition
• Ament_cmake, ament_package, testing
  along with the many suggested issues on github. I am not sure however if this adds up to a full time project

My ideas are ROS related, but I am posting this here as mentioned in OSRF’s GSoC guidelines, so please let me know if ROS discourse is a more appropriate place for this thread.

I guess as a first post after lurking around this is also my introduction to this community. Thank you to everyone for contributing to this community and making it what it is now!

[Thodoris1999]

Hello again.
I decided to take a closer look at ign_rviz. I am trying to add the PoseWithCovariance display in this branch. However, I am encountering some strange behaviour.
I am currently adding a simple covariance visual, with no scalings and offsets yet so I do not need ROS messages to test rendering. The display for now just calls the constructor shown here:

CovarianceVisual::CovarianceVisual(ignition::rendering::VisualPtr parent_node, CovarianceUserData user_data) : user_data_(user_data) 
{
  this->scene_ = parent_node->Scene();
  this->root_visual_ = this->scene_->CreateVisual();
  parent_node->AddChild(this->root_visual_);

  this->fixed_orientation_visual_ = this->scene_->CreateVisual();
  this->root_visual_->AddChild(this->fixed_orientation_visual_);

  this->position_root_visual_ = this->scene_->CreateVisual();
  if (this->user_data_.position_frame == Frame::Local)
    this->root_visual_->AddChild(this->position_root_visual_);
  else
    this->fixed_orientation_visual_->AddChild(this->position_root_visual_);

  this->orientation_root_visual_ = this->scene_->CreateVisual();
  if (this->user_data_.orientation_frame == Frame::Local)
    this->root_visual_->AddChild(this->orientation_root_visual_);
  else
    this->fixed_orientation_visual_->AddChild(this->orientation_root_visual_);

  this->createMaterials();

  this->position_visual_ = this->scene_->CreateVisual();
  this->position_visual_->SetInheritScale(false);
  this->position_ellipse_ = this->scene_->CreateSphere(); 
  this->position_visual_->AddGeometry(this->position_ellipse_);
  this->position_visual_->SetMaterial(this->materials_[kPos], false);
  this->position_root_visual_->AddChild(this->position_visual_);
  this->position_visual_->SetLocalPosition(1.0 * math::Vector3d::UnitZ);
  this->position_visual_->SetLocalRotation(math::Quaterniond(0, 0, 0));

  for (int i = 0; i < kNumOrientationShapes; i++) {
    // Visual to position and orient the shape along the axis. One for each axis.
    this->orientation_visuals_[i] = this->scene_->CreateVisual();
    // Does not inherit scale from the parent.
    // This is needed to keep the cylinders with the same height.
    // The scale is set by setOrientationScale()
    this->orientation_visuals_[i]->SetInheritScale(false);

    if (i != kYaw2D) {
      this->orientation_shapes_[i] = this->scene_->CreateCylinder();
    } else {
      this->orientation_shapes_[i] = this->scene_->CreateCone(); // TODO: use pie slice instead
    }
    this->orientation_visuals_[i]->AddGeometry(this->orientation_shapes_[i]);
    this->orientation_visuals_[i]->SetMaterial(this->materials_[i+1], false);
    this->orientation_root_visual_->AddChild(this->orientation_visuals_[i]);
  }
  
  // Position the cylinders at position 1.0 in the respective axis, and perpendicular to the axis.
  double offset = user_data_.orientation_offset;
  // x-axis (roll)
  orientation_visuals_[kRoll]->SetLocalPosition(offset * math::Vector3d::UnitX);
  orientation_visuals_[kRoll]->SetLocalRotation(math::Quaterniond(math::Vector3d::UnitY, math::Angle::HalfPi.Radian()));
  // y-axis (pitch)
  orientation_visuals_[kPitch]->SetLocalPosition(offset * math::Vector3d::UnitY);
  orientation_visuals_[kPitch]->SetLocalRotation(math::Quaterniond(math::Vector3d::UnitX, math::Angle::HalfPi.Radian()));
  // z-axis (yaw)
  orientation_visuals_[kYaw]->SetLocalPosition(offset * math::Vector3d::UnitZ);
  orientation_visuals_[kYaw]->SetLocalRotation(math::Quaterniond(0, 0, 0)); // no rotation needed
  // z-axis (yaw 2D)
  orientation_visuals_[kYaw2D]->SetLocalPosition(cone_origin_to_top * math::Vector3d::UnitX);
  orientation_visuals_[kYaw2D]->SetLocalRotation(math::Quaterniond(math::Vector3d::UnitZ, math::Angle::HalfPi.Radian()));
}

I am expecting to see one sphere, 3 cylinders and a cone when adding my display. However I see nothing. What’s perhaps weirder is that when I first add another display that draws something on the screen (e.g. AxesDisplay) before PoseWithCovariance display, something does seem to work. I do see the cylinders and the cone, and I can interact with them using their settings, but never the sphere. Even if I just change the geometries passed to orientation_visuals_ from Cylinder/Cone to Sphere, I can’t get any spheres to be drawn.
I am also getting some unpredictable glitching and crashing. Here is the gdb backtrace:

0x00007fff9e9a8f54 in Ogre::Math::calculateBasicFaceNormalWithoutNormalize(Ogre::Vector3 const&, Ogre::Vector3 const&, Ogre::Vector3 const&) ()
   from /lib/x86_64-linux-gnu/libOgreMain.so.1.9.0
(gdb) bt
#0  0x00007fff9e9a8f54 in Ogre::Math::calculateBasicFaceNormalWithoutNormalize(Ogre::Vector3 const&, Ogre::Vector3 const&, Ogre::Vector3 const&) ()
    at /lib/x86_64-linux-gnu/libOgreMain.so.1.9.0
#1  0x00007fff9e9a90a5 in Ogre::Math::intersects(Ogre::Ray const&, Ogre::Vector3 const&, Ogre::Vector3 const&, Ogre::Vector3 const&, bool, bool) ()
    at /lib/x86_64-linux-gnu/libOgreMain.so.1.9.0
#2  0x00007fff9eedcc3f in ignition::rendering::v6::OgreRayQuery::ClosestPoint() (this=0x7fff64125950)
    at /home/thodoris/src/ignition/src/ign-rendering/ogre/src/OgreRayQuery.cc:123
#3  0x00007fffc4f2e415 in ignition::gui::plugins::IgnRenderer::ScreenToScene(ignition::math::v6::Vector2<int> const&) const (this=0x55555634ec80, _screenPos=...)
    at /usr/include/c++/9/bits/shared_ptr_base.h:1020
#4  0x00007fffc4f2e5b9 in ignition::gui::plugins::IgnRenderer::BroadcastHoverPos() (this=<optimized out>)
    at /home/thodoris/src/ignition/src/ign-gui/src/plugins/scene3d/Scene3D.cc:1039
#5  0x00007fffc4f38516 in ignition::gui::plugins::IgnRenderer::HandleMouseEvent() (this=0x55555634ec80)
    at /home/thodoris/src/ignition/src/ign-gui/src/plugins/scene3d/Scene3D.cc:918
#6  0x00007fffc4f38649 in ignition::gui::plugins::IgnRenderer::Render() (this=0x55555634ec80) at /home/thodoris/src/ignition/src/ign-gui/src/plugins/scene3d/Scene3D.cc:901
#7  0x00007fffc4f3954e in ignition::gui::plugins::RenderThread::RenderNext() (this=0x55555634ec40) at /home/thodoris/src/ignition/src/ign-gui/src/plugins/scene3d/Scene3D.cc:1292
#8  0x00007ffff5f5dc2a in QObject::event(QEvent*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#9  0x00007ffff694ba66 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Widgets.so.5
#10 0x00007ffff69550f0 in QApplication::notify(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Widgets.so.5
#11 0x00007ffff5f3180a in QCoreApplication::notifyInternal2(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#12 0x00007ffff5f34488 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#13 0x00007ffff5f89e37 in  () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#14 0x00007ffff42e817d in g_main_context_dispatch () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x00007ffff42e8400 in  () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007ffff42e84a3 in g_main_context_iteration () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007ffff5f89435 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#18 0x00007ffff5f303ab in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#19 0x00007ffff5d68785 in QThread::exec() () at /lib/x86_64-linux-gnu/libQt5Core.so.5 
#20 0x00007ffff5d699d2 in  () at /lib/x86_64-linux-gnu/libQt5Core.so.5
#21 0x00007ffff6e84609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#22 0x00007ffff59db163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Any clues or better methodology in debugging such cases would be greatly appreciated. Also, regarding node visibility, is it inherited (nodes with at least one invisible parent are invisible regadless of their own value)? And also what is the bit packing of the visibility flags?

[Thodoris1999]

Problem solved! I should have made all my scene modifications inside render/preRender events…

[Thodoris1999]

PoseWIthCovariance is now implemented in ign_rviz!
It is the same as the regular pose display, but uses some additional shapes to visualize the pose’s uncertainty. I followed the approach already used by RViz, so should look familiar to new users.

• A 3D ellipsoid visualizes the position covariance, representing the position inside 2 standard deviations of error.
• 3 2D ellipsoids (one on each axis) visualize the orientation uncertainty. For small covariances, it approximately represents the ellipse that the each unit vector stays inside within two standard deviations from pure rotation.
  I noticed some errors on some orientation ellipses in RViz, as described in this issue and tested to make sure they are correct in the new implementation.
  Example 1:
  ros2 topic pub /test geometry_msgs/msg/PoseWithCovarianceStamped '{header: {frame_id: "world"}, pose: {pose: {position: {x: 1, y: 1, z: 1}, orientation: {x: 0, y: 0, z: 0.4794255, w: 0.8775826}}, covariance: [1, 0.5, 0, 0, 0, 0, 0.5, 1, 0, 0, 0, 0, 0, 0.1, 0.3, 0, 0, 0, 0, 0, 0, 0.05, 0, 0, 0, 0, 0, 0, 0.2, 0, 0, 0, 0, 0, 0, 0.2]}}'
  So large orientation covariances on pitch and yaw but small on roll. On RViz in looks like:
  
  
  
  notice that the Z axis ellipse should be twisted by 90 degrees, since the roll uncertainty is small. This is fixed now:
  
  
  
  Example 2:
  ros2 topic pub /test geometry_msgs/msg/PoseWithCovarianceStamped '{header: {frame_id: "world"}, pose: {pose: {position: {x: 1, y: 1, z: 1}, orientation: {x: 0, y: 0, z: 0.4794255, w: 0.8775826}}, covariance: [1, 0.5, 0, 0, 0, 0, 0.5, 1, 0, 0, 0, 0, 0, 0.1, 0.5, 0, 0, 0, 0, 0, 0, 0.05, 0, 0, 0, 0, 0, 0, 0.2, 0.1, 0, 0, 0, 0, 0.1, 0.2]}}'
  So now yaw error is positively correlated with pitch error. X axis ellipse becomes oriented accordingly.
  
  
  
  Example 3
  ros2 topic pub /test geometry_msgs/msg/PoseWithCovarianceStamped '{header: {frame_id: "world"}, pose: {pose: {position: {x: 1, y: 1, z: 1}, orientation: {x: 0, y: 0, z: 0.4794255, w: 0.8775826}}, covariance: [1, 0.5, 0, 0, 0, 0, 0.5, 1, 0, 0, 0, 0, 0, 0.1, 0.5, 0, 0, 0, 0, 0, 0, 0.05, 0, 0.05, 0, 0, 0, 0, 0.2, 0, 0, 0, 0, 0.05, 0, 0.2]}}'
  Positive correlation between roll and yaw. Y axis ellipse becomes oriented accordingly.
  
  
  
  Example 4
  ros2 topic pub /test geometry_msgs/msg/PoseWithCovarianceStamped '{header: {frame_id: "world"}, pose: {pose: {position: {x: 1, y: 1, z: 1}, orientation: {x: 0, y: 0, z: 0.4794255, w: 0.8775826}}, covariance: [1, 0.5, 0, 0, 0, 0, 0.5, 1, 0, 0, 0, 0, 0, 0.1, 0.5, 0, 0, 0, 0, 0, 0, 0.05, 0.05, 0, 0,0, 0, 0.05, 0.2, 0, 0, 0, 0, 0, 0, 0.2]}}'
  Positive correlation between roll and pitch. Z axis ellipse becomes oriented accordingly.
  
  
  
  Example 5
  ros2 topic pub /test geometry_msgs/msg/PoseWithCovarianceStamped '{header: {frame_id: "world"}, pose: {pose: {position: {x: 1, y: 1, z: 1}, orientation: {x: 0, y: 0, z: 0.4794255, w: 0.8775826}}, covariance: [1, 0.5, 0, 0, 0, 0, 0.5, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.2]}}'
  The 3rd, 4th and 5th items in the diagonal are zero. RViz treated this as a special case, flattening everything and using a flattened cone for the yaw variance. I copied the same behavior on my implementation:
  
  
  
  RViz:
  
  
  
  However, on my opinion, a circular “pie slice” would fit better and be able to represent deviations larger than 90 degrees, but (to my knowledge) no such mesh exists yet in ignition gazebo rendering.
  As for the shape configurations, everything supported in RViz, is supported here:
• Visibility toggles
• Shape color (including alpha and choice between using a specified color and RGB for each axis)
• Shape scales
• orientation ellipse offsets
  RViz offered to visualize orientation covariances expressed in the fixed and local frame, configurable with a dropdown menu. However the position ellipse (in purple) was always implicitly expressed in the fixed frame which I found confusing since the default for orientation was local frame. In ign_rviz, I made it so both position and orientation visuals are configurable for local and fixed frame. Here are some examples showing the difference visually
  Position cov w.r.t. local, rotation cov w.r.t. local:
  
  
  
  Position cov w.r.t. local, rotation cov w.r.t. fixed:
  
  
  
  Position cov w.r.t. fixed, rotation cov w.r.t. fixed:
  
  
  
  Position cov w.r.t. fixed, rotation cov w.r.t. local:
  
  
  
  Now that covariance visuals are implemented, it should be fairly easy to port Odometry as well!

Some issues I caught during implementation

• Choosing a topic that has been used in another plugin with a different message type causes a crash since create_subscription can’t use a topic with the wrong message type (this also crashes RViz).
• The arrow of PoseDisplay sometimes causes parts of the window to turn black. I can’t reproduce this reliably yet.
  I am also noticing a lot of duplicate code in the plugins package. As the codebase matures, it would be nice to have common code for properties and property UI, similar to RViz. For PoseWithCovariance, my QML was written with no custom QML types (except QoS config) and the file ended up over 700 LOCs even though it is very similar to Pose and it uses the same common field types (dropdown, textedit etc). This is also dangerous and hinders maintenance since the same magic numbers exist in every plugin separately. I also manually did all my UI-to-C++ data bindings.
  I will be thinking about a fitting architecture that simplifies and minimizes boilerplate plugin creation, while not being restrictive and allowing customization. If anyone wants to discuss this with me, my email is teotyrov@gmail.com.

[ahcorde]

Hi @Thodoris1999,

Super great contribution! I started to review your PRs. I will give you feedback in the PR

Feel free to open issues in the repository to discuss deeper about improvements.