Gazebo plugin development and GitHub Actions for macOS

Has anyone experience or recommendations for setting up CI for macOS Monterey using GitHub Actions?

I have a colcon build and test for a plugin project working on a macOS Monterey runner, however it is very slow as the configuration of the runner takes about 30min (using brew to install gazebo garden and other build dependencies). For comparison the equivalent configuration of a standard Ubuntu machine is a few minutes. I have tried caching the Homebrew folders under /usr/local, which does sort of work, however you need to brew link any updated packages restored from the cache, which takes a while and largely defeats any gain you might get from this approach.