Convert fetch commands to async
The improvement depends on the number of recipes, network capabilities and CPU power. This change is especially useful when there is nothing new to be actually downloaded. For instance, the $CERBERO $CERBERO_ARGS fetch-package --deps gstreamer-1.0 command takes way less time on CI for this branch than the one running on master when all packages are up to date.
https://gitlab.freedesktop.org/gstreamer/cerbero/-/jobs/251944
$ $CERBERO $CERBERO_ARGS fetch-package --deps gstreamer-1.0
WARNING: Running as root
0:00:00 Loading default configuration from /root/.cerbero/cerbero.cbc
0:00:00 WARNING: Could not recover status
0:00:00 Fetching the following recipes: libffi zlib glib libunwind gobject-introspection gstreamer-1.0 libxml2 libogg libpng pixman expat bzip2 freetype fontconfig cairo fribidi harfbuzz pango libvorbis libtheora libvisual orc opus graphene libjpeg-turbo cdparanoia gst-plugins-base-1.0 gst-shell speex tiff gdk-pixbuf gmp nettle libtasn1 gnutls glib-networking libsoup mpg123 lame wavpack flac taglib libvpx libdv gst-plugins-good-1.0 libass libkate openh264 librtmp libsrtp libdca libmms libdvdread libdvdnav libnice soundtouch vo-aacenc libcroco librsvg openjpeg openssl spandsp webrtc-audio-processing sbc ladspa srt libusrsctp gst-plugins-bad-1.0 a52dec opencore-amr x264 gst-plugins-ugly-1.0 json-glib gst-rtsp-server-1.0 ffmpeg gst-libav-1.0 gst-validate gst-editing-services-1.0
0:00:00 [(1/78) libffi -> Fetch ]
...
0:00:43 [(78/78) gst-editing-services-1.0 -> Fetch ]https://gitlab.freedesktop.org/pamarcos/cerbero/-/jobs/264049
$ $CERBERO $CERBERO_ARGS fetch-package --deps gstreamer-1.0
WARNING: Running as root
0:00:00.001968 Loading default configuration from /root/.cerbero/cerbero.cbc
0:00:00.018496 WARNING: Could not recover status
0:00:00.498276 Fetching the following recipes using 96 async job(s): libffi zlib glib libunwind gobject-introspection gstreamer-1.0 libxml2 libogg libpng pixman expat bzip2 freetype fontconfig cairo fribidi harfbuzz pango libvorbis libtheora libvisual orc opus graphene libjpeg-turbo cdparanoia gst-plugins-base-1.0 gst-shell speex tiff gdk-pixbuf gmp nettle libtasn1 gnutls glib-networking libsoup mpg123 lame wavpack flac taglib libvpx libdv gst-plugins-good-1.0 libass libkate openh264 librtmp libsrtp libdca libmms libdvdread libdvdnav libnice soundtouch vo-aacenc libcroco librsvg openjpeg openssl spandsp webrtc-audio-processing sbc ladspa srt libusrsctp gst-plugins-bad-1.0 a52dec opencore-amr x264 gst-plugins-ugly-1.0 json-glib gst-rtsp-server-1.0 ffmpeg gst-libav-1.0 gst-validate gst-editing-services-1.0
0:00:00.498356 [(1/78) libffi -> async fetch started]
...
0:00:00.586601 [(78/78) gst-editing-services-1.0 -> async fetch started]
0:00:05.381565 All async fetch jobs finishedThe --jobs/-j parameter allows to specify the number of parallel jobs to run. By default, only 2 jobs will be used:
./cerbero-uninstalled -t fetch // uses 2 jobs
./cerbero-uninstalled -t fetch-package -j // uses twice the number of logical cores
./cerbero-uninstalled -t fetch -j8 // uses 8 jobs
./cerbero-uninstalled -t fetch-bootstrap -j1 // uses 1 job
./cerbero-uninstalled -t fetch-package -j4 // uses 4 jobs
I've also added the milliseconds to the timestamp shown when -t is used to improve the precision a little bit since I'm planning on working on more performance improvements and I find it quite useful.
As a final note, I've refactored log_step_output so that it has two different flavors: one for async steps and another one for normal steps. Before, the code was effectively awaiting on every call to log_step_output, since it was an async method. This did not make any sense for non-async steps such as extract and resulted in undesired results such as not being able to call async methods from within since we can't queue tasks to a loop within an event loop. As a result of this change, the stepfunc() calls are simplified to just distinguish between async and sync ones. No more need to use stepfunc(recipe) for async recipes but stepfunc() for universal ones.