gst-curlhttpsrc multiloop download isn't "paced" to the playback and downloads the whole file
In the current state of the code if we use a pipeline uridecodebin ! convert ! resample ! alsasink (uridecodebin using gst-curlhttpsrc and not soup) on a big file (for example a 500 MB mp3):
- The calls to the create function are paced by the playback output (so in our case the alsa ringbuffer size)
- The actual download is done asynchronously by gst_curl_http_src_curl_multi_loop based on data availability in the socket
In addition the create function allocates, copies and returns the whole buffer reallocated each time the curl writefunction and not the basesource blocksize.
In our example, the first create will return a big buffer, let's say for 2MB (depending on the dl speed). They will take some time to play.
During this time the calls to curl_multi_perform will download the complete file and then the next create() call will return a 498 MB buffer.
In an embedded device with limited ram, it means that during the create() we have the 498 MB buffer, then allocate a gst_buffer of 498 MB then copy the content of the first one in the second which requires 2*498 MB which might trigger a memory overcommit and cause in the best case an OOM-killer call and in my case a kernel oops :)
I have a poc (awaiting hierarchy validation for attachment) which reduces the issue by:
Using a fixed size ringbuffer in the GstHttpSrc structure (so no realloc/free) which create() will empty basesource.blocksize by basesource.blocksize.
Making sure that gst_curl_http_src_get_chunks returns CURL_WRITEFUNC_PAUSE when this ringbuffer would be overrun by the chunk.
Calling when this happens gst_curl_setopt_generic(s, s->curl_handle, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) 1); This set the download speed to 1B/s because, afaik, curl has no other way to "wait"
- Calling curl_multi_perform on a paused connection will only put bufferization step inside curl itself.
- Removing the easy handle from the multi handle tears down the connection and would require a CURLOPT_RESUME_FROM_LARGE when we add it back to the multi handle to work but this will be problematic for radio stream which don't accept the Range option.
When create() empties the ringbuffer enough, it removes the speed limitation on the easy handle with gst_curl_setopt_generic (elt, elt->curl_handle, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) 0);
In the multi loop thread, when I know a slowed connection has reached the rebufferization threshold I call curl_easy_pause(slowed_handle, CURLPAUSE_CONT) outside of the buffer_mutex lock since this function will call gst_curl_http_src_get_chunks which locks the buffer_mutex and to make sure the locking is proper enough.