Skip to content

rusoto: s3sink: Implement timeout/retry for part uploads

Arun Raghavan requested to merge arun/gst-plugins-rs:s3sink-retries into main

Rusoto does not implement timeouts or retries for any of its HTTP requests. This is particularly problematic for the part upload stage of multipart uploads, as a blip in the network could cause part uploads to freeze for a long duration and eventually bail.

To avoid this, for part uploads, we add (a) (configurable) timeouts for each request, and (b) retries with exponential backoff, upto a configurable duration.

It is not clear if/how we want to do this for other types of requests. The creation of a multipart upload should be relatively quick, but the completion of an upload might take several minutes, so there is no one-size-fits-all configuration, necessarily.

It would likely make more sense to implement some sensible hard-coded defaults for these other sorts of requests.

Merge request reports