gstreamer nvh264enc deadlock and crashed
Submitted by alan
Assigned to Spice Bug List
Link to original bug (#103417)
Description
Spice server version 0.13.90 Host OS: Ubuntu 16.04 Guest OS: Windows 7
For hardware accelerate, I use GStreamer nvh264enc in place of xh264enc. However, it often get deadlock when GStreamer get restarted. I debugged the gstreamer plugin nvenc and I find the problem in the free_pipeline() and create_pipeline()。 Because there is no mutex lock to protect the process create/free pipeline,when the pipeline status switched in a short time,e.g start -> stop -> start, the nvenc would be in unknown status. In my case, it causes the deadlock inside the nvenc plugin.
I try to fix this bug in a simple way. In the configure_pipeline(), call free_pipeline() before create_pipeline(). This will make sure that we do free the resource of pipeline and it's safe even if the encoder->pipeline is NULL.
The source code is below. Here is the only way to call create_pipeline() except the gstreamer_encoder_new(), but it's like a initialized constructor, so we just take care about the configure_pipeline() which takes responsibility for switching the status of the pipeline.
static gboolean configure_pipeline(SpiceGstEncoder *encoder) { if (!encoder->set_pipeline) { return TRUE; }
free_pipeline(encoder->pipeline);
if (!encoder->pipeline && !create_pipeline(encoder)) {
return FALSE;
}
...
}
I can't guarantee that it doesn't bring in other bugs. I think there should be a more grace method to fix this problem.