One iceloop thread consumes about 80% single core CPU
Janus uses libnice. The thread has been identified. However, neither Janus nor libnice log messages tell what it is doing differently from other iceloop threads. The only interesting logging is that Janus logs more than 10 ICE restart detected as shown below.
Used callgrind to profile it, for about 5 minutes, this thread calls component_io_cb 10,787,769 times while other threads call at most 36,248 times ( 10,788,769 / 36,248 = 297.6), The only difference I can think of is this thread is doing both audio and video. Others may do audio only. however, we cap the video bitrate to around 500kbps. it should not make such a big difference.
Any idea? Is it possible ice restart will attach multiple sockets ( fails to detach old socket ) to the context?
Row average_bps is_audio_only
1 53478.6949596202 true
2 435241.9845097517 false
Feb 25 22:39:21 ip-10-248-115-34.ap-southeast-1.compute.internal-1 janus: Container Linux by CoreOS 1967.5.0 (Rhyolite) stable i-04117cf80523bbdc1 c5.large default-hvm ap-southeast-1a 52.77.249.234 [WARN] [7706291932605316] Didn't receive audio for more than 1 seconds... Feb 25 22:39:21 ip-10-248-115-34.ap-southeast-1.compute.internal-1 janus: Container Linux by CoreOS 1967.5.0 (Rhyolite) stable i-04117cf80523bbdc1 c5.large default-hvm ap-southeast-1a 52.77.249.234 [WARN] [7706291932605316] Didn't receive video for more than a second... Feb 25 22:39:21 ip-10-248-115-34.ap-southeast-1.compute.internal-1 janus: Container Linux by CoreOS 1967.5.0 (Rhyolite) stable i-04117cf80523bbdc1 c5.large default-hvm ap-southeast-1a 52.77.249.234 [WARN] [7706291932605316] ICE restart detected