Previously rtspsrc would set a 2s latency on rtpjitterbuffer. The problem with this is that it introduces a non-trivial latency, which cumulated with all the other sources of latency in a pipeline (decoders, parsers, ...) result in live playback being delayed by an unreasonable amount.
The whole point of the rtpjitterbuffer latency size is to be able to cope with "jitter" up to that point (i.e. the quality of your connection) and also provide some room for retransmission if activated. It is NOT to cope with the transmission delay, and it is NOT to cope with buffering.
If you don't have retransmission activated, a 200ms jitter should cover the vast majority of rtsp users. Buffering and transmission delay are handled by other elements in the pipeline.