Instead of relying on buffers after a state change to PLAYING to always start from 0 (which is not guarenteed), track the amount of time we have spent outside playing but not changed state to PAUSED.
When buffering, we may drop out of playing without fully committing to paused and once we move back to playing, would reset the base time and cause a large increase in the frame's scheduled time.