aggregator: Don't clear the pad's segment on flush
Normally, the segment field is only updated before aggregate is called, but a flush would clear it immediately. This messes with subclasses trying to use the segment after popping a buffer.
Leaving it around should be safe.
Depends on !608 (merged) (for the test) and !611 (merged).