frontend/nine: Fog and pointsize fixes
Some corrections to the pointsize code that came up when working on the code (but has no know bug it fixes). In addition a rework of pixel shader fog handling (see commit message).
I'm not adding to stable because of the risk of regressions.
Fixes: #8341 (closed)