Mmm... I'm gonna assume that when you turn around (facing Galactic South), You are viewing more rendered Chunks o -o
The lag fluctuates depending on what is on screen (So you get less Lag when looking away from stuff). Could be there is Something in the sector behind you, or your ship is Back heavy? :p
Edit: Try adjusting your settings or reducing Max rendered Segments?