Spawning the wormholes in areas near the activity but not in it would help, also being at least a few sectors away from planets, stars, stations, players, and anything that has to move through unloaded space. Since they only last for a few seconds to a few hours the inconvenience wouldn't last...