That random targeting, unpredictable nature of swarmers is what keeps them balanced. Other missiles can be shot down, but with swarm numbers, that becomes unfeasible. They need to be easily diverted.
I use swarmers myself for most of my workhorse builds, and I think the balance is just right. The only problem right now is the relative ineffectiveness of point defense due to tracking issues, and the propensity for server admins to boost maximum speeds, which seriously increases the power of missiles, as it makes them that much harder to shoot down.