Even with zero Armor HP, having a thick enough layer of armor where you expect your ship to be hit from (usually the nose on more maneuverable ships) is a huge deal. Some of the worst nightmares of ships I've fought against have basically been flying bricks with over 20 meters of advanced armor on the nose and around 4 to 5 meters of advanced armor on the sides. Trying to chew through the nose of those bricks of armor is better than trying to shoot through the sides, though, since if the ship in question is facing you, you're going to have to go through far more side armor than frontal armor because of the angle your shots will be hitting at.
I've found that heavily armored ships are an absolute nightmare to kill unless you have strong punchthrough meta guns designed exclusively for beating armor. It gets worse when the armor tank uses docked armor plates, ion effects, and shield drain beams. Add efficient point defense to the design and nothing's getting through easily. Internal armor protects vital system blocks from being destroyed and the most outright infuriating ships I've had the misfortune of fighting had their internals sandwiched between layers of advanced armor like lasagna, so my shots were hitting armor the entire time they were going through the target.
The whole "a good offense is the best defense" thing doesn't work out too well when your opponent's defenses more or less nullify your offense. The quickest way to kill an extremely heavy armor tank is with a single enormous missile+pulse+piercing system (which can and will be shot down most of the time), followed by punchthrough meta guns (which chew through armor HP, and a well designed cannon can penetrate all the way through the target with no trouble). Punchthrough meta guns tend to be somewhat delicate and taking damage to your weapons system will completely wreck your ability to do damage, though.
Starmade combat is a bit of a rock-paper-scissors thing. Every ship design can be countered by other purpose built ship designs. Shield capacity tanks are common and get splattered instantly half the time because of the prevalence of Beam+Pulse+Ion, Beam+Beam+Ion, etc. Shield regen tanks are less common and get splattered by high alpha weapons like missiles. Armor tanks are uncommon and most reliably get splattered by highly mobile glass cannons focused on hull damage, or by better armor tanks with anti-armor weapon designs.
Of course... Armor is expensive, and it takes a lot of effort to put armor tanks on the field. If an armor tank gets structure damage, it'll probably have to go home to be scrapped and replaced, which gets really expensive over time, but isn't as expensive as having to replace the entirety of a dead shield tanked ship.