Is it a completely fresh install?
Your PC is certainly powerful enough to play on full settings, but just to check a few things could you try the following please?
Place the Starmade folder + launcher in the C\: root directory.
Make sure nvidia's control panel options are default for the game. (sometimes you can force AA or other quality options, turn these off if they are one)
In the first launcher window, click the options toolbar, then open Memory Settings. Make sure they are default values. Max 1024, Initial, 512, Early 128.
In the launcher options check the following;
Max view segments; 100
normal mapping; off
shadows; off
procedural background; off
texture pack; pixel : 64
Then in advanced options;
anti aliasing; off
bloom; off
VBO; off
Vertical sync: on
Enable background; off
If it still bugs out, at least we can rule out a number of graphical options in the game.