Hello and welcome
what OS are you using ?
is the game itself starting up to the second launcher ?
when you did try to start the dedicted server, are there being files created in ./starmade/logs ?
(you could savely delete everything from the logs folder to check that)
in the first launcher under Options did you change the memory settings for dedicated server ?
looks like this
if you did change it to something above 1024 on a 32 bit OS (or while using a 32 bit Java) it may crash.