Service Outage of 3-26-15

    Nuclear Doughnut

    A Radioactive Pastry
    Joined
    Jul 2, 2013
    Messages
    157
    Reaction score
    136
    • Purchased!
    • Legacy Citizen 10
    So after receiving many complaints from members of both of the servers I run I have come here to ease everyone's fear of losing their work on the servers. As of Yet There hasn't been an explanation of the outage anywhere I can find so after some investigating with other Server Owners I hereby give you a rundown.

    1. The servers themselves were unaffected by today's SM General Outage. All your data is safe.
    2. Our hardworking Starmade Dock Staff let the SSL (Secure Sockets Layer) Certificate Expire. As most services give you MONTHS in advance notice about your SSL Certificate. This may have been a planned issue. Why they didn't inform the community is unclear at this time.
    If you don't know what SSL is here's a basic definition from Google.
    It provides a secure connection between internet browsers and websites, allowing you to transmit private data online. Sites secured with SSL display a padlock in the browsers URL and possibly a green address bar if secured by an EV Certificate.
    What does this SSL Certificate affect?
    1. Authentication on ALL Servers
    2. Starmade Registry
    3. Starmade Chat
    4. Starmade Forums (which is what we're supposed to use to contact the devs in case of a serious issue)

    This indecent has seemed to have taken out our whole community at the tick of a clock. Perhaps we need to decentralize parts of the system to ensure not such another large failure of the system.

    Some would liken the SSL to a websites' "Power Bill" so basically All of Starmade went down because the Power Bill wasn't paid.

    Unlike the Power Company. We weren't notified of this "Brownout" of service.

    With Regards from the Server Operators/Community

    Nuclear Doughnut
     
    Joined
    Jul 21, 2013
    Messages
    2,932
    Reaction score
    460
    • Hardware Store
    4. Starmade Forums (which is what we're supposed to use to contact the devs in case of a serious issue)
    One does not need the SSL to access the forums, one only needs it to login, if one already was logged in, one should still be able to use it.

    Also, I didn't notice any outage, when exactly did it happen?
     

    CyberTao

    鬼佬
    Joined
    Nov 10, 2013
    Messages
    2,564
    Reaction score
    641
    • Legacy Citizen 4
    • Railman Gold
    • Thinking Positive
    One does not need the SSL layer to access the forums, one only needs it to login, if one already was logged in, one should still be able to use it.

    Also, I didn't notice any outage, when exactly did it happen?
    It happened today, dunno when it started, ended about an hour ago or so at most I want to say. Lasted a couple hours, I got to watch the forum online member count slowly dwindle.

    Also, keep in mind that people are supposed to be auto-logged out Mega, which makes the inability to log in a bit of a problem.
     

    Nuclear Doughnut

    A Radioactive Pastry
    Joined
    Jul 2, 2013
    Messages
    157
    Reaction score
    136
    • Purchased!
    • Legacy Citizen 10
    You couldn't log into the forums if you weren't on. So there was no real way to report this. I myself and others were unable to log into the SM registry (and forums by extension).

    8pm Eastern Aproximately
     
    Joined
    Jul 31, 2013
    Messages
    387
    Reaction score
    87
    • Purchased!
    • Community Content - Bronze 1
    • Legacy Citizen 4
    I've been on and off both my server and this site hourly all day. Likewise none of my players have reported any issues. I believe you may have had localized issues. Aside from that I can almost guarantee that they did not let their SSL cert lapse. If there WAS such an issue it is more likely related to switching certificate providers, but again, the evidence says otherwise.
     

    CyberTao

    鬼佬
    Joined
    Nov 10, 2013
    Messages
    2,564
    Reaction score
    641
    • Legacy Citizen 4
    • Railman Gold
    • Thinking Positive
    I believe you may have had localized issues.
    It was a tad bit too widespread to be "localized". Starmade servers showed that almost all servers with authentication when to very low numbers (since people could not log on, but you were fine so long as you didnt log out). The Site also went pretty damn vacant, 12 members and 112 'guests' at the time it was reported to Doomsider. Most of those people ended up in the IRC channel, where many people were asking about it.

    One of the possible fixes though, was rolling back the date on your computer so that SM think's that it's oauth token hadn't expired. Worked for a few people, maybe yours' didn't expire?

    Even GeneralEagleEye popped on to ask what was up with the registry, so even people on your server experienced some issues.
     
    Joined
    Aug 8, 2013
    Messages
    403
    Reaction score
    45
    • Legacy Citizen
    • Purchased!
    • Legacy Citizen 2
    I and two other members of Ragnarock were not able to log on for a while today, neither here or the server because of the registry issue. WHen I tried to access the registry directly, it gave me an unsafe site warning. I do hope the devs give an explanation soon so that we know what happened.
     

    AndyP

    Customer Experience Manager
    Joined
    Aug 15, 2013
    Messages
    1,199
    Reaction score
    264
    • Schine
    • Wired for Logic
    [...]
    4. Starmade Forums (which is what we're supposed to use to contact the devs in case of a serious issue)
    [...]
    Beside the fact we have quite a few backup communication systems to the different serverowners, the usual way to get help if all fails would be:

    http://help.star-made.org/

    This site has an independent login system,
    beside that
    support (at) star-made.org
    is a valid mail-address to drop a mail to and get a reply.
    This only needs a working mail-program or webmail account.

    The issue was indeed an expired SSL-Cert as everyone could see.
    They are issued for a relative long time span so the expiration got missed from our side, this was a mistake on our side.

    The outage lasted round about 3 hours until the new requested cert got approved and sent back to us.
    (actually the waiting for the new cert took most time of the outage, this is no excuse, but we were at least working on it only a few minutes after it expired.)

    However, next time this may happen is:
    26. March 2018

    So feel free to drop us a message if it is the 24th of March 2018 and we did not update the Cert yet. (See this as joke, we wont let it happen again.)

    - Andy
     
    Joined
    Jul 31, 2013
    Messages
    387
    Reaction score
    87
    • Purchased!
    • Community Content - Bronze 1
    • Legacy Citizen 4
    Color me completely wrong. Not too unusual. Good work getting it back up quickly Schine.
     

    Master_Artificer

    Press F to pay respects
    Joined
    Feb 17, 2015
    Messages
    1,588
    Reaction score
    612
    • Top Forum Contributor
    • Legacy Citizen 2
    • Thinking Positive
    I'm Back!

    And thanks cyber for saying hi in the chat for me while I was trapped alone, with hundreds of other people, outside of the wall.
     
    Joined
    Oct 5, 2013
    Messages
    61
    Reaction score
    86
    Just to add to AndyP 's reply, the SSL certificate did expire. It was simply a matter of us forgetting about it. Our certificate is a wildcard certificate (it is valid for *.star-made.org), and thus the re-validation for the renewal took longer than a standard SSL cert. We have renewed it for 3 years this time.

    This should be long enough for our extended validation certificate to be issued, at which point we shouldn't have any more SSL problems.
    [DOUBLEPOST=1427500125,1427499998][/DOUBLEPOST]
    So after receiving many complaints from members of both of the servers I run I have come here to ease everyone's fear of losing their work on the servers. As of Yet There hasn't been an explanation of the outage anywhere I can find so after some investigating with other Server Owners I hereby give you a rundown.

    1. The servers themselves were unaffected by today's SM General Outage. All your data is safe.
    2. Our hardworking Starmade Dock Staff let the SSL (Secure Sockets Layer) Certificate Expire. As most services give you MONTHS in advance notice about your SSL Certificate. This may have been a planned issue. Why they didn't inform the community is unclear at this time.
    If you don't know what SSL is here's a basic definition from Google.


    What does this SSL Certificate affect?
    1. Authentication on ALL Servers
    2. Starmade Registry
    3. Starmade Chat
    4. Starmade Forums (which is what we're supposed to use to contact the devs in case of a serious issue)

    This indecent has seemed to have taken out our whole community at the tick of a clock. Perhaps we need to decentralize parts of the system to ensure not such another large failure of the system.

    Some would liken the SSL to a websites' "Power Bill" so basically All of Starmade went down because the Power Bill wasn't paid.

    Unlike the Power Company. We weren't notified of this "Brownout" of service.

    With Regards from the Server Operators/Community

    Nuclear Doughnut
    Just to clarify, the Dock's certificate did not expire. It was the certificate for the Registry that expired. When you login to the Dock, you are using your Registry credentials. Because the Registry cert couldn't be trusted any longer (due to expiration), the dock was effectively taken offline as well for people to login and authenticate with.