Starmade Server Clustering

[Lone_Puppy]

Here's a pipe dream I dreamed up after reading this post http://starmadedock.net/threads/galaxy-redesign-and-new-types-of-galaxy.21277/

A server clustering model would support this.

If you limited a server to generating a galaxy instead of a universe, you could have different galaxies operating on multiple servers.

This would require a central economic/cluster management server for these servers to synchronise with, much like VMware do with vSphere/ESX clusters and Citrix do with XenServer. It would cut down on processing an entire universe, leaving the processing distributed across multiple physical machines.

My earlier suggestion for an interserver warpgate or warp capability could be modified to accommodate a multi-galaxy/server cluster scenario if you had a Starmade server instance per galaxy. I see it operating the same way the separation between sectors works, but you engineer it to somehow seamlessly transition you from one server to another, kind of how vMotion works in VMware's vSphere product. Pretty much Starmade server clustering is where I'm heading, where the ecomonic and most likely entity positioning server would be the VMware vCenter equivilant, handling the multiple hosts/galaxies.

Example Components;

Starmade Server Cluster made of...

Starmade Centre or Managment Server
- Keeps track of GPS of galaxies within a Starmade Server cluster
- Keeps track of inter-galaxy economics
- Any inter-galaxy comms, entity relationships etc

Starmade Server
- Has a defined size like you do with sectors and is merely a galaxy.
- Has built in Entity tracking like normal
- Has access to GPS on Starmade Centre/Mgmt Server for relative interaction with other galaxies in the universe, like warp gates etc
- When transitioning from one galaxy to another, you go through a type of vMotion (VMware) from server to server. Have a transition effect. Problem may be when you have armada's or someting, but I'm sure something could be worked out.

Starmade Client
- Much the same as existing, but connects to this new clustering server environment via the Starmade Centre/Mgmt Server

 

[therimmer96]

I firmly believe that this will be possible with the modding api. Although I believe doing it on dedicated hardware rather than a VM would be better.

Local network connections between all of the servers that is seperate from the main internet connection for the moving of ship and player data between instances. Less popular galaxies can share hardware with other less popular galaxies. Work out a way for factions and blueprints to be fully synced between galaxies.

Would need a way to prompt a client to connect to a new server, but with planned downloading of mods from the server, that should not be difficult to mod in

 

[FatCobra]

Actually it would depend on the ESXi hardware. I run several ESXi clusters at work that are far to "beefy" for single instances, 16+ cores 756G of RAM etc. The network connections when running several guests on a a single host if faster than using a standard network for inter communication. I think this is a great idea but would be more of a commercial application.

For running a cluster at home I would still recommend using a single machine with some sort of visualization as Starmade is still single threaded. This would allow better utilization of multi core processors.

Overall I think this would be a cool feature.

 

[Sgtwisky]

very interesting idea there. How do you expect this to handle servers (what would be a galaxy in this setup) which have modded configs? or server rules that the admins there enforce? Or are you thinking that it should be base game only and only have "official clusters" instead of "community clusters"? Or would the idea be that only servers with the same config and "house rules" could be clustered together? If so how would you prevent someone from clustering a server with a separate set of rules, then admin deleting ships that come into their galaxy with say 10 too many turrets?

I would assume that "community clusters" could make their own standard configs and rules. maybe configs could be checked against what the cluster default configs are? This way you couldn't get one admin going "Ima make X10000 gathering so I caz Pownz the other servers!". which brings the issue of admin commands into light.

NOW.... this is all under the assumption the the cluster thing would ether be mandatory, OR an instance where a server can join a cluster. I get that moderating those that join a cluster is likely the job of the guys running the cluster hardware. I just wanted to point out that as much as I love the idea, it could make a bit of a mess for server hosts, and cluster hosts.

Anyway. Just my initial thoughts. If this somehow did make it in as a feature though, that would be crazy awesome. Even if it did have the issues I was thinking about.

 

[therimmer96]

very interesting idea there. How do you expect this to handle servers (what would be a galaxy in this setup) which have modded configs? or server rules that the admins there enforce? Or are you thinking that it should be base game only and only have "official clusters" instead of "community clusters"? Or would the idea be that only servers with the same config and "house rules" could be clustered together? If so how would you prevent someone from clustering a server with a separate set of rules, then admin deleting ships that come into their galaxy with say 10 too many turrets?

I would assume that "community clusters" could make their own standard configs and rules. maybe configs could be checked against what the cluster default configs are? This way you couldn't get one admin going "Ima make X10000 gathering so I caz Pownz the other servers!". which brings the issue of admin commands into light.

NOW.... this is all under the assumption the the cluster thing would ether be mandatory, OR an instance where a server can join a cluster. I get that moderating those that join a cluster is likely the job of the guys running the cluster hardware. I just wanted to point out that as much as I love the idea, it could make a bit of a mess for server hosts, and cluster hosts.

Anyway. Just my initial thoughts. If this somehow did make it in as a feature though, that would be crazy awesome. Even if it did have the issues I was thinking about.

The idea is not that everyone that owns their own server cna be part of the cluster. its that individuals can run their own cluster, and it behave as a single server.

 

[karkinosz]

The idea is not that everyone that owns their own server cna be part of the cluster. its that individuals can run their own cluster, and it behave as a single server.

This.
That is pretty much the entire idea we are going for... While every server is part of the greater starmade universe, randomly linking servers with vastly different admins, rules and playerbases is a bad idea.

 

[Sgtwisky]

Ah, that makes it more clear.

 

[kiraen]

Since it is planned that you can travel from server to server, I'm not sure I see the point of this.

 

[therimmer96]

Since it is planned that you can travel from server to server, I'm not sure I see the point of this.

When was this marked as planned? This is a highly argued feature.

And the point is to relieve server lag caused by someone on the other side of the universe by having them on an entirely seperate server.

 

[kiraen]

It was mentioned in one of the twitch steams.

skip to 13:10

 

[Lone_Puppy]

To hopefully clear up the idea of where I was heading, here goes...

As therimmer96 pointed out, it would be a cluster of servers participation on a single 4242 port connection. This is to have clusters of servers that would act the same way an existing server does, with only the galaxies supplied by the individual hosts in the cluster.

starmade server = as in the same way an existing server runs.
cluster = the universe. a collection of these starmade servers coordinated and managed by a central control or management server.
central control server = the one who makes the rules for the universe. Keeps the economy, admin, and whatever other controls etc.

Damn, My Win8.1 about to restart for updates. To be continued....
[DOUBLEPOST=1443916322,1443914934][/DOUBLEPOST]Now where was I?

I'm not referring to using any kind of virtualisation. I was using the way VMware uses clustering. The mechanisms and methodology they use. I could very well have used a Beowulf cluster, or Windows 2012 cluster or even a Citrix XenApp farm for that matter as an example.

Anyway, the use of VMs is not ruled out and is probably what I would build my own home Starmade cluster on.

I don't know how it would be structured in the end, but was just thinking out loud in case it was of interest to the Schine guys and wider Starmade community. Oh and that the galaxy thing sparked my train of thought.
[DOUBLEPOST=1443916442][/DOUBLEPOST]You guys are great by the way!
I love how this community has so much enthusiasm and ideas.