Link Servers
- Thread starter MacThule
- Start date
Apache Hadoop is designed for distributed data processing.... Not the same thing as a game server...
jayman38
Precentor-Primus, pro-tempore
- Joined
- Jul 13, 2014
- Messages
- 2,518
- Reaction score
- 787
Distributed processing is what this thread is all about. It might spread the processing more evenly to the rest of the cores.
- Joined
- Aug 22, 2013
- Messages
- 18
- Reaction score
- 18
Hadoop is meant for DATA processing. We're talking large databases, web server requests, management systems, and other high volumes of data from a database such as archive systems.
It is NOT the same, or intended for use in a game where the processing needed is much different, such as procedural generation, collision and physics, and high rate mining/battles.
Other than the fact the game's database is horribly inefficient in terms of size (huge headers on tiny and empty files) and sheer # of files, there is no need for Hadoop on the game's database, as only a small amount of data is read and written in a short period of time since the majority of active data is stored in memory and actively processed.
The point remains in my PREVIOUS POST that utilization of server clusters is rather pointless since the game cannot yet even distribute it's workload across the amount of processing power already available to it locally.
Last edited:
Hadoop at it's core is a distributed Map-Reduce engine and a distributed file system. If you don't know what that is I recommend doing some reading.
I shall let Wikipedia explain this one ( http://en.wikipedia.org/wiki/MapReduce )
It is designed to filter/sort/summarize/aggregate huge bodies of data like Google for example. That is a drastically different type of processing that real-time collision/physics, and the other things like logic blocks, factories etc. The other parts of Hadoop are things like a distributed database which isn't really helpful here as the game doesn't store that many files (It does store a lot but it's still well within the limits of one server). Another part (HDFS) is a distributed file system which again isn't really helpful for the same reason the database isn't
This also doesn't begin to address the fact that the overhead of Hadoop would very likely far outweigh any potential benefit.
OpenCL has benefits, but all of what needs to happen should be well enough done just multi-threading what's there. Don't really think there's much of a reason to involve OpenCL in this yet. I'm sure the devs know how they want to do it, they've just been busy with other features.
- Joined
- Jan 31, 2015
- Messages
- 1,696
- Reaction score
- 1,199
Java is a multithread language, so it's definitely *possible* to teach Starmade to multithread. Only issue is going to be how much work it'll take to improve the code.
levest28 - while I completely agree with your conclusion that SM should fully employ its local resources before expanding resources via network distribution, the fact remains that implementing network distribution may be an easier way to give SM access to two cores than re-working many segments of the code to multithread, at least as a first measure. It'd be good to keep all options on the table. Ultimately teaching SM to use 16 cores would be a far greater performance improvement, as you say.
levest28 - while I completely agree with your conclusion that SM should fully employ its local resources before expanding resources via network distribution, the fact remains that implementing network distribution may be an easier way to give SM access to two cores than re-working many segments of the code to multithread, at least as a first measure. It'd be good to keep all options on the table. Ultimately teaching SM to use 16 cores would be a far greater performance improvement, as you say.
- Joined
- Mar 10, 2015
- Messages
- 122
- Reaction score
- 50
I don't really get why there's so much fuss over this. You guys know we can have both, right?
You can totally thread Java. It's not a big deal. In order to do that, you'd have to split the universe into isolated physics systems anyway; and that's the hardest part of networking multiple computers. It's a short leap from threaded servers to networked servers.
You also aren't required to network servers. No one is going to force you to rent from a bunch of different hosts for a decent server. Being able to network would mean that, if your server admins trusted you, you could link your gaming desktop up to the server when you aren't using it for the benefit of everyone. Or an old laptop, or something like that.
Chill out yall.
You can totally thread Java. It's not a big deal. In order to do that, you'd have to split the universe into isolated physics systems anyway; and that's the hardest part of networking multiple computers. It's a short leap from threaded servers to networked servers.
You also aren't required to network servers. No one is going to force you to rent from a bunch of different hosts for a decent server. Being able to network would mean that, if your server admins trusted you, you could link your gaming desktop up to the server when you aren't using it for the benefit of everyone. Or an old laptop, or something like that.
Chill out yall.
- Joined
- Jul 21, 2013
- Messages
- 2,932
- Reaction score
- 460
Seriously? I'm completely calm, and I never said you couldn't.
All I'm saying is adding more hardware isn't the way to help this before it can use the hardware that its already been given...
Also the latency involved in linking people on home connections would probably far outweigh any potential benefit. Most people don't have the bandwidth to do that, nor would their ISP allow them too.