Ability to remove debug notifications from console log

[petrenko17]

Is it possible to add option to the server config to disable debug messages in console?
They are spammy, the amount of them is confusing and they are in general useless anyways. For examle our console is getting spamed by "path calculation error" and since the console is full of this debug message we can not find important messages such as actual exeptions or errors which caused server crash.
These messages are maily useful for developers as we can not do anything with path calculations or similar bugs on our own.
And on top of it the amount of messages literally cost server performance, especially on smaller servers.
Starmade servers are already performance heavy, please consider this feature as it is very beneficial for server owners.

 

[Erth Paradine]

Is it possible to add option to the server config to disable debug messages in console?
They are spammy, the amount of them is confusing and they are in general useless anyways. For examle our console is getting spamed by "path calculation error" and since the console is full of this debug message we can not find important messages such as actual exeptions or errors which caused server crash.
These messages are maily useful for developers as we can not do anything with path calculations or similar bugs on our own.
And on top of it the amount of messages literally cost server performance, especially on smaller servers.
Starmade servers are already performance heavy, please consider this feature as it is very beneficial for server owners.

Many of the logs are quite useful for third-party toolkits to utilize, and we in-fact use some of the log information to mitigate a number of bugs, including the asteroid respawning bug that has a tendency to lagoff all players if a colliding asteroid spawn is not VERY quickly dealt with (e.g. within 1-5 seconds). In addition to our toolkit, other servers utilize logging information to offer Mission, support a bank function, and to add features such as bounties.

Further, logging does not truly impact server performance that profoundly. If logging is in-fact impacting your server's performance, and unless you have more than 20+ active players on at the time, you most likely have an issue with your storage subsystem. Look at your disks/storage, and ensure its configured sanely; disabling atime is a good place to start (this is a feature of both Windows and Linux storage subsystems)

If your server is so active (e.g. 20+ active players), that logging is in-fact causing an issue, then consider writing server logs to a ramdisk. On linux, this can be accomplished as easily as symlinking StarMade/logs/ to /dev/shm/smlogs/

As for calling calling out truly unnecessary log entries, Schine has actually been giving this some attention over the past month. However, they need specific log entries to look at; could you be more specific? Perhaps could you paste an example log entry in this thread?

 

[petrenko17]

Many of the logs are quite useful for third-party toolkits to utilize, and we in-fact use some of the log information to mitigate a number of bugs, including the asteroid respawning bug that has a tendency to lagoff all players if a colliding asteroid spawn is not VERY quickly dealt with (e.g. within 1-5 seconds). In addition to our toolkit, other servers utilize logging information to offer Mission, support a bank function, and to add features such as bounties.

Further, logging does not truly impact server performance that profoundly. If logging is in-fact impacting your server's performance, and unless you have more than 20+ active players on at the time, you most likely have an issue with your storage subsystem. Look at your disks/storage, and ensure its configured sanely; disabling atime is a good place to start (this is a feature of both Windows and Linux storage subsystems)

If your server is so active (e.g. 20+ active players), that logging is in-fact causing an issue, then consider writing server logs to a ramdisk. On linux, this can be accomplished as easily as symlinking StarMade/logs/ to /dev/shm/smlogs/

As for calling calling out truly unnecessary log entries, Schine has actually been giving this some attention over the past month. However, they need specific log entries to look at; could you be more specific? Perhaps could you paste an example log entry in this thread?

I do agree this issue could be solved by third party software, I saw your thread. But in my opinion the game itself should have some of this basic functions. Not necessary sort of plugin comands, but cleaner console would help.
I am complaining mostly because I am used to minecraft console. So I would suggest classifying console messages into: info, warning, critical, debug... As a server owner I am interrested in:
Info messages: player joined games, his ip, chat messages, map saves.
Warning messages: sector lag, heavy collision rusilting in lag
Critical messages: exeptions, errors and similar problems which caused server crash

But I am not interrested in debug level messages such as: Loading entities, entity spawns, Npc factions ifno, rails, collisions. These messages are written in a way that only autor of starmade code can uderstand what is really going on and only he can do something with it, so why should I be interrested in these?
Also the log system is a mess. There are two logs starmade and server and console is mixture of both. Further more It is hard to find the latest log, we have to search by date. I recomend saving the latest log as a "server.log" and older logs with numbers behind.
As some new features I would appreciate filling the rest of a comand by TAB and searching in history using arrow keys.

(I am only a translator, beginner knowlidge of english, I wrote this paragraf at the instigation of our server administrator. If you find any flaws in the text I can ask him for better description. But I would recomend taking a look at minecraft console, as it is the visual presentation of our point.)

 

[Erth Paradine]

I do agree this issue could be solved by third party software, I saw your thread. But in my opinion the game itself should have some of this basic functions. Not necessary sort of plugin comands, but cleaner console would help.
I am complaining mostly because I am used to minecraft console. So I would suggest classifying console messages into: info, warning, critical, debug... As a server owner I am interrested in:
Info messages: player joined games, his ip, chat messages, map saves.
Warning messages: sector lag, heavy collision rusilting in lag
Critical messages: exeptions, errors and similar problems which caused server crash

But I am not interrested in debug level messages such as: Loading entities, entity spawns, Npc factions ifno, rails, collisions. These messages are written in a way that only autor of starmade code can uderstand what is really going on and only he can do something with it, so why should I be interrested in these?
Also the log system is a mess. There are two logs starmade and server and console is mixture of both. Further more It is hard to find the latest log, we have to search by date. I recomend saving the latest log as a "server.log" and older logs with numbers behind.
As some new features I would appreciate filling the rest of a comand by TAB and searching in history using arrow keys.

(I am only a translator, beginner knowlidge of english, I wrote this paragraf at the instigation of our server administrator. If you find any flaws in the text I can ask him for better description. But I would recomend taking a look at minecraft console, as it is the visual presentation of our point.)

Excellent points, and I agree: tiered logging would be wonderful, as it's a common feature of any modern/mature server platform, and it certainly eases administrative efforts. What's not really clear right now, is that most SM logs are somewhat tiered already (note the use of [text] classifiers)...I haven't found any documentation that concisely explains these classifiers though.

I am interested in loading, spawn, and collision information: all of this is incredibly useful, not just for debug purposes, but for general server administration. Such information is also quite useful when troubleshooting a misbehaving server, often-times after a crash. Given that the game is still in an Alpha state, the developers also often need that information too.

FWIW: for the past ~6 months we've completely ignored logfiles in StarMade/logs/ (dumping to a ramdisk eliminates excessive SSD writes, and then we simply ignore the whole mess) ...and instead we work off console output.

 

[petrenko17]

Excellent points, and I agree: tiered logging would be wonderful, as it's a common feature of any modern/mature server platform, and it certainly eases administrative efforts. What's not really clear right now, is that most SM logs are somewhat tiered already (note the use of [text] classifiers)...I haven't found any documentation that concisely explains these classifiers though.

I am interested in loading, spawn, and collision information: all of this is incredibly useful, not just for debug purposes, but for general server administration. Such information is also quite useful when troubleshooting a misbehaving server, often-times after a crash. Given that the game is still in an Alpha state, the developers also often need that information too.

FWIW: for the past ~6 months we've completely ignored logfiles in StarMade/logs/ (dumping to a ramdisk eliminates excessive SSD writes, and then we simply ignore the whole mess) ...and instead we work off console output.

I have seen your post. As I said, although it is useful, the game should already have some regulations on its own. I seem to care less than you of npc ships spawns, loading and similar events. It will not be required in the finished, polished game. So for the time beeing, I would recomend console filtering in 3 stages.

debug - current state of console (or as you poited out even more extensive)
limited - entity spawns, loadings, basickly your point of view, basic yet useful information for solving bugs
extensive - only the most basic informations, my point of view, the state of console of mostly polished game, none the less it is worth to implement it now rather than later as it will not change much.
[doublepost=1483807288,1483806268][/doublepost]This whole 266 ms lag is partly due to exeption, but also partly due to extensive console spam.

[doublepost=1483808360][/doublepost]Or another example of spam is:
[2017-01-07 17:28:56] NOT FACING:::: (23, 15, 4)[Rail Basic]o[BACK][inactive][100hp][OriencubeRightTop]
[2017-01-07 17:28:56] NOT FACING:::: (9, 19, 4)[Rail Basic]o[[WARNING] UNKNOWN SIDE 7][inactive][100hp][OriencubeLeftBottom]
[2017-01-07 17:28:56] NOT FACING:::: (9, 19, 35)[Rail Basic]o[[WARNING] UNKNOWN SIDE 7][inactive][100hp][OriencubeLeftBottom]
[2017-01-07 17:28:56] NOT FACING:::: (23, 15, 4)[Rail Basic]o[BACK][inactive][100hp][OriencubeRightTop]
[2017-01-07 17:28:56] NOT FACING:::: (9, 19, 4)[Rail Basic]o[[WARNING] UNKNOWN SIDE 7][inactive][100hp][OriencubeLeftBottom]
[2017-01-07 17:28:56] NOT FACING:::: (9, 19, 35)[Rail Basic]o[[WARNING] UNKNOWN SIDE 7][inactive][100hp][OriencubeLeftBottom]
[2017-01-07 17:28:56] NOT FACING:::: (23, 15, 4)[Rail Basic]o[BACK][inactive][100hp][OriencubeRightTop]
[2017-01-07 17:28:56] NOT FACING:::: (9, 19, 4)[Rail Basic]o[[WARNING] UNKNOWN SIDE 7][inactive][100hp][OriencubeLeftBottom]
[2017-

only in one second
And why should I care if I can not do anything about it? And I dont even know what that exactly means. Console should not contain such a debug level notifications, unless it is turned on.

 

[Erth Paradine]

Well, actually - a debug console (or logging) offers far more information than the current output. For instance see the command /npc_debug_mode

I haven't found NPC logging info to be useful yet, however we haven't looked at adding third-party functions to the NPC factions yet. Others might already been working on third-party features though.

Considering that a "polished" version of the game is most likely many years away (given past/current pace of development), a lot of existing log information is useful for admins/operators to leverage for the addition of third-party features. At some point we'll have an API too...but my guess is that's easily 2-3 years away, or further (given that "API" doesn't even appear on SM's dev timeline, and is only given a brief reference on the roadmap).

I disagree on your 266ms latency point - in the ~6 months we've been redirecting logging to syslog, the server loads imposed by console logging hasn't even registered against overall system loads (with exception to only a handful of exceptional situations). However, I do agree that your "...NOT FACING..." entries are minimaly useful....basically only unless/until an admin needs to troubleshoot a player's laggy entity. At that point, such information can be incredibly helpful. It would be nice to have an ability to toggle such verbosity on/off, on a running server, so that if/when such logging can be adjusted admins can do so without lurching through a server restart.

For determining what underlying functions are actually causing your lag, you might consider leveraging some Java profiling tools; you'll find that console/log output imposes a minimal overhead. Taking an educated guess on your 266ms example, logging likely imposed around 0.5% of the total 266ms...or ~1ms.

 

[petrenko17]

Well, actually - a debug console (or logging) offers far more information than the current output. For instance see the command /npc_debug_mode

I haven't found NPC logging info to be useful yet, however we haven't looked at adding third-party functions to the NPC factions yet. Others might already been working on third-party features though.

Considering that a "polished" version of the game is most likely many years away (given past/current pace of development), a lot of existing log information is useful for admins/operators to leverage for the addition of third-party features. At some point we'll have an API too...but my guess is that's easily 2-3 years away, or further (given that "API" doesn't even appear on SM's dev timeline, and is only given a brief reference on the roadmap).

I disagree on your 266ms latency point - in the ~6 months we've been redirecting logging to syslog, the server loads imposed by console logging hasn't even registered against overall system loads (with exception to only a handful of exceptional situations). However, I do agree that your "...NOT FACING..." entries are minimaly useful....basically only unless/until an admin needs to troubleshoot a player's laggy entity. At that point, such information can be incredibly helpful. It would be nice to have an ability to toggle such verbosity on/off, on a running server, so that if/when such logging can be adjusted admins can do so without lurching through a server restart.

For determining what underlying functions are actually causing your lag, you might consider leveraging some Java profiling tools; you'll find that console/log output imposes a minimal overhead. Taking an educated guess on your 266ms example, logging likely imposed around 0.5% of the total 266ms...or ~1ms.

I do agree that the lag is mainly caused by continous calculations of trade path. But the console spam is rather extensive.
It is also true that current information can be useful for troubleshooting, but they should have their own debug log, which could be turned off. Not every admin is capable or interrested in troubleshooting errors. Current console log should be only for those who test the game or are willing to help development. It is useless for casual players or more busy hosters, which basicly just do not have the time or interrest. For those it is only a giant confusing mess. For them console is full of same or similar nothing telling notifications, so they can not find any major events which happened.
I am suggesting these changes not as default, but rather as turnable option for causual or funmade servers. None the less, they should be noted if they decide to simplyfy the console logging via config option, their problems will not be resolved by game support.

And form my experience it is not hard to find the source of a lag, but the solution which admin should make. The usual main cause of a lag is player and his creations, but it is hard to decide if you should remove his stuff, kick him, ban him... because causing major lag in this game is not difficult at all and anyone could do it even by a mistake.
From XXX limes of console log you can find exactly what is lagging and why, but you will most likely end up with the same decisoin making at the end.
In summary not everyone bothers with troubleshooting. Those who dont shouldnt be spammed by detailed console log. Those who do might find detailed log useful.

 

[Erth Paradine]

Not every admin is capable or interrested in troubleshooting errors. Current console log should be only for those who test the game or are willing to help development. It is useless for casual players or more busy hosters, which basicly just do not have the time or interrest. For those it is only a giant confusing mess. For them console is full of same or similar nothing telling notifications, so they can not find any major events which happened.
...

SM is still alpha, and as such (IMHO), anyone hosting a server should expect to invest time/energy into this giant confusing mess, and its troubleshooting/development. That is after-all, I believe a core facet behind why alpha grade software is made publicly available.

...
I am suggesting these changes not as default, but rather as turnable option for causual or funmade servers. None the less, they should be noted if they decide to simplyfy the console logging via config option, their problems will not be resolved by game support.

And form my experience it is not hard to find the source of a lag, but the solution which admin should make. The usual main cause of a lag is player and his creations, but it is hard to decide if you should remove his stuff, kick him, ban him... because causing major lag in this game is not difficult at all and anyone could do it even by a mistake.
From XXX limes of console log you can find exactly what is lagging and why, but you will most likely end up with the same decisoin making at the end.
In summary not everyone bothers with troubleshooting. Those who dont shouldnt be spammed by detailed console log. Those who do might find detailed log useful.

If the console shouldn't include this information, then what should the console be displaying? In fact, if someone can't be bothered to help with troubleshooting, what do they hope to gain by paying any attention to the console?

 

[petrenko17]

SM is still alpha, and as such (IMHO), anyone hosting a server should expect to invest time/energy into this giant confusing mess, and its troubleshooting/development. That is after-all, I believe a core facet behind why alpha grade software is made publicly available.

If the console shouldn't include this information, then what should the console be displaying? In fact, if someone can't be bothered to help with troubleshooting, what do they hope to gain by paying any attention to the console?

I ment console of polished game should only be displaying events connected with players and their actions on server, similarly how minecraft console does. By parsing a console log you can and will find problems which have ocuured, but as you said, API is better option. Minecraft has one, yet unofficial.
I agree with your statements, this topic is description of a console after we get API and similar tools to hlep us. (At least how I imagine it). But some players just want to play with their friends on very, very small local or lan servers (4-5 people). Dont expect them to contribute to development as they usually have very little knowlidge of how java or even server itself. For them i suggest swichable simplified console described above. (Not everybody uses starmote)