StarMade Log panel

[Theck]

I've started a new project to learn some software I'll need for my daily work, and I've chosen to use StarMade as an spike project to achieve this objectives. Expect crap code as this is a tech learning project, but I think the results may help other StarMade Servers admins to control them.


The Idea
I want to use Kibana to monitor the star-made.es server data.

Project released
You can find the project page in GitHub: https://github.com/thecko/starmade-log-panel

Continue reading if you want to see the project history.

How
This is the data flow I need to configure:
StarMade logs > logstash > elasticSearch > Kibana

After some investigation, Star Made stores the server output in the log.txt.0 file, and creates new files when the size grows. I need to know if it's true, please.

LogStash is a java/ruby software that will let me monitor the server's log file, parse the lines, and finally, send it to Elastic Search.

ElasticSearch is an end-to-end search and analytics platform with a powerfull REST API and it will store the log data. Data that will be shown using Kibana.

Kibana is a web framework to consume Elastic Search data generating graphs and to search between the logs easily and without coding.


Where I am
26 of feb. 2015 - Project is finally released in an intial state. More updates will come to fix errors and to add more Kibana Dashboards.

11 of feb. 2015 - I've started installing Logstash and ElasticSearch and I've followed their "hello world" tutorials. As I've learned, I'll need to use a custom "filter" to parse the StarMade logs to separate the data, for example, store the log type (admin-command, chat line, etc.).


I wanted to share the project idea in this early version because I think it's possible that someone is using one of those tools, will have a deeper knowledge of StarMade's files and will point the errors earlier than me.

My idea is to share all the code and config files with the StarMade community as I think it could be a very useful tool to other admins.

 

[Megacrafter127]

Little hint: you don't need to monitor the logfiles. Just parsing the stdout of the JVM is enough. You'll need to start the starmade server form your parser though.
In java that's easy:
http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#exec(java.lang.String[], java.lang.String[], java.io.File)
http://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html

 

[Theck]

Many thanks Megacrafter127 for the hint, but for the real application I'll need to apply what I'll learn from this side project the source will be text files. I'll bookmark those links because they'll be very useful for other projects I'm working right now :D

 

[Theck]

Something I've seen is that not every log entry is a line. Sometimes I've found entries like this one:

[2014-11-15 07:38:30] [SIM][TAG] WRITING GROUP SimulationGroup
->[Nl->Nq]; [ENTITY_SHIP_MOB_SIM_Revenant II1416033472000_0, ENTITY_SHIP_MOB_SIM_Revenant II1416033472042_1, ENTITY_SHIP_MOB_SIM_Spirit VII1416033472044_2]

¿How do you handle those cases?

EDIT:

Well,. logstash have an input codec for this case, multiline codec allows to define patterns to distinguish between a "real" new line and the second line of a previous one :D

 

[Megacrafter127]

simple: regular expressions
a line is a new entry if it has a timestamp in the beginning of a line.
The regular expression matching the used timestamp would be:

^[\d{4,}_\d{1,2}_\d{1,2} \d{2}:\d{2}:\d{2}]

In java, you can check if a String matches a regular expression in multiple preimplemented ways. One of them would be:
http://docs.oracle.com/javase/7/docs/api/java/lang/String.html#matches(java.lang.String)
don't forget that the \ are supposed to be actual characters in the string, so if you are hardcoding the regex you'll need to replace each \ with \\

 

[Theck]

Yes, that's what I was doing in the logstash filter, but the problem was that you'll have an entry per line. With the input codec the entry will not be send until the condition matches.

This is how the stashlog config file looks like now:

input {
    #file {
    #    path => "/home/theck/areagamer/starmade/server/starmade/StarMade/logs/log.txt.0"
    #}
    stdin {
        codec => multiline {
            pattern => "^\["
            negate => true
            what => previous
        }
    }
}

filter {
    starmade {
      message => "Hello world!"
    }
}

output {
    stdout {
        codec => json
    }
}

I'm using stdin for testing purposes.

 

[Megacrafter127]

I think something like this would work:

public static final String newEntryRegex="^[\\d{4,}_\\d{1,2}_\\d{1,2} \\d{2}:\\d{2}:\\d{2}]";
public static interface LogEntryOutput {
     write(String entry);
     writePotentiallyMalformed(String entry); // #0 see comment 1
}
public static void read(File file, LogEntryOutput output) throws IOException {
    BufferedReader in=new BufferedReader(new FileReader(file));
    String buffer=null,line;
    try{
        while(null != (line=in.readLine()) {
            if(line.matches(newEntryRegex)) {
                if(buffer!=null) output.write(buffer);
                buffer=line;
            } else {
                if(buffer==null) buffer=line;
                else buffer+="\n"+line;
            }
        }
    } catch(EOFException e) { // #1 when reading from a file, no EOFExceptions would occurr normally, catchphrase only implemented for compatibility with network-streams.
        output.writePotentiallyMalformed(buffer);
        throw e;
    }
    if(buffer!=null) output.write(buffer);
}

 

[Theck]

After some problems with the kibana's configuration in my laptop, I've been able to make it work.

Then I've installed on the real server but there I've installed logstash as a service. I've need some hours of research to find out what was the problem (there was no error popping up), but now, I can say I have a first version of the Log Dashboard:

I'll try to write open a repository in my github profile with the instructions and needed files :D

 

[Theck]

Hello everyone,

Finally I've been able to find some time to remember, organize and write down some instructions to create your own StarMade log panels. I've tried to make it short, and you'll find only a few lines about installing the software (because there is a lot of information in their official websites). I've tried to focus on what is special for StarMade, the LogStash filter and the Kibana Dashboards.

Link: https://github.com/thecko/starmade-log-panel

I'll be watching this thread for any suggestions and I'll update as soon as something changes, or when I add any new Dashboard. I'm new to Kibana and there's more to know to be able to create better analytic Dashboards.

Many thanks and I hope this will be useful to any server admins out there