"Fossies" - the Fresh Open Source Software Archive

Member "log_analysis-0.46/doc/Tutorial" (20 Sep 2006, 4775 Bytes) of package /linux/privat/old/log_analysis-0.46.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 
    2 Here's a tutorial on writing useful configs:
    3 
    4 First, every log_analysis config should contain a "config_version"
    5 directive.  So, start your config (say, log_analysis.conf) like so:
    6 
    7 config_version 0.44
    8 
    9 Then run log_analysis on your syslog file and check the output, ie:
   10 
   11 log_analysis /var/log/syslog.20020719
   12 
   13 If you use a non-standard basename (for example,
   14 /var/log/routers.20020719), log_analysis won't know what kind of log
   15 type it is, and will complain.  You can force the issue by adding this
   16 to your config:
   17 
   18 add arr syslog_filenames=
   19 routers
   20 
   21 And then run:
   22 
   23 log_analysis -f log_analysis.conf /var/log/routers.20020719
   24 
   25 So far, so good.  Now, suppose log_analysis complains that there are a
   26 bunch of unknowns, like, say, 
   27 
   28 Unknowns for type syslog:
   29 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:4745 255.255.255.255:137 (#22)
   30 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#23)
   31 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#24)
   32 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#25)
   33 
   34 Suppose you want those four messages to go under a common heading,
   35 like maybe "firewall denied".  You also want them rewritten so that
   36 the three identical port 68 packets can correlate.  For example, you
   37 have this output as your goal:
   38 
   39 firewall denied:
   40 3          from 10.128.104.1 to 255.255.255.255 port 68
   41 1          from 10.128.104.1 to 255.255.255.255 port 137
   42 
   43 
   44 Start off by adding a line to your config that says:
   45 
   46 logtype: syslog
   47 
   48 Add one of the raw data parts of an above line to your file, with the
   49 keyword "pattern:", like so:
   50 
   51 pattern: kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#24)
   52 
   53 Next, edit that line, and escape (ie. put a backslash, "\") in front
   54 of any punctuation characters or special characters in the data.  [1]
   55 For example:
   56 
   57 pattern: kernel\: firewall log\: inp DENY eth0 PROTO\=17 10\.128\.104\.1\:67 255\.255\.255\.255\:68 \(\#24\)
   58 
   59 Now, figure out which parts are variable.  For example, "eth0" is an
   60 interface, the "17" is an IP protocol, "10.128.104.1" is a source IP,
   61 "67" is a port, "255.255.255.255" is an IP, "68" is a port, and "24"
   62 is some sort of counter.  Replace each variable with something like
   63 ($pat{ip}), ($pat{int}), etc.  You can get a list of available "pats"
   64 by running log_analysis -I pat.  Note that the parentheses here should
   65 *not* be escaped.  [2] For example:
   66 
   67 pattern: kernel\: firewall log\: inp DENY ($pat{word}) PROTO\=($pat{int}) ($pat{ip})\:($pat{int}) ($pat{ip}):($pat{int}) \(\#($pat{int})\)
   68 
   69 Each variable was saved into a special token -- $1, $2, $3, etc.  So,
   70 the interface would be in $1 (ie. "eth0"), the protocol in $2, source
   71 IP in $3, etc.  
   72 
   73 Next, decide what format you want the data to be rewritten in.  For
   74 example, you might want the per-entry format to be "from SOURCE_IP to
   75 DEST_IP port DEST_PORT", leaving out the source port (which is usually
   76 uninteresting) and the counter (which is useless info).  So, add a
   77 "format" line:
   78 
   79 format: from $3 to $5 port $6
   80 
   81 Now, you just need the category name to log to.  If you like "firewall
   82 denied", you'd say:
   83 
   84 dest: firewall denied
   85 
   86 And you're done with this type of log message.  If you have more
   87 unknowns, just keep addings patterns, formats, and dests.
   88 
   89 If you have a *lot* of unknowns, or log_analysis is taking a long time
   90 to run, you might find the "-u unknowns" and "-U" options useful.
   91 Here's how I use them:
   92 
   93 log_analysis -f myconfig.conf -u unknowns -U syslogfile
   94 <control-c after a few seconds>
   95 log_analysis -f myconfig.conf -u unknowns syslogfile
   96 <edit myconfig.conf as described above>
   97 log_analysis -f myconfig.conf -u unknowns syslogfile
   98 <edit and repeat until not many unknowns left>
   99 rm -r unknowns
  100 log_analysis -f myconfig.conf -u unknowns -U syslogfile
  101 <control-c after a few more seconds, and repeat>
  102 
  103 log_analysis can do quite a bit more, but this should get you started.
  104 If something here isn't clear, feel free to post a few lines of sample
  105 logs and describe what you want done with them.
  106 
  107 
  108 [1] Strictly speaking, you only need to escape characters that have a
  109 special meaning in perl regexes, such as "(" or ")", not ":" or "=".
  110 If you want to be lazy, read perl documentation or experiment.  When
  111 in doubt, escape.  From a future compatibility perspective, escaping
  112 is an even better idea.
  113 
  114 [2] You can actually use any perl regex here if you want to; just
  115 remember to put it in parentheses.  [3]
  116 
  117 [3] Strictly speaking, you only need to put things in parentheses if
  118 you might want to refer to them later, for example, in a format.  Only
  119 things in parentheses get saved to $1, $2, etc.  But I recommend
  120 putting all variable parts in parentheses.
  121