"Fossies" - the Fresh Open Source Software Archive

Member "log_analysis-0.46/doc/Tutorial,v" (20 Sep 2006, 5333 Bytes) of package /linux/privat/old/log_analysis-0.46.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 head	1.3;
    2 access;
    3 symbols;
    4 locks
    5 	morty:1.3; strict;
    6 comment	@# @;
    7 
    8 
    9 1.3
   10 date	2006.09.20.08.19.01;	author morty;	state Exp;
   11 branches;
   12 next	1.2;
   13 
   14 1.2
   15 date	2005.08.12.00.03.43;	author morty;	state Exp;
   16 branches;
   17 next	1.1;
   18 
   19 1.1
   20 date	2005.08.12.00.02.34;	author morty;	state Exp;
   21 branches;
   22 next	;
   23 
   24 
   25 desc
   26 @log_analysis tutorial
   27 @
   28 
   29 
   30 1.3
   31 log
   32 @*** empty log message ***
   33 @
   34 text
   35 @
   36 Here's a tutorial on writing useful configs:
   37 
   38 First, every log_analysis config should contain a "config_version"
   39 directive.  So, start your config (say, log_analysis.conf) like so:
   40 
   41 config_version 0.44
   42 
   43 Then run log_analysis on your syslog file and check the output, ie:
   44 
   45 log_analysis /var/log/syslog.20020719
   46 
   47 If you use a non-standard basename (for example,
   48 /var/log/routers.20020719), log_analysis won't know what kind of log
   49 type it is, and will complain.  You can force the issue by adding this
   50 to your config:
   51 
   52 add arr syslog_filenames=
   53 routers
   54 
   55 And then run:
   56 
   57 log_analysis -f log_analysis.conf /var/log/routers.20020719
   58 
   59 So far, so good.  Now, suppose log_analysis complains that there are a
   60 bunch of unknowns, like, say, 
   61 
   62 Unknowns for type syslog:
   63 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:4745 255.255.255.255:137 (#22)
   64 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#23)
   65 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#24)
   66 1          kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#25)
   67 
   68 Suppose you want those four messages to go under a common heading,
   69 like maybe "firewall denied".  You also want them rewritten so that
   70 the three identical port 68 packets can correlate.  For example, you
   71 have this output as your goal:
   72 
   73 firewall denied:
   74 3          from 10.128.104.1 to 255.255.255.255 port 68
   75 1          from 10.128.104.1 to 255.255.255.255 port 137
   76 
   77 
   78 Start off by adding a line to your config that says:
   79 
   80 logtype: syslog
   81 
   82 Add one of the raw data parts of an above line to your file, with the
   83 keyword "pattern:", like so:
   84 
   85 pattern: kernel: firewall log: inp DENY eth0 PROTO=17 10.128.104.1:67 255.255.255.255:68 (#24)
   86 
   87 Next, edit that line, and escape (ie. put a backslash, "\") in front
   88 of any punctuation characters or special characters in the data.  [1]
   89 For example:
   90 
   91 pattern: kernel\: firewall log\: inp DENY eth0 PROTO\=17 10\.128\.104\.1\:67 255\.255\.255\.255\:68 \(\#24\)
   92 
   93 Now, figure out which parts are variable.  For example, "eth0" is an
   94 interface, the "17" is an IP protocol, "10.128.104.1" is a source IP,
   95 "67" is a port, "255.255.255.255" is an IP, "68" is a port, and "24"
   96 is some sort of counter.  Replace each variable with something like
   97 ($pat{ip}), ($pat{int}), etc.  You can get a list of available "pats"
   98 by running log_analysis -I pat.  Note that the parentheses here should
   99 *not* be escaped.  [2] For example:
  100 
  101 pattern: kernel\: firewall log\: inp DENY ($pat{word}) PROTO\=($pat{int}) ($pat{ip})\:($pat{int}) ($pat{ip}):($pat{int}) \(\#($pat{int})\)
  102 
  103 Each variable was saved into a special token -- $1, $2, $3, etc.  So,
  104 the interface would be in $1 (ie. "eth0"), the protocol in $2, source
  105 IP in $3, etc.  
  106 
  107 Next, decide what format you want the data to be rewritten in.  For
  108 example, you might want the per-entry format to be "from SOURCE_IP to
  109 DEST_IP port DEST_PORT", leaving out the source port (which is usually
  110 uninteresting) and the counter (which is useless info).  So, add a
  111 "format" line:
  112 
  113 format: from $3 to $5 port $6
  114 
  115 Now, you just need the category name to log to.  If you like "firewall
  116 denied", you'd say:
  117 
  118 dest: firewall denied
  119 
  120 And you're done with this type of log message.  If you have more
  121 unknowns, just keep addings patterns, formats, and dests.
  122 
  123 If you have a *lot* of unknowns, or log_analysis is taking a long time
  124 to run, you might find the "-u unknowns" and "-U" options useful.
  125 Here's how I use them:
  126 
  127 log_analysis -f myconfig.conf -u unknowns -U syslogfile
  128 <control-c after a few seconds>
  129 log_analysis -f myconfig.conf -u unknowns syslogfile
  130 <edit myconfig.conf as described above>
  131 log_analysis -f myconfig.conf -u unknowns syslogfile
  132 <edit and repeat until not many unknowns left>
  133 rm -r unknowns
  134 log_analysis -f myconfig.conf -u unknowns -U syslogfile
  135 <control-c after a few more seconds, and repeat>
  136 
  137 log_analysis can do quite a bit more, but this should get you started.
  138 If something here isn't clear, feel free to post a few lines of sample
  139 logs and describe what you want done with them.
  140 
  141 
  142 [1] Strictly speaking, you only need to escape characters that have a
  143 special meaning in perl regexes, such as "(" or ")", not ":" or "=".
  144 If you want to be lazy, read perl documentation or experiment.  When
  145 in doubt, escape.  From a future compatibility perspective, escaping
  146 is an even better idea.
  147 
  148 [2] You can actually use any perl regex here if you want to; just
  149 remember to put it in parentheses.  [3]
  150 
  151 [3] Strictly speaking, you only need to put things in parentheses if
  152 you might want to refer to them later, for example, in a format.  Only
  153 things in parentheses get saved to $1, $2, etc.  But I recommend
  154 putting all variable parts in parentheses.
  155 
  156 @
  157 
  158 
  159 1.2
  160 log
  161 @correct typo
  162 @
  163 text
  164 @d7 1
  165 a7 1
  166 config_version 0.41
  167 @
  168 
  169 
  170 1.1
  171 log
  172 @Initial revision
  173 @
  174 text
  175 @d111 1
  176 a111 1
  177 in double, escape.  From a future compatibility perspective, escaping
  178 @