"Fossies" - the Fresh Open Source Software Archive

Member "log_analysis-0.46/README.long" (17 Apr 2012, 6480 Bytes) of package /linux/privat/old/log_analysis-0.46.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 
    2 
    3                                   log_analysis
    4 
    5    Homepage:        [1]log_analysis
    6    Current version: 0.46
    7    Download:        [2]log_analysis-0.46.tar.gz
    8    Requires:        perl 5.00503
    9 
   10   Rationale
   11 
   12    I like to go through my system logs. This often lets me know if there's
   13    a system problem before it becomes a major issue, and it shows me
   14    security issues.
   15 
   16    Reviewing system logs has problems:
   17      * Logs contain lots of extraneous stuff that I want to be logged, but
   18        that I don't want to sift through when I review logs (ie. routine,
   19        error-free daemon operation.)
   20      * Logs contain a lot of repetition, which drowns out the interesting
   21        entries.
   22      * Noting repetition can be tricky because each entry usually has
   23        extra features to make it unique, such as a date, maybe a PID (ie.
   24        for syslog), and maybe application-specific information (ie.
   25        sendmail queue IDs.)
   26      * One needs to remember to review them. :)
   27      * One needs to be root to looks at logs for some OSs.
   28      * On most systems, looking at the logs for just one day can be a
   29        pain.
   30      * If I attack each box I deal with and write a separate script to do
   31        all this, I'll waste a lot of time duplicating effort.
   32      * Writing patterns is a pain even if you know regular expressions.
   33 
   34    log_analysis is my solution to these problems. It goes through several
   35    different kinds of logs (currently syslog, wtmp, and sulog), over some
   36    period (defaults to yesterday). It strips out the date and PID, and
   37    throws away certain entries. Then it tries each entry against a list of
   38    perl regular expressions. Each perl regular expression is associated
   39    with a category name and a rule for extracting data. When there's a
   40    match, the data-extracting rule is applied, and filed under the
   41    category. If a log entry is unknown, it's filed under a special
   42    category for unknowns. Identical entries for a given category are
   43    sorted and counted. There's an option to mail the output, so you can
   44    just run it out of cron. You can also save a local copy of the output.
   45    If you prefer to PGP-mail yourself the output, you can do this, too.
   46    The whole thing is designed to be easily extended, complete with an
   47    easy plug-in interface. The default mode is for reporting, but it also
   48    "real" and "gui" modes for continuous monitoring, complete with action
   49    support. Oh, and you can edit patterns in a GUI that helps write
   50    regular expressions quickly and easily.
   51 
   52   Security
   53 
   54    The program needs to run with permissions to read your log files in
   55    order to be useful, which usually means root. It does not default to
   56    SUID root, and I recommend not making it SUID, so just run it as root
   57    (ie. manually or out of cron). I've tried to avoid temp files
   58    everywhere that I can, and in the one case where I do use a temp file,
   59    I make sure to use the POSIX tmpnam function instead of trying to make
   60    up my own temp file algorithm. The default umask is 077. If you use
   61    action commands, there is nothing to stop you from using parts of the
   62    log message in insecure ways, so for goodness' sake, be careful.
   63 
   64   Local extensions
   65 
   66    log_analysis already has lots of rules, but chances are that you have
   67    log entries that aren't already covered. So, log_analysis can easily be
   68    extended via a local config file, as documented in the log_analysis
   69    manpage. There's even an easy way to do modular plug-ins.
   70 
   71   Mailing list
   72 
   73    We have a mailing list. It's run by majordomo, off the frakir.org
   74    server, and it's called log_analysis. That should give you enough info
   75    to figure out how to subscribe. :)
   76 
   77   Change Log
   78 
   79    Here is the [3]ChangeLog.
   80 
   81   Incompatible changes
   82 
   83    0.44
   84 
   85           + no more gui_mode_configuration_disabled or
   86             gui_mode_ignore_disabled
   87           + using the "gui_mode_config_savelocal" option will not
   88             recognize local modifications made from earlier versions of
   89             log_analysis. Be careful!
   90 
   91    0.42
   92 
   93           + dests may no longer contain backslash
   94 
   95    0.38
   96 
   97           + date_format defaults to %Y_%m_%d
   98           + -o no longer also outputs to standard out. Add -O for the old
   99             behavior.
  100           + config_version is now mandatory
  101 
  102    0.37
  103 
  104           + end has been replaced with @@end
  105           + the new default sort is "funky". It handles numbers and IPs
  106             more cleanly, although it takes longer for large data sets.
  107 
  108    0.36
  109 
  110           + -F has been changed to -I internal_config
  111           + -D has been changed to -I evals
  112 
  113    0.35
  114 
  115           + raw_rules no longer allows an optional code hook as its 4th
  116             field. If you were using this, please revert to the old
  117             version and let me know.
  118 
  119    0.34
  120 
  121           + Took required_log_file out of the config; supporting it was
  122             too messy, and everyone I asked wasn't using it, anyway. If
  123             you were using it, please revert to the old version and let me
  124             know.
  125 
  126    0.29
  127 
  128           + Some directive names used "-" instead of "_". These have been
  129             changed. The new names are include_dir, include_dir_if_exists,
  130             include_if_exists, and block_comment.
  131 
  132   Future plans/TODO
  133 
  134    Major things I would really like to see happen, but don't necessarily
  135    have time to implement in the foreseeable future:
  136      * Statistics support. That is, some way to keep track of information
  137        beyond simple message counts -- say, so we can know not only how
  138        many mail message lines we've seen, but also how many bytes those
  139        messages contained. This probably includes the ability to allow
  140        each message to count towards multiple categories.
  141      * Persistence. This goes with statistics -- once I can know, say,
  142        many bytes I get per day, I would now like a way for log_analysis
  143        to keep track of that, and tell me how many we saw this month or
  144        this year.
  145      * Support for "state" across lines. That is, some kinds of log
  146        messages (most notably sendmail) don't provide all the relevant
  147        info in each log message, so you need to keep track of previous log
  148        messages. There is no mechanism to handle this.
  149      * A config syntax sophisticated enough to keep up with all this.
  150      * This page sucks. :) Rewrite to be more useful.
  151      __________________________________________________________________
  152 
  153    by Mordechai T. Abzug <[4]morty+www@frakir.org>
  154    [5]Main page