"Fossies" - the Fresh Open Source Software Archive

Member "squidanalyzer-6.6/README" (7 May 2017, 23617 Bytes) of package /linux/privat/squidanalyzer-6.6.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "README": 6.5_vs_6.6.

    1 NAME
    2     SquidAnalyzer - Squid access log report generation tool
    3 
    4 DESCRIPTION
    5     SquidAnalyzer parse native access log format of the Squid proxy and
    6     generate general statistics about hits, bytes, users, networks, top url,
    7     top second level domain and denied URLs. Common and combined log format
    8     are also supported. SquidGuard logs can also be parsed and ACL's
    9     redirection reported into denied URLs report.
   10 
   11     Statistic reports are oriented to user and bandwidth control, this is
   12     not a pure cache statistics generator. SquidAnalyzer use flat files to
   13     store data and don't need any SQL, SQL Lite or Berkeley databases.
   14 
   15     This analyzer is incremental so it should be run in a daily cron. Take
   16     care if you have rotate log enable to run it before rotation is done.
   17 
   18 REQUIREMENT
   19     Nothing is required than a modern perl version 5.8 or higher. Graphics
   20     are based on the Flotr2 Javascript library so they are drawn at your
   21     browser side without extra installation required.
   22 
   23 INSTALLATION
   24   Generic install
   25     If you want the package to be intalled into the Perl distribution just
   26     do the following:
   27 
   28         perl Makefile.PL
   29         make
   30         make install
   31 
   32     Follow the instruction given at the end of install. With this default
   33     install everything configurable will be installed under
   34     /etc/squidanalyzer. The Perl library SquidAnalyzer.pm will be installed
   35     under your site_perl directory and the squid-analyzer Perl script will
   36     be copied under /usr/local/bin.
   37 
   38     The default output directory for html reports will be
   39     /var/www/squidanalyzer/.
   40 
   41     On FreeBSD, if make install is freezing and you have the following
   42     messages:
   43 
   44             FreeBSD: Registering installation in the package database
   45             FreeBSD: Cannot determine short module description
   46             FreeBSD: Cannot determine module description
   47 
   48     please proceed as follow:
   49 
   50             perl Makefile.PL INSTALLDIRS=site
   51             make
   52             make install
   53 
   54     as the issue is related to an install into the default Perl vendor
   55     installdirs it will then use Perl site installdirs.
   56 
   57     Note: you may not encountered this issue any more, since v6.6
   58     SquidAnalyzer use site as default installation directory.
   59 
   60   Custom install
   61     You can create your fully customized SquidAnalyzer installation by using
   62     the Makefile.PL Perl script. Here is a sample:
   63 
   64             perl Makefile.PL \
   65                     LOGFILE=/var/log/squid3/access.log \
   66                     BINDIR=/usr/bin \
   67                     CONFDIR=/etc \
   68                     HTMLDIR=/var/www/squidreport \
   69                     BASEURL=/squidreport \
   70                     MANDIR=/usr/share/man/man3 \
   71                     DOCDIR=/usr/share/doc/squidanalyzer
   72 
   73     If you want to build a distro package, there are two other options that
   74     you may use. The QUIET option is to tell to Makefile.PL to not show the
   75     default post install README. The DESTDIR is to create and install all
   76     files in a package build base directory. For example for Fedora RPM,
   77     thing may look like that:
   78 
   79             # Make Perl and SendmailAnalyzer distrib files
   80             %{__perl} Makefile.PL \
   81                 INSTALLDIRS=vendor \
   82                 QUIET=1 \
   83                 LOGFILE=/var/log/squid/access.log \
   84                 BINDIR=%{_bindir} \
   85                 CONFDIR=%{_sysconfdir} \
   86                 BASEDIR=%{_localstatedir}/lib/%{uname} \
   87                 HTMLDIR=%{webdir} \
   88                 MANDIR=%{_mandir}/man3 \
   89                 DOCDIR=%{_docdir}/%{uname}-%{version} \
   90                 DESTDIR=%{buildroot} < /dev/null
   91 
   92     See spec file in packaging/RPM for full RPM build script.
   93 
   94   Local install
   95     You can also have a custom installation. Just copy the SquidAnalyzer.pm
   96     and the squid-analyzer perl script into a directory, copy and modify the
   97     configuration file and run the script from here with the -c option.
   98 
   99     Then copy files sorttable.js, squidanalyzer.css and
  100     logo-squidanalyzer.png into the output directory.
  101 
  102   Post installation
  103     1. Modify your httpd.conf to allow access to HTML output like follow:
  104 
  105             Alias /squidreport /var/www/squidanalyzer
  106             <Directory /var/www/squidanalyzer>
  107                 Options -Indexes FollowSymLinks MultiViews
  108                 AllowOverride None
  109                 Order deny,allow
  110                 Deny from all
  111                 Allow from 127.0.0.1
  112             </Directory>
  113 
  114     2. If necessary, give additional host access to SquidAnalyzer in
  115     httpd.conf. Restart and ensure that httpd is running.
  116 
  117     3. Browse to http://my.host.dom/squidreport/ to ensure that things are
  118     working properly.
  119 
  120     4. Setup a cronjob to run squid-analyzer daily or more often:
  121 
  122             # SquidAnalyzer log reporting daily
  123             0 2 * * * /usr/local/bin/squid-analyzer > /dev/null 2>&1
  124 
  125     or run it manually. For more information, see README file.
  126 
  127     If your squid logfiles are rotated then cron isn't going to give the
  128     expected result as there exists a time between when the cron is run and
  129     the logfiles are rotated. It would be better to call squid-analyzer from
  130     logrotate, eg:
  131 
  132             /var/log/proxy/squid-access.log {
  133                 daily
  134                 compress
  135                 rotate 730
  136                 missingok
  137                 nocreate
  138                 sharedscripts
  139                 postrotate
  140                     test ! -e /var/run/squid.pid || /usr/sbin/squid -k rotate
  141                     /usr/bin/squid-analyzer -d -l /var/log/proxy/squid-access.log.1
  142                 endscript
  143             }
  144 
  145     You can also use network name instead of network ip addresses by using
  146     the network-aliases file. Also if you don't have authentication enable
  147     and want to replace client ip addresses by some know user or computer
  148     you can use the user-aliases file to do so.
  149 
  150     See the file squidanalyzer.conf to customized your output statistics and
  151     match your network and file system configuration.
  152 
  153 USAGE
  154     SquidAnalyzer can be run manually or by cron job using the
  155     squid-analyzer Perl script. Here are authorized usage:
  156 
  157     Usage: squid-analyzer [ -c squidanalyzer.conf ] [logfile(s)]
  158 
  159         -c | --configfile filename : path to the SquidAnalyzer configuration file.
  160                                      By default: /etc/squidanalyzer/squidanalyzer.conf
  161         -b | --build_date date     : set the date to be rebuilt, format: yyyy-mm-dd
  162                                      or yyyy-mm or yyyy. Used with -r or --rebuild.
  163         -d | --debug               : show debug information.
  164         -h | --help                : show this message and exit.
  165         -j | --jobs number         : number of jobs to run at same time. Default
  166                                      is 1, run as single process.
  167         -p | --preserve number     : used to set the statistic obsolescence in
  168                                      number of month. Older stats will be removed.
  169         -P | --pid_dir directory   : set directory where pid file will be stored.
  170                                      Default /tmp/
  171         -r | --rebuild             : use this option to rebuild all html and graphs
  172                                      output from all data files.
  173         -s | --start HH:MM         : log lines before this time will not be parsed.
  174         -S | --stop  HH:MM         : log lines after this time will not be parsed.
  175         -t | --timezone +/-HH      : set number of hours from GMT of the timezone.
  176                                      Use this to adjust date/time of SquidAnalyzer
  177                                      output when it is run on a different timezone
  178                                      than the squid server.
  179         -v | version               : show version and exit.
  180         --no-year-stat             : disable years statistics, reports will start
  181                                      from month level only.
  182         --no-week-stat             : disable weekly statistics.
  183 
  184     Log files to parse can be given as command line arguments or as a comma
  185     separated list of file for the LogFile configuration directive. By
  186     default SquidAnalyer will use file: /var/log/squid/access.log
  187 
  188     There is special options like --rebuild that force SquidAnalyzer to
  189     rebuild all HTML reports, useful after an new feature or a bug fix. If
  190     you want to limit the rebuild to a single day, a single month or year,
  191     you can use the --build_date option by specifying the date part to
  192     rebuild, format: yyyy-mm-dd, yyyy-mm or yyyy.
  193 
  194     The --preserve option should be used if you want to rotate your
  195     statistics and data. The value is the number of months to keep, older
  196     reports and data will be removed from the filesystem. Useful to preserve
  197     space, for example:
  198 
  199             squid-analyzer -p 6 -c /etc/squidanalyzer/squidanalyzer.conf
  200 
  201     will only preserve six month of statistics from the last run of
  202     squidanalyzer.
  203 
  204     If you have a SquidGuard log you can add it to the list of file to be
  205     parsed, either in the LogFile configuration directive log list, either
  206     at command line:
  207 
  208             squid-analyzer /var/log/squid3/access.log /var/log/squid/SquidGuard.log
  209 
  210     SquidAnalyzer will automatically detect the log format and report
  211     SquidGuard ACL's redirection to the Denied Urls report.
  212 
  213 MULTIPROCESS
  214     If you have huges squid access.log you will be interested by using
  215     multiprocess with SquidAnalyzer. Using the -j or --jobs command line
  216     option you can force SquidAnalyzer to use as many cores/cpus as wanted.
  217 
  218             squid-analyzer -j 8 -l /var/log/squid3/huge_access.log
  219 
  220     Here SquidAnalyzer will use 8 cpus to parse the file and compute all
  221     statistics reports. It will also use much more memory at the same time.
  222 
  223 LOGFORMAT
  224     SquidAnalyzer supports the following predefined log format:
  225 
  226         logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
  227         logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
  228         logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
  229 
  230     The common and combined log format can have one more field to add
  231     mime-type report like with the native squid log format:
  232 
  233         logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh %mt
  234         logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh %mt
  235 
  236     Those are the default format used by squid, you can switch to any of the
  237     three log format by giving the name at end of the access_log directive:
  238 
  239             access_log /var/log/squid3/access.log squid
  240 
  241     or
  242 
  243             access_log /var/log/squid3/access.log common
  244 
  245 CONFIGURATION
  246     Unless previous version customization of SquidAnalyzer is now done by a
  247     single configuration file squidanalyzer.conf.
  248 
  249     Here follow the configuration directives used by Squid Analyzer.
  250 
  251     Output output_directory
  252         Where SquidAnalyzer should dump all HTML, data and images files. You
  253         should give a path that can be read by a Web browser.
  254 
  255     WebUrl
  256         The URL of the SquidAnalyzer javascript, HTML and images files.
  257         Default: /squidreport
  258 
  259     CustomHeader
  260         This directive allow you to replace the SquidAnalyze logo by your
  261         custom logo. The default value is defined as follow:
  262 
  263                 <a href="$self->{WebUrl}">
  264                 <img src="$self->{WebUrl}images/logo-squidanalyzer.png" title="SquidAnalyzer $VERSION" border="0">
  265                 </a> SquidAnalyzer
  266 
  267         Feel free to define your own header but take care to not break
  268         current design. For example:
  269 
  270                 CustomHeader   <a href="http://my.isp.dom/"><img src="http://my.isp.dom/logo.png" title="My ISP link" border="0" width="100" height="110"></a> My ISP Company
  271                                                                                                    126,1         Bas
  272 
  273     LogFile squid_access_log_file
  274         Set the path to the Squid log file. This can be a comma separated
  275         list of files to process several files at the same time. If the
  276         files comes from differents Squid servers, they will be merges in a
  277         single reports. You can also add to the list a SquidGuard log file,
  278         SquidAnalyzer will automatically detect the format.
  279 
  280     UseClientDNSName 0
  281         If you want to use DNS name instead of client Ip address as username
  282         enable this directive. When you don't have authentication, the
  283         username is set to the client ip address, this allow you to use the
  284         DNS name instead. Note that you must have a working DNS resolution
  285         and that it can really slow down the generation of reports.
  286 
  287     DNSLookupTimeout 0.0001
  288         If you have enabled UseClientDNSName and have lot of ip addresses
  289         that do not resolve you may want to increase the DNS lookup timeout.
  290         By default SquidAnalyzer will stop to lookup a DNS name after 0.0001
  291         second (100 ms).
  292 
  293     NetworkAlias network-aliases_file
  294         Set path to the file containing network alias name. Network are show
  295         as Ip addresses so if you want to display name instead create a file
  296         with this format:
  297 
  298             LOCATION_NAME IP_NETWORK_ADDRESS
  299 
  300         Separator must be a tabulation this allow the use of space character
  301         in the network alias name.
  302 
  303         You can use regex to match and group some network addresses. See
  304         network-aliases file for examples.
  305 
  306     UserAlias user-aliases_file
  307         Set path to the file containing user alias name. If you don't have
  308         auth_proxy enable users are seen as ip addresses. So if you want to
  309         show username or computer name instead, create a file with this
  310         format:
  311 
  312             FULL_USERNAME IP_ADDRESS
  313 
  314         When 'UseClientDNSName' is ebabled you can replace ip address by a
  315         DNS name.
  316 
  317         If you have auth_proxy enable but want to replace login name by full
  318         user name for example, create a file with this format:
  319 
  320             FULL_USERNAME LOGIN_NAME
  321 
  322         Separator for both must be a tabulation this allow the use of space
  323         character in the user alias name.
  324 
  325         You can use regex to match and group some user login or ip
  326         addresses. See user-aliases file for examples.
  327 
  328     AnonymizeLogin 0
  329         Set this to 1 if you want to anonymize all user login. The username
  330         will be replaced by an unique id that change at each squid-analyzer
  331         run. Default disable.
  332 
  333     OrderNetwork bytes|hits|duration
  334     OrderUser bytes|hits|duration
  335     OrderUrl bytes|hits|duration
  336         Used to set how SquidAnalyzer sort Network, User and User detailed
  337         Urls reports screen. Value can be: bytes, hits or duration. Default
  338         is bytes. Note that OrderUrl is limited to User detailed Urls
  339         reports and does not apply to Top Url and Top domain report where
  340         there is three reports each already ordered.
  341 
  342     OrderMime bytes|hits
  343         Used to set how SquidAnalyzer sort Mime types report screen Value
  344         can be: bytes or hits. Default is bytes.
  345 
  346     UrlReport 0|1
  347         Should SquidAnalyzer display user url details. This will show all
  348         URL read by user. Take care to have enougth space disk for large
  349         user. Default is 0, no url detail report.
  350 
  351     UserReport 0|1
  352         Should SquidAnalyzer display user details. This will show statistics
  353         about user. Default is 1, show user detail report. Disable it to be
  354         able to remove any user related reports, statistics about URL and
  355         domains will remain.
  356 
  357     UrlHitsOnly 0|1
  358         Enable this directive if you don't want the tree Top URL and Domain
  359         tables. You will just have the table of Url/Domain ordered per hits
  360         then you can still sort the URL/Domain order by clicking on each
  361         column. This is useful when you have set a high value to TopNumber.
  362 
  363     QuietMode 0|1
  364         Run in quiet mode for batch processing or print debug information.
  365         Default is 0, verbose mode.
  366 
  367     CostPrice price/Mb
  368         Used to set a cost of the bandwidth per Mb. If you want to generate
  369         invoice per Mb for bandwidth traffic this can help you. Value 0 mean
  370         no cost, this is the default value, the "Cost" column is not
  371         displayed
  372 
  373     Currency currency_abreviation
  374         Used to set the currency of the bandwidth cost. Preferably the html
  375         special character. Default is &euro;
  376 
  377     TopNumber number
  378         Used to set the number of top url and second level domain to show.
  379         Default is top 100.
  380 
  381     TopStorage number
  382         Top number of url to preserve on each data file sorted by OrderUrl.
  383         On huge access log it will improve a lot the performances but you
  384         will have less precision in the top url. Default to 0, all url will
  385         be stored.
  386 
  387     TopUrlUser Use this directive to show the top N users that look at an
  388     URL or a domain. Set it to 0 to disable this feature. Default is top 10.
  389     Exclude exclusion_file
  390         Used to set client ip addresses, network addresses, auth login or
  391         uri to exclude from report.
  392 
  393         You can define one by line exclusion by specifying first the type of
  394         the exclusion (USER, CLIENT or URI) and a space separated list of
  395         valid regex.
  396 
  397         You can also use the NETWORK type to define network address with
  398         netmask using the CIDR notation: xxx.xxx.xxx.xxx/n
  399 
  400         See example below:
  401 
  402                 NETWORK        192.168.1.0/24 10.10.0.0/16
  403                 CLIENT         192\.168\.1\.2 
  404                 CLIENT         10\.169\.1\.\d+ 192\.168\.10\..*
  405                 USER           myloginstr
  406                 USER           guestlogin\d+ guestdemo
  407                 URI            http:\/\/myinternetdomain.dom.*
  408                 URI            .*\.webmail\.com\/.*\/login\.php.*
  409 
  410         you can have multiple line of the same exclusion type.
  411 
  412     Include inclusion_file
  413         Used to set client ip addresses, network addresses or auth login to
  414         include into the report. All others will not be included. It works
  415         as the opposite of the Include parameter.
  416 
  417         You can define one by line inclusion by specifying first the type of
  418         the inclusion (USER or CLIENT) and a space separated list of valid
  419         regex.
  420 
  421         You can also use the NETWORK type to define network address with
  422         netmask using the CIDR notation: xxx.xxx.xxx.xxx/n
  423 
  424         See example below:
  425 
  426                 NETWORK        192.168.1.0/24 10.10.0.0/16
  427                 CLIENT         192\.168\.1\.2 
  428                 CLIENT         10\.169\.1\.\d+ 192\.168\.10\..*
  429                 USER           myloginstr
  430                 USER           guestlogin\d+ guestdemo
  431                 URI            http:\/\/myinternetdomain.dom.*
  432                 URI            .*\.webmail\.com\/.*\/login\.php.*
  433 
  434         you can have multiple line of the same inclusion type.
  435 
  436     ExcludedMethods
  437         This directive allow exclusion of some unwanted methods in report
  438         statistics like HEAD, POST, CONNECT, etc. Can be a comma separated
  439         list of methods.
  440 
  441     ExcludedMimes
  442         This directive allow exclusion of some unwanted mimetypes in report
  443         statistics like text/html, text/plain, or more generally text/*,
  444         etc. Can be a comma separated list of perl regular expression. Ex:
  445 
  446                 ExcludedMimes   text/.*,image/.*
  447 
  448     Lang
  449         Used to set the translation file to be used. Value must be set to a
  450         file containing all string translated. See the lang directory for
  451         translation files. Default is defined internally in English.
  452 
  453     ExcludedCodes
  454         This directive allow exclusion of some unwanted codes in report
  455         statistics like TCP_DENIED/403 which are generated when a user
  456         accesses a page the first time without authentication. Can be a
  457         comma separated list of methods. Default is none, all codes will be
  458         parsed.
  459 
  460     DateFormat
  461         Date format used to display date (year = %y, month = %m and day =
  462         %d) You can also use %M to replace month by its 3 letters
  463         abbreviation. Default: %y-%m-%d
  464 
  465     SiblingHit
  466         Adds peer cache hit (CD_SIBLING_HIT) to be taken has local cache
  467         hit. Enabled by default, you must disabled it if you don't want to
  468         report peer cache hit onto your stats.
  469 
  470     TransfertUnit
  471         Allow one to change the default unit used to display transfert size.
  472         Default is BYTES, other possible values are KB, MB and GB.
  473 
  474     MinPie
  475         Minimum percentage of data in pie's graphs to not be placed in the
  476         others item. Lower values will be summarized into the others item.
  477 
  478     Locale
  479         Set this to your locale to display generated date in your language.
  480         Default is to use the current locale of the system. If you want date
  481         in German for example, set it to de_DE.
  482 
  483                 Rapport genere le mardi 11 decembre 2012, 15:13:09 (UTC+0100).
  484 
  485         with a Locale set to fr_FR.
  486 
  487     MaxFormatError
  488         When SquidAnalyzer find a corrupted line in his data file, it exit
  489         immedialtly. You can force him to wait for a certain amount of
  490         errors before exiting. Of course you might want to remove the
  491         corrupted line before the next run. This can be useful if you have
  492         special characters in some fields like mime type.
  493 
  494     TimeZone
  495         Set timezone to use when SquidAnalyzer is used in a different server
  496         than the one running squid and there is a different timezone between
  497         these two machines. The value must follow format: +/-HH. Default is
  498         to use local time. For example:
  499 
  500                 TimeZone        +01
  501 
  502         for a log file generated on zone Europe/Paris with UTC+0100 and
  503         parsed on a computer with different timezone.
  504 
  505     UseUrlPort
  506         Enable this directive if you want to include port number into Url
  507         statistics. Default is to remove the port information from the Url.
  508 
  509     UpdateAlias
  510         Enable this directive if you want to apply immedialtly the changes
  511         made in aliases files to avoid duplicates. You still have to use
  512         --rebuild to recreate previous reports with new aliases. Enabling
  513         this will imply a lost of performances with huges log files.
  514 
  515     TimeStart and TimeStop
  516         The two following configuration directive allow you to specify a
  517         start and stop time. Log line out of this time range will not be
  518         parsed. The format of the value is HH:MM
  519 
  520 SUPPORT
  521   Release announcement
  522     Please follow us on twitter to receive release announcement and latest
  523     news : https://twitter.com/SquidAnalyzer
  524 
  525   Bugs and Feature requests
  526     Please report any bugs, patches, discussion and feature request using
  527     tools on the git repository at https://github.com/darold/squidanalyzer.
  528 
  529   How to contribute ?
  530     Any contribution to build a better tool is welcome, you just have to
  531     send me your ideas, features request or patches using the tools on the
  532     git repository at https://github.com/darold/squidanalyzer
  533 
  534     You can also support the developer by donate some contribution by
  535     clicking on the "Donate" button on the SquidAnalyzer web site at
  536     http://squidanalyzer.darold.net/
  537 
  538 AUTHOR
  539     Gilles DAROLD <gilles@darold.net>
  540 
  541 COPYRIGHT
  542     Copyright (c) 2001-2017 Gilles DAROLD
  543 
  544     This package is free software and published under the GPL v3 or above
  545     license.
  546