"Fossies" - the Fresh Open Source Software Archive 
Member "squidanalyzer-6.6/README" (7 May 2017, 23617 Bytes) of package /linux/privat/squidanalyzer-6.6.tar.gz:
As a special service "Fossies" has tried to format the requested text file into HTML format (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
See also the latest
Fossies "Diffs" side-by-side code changes report for "README":
6.5_vs_6.6.
1 NAME
2 SquidAnalyzer - Squid access log report generation tool
3
4 DESCRIPTION
5 SquidAnalyzer parse native access log format of the Squid proxy and
6 generate general statistics about hits, bytes, users, networks, top url,
7 top second level domain and denied URLs. Common and combined log format
8 are also supported. SquidGuard logs can also be parsed and ACL's
9 redirection reported into denied URLs report.
10
11 Statistic reports are oriented to user and bandwidth control, this is
12 not a pure cache statistics generator. SquidAnalyzer use flat files to
13 store data and don't need any SQL, SQL Lite or Berkeley databases.
14
15 This analyzer is incremental so it should be run in a daily cron. Take
16 care if you have rotate log enable to run it before rotation is done.
17
18 REQUIREMENT
19 Nothing is required than a modern perl version 5.8 or higher. Graphics
20 are based on the Flotr2 Javascript library so they are drawn at your
21 browser side without extra installation required.
22
23 INSTALLATION
24 Generic install
25 If you want the package to be intalled into the Perl distribution just
26 do the following:
27
28 perl Makefile.PL
29 make
30 make install
31
32 Follow the instruction given at the end of install. With this default
33 install everything configurable will be installed under
34 /etc/squidanalyzer. The Perl library SquidAnalyzer.pm will be installed
35 under your site_perl directory and the squid-analyzer Perl script will
36 be copied under /usr/local/bin.
37
38 The default output directory for html reports will be
39 /var/www/squidanalyzer/.
40
41 On FreeBSD, if make install is freezing and you have the following
42 messages:
43
44 FreeBSD: Registering installation in the package database
45 FreeBSD: Cannot determine short module description
46 FreeBSD: Cannot determine module description
47
48 please proceed as follow:
49
50 perl Makefile.PL INSTALLDIRS=site
51 make
52 make install
53
54 as the issue is related to an install into the default Perl vendor
55 installdirs it will then use Perl site installdirs.
56
57 Note: you may not encountered this issue any more, since v6.6
58 SquidAnalyzer use site as default installation directory.
59
60 Custom install
61 You can create your fully customized SquidAnalyzer installation by using
62 the Makefile.PL Perl script. Here is a sample:
63
64 perl Makefile.PL \
65 LOGFILE=/var/log/squid3/access.log \
66 BINDIR=/usr/bin \
67 CONFDIR=/etc \
68 HTMLDIR=/var/www/squidreport \
69 BASEURL=/squidreport \
70 MANDIR=/usr/share/man/man3 \
71 DOCDIR=/usr/share/doc/squidanalyzer
72
73 If you want to build a distro package, there are two other options that
74 you may use. The QUIET option is to tell to Makefile.PL to not show the
75 default post install README. The DESTDIR is to create and install all
76 files in a package build base directory. For example for Fedora RPM,
77 thing may look like that:
78
79 # Make Perl and SendmailAnalyzer distrib files
80 %{__perl} Makefile.PL \
81 INSTALLDIRS=vendor \
82 QUIET=1 \
83 LOGFILE=/var/log/squid/access.log \
84 BINDIR=%{_bindir} \
85 CONFDIR=%{_sysconfdir} \
86 BASEDIR=%{_localstatedir}/lib/%{uname} \
87 HTMLDIR=%{webdir} \
88 MANDIR=%{_mandir}/man3 \
89 DOCDIR=%{_docdir}/%{uname}-%{version} \
90 DESTDIR=%{buildroot} < /dev/null
91
92 See spec file in packaging/RPM for full RPM build script.
93
94 Local install
95 You can also have a custom installation. Just copy the SquidAnalyzer.pm
96 and the squid-analyzer perl script into a directory, copy and modify the
97 configuration file and run the script from here with the -c option.
98
99 Then copy files sorttable.js, squidanalyzer.css and
100 logo-squidanalyzer.png into the output directory.
101
102 Post installation
103 1. Modify your httpd.conf to allow access to HTML output like follow:
104
105 Alias /squidreport /var/www/squidanalyzer
106 <Directory /var/www/squidanalyzer>
107 Options -Indexes FollowSymLinks MultiViews
108 AllowOverride None
109 Order deny,allow
110 Deny from all
111 Allow from 127.0.0.1
112 </Directory>
113
114 2. If necessary, give additional host access to SquidAnalyzer in
115 httpd.conf. Restart and ensure that httpd is running.
116
117 3. Browse to http://my.host.dom/squidreport/ to ensure that things are
118 working properly.
119
120 4. Setup a cronjob to run squid-analyzer daily or more often:
121
122 # SquidAnalyzer log reporting daily
123 0 2 * * * /usr/local/bin/squid-analyzer > /dev/null 2>&1
124
125 or run it manually. For more information, see README file.
126
127 If your squid logfiles are rotated then cron isn't going to give the
128 expected result as there exists a time between when the cron is run and
129 the logfiles are rotated. It would be better to call squid-analyzer from
130 logrotate, eg:
131
132 /var/log/proxy/squid-access.log {
133 daily
134 compress
135 rotate 730
136 missingok
137 nocreate
138 sharedscripts
139 postrotate
140 test ! -e /var/run/squid.pid || /usr/sbin/squid -k rotate
141 /usr/bin/squid-analyzer -d -l /var/log/proxy/squid-access.log.1
142 endscript
143 }
144
145 You can also use network name instead of network ip addresses by using
146 the network-aliases file. Also if you don't have authentication enable
147 and want to replace client ip addresses by some know user or computer
148 you can use the user-aliases file to do so.
149
150 See the file squidanalyzer.conf to customized your output statistics and
151 match your network and file system configuration.
152
153 USAGE
154 SquidAnalyzer can be run manually or by cron job using the
155 squid-analyzer Perl script. Here are authorized usage:
156
157 Usage: squid-analyzer [ -c squidanalyzer.conf ] [logfile(s)]
158
159 -c | --configfile filename : path to the SquidAnalyzer configuration file.
160 By default: /etc/squidanalyzer/squidanalyzer.conf
161 -b | --build_date date : set the date to be rebuilt, format: yyyy-mm-dd
162 or yyyy-mm or yyyy. Used with -r or --rebuild.
163 -d | --debug : show debug information.
164 -h | --help : show this message and exit.
165 -j | --jobs number : number of jobs to run at same time. Default
166 is 1, run as single process.
167 -p | --preserve number : used to set the statistic obsolescence in
168 number of month. Older stats will be removed.
169 -P | --pid_dir directory : set directory where pid file will be stored.
170 Default /tmp/
171 -r | --rebuild : use this option to rebuild all html and graphs
172 output from all data files.
173 -s | --start HH:MM : log lines before this time will not be parsed.
174 -S | --stop HH:MM : log lines after this time will not be parsed.
175 -t | --timezone +/-HH : set number of hours from GMT of the timezone.
176 Use this to adjust date/time of SquidAnalyzer
177 output when it is run on a different timezone
178 than the squid server.
179 -v | version : show version and exit.
180 --no-year-stat : disable years statistics, reports will start
181 from month level only.
182 --no-week-stat : disable weekly statistics.
183
184 Log files to parse can be given as command line arguments or as a comma
185 separated list of file for the LogFile configuration directive. By
186 default SquidAnalyer will use file: /var/log/squid/access.log
187
188 There is special options like --rebuild that force SquidAnalyzer to
189 rebuild all HTML reports, useful after an new feature or a bug fix. If
190 you want to limit the rebuild to a single day, a single month or year,
191 you can use the --build_date option by specifying the date part to
192 rebuild, format: yyyy-mm-dd, yyyy-mm or yyyy.
193
194 The --preserve option should be used if you want to rotate your
195 statistics and data. The value is the number of months to keep, older
196 reports and data will be removed from the filesystem. Useful to preserve
197 space, for example:
198
199 squid-analyzer -p 6 -c /etc/squidanalyzer/squidanalyzer.conf
200
201 will only preserve six month of statistics from the last run of
202 squidanalyzer.
203
204 If you have a SquidGuard log you can add it to the list of file to be
205 parsed, either in the LogFile configuration directive log list, either
206 at command line:
207
208 squid-analyzer /var/log/squid3/access.log /var/log/squid/SquidGuard.log
209
210 SquidAnalyzer will automatically detect the log format and report
211 SquidGuard ACL's redirection to the Denied Urls report.
212
213 MULTIPROCESS
214 If you have huges squid access.log you will be interested by using
215 multiprocess with SquidAnalyzer. Using the -j or --jobs command line
216 option you can force SquidAnalyzer to use as many cores/cpus as wanted.
217
218 squid-analyzer -j 8 -l /var/log/squid3/huge_access.log
219
220 Here SquidAnalyzer will use 8 cpus to parse the file and compute all
221 statistics reports. It will also use much more memory at the same time.
222
223 LOGFORMAT
224 SquidAnalyzer supports the following predefined log format:
225
226 logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
227 logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
228 logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
229
230 The common and combined log format can have one more field to add
231 mime-type report like with the native squid log format:
232
233 logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh %mt
234 logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh %mt
235
236 Those are the default format used by squid, you can switch to any of the
237 three log format by giving the name at end of the access_log directive:
238
239 access_log /var/log/squid3/access.log squid
240
241 or
242
243 access_log /var/log/squid3/access.log common
244
245 CONFIGURATION
246 Unless previous version customization of SquidAnalyzer is now done by a
247 single configuration file squidanalyzer.conf.
248
249 Here follow the configuration directives used by Squid Analyzer.
250
251 Output output_directory
252 Where SquidAnalyzer should dump all HTML, data and images files. You
253 should give a path that can be read by a Web browser.
254
255 WebUrl
256 The URL of the SquidAnalyzer javascript, HTML and images files.
257 Default: /squidreport
258
259 CustomHeader
260 This directive allow you to replace the SquidAnalyze logo by your
261 custom logo. The default value is defined as follow:
262
263 <a href="$self->{WebUrl}">
264 <img src="$self->{WebUrl}images/logo-squidanalyzer.png" title="SquidAnalyzer $VERSION" border="0">
265 </a> SquidAnalyzer
266
267 Feel free to define your own header but take care to not break
268 current design. For example:
269
270 CustomHeader <a href="http://my.isp.dom/"><img src="http://my.isp.dom/logo.png" title="My ISP link" border="0" width="100" height="110"></a> My ISP Company
271 126,1 Bas
272
273 LogFile squid_access_log_file
274 Set the path to the Squid log file. This can be a comma separated
275 list of files to process several files at the same time. If the
276 files comes from differents Squid servers, they will be merges in a
277 single reports. You can also add to the list a SquidGuard log file,
278 SquidAnalyzer will automatically detect the format.
279
280 UseClientDNSName 0
281 If you want to use DNS name instead of client Ip address as username
282 enable this directive. When you don't have authentication, the
283 username is set to the client ip address, this allow you to use the
284 DNS name instead. Note that you must have a working DNS resolution
285 and that it can really slow down the generation of reports.
286
287 DNSLookupTimeout 0.0001
288 If you have enabled UseClientDNSName and have lot of ip addresses
289 that do not resolve you may want to increase the DNS lookup timeout.
290 By default SquidAnalyzer will stop to lookup a DNS name after 0.0001
291 second (100 ms).
292
293 NetworkAlias network-aliases_file
294 Set path to the file containing network alias name. Network are show
295 as Ip addresses so if you want to display name instead create a file
296 with this format:
297
298 LOCATION_NAME IP_NETWORK_ADDRESS
299
300 Separator must be a tabulation this allow the use of space character
301 in the network alias name.
302
303 You can use regex to match and group some network addresses. See
304 network-aliases file for examples.
305
306 UserAlias user-aliases_file
307 Set path to the file containing user alias name. If you don't have
308 auth_proxy enable users are seen as ip addresses. So if you want to
309 show username or computer name instead, create a file with this
310 format:
311
312 FULL_USERNAME IP_ADDRESS
313
314 When 'UseClientDNSName' is ebabled you can replace ip address by a
315 DNS name.
316
317 If you have auth_proxy enable but want to replace login name by full
318 user name for example, create a file with this format:
319
320 FULL_USERNAME LOGIN_NAME
321
322 Separator for both must be a tabulation this allow the use of space
323 character in the user alias name.
324
325 You can use regex to match and group some user login or ip
326 addresses. See user-aliases file for examples.
327
328 AnonymizeLogin 0
329 Set this to 1 if you want to anonymize all user login. The username
330 will be replaced by an unique id that change at each squid-analyzer
331 run. Default disable.
332
333 OrderNetwork bytes|hits|duration
334 OrderUser bytes|hits|duration
335 OrderUrl bytes|hits|duration
336 Used to set how SquidAnalyzer sort Network, User and User detailed
337 Urls reports screen. Value can be: bytes, hits or duration. Default
338 is bytes. Note that OrderUrl is limited to User detailed Urls
339 reports and does not apply to Top Url and Top domain report where
340 there is three reports each already ordered.
341
342 OrderMime bytes|hits
343 Used to set how SquidAnalyzer sort Mime types report screen Value
344 can be: bytes or hits. Default is bytes.
345
346 UrlReport 0|1
347 Should SquidAnalyzer display user url details. This will show all
348 URL read by user. Take care to have enougth space disk for large
349 user. Default is 0, no url detail report.
350
351 UserReport 0|1
352 Should SquidAnalyzer display user details. This will show statistics
353 about user. Default is 1, show user detail report. Disable it to be
354 able to remove any user related reports, statistics about URL and
355 domains will remain.
356
357 UrlHitsOnly 0|1
358 Enable this directive if you don't want the tree Top URL and Domain
359 tables. You will just have the table of Url/Domain ordered per hits
360 then you can still sort the URL/Domain order by clicking on each
361 column. This is useful when you have set a high value to TopNumber.
362
363 QuietMode 0|1
364 Run in quiet mode for batch processing or print debug information.
365 Default is 0, verbose mode.
366
367 CostPrice price/Mb
368 Used to set a cost of the bandwidth per Mb. If you want to generate
369 invoice per Mb for bandwidth traffic this can help you. Value 0 mean
370 no cost, this is the default value, the "Cost" column is not
371 displayed
372
373 Currency currency_abreviation
374 Used to set the currency of the bandwidth cost. Preferably the html
375 special character. Default is €
376
377 TopNumber number
378 Used to set the number of top url and second level domain to show.
379 Default is top 100.
380
381 TopStorage number
382 Top number of url to preserve on each data file sorted by OrderUrl.
383 On huge access log it will improve a lot the performances but you
384 will have less precision in the top url. Default to 0, all url will
385 be stored.
386
387 TopUrlUser Use this directive to show the top N users that look at an
388 URL or a domain. Set it to 0 to disable this feature. Default is top 10.
389 Exclude exclusion_file
390 Used to set client ip addresses, network addresses, auth login or
391 uri to exclude from report.
392
393 You can define one by line exclusion by specifying first the type of
394 the exclusion (USER, CLIENT or URI) and a space separated list of
395 valid regex.
396
397 You can also use the NETWORK type to define network address with
398 netmask using the CIDR notation: xxx.xxx.xxx.xxx/n
399
400 See example below:
401
402 NETWORK 192.168.1.0/24 10.10.0.0/16
403 CLIENT 192\.168\.1\.2
404 CLIENT 10\.169\.1\.\d+ 192\.168\.10\..*
405 USER myloginstr
406 USER guestlogin\d+ guestdemo
407 URI http:\/\/myinternetdomain.dom.*
408 URI .*\.webmail\.com\/.*\/login\.php.*
409
410 you can have multiple line of the same exclusion type.
411
412 Include inclusion_file
413 Used to set client ip addresses, network addresses or auth login to
414 include into the report. All others will not be included. It works
415 as the opposite of the Include parameter.
416
417 You can define one by line inclusion by specifying first the type of
418 the inclusion (USER or CLIENT) and a space separated list of valid
419 regex.
420
421 You can also use the NETWORK type to define network address with
422 netmask using the CIDR notation: xxx.xxx.xxx.xxx/n
423
424 See example below:
425
426 NETWORK 192.168.1.0/24 10.10.0.0/16
427 CLIENT 192\.168\.1\.2
428 CLIENT 10\.169\.1\.\d+ 192\.168\.10\..*
429 USER myloginstr
430 USER guestlogin\d+ guestdemo
431 URI http:\/\/myinternetdomain.dom.*
432 URI .*\.webmail\.com\/.*\/login\.php.*
433
434 you can have multiple line of the same inclusion type.
435
436 ExcludedMethods
437 This directive allow exclusion of some unwanted methods in report
438 statistics like HEAD, POST, CONNECT, etc. Can be a comma separated
439 list of methods.
440
441 ExcludedMimes
442 This directive allow exclusion of some unwanted mimetypes in report
443 statistics like text/html, text/plain, or more generally text/*,
444 etc. Can be a comma separated list of perl regular expression. Ex:
445
446 ExcludedMimes text/.*,image/.*
447
448 Lang
449 Used to set the translation file to be used. Value must be set to a
450 file containing all string translated. See the lang directory for
451 translation files. Default is defined internally in English.
452
453 ExcludedCodes
454 This directive allow exclusion of some unwanted codes in report
455 statistics like TCP_DENIED/403 which are generated when a user
456 accesses a page the first time without authentication. Can be a
457 comma separated list of methods. Default is none, all codes will be
458 parsed.
459
460 DateFormat
461 Date format used to display date (year = %y, month = %m and day =
462 %d) You can also use %M to replace month by its 3 letters
463 abbreviation. Default: %y-%m-%d
464
465 SiblingHit
466 Adds peer cache hit (CD_SIBLING_HIT) to be taken has local cache
467 hit. Enabled by default, you must disabled it if you don't want to
468 report peer cache hit onto your stats.
469
470 TransfertUnit
471 Allow one to change the default unit used to display transfert size.
472 Default is BYTES, other possible values are KB, MB and GB.
473
474 MinPie
475 Minimum percentage of data in pie's graphs to not be placed in the
476 others item. Lower values will be summarized into the others item.
477
478 Locale
479 Set this to your locale to display generated date in your language.
480 Default is to use the current locale of the system. If you want date
481 in German for example, set it to de_DE.
482
483 Rapport genere le mardi 11 decembre 2012, 15:13:09 (UTC+0100).
484
485 with a Locale set to fr_FR.
486
487 MaxFormatError
488 When SquidAnalyzer find a corrupted line in his data file, it exit
489 immedialtly. You can force him to wait for a certain amount of
490 errors before exiting. Of course you might want to remove the
491 corrupted line before the next run. This can be useful if you have
492 special characters in some fields like mime type.
493
494 TimeZone
495 Set timezone to use when SquidAnalyzer is used in a different server
496 than the one running squid and there is a different timezone between
497 these two machines. The value must follow format: +/-HH. Default is
498 to use local time. For example:
499
500 TimeZone +01
501
502 for a log file generated on zone Europe/Paris with UTC+0100 and
503 parsed on a computer with different timezone.
504
505 UseUrlPort
506 Enable this directive if you want to include port number into Url
507 statistics. Default is to remove the port information from the Url.
508
509 UpdateAlias
510 Enable this directive if you want to apply immedialtly the changes
511 made in aliases files to avoid duplicates. You still have to use
512 --rebuild to recreate previous reports with new aliases. Enabling
513 this will imply a lost of performances with huges log files.
514
515 TimeStart and TimeStop
516 The two following configuration directive allow you to specify a
517 start and stop time. Log line out of this time range will not be
518 parsed. The format of the value is HH:MM
519
520 SUPPORT
521 Release announcement
522 Please follow us on twitter to receive release announcement and latest
523 news : https://twitter.com/SquidAnalyzer
524
525 Bugs and Feature requests
526 Please report any bugs, patches, discussion and feature request using
527 tools on the git repository at https://github.com/darold/squidanalyzer.
528
529 How to contribute ?
530 Any contribution to build a better tool is welcome, you just have to
531 send me your ideas, features request or patches using the tools on the
532 git repository at https://github.com/darold/squidanalyzer
533
534 You can also support the developer by donate some contribution by
535 clicking on the "Donate" button on the SquidAnalyzer web site at
536 http://squidanalyzer.darold.net/
537
538 AUTHOR
539 Gilles DAROLD <gilles@darold.net>
540
541 COPYRIGHT
542 Copyright (c) 2001-2017 Gilles DAROLD
543
544 This package is free software and published under the GPL v3 or above
545 license.
546