Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
awffull(1)							    awffull(1)

NAME
       AWFFull - A Webalizer Fork, Full	o' features

SYNOPSIS
       awffull [...] [log-file]

DESCRIPTION
       AWFFull	is  a  web server log analysis program based on	The Webalizer.
       AWFFull produces	usage statistics in HTML format	 for  viewing  with  a
       browser.	 The results are presented in both columnar and	graphical for-
       mat, which  facilitates	interpretation.	 Yearly,  monthly,  daily  and
       hourly  usage  statistics are presented,	along with the ability to dis-
       play usage by site, URL,	referrer, user	agent  (browser),  user	 name,
       search strings, entry/exit pages, and country (some information may not
       be available if not present in the log file being processed).

       AWFFull supports	the following log formats shown	in the following vari-
       able list:

       CLF    (common log format) log files

       Combined
	      log  formats  as	defined	 by NCSA and others, and variations of
	      these which it attempts to handle	intelligently

       xferlog
	      wu-ftpd formatted	log files allowing analysis  of	 ftp  servers,
	      and squid	proxy logs.
	      Note

	      Logs  may	also be	compressed, via	gzip. If a compressed log file
	      is detected, it will be automatically uncompressed while	it  is
	      read.  Compressed	 logs must have	the standard gzip extension of
	      .gz.

       This documentation applies to AWFFull Version 3.8.2

CHANGES	FROM WEBALIZER
       AWFFull is based	on The Webalizer code and has a	number	of  large  and
       small changes.  These include:

       o Beyond	 the  raw statistics: Making use of published formulae to pro-
	 vide additional insights into site usage

       o GeoIP IP Address look-ups for more accurate country detection

       o Resizable graphs

       o Integration with GNU gettext allowing for ease	of translations.  Cur-
	 rently	32 languages are supported.

       o Display more than 12 months of	the site history on the	front page.

       o Additional page count tracking	and sort by same.

       o Some  minor visual tweaks, including Geolizer's use of	Kb, Mb etc for
	 Volumes

       o Additional Pie	Charts for URL counts, Entry and Exit Pages, and Sites

       o Horizontal lines on graphs that are more sensible and easier to read

       o User Agent and	Referral tracking is now calculated via	PAGES not HITS

       o GNU style long	command	line options are now supported (eg --help)

       o Can choose what is a page by excluding	`what isn't' vs	 the  original
	 `what is' method

       o Requests  to  the site	being analysed are displayed with the matching
	 referring URL

       o A Table of 404	Errors,	and the	referring URL can be generated

       o An external CSS file can be used with the generated html

       o Manual	performance optimisation of the	config file is now easier with
	 a post	analysis summary output

       o Specified IP Addresses	can be assigned	to a given country

       o Additional Dump options for detailed analysis with other tools

       o Lotus Domino v6 logs are now detected and processed

       Additional  changes  and	improvements are planned and undergoing	imple-
       mentation. See the TODO file for	details.

NEW REPORT MEASUREMENTS
       With version 3.8.1 of AWFFull, several new measured results  have  been
       added to	the detailed report monthly page.

       Single Access
	      Single Access Pages - the	only page seen within a	given visit

       Stickiness
	      How  useful  a  given entry page is to draw Visitors deeper into
	      your site

       Popularity
	      The Ratio	of Page	Entries	to Page	Exits

       These metrics can help towards improving	insight	in the	usage  of  the
       processed  web  site.  And  hence allow the site	owner to make positive
       change to make the site more useful to site visitors. All three metrics
       appear  in  the `Entry Pages' Report. `Popularity' is also on the `Exit
       Pages' Report.

       Single Access

       More completely:	`Single	Access Pages'. This is a report	on the	number
       of  times that a	given page was the only	page viewed within a Visit. Or
       in English, Someone came	to your	website. They only  viewed  one	 page.
       The number is the cumulative count of people who	did this for that par-
       ticular page. Why is this useful? Identifying those  entry  pages  that
       don't  draw  visitors deeper into your site. Or seeing entry pages that
       shouldn't be entry pages. It's also a reality check  against  the  next
       two  values which are calculated	from this number. The number generated
       should be a subset of the `Entry	Page Views' and/or `Exit  Page	Views'
       metric. If it isn't? Let	me know, we have a bug.	:-)

       Stickiness

       Is  calculated as 1 - (Single Access / Entry Page Views)	expressed as a
       percentage. In essence Stickiness describes how useful  a  given	 entry
       page  is	to draw	Visitors deeper	into your site.	The stickier the page,
       the more	folk are caught	by it. :-) The closer to 100% the better. Gen-
       erally.	Certain	pages within YOUR website may not make sense to	have a
       high stickiness or even > 5%. This measurement is a clue	to understand-
       ing  how	 your  site is used, it	is not a rule. How is this useful? How
       and where are people entering your web  site.  Does  that  make	sense?
       Should  it  be here or there? What can you change to fix	this and hence
       improve their use of your website.

       Popularity

       Popularity is the Ratio of Page Entries to Page Exits. o	If  it	equals
       1.0?  Then  the	number	of visitors to your site who started with that
       page, equals the	number who left	at that	page. o	If greater  then  1.0,
       then  more people entered here then left. o If less then	0? More	people
       left from here then entered. I personally find this metric one  of  the
       more useful "At a Glance: How are Pages Performing" metrics. One	of the
       difficulties with using this particular metric is that certain  numbers
       will  NOT  make	sense for YOUR site. In	that a natural exit page would
       expect to have a	very low Popularity. It's an exit page,	not  an	 entry
       page.  So  if  an exit page has a high popularity, then you have	a real
       problem.	Likewise, a low	Popularity for an entry	page is	unlikely to be
       a Good Thing(tm).

       "Where  &  Why?"	 All three of these metrics are	covered	very nicely in
       Hack #58	from "Web Site Measurement Hacks" [1]. Which is	where,	credit
       where credit due, the inspiration to add	these metrics came from.

RUNNING	AWFFULL
       AWFFull	is  designed to	be run from a Unix command line	prompt or as a
       crond(8)	job.  There is no need to run with super-user  privleges,  and
       indeed, is preferable NOT to.

       Once executed, the general flow of the program is:

       A  default  configuration  file	is  scanned  for,  /usr/local/etc/awf-
       full.conf and, if found,	is used.

       Any command line	arguments given	to the program are  parsed.  This  may
       include the specification of one	or more	configuration files, which are
       processed at the	time it	is encountered.

       It can be useful	to have	multiple config	files. A master	used for  mul-
       tiple sites, and	individualised config files. Do	be aware that last op-
       tion set	wins. So last config file, or if after a config	file,  command
       line  options.  Useful if you desire to send the	output to an alternate
       directory.

       If a log	file was specified, it is opened and made ready	 for  process-
       ing.  If	 no  log  file	was given, STDIN is used for input. If the log
       filename	'-' is specified, STDIN	will be	forced.

       If an output directory was specified, AWFFull changes to	that directory
       in preparation for generating output. If	no output directory was	given,
       the current directory is	used.

       If no hostname was given, the program attempts to get the hostname  us-
       ing a uname(2) system call. If that fails, localhost is used.

       A  history file is searched for in the current directory	(output	direc-
       tory) and read if found.	This file keeps	totals	for  previous  months,
       which  is used in the main index.html HTML document. Note: The file lo-
       cation can now be specified with	the HistoryName	configuration option.

       If incremental processing was specified,	a data file  is	 searched  for
       and  loaded  if found, containing the 'internal state' data of the pro-
       gram at the end of a previous run. Note:	The file location can  now  be
       specified with the IncrementalName configuration	option.

       Main  processing	 begins	 on  the  log  file. If	the log	spans multiple
       months, a separate HTML document	is created for each month.  After main
       processing,  the	 main  index.html page is created, which has totals by
       month and links to each months HTML document.

       A new history file is saved to disk, which includes totals generated by
       AWFFull during the current run.

       If  incremental	processing  was	specified, a data file is written that
       contains	the 'internal state' data at the end of	this run.

INCREMENTAL PROCESSING
       Version 1.2x of The Webalizer added incremental run capability.	Simply
       put,  this  allows  processing large log	files by breaking them up into
       smaller pieces, and processing these pieces instead. What this means in
       real  terms  is	that you can now rotate	your log files as often	as you
       want, and still be able to produce monthly usage	statistics without the
       loss  of	any detail. Basically, AWFFull saves and restores all internal
       data in a file named awffull.current. This allows the program to	'start
       where  it  left off' so to speak, and allows the	preservation of	detail
       from one	run to the next. The data file is placed in the	current	output
       directory,  and	is a plain ASCII text file that	can be viewed with any
       standard	text editor. It's location and name may	be changed  using  the
       IncrementalName configuration keyword.

       Some  special  precautions  need	to be taken when using the incremental
       run capability of AWFFull. Configuration	options	should not be  changed
       between	runs,  as  that	 could	cause  corruption of the internal data
       stored. For example, changing the MangleAgents level will cause differ-
       ent  representations of user agents to be stored, producing invalid re-
       sults in	the user agents	section	of the report. If you need  to	change
       configuration  options, do it at	the end	of the month after normal pro-
       cessing of the previous month and before	processing the current	month.
       You may also want to delete the awffull.current file as well.

       AWFFull	also  attempts to prevent data duplication by keeping track of
       the timestamp of	the last record	processed. This	timestamp is then com-
       pared  to  current  records  being processed, and any records that were
       logged previous to that timestamp are ignored. This, in theory,	should
       allow  you  to  re-process  logs	 that  have already been processed, or
       process logs that contain a mix of processed/not	yet processed records,
       and not produce duplication of statistics.

       The only	time this may break is if you have duplicate timestamps	in two
       separate	log files. Any records in the second log file that do have the
       same  timestamp	as the last record in the previous log file processed,
       will be discarded as if they had	already	been processed.	There are lots
       of  ways	 to prevent this however, for example, stopping	the web	server
       before rotating logs will prevent this situation, or using a tool  such
       as  cronolog  (<http://cronolog.org/>).	 This  setup also necessitates
       that you	always process logs in	chronological  order,  otherwise  data
       loss will occur as a result of the timestamp compare.

REVERSE	DNS LOOKUPS
       AWFFull	no longer supports DNS lookups.	Please use an external program
       such as DNShistory or DNSTran instead.

       o <http://www.summary.net/soft/dnstran.html>

       o <http://www.stedee.id.au/dnshistory>

       With version 3.7.1 of AWFFull, GeoIP capability can  be	used  for  im-
       proved country detection.

COMMAND	LINE OPTIONS
       AWFFull	supports  many different configuration options that will alter
       the way the program behaves and generates output. Most of these can  be
       specified  on  the  command line, while some can	only be	specified in a
       configuration file. The command line options  are  listed  below,  with
       references  to  the corresponding configuration file keywords. See also
       awffull.conf(5).

       General Options

       -h, --help
	      Display all available command line options and exit program

       -V, --version
	      Display program version and exit program

       -v --verbose
	      Verbosity	Display	debugging information for errors and warnings.
	      Multiple v's will	increase the amount of information displayed.

       --match_counts
	      Display optimisation useful information pertaining to the	number
	      of matches against various Group,	Hide and Ignore	options.

       -i --ignore_history
	      IgnoreHist Ignore	history. USE WITH CAUTION.   This  will	 cause
	      AWFFull to ignore	any previous monthly history file only.	Incre-
	      mental data (if present) is still	processed.

       -p --preserve_state
	      Incremental Preserve internal data between runs.

       -T --timing
	      TimeMe. Force display of timing information at end  of  process-
	      ing.

       -c --config=FILE
	      Use configuration	file FILE

       -n NAME
	      HostName.	Use the	hostname NAME.

       -o --output=DIR
	      OutputDir. Use output directory DIR.

       -t NAME
	      ReportTitle. Use NAME for	report title.

       F --logtype=TYPE
	      LogType. Specify log type	to be processed.  Value	can be one of:
	      auto, clf, combined, domino, ftp or squid	format.	If not	speci-
	      fied,  will default to auto format. FTP logs must	be in standard
	      wu-ftpd xferlog format. A	value of `auto'	states	that  the  log
	      format automatically ascertained.

       -f --fold
	      FoldSeqErr. Fold out of sequence log records back	into analysis,
	      by treating as if	they were the same date/time as	the last  good
	      record.  Normally,  out  of  sequence log	records	are simply ig-
	      nored.

       -Y     CountryGraph. Suppress country graph.

       -G     HourlyGraph. Suppress hourly graph.

       -x NAME
	      HTMLExtension. Defines the HTML file extension  to  use  on  the
	      created report files. If not specified, defaults to html.	Do not
	      include the leading period.

       -H     HourlyStats. Suppress hourly statistics.

       -L     GraphLegend. Suppress color coded	graph legends.

       -l NUM GraphLines. Use background lines.	The number of lines and	 where
	      to place are automatically calculated. For backwards compatibil-
	      ity, any number >	0 enables.  Use	 zero  ('0')  to  disable  the
	      lines.

       -P NAME"
	      PageType.	 Specify  file	extensions  that are considered	pages.
	      Sometimes	referred to as pageviews.

       -m NUM VisitTimeout. Specify the	Visit  timeout	period.	 Specified  in
	      number  of  seconds. Default is 1800 seconds (30 minutes). Some-
	      times referred to	as sessions.

       -I NAME
	      IndexAlias. Use the filename name	as an additional alias for in-
	      dex.

       -M NUM MangleAgents.  Mangle  user  agent names according to the	mangle
	      level specified by num.

	      Mangle levels are:

	      5	- Browser name and major version

	      4	- Browser name,	major and minor	version

	      3	- Browser name,	major version, minor version  to  two  decimal
	      places

	      2	- Browser name,	major and minor	versions and sub-version

	      1	- Browser name,	version	and machine type if possible

	      0	- All information (left	unchanged).

       -g NUM GroupDomains.  Automatically group sites by domain. The grouping
	      level specified by num can be thought of as 'the number of dots'
	      to  display in the grouping. The default value of	0 disables any
	      domain grouping.

       Hide Options

       -a NAME
	      HideAgent. Hide user agents matching name.

       -r NAME
	      HideReferrer. Hide referrer matching name.

       -s NAME
	      HideSite.	Hide site matching name.

       -X NAME
	      HideAllSites. Hide all individual	sites (only display groups).

       -u NAME
	      HideURL. Hide URL	matching name.

       Table size options

       -A --top_agents=NUM
	      TopAgents. Display the top num user agents table.

       -R --top_refers=NUM
	      TopReferrers. Display the	top num	referrers table.

       -S --top_sites=NUM
	      TopSites.	Display	the top	num sites table.

       -U --top_urls=NUM
	      TopURLs. Display the top num URL's table.

       -C --top_countries=NUM
	      TopCountries. Display the	top num	countries table.

       -e --top_entry=NUM
	      TopEntry.	Display	the top	num entry pages	table.

       -E --top_exit=NUM
	      TopExit. Display the top num exit	pages table.

       Other Options

       --use_geoip
	      Enables the use of the Maxmind GeoIP capability for  more	 accu-
	      rate detection of	countries.

	      NOTE! Do not enable GeoIP	if you analyse files that have had the
	      IP Address translated to a Fully Qualified Host Name.   Use  ei-
	      ther  raw	 IP  Addresses	and GeoIP, or Names and	disable	GeoIP.
	      ie. Don't	use GeoIP AND DNShistory.

       --match_counts
	      Display the various Group/Hide etc Match Counts. This option  is
	      ideal for	optimisation of	the awffull.conf file. Just be careful
	      with optimising Agents in	particular, as the order is  typically
	      important.

CONFIGURATION FILES
       See the awffull.conf(5) man page	for complete details of	all configura-
       tion options.

       Configuration files are standard	ASCII(7) text files that may be	creat-
       ed or edited using any standard editor.

       Blank lines and lines that begin	with a pound sign ('#')	are ignored.

       Any  other lines	are considered to be configuration lines, and have the
       form `Keyword Value', where the	`Keyword'  is  one  of	the  currently
       available  configuration	keywords (see awffull.conf(5)),	and `Value' is
       the value to assign to that particular option.

       Any text	found after the	keyword	up to the end of the line  is  consid-
       ered  the keyword's value, so you should	not include anything after the
       actual value on the line	that is	not actually part of the  value	 being
       assigned.  The file sample.conf provided	with the distribution contains
       lots of useful documentation and	examples as well.

       Certain "Keywords" will accept a	2nd value. In  those  situations,  the
       first  value  may  be enclosed in double	quotes (") to allow for	white-
       space.

SEE ALSO
       awffull.conf(5)

BUGS
       None currently known. YMMV....

       Report bugs to <https://bugs.launchpad.net/awffull>, or use  the	 email
       discussion list:	<awffull@stedee.id.au>

NOTES
       In  case	 it is not obvious: AWFFull is a play/pun on the word `awful',
       and is pronounced the same way. Yes it was deliberate.

REFERENCES
       [1] Web Site Measurement	Hacks. Eric T. Peterson	(and others).  O'Reil-
       ly. ISBN	0-596-00988-7.

				  2008-Dec-13			    awffull(1)

NAME | SYNOPSIS | DESCRIPTION | CHANGES FROM WEBALIZER | NEW REPORT MEASUREMENTS | RUNNING AWFFULL | INCREMENTAL PROCESSING | REVERSE DNS LOOKUPS | COMMAND LINE OPTIONS | CONFIGURATION FILES | SEE ALSO | BUGS | NOTES | REFERENCES

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=awffull&sektion=1&manpath=FreeBSD+12.1-RELEASE+and+Ports>

home | help