Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
DNSHISTORY(1)			 User Commands			 DNSHISTORY(1)

       dnshistory  -  processes	 various log file formats doing	dns IP Address
       lookups.	Store these pairs in a database	for later retrieval.

       dnshistory [OPTION]... [--file=FILE]

       dnshistory is Copyright (C) 2004, 2005, 2006, 2007 Stephen McInerney

       dnshistory currently processes Apache CLF and Combined logs, Squid  ac-
       cess  logs,  FTP	xferlog	files and iptables based logs.	The log	format
       is auto-detected.

       dnshistory has five modes of operation:

       +o      Do Lookups. The default mode. Given a web	log  file,  dnshistory
	      will  perform  DNS reverse lookups on each unique	IP Address and
	      store the	results	in a history database.

       +o      Do Translations. Given a raw web log file, dnshistory will  make
	      use  of a	previously created history database and	send to	STDOUT
	      the same web log but with	addresses replaced by the Fully	Quali-
	      fied Domain Name as previously looked up.

       +o      Do  Recombining. Given two web log files,	one raw	and one	previ-
	      ously translated (eg. by using dnstran): Create a	history	 data-
	      base from	the values in these separate log files.

       +o      Do Dump. Dump a given history database to	STDOUT.

       +o      Do  Import.  Import  a previously	dumped history database	from a
	      given file.

       +o      Show History. Given one or more  IP  Addresses  on  the  command
	      line, show the history of	those addresses.

       The lookups make	use of threads for maximum speed, and use the standard
       resolution libraries on a system. Thus hosts files, NIS,	LDAP and other
       name resolution methods should work transparently.

       It  is  strongly	 recommended  that for massive lookups a DNS server is
       "nearby". Preferably not	a forwarding server.

       dnshistory can read .gz files. STDIN is assumed to not be gz encoded.

       -L --dolookups
	      The default mode.	Given a	log file,  either  via	STDIN  or  via
	      '-f', do the lookups and store the results.

       -T --dotranslate
	      Given  a	log file, either via STDIN or via '-f',	lookup each IP
	      Address from the history database; replace the IP	 Address  with
	      the FQDN and send	the newly updated log line to STDOUT.

       -R --dorecombine=FILE
	      Given  a	previously  translated file (eg. via dnstran) via this
	      option for the names, do the lookups for a file given via	 STDIN
	      or '-f' and store	the results. This file can be gz encoded.

	      The  date/time of	each stored entry is taken as being the	actual
	      time for the lookup stored in the	recombine log  file.  This  is
	      probably incorrect, but "Good Enough".

       -D --dodump
	      Dump the history database	to STDOUT.

       -I --doimport=FILE
	      Given a previously dumped	database, import that into a new data-
	      base. Will fail and exit if the chosen database already exists.

       -S --showhistory
	      Given one	or more	IP Addresses show their	history.  Address  are
	      the  last	item(s)	on the command line.  Addresses	with no	as yet
	      discovered FQDN will display 'NONAME'.  The Date/Time  displayed
	      is  formatted  as	 YYYY-MM-DD:hh:mm:ss,  vs  the	'seconds since
	      epoch' for "--dodump"

	      By default dnshistory will attempt to autodetect	what  type  of
	      logfile is being processed. By using this	option,	the autodetec-
	      tion is overridden. The choices are: auto, clf  or  www,	squid,
	      ftp or iptables.

       -c --cache=SIZE
	      Set  the	size  of the memory cache to use. Value	is in Mb.  De-
	      fault is 20Mb.

       -d --database=FILE
	      Change the default database file to use to store stateful	data.

       -f --file=FILE
	      Web Log File to process. This file can be	gz encoded.
	      Will use STDIN if	not set

       -h --help
	      Help screen. Very	brief.

       -l --maxlookups=NUMBER
	      The maximum number of lookups to attempt.	The default is 1. This
	      has not shown to be at all useful	in testing...

       -m --maxthreads=NUMBER
	      How  many	 name lookup threads to	spawn off. The default is 100.
	      Setting this too high can	do evil	things to  bandwidth  and  the
	      CPU usage	of any queried DNS server(s).

	      If  doing	 lots of DNS queries, setting this too high can	have a
	      very negative impact on the ability to successfully resolve any-

       -t --timeout=VALUE
	      The  time	 in seconds before a stored DNS	value is deemed	"old".
	      The default is 7 days.

       -v --verbose
	      Verboseness of a run. More v's will increase the level  of  ver-
	      bosity,  up to a maximum of 5. All of the	higher levels are only
	      of value for debugging purposes.

       -V --version
	      Display the version information and exit

       -w --wait
	      Delay time between query retries within a	single run

       At verbose level	1 (-v) some  success/failure  counters	will  be  dis-
       played. As well as any problematic log lines to STDERR.
       At  verbose  level  2  (-vv)  lines  that  may not match	up (eg.	Due to
       dnstran modifying referrers or URL's) will be sent to STDERR.

       A typical run, using a database in /tmp/	(/tmp/c.db), and a log file in
       the current directory (test.log). First,	do the lookups:

	   dnshistory -d /tmp/c.db -f test.log

       Then  the  translation run for input into, for example, a web log anal-

	   dnshistory -T -d /tmp/c.db -f test.log | webalizer ....

       Do three	attempts on failed queries, with a 2 second delay between  re-

	   dnshistory -l 3 -w 2	-d /tmp/c.db -f	test.log

       Lookup	 and	Display	  the	history	  of   three   IP   Addresses:,,

	   dnshistory -S -d /tmp/c.db

       Import a	previously dumped database via dnsdb.dump

	   dnshistory -I dnsdb.dump -d /tmp/d.db

	   The default history database	file.

       +o      Ignores IP Addresses located in the URL and Referrer fields.

       Stephen McInerney <>

Linux				 JANUARY 2007			 DNSHISTORY(1)


Want to link to this manual page? Use this URL:

home | help