Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
FIFOLOG(1)		  BSD General Commands Manual		    FIFOLOG(1)

     fifolog_create, fifolog_writer, fifolog_reader -- initialize, write, seek
     and extract data from a fifolog

     fifolog_create [-l	record-size] [-r record-count] [-s size] file
     fifolog_reader [-t] [-b tstart] [-B Tstart] [-e tend] [-E Tend]
		    [-o	ofile] [-R regexp] [-T timefmt]	file
     fifolog_writer [-w	write-rate] [-s	sync-rate] [-z compression] file

     Fifologs provide a	compact	round-robin circular storage for recording
     text and binary information to permanent storage in a bounded and pre-
     dictable fashion, time and	space wise.

     A fifolog can be stored either directly on	a disk partition or in a regu-
     lar file.

     The input data stream is encoded, compressed and marked up	with time-
     stamps before it is written to storage, such that it is possible to seek
     out a particular time interval in the stored data,	without	having to de-
     compress the entire logfile.

     The fifolog_create	utility	is used	to initialize the first	sector of a
     disk device or file system	file to	make it	a fifolog and should be	called
     only once.

     Running fifolog_create on an existing fifolog will	reset it so that
     fifolog_reader and	fifolog_writer will not	see the	previous contents.
     (The previous contents are	not physically erased, and with	a bit of hand-
     work all but the first record can be easily recovered.)

     If	the file does not already exist, fifolog_create	will attempt to	create
     and ftruncate(2) it to the	specified size,	defaulting to 86400 records of
     512 bytes if the -r, -l or	-s options do not specify otherwise.

     The fifolog_writer	utility	will read standard input and write it to the
     end of the	fifolog	according to the parameters given.

     Writes happen whenever the	output buffer is filled	with compressed	data
     or	when either of two timers expire, forcing a partially filled buffer to
     be	written.

     The first and faster timer, -w write-rate,	forces available data to be
     written but does not flush	and reset the compression dictionary.  This
     timer is intended to minimize the amount of logdata lost in RAM in	case
     of	a crash	and by default it fires	10 seconds after the previous write.

     The second	and slower timer, -s sync-rate,	forces a full flush and	reset
     of	the compression	engine and causes the next record written to be	a syn-
     chronization point	with an	uncompressed timestamp,	making it possible to
     start reading the logfile from that record.  By default this timer	fires
     a minute after the	previous sync.

     The -z compression	option controls	the zlib(3) compression	level; legal
     values are	zero to	nine which is the default.

     The fifolog_reader	utility	will retrieve records from the fifolog accord-
     ing to the	specified parameters and write them either to standard output
     or	the file specified with	-o.

     It	is possible to specify a start and end time to limit the amount	of
     data fifolog_reader will report.  The lower-case variants -b and -e take
     a time_t value, whereas the upper-case variants -B	and -E take human-
     readable specifications such as "1	hour ago".

     The -t option forces timestamps to	be formatted as	"YYYYMMDDhhmmss" in-
     stead of as time_t, and -T	allows the specification of an strftime(3)
     formatting	string.

     Finally, records can be filtered such that	only records matching the
     (REG_BASIC) regular expression specified with -R are output.

     The data stored in	the fifolog consists of	three layers, an outer layer
     that allows searches to synchronization points based on timestamps	with-
     out having	to decompress and decode the actual contents, a	compression
     layer implemented with zlib(3), and an inner serialization	and timestamp-
     ing layer.

     The exact encoding	is described in	the fifolog.h file.

     Fifolog is	particularly well suited for use on Flash based	media, where
     it	results	in much	lower write-wear, than a file system with regular log
     files rotated with	newsyslog(8) etc.

     Create a fifolog with 1024*1024 records of	512 bytes:

	   fifolog_create -r 10m /tmp/fifolog

     Write a single record to this file:

	   date	| fifolog_writer /tmp/fifolog

     Read it back with human readable timestamps:

	   fifolog_reader -t /tmp/fifolog

     One particular useful use of fifolog_writer is with syslogd(8) using a
     line such as this in syslog.conf(5):

	   *.* |fifolog_writer /var/log/syslog_fifolog

     The fifolog tools have been liberated from	an open	source SCADA applica-
     tions called "measured", which monitors and controls remote radio naviga-
     tion transmitters for the Danish Air Traffic Control system.

     The fifolog tools were written by Poul-Henning Kamp.

BSD			       February	9, 2008				   BSD


Want to link to this manual page? Use this URL:

home | help