Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
rwflowpack(8)			SiLK Tool Suite			 rwflowpack(8)

NAME
       rwflowpack - Collect flow data and store	it in binary SiLK Flow files

SYNOPSIS
	 rwflowpack [--input-mode=MODE]	INPUT_MODE_SPECIFIC_SWITCHES
	       [--output-mode=MODE] OUTPUT_MODE_SPECIFIC_SWITCHES
	       { --log-destination=DESTINATION
		 | --log-pathname=FILE_PATH
		 | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]
		   [--log-post-rotate=COMMAND] }
	       [--no-file-locking] [--flush-timeout=VAL]
	       [--file-cache-size=VAL] [--pack-interfaces]
	       [--byte-order=ENDIAN] [--compression-method=COMP_METHOD]
	       [--error-directory=DIR_PATH] [--archive-directory=DIR_PATH]
	       [--flat-archive]	[--post-archive-command=COMMAND]
	       [--site-config-file=FILENAME] [--log-level=LEVEL]
	       [--log-sysfacility=NUMBER] [--pidfile=FILE_PATH]
	       [--no-chdir] [--no-daemon]

       To collect flow data over the network or	directory polling (default):

	 rwflowpack [--input-mode=stream] --sensor-configuration=FILE_PATH
	       [--packing-logic=PLUGIN]	[--sensor-name=SENSOR]
	       [--polling-interval=NUMBER] ...

       To collect from local files containing flows created by flowcap(8):

	 rwflowpack --input-mode=fcfiles --incoming-directory=DIR_PATH
	       --sensor-configuration=FILE_PATH	[--packing-logic=PLUGIN]
	       [--polling-interval=NUMBER] ...

       To collect from a single	file containing	NetFlow	v5 PDUs:

	 rwflowpack --input-mode=pdufile --netflow-file=FILE_PATH
	       --sensor-configuration=FILE_PATH	[--packing-logic=PLUGIN]
	       [--sensor-name=SENSOR] ...

       To respool SiLK Flows without modifying the class, type,	or sensor:

	 rwflowpack --input-mode=respool --incoming-directory=DIR_PATH
	       [--polling-interval=NUMBER] ...

       To store	the SiLK Flow files on the local machine (default):

	 rwflowpack ...	[--output-mode=local-storage]
	       --root-directory=DIR_PATH ...

       To create incremental files to be processed by rwflowappend(8):

	 rwflowpack ...	--output-mode=incremental-files
	       --incremental-directory=DIR_PATH	...

       To create incremental files to be processed by rwflowappend
       (deprecated):

	 rwflowpack ...	--output-mode=sending --sender-directory=DIR_PATH
	       --incremental-directory=DIR_PATH	...

       Help options:

	 rwflowpack --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
	       { --verify-sensor-config	| --verify-sensor-config=VERBOSE }

	 rwflowpack --help

	 rwflowpack --version

DESCRIPTION
       rwflowpack is a daemon that runs	as part	of the SiLK flow collection
       and packing tool-chain.	The primary job	of rwflowpack is to convert
       each incoming flow record to the	SiLK Flow format, categorize each
       incoming	flow record (e.g., as incoming or outgoing), set the sensor
       value for the record, and determine which hourly	file will ultimately
       store the record.

       The settings that rwflowpack uses to categorize each flow record	are
       determined by two textual configuration files and compiled code that is
       referred	to as packing logic.

       The first of the	configuration files is silk.conf(5) which specifies
       the classes, types, and sensors to use at your site.  There are several
       different ways to specify the location of this file as detailed in the
       "FILES" section below.

       The second configuration	file is	the sensor.conf(5) file, whose
       location	is specified via the --sensor-configuration switch.  This file
       contains	multiple sensor	blocks,	where each block contains information
       used to categorize flow records captured	at that	particular sensor.
       This file also contains probe blocks which specify how to collect
       NetFlow v5, IPFIX, and/or NetFlow v9 flow records, and a	mapping	of
       probes to sensors.  See the sensor.conf(5) manual page for details.

       The packing logic uses the information from the silk.conf and
       sensor.conf files to categorize each flow record.  By categorizing a
       flow record, we mean determine whether the record is incoming or
       outgoing	and assign a class/type	pair (also called a flowtype) to the
       record.	The flowtype along with	the starting hour of the record	and
       the sensor where	the record was collected form a	triple which
       determines into which file a flow record	is stored.  The	files that
       rwflowpack produces have	the form flowType-sensorName_YYYYMMDD.HH where
       flowType	encodes	the class/type pair, sensorName	is the sensor where
       the flows were collected, and YYYYMMDD.HH is the	year, month, day, and
       hour when the flow started.

       For more	information on how rwflowpack categorizes each flow record and
       converts	data to	the SiLK Flow format, see the SiLK Installation
       Handbook, the sensor.conf(5) manual page, and the manual	page for the
       packing logic plug-in that rwflowpack is	using (e.g.,
       packlogic-twoway(3) is the default, packlogic-generic(3)).

       The compiled code for the packing-logic is normally loaded into
       rwflowpack as a run-time.  (The administrator may choose	to compile the
       packing logic into rwflowpack when building SiLK.  See the SiLK
       Installation Handbook for details.)  The	name of	this plug-in is
       specified in the	packing-logic statement	in the silk.conf file.	A
       different location may be specified using the --packing-logic switch.

   Input Modes
       There are several ways to input data to rwflowpack, and the method to
       use is determined by the	--input-mode switch, with "stream" being the
       default when the	switch is not provided.

       "stream"
	   In "stream" input-mode, rwflowpack processes	the probe statements
	   in the sensor.conf(5) file which specify how	rwflowpack is to
	   capture data	from one or more sources.  The data is assigned	to a
	   sensor based	on the probe-sensor mapping in the sensor.conf file.
	   rwflowpack then categorizes the records, converts them to the SiLK
	   Flow	format,	and writes them	to files.

	   The sources of data that rwflowpack supports	are:

	   o   listening for NetFlow v5	packets	on a UDP socket

	   o   listening for IPFIX (Internet Protocol Flow Information eXport)
	       packets on a TCP	or a UDP socket

	   o   listening for NetFlow v9	packets	on a UDP socket

	   o   listening for sFlow v5 packets on a UDP socket

	   o   polling a directory for files containing	NetFlow	v5 records
	       (see the	description of the "pdufile" input-mode	for the
	       required	format of these	files)

	   o   polling a directory for files containing	IPFIX records (as
	       generated by yaf(1))

	   o   polling a directory for files containing	SiLK Flow records
	       (compare	to the "respool" input-mode)

	   Multiple sources may	be specified.

	   Processing of IPFIX,	NetFlow	v9, or sFlow is	only available when
	   SiLK	is compiled with support for libfixbuf-1.7.0 or	later.
	   Processing of sFlow records was added in SiLK 3.9.0.	 libfixbuf is
	   available from <http://tools.netsa.cert.org/fixbuf/>.

	   Configuration of "stream" input-mode	is specified in	the "Stream
	   Collection Switches (--input-mode=stream)" section below.

       "fcfiles"
	   Instead of having rwflowpack	capture	data itself, you may configure
	   rwflowpack to work in conjunction with one or more flowcap(8)
	   daemons by specifying "fcfiles" as the input-mode.

	   In this configuration, each flowcap uses the	probe statements in
	   the sensor.conf(5) file to determine	how to collect the data.
	   flowcap supports the	network-based capture methods specified	for
	   the "stream"	input-mode---i.e., flowcap does	not support directory
	   polling.  flowcap writes the	data it	captures into small files and
	   includes the	probe name in each file's header.

	   Typically, the flowcap processes run	on separate machines near the
	   router or flow meter	that is	generating the records.	 The
	   rwsender(8) and rwreceiver(8) daemons are often used	to move	the
	   files from the flowcap machines to the rwflowpack machine.

	   rwflowpack polls a directory	for the	files created by flowcap.
	   Once	it finds a file, it assigns those records a sensor based on
	   the probe-sensor mapping in the sensor.conf file, it	categorizes
	   the records,	and it writes the records to one or more output	files.

	   Since rwflowpack does not capture data in "fcfiles" input-mode,
	   rwflowpack does not use the probe statements	in the sensor.conf
	   file, and the statements do not need	to be present.

	   The switches	that rwflowpack	uses in	"fcfiles" input-mode are given
	   below ("Flowcap Files Collection Switches (--input-mode=fcfiles)").

       "pdufile"
	   Setting the input-mode to "pdufile" tells rwflowpack	to read	a
	   single file containing NetFlow v5 PDU records and then exit.
	   rwflowpack does not become a	daemon in this input-mode; instead it
	   remains in the foreground, processes	the NetFlow file, and exits.

	   The file must be in the format created by NetFlow Collector:	The
	   file's size must be an integer multiple of 1464, where each 1464
	   byte	chunk contains a 24-byte NetFlow v5 header and space for
	   thirty 48-byte NetFlow records.  The	number of valid	records	per
	   chunk is specified in the chunk's header.  (This is also the	format
	   that	rwflowpack requires in "stream"	input-mode when	it is polling
	   a directory for NetFlow v5 files.)

	   To convert single PDU file to a stream of SiLK Flow records,
	   consider using rwpdu2silk(1).

	   In "pdufile"	input-mode, the	sensor.conf file must define a sensor
	   that	maps to	a probe	that uses the read-from-file statement.
	   However, the	argument to that statement is ignored, and the
	   argument is typically set to	/dev/null.  The	NetFlow	v5 file's
	   location must be specified with the --netflow-file switch.  If
	   sensor.conf contains	multiple sensor	blocks,	the --sensor-name
	   switch is required to tell rwflowpack which sensor to use.

	   See the "PDU	File Collection	Switches (--input-mode=pdufile)"
	   section below for additional	configuration information.

       "respool"
	   Sometimes it	is desirable to	pull existing SiLK Flow	records	from
	   one data repository and use them to create a	"mini" data repository
	   (for	example, for testing).	The "respool" input-mode is one	way to
	   accomplish this.

	   In this configuration, rwflowpack polls a directory for SiLK	flow
	   files and it	uses the existing class/type pair and sensor values to
	   determine where to store the	flow record.  That is, rwflowpack puts
	   the data into appropriate hourly file, but it does not change any
	   other settings on the flow records.

	   To contrast "respool" input-mode with rwflowpack processing SiLK
	   Flow	files in "stream" input-mode: In "respool" input-mode, the
	   existing class, type, and sensor values are used to determine each
	   record's destination.  In the latter, rwflowpack treats the records
	   as it would any other newly arrived data, assigning the data	to a
	   sensor and re-categorizing the records to assign a class/type pair
	   to them.

	   Since no categorization occurs in "respool" input-mode, the
	   --sensor-configuration and --packing-logic switches are not
	   required and	not allowed, and their presence	causes rwflowpack to
	   exit	with an	error code.

   Output Modes
       As mentioned above, after rwflowpack categorizes	a flow record (that
       is, determines the sensor, class/type, and starting hour	for the
       record),	it uses	those values to	generate the name of the hourly	file
       that will contain that record, and it writes the	record to that file.

       In order	for the	record in that file to be retrievable by rwfilter(1),
       the file	must be	stored in a SiLK data repository, which	is a directory
       tree of binary SiLK Flow	files.	The root of this directory tree	is
       called the SILK_DATA_ROOTDIR.  The structure of the tree	under the root
       is determined by	the path-format	setting	in the silk.conf(5) file.

       There are two ways to get the files into	the SILK_DATA_ROOTDIR; which
       method is used is determined by rwflowpack's --output-mode switch.
       This switch supports the	following values:

       "local-storage"
	   In "local-storage" output-mode, rwflowpack creates the hourly SiLK
	   Flow	files directly in the data repository, and it writes the
	   records into	these files.  rwflowpack uses "local-storage" output-
	   mode	when the --output-mode switch is not provided.

       "incremental-files"
	   When	the output-mode	is "incremental-files",	rwflowpack does	not
	   create hourly data files directly.  Instead,	rwflowpack creates
	   smaller files (called incremental files), and rwflowpack relies on
	   the rwflowappend(8) daemon to combine the incremental files into
	   hourly files	in the final data repository.

	   In "incremental-files" output-mode, rwflowpack uses a single
	   destination directory whose location	is specified by	the
	   --incremental-directory switch.  In this directory, rwflowpack
	   creates a zero-byte place holder file and a corresponding working
	   file.  The name of the place	holder file has	a unique, random
	   suffix, and the name	of the working file is a dot (".") followed by
	   the name of the place holder	file.

	   rwflowpack first writes records to the working files.  Periodically
	   (as determined by the value specified to --flush-timeout),
	   rwflowpack closes all the working files and moves them over the
	   place holder	files.	Once rwflowpack	has closed and moved a working
	   file, it no longer takes responsibility for it.

	   The rwflowappend process may	poll the incremental-directory itself,
	   or another process may poll that directory and pass the files to
	   rwflowappend.  If rwflowpack	and rwflowappend are on	different
	   machines, an	rwsender(8)/rwreceiver(8) pair may be used to move the
	   files between the machines.

	   Additional reasons one may want to use "incremental-files" output-
	   mode	are to support having multiple data repositories or to allow
	   additional processing of the	SiLK Flow records (such	as by the
	   Analysis Pipeline
	   (<http://tools.netsa.cert.org/analysis-pipeline/>)).

	   Note: This output-mode was introduced in SiLK 3.6.0.	 This mode is
	   similar to the "sending" output-mode	and is meant to	replace	it.
	   In "incremental-files" output-mode, all writing occurs within the
	   incremental-directory, while	the "sending" output-mode uses two
	   directories.

       "sending"
	   This	output-mode works similarly to "incremental-files".  The
	   difference is that when rwflowpack flushes the open files, it moves
	   the files from the incremental-directory and	to the directory
	   specified by	the --sender-directory switch.	Once a file is moved
	   to the sender-directory, rwflowpack no longer takes responsibility
	   for it.

	   As of SiLK 3.6.0, the "sending" output-mode is deprecated.

OPTIONS
       Option names may	be abbreviated if the abbreviation is unique or	is an
       exact match for an option.  A parameter to an option may	be specified
       as --arg=param or --arg param, though the first form is required	for
       options that take optional parameters.

   Input and Output Mode
       As described in the "Input Modes" section above,	rwflowpack has
       multiple	ways of	getting	data, and that data may	be stored in one of
       two methods (c.f., "Output Modes").  Choosing the modes and configuring
       each mode are described in the sections below.

       --input-mode=MODE
	   Determine how rwflowpack gathers data.  The default input MODE is
	   "stream".  The available input-modes	are "stream", "fcfiles",
	   "pdufile", and "respool".

       --output-mode=MODE
	   Determine what rwflowpack does with the data	as it is packed	into
	   SiLK	binary files.  The default output MODE is "local-storage".
	   The available output-modes are "local-storage", "incremental-files"
	   and "sending".

   Stream Collection Switches (--input-mode=stream)
       In "stream" input-mode, rwflowpack uses the probe statements in the
       sensor.conf file	to capture data, and then rwflowpack categorizes the
       data.  The "stream" input-mode is the default when the --input-mode
       switch is not provided.	This input-mode	accepts	the following
       switches; the --sensor-configuration switch is required,	and all	other
       switches	are optional.

       --sensor-configuration=FILE_PATH
	   Give	the path to the	configuration file that	specifies how
	   rwflowpack is to capture data and that tells	rwflowpack whether a
	   record represents an	incoming or outgoing flow.  The	complete
	   syntax of the configuration file is described in the	sensor.conf(5)
	   manual page;	see also the SiLK Installation Handbook.

       --packing-logic=PLUGIN
	   Specify the plug-in that rwflowpack should load, where the plug-in
	   provides functions that determine into which	class and type each
	   flow	record will be categorized and the format of the files that
	   rwflowpack will write.  When	SiLK has been configured with hard-
	   coded packing logic (i.e., when --enable-packing-logic was
	   specified to	the configure script), this switch will	not be present
	   on rwflowpack.  A default value for this switch may be specified in
	   the silk.conf(5) site configuration file (see the description of
	   the --site-config-file switch).  When PLUGIN	does not contain a
	   slash ("/"),	rwflowpack attempts to find a file named PLUGIN	in the
	   directories listed in the "FILES" section.  If rwflowpack finds the
	   file, it uses that path.  If	PLUGIN contains	a slash	or if
	   rwflowpack does not find the	file, rwflowpack relies	on your
	   operating system's dlopen(3)	call to	find the file.	When the
	   SILK_PLUGIN_DEBUG environment variable is non-empty,	rwflowpack
	   prints status messages to the standard error	as it attempts to find
	   and open each of its	plug-ins.  rwflowpack exits if it does not
	   have	access to the packing logic functions.

       --sensor-name=SENSOR
	   Cause rwflowpack to ignore all probes in the	sensor configuration
	   file	except the probes for SENSOR.  Only data for SENSOR is
	   collected.  This allows a common configuration file to be used by
	   multiple rwflowpack invocations, yet	also allow each	rwflowpack
	   instance only collect data for a single sensor.  There must be a
	   sensor definition for SENSOR	in the configuration file.  When this
	   switch is not present, rwflowpack collects and packs	data for all
	   sensors.

       --polling-interval=NUMBER
	   Specify the number of seconds rwflowpack waits between scans	of the
	   directories specified by the	"poll-directory" settings of the
	   probes in the sensor.conf file.  The	default	is 15 seconds.

   Flowcap Files Collection Switches (--input-mode=fcfiles)
       As described in the "Input Modes" section above,	in "fcfiles" input-
       mode, rwflowpack	processes files	created	by the flowcap(8) daemon.  In
       "fcfiles" input-mode, the --sensor-configuration	and
       --incoming-directory switches are required.

       --sensor-configuration=FILE_PATH
	   Give	the path to the	configuration file that	rwflowpack consults to
	   determine whether a record represents an incoming or	outgoing flow.
	   The complete	syntax of the configuration file is described in the
	   sensor.conf(5) manual page; see also	the SiLK Installation
	   Handbook.

       --incoming-directory=DIR_PATH
	   Periodically	scan the directory DIR_PATH for	files that have	been
	   created by flowcap.	As rwflowpack scans DIR_PATH, it ignores a
	   file	if its name begins with	a dot (".") or if its size is 0	bytes.
	   When	a file is first	detected, rwflowpack records its size, and the
	   file	must have the same size	for two	consecutive scans before
	   rwflowpack processes	it.  After the file is successfully processed,
	   rwflowpack either moves it to the archive-directory or deletes it.
	   The interval	between	scans is set by	--polling-interval.  DIR_PATH
	   must	be a complete directory	path.

       --packing-logic=PLUGIN
	   Specify the plug-in that rwflowpack should load for the packing
	   logic.  For more detail, see	the description	above.

       --polling-interval=NUMBER
	   Specify the number of seconds rwflowpack waits between polls	of the
	   incoming-directory for new files created by flowcap.	 If not	given,
	   the default value is	15 seconds.

   PDU File Collection Switches	(--input-mode=pdufile)
       In this input-mode, rwflowpack stays in the foreground, processes a
       single file of NetFlow v5 data, and exits.  The --sensor-configuration
       and --netflow-file switches are required.  The --sensor-name switch is
       also required when the sensor.conf file contains	more than one sensor.
       The following switches are available in "pdufile" input-mode:

       --sensor-configuration=FILE_PATH
	   Give	the path to the	configuration file that	rwflowpack consults to
	   determine whether a record represents an incoming or	outgoing flow.

       --netflow-file=FILE_PATH
	   Name	the full path of the file from which rwflowpack	reads NetFlow
	   v5 PDUs.  This switch is required in	PDU File input-mode.

       --sensor-name=SENSOR
	   Cause rwflowpack to ignore all probes in the	sensor configuration
	   file	except the probes for SENSOR.  There must be a sensor
	   definition for SENSOR in the	configuration file.  This switch is
	   required in this input-mode unless the sensor.conf file only
	   defines a single sensor.

       --packing-logic=PLUGIN
	   Specify the plug-in that rwflowpack should load for the packing
	   logic.  For more detail, see	the description	of this	switch in the
	   "stream" input-mode.

   Respooling Switches (--input-mode=respool)
       When the	--input-mode="respool" switch is provided, rwflowpack polls a
       directory for SiLK Flow files, and writes the records it	finds into new
       hourly files, leaving the sensor	and class/type values unchanged	in the
       records.	 (See "Input Modes" above for additional details.)  The	first
       of the following	switches is required:

       --incoming-directory=DIR_PATH
	   Periodically	scan the directory DIR_PATH for	SiLK Flow files	to
	   process.  As	rwflowpack scans DIR_PATH, it ignores a	file if	its
	   name	begins with a dot (".")	or if its size is 0 bytes.  When a
	   file	is first detected, rwflowpack records its size,	and the	file
	   must	have the same size for two consecutive scans before rwflowpack
	   will	process	it.  After the file is successfully processed,
	   rwflowpack either moves it to the archive-directory or deletes it.
	   The interval	between	scans is set by	--polling-interval.  DIR_PATH
	   must	be a complete directory	path.

       --polling-interval=NUMBER
	   Specify the number of seconds rwflowpack waits between polls	of the
	   incoming-directory.	If not given, the default value	is 15 seconds.

   Local Storage Switches (--output-mode=local-storage)
       In "local-storage" output-mode, rwflowpack stores SiLK Flow records
       directly	in the data repository.	 This is the default when the
       --output-mode switch is not provided.  When operating in	this output-
       mode, the following switch is required:

       --root-directory=DIR_PATH
	   Name	the full path of the SILK_DATA_ROOTDIR;	that is, the directory
	   under which the files containing the	packed SiLK Flow records are
	   stored.  rwflowpack creates subdirectories below DIR_PATH based on
	   the data received and the path-format setting in the	silk.conf(5)
	   file.

   Incremental-Files Switches (--output-mode=incremental-files)
       As described in the "Output Modes" section above, the
       "incremental-files" output-mode tells rwflowpack	to write incremental-
       files.  The rwflowappend(8) daemon combines these incremental-files to
       create a	SiLK data repository.  When running in "incremental-files"
       output-mode, the	following switch must be provided:

       --incremental-directory=DIR_PATH
	   Name	the full path of the destination directory where incremental-
	   files are both created and stored to	await action by	another
	   process such	as rwflowappend	or rwsender.  It is recommended	that
	   only	a single rwflowpack process write to this directory.

   Sending Mode	Switches (--output-mode=sending)
       This output-mode	is deprecated as of SiLK 3.6.0.	 This output-mode
       works similarly to the "incremental-files" output-mode, except
       rwflowpack moves	the files to a second directory	periodically.  Both
       the following switches are required in "sending"	output-mode:

       --incremental-directory=DIR_PATH
	   Name	the full path of the working directory under which
	   incremental-files are initially created.  Periodically (as
	   determined by the --flush-timeout switch), rwflowpack closes	the
	   files in this directory and moves them to the sender-directory.  An
	   rwflowpack invocation assumes it has	full control over the files in
	   this	directory.  When rwflowpack starts, any	files in this
	   directory are moved to the sender-directory.

       --sender-directory=DIR_PATH
	   Name	the full path of the destination directory where incremental-
	   files are moved to await action by another process such as
	   rwflowappend	or rwsender.  Once rwflowpack moves files to this
	   directory, it no longer takes responsibility	for them.  The other
	   process (e.g., rwsender) is responsible for removing	files from
	   this	directory.  Multiple rwflowpack	invocations may	use a single
	   sender-directory.

   General Configuration
       The following switches are optional:

       --no-file-locking
	   Do not use advisory write locks.  Normally, rwflowpack obtains a
	   write lock on an data file prior to writing records to it.  The
	   write lock prevents two instances of	rwflowpack from	writing	to the
	   same	data file simultaneously.  However, attempting to use a	write
	   lock	on some	file systems causes rwflowpack to exit with an error,
	   and this switch may be used when writing data to these file
	   systems.

       --flush-timeout=VAL
	   Set the timeout for flushing	any in-memory records to disk to VAL
	   seconds.  If	not specified, the default is 2	minutes	(120 seconds).
	   When	the output-mode	is "local-storage", this value specifies how
	   often the files are flushed to disk to ensure that any records in
	   memory are written to disk.	When using the "incremental-files" or
	   "sending" output-mode, this value specifies how often to close and
	   move	the incremental	files.	See the	"Output	Modes" section for
	   details.

       --file-cache-size=VAL
	   Set the maximum number of data files	to have	open for writing at
	   any one time	to VAL.	 If not	specified, the default is 128 files.
	   The minimum file cache size is 4.  This switch also determines how
	   many	files rwflowpack reads from simultaneously when	using probes
	   that	poll directories for files (see	sensor.conf(5)).  The maximum
	   number of input files open at any one time is limited to one	eighth
	   of VAL (with	a minimum of 2), and the number	of directory polling
	   operations to perform simultaneously	is limited to one sixteenth of
	   VAL (minimum	is 1).

       --pack-interfaces
	   Allow one to	override the default file output formats of the	packed
	   SiLK	Flow files that	rwflowpack writes.  When this switch is
	   present, rwflowpack writes additional information into the packed
	   files: the router's SNMP input and output interfaces	and the	next-
	   hop IP address.  (When the sensor.conf file contains	an
	   "interface-values" attribute	whose value is "vlan", the input and
	   output fields contain the vlan IDs instead of SNMP interface
	   values.)  The extra data produced by	this switch is useful for
	   determining why traffic is being stored in certain files.  Note
	   that	this switch only affects newly created files.  New records are
	   always appended to an existing file in the file's current output
	   format to maintain file integrity.

       --byte-order=ENDIAN
	   Set the byte	order for newly	created	SiLK Flow files.  When
	   appending records to	an existing file, the byte order of the	file
	   is maintained.  The argument	is one of the following:

	   "native"
	       Use the byte order of the machine where rwflowpack is running.
	       This is the default.

	   "big"
	       Use network byte	order (big endian) for the flow	files.

	   "little"
	       Write the flow files in little endian format.

       --compression-method=COMP_METHOD
	   Specify the compression library to use when creating	new files.
	   When	this switch is not given, newly	created	files are compressed
	   using the default chosen when SiLK was compiled.  When appending
	   records to an existing file,	the compression	method of the file is
	   maintained.	The valid values for COMP_METHOD are determined	by
	   which external libraries were found when SiLK was compiled.	To see
	   the available compression methods and the default method, use the
	   --help or --version switch. SiLK can	support	the following
	   COMP_METHOD values when the required	libraries are available.

	   none
	       Do not compress the output using	an external library.

	   zlib
	       Use the zlib(3) library for compressing the output.  Using zlib
	       produces	the smallest output files at the cost of speed.

	   lzo1x
	       Use the lzo1x algorithm from the	LZO real time compression
	       library for compression.	 This compression provides good
	       compression with	less memory and	CPU overhead.

	   snappy
	       Use the snappy library for compression, and always compress the
	       output regardless of the	destination.  This compression
	       provides	good compression with less memory and CPU overhead.
	       Since SiLK 3.13.0.

	   best
	       Use lzo1x if available, otherwise use snappy if available,
	       otherwise use zlib if available.

       --site-config-file=FILENAME
	   Read	the SiLK site configuration from the named file	FILENAME.
	   When	this switch is not provided, rwflowpack	searches for the site
	   configuration file in the locations specified in the	"FILES"
	   section.

   Disposal of Input Flow Files
       The following switches determine	how rwflowpack handles input files
       once it has processed them.  These switches have	no effect when
       rwflowpack reads	all of its data	directly from the network.  Otherwise,
       the switches affect the named --netflow-file in "pdufile" input-mode,
       the files read from the --incoming-directory in "fcfiles" and "respool"
       input-mode, and files read from probes that have	a "poll-directory"
       statement (c.f. sensor.conf(5)) in "stream" input-mode.

       --error-directory=DIR_PATH
	   Move	input files that cannot	be opened, have	an unexpected format,
	   contain an unrecognized probe name in "fcfiles" input-mode, or are
	   not successfully processed into the directory DIR_PATH.  DIR_PATH
	   must	be a complete directory	path.  If this switch is not provided,
	   problem files remain	in place and cause rwflowpack to exit.

       --archive-directory=DIR_PATH
	   Move	input files that rwflowpack processes successfully into	the
	   directory DIR_PATH.	DIR_PATH must be a complete directory path.
	   When	this switch is not provided and	the input-mode is "pdufile",
	   the original	NetFlow	source file is not modified, moved, or
	   deleted.  In	all other input-modes, no --archive-directory switch
	   causes rwflowpack to	delete each input file after successfully
	   processing it.  When	the --flat-archive switch is also provided,
	   incoming files are moved into the top of DIR_PATH; when
	   --flat-archive is not given,	each file is moved to a	subdirectory
	   based on the	current	UTC time: DIR_PATH/YEAR/MONTH/DAY/HOUR/.
	   Removing files from the archive-directory is	not the	job of
	   rwflowpack; the system administrator	should implement a separate
	   process to clean this directory.  This switch is required when the
	   --post-archive-command switch is present.

       --flat-archive
	   When	archiving input	files via the --archive-directory switch, move
	   the files into the top of the archive-directory, not	into
	   subdirectories of the archive-directory.  This switch has no	effect
	   if --archive-directory is not also specified.  This switch may be
	   used	to allow another process to watch for new files	appearing in
	   the archive-directory.

       --post-archive-command=COMMAND
	   Run COMMAND on each input file after	rwflowpack has successfully
	   processed the file and moved	the file into the archive-directory.
	   Each	occurrence of the string %s in COMMAND is replaced with	the
	   full	path to	the input file in the archive-directory, and each
	   occurrence of "%%" is replaced with "%".  If	any other character
	   follows "%",	rwflowpack exits with an error.	 When using this
	   feature, the	--archive-directory switch must	be specified.  See
	   also	the rwpollexec(8) daemon.

   Logging and Daemon Configuration
       One of the following mutually-exclusive switches	is required:

       --log-destination=DESTINATION
	   Specify the destination where logging messages are written.	When
	   DESTINATION begins with a slash "/",	it is treated as a file	system
	   path	and all	log messages are written to that file; there is	no log
	   rotation.  When DESTINATION does not	begin with "/",	it must	be one
	   of the following strings:

	   "none"
	       Messages	are not	written	anywhere.

	   "stdout"
	       Messages	are written to the standard output.

	   "stderr"
	       Messages	are written to the standard error.

	   "syslog"
	       Messages	are written using the syslog(3)	facility.

	   "both"
	       Messages	are written to the syslog facility and to the standard
	       error (this option is not available on all platforms).

       --log-directory=DIR_PATH
	   Use DIR_PATH	as the directory where the log files are written.
	   DIR_PATH must be a complete directory path.	The log	files have the
	   form

	    DIR_PATH/LOG_BASENAME-YYYYMMDD.log

	   where YYYYMMDD is the current date and LOG_BASENAME is the
	   application name or the value passed	to the --log-basename switch
	   when	provided.  The log files are rotated: At midnight local	time,
	   a new log is	opened,	the previous file is closed, and the command
	   specified by	--log-post-rotate is invoked on	the previous day's log
	   file.  (Old log files are not removed by rwflowpack;	the
	   administrator should	use another tool to remove them.)  When	this
	   switch is provided, a process-ID file (PID) is also written in this
	   directory unless the	--pidfile switch is provided.

       --log-pathname=FILE_PATH
	   Use FILE_PATH as the	complete path to the log file.	The log	file
	   is not rotated.

       The following set of switches is	optional:

       --log-level=LEVEL
	   Set the severity of messages	that are logged.  The levels from most
	   severe to least are:	"emerg", "alert", "crit", "err", "warning",
	   "notice", "info", "debug".  The default is "info".

       --log-sysfacility=NUMBER
	   Set the facility that syslog(3) uses	for logging messages.  This
	   switch takes	a number as an argument.  The default is a value that
	   corresponds to "LOG_USER" on	the system where rwflowpack is
	   running.  This switch produces an error unless
	   --log-destination=syslog is specified.

       --log-basename=LOG_BASENAME
	   Use LOG_BASENAME in place of	the application	name in	the name of
	   log files in	the log	directory.  See	the description	of the
	   --log-directory switch.  This switch	does not affect	the name of
	   the process-ID file.

       --log-post-rotate=COMMAND
	   Run COMMAND on the previous day's log file after log	rotation.
	   When	this switch is not specified, the previous day's log file is
	   compressed with gzip(1).  When the switch is	specified and COMMAND
	   is the empty	string,	no action is taken on the log file.  Each
	   occurrence of the string %s in COMMAND is replaced with the full
	   path	to the log file, and each occurrence of	"%%" is	replaced with
	   "%".	 If any	other character	follows	"%", rwflowpack	exits with an
	   error.  Specifying this switch without also using --log-directory
	   is an error.

       --pidfile=FILE_PATH
	   Set the complete path to the	file in	which rwflowpack writes	its
	   process ID (PID) when it is running as a daemon.  No	PID file is
	   written when	--no-daemon is given.  When this switch	is not
	   present, no PID file	is written unless the --log-directory switch
	   is specified, in which case the PID is written to
	   LOGPATH/rwflowpack.pid.

       --no-chdir
	   Do not change directory to the root directory.  When	rwflowpack
	   becomes a daemon process, it	changes	its current directory to the
	   root	directory so as	to avoid potentially running on	a mounted file
	   system.  Specifying --no-chdir prevents this	behavior, which	may be
	   useful during debugging.  The application does not change its
	   directory when --no-daemon is given.

       --no-daemon
	   Force rwflowpack to run in the foreground---it does not become a
	   daemon process.  This may be	useful during debugging.

   Help	Options
       --verify-sensor-config
       --verify-sensor-config=VERBOSE
	   Verify that the syntax of the sensor	configuration file is correct
	   and then exit rwflowpack.  If the file is incorrect or if it	does
	   not define any sensors, an error message is printed and rwflowpack
	   exits abnormally.  If the file is correct and no argument is
	   provided to the --verify-sensor-config switch, rwflowpack simply
	   exits with status 0.	 If an argument	(other than the	empty string
	   and 0) is provided to the switch, the names of the probes and
	   sensors found in the	sensor configuration file are printed to the
	   standard output, and	then rwflowpack	exits.

       --help
	   Print the available options and exit.

       --version
	   Print the version number and	information about how SiLK was
	   configured, then exit the application.

ENVIRONMENT
       SILK_IPFIX_PRINT_TEMPLATES
	   When	set to 1, rwflowpack writes messages to	the log	file
	   describing each IPFIX and NetFlow v9	template it receives.  This is
	   equivalent to adding	"show-templates" to the	"log-flags" setting
	   for each probe in the sensor.conf file.  See	the sensor.conf(5)
	   manual page for the format of these messages.  Since	SiLK 3.8.2.

       SILK_LIBFIXBUF_SUPPRESS_WARNINGS
	   When	set to 1, rwflowpack disables all warning messages generated
	   by libfixbuf.  These	warning	messages include out-of-sequence
	   packets, data records not having a corresponding template, record
	   count discrepancies,	and issues decoding list elements.  Since SiLK
	   3.10.0.

       SILK_CONFIG_FILE
	   This	environment variable is	used as	the value for the
	   --site-config-file when that	switch is not provided.

       SILK_DATA_ROOTDIR
	   This	environment variable specifies the root	directory of data
	   repository.	When the output-mode is	"sending", rwflowpack may use
	   this	environment variable when searching for	the SiLK site
	   configuration file.	See the	"FILES"	section	for details.

       SILK_PATH
	   This	environment variable gives the root of the install tree.  When
	   searching for configuration files, rwflowpack may use this
	   environment variable.  See the "FILES" section for details.

       SILK_PLUGIN_DEBUG
	   When	set to 1, rwflowpack print status messages to the standard
	   error as it tries to	open the packing logic plug-in.

FILES
       sensor.conf
	   The location	of this	file must be specified by the
	   --sensor-configuration switch.  This	file specifies probe blocks
	   that	tell rwflowpack	how to capture data when the --input-mode is
	   "stream".  The file also contains sensor blocks that	map probes to
	   sensors and that the	packing-logic uses to determine	the category
	   of each flow	record.	 The syntax of this file is described in the
	   sensor.conf(5) manual page.

       ${SILK_CONFIG_FILE}
       ROOT_DIRECTORY/silk.conf
       ${SILK_PATH}/share/silk/silk.conf
       ${SILK_PATH}/share/silk.conf
       /usr/local/share/silk/silk.conf
       /usr/local/share/silk.conf
	   Possible locations for the SiLK site	configuration file which are
	   checked when	the --site-config-file switch is not provided.	When
	   rwflowpack is running in "local-storage" output-mode,
	   ROOT_DIRECTORY/ is the directory specified to the --root-directory
	   switch.  When the output-mode is "sending", ROOT_DIRECTORY/ is
	   either the value specified in the SILK_DATA_ROOTDIR environment
	   variable or the default data	repository directory compiled into
	   rwflowpack (/data).

       ${SILK_PATH}/lib64/silk/
       ${SILK_PATH}/lib64/
       ${SILK_PATH}/lib/silk/
       ${SILK_PATH}/lib/
       /usr/local/lib64/silk/
       /usr/local/lib64/
       /usr/local/lib/silk/
       /usr/local/lib/
	   Directories that rwflowpack checks when attempting to load the
	   packing-logic plug-in.

SEE ALSO
       SiLK Installation Handbook, sensor.conf(5), silk.conf(5),
       packlogic-twoway(3), packlogic-generic(3), flowcap(8), rwfilter(1),
       rwflowappend(8),	rwreceiver(8), rwsender(8), rwpollexec(8),
       rwpdu2silk(1), rwpackchecker(8),	silk(7), gzip(1), yaf(1), dlopen(3),
       zlib(3),	syslog(3)

NOTES
       As SiLK 3.6.0, the "incremental-files" output-mode should be used in
       place of	the "sending" output-mode that existed in prior	releases of
       rwflowpack.  See	"Output	Modes" for details.

       For administrators that use the "sending" output-mode in	SiLK 3.5 or
       older and upgrade to SiLK 3.6 or	later: Any incremental files that the
       older version of	rwflowpack leaves in the incremental-directory will
       not be moved to the sender-directory by the new version of rwflowpack.
       Those files will	need to	be moved by hand.

       rwflowpack does not check the integrity of the data file	before
       appending records to it.

       When the	disk becomes full or other write errors	occur, rwflowpack may
       leave partially written records or partially written compressed blocks
       in the files it has open.  For each file	where a	partially written
       compressed block	remains	and additional compressed blocks are appended,
       the newly appended compressed blocks are	not readable by	other SiLK
       tools.  For each	file where a partially written record remains and
       additional records are appended,	other SiLK tools will read the
       unaligned data as if it were aligned and	produce	garbage	records.
       Partially writes	may also occur if rwflowpack is	suddenly killed	(e.g.,
       by "kill	-9").

       When a write error occurs, rwflowpack may leave a zero byte file	in the
       data repository.	 Such files do affect the exit status of rwfilter(1),
       though rwfilter warns about being unable	to read	the header from	the
       file.

       rwflowpack obtains an advisory write lock on the	hourly file it is
       writing,	allowing multiple rwflowpack processes to write	to the same
       hourly file.  File locking may be disabled by using the
       --no-file-locking switch.  If this switch is enabled, the administrator
       must ensure that	multiple rwflowpack processes do not attempt to	write
       to the same hourly file simultaneously.

SiLK 3.19.1			  2020-08-27			 rwflowpack(8)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | ENVIRONMENT | FILES | SEE ALSO | NOTES

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=rwflowpack&sektion=8&manpath=FreeBSD+12.2-RELEASE+and+Ports>

home | help