Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
rwaggbagtool(1)			SiLK Tool Suite		       rwaggbagtool(1)

NAME
       rwaggbagtool - Manipulate binary	Aggregate Bag files

SYNOPSIS
	 rwaggbagtool [{ --add | --subtract }]
	       [--insert-field=FIELD=VALUE [--insert-field=FIELD2=VALUE2...]]
	       [{ --remove-fields=REMOVE_LIST |	--select-fields=SELECT_LIST
		  | --to-ipset=FIELD [--ipset-record-version=VERSION]
		  | --to-bag=BAG_KEY,BAG_COUNTER }]
	       [--min-field=FIELD=VALUE	[--min-field=FIELD=VALUE...]]
	       [--max-field=FIELD=VALUE	[--max-field=FIELD=VALUE...]]
	       [--set-intersect=FIELD=FILE [--set-intersect=FIELD=FILE...]]
	       [--set-complement=FIELD=FILE [--set-complement=FIELD=FILE...]]
	       [--output-path=PATH]
	       [--note-strip] [--note-add=TEXT]	[--note-file-add=FILE]
	       [--compression-method=COMP_METHOD]
	       [--site-config-file=FILENAME]
	       [AGGBAG_FILE [AGGBAG_FILE ...]]

	 rwaggbagtool --help

	 rwaggbagtool --version

DESCRIPTION
       rwaggbagtool performs operations	on one or more Aggregate Bag files and
       creates a new Aggregate Bag file.  An Aggregate Bag is a	binary file
       that maps a key to a counter, where the key and the counter are both
       composed	of one or more fields.	rwaggbag(1) and	rwaggbagbuild(1) are
       the primary tools used to create	an Aggregate Bag file.	rwaggbagcat(1)
       prints a	binary Aggregate Bag file as text.

       rwaggbagtool processes the Aggregate Bag	files listed on	the command
       line.  When no file names are specified,	rwaggbagtool attempts to read
       an Aggregate Bag	from the standard input.  To read the standard input
       in addition to the named	files, use "-" or "stdin" as a file name.  If
       any input is not	an Aggregate Bag file, rwaggbagtool prints an error to
       the standard error and exits with an error status.

OPTIONS
       Option names may	be abbreviated if the abbreviation is unique or	is an
       exact match for an option.  A parameter to an option may	be specified
       as --arg=param or --arg param, though the first form is required	for
       options that take optional parameters.

       --add
	   Sum each of the counters for	each key for all the Aggregate Bag
	   input files.	 All the Aggregate Bag files must have the same	set of
	   key fields and counter fields.  (The	values of the keys may differ,
	   but the set of fields that comprise the key must match.)  If	no
	   other operation is specified, the add operation is the default.

       --subtract
	   Subtract from the first Aggregate Bag file all subsequent Aggregate
	   Bag files.  All the Aggregate Bag files must	have the same set of
	   key fields and counter fields.  If a	key does not appear in the
	   first Aggregate Bag file, rwaggbagtool assumes it has a value of 0.
	   If any counter subtraction results in a negative number, the	key
	   will	not appear in the resulting Aggregate Bag file.

   Field manipulation switches
       The following switches allow modification of the	fields in the
       Aggregate Bag file.  The	--remove-fields	and --select-fields switches
       are mutually exclusive, and they	reduce the number of fields in the
       Aggregate Bag input files.  The --insert-field switch is	applied	after
       --remove-fields or --select-fields, and it adds a field unless that
       field is	already	present.

       --insert-field=FIELD=VALUE
	   For each entry read from an Aggregate Bag input file, insert	a
	   field named FIELD and set its value to VALUE	if one of the
	   following is	true: (1)the input file	does not contain a field named
	   FIELD or (2)the input file does have	a field	named FIELD but	it was
	   removed by either (2a)being listed in the --remove-fields list or
	   (2b)not being listed	in the --select-fields list.  That is, this
	   switch only inserts FIELD when FIELD	is not present in the input
	   Aggregate Bag, but specifying FIELD in --remove-fields removes it
	   from	the input.  VALUE is a textual representation of the field's
	   value as described in the description of the	--fields switch	in the
	   rwaggbagbuild(1) tool.  This	switch may be repeated in order	to
	   insert multiple fields.

       --remove-fields=REMOVE_LIST
	   Remove the fields specified in REMOVE_LIST from each	of the
	   Aggregate Bag input files, where REMOVE_LIST	is a comma-separated
	   list	of field names.	 This switch may include field names that are
	   not in an Aggregate Bag input, and those field names	are ignored.
	   If a	field name is included in this list and	in a --insert-field
	   switch, the field is	given the value	specified by the
	   --insert-field switch, and the field	is included in the output
	   Aggregate Bag file.	If removing a key field	produces multiple
	   copies of a key, the	counters of those keys are merged.
	   rwaggbagbuild exits with an error when this switch is used with
	   --select-fields, --to-ipset,	or --to-bag.

       --select-fields=SELECT_LIST
	   For each Aggregate Bag input	file, only use the fields in
	   SELECT_LIST,	a comma-separated list of field	names.	Alternatively,
	   consider this switch	as removing all	fields that are	not included
	   in SELECT_LIST.  This switch	may include field names	that are not
	   in an Aggregate Bag input, and those	field names are	ignored.  When
	   a field name	is included in this list and in	a --insert-field
	   switch, the field uses its value from the input Aggregate Bag file
	   if present, and it uses the value specified in the --insert-field
	   switch otherwise.  If selecting only	some key fields	produces
	   multiple copies of a	key, the counters of those keys	are merged.
	   rwaggbagbuild exits with an error when this switch is used with
	   --remove-fields, --to-ipset,	or --to-bag.

   Filtering switches
       The following switches remove entries from the Aggregate	Bag file based
       on a field's value.  These switches are applied immediately before the
       output is generated.

       --min-field=FIELD=VALUE
	   Remove from the Aggregate Bag file all entries where	the value of
	   the field FIELD is less than	VALUE, where VALUE is a	textual
	   representation of the field's value as described in the description
	   of the --fields switch in the rwaggbagbuild(1) tool.	 This switch
	   is ignored if FIELD is not present in the Aggregate Bag.  This
	   switch may be repeated.  Since SiLK 3.17.0.

       --max-field=FIELD=VALUE
	   Remove from the Aggregate Bag file all entries where	the value of
	   the field FIELD is greater than VALUE, where	VALUE is a textual
	   representation of the field's value as described in the description
	   of the --fields switch in the rwaggbagbuild(1) tool.	 This switch
	   is ignored if FIELD is not present in the Aggregate Bag.  This
	   switch may be repeated.  Since SiLK 3.17.0.

       --set-intersect=FIELD=SET_FILE
	   Read	an IPset from the stream SET_FILE, and remove from the
	   Aggregate Bag file all entries where	the value of the field FIELD
	   is not present in the IPset.	 SET_FILE may be the name a file or
	   the string "-" or "stdin" to	read the IPset from the	standard
	   input.  This	switch is ignored if FIELD is not present in the
	   Aggregate Bag.  This	switch may be repeated.	 Since SiLK 3.17.0.

       --set-complement=FIELD=SET_FILE
	   Read	an IPset from the stream SET_FILE, and remove from the
	   Aggregate Bag file all entries where	the value of the field FIELD
	   is present in the IPset.  SET_FILE may be the name a	file or	the
	   string "-" or "stdin" to read the IPset from	the standard input.
	   This	switch is ignored if FIELD is not present in the Aggregate
	   Bag.	 This switch may be repeated.  Since SiLK 3.17.0.

   Output switches
       The following switches control the output.

       --to-ipset=FIELD
	   After operating on the Aggregate Bag	input files, create an IPset
	   file	from the resulting Aggregate Bag by treating the values	in the
	   field named FIELD as	IP addresses, inserting	the IP addresses into
	   the IPset, and writing the IPset to the standard output or the
	   destination specified by --output-path.  When this switch is	used,
	   the only legal field	name that may be used in the --insert-field
	   switch is FIELD.  rwaggbagbuild exits with an error when this
	   switch is used with --remove-fields,	--select-fields, or --to-bag.

       --ipset-record-version=VERSION
	   Specify the format of the IPset records that	are written to the
	   output when the --to-ipset switch is	used.  VERSION may be 2, 3, 4,
	   5 or	the special value 0.  When the switch is not provided, the
	   SILK_IPSET_RECORD_VERSION environment variable is checked for a
	   version.  The default version is 0.

	    0	  Use the default version for an IPv4 IPset and	an IPv6	IPset.
		  Use the --help switch	to see the versions used for your SiLK
		  installation.

	    2	  Create a file	that may hold only IPv4	addresses and is
		  readable by all versions of SiLK.

	    3	  Create a file	that may hold IPv4 or IPv6 addresses and is
		  readable by SiLK 3.0 and later.

	    4	  Create a file	that may hold IPv4 or IPv6 addresses and is
		  readable by SiLK 3.7 and later.  These files are more
		  compact that version 3 and often more	compact	than version
		  2.

	    5	  Create a file	that may hold only IPv6	addresses and is
		  readable by SiLK 3.14	and later.  When this version is
		  specified, IPsets containing only IPv4 addresses are written
		  in version 4.	 These files are usually more compact that
		  version 4.

       --to-bag=BAG_KEY,BAG_COUNTER
	   After operating on the Aggregate Bag	input files, create a (normal)
	   Bag file from the resulting Aggregate Bag.  Use the BAG_KEY field
	   as the key of the Bag, and the BAG_COUNTER field as the counter of
	   the Bag.  Write the Bag to the standard output or the destination
	   specified by	--output-path.	When this switch is used, the only
	   legal field names that may be used in the --insert-field switch are
	   BAG_KEY and BAG_COUNTER.  rwaggbagbuild exits with an error when
	   this	switch is used with --remove-fields, --select-fields, or
	   --to-ipset.

       --output-path=PATH
	   Write the resulting Aggregate Bag, IPset (see --to-ipset), or Bag
	   (see	--to-bag) to PATH, where PATH is a filename, a named pipe, the
	   keyword "stderr" to write the output	to the standard	error, or the
	   keyword "stdout" or "-" to write the	output to the standard output.
	   If PATH names an existing file, rwaggbagtool	exits with an error
	   unless the SILK_CLOBBER environment variable	is set,	in which case
	   PATH	is overwritten.	 If this switch	is not given, the output is
	   written to the standard output.  Attempting to write	the binary
	   output to a terminal	causes rwaggbagtool to exit with an error.

       --note-strip
	   Do not copy the notes (annotations) from the	input files to the
	   output file.	 Normally notes	from the input files are copied	to the
	   output.

       --note-add=TEXT
	   Add the specified TEXT to the header	of the output file as an
	   annotation.	This switch may	be repeated to add multiple
	   annotations to a file.  To view the annotations, use	the
	   rwfileinfo(1) tool.

       --note-file-add=FILENAME
	   Open	FILENAME and add the contents of that file to the header of
	   the output file as an annotation.	This switch may	be repeated to
	   add multiple	annotations.  Currently	the application	makes no
	   effort to ensure that FILENAME contains text; be careful that you
	   do not attempt to add a SiLK	data file as an	annotation.

       --compression-method=COMP_METHOD
	   Specify the compression library to use when writing output files.
	   If this switch is not given,	the value in the
	   SILK_COMPRESSION_METHOD environment variable	is used	if the value
	   names an available compression method.  When	no compression method
	   is specified, output	to the standard	output or to named pipes is
	   not compressed, and output to files is compressed using the default
	   chosen when SiLK was	compiled.  The valid values for	COMP_METHOD
	   are determined by which external libraries were found when SiLK was
	   compiled.  To see the available compression methods and the default
	   method, use the --help or --version switch.	SiLK can support the
	   following COMP_METHOD values	when the required libraries are
	   available.

	   none
	       Do not compress the output using	an external library.

	   zlib
	       Use the zlib(3) library for compressing the output, and always
	       compress	the output regardless of the destination.  Using zlib
	       produces	the smallest output files at the cost of speed.

	   lzo1x
	       Use the lzo1x algorithm from the	LZO real time compression
	       library for compression,	and always compress the	output
	       regardless of the destination.  This compression	provides good
	       compression with	less memory and	CPU overhead.

	   snappy
	       Use the snappy library for compression, and always compress the
	       output regardless of the	destination.  This compression
	       provides	good compression with less memory and CPU overhead.

	   best
	       Use lzo1x if available, otherwise use snappy if available,
	       otherwise use zlib if available.	 Only compress the output when
	       writing to a file.

   Miscellaneous switches
       --site-config-file=FILENAME
	   Read	the SiLK site configuration from the named file	FILENAME.
	   When	this switch is not provided, rwaggbagtool searches for the
	   site	configuration file in the locations specified in the "FILES"
	   section.

       --help
	   Print the available options and exit.

       --version
	   Print the version number and	information about how SiLK was
	   configured, then exit the application.

EXAMPLES
       In the following	examples, the dollar sign ("$")	represents the shell
       prompt.	The text after the dollar sign represents the command line.
       Lines have been wrapped for improved readability, and the back slash
       ("\") is	used to	indicate a wrapped line.

       To create two Aggregate Bag files, in.aggbag and	inweb.aggbag, and then
       add the counters	to create total.aggbag:

	$ rwfilter --type=in --pass=-				   \
	  | rwaggbag --key=sport,dport,proto --counter=records	   \
	       --output-path=in.aggbag
	$ rwfilter --type=inweb	--pass=-			   \
	  | rwaggbag --key=sport,dport,proto --counter=records	   \
	       --output-path=inweb.aggbag
	$ rwaggbagtool --add in.aggbag inweb.aggbag --output-path=total.aggbag
	$ rwaggbagcat total.aggbag

       To subtract inweb.aggbag	from total.aggbag:

	$ rwaggbagtool --subtract total.aggbag inweb.aggbag    \
	  | rwaggbagcat

       Create an Aggregate Bag file:

	$ rwaggbag --key=sport,dport			   \
	       --counter=sum-bytes,sum-packets data.rw	   \
	       --output-path=my-ab.aggbag

       To get just the source port and byte count from the file	my-ab.aggbag,
       you may either remove the destination port and packet count:

	$ rwaggbagtool --remove=dport,sum-packets my-ab.aggbag	\
	       --output-path=source-bytes.aggbag

       or you may select the source port and byte count:

	$ rwaggbagtool --select=sport,sum-bytes	my-ag.aggbag	\
	       --output-path=source-bytes.aggbag

       To replace the packet count in my-ab.aggbag with	zeros, remove the
       field and insert	it with	the value you want:

	$ rwaggbagtool --remove=sum-packets --insert=sum-packets=0  \
	       my-ab.aggbag --output-path=zero-packets.aggbag

       To create a regular Bag with the	source port and	byte count from
       my-ab.aggbag, use the --to-bag switch:

	$ rwaggbagtool --to-bag=sport,sum-bytes	my-ab.aggbag  \
	       --output-path=sport-byte.bag

       The --to-ipset switch works similarly:

	$ rwaggbag --key=sipv6,dipv6 --counter=records data-v6.rw  \
	       --output-path=ips.aggbag
	$ rwaggbagtool --to-ipset=dipv6	--output-path=dip.set

ENVIRONMENT
       SILK_IPSET_RECORD_VERSION
	   This	environment variable is	used as	the value for the
	   --ipset-record-version when that switch is not provided.

       SILK_CLOBBER
	   The SiLK tools normally refuse to overwrite existing	files.
	   Setting SILK_CLOBBER	to a non-empty value removes this restriction.

       SILK_COMPRESSION_METHOD
	   This	environment variable is	used as	the value for
	   --compression-method	when that switch is not	provided.

       SILK_CONFIG_FILE
	   This	environment variable is	used as	the value for the
	   --site-config-file when that	switch is not provided.

       SILK_DATA_ROOTDIR
	   This	environment variable specifies the root	directory of data
	   repository.	As described in	the "FILES" section, rwaggbagtool may
	   use this environment	variable when searching	for the	SiLK site
	   configuration file.

       SILK_PATH
	   This	environment variable gives the root of the install tree.  When
	   searching for configuration files, rwaggbagtool may use this
	   environment variable.  See the "FILES" section for details.

FILES
       ${SILK_CONFIG_FILE}
       ${SILK_DATA_ROOTDIR}/silk.conf
       /data/silk.conf
       ${SILK_PATH}/share/silk/silk.conf
       ${SILK_PATH}/share/silk.conf
       /usr/local/share/silk/silk.conf
       /usr/local/share/silk.conf
	   Possible locations for the SiLK site	configuration file which are
	   checked when	the --site-config-file switch is not provided.

SEE ALSO
       rwaggbag(1), rwaggbagbuild(1), rwaggbagcat(1), rwfilter(1),
       rwfileinfo(1), silk(7), zlib(3)

SiLK 3.19.1			  2021-09-21		       rwaggbagtool(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | EXAMPLES | ENVIRONMENT | FILES | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=rwaggbagtool&sektion=1&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help