Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
rwptoflow(1)			SiLK Tool Suite			  rwptoflow(1)

NAME
       rwptoflow - Generate SiLK Flow records from packet data

SYNOPSIS
	 rwptoflow [--plugin=PLUGIN [--plugin=PLUGIN ...]]
	       [--active-time=YYYY/MM/DD:hh:dd:mm:ss.uuuuuu-YYYY/MM/DD:hh:dd:mm:ss.uuuuuu]
	       [--flow-output=FLOW_PATH] [--packet-pass-output=PCKTS_PASS]
	       [--packet-reject-output=PCKTS_REJECT]
	       [--reject-all-fragments]	[--reject-nonzero-fragments]
	       [--reject-incomplete] [--set-sensorid=SCALAR]
	       [--set-inputindex=SCALAR] [--set-outputindex=SCALAR]
	       [--set-nexthopip=IP_ADDRESS] [--print-statistics]
	       [--note-add=TEXT] [--note-file-add=FILE]
	       [--compression-method=COMP_METHOD] TCPDUMP_INPUT

	 rwptoflow [--plugin=PLUGIN ...] --help

	 rwptoflow --version

DESCRIPTION
       rwptoflow attempts to generate a	SiLK Flow record for every Ethernet IP
       IPv4 packet in the pcap(3) (tcpdump(1)) capture file TCPDUMP_INPUT.
       TCPDUMP_INPUT must contain data captured	from an	Ethernet datalink.

       rwptoflow does not attempt to reassemble	fragmented packets or to
       combine multiple	packets	into a single flow record.  rwptoflow is a
       simple program that creates one SiLK Flow record	for every IPv4 packet
       in TCPDUMP_INPUT.  (For an alternate approach, consider using the
       rwp2yaf2silk(1) tool as described at the	end of this section.)

       rwptoflow will read from	its standard input if TCPDUMP_INPUT is
       specified as "stdin".  The SiLK Flow records are	written	to the
       specified flow-output file or to	the standard output.  The application
       will fail when attempting to read or write binary data from or to a
       terminal.

       Packets outside of a user-specified active-time window can be ignored.
       Additional filtering on the TCPDUMP_INPUT can be	performed by using
       tcpdump with an "expression" filter and piping tcpdump's	output into
       rwptoflow.

       In addition to generating flow records, rwptoflow can write pcap	files
       containing the packets that it used to generate each flow, and/or the
       packets that were rejected.  Note that packets falling outside the
       active-time window are ignored and are not written to the packet-
       reject-output.

       Statistics of the number	of packets read, rejected, and written can be
       printed.

       rwptoflow will reject any packet	that is	not an IPv4 Ethernet packet
       and any packet that is too short	to contain the Ethernet	and IP
       headers.	 At the	user's request,	packets	may be rejected	when

       o   they	are fragmented---either	the initial (zero-offset) fragment or
	   a subsequent	fragment

       o   they	have a non-zero	fragment offset

       o   they	are not	fragmented or they are the zero-fragment but the
	   capture file	does not contain enough	information about the packet
	   to set protocol-specific information---namely the ICMP type and
	   code, the UDP source	and destination	ports, or the TCP source and
	   destination ports and flags

       Since the input packet formats do not contain some fields normally
       found in	NetFlow	data, rwptoflow	provides a way to set those flow
       values in all packets.  For example, it is possible to set the sensor-
       id manually for a tcpdump source, so that flow data can be filtered or
       sorted by that value later.

   Alternative to rwptoflow
       As mentioned above, rwptoflow is	a simple program for processing
       Ethernet	IP IPv4	packets.  rwptoflow does not:

       o   reassemble fragmented packets

       o   support IPv6	packets

       o   combine multiple packets into a single flow record

       o   support any decoding	of packets (e.g., 802.1q)

       For these features (and others),	you should use the yaf(1) application
       (<http://tools.netsa.cert.org/yaf/>) to read the	pcap file and generate
       an IPFIX	stream,	and pipe the IPFIX stream into rwipfix2silk(1) to
       convert it to SiLK Flow records.

       The rwp2yaf2silk(1) script makes	this common usage more convenient by
       wrapping	the invocation of yaf and rwipfix2silk.	 You give rwp2yaf2silk
       a pcap file and it writes SiLK Flow records.

       By default, rwptoflow creates a flow record for every packet, fragments
       and all.	 You can almost	force yaf to create a flow record for every
       packet: When you	give yaf the --idle-timeout=0 switch, yaf creates a
       flow record for every complete packet and for each packet that it is
       able to completely reassemble from packet fragments.  Any fragmented
       packets that yaf	cannot reassemble are dropped.

OPTIONS
       Option names may	be abbreviated if the abbreviation is unique or	is an
       exact match for an option.  A parameter to an option may	be specified
       as --arg=param or --arg param, though the first form is required	for
       options that take optional parameters.

       --plugin=PLUGIN
	   Use the specified plug-in to	ignore or reject packets or to modify
	   the flow record that	is generated from the packet.  The switch may
	   be repeated to load multiple	plug-ins.  See the "PLUG-IN SUPPORT"
	   section below for details.

       --active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
       --active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]-YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
	   Ignore all packets whose time falls outside the specified range.
	   The times must be specified to at least day precision.  The start
	   time	is required; when the end-time is not present, it is treated
	   as infinite.	 The end-time will be rounded-up to instant before the
	   next	time unit; i.e., an end-time of	2006/08/31:15 is treated as
	   2006/08/31:15:59:59.999999.

       --flow-output=FLOW_PATH
	   Write the generated SiLK Flow records to the	specified file at
	   FLOW_PATH.  When this switch	is not provided, the flows are written
	   to the standard output.

       --packet-pass-output=PCKTS_PASS
	   For each generated SiLK Flow	record,	write the packet that
	   generated the flow to the pcap file specified by PCKTS_PASS.	 Use
	   "stdout" to write the packets to the	standard output.

       --packet-reject-output=PCKTS_REJECT
	   Write each packet that occurs within	the active-time	window but for
	   which a SiLK	Flow record was	not generated to the pcap file
	   specified by	PCKTS_REJECT.  Use "stdout" to write the packets to
	   the standard	output.

	   The packets that get	written	to this	file may include packets that
	   were	shorter	than that required to get the IP header, non-IPv4
	   packets, and	packets	that get treated as "reject" packets by	the
	   following switches.

       --reject-all-fragments
	   Do not generate a SiLK Flow record for the packet when the packet
	   is fragmented.  This	includes the initial (zero-offset) fragment
	   and all subsequent fragments.  If --packet-reject-output is
	   specified, the packet will be written to that file.

       --reject-nonzero-fragments
	   Do not generate a SiLK Flow record for the packet when the packet
	   is fragmented unless	this is	the initial fragment.  That is,	reject
	   all packets that have a non-zero fragmentation offset.  Normally
	   flow	records	are generated for these	packets, but the ports and TCP
	   flag	information is set to zero.  If	--packet-reject-output is
	   specified, the packet will be written to that file.

       --reject-incomplete
	   Do not generate a SiLK Flow record for the packet when the packet's
	   fragmentation-offset	is zero	yet the	packet does not	contain	enough
	   information to completely specify an	ICMP, UDP, or TCP record (that
	   is, the packet is too short to set the ICMP type and	code, the UDP
	   or TCP source or destination	port, or the TCP flags).  Normally,
	   flow	records	are generated for these	packets	but the	ports and TCP
	   flag	information is set to zero.  This switch has no	effect on
	   packets where the protocol is not 1,6, or 17.

	   This	switch does not	imply --reject-nonzero-fragments; to indicate
	   that	all generated flow records must	have valid port	and TCP	flag
	   information,	specify	--reject-nonzero-fragments
	   --reject-incomplete.

       --set-sensorid=SCALAR
	   Set the sensor ID for all flows to SCALAR.  SCALAR should be	an
	   integer value between 0 and 65534, inclusive.  When not specified,
	   the sensor ID is set	to 65535.

       --set-inputindex=SCALAR
	   Set the input SNMP index value for all flows	to SCALAR.  SCALAR
	   should be an	integer	value between 0	and 65535, inclusive.  When
	   not specified, the SNMP input is set	to 0.

       --set-outputindex=SCALAR
	   Set the output SNMP index value for all flows to SCALAR.  SCALAR
	   should be an	integer	value between 0	and 65535, inclusive.  When
	   not specified, the SNMP output is set to 0.

       --set-nexthopip=IP_ADDRESS
	   Set the next-hop IP address for all flows to	IP_ADDRESS; IP_ADDRESS
	   may be in its canonical form	or an integer.	When not specified,
	   the next-hop	IP is set to 0.0.0.0.

       --print-statistics
	   Print a summary of the packets that were processed.	This summary
	   includes

	   o   the total number	of packets read

	   o   the number that fell outside the	time-window

	   o   the number that were too	short to get the IP header

	   o   the number that were not	IPv4

	   o   the number that were discarded by a plug-in

	   o   the total number	of fragmented packets

	   o   the number of fragments where the offset	was zero

	   o   the number of zero-offset packets that were incomplete

	   o   the number of flows written to the output

       --note-add=TEXT
	   Add the specified TEXT to the header	of the output file as an
	   annotation.	This switch may	be repeated to add multiple
	   annotations to a file.  To view the annotations, use	the
	   rwfileinfo(1) tool.

       --note-file-add=FILENAME
	   Open	FILENAME and add the contents of that file to the header of
	   the output file as an annotation.	This switch may	be repeated to
	   add multiple	annotations.  Currently	the application	makes no
	   effort to ensure that FILENAME contains text; be careful that you
	   do not attempt to add a SiLK	data file as an	annotation.

       --compression-method=COMP_METHOD
	   Specify the compression library to use when writing output files.
	   If this switch is not given,	the value in the
	   SILK_COMPRESSION_METHOD environment variable	is used	if the value
	   names an available compression method.  When	no compression method
	   is specified, output	to the standard	output or to named pipes is
	   not compressed, and output to files is compressed using the default
	   chosen when SiLK was	compiled.  The valid values for	COMP_METHOD
	   are determined by which external libraries were found when SiLK was
	   compiled.  To see the available compression methods and the default
	   method, use the --help or --version switch.	SiLK can support the
	   following COMP_METHOD values	when the required libraries are
	   available.

	   none
	       Do not compress the output using	an external library.

	   zlib
	       Use the zlib(3) library for compressing the output, and always
	       compress	the output regardless of the destination.  Using zlib
	       produces	the smallest output files at the cost of speed.

	   lzo1x
	       Use the lzo1x algorithm from the	LZO real time compression
	       library for compression,	and always compress the	output
	       regardless of the destination.  This compression	provides good
	       compression with	less memory and	CPU overhead.

	   snappy
	       Use the snappy library for compression, and always compress the
	       output regardless of the	destination.  This compression
	       provides	good compression with less memory and CPU overhead.
	       Since SiLK 3.13.0.

	   best
	       Use lzo1x if available, otherwise use snappy if available,
	       otherwise use zlib if available.	 Only compress the output when
	       writing to a file.

       --help
	   Print the available options and exit.  Options that add fields can
	   be specified	before --help so that the new options appear in	the
	   output.

       --version
	   Print the version number and	information about how SiLK was
	   configured, then exit the application.

PLUG-IN	SUPPORT
       rwptoflow allows	the user to provide additional logic to	ignore or
       reject packets, or to modify the	flow record that is generated from the
       packet.	To do this, the	user creates a plug-in that gets loaded	at
       run-time	by giving rwptoflow the	--plugin switch	with the path to the
       plug-in as the parameter	to the switch.

       A plug-in is a shared object file (a.k.a. dynamic library) that is
       compiled	from C source code.  The plug-in should	have four subroutines
       defined:

       setup()
	   is called when the object is	first loaded.  This is the place to
	   initialize global variables to their	default	values.	 If the	plug-
	   in provides switches	of its own, they must be registered in this
	   subroutine.

       initialize()
	   gets	called after all options have been processed but before	any
	   packets are read from the input.  If	this subroutine	does not
	   return 0, the application will quit.

       ptoflow()
	   will	be called for every packet that	rwptoflow is able to convert
	   into	a flow record just before the flow record is written.  This
	   subroutine will not see packets that	are short or that are not
	   IPv4; it will also not see fragmented packets if
	   --reject-all-fragments is specified.

	   The "ptoflow()" function is called with two parameters:

	   o   a pointer to the	"rwRec"	object that rwptoflow created from the
	       packet.	The subroutine may modify the record as	it sees	fit.

	   o   a void pointer that the function	may cast to a pointer to the C
	       structure:

		 typedef struct	_sk_pktsrc_t {
		     /*	the source of the packets */
		     pcap_t			*pcap_src;
		     /*	the pcap header	as returned from pcap_next() */
		     const struct pcap_pkthdr	*pcap_hdr;
		     /*	the packet as returned from pcap_next()	*/
		     const u_char		*pcap_data;
		 } sk_pktsrc_t;

	       This structure gives the	user access to all the information
	       about the packet.

	   The following return	values from "ptoflow()"	determines whether
	   rwptoflow writes the	flow and the packet:

	    0	  Write	the flow record	to the flow-output and the packet to
		  the PCKTS_PASS unless	another	plug-in	instructs otherwise.

	    1	  Write	the flow record	to the flow-output and the packet to
		  the PCKTS_PASS immediately; do not call the "ptoflow()"
		  routine on any other plug-in.

	    2	  Treat	the packet as a	reject:	Do not write the flow record;
		  write	the packet to the PCKTS_REJECT immediately; do not
		  call the "ptoflow()" routine on any other plug-in.

	    3	  Ignore the packet immediately: Do not	write the flow record
		  nor the packet; do not call the "ptoflow()" routine on any
		  other	plug-in.

	   If "ptoflow()" returns any other value, the rwptoflow application
	   will	terminate with an error.

       teardown()
	   is called as	the application	exits.	The user can use this routine
	   to print results and	to free() any data structures that were	used.

       rwptoflow uses the following rules to find the plug-in: When PLUGIN
       contains	a slash	("/"), rwptoflow assumes the path to PLUGIN is
       correct.	 Otherwise, rwptoflow will attempt to find the file in
       $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these
       directories parallel to the application's directory: lib/silk,
       share/lib, and lib.  If rwptoflow does not find the file, it assumes
       the plug-in is in the current directory.	 To force rwptoflow to look in
       the current directory first, specify --plugin=./PLUGIN.	When the
       SILK_PLUGIN_DEBUG environment variable is non-empty, rwptoflow prints
       status messages to the standard error as	it tries to open each of its
       plug-ins.

EXAMPLES
       In the following	examples, the dollar sign ("$")	represents the shell
       prompt.	The text after the dollar sign represents the command line.

       Given the packet	capture	file data.pcap,	convert	it to a	SiLK flow
       file, data.rw, and copy the packets that	rwptoflow understands to the
       file good.pcap:

	$ rwptoflow data.pcap --packet-pass=good.pcap --flow-out=data.rw

       Use rwfilter(1) to partition the	SiLK Flows records, writing those
       records whose source IPs	are found in the IPset file sip.set to
       filtered.rw:

	$ rwfilter --sipset=sip.set --pass=filtered.rw	data.rw

       Use rwpmatch(1) to match	the capture file, good.pcap, against the
       filtered	SiLK file, in affect generating	a capture file which has been
       filtered	by sip.set:

	$ rwpmatch --flow-file=filtered.rw good.pcap > filtered.pcap

ENVIRONMENT
       SILK_PLUGIN_DEBUG
	   When	set to 1, rwptoflow print status messages to the standard
	   error as it tries to	open each of its plug-ins.

       SILK_CLOBBER
	   The SiLK tools normally refuse to overwrite existing	files.
	   Setting SILK_CLOBBER	to a non-empty value removes this restriction.

       SILK_COMPRESSION_METHOD
	   This	environment variable is	used as	the value for
	   --compression-method	when that switch is not	provided.  Since SiLK
	   3.13.0.

       SILK_PATH
	   This	environment variable gives the root of the install tree.  When
	   searching for a plug-in, rwptoflow may use this environment
	   variable.

SEE ALSO
       rwpmatch(1), rwpdedupe(1), rwfilter(1), rwfileinfo(1), rwp2yaf2silk(1),
       rwipfix2silk(1),	silk(7), yaf(1), tcpdump(1), pcap(3), mergecap(1),
       zlib(3)

NOTES
       SiLK supports millisecond timestamps.  When reading packets whose
       timestamps have finer precision,	the times are truncated	at the
       millisecond position.

       The mergecap(1) or rwpdedupe(1) programs	can be used to join multiple
       tcpdump capture files in	order to convert into a	single flow file.

SiLK 3.19.1			  2021-02-28			  rwptoflow(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | PLUG-IN SUPPORT | EXAMPLES | ENVIRONMENT | SEE ALSO | NOTES

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=rwptoflow&sektion=1&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help