Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
SUPER_MEDIATOR(1)		ipfix mediator		     SUPER_MEDIATOR(1)

NAME
       super_mediator -	IPFIX Mediator

SYNOPSIS
	    super_mediator   [--config CONFIG_FILE_NAME]
			     [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER]
			     [--ipfix-port PORT] [--export-port	PORT]
			     [--ipfix-input TRANSPORT_PROTOCOL]
			     [--output-mode TRANSPORT_PROTOCOL]
			     [--watch POLL_TIME]
			     [--move PROCESSED_INPUT_DIRECTORY]	[--lock]
			     [--rotate ROTATE_SECONDS]
			     [--no-stats] [--dns-dedup]
			     [--groups SPREAD_GROUPS]
			     [--udp-temp-timeout TIMEOUT_SECONDS]
			     [--log LOG_SPECIFIER]
			     [--log-dir	LOG_PATH]
			     [--daemonize] [--pidfile PIDFILE_NAME]
			     [--fields FIELD_LIST]
			     [--print-headers]
			     [--sleep MICROSECONDS]
			     [--verbose] [--quiet] [--version]
			     [--become-user UNPRIVILEGED_USER]
			     [--become-group UNPRIVILEGED_GROUP]
			     [--metadata-export]

DESCRIPTION
       super_mediator is an IPFIX Mediator to be used as a manifold for	yaf(1)
       IPFIX export data.  It has multiple means of reading and	writing	IPFIX.
       A limited number	of options can be specified on the command line.  For
       advanced	configuration with multiple collectors,	multiple exporters,
       and filtering, the configuration	file super_mediator.conf(1) should be
       used.  Command line arguments for collection will override collectors
       defined in the configuration file. Command line arguments for exporters
       will be ignored if a configuration file is provided.

       super_mediator can read from an IPFIX file, watch for files in a	given
       directory, collect IPFIX	from yaf(1) via	TCP or UDP on a	given host and
       port, or	subscribe to a series of Spread	Groups.	 It can	have multiple
       incoming	sources	and export to multiple collectors.  It can export to a
       single or rotating IPFIX, CSV, or JSON files, via UDP or	TCP, and to
       multiple	Spread Groups.

       super_mediator can filter on import and/or export.  Filters can be
       defined in the configuration file.

       super_mediator is capable of collecting all of YAF Deep Packet
       Inspection data (see yafdpi(1)).	For advanced configuration of the DPI
       to CSV output, see super_mediator.conf(1).  For DNS, the	super_mediator
       can perform de-duplication on the incoming DNS resource records.	 It
       will cache a key	containing the resource	record name (rrname), resource
       record type (rrtype), and the resource record value (rrval).  When a
       resource	record is seen matching	this key, the super_mediator will
       increment the hit count and the last_seen timestamp associated with the
       record.	If it is a new record, the record will be exported. Each
       record will be exported to the appropriate exporter.  The text format
       is:

	   first_seen |	rrtype | rrname	| rrval

       (see below for IPFIX template)

       Records will be periodically flushed.  The default is to	flush a	record
       from the	buffer once it has reached the desired hit count of 500	or a
       new record has not been seen in 5 minutes.  These default behaviors can
       be modified in the configuration	file.  Similarly, a file can be
       written upon flush time.	 If this is the	case, the text format would
       be:

	   first_seen |	last_seen | rrtype | rrname | hitcount | rrval

       There is	also the option	to BASE	64 Encode all DNS domain names in a
       text file.  This	option can be specified	in the configuration file.

OPTIONS
   Configuration File Option
       When possible, the configuration	file should be used when running the
       super_mediator.	However, some command line options were	given as a
       convenience for quick analysis.

       --config	CONFIGURATION_FILE
	   CONFIGURATION_FILE is the configuration file	following the
	   guidelines given in the super_mediator.conf man page.  If this
	   option is given, it should be the only command line option given.

   Input Options
       These options control where super_mediator will take its	input from.
       super_mediator can read packets from a file, directory, live on a TCP
       or UDP port, or by subscribing to Spread	group(s) through the Spread
       daemon.	By default, if no input	options	are given, super_mediator
       reads an	IPFIX file on standard input.

       --in INPUT_SPECIFIER
	   INPUT_SPECIFIER is an input specifier.  If --ipfix-input is given
	   and set to TCP or UDP, INPUT_SPECIFIER is the hostname or IP
	   Address of the host to listen on.  If --ipfix-input is SPREAD,
	   INPUT_SPECIFIER is the daemon name of the Spread daemon to connect
	   to.	If the Spread daemon is	running	on remote host,	the remote
	   host	or IP should be	given in the form daemon_name@hostname.	 If
	   --watch is given, INPUT_SPECIFIER should be a file glob pattern,
	   which must be escaped or quoted to prevent shell expansion.	Files
	   that	match this pattern will	be processed by	the super_mediator.
	   If no other options are given, the super_mediator assumes
	   INPUT_SPECIFIER is an IPFIX file.

       --ipfix-port PORT
	   If <--ipfix-input> is present, export flows to TCP or UDP on	port
	   port.  If not present, the default port 18000 is used.

       --ipfix-input TRANSPORT_PROTOCOL
	   If present, causes super_mediator to	operate	as an IPFIX collector,
	   listening for connections via the sepecified	protocol
	   TRANSPORT_PROTOCOL from a yaf(1) exporter named in the
	   INPUT_SPECIFIER.  Valid TRANSPORT_PROTOCOL values are tcp, udp, and
	   spread;  spread is only available if	super_mediator was built with
	   Spread support.  UDP	is not recommended, as it is not a reliable
	   transport protocol, and cannot guarantee delivery of	messages.  If
	   spread is specified,	--groups must also be present.

       --watch POLL_TIME
	   If present, process files that match	the pattern given to --in
	   every POLL_TIME seconds.  super_mediator will run forever waiting
	   for files that match	the pattern.  --watch should be	used with
	   --move.  If --move is not present, files will be deleted after they
	   have	been processed.

       --move PROCESSED_INPUT_DIRECTORY
	   If present, input files will	be moved to PROCESSED_INPUT_DIRECTORY
	   after they have been	successfully processed.	 If this is not
	   present with	--watch, files will be deleted after they are
	   processed.

       --lock
	   If present, super_mediator will not read files that are locked,
	   which means they have the extension ".lock" appended	to the end of
	   the filename.  This can be used if super_mediator is	reading	from a
	   yaf(1) export directory and yaf(1) is run with --lock.  This	will
	   prevent super_mediator from removing	the files out from under
	   yaf(1).  This does not lock files that the super_mediator is
	   writing to.	Use the	super_mediator configuration file to enable
	   locking of output files.

       --groups	SPREAD_GROUPS
	   If --ipfix-input is present and set to spread, use --group to
	   specify the the Spread group	name(s)	to subscribe to.  It is
	   possible to list more than one group	name in	a comma-seperated
	   list.  See the Spread Documentation,	www.spread.org,	for more
	   details on Spread.

   Output Options
       These options control where super_mediator will send its	output.
       super_mediator can write	flows to an IPFIX file,	text file, or to an
       IPFIX collector over TCP, UDP, or Spread.  By default, if no options
       are given, yaf(1) writes	IPFIX to standard out.

       --out OUTPUT_SPECIFIER
	   OUTPUT_SPECIFIER is an output specifier.  If	--output-mode is
	   present, and	set to TCP or UDP, the OUTPUT_SPECIFIER	specifies the
	   hostname or IP address of the collector to which the	flows will be
	   exported.  Otherwise, if --output-mode is set to SPREAD, the
	   OUTPUT_SPECIFIER should be the Spread daemon	name.  If the Spread
	   daemon name is running on a remote host, it should be in the	form
	   daemon_name@hostname.  If --output-mode is set to TEXT,
	   OUTPUT_SPECIFIER is a filename in which the flows will be written
	   in pipe-delimited format.  Otherwise, OUTPUT_SPECIFIER is a
	   filename in which flows will	be written in IPFIX Format.  The
	   string - may	be used	to write to standard output (the default).  If
	   <--rotate> is present, OUTPUT_SPECIFIER is the prefix name of each
	   output file to write	to.  super_mediator must be able to make an
	   initial connection to the OUTPUT_SPECIFIER for super_mediator to
	   start.  If the connection is	lost after the initial connection,
	   super_mediator will immediately retry the connection	after
	   reporting a warning message to the log.  If the retry is
	   unsuccessful, super_mediator	will retry the connection every	15
	   seconds until the connection	is successful.	Flows will be lost
	   while the connection	is down.

       --export-port PORT
	   If --output-mode is present and set to TCP or UDP, export flows to
	   port	PORT.  If not present, the default port	18001 will be used.

       --output-mode TRANSPORT_PROTOCOL
	   If present, causes super_mediator to	operate	as an IPFIX or TEXT
	   Exporter, exporting via the specified protocol TRANSPORT_PROTOCOL
	   to a	collector (e.g rwflowpack, flowcap) named in the
	   OUTPUT_SPECIFIER.  Valid TRANSPORT_PROTOCOL values are tcp, udp,
	   text, and json; spread is only available through the	configuration
	   file.  UDP is not recommended, as it	is not a reliable transport
	   protocol, and cannot	guarantee delivery of messages.

       --udp-temp-timeout TIMEOUT_SECS
	   Set UDP template timeout in seconds if --ipfix is set to udp.  As
	   per RFC 5101	recommendations, super_mediator	will attempt to	export
	   templates three times within	TEMPLATE_TIMEOUT.  The default
	   template timeout period is 600 seconds (10 minutes).

       --no-stats
	   If present, super_mediator will not forward yaf(1) process
	   statistics records
	    or log statistics.	It is possible to configure certain exporters
	   to process stats while others ignore	stats messages.	 This must be
	   done	with through the super_mediator	configuration file.

       --sleep MICROSECONDS
	   If present, super_mediator will sleep for MICROSECONDS between each
	   call	to fBufAppend, which appends the IPFIX messages	to the output
	   source.  This is useful if super_mediator is	reading	an IPFIX file
	   and transmitting IPFIX over UDP.  super_mediator may	send the
	   messages too	quickly	for the	IPFIX Collector	to receive them
	   (possibly dropping messages.)  This option is only available	with
	   one collector and one exporter when executing super_mediator	from
	   the command line.

       --fields	FIELD_LIST
	   If present and --output-mode=TEXT is	also present, write only the
	   fields given	in FIELD_LIST. FIELD_LIST is a list of integers
	   corresponding to flow fields, separated by a	comma.	The list of
	   acceptable fields are listed	in super_mediator.conf(1) under
	   FIELDS.  Only the integer representation of the field is accepted.
	   The below example will print:
	   stime|etime|sip|dip|sport|dport|protocol|applabel for each flow to
	   the file given to --out.

	   --fields 18,19,0,1,4,5,6,7
       --metadata-export
	   If present, super_mediator will include information element and
	   template metadata in	output.	 It is possible	to configure certain
	   exporters to	include	this metadata while others do not via the
	   super_mediator configuration	file.

       --print-headers
	   If present for TEXT Exporters, the super_mediator will write	a
	   header for delimited	flow data.  If files rotate, it	will write one
	   header at the top of	each flow data file.  Ignored for custom field
	   lists.

   Logging and Daemon Configuration
       --log LOG_SPECIFIER
	   Specifies the destination for log messages.	LOG_FILE can be	a
	   syslog(3) facility name, the	special	value stderr for standard
	   error, or the absolute path to a file for file logging.  The
	   default log specifier is stderr.  The log level can be specified by
	   the LOGLEVEL	keyword	in the Super_mediator configuration file or by
	   using one of	the following two options.  Default level is WARNING.

       --log-dir LOG_PATH
	   If present, super_mediator will write log files to LOG_PATH.
	   LOG_PATH must be a complete directory path.	The log	files have the
	   form

	   LOG_PATH/sm-YYYYMMDD.log

	   where YYYYMMDD is the current date.	The log	files are rotated at
	   midnight local time.	 When the log files are	rotated	a new log is
	   opened, the previous	file is	closed,	and gzip(1) is invoked on the
	   previous day's log file.  (Old log files will not be	removed	by
	   super_mediator.)

       --verbose
	   If present, log all messages.  The default log level	is WARNING.
	   This	option will change the log level to DEBUG and log all yaf(1)
	   and super_mediator process statistics, along	with any IO
	   operations.

       --quiet
	   If present, turn off	logging	completely.  super_mediator will not
	   log errors.

       --daemonize
	   If present, super_mediator will become a daemon.

       --pidfile PIDFILE_NAME
	   Set the complete path to the	file in	which super_mediator writes
	   its process ID (pid)	when running as	a daemon.  --pid-file is
	   ignored if --daemon is not present.

   Privilege Options
       These options are used to cause super_mediator to drop privileges when
       running as root for live	capture	purposes.

       --become-user UNPRIVILEGED_USER
	   After opening the live capture device in --live mode, drop
	   privilege to	the named user.	Using --become-user requires
	   super_mediator to be	run  as	root or	setuid root. This option will
	   cause all files written by super_mediator to	be owned by the	user
	   UNPRIVILEGED_USER and the user's primary group; use --become-group
	   as well to change the group super_mediator runs as for output
	   purposes.

	   If running as root for live capture purposes	and --become-user is
	   not present,	super_mediator will warn that privilege	is not being
	   dropped. We highly recommend	the use	of this	option,	especially in
	   production environments, for	security purposes.

       --become-group UNPRIVILEGED_GROUP
	   --become-group can be used to change	the group from the default of
	   the user given in --become-user.  This option has no	effect if
	   given without the --become-user option as well.

   DNS Deduplication
       --dns-dedup
	   If preset, super_mediator will cache	DNS resource records and emit
	   records only	when new ones are seen or super_mediator has seen 500
	   of the same records,	by default.  DNS de-duplication	can be further
	   configured in the super_mediator configuration file.

   IPFIX Export	Templates
       The following IPFIX fields are exported by the super_mediator.  DPI
       information will	be exported in the same	format as received by yaf(1),
       with the	exception of the de-duplicated DNS records.  Any statistics
       option messages will also be formatted in the same format as they are
       received.

       FLOW_ONLY Record

       Some fields are only exported if	they are non-zero and were enabled in
       yaf.  super_mediator exports many variations of the following template.
       The following list contains all of the possible elements	that could
       exist in	the FLOW_ONLY record.  Reverse elements	are only exported if
       reversePacketTotalCount or reversePacketDeltaCount is nonzero.

	   flowStartMilliseconds, IE 152, 8 octets, unsigned

	   flowEndMilliseconds,	IE 153,	8 octets, unsigned

	   octetTotalCount, IE 85, 8 octets, unsigned

	   reverseOctetTotalCount Reverse IE 85	(PEN 29305), 8 octets, unsigned

	   packetTotalcount, IE	86, 8 octets, unsigned

	   reversePacketTotalcount, Reverse IE 86 (PEN 29305), 8 octets, unsigned

	   octetDeltaCount, IE 1, 8 octets, unsigned

	   reverseOctetDeltaCount, reverse IE 1	(PEN 29305), 8 octets, unsigned

	   packetDeltaCount, IE	2, 8 octets, unsigned

	   reversePacketDeltaCount, reverse IE 2 (PEN 29305), 8	octets,	unsigned

	   sourceIPv6Address, IE 27, 16	octets,	unsigned

	   destinationIPv6Address, IE 28, 16 octets, unsigned

	   sourceIPv4Address, IE 8, 4 octets, unsigned

	   destinationIPv4Address, IE 12, 4 octets, unsigned

	   sourceTransportPort,	IE 7, 2	octets,	unsigned

	   destinationTransportPort, IE	11, 2 octets, unsigned

	   flowAttributes, CERT	(PEN 6871) IE 40, 2 octets, unsigned

	   reverseFlowAttributes, CERT (PEN 6871) IE 16424, 2 octets, unsigned

	   protocolIdentifier, IE 4, 1 octet, unsigned

	   flowEndReason, IE 136, 1 octet, unsigned

	   silkAppLabel, CERT (PEN 6871) IE 33,	2 octets, unsigned

	   reverseFlowDeltaMilliseconds, CERT (PEN 6871) IE 21,	4 octets, unsigned

	   tcpSequenceNumber, IE 184, 4	octets,	unsigned

	   reverseTcpSequenceNumber, Reverse IE	184 (PEN 29305), 4 octets, unsigned

	   initialTCPFlags, CERT (PEN 6871) IE 14, 1 octet, unsigned

	   unionTCPFlags, CERT (PEN 6871) IE 15, 1 octet, unsigned

	   reverseInitialTCPFlags, CERT	(PEN 6871) IE 16398, 1 octet, unsigned

	   reverseUnionTCPFlags, CERT (PEN 6871) IE 16399, 1 octet, unsigned

	   vlanId, IE 58, 2 octets, unsigned

	   reverseVlanId, Reverse IE 58	(PEN 29305), 2 octets, unsigned

	   ipClassOfService, IE	5, 1 octet, unsigned

	   reverseIpClassOfService, Reverse IE 15 (PEN 29305), 1 octet,	unsigned

	   mplsTopLabelStackSection, IE	70, 3 octets, unsigned

	   mplsLabelStackSection2, IE 71, 3 octets, unsigned

	   mplsLabelStackSection3, IE 72, 3 octets, unsigned

	   observationDomainId,	IE 149,	4 octets, unsigned

	       The observation domain ID of the	yaf(1) process that
	       generated the flow.

	   subTemplateMultiList, IE 293, Variable Length

       DNS De-duplicated Record

	   flowStartMilliseconds, IE 152, 8 octets, unsigned

	       The time	in milliseconds	of when	the DNS	resource record	was
	       first seen.

	   flowEndMilliseconds,	IE 153,	8 octets, unsigned

	       The time	in milliseconds	of the last record seen	by the
	       B<super_mediator>. This is only exported	if the
	       B<super_mediator> is configured to LAST_SEEN.

	   sourceIPv4Address, IE 8, 4 octets, unsigned

	       The IPv4	Address	found in a DNS A Record.

	   dnsTTL, CERT	(PEN 6871) IE 199, 4 octets, unsigned

	       The maximum TTL seen for	the aggregated DNS records.
	       This is only exported if	the B<super_mediator> is
	       configured to LAST_SEEN.

	   dnsQRType, CERT (PEN	6871) IE 175, 2	octets,	unsigned

	       The type	of Resource Record.  This corresponds
	       with the	QTYPE filed in the DNS Question	Section
	       or the TYP field	in the DNS Resource Record Section.

	   dnsHitCount,	CERT (PEN 6871)	IE 228,	2 octets, unsigned

	       The number of times the B<super_mediator> saw
	       this record in the FLUSH	TIMEOUT	period.
	       This is only exported if	the B<super_mediator> is
	       configured to LAST_SEEN.

	   dnsQName, CERT (PEN 6871) IE	179, variable length

	       A DNS Response Name.  This field	corresponds
	       with the	NAME field in the DNS Resource Record Section.

	   dnsRName, CERT (PEN 6871) IE	927, variable length

	      A	DNS Resource Record Data Element.  This	field
	      corresponds with the RDATA in the	DNS Resource
	      Record Section.  For CNAME records, this will be the
	      canonical	name. For NS Records, this will	be the
	      name server name.	 For AAAA records, this	will be	an
	      IPv6 Address, etc.

	   observationDomainName IE 300, variable length

	      This field is populated if the DEDUP_CONFIG block	was
	      configured with B<ADD_EXPORTER_NAME>
	      or DNS_DEDUP was configured with a B<VLAN_MAP> or	B<OBID_MAP>.
	      See super_mediator.conf(1) for more information on MAPS.

       Tombstone Options Template

       Super_mediator will forward and can optionally generate tombstone
       records using IPFIX Options Template Records.  These records are
       intended	to allow the analysis of the time it takes for records to be
       processed by each tool (eg.  YAF, Super Mediator, SiLK) in your
       environment.  Each tombstone record generated by	super_mediator
       consists	of four	information elements: a	user-settable ID, an ID	that
       is randomly generated for each run of super_mediator, a sequentially
       increasing ID for each record, and a subTemplateList of the time	each
       program interacted with the tombstone record.  With ideal randomness,
       the 3 IDs taken together	should uniquely	specify	a record. Tombstone
       record generation is not	on by default and can be turned	on using the
       TOMBSTONE_CONFIG	section	of the config file. Tombstone Records are only
       sent/forwarded to exporters configured to receive stats records.

       The following Information Elements will be exported:

	   tombstoneId,	CERT (PEN 6871)	IE 550,	4 octets, unsigned

	      A	sequentially increasing	identification number unique to	each tombstone
	      record in	a run of a given program.

	   exporterConfiguredId, CERT (PEN 6871) IE 551, 2 octets, unsigned

	      An identification	number for the record that is user specifiable at
	      runtime and shared across	all records in a run of	the given program.

	   exporterUniqueId, CERT (PEN 6871) IE	552, 2 octets, unsigned

	      An identification	number for the record that is randomly generated
	      at runtime and shared across all records in a run	of the given program.

	   subTemplateList, IE 292, variable length

	      A	subTemplateList	consisting of Tombstone	Access Templates (see below)
	      that specify when	each program that supports tombstone timestamping
	      interacted with the tombstone record.

       Tombstone Access	Template

       The following two Information Elements will be exported as a template
       within the subTemplateList of a Tombstone Options Template:

	   exportingProcessId, IE 161, 4 octets, unsigned

	      The identification number	of the program that interacted with the	record.
	      B<super_mediator>	has and	ID of 2.

	   observationTimeSeconds, IE 322, 4 octets, dateTimeSeconds

	      The UNIX timestamp of when the program interacted	with the record.

EXAMPLES
       To run super_mediator with the configuration file:

       "super_mediator -c /usr/local/etc/super_mediator.conf"

       To read a yaf(1)	file and write delimited text to stdout:

       "super_mediator -i yaffile.yaf -o - -m text"

       To listen for connections from yaf(1) via TCP and write to rotating
       text files:

       "super_mediator -i localhost --ipfix-port 18000 --ipfix-input TCP --out
       /tmp/mediator --rotate 120 --log	/tmp/mediator.log -v"

       To listen for connections from yaf(1) via UDP and send output to	a
       rwflowpack process running on TCP port 18001:

       "super_mediator -p 18000	--ipfix-input UDP -o localhost -m TCP
       --export-port 18001"

       To only print the time and 5-tuple for each flow	to stdout:

       "super_mediator -i /myyaffile.yaf -o - -m TEXT
       --fields=18,19,0,1,4,5,6"

KNOWN ISSUES
       super_mediator requires libfixbuf 1.7.0 or later.

       super_mediator will not create new file directories.  All output
       directories must	exist before you start super_mediator.

KNOWN ISSUES
       yaf presently encodes the ICMP type and code information	into the
       destinationTransportPort	information element for	ICMP and ICMP6 flows.
       super_mediator running in TEXT output mode writes the type in the
       sourceTransportPort field and the ICMP code in the
       destinationTransportPort	field.

AUTHORS
       Emily Sarneso and the CERT Network Situational Awareness	Group
       Engineering Team	<http://www.cert.org/netsa>.  Bug reports and feature
       requests	may be sent via	email to <netsa-help@cert.org>.

SEE ALSO
       yaf(1), Spread Documentation at www.spread.org, and the following IETF
       Internet	RFCs: Specification of the IPFIX Protocol for Exchange of IP
       Traffic Flow Information	RFC 5101, Information Model for	IP Flow
       Information Export RFC 5102, Export of Structured Data in RFC 6313.

1.6.0				   5-Nov-2021		     SUPER_MEDIATOR(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | EXAMPLES | KNOWN ISSUES | KNOWN ISSUES | AUTHORS | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=super_mediator&sektion=1&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help