Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
SHMUX(8)		    System Manager's Manual		      SHMUX(8)

       shmux - Shell Multiplexor

       shmux [ -h | -V ]

       shmux  [	-bBdFmpqQstv ] [ -C timeout ] [	-M max ] [ -r rcmd ] [ -S mode
       ] [ -e list ] [ -E list ] [ -a analyzer ] [ -A condition	] [ -o dir ] [
       -P timeout ] [ -T timeout ] -c command [	- | targets... ]

       shmux is	program	for executing the same command on many hosts in	paral-
       lel.  For each target, a	child process is spawned by shmux, and a shell
       on  the	target obtained	one of the supported methods: rsh, ssh,	or sh.
       The output produced by the children is received	by  shmux  and	either
       (optionally)  output  in	 turn to the user, and/or written to files for
       later processing.  The list of targets may either be specified  on  the
       command line or be read from the	standard input.

       The  default method may be set by using the -r option, and may be over-
       ridden for any target by	prefixing the target name by the method	and  a
       colon.	Two  special  methods, ssh1 and	ssh2 may be used to select the
       SSH protocol version 1 or 2, respectively.  For the sh method, it is up
       to  the specified command to contact the	target.	 In order for the exe-
       cuted command to	know  which  target  it	 is  being  invoked  for,  the
       SHMUX_TARGET environment	variable is set	by shmux.

       Before executing	the specified command, shmux will optionally ping each
       target to ensure	that it	can be reached,	and/or run a dummy  test  com-
       mand  to	 make  sure  that the target not only is alive,	but that it is
       possible	to cleanly execute a command on	it.  Both these	tests are typ-
       ically  run  with a fairly short	timeout	to quickly dismiss unavailable
       targets rather than waiting for the standard (longer) network timeout.

       After executing the specified command, shmux  will  optionally  process
       the  output  or	run  a user specified command to determine whether the
       command was successful or resulted in an	error.	Optionally,  the  com-
       mand output is suppressed for successful	targets.

       Finally	shmux will print out a summary of the results and list targets
       with errors.

       -h     Display a	brief help message.

       -V     Display the version information.

       -c command
	      Specify the command to execute on	targets.

       -C timeout
	      Specify a	timeout	for the	command	 being	executed  on  targets.
	      This  should be a	number followed	by a time unit.	 The following
	      are valid	time  units:  s(econds),  m(inutes),  h(our),  d(ays),
	      w(eeks).	 See the PROCESS MANAGEMENT section for	details	on how
	      this is handled.

       -M max Defines the maximum number of spawned processes.	While there is
	      no  real	(or  hard  coded)  limitation for this,	the system re-
	      sources are typically bounded and	this affects  shmux  in	 turn.
	      The most critical	resource is the	maximum	number of open files a
	      process may have.	 But one should	also consider the load imposed
	      on the system by the creation of a (potentially) large number of

       -r rcmd
	      Defines the default method used to run a shell on	targets.

       -S mode
	      This options defines the strategy	used by	 shmux	when  spawning
	      children	to  run	the command on targets.	 If the	mode is	set to
	      "all", then processes will be spawned as fast as possible.  When
	      set  to  "check",	 processes will	be spawned as long as commands
	      succeed.	Finally, the default mode, "one", is to	spawn only one
	      process  at  first,  and	wait for it to succeed,	at which point
	      shmux will switch	to the "check" mode.  In the "one" or  "check"
	      modes,  when  the	 command  fails	 on  a target, shmux will stop
	      spawning new processes.

       -F     When using either	the "one" or "check"  spawn  modes  (see  -S),
	      shmux will stop spawning new processes if	the command fails on a
	      target.  By default, shmux then waits for	user input  if	avail-
	      able,  but reverts to spawning processes if such input cannot be
	      obtained (because	the interactive	mode is	disabled,  either  via
	      -B or lack of a terminal).  Using	the -F option changes this be-
	      haviour to force shmux to	quit once all existing processes  have
	      terminated after such a failure.

       -e list
	      Defines which command exit codes should be considered errors and
	      reported as such.	 The list should be a comma separated list  of
	      ranges  of  the  form "x", "x-y",	"x-" or	"-y" where "x" and "y"
	      are numbers between 0 and	255.

       -E list
	      Defines which command exit codes should always be	output to  the
	      screen.	Codes  that  are considered errors (see	the -e option)
	      are always shown,	but by default,	 other	exit  codes  are  only
	      shown if the -v option is	specified.  This allows	more flexibil-

       -a analyzer
	      Defines how output should	be analyzed by shmux after the command
	      completes	 on a target.  By default, nothing is done.  Valid op-
	      tions are: lnregex, lnpcre, regex, pcre and  run.	  This	option
	      requires -o to be	used as	well.

       -A condition
	      When  the	 -a  option is used, it	is also	necessary to configure
	      the chosen analyzer.

	      For line based regular expressions (lnregex and lnpcre), the  -A
	      option  should be	specified twice:  once for the standard	output
	      and once for the standard	error output.  In both cases, the  ar-
	      gument should be the name	of a file containing a list of regular
	      expressions, one per line.  Each regular expression must be pre-
	      fixed by the character '=' or '!'.  The output of	the command is
	      analyzed line per	line by	checking whether it matches any	of the
	      specified	 regular  expressions.	 If the	first matching regular
	      expression is prefixed by	'=', then the command is considered to
	      be  successful  (so far).	 Any other result (e.g.	no match, or a
	      match prefixed by	'!') indicates an error	 condition.   The  de-
	      fault  is	to consider standard output normal, and	standard error
	      output indicative	of an error.

	      For other	regular	expressions (regex and pcre),  the  -A	option
	      should be	specified twice: once for the standard output and once
	      for the standard error output.  This option allows defining  the
	      expression to be applied to the output.  If the condition	starts
	      with the character '=' then a match indicates that  the  command
	      completed	without	error.	If the condition starts	with the char-
	      acter '!'	then a failure to match	 indicates  that  the  command
	      completed	 without error.	 The second character of the condition
	      should either be '=' if the regular expression follows or	'<' if
	      the  regular  expression should be read from the file which name
	      follows.	(The '=' characters may	be omitted.)  A	condition must
	      be  specified for	the standard output, but is optional for stan-
	      dard error output.  The default for the latter  is  to  consider
	      any output as indicative of an error.

	      For  the	run  analyzer,	the -A must be specified at least once
	      with the name of a program to run, and optionally	a second  time
	      to  specify  a  timeout  which  defaults to 15 seconds if	unset.
	      This timeout is implemented using	alarm(3) and will  not	other-
	      wise  be	enforced by shmux.  The	program	will be	called after a
	      command completes	on a target unless the command	exit  code  is
	      considered an error (see -e) with	two arguments: the target name
	      and the directory	specified with the -o option.  The  output  of
	      the  program will	always be shown	to the user.  Exit codes other
	      than 0 indicate that the command failed for the particular  tar-

       -o dir If specified, shmux will place the output	and (for normal	termi-
	      nations) the exit	code of	the executed commands in  files	 under
	      this  directory.	For each target, the files will	be named after
	      the target name so as to make it easy to	identify.   The	 files
	      must  not	already	exist, so it is	recommended that the directory
	      be empty.	 This also means that each target must be unique.  The
	      directory	will be	created	if it does not already exist.

       -p     Ping  targets  to	 verify	 they are alive	before doing anything.
	      The target names must be unique or bad things will happen.

       -P timeout
	      Defines the initial target ping  timeout	in  milliseconds,  see
	      fping(8).	 (Implies -p.)

       -t     Before executing the specified command for a target, send	a sim-
	      ple "echo" test and verify that the  output  is  correct.	  This
	      step  goes beyond	the simple ping	test as	it verifies that it is
	      possible to get a	shell on the target.  If this option is	speci-
	      fied  twice  (or	once when -T is	used), output for failed tests
	      will be displayed	to help	the user understand what went wrong.

       -T timeout
	      Defines the test timeout in seconds.  (Implies -t.)

       -m     By default the output is displayed as soon as  it	 is  received.
	      For  multi-line  outputs,	 this  will typically result in	output
	      from several targets being mixed.	 This option may  be  used  to
	      keep each	target output separate.

       -b     Show  the	bare output from the executed commands instead of pre-
	      fixing each line by the corresponding target name.

       -B     Enables batch mode which	forcefully  disables  the  interactive
	      mode, ignoring input from	the terminal.  See also	-s and -F.

       -s     Suppress the progress status line.  See also -B.

       -q     Used once, this will suppress the	output from successful targets
	      (as defined by the use of	options	options	-e, -a and -A.	If us-
	      ing  the	run analyzer, the output of all	targets	is suppressed.
	      Note that	using this option effectively implies -m in most cases
	      as  failure is ultimately	determined upon	completion of the com-
	      mand.  (Only the lnregex and lnpcre analyzers are	able to	deter-
	      mine  errors  while  the	command	 is  running.)	When specified
	      twice, this option allows	to completely suppress output from the
	      command  run  on	targets.   This	also changes the default spawn
	      stategy to "all".	 Using this option (once  or  twice)  requires
	      using -o.

       -Q     Suppress the final summary of results.

       -v     Display internal status messages.

       -D     Display internal debug messages.

       shmux  will  optionally report the exit code of the command it runs for
       each target.  For the sh	command, this will be the same as the  command
       itself,	but  this  is the exception.  For other	methods, the exit code
       reported	by shmux really	is the code returned by	the  program  used  to
       spawn command for that method.

       In particular, rsh typically only returns either	0 or 1 while different
       implementations of ssh have different behaviours	which may include  re-
       turning the actual exit code of the remotely executed command or	not.

       Children	processes spawned by shmux are made process group leaders (us-
       ing setpgid(2)) in order	to group all of	their descendants in a	single
       process	group which allows to easily send signals to all of these pro-
       cesses.	This is	especially useful for the sh method which can generate
       many processes locally, but has no effect on remote processes.

       shmux  uses  pipe(2)  to	 capture  the output generated by its children
       (even when this output is redirected to a file with -o) and  will  read
       from  these  until they are closed (by the child	as well	as its descen-
       dants, if any).	Once there is nothing left to read,  shmux  will  wait
       for  the	child and its descendants to terminate before considering that
       particular target done with.

       The timeout is implemented by using alarm(3) which (unless canceled  or
       intercepted)  will  terminate the child process upon receipt but	is not
       inherited by descendants	of that	process.  For that reason, shmux  will
       send  a	SIGALRM	signal to the child's process group when the child it-
       self terminates upon such signal.

       After the scheduled SIGALRM time, shmux will wait an additional 5  sec-
       onds  for  all  of  the processes in the	group to terminate to give the
       child a chance to handle	the timeout more gracefully  if	 desired.   It
       will  then  send	 a SIGTERM signal to the child's process group.	 After
       another 5 seconds, if any process is still alive	in the group, a	 final
       SIGKILL signal is sent to the process group.

       Note  that  only	 commands which	terminate because of an	ALRM signal or
       because of a TERM or KILL signal	after such signal has been  issued  by
       shmux  will be reported as having timed out in the final	summary	report
       produced	by shmux upon exiting.	This is	not as accurate	as  it	should

       By default, shmux offers	a minimal "interactive mode" while running: it
       reads commands from the terminal	and acts upon them accordingly.	  This
       can  be	disabled using the -B option.  The following commands are cur-
       rently recognized:

       q      Prevent shmux from spawning any more children,  wen  for	active
	      children to complete, and	then quit.

       Q      Quit  immediately	(e.g. abort), not waiting for any active chil-
	      dren to terminate.  This isn't recommended as  children  may  be
	      left  running unattended.	 A better alternative is to press con-
	      trol-C which will	be received by shmux and all running children.
	      These  will  typically  abort,  and shmux	terminate in a cleaner

       space  Initiate a pause,	this prevents shmux  from  spawning  any  more
	      children	to  do	its work.  The following three commands	may be
	      used to resume work.

       1      This is equivalent to specifying "-S more" on the	command	 line,
	      but may be used at any time.

       return This is equivalent to specifying "-S check" on the command line,
	      but may be used at any time.

       +      This is equivalent to specifying "-S all"	on the	command	 line,
	      but may be used at any time.

       F      This  toggles  the  failure mode for the "one" and "check" spawn
	      modes (see -S) between "pause" and "quit"	(see -F).

       p      Display the list of pending targets.

       r      Display the list of targets for which a children process is cur-
	      rently running.

       f      Display  the  list  of targets for which shmux failed to run the

       e      Display the list of targets for which the	command	completed with

       s      Display the list of targets for which the	command	completed suc-

       a      Display the list of all targets with their status.

       k      Allows sending a signal to a target.

       v      Toggles whether internal messages	are displayed or not.

       D      Toggles whether debug messages are displayed or not.

       shmux will use the following environment	variables if set:

	      Specifies	the maximum number of processes	 to  spawn  simultane-
	      ously.   If  the -M option is specified, it overrides this vari-

	      Specifies	the default command used to run	a  shell  on  targets.
	      If the -r	option is specified, it	overrides this variable.

	      Specify  the shell to use	for targets using the sh method.  (De-
	      fault is "/bin/sh".)

	      Specify an alternative command to	"rsh" to use for targets using
	      the rsh method.

	      Specify an alternative command to	"ssh" to use for targets using
	      the ssh, ssh1 and	 ssh2  method  (respectively).	 The  variable
	      SHMUX_SSH	 is  used for all methods, but superseded by the other
	      specific variables.

	      The variables may	be used	to specify replace  the	 default  "-xa
	      -oLogLevel=ERROR"	 options  passed  to  "ssh", "ssh1" and	"ssh2"
	      (respectively).  The variable SHMUX_SSH_OPTS  is	used  for  all
	      methods, but superseded by the other specific variables.

	      Multiple arguments may be	supplied, using	white-space characters
	      to separate them from each other.	 You can  use  single-	and/or
	      double-quotes  to	 preserve white-spaces in an argument (but not
	      backslashes).  Note that doubling	 a single-quote	inside a  sin-
	      gle-quoted string	gives you a single-quote; likewise for double-
	      quotes (though you  need	to  pay	attention   to	which	quotes
	      your  shell  is parsing and which	quotes shmux is	parsing).

	      Specifies	 the  default  spawn mode.  If the -S option is	speci-
	      fied, it overrides this variable.

	      Specifies	the default list of exit codes to  be  considered  er-
	      rors.   If  the  -e option is specified, it overrides this vari-

	      Specifies	the default list of exit codes to be  always  display.
	      If the -E	option is specified, it	overrides this variable.

       The following exit values are returned:

       0      indicates	 that  shmux  was able to complete its work.  However,
	      this does	not mean that all targets successfully	ran  the  com-
	      mand, nor	that shmux ran without any error.

       1      indicates	 shmux	was cleanly interrupted	before it could	finish
	      its work.

       2      indicates	shmux aborted upon user	request, possibly leaving some
	      children running.

       3      indicates	 shmux	encountered  a	fatal error which caused it to
	      abort, possibly leaving some children running.

       4      indicates	a syntax error or a fatal error	which caused shmux  to
	      abort before it could spawn any children.

       fping(8), sh(1),	rsh(1),	ssh(1),	regex(3), pcre(3).

       The  latest  official  release  of  shmux  is  available	 on  GitHub at

       Christophe Kalt <>

       All rsh and some	ssh implementations effectively	achieve	user authenti-
       cation  by  using  privileged  ports.   Since  there are	only 1024 such
       ports, there is an obvious limitation on	how many  connections  may  be
       open  using  these  programs.   To make matters worse such ports	may be
       held for	four minutes (by default) as per the TCP specification.	  When
       using shmux, one	can very quickly run out of privileged ports, at which
       point failures will start happening.

       All rsh and ssh implementations fail to accurately  return  the	remote
       command exit code to the	caller.	 Typically, rsh	will simply not	bother
       to return this exit code, and ssh will try its best but is  limited  by
       the  fact it also tries to return an error code for SSH related errors.
       In either case, this may	pose problems when using -e.

       Under Cygwin, child processes do	not inherit alarm(3) settings from the
       parent disabling	this simple timeout enforcement	system used by shmux.

       Some  shells  will ignore SIGALRM, others die upon its receipt (regard-
       less of any trap).  Again, this will affect the simple timeout enforce-
       ment system used	by shmux.

       For  bug	reports, create	an issue at
       sues, or	send an	email to

				  2017-01-13			      SHMUX(8)


Want to link to this manual page? Use this URL:

home | help