Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
just-man-pages/condor_submitGeneral Commands Majust-man-pages/condor_submit(1)

Name
       condor_submit Queue jobs	for execution under HTCondor

Synopsis
       condor_submit  [	-verbose ] [ -unused ] [ -name schedd_name ] [ -remote
       schedd_name ] [ -addr _ip:port_ ] [ -pool pool_name ] [	-disable  ]  [
       -password passphrase ] [	-debug ] [ -append command ...	] [ -spool ] [
       -dump filename ]	[ -interactive ] [ -dry-run ] [	 -maxjobs  _number-of-
       jobs_  ]	[ -single-cluster ] [ <submit-variable>=<value>	] [ submit de-
       scription file ]	[ -queue queue_arguments ]

Description
       condor_submit is	the program for	submitting jobs	 for  execution	 under
       HTCondor.   condor_submit requires a submit description file which con-
       tains commands to direct	the queuing of jobs.  One  submit  description
       file  may  contain specifications for the queuing of many HTCondor jobs
       at once.	A single invocation of condor_submit may  cause	 one  or  more
       clusters.  A  cluster is	a set of jobs specified	in the submit descrip-
       tion file between queue	commands  for  which  the  executable  is  not
       changed.	It is advantageous to submit multiple jobs as a	single cluster
       because:

	  * Only one copy of the checkpoint file is needed  to	represent  all
	  jobs in a cluster until they begin execution.

	  *  There  is	much  less overhead involved for HTCondor to start the
	  next job in a	cluster	than for HTCondor to start a new cluster. This
	  can make a big difference when submitting lots of short jobs.

       Multiple	 clusters  may be specified within a single submit description
       file. Each cluster must specify a single	executable.

       The job ClassAd attribute  ClusterId identifies a cluster.

       The submit description file argument is the path	and file name  of  the
       submit description file.	If this	optional argument is missing or	is the
       dash character (	- ), then the commands are taken from standard input.

       Note that submission of jobs from a Windows machine requires a  stashed
       password	 to allow HTCondor to impersonate the user submitting the job.
       To stash	a password, use	the condor_store_cred command. See the	manual
       page for	details.

       For  lengthy lines within the submit description	file, the backslash ()
       is a line continuation character. Placing the backslash at the end of a
       line  causes  the  current line's command to be continued with the next
       line of the file. Submit	description files may contain comments.	A com-
       ment is any line	beginning with a pound character ( # ).

Options
       -verbose

	  Verbose output - display the created job ClassAd

       -unused

	  As  a	 default,  causes  no warnings to be issued about user-defined
	  macros not being used	within the submit description file. The	 mean-
	  ing  reverses	(toggles) when the configuration variable  WARN_ON_UN-
	  USED_SUBMIT_FILE_MACROS is set to the	non default value of  False  .
	  Printing  the	 warnings  can help identify spelling errors of	submit
	  description file commands. The warnings are sent to stderr.

       -name schedd_name

	  Submit to the	specified condor_schedd	. Use this option to submit to
	  a  condor_schedd  other  than	the default local one.	schedd_name is
	  the value of the  Name ClassAd attribute on the  machine  where  the
	  condor_schedd	daemon runs.

       -remote schedd_name

	  Submit  to the specified condor_schedd , spooling all	required input
	  files	over the network connection.  schedd_name is the value of  the
	  Name ClassAd attribute on the	machine	where the condor_schedd	daemon
	  runs.	This option is equivalent to using both	-name and -spool .

       -addr _ip:port_

	  Submit to the	condor_schedd at the IP	address	and port given by  the
	  sinful string	argument _ip:port_ .

       -pool pool_name

	  Look	in the specified pool for the condor_schedd to submit to. This
	  option is used with -name or -remote .

       -disable

	  Disable file permission checks when submitting a job for  read  per-
	  missions on all input	files, such as those defined by	commands input
	  and transfer_input_files , as	well as	 write	permission  to	output
	  files,  such	as  a log file defined by log and output files defined
	  with output or transfer_output_files .

       -password passphrase

	  Specify a password to	the MyProxy server.

       -debug

	  Cause	debugging information to be sent to  stderr  ,	based  on  the
	  value	of the configuration variable  TOOL_DEBUG .

       -append command

	  Augment  the	commands in the	submit description file	with the given
	  command . This command will be considered to immediately precede the
	  queue	command	within the submit description file, and	come after all
	  other	previous commands. If the command specifies a  queue  command,
	  as in	the example

	   condor_submit mysubmitfile -append "queue input in A, B, C"

	  then	the  entire  -append command line option and its arguments are
	  converted to

	   condor_submit mysubmitfile -queue input in A, B, C

	  The submit description file is not modified. Multiple	 commands  are
	  specified  by	using the -append option multiple times. Each new com-
	  mand is given	in a separate -append option. Commands with spaces  in
	  them will need to be enclosed	in double quote	marks.

       -spool

	  Spool	 all  required	input files, job event log, and	proxy over the
	  connection to	the condor_schedd .  After  submission,	 modify	 local
	  copies  of  the  files without affecting your	jobs. Any output files
	  for completed	jobs need to be	retrieved with condor_transfer_data .

       -dump filename

	  Sends	all ClassAds to	the specified file, instead  of	 to  the  con-
	  dor_schedd .

       -interactive

	  Indicates that the user wants	to run an interactive shell on an exe-
	  cute machine in the pool. This is equivalent to  creating  a	submit
	  description  file  of	a vanilla universe sleep job, and then running
	  condor_ssh_to_job by hand. Without any  additional  arguments,  con-
	  dor_submit  with  the	-interactive flag creates a dummy vanilla uni-
	  verse	job that sleeps, submits it to the local scheduler, waits  for
	  the  job to run, and then launches condor_ssh_to_job to run a	shell.
	  If the user would like to run	the shell on a machine that matches  a
	  particular  requirements  expression,	the submit description file is
	  specified, and it will contain the expression. Note that all	policy
	  expressions  specified  in  the submit description file are honored,
	  but any executable or	universe commands are overwritten to be	 sleep
	  and  vanilla.	 The  job  ClassAd attribute  InteractiveJob is	set to
	  True to identify interactive jobs for	condor_startd policy usage.

       -dry-run	<file>

	  Parse	the submit description file, sending the job  ClassAd  to  the
	  file given by	<file> , but do	not submit the job(s). This permits an
	  observation of job specification, and	it facilitates	debugging  the
	  submit  description  file  contents.	if  <file> is -, the output is
	  written to  stdout .

       -maxjobs	_number-of-jobs_

	  If the total number of jobs specified	by the submit description file
	  is  more  than the integer value given by <number-of-jobs> , then no
	  jobs are submitted for execution and an error	message	is  generated.
	  A 0 or negative value	for the	<number-of-jobs> causes	no limit to be
	  imposed.

       -single-cluster

	  If the jobs specified	by the submit  description  file  causes  more
	  than a single	cluster	value to be assigned, then no jobs are submit-
	  ted for execution and	an error message is generated.

       <submit-variable>=<value>

	  Defines a submit command or submit variable with a value, and	parses
	  it  as  if  it was placed at the beginning of	the submit description
	  file.	The submit description file is not changed. To correctly parse
	  the  condor_submit command line, this	option must be specified with-
	  out white space characters before and	after the equals sign (	 =  ),
	  or the entire	option must be surrounded by double quote marks.

       -queue queue_arguments

	  A  command  line  specification  of how many jobs to queue, which is
	  only permitted if the	submit description file	does not have a	 queue
	  command.  The	queue_arguments	are the	same as	may be within a	submit
	  description file. The	parsing	of the queue_arguments finishes	at the
	  end  of  the	line  or  when	a dash character ( - ) is encountered.
	  Therefore, its best placement	within the command line	will be	at the
	  end of the command line.

	  On  a	Unix command line, the shell expands file globs	before parsing
	  occurs.

Submit Description File	Commands
       Each submit description file describes one cluster of jobs to be	placed
       in  the	HTCondor  execution pool. All jobs in a	cluster	must share the
       same executable,	but they may have different input  and	output	files,
       and  different  program	arguments.  The	submit description file	is the
       only command-line argument to condor_submit . If	the submit description
       file  argument  is omitted, condor_submit will read the submit descrip-
       tion from standard input.

       The submit description file must	contain	one executable command and  at
       least  one  queue  command.  All	of the other commands have default ac-
       tions.

       The commands which can appear in	the submit description file are	numer-
       ous. They are listed here in alphabetical order by category.

       BASIC COMMANDS

       arguments = <argument_list>

	  List	of  arguments  to be supplied to the executable	as part	of the
	  command line.

	  In the java universe,	the first argument must	be  the	 name  of  the
	  class	containing  main .

	  There	 are two permissible formats for specifying arguments, identi-
	  fied as the old syntax and the new syntax. The old  syntax  supports
	  white	 space	characters  within  arguments  only in special circum-
	  stances; when	used, the command line arguments  are  represented  in
	  the  job  ClassAd  attribute	Args . The new syntax supports uniform
	  quoting of white space characters within arguments; when  used,  the
	  command  line	arguments are represented in the job ClassAd attribute
	  Arguments .

	  Old Syntax

	  In the old syntax, individual	command	line arguments	are  delimited
	  (separated)  by space	characters. To allow a double quote mark in an
	  argument, it is escaped with a backslash; that is, the two character
	  sequence  \" becomes a single	double quote mark within an argument.

	  Further  interpretation  of the argument string differs depending on
	  the operating	system.	On Windows,  the  entire  argument  string  is
	  passed  verbatim  (other than	the backslash in front of double quote
	  marks) to the	Windows	application. Most  Windows  applications  will
	  allow	 spaces	 within	 an argument value by surrounding the argument
	  with double quotes marks. In all other cases,	there  is  no  further
	  interpretation of the	arguments.

	  Example:

       arguments = one \"two\" 'three'

	  Produces in Unix vanilla universe:

       argument	1: one
       argument	2: "two"
       argument	3: 'three'

	  New Syntax

	  Here are the rules for using the new syntax:

	     1.	 The  entire string representing the command line arguments is
	     surrounded	by double quote	marks. This permits  the  white	 space
	     characters	of spaces and tabs to potentially be embedded within a
	     single argument. Putting the double quote mark within  the	 argu-
	     ments  is	accomplished  by escaping it with another double quote
	     mark.

	     2.	The white space	characters of spaces  or  tabs	delimit	 argu-
	     ments.

	     3.	 To  embed  white  space characters of spaces or tabs within a
	     single argument, surround the entire argument with	 single	 quote
	     marks.

	     4.	To insert a literal single quote mark, escape it within	an ar-
	     gument already delimited by single	quote marks by adding  another
	     single quote mark.

	  Example:

       arguments = "3 simple arguments"

	  Produces:

       argument	1: 3
       argument	2: simple
       argument	3: arguments

	  Another example:

       arguments = "one	'two with spaces' 3"

	  Produces:

       argument	1: one
       argument	2: two with spaces
       argument	3: 3

	  And yet another example:

       arguments = "one	""two""	'spacey	''quoted'' argument'"

	  Produces:

       argument	1: one
       argument	2: "two"
       argument	3: spacey 'quoted' argument

	  Notice that in the new syntax, the backslash has no special meaning.
	  This is for the convenience of Windows users.

       environment = <parameter_list>

	  List of environment variables.

	  There	are two	different formats for specifying the environment vari-
	  ables: the old format	and the	new format. The	old format is retained
	  for backward-compatibility. It  suffers  from	 a  platform-dependent
	  syntax  and the inability to insert some special characters into the
	  environment.

	  The new syntax for specifying	environment values:

	     1.	Put double quote marks around the entire argument string. This
	     distinguishes  the	 new  syntax from the old. The old syntax does
	     not have double quote marks around	it. Any	literal	 double	 quote
	     marks  within  the	string must be escaped by repeating the	double
	     quote mark.

	     2.	Each environment entry has the form

       <name>=<value>

	     3.	Use white space	(space or tab characters) to separate environ-
	     ment entries.

	     4.	 To  put any white space in an environment entry, surround the
	     space and as much of the surrounding entry	as desired with	single
	     quote marks.

	     5.	To insert a literal single quote mark, repeat the single quote
	     mark anywhere inside of a	section	 surrounded  by	 single	 quote
	     marks.

	  Example:

       environment = "one=1 two=""2"" three='spacey ''quoted'' value'"

	  Produces the following environment entries:

       one=1
       two="2"
       three=spacey 'quoted' value

	  Under	 the  old  syntax, there are no	double quote marks surrounding
	  the environment specification. Each environment entry	remains	of the
	  form

       <name>=<value>

	  Under	 Unix,	list  multiple	environment entries by separating them
	  with a semicolon (;).	Under Windows, separate	multiple entries  with
	  a  vertical  bar (| ). There is no way to insert a literal semicolon
	  under	Unix or	a literal vertical bar under Windows. Note that	spaces
	  are  accepted, but rarely desired, characters	within parameter names
	  and values, because they are treated as literal characters, not sep-
	  arators  or  ignored	white space. Place spaces within the parameter
	  list only if required.

	  A Unix example:

       environment = one=1;two=2;three="quotes have no 'special' meaning"

	  This produces	the following:

       one=1
       two=2
       three="quotes have no 'special' meaning"

	  If the environment is	set with the environment command and getenv is
	  also	set to true, values specified with environment override	values
	  in the submitter's environment (regardless of	the order of the envi-
	  ronment and getenv commands).

       error = <pathname>

	  A  path and file name	used by	HTCondor to capture any	error messages
	  the program would normally write to the screen (that is,  this  file
	  becomes   stderr  ). A path is given with respect to the file	system
	  of the machine on which the job is submitted.	The  file  is  written
	  (by  the  job)  in the remote	scratch	directory of the machine where
	  the job is executed. When the	 job  exits,  the  resulting  file  is
	  transferred back to the machine where	the job	was submitted, and the
	  path is utilized for file placement. If not specified,  the  default
	  value	of  /dev/null is used for submission to	a Unix machine.	If not
	  specified, error messages are	ignored	for submission	to  a  Windows
	  machine. More	than one job should not	use the	same error file, since
	  this will cause one job to overwrite the errors of another. The  er-
	  ror file and the output file should not be the same file as the out-
	  puts will overwrite each other or be lost. For grid  universe	 jobs,
	  error	may be a URL that the Globus tool globus_url_copy understands.

       executable = <pathname>

	  An optional path and a required file name of the executable file for
	  this job cluster. Only one executable	command	within	a  submit  de-
	  scription  file  is guaranteed to work properly. More	than one often
	  works.

	  If no	path or	a relative path	is used, then the executable  file  is
	  presumed to be relative to the current working directory of the user
	  as the condor_submit command is issued.

	  If submitting	into the standard universe, then the named  executable
	  must	have  been  re-linked with the HTCondor	libraries (such	as via
	  the condor_compile command). If submitting into the vanilla universe
	  (the	default),  then	the named executable need not be re-linked and
	  can be any process which can run in the  background  (shell  scripts
	  work	fine  as well).	If submitting into the Java universe, then the
	  argument must	be a compiled  .class file.

       getenv =	<True |	False>

	  If getenv is set to  True , then condor_submit will copy all of  the
	  user's  current  shell environment variables at the time of job sub-
	  mission into the job ClassAd.	The job	will  therefore	 execute  with
	  the  same  set  of environment variables that	the user had at	submit
	  time.	Defaults to  False .

	  If the environment is	set with the environment command and getenv is
	  also	set to true, values specified with environment override	values
	  in the submitter's environment (regardless of	the order of the envi-
	  ronment and getenv commands).

       input = <pathname>

	  HTCondor  assumes  that its jobs are long-running, and that the user
	  will not wait	at the terminal	for their completion. Because of this,
	  the  standard	 files	which  normally	access the terminal, ( stdin ,
	  stdout , and	stderr ), must refer to	files.	Thus,  the  file  name
	  specified  with  input should	contain	any keyboard input the program
	  requires (that is, this file becomes	stdin ). A path	is given  with
	  respect  to  the file	system of the machine on which the job is sub-
	  mitted. The file is  transferred  before  execution  to  the	remote
	  scratch  directory  of the machine where the job is executed.	If not
	  specified, the default value of  /dev/null is	used for submission to
	  a Unix machine. If not specified, input is ignored for submission to
	  a Windows machine. For grid universe jobs, input may be a  URL  that
	  the Globus tool globus_url_copy understands.

	  Note	that this command does not refer to the	command-line arguments
	  of the program. The command-line arguments are specified by the  ar-
	  guments command.

       log = <pathname>

	  Use  log to specify a	file name where	HTCondor will write a log file
	  of what is happening with this job cluster, called a job event  log.
	  For example, HTCondor	will place a log entry into this file when and
	  where	the job	begins running,	when the job produces a	checkpoint, or
	  moves	 (migrates)  to	 another  machine, and when the	job completes.
	  Most users find specifying a log file	to be handy; its use is	recom-
	  mended. If no	log entry is specified,	HTCondor does not create a log
	  for this cluster. If a relative path is specified, it	is relative to
	  the  current working directory as the	job is submitted or the	direc-
	  tory specified by submit command initialdir on the submit machine.

       log_xml = <True | False>

	  If log_xml is	 True ,	then the job event log file will be written in
	  ClassAd  XML.	 If not	specified, XML is not used. Note that the file
	  is an	XML fragment; it is missing the	file header and	footer.	Do not
	  mix  XML and non-XML within a	single file. If	multiple jobs write to
	  a single job event log file, ensure that all	of  the	 jobs  specify
	  this option in the same way.

       notification = <Always |	Complete | Error | Never>

	  Owners  of  HTCondor jobs are	notified by e-mail when	certain	events
	  occur. If defined by Always ,	the owner will	be  notified  whenever
	  the job produces a checkpoint, as well as when the job completes. If
	  defined by Complete ,	the owner will be notified when	the job	termi-
	  nates.  If defined by	Error ,	the owner will only be notified	if the
	  job terminates abnormally, or	if the job is placed on	 hold  because
	  of  a	failure, and not by user request. If defined by	Never (the de-
	  fault), the owner will not receive e-mail, regardless	to  what  hap-
	  pens to the job. The HTCondor	User's manual documents	statistics in-
	  cluded in the	e-mail.

       notify_user = <email-address>

	  Used to specify the e-mail address to	use when HTCondor sends	e-mail
	  about	a job. If not specified, HTCondor defaults to using the	e-mail
	  address defined by

       job-owner@UID_DOMAIN

	  where	the configuration variable  UID_DOMAIN is specified by the HT-
	  Condor  site	administrator.	If  UID_DOMAIN has not been specified,
	  HTCondor sends the e-mail to:

       job-owner@submit-machine-name

       output =	<pathname>

	  The output file captures any information the program would  ordinar-
	  ily  write  to  the  screen (that is,	this file becomes  stdout ). A
	  path is given	with respect to	the file  system  of  the  machine  on
	  which	 the job is submitted. The file	is written (by the job)	in the
	  remote scratch directory of the machine where	the job	 is  executed.
	  When	the  job  exits, the resulting file is transferred back	to the
	  machine where	the job	was submitted, and the path  is	 utilized  for
	  file placement. If not specified, the	default	value of  /dev/null is
	  used for submission to a Unix	machine. If not	specified,  output  is
	  ignored  for	submission  to a Windows machine. Multiple jobs	should
	  not use the same output file,	since this will	cause one job to over-
	  write	 the  output  of  another.  The	output file and	the error file
	  should not be	the same file as the outputs will overwrite each other
	  or  be  lost.	 For  grid universe jobs, output may be	a URL that the
	  Globus tool globus_url_copy understands.

	  Note that if a program explicitly opens and writes to	a  file,  that
	  file should not be specified as the output file.

       priority	= <integer>

	  An  HTCondor	job  priority can be any integer, with 0 being the de-
	  fault. Jobs with higher numerical priority will run before jobs with
	  lower	 numerical  priority. Note that	this priority is on a per user
	  basis. One user with many jobs may use this command to order his/her
	  own  jobs, and this will have	no effect on whether or	not these jobs
	  will run ahead of another user's jobs.

       queue [ <int expr> ]

	  Places zero or more copies of	the job	into the HTCondor queue.

       queue

	  [ <int expr> ] [ <varname> ] in [ slice ]  <list  of	items>	Places
	  zero	or  more  copies  of  the job in the queue based on items in a
	  <list	of items>

       queue

	  [ <int expr> ] [ <varname> ] matching	[ files	| dirs	]  [  slice  ]
	  <list	 of  items with	file globbing> ] Places	zero or	more copies of
	  the job in the queue based on	files that match a <list of items with
	  file globbing>

       queue

	  [  <int  expr> ] [ <list of varnames>	] from [ slice ] <file name> |
	  <list	of items> ] Places zero	or more	copies of the job in the queue
	  based	on lines from the submit file or from <file name>

	  The  optional	argument _int expr_ specifies how many times to	repeat
	  the job submission for a given set of	arguments. It may be an	 inte-
	  ger  or  an expression that evaluates	to an integer, and it defaults
	  to 1.	All but	the first form of this command	are  various  ways  of
	  specifying  a	 list  of  items. When these forms are used _int expr_
	  jobs will be queued for each item in the list. The in	, matching and
	  from keyword indicates how the list will be specified.

	     *	in  The	 list of items is an explicit comma and/or space sepa-
	     rated <list of items> . If	the <list of  items>  begins  with  an
	     open  paren,  and	the close paren	is not on the same line	as the
	     open, then	the list continues until a line	 that  begins  with  a
	     close paren is read from the submit file.

	     *	matching  Each	item in	the <list of items with	file globbing>
	     will be matched against the names of files	and directories	 rela-
	     tive  to  the current directory, the set of matching names	is the
	     resulting list of items.

		* files	Only filenames will matched.

		* dirs Only directory names will be matched.

	     * from <file name>	| <list	of items> Each line from  <file	 name>
	     or	 <list	of  items>  is a single	item, this allows for multiple
	     variables to be set for each item.	 Lines	from  <file  name>  or
	     <list  of	items> will be split on	comma and/or space until there
	     are values	for each of the	variables specified in <list  of  var-
	     names>  .	The  last  variable  will contain the remainder	of the
	     line. When	the <list of items> form is used, the  list  continues
	     until  the	 first	line that begins with a	close paren, and lines
	     beginning with pound sign ('#') will be skipped. When  using  the
	     <file name> form, if the <file name> ends with |, then it will be
	     executed as a script whatever the script writes to	  stdout  will
	     be	the list of items. The optional	argument _varname_ or _list of
	     varnames_ is the name or names of of variables that will  be  set
	     to	 the  value  of	 the  current item when	queuing	the job. If no
	     _varname_ is specified the	variable ITEM will  be	used.  Leading
	     and trailing whitespace be	trimmed. The optional argument _slice_
	     is	a python style slice selecting only some of the	items  in  the
	     list of items. Negative step values are not supported.

	  A  submit file may contain more than one queue statement, and	if de-
	  sired, any commands may be placed between subsequent queue commands,
	  such	as new input , output ,	error ,	initialdir , or	arguments com-
	  mands. This is handy when submitting multiple	runs into one  cluster
	  with one submit description file.

       universe	= <vanilla | standard |	scheduler | local | grid | java	| vm |
       docker>

	  Specifies which HTCondor universe to use when	running	this job.  The
	  HTCondor universe specifies an HTCondor execution environment.

	  The  vanilla universe	is the default (except where the configuration
	  variable  DEFAULT_UNIVERSE defines it	otherwise), and	is  an	execu-
	  tion environment for jobs which do not use HTCondor's	mechanisms for
	  taking checkpoints; these are	ones that have not  been  linked  with
	  the  HTCondor	 libraries.  Use  the vanilla universe to submit shell
	  scripts to HTCondor.

	  The standard universe	tells HTCondor that  this  job	has  been  re-
	  linked  via condor_compile with the HTCondor libraries and therefore
	  supports taking checkpoints and remote system	calls.

	  The scheduler	universe is for	a job that is to run  on  the  machine
	  where	the job	is submitted. This universe is intended	for a job that
	  acts as a metascheduler and will not be preempted.

	  The local universe is	for a job that is to run on the	machine	 where
	  the  job  is	submitted.  This universe runs the job immediately and
	  will not preempt the job.

	  The grid universe forwards the job to	 an  external  job  management
	  system.  Further specification of the	grid universe is done with the
	  grid_resource	command.

	  The java universe is for programs written to the  Java  Virtual  Ma-
	  chine.

	  The vm universe facilitates the execution of a virtual machine.

	  The docker universe runs a docker container as an HTCondor job.

       COMMANDS	FOR MATCHMAKING

       rank = <ClassAd Float Expression>

	  A ClassAd Floating-Point expression that states how to rank machines
	  which	have already met  the  requirements  expression.  Essentially,
	  rank	expresses  preference.	A  higher  numeric value equals	better
	  rank.	HTCondor will give the job the machine with the	highest	 rank.
	  For example,

	       request_memory =	max({60, Target.TotalSlotMemory})
	       rank = Memory

	  asks	HTCondor  to  find  all	 available  machines with more than 60
	  megabytes of memory and give to the job the machine  with  the  most
	  amount  of  memory. The HTCondor User's Manual contains complete in-
	  formation on the syntax and available	attributes that	can be used in
	  the ClassAd expression.

       request_cpus = <num-cpus>

	  A requested number of	CPUs (cores). If not specified,	the number re-
	  quested will be 1. If	specified, the expression

	 && (RequestCpus <= Target.Cpus)

	  is appended to the requirements expression for the job.

	  For pools that enable	dynamic	condor_startd provisioning,  specifies
	  the  minimum	number	of CPUs	requested for this job,	resulting in a
	  dynamic slot being created with this many cores.

       request_disk = <quantity>

	  The requested	amount of disk space in	KiB requested for this job. If
	  not  specified,  it  will  be	 set  to  the  job  ClassAd  attribute
	  DiskUsage . The expression

	 && (RequestDisk <= Target.Disk)

	  is appended to the requirements expression for the job.

	  For pools that enable	dynamic	condor_startd provisioning, a  dynamic
	  slot will be created with at least this much disk space.

	  Characters  may  be appended to a numerical value to indicate	units.
	  K or	KB indicates KiB, numbers of bytes.  M or  MB  indicates  MiB,
	  numbers  of  bytes.  G or  GB	indicates GiB, numbers of bytes.  T or
	  TB indicates TiB, numbers of bytes.

       request_memory =	<quantity>

	  The required amount of memory	in MiB that this job  needs  to	 avoid
	  excessive  swapping. If not specified	and the	submit command vm_mem-
	  ory is specified, then the value specified for vm_memory defines re-
	  quest_memory . If neither request_memory nor vm_memory is specified,
	  the value is set by the configuration	variable  JOB_DEFAULT_REQUEST-
	  MEMORY . The actual amount of	memory used by a job is	represented by
	  the job ClassAd attribute  MemoryUsage .

	  For pools that enable	dynamic	condor_startd provisioning, a  dynamic
	  slot will be created with at least this much RAM.

	  The expression

	 && (RequestMemory <= Target.Memory)

	  is appended to the requirements expression for the job.

	  Characters  may  be appended to a numerical value to indicate	units.
	  K or	KB indicates KiB, numbers of bytes.  M or  MB  indicates  MiB,
	  numbers  of  bytes.  G or  GB	indicates GiB, numbers of bytes.  T or
	  TB indicates TiB, numbers of bytes.

       request_<name> =	<quantity>

	  The required amount of the custom  machine  resource	identified  by
	  <name>  that	this job needs.	The custom machine resource is defined
	  in the machine's configuration. Machines that	 have  available  GPUs
	  will define  <name> to be  GPUs .

       requirements = <ClassAd Boolean Expression>

	  The  requirements command is a boolean ClassAd expression which uses
	  C-like operators. In order for any job in this cluster to run	 on  a
	  given	machine, this requirements expression must evaluate to true on
	  the given machine.

	  For scheduler	and local universe jobs, the  requirements  expression
	  is evaluated against the  Scheduler ClassAd which represents the the
	  condor_schedd	daemon running on the submit machine,  rather  than  a
	  remote machine. Like all commands in the submit description file, if
	  multiple requirements	commands are present, all but the last one are
	  ignored.  By default,	condor_submit appends the following clauses to
	  the requirements expression:

	     1.	Arch and OpSys are set equal to	the Arch and OpSys of the sub-
	     mit machine. In other words: unless you request otherwise,	HTCon-
	     dor will give your	job machines with the  same  architecture  and
	     operating system version as the machine running condor_submit .

	     2.	 Cpus >=RequestCpus, if	the job	ClassAd	attribute  RequestCpus
	     is	defined.

	     3.	Disk >=RequestDisk, if the job ClassAd attribute   RequestDisk
	     is	 defined.  Otherwise,  Disk >=DiskUsage	is appended to the re-
	     quirements. The  DiskUsage	attribute is initialized to  the  size
	     of	 the  executable  plus	the  size  of any files	specified in a
	     transfer_input_files command. It exists to	ensure there is	enough
	     disk  space  on the target	machine	for HTCondor to	copy over both
	     the executable and	needed input files. The	  DiskUsage  attribute
	     represents	the maximum amount of total disk space required	by the
	     job in kilobytes. HTCondor	automatically updates  the   DiskUsage
	     attribute	approximately every 20 minutes while the job runs with
	     the amount	of space being used by the job on the execute machine.

	     4.	Memory >=RequestMemory,	if the job ClassAd attribute  Request-
	     Memory is defined.

	     5.	 If  Universe is set to	Vanilla, FileSystemDomain is set equal
	     to	the submit machine's FileSystemDomain. View  the  requirements
	     of	 a job which has already been submitted	(along with everything
	     else about	the job	ClassAd) with the command condor_q  -l	;  see
	     the command reference for condor_q	on page	. Also,	see the	HTCon-
	     dor Users Manual for  complete  information  on  the  syntax  and
	     available attributes that can be used in the ClassAd expression.

       FILE TRANSFER COMMANDS

       dont_encrypt_input_files	= <file1,file2,file... >

	  A  comma  and/or space separated list	of input files that are	not to
	  be network encrypted when transferred	with the file transfer	mecha-
	  nism.	 Specification of files	in this	manner overrides configuration
	  that would use encryption. Each input	file must also be in the  list
	  given	 by transfer_input_files . When	a path to an input file	or di-
	  rectory is specified,	this specifies the path	to  the	 file  on  the
	  submit  side.	A single wild card character ( * ) may be used in each
	  file name.

       dont_encrypt_output_files = <file1,file2,file...	>

	  A comma and/or space separated list of output	files that are not  to
	  be  network  encrypted  when transferred back	with the file transfer
	  mechanism. Specification of files in this manner overrides  configu-
	  ration  that	would use encryption. The output file(s) must also ei-
	  ther be in the list given by transfer_output_files or	be  discovered
	  and  to be transferred back with the file transfer mechanism.	When a
	  path to an output file or directory is specified, this specifies the
	  path to the file on the execute side.	A single wild card character (
	  * ) may be used in each file name.

       encrypt_execute_directory = <True | False>

	  Defaults to  False . If set to  True ,  HTCondor  will  encrypt  the
	  contents  of	the  remote scratch directory of the machine where the
	  job is executed. This	encryption is transparent to the  job  itself,
	  but  ensures that files left behind on the local disk	of the execute
	  machine, perhaps due to a system crash, will remain private. In  ad-
	  dition,  condor_submit will append to	the job's requirements expres-
	  sion

	 && (TARGET.HasEncryptExecuteDirectory)

	  to ensure the	job is matched to a machine that  is  capable  of  en-
	  crypting the contents	of the execute directory. This support is lim-
	  ited to Windows platforms that use the NTFS file  system  and	 Linux
	  platforms with the ecryptfs-utils package installed.

       encrypt_input_files = <file1,file2,file... >

	  A  comma  and/or  space separated list of input files	that are to be
	  network encrypted when transferred with the file transfer mechanism.
	  Specification	 of  files in this manner overrides configuration that
	  would	not use	encryption. Each input file must also be in  the  list
	  given	 by transfer_input_files . When	a path to an input file	or di-
	  rectory is specified,	this specifies the path	to  the	 file  on  the
	  submit  side.	A single wild card character ( * ) may be used in each
	  file name. The method	of encryption utilized will be as agreed  upon
	  in  security	negotiation; if	that negotiation failed, then the file
	  transfer mechanism must also fail for	files to be network encrypted.

       encrypt_output_files = <file1,file2,file... >

	  A comma and/or space separated list of output	files that are	to  be
	  network encrypted when transferred back with the file	transfer mech-
	  anism. Specification of files	in this	manner overrides configuration
	  that	would  not use encryption. The output file(s) must also	either
	  be in	the list given by transfer_output_files	or be  discovered  and
	  to be	transferred back with the file transfer	mechanism. When	a path
	  to an	output file or directory is specified, this specifies the path
	  to  the file on the execute side. A single wild card character ( * )
	  may be used in each file name. The  method  of  encryption  utilized
	  will	be as agreed upon in security negotiation; if that negotiation
	  failed, then the file	transfer mechanism must	also fail for files to
	  be network encrypted.

       max_transfer_input_mb = <ClassAd	Integer	Expression>

	  This	integer	expression specifies the maximum allowed total size in
	  MiB of the input files that are transferred for a job. This  expres-
	  sion	does  not  apply to grid universe, standard universe, or files
	  transferred via file transfer	plug-ins. The expression may refer  to
	  attributes  of  the job. The special value -1	indicates no limit. If
	  not defined, the value set  by  configuration	 variable   MAX_TRANS-
	  FER_INPUT_MB	is  used.  If  the observed size of all	input files at
	  submit time is larger	than the limit,	the job	 will  be  immediately
	  placed on hold with a	 HoldReasonCode	value of 32. If	the job	passes
	  this initial test, but the size of the input files increases or  the
	  limit	 decreases  so	that  the  limit  is violated, the job will be
	  placed on hold at the	time when the file transfer is attempted.

       max_transfer_output_mb =	<ClassAd Integer Expression>

	  This integer expression specifies the	maximum	allowed	total size  in
	  MiB of the output files that are transferred for a job. This expres-
	  sion does not	apply to grid universe,	standard  universe,  or	 files
	  transferred  via file	transfer plug-ins. The expression may refer to
	  attributes of	the job. The special value -1 indicates	no  limit.  If
	  not  set, the	value set by configuration variable  MAX_TRANSFER_OUT-
	  PUT_MB is used. If the total size of the job's output	 files	to  be
	  transferred is larger	than the limit,	the job	will be	placed on hold
	  with a  HoldReasonCode value of 33. The output will  be  transferred
	  up  to  the  point when the limit is hit, so some files may be fully
	  transferred, some partially, and some	not at all.

       output_destination = <destination-URL>

	  When present,	defines	a URL that specifies both a plug-in and	a des-
	  tination  for	 the transfer of the entire output sandbox or a	subset
	  of output files as specified by  the	submit	command	 transfer_out-
	  put_files . The plug-in does the transfer of files, and no files are
	  sent back to the submit machine. The HTCondor	Administrator's	manual
	  has full details.

       should_transfer_files = <YES | NO | IF_NEEDED >

	  The  should_transfer_files  setting  is  used	 to define if HTCondor
	  should transfer files	to and from the	remote machine where  the  job
	  runs.	 The file transfer mechanism is	used to	run jobs which are not
	  in the standard universe (and	can therefore use remote system	 calls
	  for  file access) on machines	which do not have a shared file	system
	  with the submit machine.  should_transfer_files equal	 to  YES  will
	  cause	 HTCondor  to  always transfer files for the job.  NO disables
	  HTCondor's file transfer mechanism.	IF_NEEDED  will	 not  transfer
	  files	 for  the  job	if  it	is matched with	a resource in the same
	  FileSystemDomain as the submit machine (and therefore, on a  machine
	  with	the same shared	file system). If the job is matched with a re-
	  mote resource	in  a  different   FileSystemDomain  ,	HTCondor  will
	  transfer the necessary files.

	  For more information about this and other settings related to	trans-
	  ferring files, see the HTCondor User's manual	section	 on  the  file
	  transfer mechanism.

	  Note	that should_transfer_files is not supported for	jobs submitted
	  to the grid universe.

       skip_filechecks = <True | False>

	  When	True , file permission checks for the submitted	job  are  dis-
	  abled.  When	 False , file permissions are checked; this is the be-
	  havior when this command is not present in  the  submit  description
	  file.	File permissions are checked for read permissions on all input
	  files, such as those defined	by  commands  input  and  transfer_in-
	  put_files  , and for write permission	to output files, such as a log
	  file defined by log and output files defined with output  or	trans-
	  fer_output_files .

       stream_error = <True | False>

	  If   True  , then  stderr is streamed	back to	the machine from which
	  the job was submitted. If  False ,  stderr  is  stored  locally  and
	  transferred  back when the job completes. This command is ignored if
	  the job ClassAd attribute  TransferErr is  False . The default value
	  is   False  .	 This command must be used in conjunction with error ,
	  otherwise  stderr will sent to  /dev/null on Unix machines  and  ig-
	  nored	on Windows machines.

       stream_input = <True | False>

	  If  True , then  stdin is streamed from the machine on which the job
	  was submitted. The default value is  False .	The  command  is  only
	  relevant for jobs submitted to the vanilla or	java universes,	and it
	  is ignored by	the grid universe. This	command	must be	used  in  con-
	  junction  with  input	 , otherwise  stdin will be  /dev/null on Unix
	  machines and ignored on Windows machines.

       stream_output = <True | False>

	  If  True , then  stdout is streamed back to the machine  from	 which
	  the  job  was	 submitted.  If	 False ,  stdout is stored locally and
	  transferred back when	the job	completes. This	command	is ignored  if
	  the job ClassAd attribute  TransferOut is  False . The default value
	  is  False . This command must	be used	in conjunction with  output  ,
	  otherwise   stdout  will sent	to  /dev/null on Unix machines and ig-
	  nored	on Windows machines.

       transfer_executable = <True | False>

	  This command is applicable to	jobs submitted to the grid and vanilla
	  universes.  If  transfer_executable is set to	 False , then HTCondor
	  looks	for the	executable on the remote machine, and does not	trans-
	  fer  the  executable	over. This is useful for an already pre-staged
	  executable; HTCondor behaves more like rsh.  The  default  value  is
	  True .

       transfer_input_files = <file1,file2,file... >

	  A comma-delimited list of all	the files and directories to be	trans-
	  ferred into the working directory for	the job,  before  the  job  is
	  started.  By	default,  the file specified in	the executable command
	  and any file specified in the	input command (for example,   stdin  )
	  are transferred.

	  When	a path to an input file	or directory is	specified, this	speci-
	  fies the path	to the file on the submit side.	The file is placed  in
	  the job's temporary scratch directory	on the execute side, and it is
	  named	using the  base	 name  of  the	original  path.	 For  example,
	  /path/to/input_file  becomes	input_file in the job's	scratch	direc-
	  tory.

	  A directory may be specified by appending the	forward	slash  charac-
	  ter  (/)  as a trailing path separator. This syntax is used for both
	  Windows and Linux submit hosts. A directory example using a trailing
	  path	separator is  input_data/ . When a directory is	specified with
	  the trailing path separator,	the  contents  of  the	directory  are
	  transferred,	but  the directory itself is not transferred. It is as
	  if each of the items within the directory were listed	in the	trans-
	  fer list. When there is no trailing path separator, the directory is
	  transferred, its contents are	transferred, and  these	 contents  are
	  placed inside	the transferred	directory.

	  For grid universe jobs other than HTCondor-C,	the transfer of	direc-
	  tories is not	currently supported.

	  Symbolic links to files are transferred as the files they point  to.
	  Transfer  of	symbolic  links	 to  directories is not	currently sup-
	  ported.

	  For vanilla and vm universe jobs only, a file	may  be	 specified  by
	  giving  a  URL,  instead  of a file name. The	implementation for URL
	  transfers requires both configuration	and available plug-in.

       transfer_output_files = <file1,file2,file... >

	  This command forms an	explicit list of output	files and  directories
	  to  be  transferred back from	the temporary working directory	on the
	  execute machine to the submit	machine. If there are multiple	files,
	  they must be delimited with commas. Setting transfer_output_files to
	  the empty string ( ""	) means	that no	files are to be	transferred.

	  For HTCondor-C jobs and all other non-grid universe jobs, if	trans-
	  fer_output_files  is	not  specified,	 HTCondor  will	 automatically
	  transfer back	all files in the  job's	 temporary  working  directory
	  which	 have  been modified or	created	by the job. Subdirectories are
	  not scanned for output, so if	output from subdirectories is desired,
	  the output list must be explicitly specified.	For grid universe jobs
	  other	than HTCondor-C, desired output	files must also	be  explicitly
	  listed.  Another reason to explicitly	list output files is for a job
	  that creates many files, and the user	wants  only  a	subset	trans-
	  ferred back.

	  For  grid  universe  jobs other than with grid type condor , to have
	  files	other than standard output and standard	error transferred from
	  the execute machine back to the submit machine, do use transfer_out-
	  put_files , listing all files	to be  transferred.  These  files  are
	  found	on the execute machine in the working directory	of the job.

	  When	a  path	to an output file or directory is specified, it	speci-
	  fies the path	to the file on the execute side. As a  destination  on
	  the submit side, the file is placed in the job's initial working di-
	  rectory, and it is named using the base name of the  original	 path.
	  For  example,	 path/to/output_file becomes  output_file in the job's
	  initial working directory. The name and path of  the	file  that  is
	  written  on  the  submit side	may be modified	by using transfer_out-
	  put_remaps . Note that this remap function only works	with files but
	  not with directories.

	  A directory may be specified using a trailing	path separator.	An ex-
	  ample	of a trailing path separator is	the slash  character  on  Unix
	  platforms;  a	 directory  example using a trailing path separator is
	  input_data/ .	When a directory is specified  with  a	trailing  path
	  separator,  the  contents  of	the directory are transferred, but the
	  directory itself is not transferred. It is as	if each	of  the	 items
	  within the directory were listed in the transfer list. When there is
	  no trailing path separator, the directory is transferred,  its  con-
	  tents	 are  transferred,  and	 these	contents are placed inside the
	  transferred directory.

	  For grid universe jobs other than HTCondor-C,	the transfer of	direc-
	  tories is not	currently supported.

	  Symbolic  links to files are transferred as the files	they point to.
	  Transfer of symbolic links to	 directories  is  not  currently  sup-
	  ported.

       transfer_output_remaps <`` name newname ; name2 newname2	... ''>

	  This	specifies the name (and	optionally path) to use	when download-
	  ing output files from	the completed job. Normally, output files  are
	  transferred back to the initial working directory with the same name
	  they had in the execution directory. This gives you  the  option  to
	  save	them  with a different path or name. If	you specify a relative
	  path,	the final path will be relative	to the job's  initial  working
	  directory.

	  name describes an output file	name produced by your job, and newname
	  describes the	file name it should be downloaded to. Multiple	remaps
	  can be specified by separating each with a semicolon.	If you wish to
	  remap	file names that	contain	equals signs or	semicolons, these spe-
	  cial	characters may be escaped with a backslash. You	cannot specify
	  directories to be remapped.

       when_to_transfer_output = <ON_EXIT | ON_EXIT_OR_EVICT >

	  Setting when_to_transfer_output equal	to ON_EXIT will	cause HTCondor
	  to  transfer	the  job's output files	back to	the submitting machine
	  only when the	job completes (exits on	its own).

	  The ON_EXIT_OR_EVICT option is  intended  for	 fault	tolerant  jobs
	  which	 periodically  save their own state and	can restart where they
	  left off. In this case, files	are spooled to the submit machine  any
	  time	the  job leaves	a remote site, either because it exited	on its
	  own, or was evicted by the HTCondor system for any reason  prior  to
	  job completion. The files spooled back are placed in a directory de-
	  fined	by the value of	the  SPOOL configuration variable. Any	output
	  files	 transferred back to the submit	machine	are automatically sent
	  back out again as input files	if the job restarts.

       POLICY COMMANDS

       hold = <True | False>

	  If hold is set to  True , then the submitted job will	be placed into
	  the  Hold  state. Jobs in the	Hold state will	not run	until released
	  by condor_release . Defaults to  False .

       keep_claim_idle = <integer>

	  An integer number of seconds that a job requests  the	 condor_schedd
	  to  wait before releasing its	claim after the	job exits or after the
	  job is removed.

	  The process by which the condor_schedd  claims  a  condor_startd  is
	  somewhat  time-consuming.  To	 amortize this cost, the condor_schedd
	  tries	to reuse claims	to run subsequent jobs,	after a	 job  using  a
	  claim	 is done. However, it can only do this if there	is an idle job
	  in the queue at the moment the previous  job	completes.  Sometimes,
	  and  especially for the node jobs when using DAGMan, there is	a sub-
	  sequent job about to be submitted, but it has	not yet	arrived	in the
	  queue	 when  the  previous  job  completes.  As  a  result, the con-
	  dor_schedd releases the claim, and the next job must wait an	entire
	  negotiation cycle to start. When this	submit command is defined with
	  a non-negative integer, when the job exits, the condor_schedd	 tries
	  as  usual to reuse the claim.	If it cannot, instead of releasing the
	  claim, the condor_schedd keeps the claim until either	the number  of
	  seconds  given as a parameter, or a new job which matches that claim
	  arrives, whichever comes first. The condor_startd in	question  will
	  remain  in  the  Claimed/Idle	 state,	 and  the original job will be
	  "charged" (in	terms of priority) for the time	in this	state.

       leave_in_queue =	<ClassAd Boolean Expression>

	  When the ClassAd Expression evaluates	to  True , the job is not  re-
	  moved	 from the queue	upon completion. This allows the user of a re-
	  motely spooled job to	retrieve output	files in cases where  HTCondor
	  would	 have removed them as part of the cleanup associated with com-
	  pletion. The job will	only exit the queue once it  has  been	marked
	  for removal (via condor_rm , for example) and	the leave_in_queue ex-
	  pression has become  False .	leave_in_queue defaults	to  False .

	  As an	example, if the	job is to be removed once the  output  is  re-
	  trieved with condor_transfer_data , then use

       leave_in_queue  =  (JobStatus  == 4) && ((StageOutFinish	=?= UNDEFINED)
       ||\
			(StageOutFinish	== 0))

       next_job_start_delay = <ClassAd Boolean Expression>

	  This expression specifies the	 number	 of  seconds  to  delay	 after
	  starting up this job before the next job is started. The maximum al-
	  lowed	delay is specified  by	the  HTCondor  configuration  variable
	  MAX_NEXT_JOB_START_DELAY  ,  which defaults to 10 minutes. This com-
	  mand does not	apply to scheduler or local universe jobs.

	  This command has been	historically used to implement a form  of  job
	  start	throttling from	the job	submitter's perspective. It was	effec-
	  tive for the case of multiple	job submission where the  transfer  of
	  extremely  large  input  data	sets to	the execute machine caused ma-
	  chine	performance to suffer. This command is no  longer  useful,  as
	  throttling  should be	accomplished through configuration of the con-
	  dor_schedd daemon.

       on_exit_hold = <ClassAd Boolean Expression>

	  The ClassAd expression is checked when the job exits,	and if	True ,
	  places  the  job  into  the Hold state. If  False (the default value
	  when not defined), then nothing happens and the  on_exit_remove  ex-
	  pression is checked to determine if that needs to be applied.

	  For example: Suppose a job is	known to run for a minimum of an hour.
	  If the job exits after less than an hour, the	job should  be	placed
	  on hold and an e-mail	notification sent, instead of being allowed to
	  leave	the queue.

	 on_exit_hold =	(time()	- JobStartDate)	< (60 *	$(MINUTE))

	  This expression places the job on hold if it exits  for  any	reason
	  before  running  for an hour.	An e-mail will be sent to the user ex-
	  plaining that	the job	was placed on hold because this	expression be-
	  came	True .

	   periodic_* expressions take precedence over	on_exit_* expressions,
	  and  *_hold expressions take precedence  over	 a   *_remove  expres-
	  sions.

	  Only	job ClassAd attributes will be defined for use by this ClassAd
	  expression. This expression is available for the vanilla, java, par-
	  allel,  grid,	 local	and  scheduler	universes.  It is additionally
	  available, when submitted from a Unix	machine, for the standard uni-
	  verse.

       on_exit_hold_reason = <ClassAd String Expression>

	  When	the  job  is placed on hold due	to the on_exit_hold expression
	  becoming  True , this	expression is evaluated	to set	the  value  of
	  HoldReason  in  the job ClassAd. If this expression is  UNDEFINED or
	  produces an empty or invalid string, a default description is	used.

       on_exit_hold_subcode = <ClassAd Integer Expression>

	  When the job is placed on hold due to	 the  on_exit_hold  expression
	  becoming   True  ,  this expression is evaluated to set the value of
	  HoldReasonSubCode in the job ClassAd.	The default subcode is 0.  The
	  HoldReasonCode  will	be set to 3, which indicates that the job went
	  on hold due to a job policy expression.

       on_exit_remove =	<ClassAd Boolean Expression>

	  The ClassAd expression is checked when the job exits,	and  if	  True
	  (the	default	value when undefined), then it allows the job to leave
	  the queue normally. If  False	, then the job is placed back into the
	  Idle	state.	If  the	user job runs under the	vanilla	universe, then
	  the job restarts from	the beginning. If the user job runs under  the
	  standard  universe,  then it continues from where it left off, using
	  the last checkpoint.

	  For example, suppose a job occasionally segfaults, but  chances  are
	  that	the  job will finish successfully if the job is	run again with
	  the same data. The on_exit_remove expression can cause  the  job  to
	  run again with the following command.	Assume that the	signal identi-
	  fier for the segmentation fault is 11	on the platform	where the  job
	  will be running.

	 on_exit_remove	= (ExitBySignal	== False) || (ExitSignal != 11)

	  This	expression  lets  the  job  leave the queue if the job was not
	  killed by a signal or	if it was killed by a signal  other  than  11,
	  representing	segmentation  fault in this example. So, if the	exited
	  due to signal	11, it will stay in the	job queue. In any  other  case
	  of  the  job	exiting,  the  job will	leave the queue	as it normally
	  would	have done.

	  As another example, if the job should	only leave the queue if	it ex-
	  ited	on its own with	status 0, this on_exit_remove expression works
	  well:

	 on_exit_remove	= (ExitBySignal	== False) && (ExitCode == 0)

	  If the job was killed	by a signal or exited  with  a	non-zero  exit
	  status, HTCondor would leave the job in the queue to run again.

	   periodic_* expressions take precedence over	on_exit_* expressions,
	  and  *_hold expressions take precedence  over	 a   *_remove  expres-
	  sions.

	  Only	job ClassAd attributes will be defined for use by this ClassAd
	  expression. This expression is available for the vanilla, java, par-
	  allel,  grid,	 local	and  scheduler	universes.  It is additionally
	  available, when submitted from a Unix	machine, for the standard uni-
	  verse.  Note	that the condor_schedd daemon, by default, only	checks
	  these	periodic expressions once every	300  seconds.  The  period  of
	  these	 evaluations can be adjusted by	setting	the  PERIODIC_EXPR_IN-
	  TERVAL configuration macro.

       periodic_hold = <ClassAd	Boolean	Expression>

	  This expression is checked periodically at an	interval of the	number
	  of seconds set by the	configuration variable	PERIODIC_EXPR_INTERVAL
	  . If it becomes  True	, the job will be placed on hold. If  unspeci-
	  fied,	the default value is  False .

	   periodic_* expressions take precedence over	on_exit_* expressions,
	  and  *_hold expressions take precedence  over	 a   *_remove  expres-
	  sions.

	  Only	job ClassAd attributes will be defined for use by this ClassAd
	  expression. This expression is available for the vanilla, java, par-
	  allel,  grid,	 local	and  scheduler	universes.  It is additionally
	  available, when submitted from a Unix	machine, for the standard uni-
	  verse.  Note	that the condor_schedd daemon, by default, only	checks
	  these	periodic expressions once every	300  seconds.  The  period  of
	  these	 evaluations can be adjusted by	setting	the  PERIODIC_EXPR_IN-
	  TERVAL configuration macro.

       periodic_hold_reason = <ClassAd String Expression>

	  When the job is placed on hold due to	the  periodic_hold  expression
	  becoming   True  ,  this expression is evaluated to set the value of
	  HoldReason in	the job	ClassAd. If this expression is	 UNDEFINED  or
	  produces an empty or invalid string, a default description is	used.

       periodic_hold_subcode = <ClassAd	Integer	Expression>

	  When	the  job is placed on hold due to the periodic_hold expression
	  becoming true, this expression is evaluated  to  set	the  value  of
	  HoldReasonSubCode  in	the job	ClassAd. The default subcode is	0. The
	  HoldReasonCode will be set to	3, which indicates that	the  job  went
	  on hold due to a job policy expression.

       periodic_release	= <ClassAd Boolean Expression>

	  This expression is checked periodically at an	interval of the	number
	  of seconds set by the	configuration variable	PERIODIC_EXPR_INTERVAL
	  while	 the job is in the Hold	state. If the expression becomes  True
	  , the	job will be released.

	  Only job ClassAd attributes will be defined for use by this  ClassAd
	  expression. This expression is available for the vanilla, java, par-
	  allel, grid, local  and  scheduler  universes.  It  is  additionally
	  available, when submitted from a Unix	machine, for the standard uni-
	  verse. Note that the condor_schedd daemon, by	default,  only	checks
	  periodic  expressions	 once  every  300 seconds. The period of these
	  evaluations can be adjusted by setting  the	PERIODIC_EXPR_INTERVAL
	  configuration	macro.

       periodic_remove = <ClassAd Boolean Expression>

	  This expression is checked periodically at an	interval of the	number
	  of seconds set by the	configuration variable	PERIODIC_EXPR_INTERVAL
	  .  If	 it  becomes  True , the job is	removed	from the queue.	If un-
	  specified, the default value is  False .

	  See the Examples section of this manual page for an example of a pe-
	  riodic_remove	expression.

	   periodic_* expressions take precedence over	on_exit_* expressions,
	  and  *_hold expressions take precedence  over	 a   *_remove  expres-
	  sions.  So, the  periodic_remove expression takes precedent over the
	  on_exit_remove expression, if	the two	describe conflicting actions.

	  Only job ClassAd attributes will be defined for use by this  ClassAd
	  expression. This expression is available for the vanilla, java, par-
	  allel, grid, local  and  scheduler  universes.  It  is  additionally
	  available, when submitted from a Unix	machine, for the standard uni-
	  verse. Note that the condor_schedd daemon, by	default,  only	checks
	  periodic  expressions	 once  every  300 seconds. The period of these
	  evaluations can be adjusted by setting  the	PERIODIC_EXPR_INTERVAL
	  configuration	macro.

       COMMANDS	SPECIFIC TO THE	STANDARD UNIVERSE

       allow_startup_script = <True | False>

	  If  True,  a	standard universe job will execute a script instead of
	  submitting the job, and the consistency check	to  see	 if  the  exe-
	  cutable  has	been  linked using condor_compile is omitted. The exe-
	  cutable command within the submit  description  file	specifies  the
	  name	of  the	 script. The script is used to do preprocessing	before
	  the job is submitted.	The shell script ends with an exec of the  job
	  executable,  such  that the process id of the	executable is the same
	  as that of the shell script. Here is an example script that  gets  a
	  copy of a machine-specific executable	before the exec	.

	  #! /bin/sh

	  # get	the host name of the machine
	  $host=`uname -n`

	  # grab a standard universe executable	designed specifically
	  # for	this host
	  scp elsewhere@cs.wisc.edu:${host} executable

	  #  The  PID  MUST  stay  the same, so	exec the new standard universe
       process.
	  exec executable ${1+"$@"} If this command is not present  (defined),
       then the	value defaults to false.

       append_files = file1, file2, ...

	  If your job attempts to access a file	mentioned in this list,	HTCon-
	  dor will force all writes to that file to be appended	 to  the  end.
	  Furthermore,	condor_submit will not truncate	it. This list uses the
	  same syntax as compress_files, shown above.

	  This option may yield	some surprising	results. If several  jobs  at-
	  tempt	 to write to the same file, their output may be	intermixed. If
	  a job	is evicted from	one or more machines during the	course of  its
	  lifetime,  such  an  output file might contain several copies	of the
	  results. This	option should be only be used when you wish a  certain
	  file to be treated as	a running log instead of a precise result.

	  This option only applies to standard-universe	jobs.

       buffer_files  <``  name (size,block-size) ; name2 (size,block-size) ...
       '' >

       buffer_size <bytes-in-buffer>

       buffer_block_size <bytes-in-block>

	  HTCondor keeps a buffer of recently-used data	for each  file	a  job
	  accesses.  This  buffer is used both to cache	commonly-used data and
	  to consolidate small reads and writes	into  larger  operations  that
	  get  better  throughput. The default settings	should produce reason-
	  able results for most	programs.

	  These	options	only apply to standard-universe	jobs.

	  If needed, you may set the buffer  controls  individually  for  each
	  file	using  the buffer_files	option.	For example, to	set the	buffer
	  size to 1 MiB	and the	block size to 256 KiB for the file  input.data
	  , use	this command:

       buffer_files = "input.data=(1000000,256000)"

	  Alternatively,  you  may  use	 these	two options to set the default
	  sizes	for all	files used by your job:

       buffer_size = 1000000
       buffer_block_size = 256000

	  If you do not	set these, HTCondor will use the values	given by these
	  two configuration file macros:

       DEFAULT_IO_BUFFER_SIZE =	1000000
       DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000

	  Finally,  if no other	settings are present, HTCondor will use	a buf-
	  fer of 512 KiB and a block size of 32	KiB.

       compress_files =	file1, file2, ...

	  If your job attempts to access any of	the files  mentioned  in  this
	  list,	 HTCondor will automatically compress them (if writing)	or de-
	  compress them	(if reading). The compress format is the same as  used
	  by GNU gzip.

	  The  files  given  in	this list may be simple	file names or complete
	  paths	and may	include	as a wild card.	For example, this list	causes
	  the  file /tmp/data.gz, any file named event.gz, and any file	ending
	  in .gzip to be automatically compressed or decompressed as needed:

       compress_files =	/tmp/data.gz, event.gz,	*.gzip

	  Due to the nature of the compression format, compressed  files  must
	  only	be accessed sequentially. Random access	reading	is allowed but
	  is very slow,	while random access writing is	simply	not  possible.
	  This	restriction  may  be  avoided by using both compress_files and
	  fetch_files at the same time.	When this is done, a file is  kept  in
	  the  decompressed  state at the execution machine, but is compressed
	  for transfer to its original location.

	  This option only applies to standard universe	jobs.

       fetch_files = file1, file2, ...

	  If your job attempts to access a file	mentioned in this list,	HTCon-
	  dor will automatically copy the whole	file to	the executing machine,
	  where	it can be accessed quickly. When your job closes the file,  it
	  will	be  copied  back  to its original location. This list uses the
	  same syntax as compress_files, shown above.

	  This option only applies to standard universe	jobs.

       file_remaps <`` name newname ; name2 newname2 ... ''>

	  Directs HTCondor to use a new	file name in  place  of	 an  old  one.
	  name	describes  a  file name	that your job may attempt to open, and
	  newname describes the	file name it should be replaced	with.  newname
	  may  include	an  optional leading access specifier,	local: or  re-
	  mote:	. If left unspecified, the default access  specifier  is   re-
	  mote:	 .  Multiple remaps can	be specified by	separating each	with a
	  semicolon.

	  This option only applies to standard universe	jobs.

	  If you wish to remap file names that contain equals signs  or	 semi-
	  colons, these	special	characters may be escaped with a backslash.

	  Example One:

	     Suppose that your job reads a file	named  dataset.1 . To instruct
	     HTCondor to force your job	to read	  other.dataset	 instead,  add
	     this to the submit	file:

       file_remaps = "dataset.1=other.dataset"

	  Example Two:

	     Suppose  that your	run many jobs which all	read in	the same large
	     file, called  very.big . If this file can be found	 in  the  same
	     place  on	a  local  disk	in  every  machine  in	the pool, (say
	     /bigdisk/bigfile ,) you can instruct HTCondor  of	this  fact  by
	     remapping	 very.big to  /bigdisk/bigfile and specifying that the
	     file is to	be read	locally, which will be much faster than	 read-
	     ing over the network.

       file_remaps = "very.big = local:/bigdisk/bigfile"

	  Example Three:

	     Several  remaps  can be applied at	once by	separating each	with a
	     semicolon.

       file_remaps  =  "very.big  =  local:/bigdisk/bigfile  ;	 dataset.1   =
       other.dataset"

       local_files = file1, file2, ...

	  If your job attempts to access a file	mentioned in this list,	HTCon-
	  dor will cause it to be read or written at  the  execution  machine.
	  This	is  most useful	for temporary files not	used for input or out-
	  put. This list uses the same syntax as compress_files, shown above.

       local_files = /tmp/*

	  This option only applies to standard universe	jobs.

       want_remote_io =	<True |	False>

	  This option controls how a file is opened and	manipulated in a stan-
	  dard	universe  job.	If  this option	is true, which is the default,
	  then the condor_shadow makes all decisions about how each and	 every
	  file	should	be opened by the executing job.	This entails a network
	  round	trip (or more) from the	job  to	 the  condor_shadow  and  back
	  again	 for every single  open() in addition to other needed informa-
	  tion about the file. If set to false,	then when the job queries  the
	  condor_shadow	 for the first time about how to open a	file, the con-
	  dor_shadow will inform the job to automatically perform all  of  its
	  file	manipulation  on  the local file system	on the execute machine
	  and any file remapping will be ignored. This means that  there  must
	  be a shared file system (such	as NFS or AFS) between the execute ma-
	  chine	and the	submit machine and that	ALL paths that the  job	 could
	  open	on the execute machine must be valid. The ability of the stan-
	  dard universe	job to checkpoint, possibly to a checkpoint server, is
	  not  affected	 by  this  attribute. However, when the	job resumes it
	  will be expecting the	same file system conditions that were  present
	  when the job checkpointed.

       COMMANDS	FOR THE	GRID

       batch_queue = <queuename>

	  Used	for pbs	, lsf ,	and sge	grid universe jobs. Specifies the name
	  of the PBS/LSF/SGE job queue into which the job should be submitted.
	  If not specified, the	default	queue is used.

       boinc_authenticator_file	= <pathname>

	  For  grid type boinc jobs, specifies a path and file name of the au-
	  thorization file that	grants permission  for	HTCondor  to  use  the
	  BOINC	service. There is no default value when	not specified.

       cream_attributes	= <name=value;... ;name=value>

	  Provides  a  list  of	attribute/value	pairs to be set	in a CREAM job
	  description of a grid	universe job destined for the CREAM grid  sys-
	  tem. The pairs are separated by semicolons, and written in New Clas-
	  sAd syntax.

       delegate_job_GSI_credentials_lifetime = <seconds>

	  Specifies the	maximum	number of seconds for which delegated  proxies
	  should be valid. The default behavior	when this command is not spec-
	  ified	 is  determined	  by   the   configuration   variable	 DELE-
	  GATE_JOB_GSI_CREDENTIALS_LIFETIME  ,	which  defaults	 to one	day. A
	  value	of 0 indicates that the	delegated proxy	should be valid	for as
	  long	as  allowed  by	 the credential	used to	create the proxy. This
	  setting currently only applies to  proxies  delegated	 for  non-grid
	  jobs	and for	HTCondor-C jobs. It does not currently apply to	globus
	  grid jobs, which always behave as though this	setting	were  0.  This
	  variable   has  no  effect  if  the  configuration  variable	 DELE-
	  GATE_JOB_GSI_CREDENTIALS is  False , because in that	case  the  job
	  proxy	is copied rather than delegated.

       deltacloud_hardware_profile = <Deltacloud profile name>

	  Used	for deltacloud jobs. An	optional identifier for	the type of VM
	  desired. If not provided, a service-defined default is used.

       deltacloud_hardware_profile_cpu = <cpu details>

	  Used for deltacloud jobs. An optional	description of	the  CPUs  de-
	  sired	for the	VM, overriding the selected hardware profile.

       deltacloud_hardware_profile_memory = <memory details>

	  Used	for  deltacloud	 jobs.	An  optional description of the	memory
	  (RAM)	desired	for the	VM, overriding the selected hardware profile.

       deltacloud_hardware_profile_storage = <storage details>

	  Used for deltacloud jobs. An optional	 description  of  the  storage
	  (disk) desired for the VM, overriding	the selected hardware profile.

       deltacloud_image_id = <Deltacloud image ID>

	  Used for deltacloud jobs. Identifier of the VM image to run.

       deltacloud_keyname = <Deltacloud	key name>

	  Used for deltacloud jobs. Identifier of the SSH key pair that	should
	  be used to allow remote login	to the running instance. The key  pair
	  needs	to be created before submission.

       deltacloud_password_file	= <pathname>

	  Used	for  deltacloud	 jobs. Path and	file name of a file containing
	  the secret key to be used to authenticate with a Deltacloud service.

       deltacloud_realm_id = <Deltacloud realm ID>

	  Used for deltacloud jobs. An optional	identifier specifying which of
	  multiple  locations within a cloud service should be used to run the
	  VM. If not provided, a service-selected default is used.

       deltacloud_user_data = <data>

	  Used for deltacloud jobs. A string, representing  a  block  of  data
	  that	can  be	 accessed  by the virtual machine job inside the cloud
	  service.

       deltacloud_username = <Deltacloud username>

	  Used for deltacloud jobs. The	user name to be	used  to  authenticate
	  with a Deltacloud service.

       ec2_access_key_id = <pathname>

	  For  grid  type  ec2 jobs, identifies	the file containing the	access
	  key.

       ec2_ami_id = <EC2 xMI ID>

	  For grid type	ec2 jobs, identifies the machine image.	Services  com-
	  patible with the EC2 Query API may refer to these with abbreviations
	  other	than  AMI , for	example	 EMI is	valid for Eucalyptus.

       ec2_availability_zone = <zone name>

	  For grid type	ec2 jobs, specifies the	Availability Zone that the in-
	  stance   should   be	run  in.  This	command	 is  optional,	unless
	  ec2_ebs_volumes is set. As an	example,  one  current	zone  is   us-
	  east-1b .

       ec2_block_device_mapping	  =  <block-device>:<kernel-device>,<block-de-
       vice>:<kernel-device>, ...

	  For grid type	ec2 jobs, specifies the	block device to	kernel	device
	  mapping. This	command	is optional.

       ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device name>,...

	  For grid type	ec2 jobs, optionally specifies a list of Elastic Block
	  Store	(EBS) volumes to be made available to the instance and the de-
	  vice names they should have in the instance.

       ec2_elastic_ip =	<elastic IP address>

	  For  grid type ec2 jobs, and optional	specification of an Elastic IP
	  address that should be assigned to this instance.

       ec2_iam_profile_arn = <IAM profile ARN>

	  For grid type	ec2 jobs, an Amazon Resource  Name  (ARN)  identifying
	  which	Identity and Access Management (IAM) (instance)	profile	to as-
	  sociate with the instance.

       ec2_iam_profile_name= <IAM profile name>

	  For grid type	ec2 jobs, a name identifying which Identity and	Access
	  Management (IAM) (instance) profile to associate with	the instance.

       ec2_instance_type = <instance type>

	  For grid type	ec2 jobs, identifies the instance type.	Different ser-
	  vices	may offer different instance types, so	no  default  value  is
	  set.

       ec2_keypair = <ssh key-pair name>

	  For  grid  type ec2 jobs, specifies the name of an SSH key-pair that
	  is already registered	with the EC2 service. The  associated  private
	  key can be used to ssh into the virtual machine once it is running.

       ec2_keypair_file	= <pathname>

	  For grid type	ec2 jobs, specifies the	complete path and file name of
	  a file into which HTCondor will write	an SSH key for	use  with  ec2
	  jobs.	The key	can be used to ssh into	the virtual machine once it is
	  running. If ec2_keypair is specified for a job, ec2_keypair_file  is
	  ignored.

       ec2_parameter_names = ParameterName1, ParameterName2, ...

	  For grid type	ec2 jobs, a space or comma separated list of the names
	  of additional	parameters to pass when	instantiating an instance.

       ec2_parameter_<name>= <value>

	  For grid type	ec2 jobs, specifies the	value for the  correspondingly
	  named	 (instance  instantiation) parameter.  <name> is the parameter
	  name specified in the	submit command ec2_parameter_names , but  with
	  any periods replaced by underscores.

       ec2_secret_access_key = <pathname>

	  For  grid type ec2 jobs, specifies the path and file name containing
	  the secret access key.

       ec2_security_groups = group1, group2, ...

	  For grid type	ec2 jobs, defines the  list  of	 EC2  security	groups
	  which	should be associated with the job.

       ec2_security_ids	= id1, id2, ...

	  For  grid  type ec2 jobs, defines the	list of	EC2 security group IDs
	  which	should be associated with the job.

       ec2_spot_price =	<bid>

	  For grid type	ec2 jobs, specifies the	spot instance  bid,  which  is
	  the  most  that  the job submitter is	willing	to pay per hour	to run
	  this job.

       ec2_tag_names = <name0,name1,name...>

	  For grid type	ec2 jobs, specifies the	case of	tag names that will be
	  associated  with  the	 running instance. This	is only	necessary if a
	  tag name case	matters. By default the	 list  will  be	 automatically
	  generated.

       ec2_tag_<name>= <value>

	  For  grid  type  ec2 jobs, specifies a tag to	be associated with the
	  running  instance.  The  tag	 name	will   be   lower-cased,   use
	  ec2_tag_names	to change the case.

       ec2_user_data = <data>

	  For  grid  type  ec2	jobs, provides a block of data that can	be ac-
	  cessed  by  the  virtual  machine.   If   both   ec2_user_data   and
	  ec2_user_data_file  are  specified for a job,	the two	blocks of data
	  are concatenated, with the data from this ec2_user_data submit  com-
	  mand occurring first.

       ec2_user_data_file = <pathname>

	  For  grid  type  ec2 jobs, specifies a path and file name whose con-
	  tents	can be accessed	by the virtual machine.	If both	 ec2_user_data
	  and  ec2_user_data_file  are	specified for a	job, the two blocks of
	  data are concatenated, with the data from that ec2_user_data	submit
	  command occurring first.

       ec2_vpc_ip = <a.b.c.d>

	  For  grid  type  ec2	jobs, that are part of a Virtual Private Cloud
	  (VPC), an optional specification of the IP  address  that  this  in-
	  stance should	have within the	VPC.

       ec2_vpc_subnet =	<subnet	specification string>

	  For  grid  type  ec2	jobs, an optional specification	of the Virtual
	  Private Cloud	(VPC) that this	instance should	be a part of.

       gce_auth_file = <pathname>

	  For grid type	gce jobs, specifies a path and file name of the	autho-
	  rization  file that grants permission	for HTCondor to	use the	Google
	  account. There is no default value when not specified.

       gce_image = <image id>

	  For grid type	gce jobs, the identifier of the	virtual	machine	 image
	  representing	the HTCondor job to be run. This virtual machine image
	  must already be register with	GCE and	reside in Google's Cloud Stor-
	  age service.

       gce_machine_type	= <machine type>

	  For  grid type gce jobs, the long form of the	URL that describes the
	  machine configuration	that the virtual machine instance  is  to  run
	  on.

       gce_metadata = <name=value,... ,name=value>

	  For  grid  type  gce	jobs, a	comma separated	list of	name and value
	  pairs	that define metadata for a virtual machine instance that is an
	  HTCondor job.

       gce_metadata_file = <pathname>

	  For  grid  type gce jobs, specifies a	path and file name of the file
	  that contains	metadata for a virtual machine instance	that is	an HT-
	  Condor  job. Within the file,	each name and value pair is on its own
	  line;	so, the	pairs are separated by the newline character.

       globus_rematch =	<ClassAd Boolean Expression>

	  This expression is evaluated by the condor_gridmanager whenever:

	     1.	the globus_resubmit expression evaluates to  True

	     2.	the condor_gridmanager decides it needs	to retry a  submission
	     (as  when	a  previous submission failed to commit) If globus_re-
	     match evaluates to	 True ,	then before the	job is submitted again
	     to	 globus,  the  condor_gridmanager  will	 request that the con-
	     dor_schedd	daemon renegotiate with	the matchmaker (the condor_ne-
	     gotiator ). The result is this job	will be	matched	again.

       globus_resubmit = <ClassAd Boolean Expression>

	  The  expression is evaluated by the condor_gridmanager each time the
	  condor_gridmanager gets a job	ad to manage. Therefore,  the  expres-
	  sion is evaluated:

	     1.	when a grid universe job is first submitted to HTCondor-G

	     2.	when a grid universe job is released from the hold state

	     3.	 when HTCondor-G is restarted (specifically, whenever the con-
	     dor_gridmanager is	restarted)  If	the  expression	 evaluates  to
	     True  , then any previous submission to the grid universe will be
	     forgotten and this	job will be submitted again as a fresh submis-
	     sion  to  the grid	universe. This may be useful if	there is a de-
	     sire to give up on	a previous submission and try again. Note that
	     this  may	result	in the same job	running	more than once.	Do not
	     treat this	operation lightly.

       globus_rsl = <RSL-string>

	  Used to provide any additional Globus	RSL  string  attributes	 which
	  are not covered by other submit description file commands or job at-
	  tributes. Used for grid universe jobs, where the grid	resource has a
	  grid-type-string of gt2 .

       grid_resource = <grid-type-string><grid-specific-parameter-list>

	  For  each  grid-type-string  value,  there are further type-specific
	  values that must specified. This submit description file command al-
	  lows	each  to  be  given in a space-separated list. Allowable grid-
	  type-string values are batch , condor	, cream	, deltacloud ,	ec2  ,
	  gt2 ,	gt5 , lsf , nordugrid ,	pbs , sge , and	unicore	. The HTCondor
	  manual chapter on Grid Computing details the variety of grid types.

	  For a	grid-type-string of batch , the	single parameter is  the  name
	  of  the local	batch system, and will be one of  pbs ,	 lsf , or  sge
	  .

	  For a	grid-type-string of condor , the first parameter is  the  name
	  of the remote	condor_schedd daemon. The second parameter is the name
	  of the pool to which the remote condor_schedd	daemon belongs.

	  For a	grid-type-string of cream , there are  three  parameters.  The
	  first	parameter is the web services address of the CREAM server. The
	  second parameter is the name of the batch system  that  sits	behind
	  the  CREAM  server.  The  third parameter identifies a site-specific
	  queue	within the batch system.

	  For a	grid-type-string of deltacloud , the single parameter  is  the
	  URL of the deltacloud	service	requested.

	  For  a  grid-type-string of ec2 , one	additional parameter specifies
	  the EC2 URL.

	  For a	grid-type-string of gt2	, the single parameter is the name  of
	  the pre-WS GRAM resource to be used.

	  For  a grid-type-string of gt5 , the single parameter	is the name of
	  the pre-WS GRAM resource to be used, which is	the same  as  for  the
	  grid-type-string of gt2 .

	  For a	grid-type-string of lsf	, no additional	parameters are used.

	  For  a  grid-type-string  of nordugrid , the single parameter	is the
	  name of the NorduGrid	resource to be used.

	  For a	grid-type-string of pbs	, no additional	parameters are used.

	  For a	grid-type-string of sge	, no additional	parameters are used.

	  For a	grid-type-string of unicore , the first	parameter is the  name
	  of the Unicore Usite to be used. The second parameter	is the name of
	  the Unicore Vsite to be used.

       keystore_alias =	<name>

	  A string to locate the certificate in	a Java keystore	file, as  used
	  for a	unicore	job.

       keystore_file = <pathname>

	  The complete path and	file name of the Java keystore file containing
	  the certificate to be	used for a unicore job.

       keystore_passphrase_file	= <pathname>

	  The  complete	 path  and  file  name	to  the	 file  containing  the
	  passphrase  protecting  a Java keystore file containing the certifi-
	  cate.	Relevant for a unicore job.

       MyProxyCredentialName = <symbolic name>

	  The symbolic name  that  identifies  a  credential  to  the  MyProxy
	  server.  This	 symbolic  name	 is set	as the credential is initially
	  stored on the	server (using myproxy-init ).

       MyProxyHost = <host>:<port>

	  The Internet address of the host that	is  the	 MyProxy  server.  The
	  host may be specified	by either a host name (as in  head.example.com
	  ) or an IP address (of the form 123.456.7.8).	The port number	is  an
	  integer.

       MyProxyNewProxyLifetime = <number-of-minutes>

	  The new lifetime (in minutes)	of the proxy after it is refreshed.

       MyProxyPassword = <password>

	  The  password	 needed	to refresh a credential	on the MyProxy server.
	  This password	is set when the	user initially stores  credentials  on
	  the server (using myproxy-init ). As an alternative to using MyProx-
	  yPassword in the submit description file, the	password may be	speci-
	  fied	as a command line argument to condor_submit with the -password
	  argument.

       MyProxyRefreshThreshold = <number-of-seconds>

	  The time (in seconds)	before the expiration  of  a  proxy  that  the
	  proxy	 should	 be refreshed. For example, if MyProxyRefreshThreshold
	  is set to the	value 600, the proxy will be refreshed 10 minutes  be-
	  fore it expires.

       MyProxyServerDN = <credential subject>

	  A  string that specifies the expected	Distinguished Name (credential
	  subject, abbreviated DN) of the MyProxy server. It must be specified
	  when	the  MyProxy server DN does not	follow the conventional	naming
	  scheme of a host credential. This  occurs,  for  example,  when  the
	  MyProxy server DN begins with	a user credential.

       nordugrid_rsl = <RSL-string>

	  Used	to  provide any	additional RSL string attributes which are not
	  covered by regular submit description	file parameters. Used when the
	  universe is grid , and the type of grid system is nordugrid .

       transfer_error =	<True |	False>

	  For  jobs  submitted	to the grid universe only. If  True , then the
	  error	output (from  stderr ) from the	job is	transferred  from  the
	  remote  machine back to the submit machine. The name of the file af-
	  ter transfer is given	by the error command. If  False	, no  transfer
	  takes	 place	(from  the  remote machine to submit machine), and the
	  name of the file is given by the error command. The default value is
	  True .

       transfer_input =	<True |	False>

	  For  jobs  submitted	to the grid universe only. If  True , then the
	  job input ( stdin ) is transferred from the machine  where  the  job
	  was  submitted  to  the remote machine. The name of the file that is
	  transferred is given by the input command.  If   False  ,  then  the
	  job's	 input	is taken from a	pre-staged file	on the remote machine,
	  and the name of the file is given by the input command. The  default
	  value	is  True .

	  For  transferring files other	than  stdin , see transfer_input_files
	  .

       transfer_output = <True | False>

	  For jobs submitted to	the grid universe only.	If  True  ,  then  the
	  output  (from	  stdout ) from	the job	is transferred from the	remote
	  machine back to the submit machine.  The  name  of  the  file	 after
	  transfer  is	given  by  the output command. If  False , no transfer
	  takes	place (from the	remote machine to  submit  machine),  and  the
	  name	of  the	file is	given by the output command. The default value
	  is  True .

	  For transferring  files  other  than	 stdout	 ,  see	 transfer_out-
	  put_files .

       use_x509userproxy = <True | False>

	  Set this command to  True to indicate	that the job requires an X.509
	  user proxy. If x509userproxy is set, then that file is used for  the
	  proxy. Otherwise, the	proxy is looked	for in the standard locations.
	  If x509userproxy is set or if	the job	is a grid universe job of grid
	  type	gt2  ,	gt5  ,	cream  ,  or  nordugrid	 ,  then  the value of
	  use_x509userproxy is forced to  True . Defaults to  False .

       x509userproxy = <full-pathname>

	  Used to override the default path name for X.509 user	 certificates.
	  The default location for X.509 proxies is the	 /tmp directory, which
	  is generally a local file system. Setting this value would allow HT-
	  Condor  to  access  the  proxy in a shared file system (for example,
	  AFS).	HTCondor will use the proxy specified in the  submit  descrip-
	  tion	file  first. If	nothing	is specified in	the submit description
	  file,	it will	use the	environment variable X509_USER_PROXY. If  that
	  variable is not present, it will search in the default location.

	  x509userproxy	is relevant when the universe is vanilla , or when the
	  universe is grid and the type	of grid	system is one of gt2 ,	gt5  ,
	  condor , cream , or nordugrid	. Defining a value causes the proxy to
	  be delegated to the execute machine. Further,	 VOMS  attributes  de-
	  fined	in the proxy will appear in the	job ClassAd.

       COMMANDS	FOR PARALLEL, JAVA, and	SCHEDULER UNIVERSES

       hold_kill_sig = <signal-number>

	  For  the scheduler universe only, signal-number is the signal	deliv-
	  ered to the job when the job is put on hold with condor_hold .  sig-
	  nal-number  may be either the	platform-specific name or value	of the
	  signal. If this command is not present, the  value  of  kill_sig  is
	  used.

       jar_files = <file_list>

	  Specifies  a	list of	additional JAR files to	include	when using the
	  Java universe. JAR files will	be transferred	along  with  the  exe-
	  cutable and automatically added to the classpath.

       java_vm_args = <argument_list>

	  Specifies a list of additional arguments to the Java VM itself, When
	  HTCondor runs	the Java program, these	are the	arguments that go  be-
	  fore	the  class name. This can be used to set VM-specific arguments
	  like stack size, garbage-collector arguments	and  initial  property
	  values.

       machine_count = <max>

	  For the parallel universe, a single value ( max ) is required. It is
	  neither a maximum or minimum,	but the	number of machines to be dedi-
	  cated	toward running the job.

       remove_kill_sig = <signal-number>

	  For  the scheduler universe only, signal-number is the signal	deliv-
	  ered to the job when the job is removed with	condor_rm  .   signal-
	  number may be	either the platform-specific name or value of the sig-
	  nal. This example shows it both ways for a Linux signal:

       remove_kill_sig = SIGUSR1
       remove_kill_sig = 10

	  If this command is not present, the value of kill_sig	is used.

       COMMANDS	FOR THE	VM UNIVERSE

       vm_disk = file1:device1:permission1, file2:device2:permission2:format2,
       ...

	  A list of comma separated disk files.	Each disk file is specified by
	  4 colon separated fields. The	first field is the path	and file  name
	  of  the  disk	file. The second field specifies the device. The third
	  field	specifies permissions, and the optional	fourth field specifies
	  the image format.

	  An example that specifies two	disk files:

       vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w

       vm_checkpoint = <True | False>

	  A  boolean  value  specifying	whether	or not to take checkpoints. If
	  not specified, the default value is  False . In the  current	imple-
	  mentation,  setting  both  vm_checkpoint  and	vm_networking to  True
	  does not yet work in all cases. Networking cannot be used  if	 a  vm
	  universe  job	uses a checkpoint in order to continue execution after
	  migration to another machine.

       vm_macaddr = <MACAddr>

	  Defines that MAC address that	the virtual machine's  network	inter-
	  face	should have, in	the standard format of six groups of two hexa-
	  decimal digits separated by colons.

       vm_memory = <MBytes-of-memory>

	  The amount of	memory in MBytes that a	vm universe job	requires.

       vm_networking = <True | False>

	  Specifies whether to use networking or not. In the current implemen-
	  tation,  setting  both vm_checkpoint and vm_networking to  True does
	  not yet work in all cases. Networking	cannot be used if  a  vm  uni-
	  verse	job uses a checkpoint in order to continue execution after mi-
	  gration to another machine.

       vm_networking_type = <nat | bridge >

	  When vm_networking is	 True ,	this definition	augments the job's re-
	  quirements  to match only machines with the specified	networking. If
	  not specified, then either networking	type matches.

       vm_no_output_vm = <True | False>

	  When	True , prevents	HTCondor from transferring output  files  back
	  to  the machine from which the vm universe job was submitted.	If not
	  specified, the default value is  False .

       vm_type = <vmware | xen | kvm>

	  Specifies the	underlying virtual machine software that this job  ex-
	  pects.

       vmware_dir = <pathname>

	  The  complete	 path  and name	of the directory where VMware-specific
	  files	and applications such as the VMDK (Virtual Machine  Disk  For-
	  mat) and VMX (Virtual	Machine	Configuration) reside. This command is
	  optional; when not specified,	all relevant VMware image files	are to
	  be listed using transfer_input_files .

       vmware_should_transfer_files = <True | False>

	  Specifies  whether  HTCondor will transfer VMware-specific files lo-
	  cated	as specified by	vmware_dir to the execute machine ( True )  or
	  rely	on  access through a shared file system	( False	). Omission of
	  this required	command	(for VMware vm universe	jobs)  results	in  an
	  error	 message  from condor_submit , and the job will	not be submit-
	  ted.

       vmware_snapshot_disk = <True | False>

	  When	True , causes HTCondor to utilize a VMware snapshot  disk  for
	  new  or modified files. If not specified, the	default	value is  True
	  .

       xen_initrd = <image-file>

	  When xen_kernel gives	a path and file	name for the kernel  image  to
	  use,	this optional command may specify a path to and	ramdisk	( ini-
	  trd )	image file.

       xen_kernel = <included |	path-to-kernel>

	  A value of included specifies	that the kernel	 is  included  in  the
	  disk	file. If not one of these values, then the value is a path and
	  file name of the kernel to be	used.

       xen_kernel_params = <string>

	  A string that	is appended to the Xen kernel command line.

       xen_root	= <string>

	  A string that	is appended to the Xen kernel command line to  specify
	  the  root  device.  This  string is required when xen_kernel gives a
	  path to a kernel. Omission for this required case results in an  er-
	  ror message during submission.

       COMMANDS	FOR THE	DOCKER UNIVERSE

       docker_image = <image-name >

	  Defines  the	name  of  the  Docker  image that is the basis for the
	  docker container.

       ADVANCED	COMMANDS

       accounting_group	= <accounting-group-name>

	  Causes jobs to negotiate under  the  given  accounting  group.  This
	  value	 is advertised in the job ClassAd as  AcctGroup	. The HTCondor
	  Administrator's manual contains more	information  about  accounting
	  groups.

       accounting_group_user = <accounting-group-user-name>

	  Sets the user	name associated	with the accounting group name for re-
	  source usage accounting purposes. If not set,	defaults to the	 value
	  of  the  job	ClassAd	attribute  Owner . This	value is advertised in
	  the job ClassAd as  AcctGroupUser . If an accounting group  has  not
	  been set with	the accounting_group command, this command is ignored.

       concurrency_limits = <string-list>

	  A  list of resources that this job needs. The	resources are presumed
	  to have concurrency limits placed upon them,	thereby	 limiting  the
	  number  of  concurrent  jobs	in  execution which need the named re-
	  source. Commas and space characters delimit the items	in  the	 list.
	  Each	item  in the list is a string that identifies the limit, or it
	  is a ClassAd expression that evaluates to a string, and it is	evalu-
	  ated	in the context of machine ClassAd being	considered as a	match.
	  Each item in the list	also may specify a numerical value identifying
	  the  integer	number	of  resources required for the job. The	syntax
	  follows the resource name by a colon character ( : ) and the numeri-
	  cal  value. Details on concurrency limits are	in the HTCondor	Admin-
	  istrator's manual.

       concurrency_limits_expr = <ClassAd String Expression>

	  A ClassAd expression that represents the list	of resources that this
	  job  needs  after evaluation.	The ClassAd expression may specify ma-
	  chine	ClassAd	attributes that	are evaluated against  a  matched  ma-
	  chine. After evaluation, the list sets concurrency_limits .

       copy_to_spool = <True | False>

	  If copy_to_spool is  True , then condor_submit copies	the executable
	  to the local spool directory before running it on a remote host.  As
	  copying  can	be  quite  time	consuming and unnecessary, the default
	  value	is  False for all job universes	other than the	standard  uni-
	  verse. When  False , condor_submit does not copy the executable to a
	  local	spool directory. The default is	 True  in  standard  universe,
	  because  resuming execution from a checkpoint	can only be guaranteed
	  to work using	precisely the same executable that created the	check-
	  point.

       coresize	= <size>

	  Should  the  user's  program abort and produce a core	file, coresize
	  specifies the	maximum	size in	bytes of the core file which the  user
	  wishes  to  keep.  If	coresize is not	specified in the command file,
	  the system's user resource limit  coredumpsize is used. A  value  of
	  -1 results in	no limits being	applied	to the core file size.

       cron_day_of_month = <Cron-evaluated Day>

	  The  set of days of the month	for which a deferral time applies. The
	  HTCondor User's manual section on Time Scheduling for	Job  Execution
	  has further details.

       cron_day_of_week	= <Cron-evaluated Day>

	  The  set  of days of the week	for which a deferral time applies. The
	  HTCondor User's manual section on Time Scheduling for	Job  Execution
	  has further details.

       cron_hour = <Cron-evaluated Hour>

	  The  set  of hours of	the day	for which a deferral time applies. The
	  HTCondor User's manual section on Time Scheduling for	Job  Execution
	  has further details.

       cron_minute = <Cron-evaluated Minute>

	  The set of minutes within an hour for	which a	deferral time applies.
	  The HTCondor User's manual section on	Time Scheduling	for Job	Execu-
	  tion has further details.

       cron_month = <Cron-evaluated Month>

	  The  set  of months within a year for	which a	deferral time applies.
	  The HTCondor User's manual section on	Time Scheduling	for Job	Execu-
	  tion has further details.

       cron_prep_time =	<ClassAd Integer Expression>

	  Analogous  to	 deferral_prep_time . The number of seconds prior to a
	  job's	deferral time that the job may be matched and sent to an  exe-
	  cution machine.

       cron_window = <ClassAd Integer Expression>

	  Analogous  to	 the  submit  command deferral_window .	It allows cron
	  jobs that miss their deferral	time to	begin execution.

	  The HTCondor User's manual section on	Time Scheduling	for Job	Execu-
	  tion has further details.

       dagman_log = <pathname>

	  DAGMan  inserts this command to specify an event log that it watches
	  to maintain the state	of the DAG. If the log command is  not	speci-
	  fied	in the submit file, DAGMan uses	the log	command	to specify the
	  event	log.

       deferral_prep_time = <ClassAd Integer Expression>

	  The number of	seconds	prior to a job's deferral time	that  the  job
	  may be matched and sent to an	execution machine.

	  The HTCondor User's manual section on	Time Scheduling	for Job	Execu-
	  tion has further details.

       deferral_time = <ClassAd	Integer	Expression>

	  Allows a job to specify the time at which its	execution is to	begin,
	  instead  of  beginning execution as soon as it arrives at the	execu-
	  tion machine.	The deferral time is an	expression that	evaluates to a
	  Unix	Epoch  timestamp (the number of	seconds	elapsed	since 00:00:00
	  on January 1,	1970, Coordinated Universal Time).  Deferral  time  is
	  evaluated  with respect to the execution machine. This option	delays
	  the start of execution, but not the matching and claiming of	a  ma-
	  chine	 for  the  job.	If the job is not available and	ready to begin
	  execution at the deferral time, it has missed	its deferral  time.  A
	  job that misses its deferral time will be put	on hold	in the queue.

	  The HTCondor User's manual section on	Time Scheduling	for Job	Execu-
	  tion has further details.

	  Due to implementation	details, a deferral time may not be  used  for
	  scheduler universe jobs.

       deferral_window = <ClassAd Integer Expression>

	  The  deferral	 window	 is used in conjunction	with the deferral_time
	  command to allow jobs	that miss their	deferral time to begin	execu-
	  tion.

	  The HTCondor User's manual section on	Time Scheduling	for Job	Execu-
	  tion has further details.

       description = <string>

	  A string that	sets the value of the job  ClassAd  attribute	JobDe-
	  scription  .	When  set,  tools which	display	the executable such as
	  condor_q will	instead	use this string.

       email_attributes	= <list-of-job-ad-attributes>

	  A comma-separated list of attributes from the	job ClassAd. These at-
	  tributes  and	 their values will be included in the e-mail notifica-
	  tion of job completion.

       image_size = <size>

	  Advice to HTCondor specifying	the  maximum  virtual  image  size  to
	  which	the job	will grow during its execution.	HTCondor will then ex-
	  ecute	the job	only on	machines which have enough resources, (such as
	  virtual memory), to support executing	the job. If not	specified, HT-
	  Condor will automatically  make  a  (reasonably  accurate)  estimate
	  about	 the  job's size and adjust this estimate as the program runs.
	  If specified and underestimated, the job may crash due  to  the  in-
	  ability  to  acquire	more  address space; for example, if  malloc()
	  fails. If the	image size is overestimated, HTCondor may have	diffi-
	  culty	 finding  machines which have the required resources.  size is
	  specified in KiB. For	example, for an	image  size  of	 8  MiB,  size
	  should be 8000.

       initialdir = <directory-path>

	  Used to give jobs a directory	with respect to	file input and output.
	  Also provides	a directory (on	the machine from which the job is sub-
	  mitted) for the job event log, when a	full path is not specified.

	  For vanilla universe jobs where there	is a shared file system, it is
	  the current working directory	on the machine where the job  is  exe-
	  cuted.

	  For vanilla or grid universe jobs where file transfer	mechanisms are
	  utilized (there is not a shared file system),	it is the directory on
	  the  machine	from  which the	job is submitted where the input files
	  come from, and where the job's output	files go to.

	  For standard universe	jobs, it is the	directory on the machine  from
	  which	 the job is submitted where the	condor_shadow daemon runs; the
	  current working directory for	file  input  and  output  accomplished
	  through remote system	calls.

	  For scheduler	universe jobs, it is the directory on the machine from
	  which	the job	is submitted where the job runs; the  current  working
	  directory  for  file	input and output with respect to relative path
	  names.

	  Note that the	path to	the executable is not relative to initialdir ;
	  if  it  is a relative	path, it is relative to	the directory in which
	  the condor_submit command is run.

       job_ad_information_attrs	= <attribute-list>

	  A comma-separated list of job	ClassAd	attribute names. The named at-
	  tributes  and	their values are written to the	job event log whenever
	  any event is being written to	the  log.  This	 implements  the  same
	  thing	 as  the  configuration	 variable  EVENT_LOG_INFORMATION_ATTRS
	  (see page ), but it applies to the job event	log,  instead  of  the
	  system event log.

       job_lease_duration = <number-of-seconds>

	  For vanilla, parallel, VM, and java universe jobs only, the duration
	  in seconds of	a job lease. The default value is 2,400, or forty min-
	  utes.	If a job lease is not desired, the value can be	explicitly set
	  to 0 to disable the job lease	semantics. The value  can  also	 be  a
	  ClassAd expression that evaluates to an integer. The HTCondor	User's
	  manual section on Special Environment	Considerations has further de-
	  tails.

       job_machine_attrs = <attr1, attr2, ... >

	  A  comma and/or space	separated list of machine attribute names that
	  should be recorded in	the job	ClassAd	in addition to the ones	speci-
	  fied	by  the	 condor_schedd	daemon's system	configuration variable
	  SYSTEM_JOB_MACHINE_ATTRS . When there	 are  multiple	run  attempts,
	  history  of  machine	attributes  from  previous run attempts	may be
	  kept.	The number of run attempts to store may	be extended beyond the
	  system-specified  history  length  by	 using the submit file command
	  job_machine_attrs_history_length . A machine attribute named	X will
	  be  inserted	into the job ClassAd as	an attribute named  MachineAt-
	  trX0 . The previous value of	this  attribute	 will  be  named   Ma-
	  chineAttrX1  ,  the  previous	to that	will be	named  MachineAttrX2 ,
	  and so on, up	to the specified history length. A history of length 1
	  means	 that only  MachineAttrX0 will be recorded. The	value recorded
	  in the job ClassAd is	the evaluation of the machine attribute	in the
	  context  of  the job ClassAd when the	condor_schedd daemon initiates
	  the start up of the job. If the evaluation results in	an   Undefined
	  or   Error  result,  the value recorded in the job ad	will be	 Unde-
	  fined	or  Error , respectively.

       want_graceful_removal = <boolean	expression>

	  When	True , this causes a graceful shutdown of the job when the job
	  is removed or	put on hold, giving it time to clean up	or save	state.
	  Otherwise, the job is	abruptly killed. The default is	 false .

       kill_sig	= <signal-number>

	  When HTCondor	needs to kick a	job off	of a machine, it will send the
	  job  the signal specified by signal-number .	signal-number needs to
	  be an	integer	which represents a valid signal	on the	execution  ma-
	  chine.  For  jobs  submitted	to  the	standard universe, the default
	  value	is the number for  SIGTSTP which tells the HTCondor  libraries
	  to initiate a	checkpoint of the process. For jobs submitted to other
	  universes, the default value,	when not defined, is  SIGTERM ,	 which
	  is the standard way to terminate a program in	Unix.

       kill_sig_timeout	= <seconds>

	  This	submit command should no longer	be used	as of HTCondor version
	  7.7.3; use job_max_vacate_time instead.  If  job_max_vacate_time  is
	  not defined, this defines the	number of seconds that HTCondor	should
	  wait following the sending of	the kill signal	 defined  by  kill_sig
	  and  forcibly	 killing  the  job.  The actual	amount of time between
	  sending the signal and forcibly killing the job is the  smallest  of
	  this	value and the configuration variable  KILLING_TIMEOUT ,	as de-
	  fined	on the execute machine.

       load_profile = <True | False>

	  When	True , loads the account profile of the	dedicated run  account
	  for Windows jobs. May	not be used with run_as_owner .

       match_list_length = <integer value>

	  Defaults  to	the  value zero	(0). When match_list_length is defined
	  with an integer value	greater	than zero (0), attributes are inserted
	  into	the  job  ClassAd. The maximum number of attributes defined is
	  given	by the integer value. The job ClassAds introduced are given as

       LastMatchName0 =	"most-recent-Name"
       LastMatchName1 =	"next-most-recent-Name"

	  The value for	each introduced	ClassAd	is given by the	value  of  the
	  Name	attribute  from	 the  machine  ClassAd of a previous execution
	  (match). As a	job is matched,	the definitions	for  these  attributes
	  will	roll,  with   LastMatchName1 becoming  LastMatchName2 ,	 Last-
	  MatchName0 becoming  LastMatchName1 ,	and  LastMatchName0 being  set
	  by the most recent value of the  Name	attribute.

	  An  intended	use of these job attributes is in the requirements ex-
	  pression. The	requirements can allow a job to	prefer	a  match  with
	  either the same or a different resource than a previous match.

       job_max_vacate_time = <integer expression>

	  An integer-valued expression (in seconds) that may be	used to	adjust
	  the time given to an evicted job for gracefully  shutting  down.  If
	  the  job's setting is	less than the machine's, the job's is used. If
	  the job's setting is larger than the machine's, the  result  depends
	  on  whether  the  job	has any	excess retirement time.	If the job has
	  more retirement time left than the machine's max  vacate  time  set-
	  ting,	 then retirement time will be converted	into vacating time, up
	  to the amount	requested by the job.

	  Setting this expression does not affect the job's resource  require-
	  ments	or preferences.	For a job to only run on a machine with	a min-
	  imum	MachineMaxVacateTime , or to preferentially run	 on  such  ma-
	  chines,  explicitly specify this in the requirements and/or rank ex-
	  pressions.

       max_job_retirement_time = <integer expression>

	  An integer-valued expression (in seconds) that does  nothing	unless
	  the machine that runs	the job	has been configured to provide retire-
	  ment time. Retirement	time is	a grace	period given to	a job to  fin-
	  ish  when a resource claim is	about to be preempted. The default be-
	  havior in many cases is to take as much retirement time as  the  ma-
	  chine	 offers,  so  this  command will rarely	appear in a submit de-
	  scription file.

	  When a resource claim	is to be preempted,  this  expression  in  the
	  submit  file	specifies the maximum run time of the job (in seconds,
	  since	the job	started). This expression has  no  effect,  if	it  is
	  greater  than	 the  maximum  retirement time provided	by the machine
	  policy. If the resource claim	is not preempted, this expression  and
	  the  machine retirement policy are irrelevant. If the	resource claim
	  is preempted the job will be allowed to  run	until  the  retirement
	  time	expires,  at  which  point  it is hard-killed. The job will be
	  soft-killed when it is getting close to the end of retirement	in or-
	  der to give it time to gracefully shut down. The amount of lead-time
	  for soft-killing is determined by the	maximum	vacating time  granted
	  to the job.

	  Standard  universe jobs and any jobs running with nice_user priority
	  have a default max_job_retirement_time of 0, so no  retirement  time
	  is utilized by default. In all other cases, no default value is pro-
	  vided, so the	maximum	amount of retirement time is utilized  by  de-
	  fault.

	  Setting  this	expression does	not affect the job's resource require-
	  ments	or preferences.	For a job to only run on a machine with	a min-
	  imum	 MaxJobRetirementTime  ,  or to	preferentially run on such ma-
	  chines, explicitly specify this in the requirements and/or rank  ex-
	  pressions.

       nice_user = <True | False>

	  Normally, when a machine becomes available to	HTCondor, HTCondor de-
	  cides	which job to run based upon user and job  priorities.  Setting
	  nice_user equal to  True tells HTCondor not to use your regular user
	  priority, but	that this job should  have  last  priority  among  all
	  users	 and  all  jobs. So jobs submitted in this fashion run only on
	  machines which no other non-nice_user	job wants --  a	 true  bottom-
	  feeder  job! This is very handy if a user has	some jobs they wish to
	  run, but do not wish to use resources	that could instead be used  to
	  run  other  people's	HTCondor  jobs.	Jobs submitted in this fashion
	  have	"nice-user." prepended to the owner name when viewed from con-
	  dor_q	or condor_userprio . The default value is  False .

       noop_job	= <ClassAd Boolean Expression>

	  When	this  boolean expression is  True , the	job is immediately re-
	  moved	from the queue,	and HTCondor makes no attempt at  running  the
	  job.	The log	file for the job will show a job submitted event and a
	  job terminated event,	along with an exit code	of 0, unless the  user
	  specifies a different	signal or exit code.

       noop_job_exit_code = <return value>

	  When	noop_job  is  in  the submit description file and evaluates to
	  True , this command allows the job to	specify	the  return  value  as
	  shown	 in the	job's log file job terminated event. If	not specified,
	  the job will show as having terminated with status 0.	This overrides
	  any value specified with noop_job_exit_signal	.

       noop_job_exit_signal = <signal number>

	  When	noop_job  is  in  the submit description file and evaluates to
	  True , this command allows the job to	specify	the signal number that
	  the job's log	event will show	the job	having terminated with.

       remote_initialdir = <directory-path>

	  The  path specifies the directory in which the job is	to be executed
	  on the remote	machine. This is currently supported in	all  universes
	  except for the standard universe.

       rendezvousdir = <directory-path>

	  Used to specify the shared file system directory to be used for file
	  system authentication	when submitting	to a remote scheduler.	Should
	  be a path to a preexisting directory.

       run_as_owner = <True | False>

	  A boolean value that causes the job to be run	under the login	of the
	  submitter, if	supported by the joint configuration of	the submit and
	  execute machines. On Unix platforms, this defaults to	 True ,	and on
	  Windows platforms, it	defaults to  False .  May  not	be  used  with
	  load_profile . See the HTCondor manual Platform-Specific Information
	  chapter for administrative details on	configuring Windows to support
	  this option.

       stack_size = <size in bytes>

	  This	command	applies	only to	Linux platform jobs that are not stan-
	  dard universe	jobs. An integer number	 of  bytes,  representing  the
	  amount  of  stack  space to be allocated for the job.	This value re-
	  places the default allocation	of stack space,	which is unlimited  in
	  size.

       submit_event_notes = <note>

	  A string that	is appended to the submit event	in the job's log file.
	  For DAGMan jobs, the string  DAG Node: and the node's	name is	 auto-
	  matically defined for	submit_event_notes , causing the logged	submit
	  event	to identify the	DAG node job submitted.

       +<attribute>= <value>

	  A line which begins with  a  '+'  (plus)  character  instructs  con-
	  dor_submit  to  insert  the following	attribute into the job ClassAd
	  with the given value .

       PRE AND POST SCRIPTS IMPLEMENTED	WITH SPECIALLY-NAMED ATTRIBUTES

       +PreCmd =  <executable>

	  A vanilla universe job may specify that a script is to be run	on the
	  execute machine before the job, and this is called a prescript. Def-
	  inition of this  specifically-named  attribute  causes  the  script,
	  identified  by  path	and  file  name, to be executed. The prescript
	  could	prepare	or initialize the job. Note that this definition of  a
	  prescript  is	different from the PRE script described	in DAGMan. The
	  prescript is not automatically transferred with the job, as the main
	  executable  is,  so it must be entered into the transfer_input_files
	  list,	when file transfer is enabled.

       +PreArgs	=  <argument_list>

	  Defines command line arguments for the prescript, presuming the  Old
	  argument syntax.

       +PreArguments =	<argument_list>

	  Defines  command line	arguments for the prescript, presuming the New
	  argument syntax. An exception	to the syntax is  that	double	quotes
	  must be escaped with a backslash instead of another double quote.

	  Note	that  if  both	+PreArgs  and +PreArguments are	specified, the
	  +PreArguments	value is used and the +PreArgs value is	ignored.

       +PreEnv =  <environment_vars>

	  Defines the environment for the prescript, presuming the  Old	 envi-
	  ronment syntax.

       +PreEnvironment =  <environment_vars>

	  Defines  the	environment for	the prescript, presuming the New envi-
	  ronment syntax.

	  Note that if both +PreEnv and	 +PreEnvironment  are  specified,  the
	  +PreEnvironment value	is used	and the	+PreEnv	value is ignored.

       +PostCmd	=  <executable>

	  A vanilla universe job may specify that a script is to be run	on the
	  execute machine after	the job	exits, and  this  is  called  a	 post-
	  script.  Definition  of this specifically-named attribute causes the
	  script, identified by	path and file name, to be executed. The	 post-
	  script  is run if the	job exits, but not if the job is evicted. Note
	  that this definition of a postscript	is  different  from  the  POST
	  script  described  in	 DAGMan.  The  postscript is not automatically
	  transferred with the job, as the main	executable is, so it  must  be
	  entered  into	 the  transfer_input_files list, when file transfer is
	  enabled.

       +PostArgs =  <argument_list>

	  Defines command line arguments for the postscript, presuming the Old
	  argument syntax.

       +PostArguments =	 <argument_list>

	  Defines command line arguments for the postscript, presuming the New
	  argument syntax. An exception	to the syntax is  that	double	quotes
	  must	be  escaped  with  a backslash instead of another double quote
	  mark.

	  Note that if both +PostArgs and +PostArguments  are  specified,  the
	  +PostArguments value is used and the +PostArgs value is ignored.

       +PostEnv	=  <environment_vars>

	  Defines  the environment for the postscript, presuming the Old envi-
	  ronment syntax.

       +PostEnvironment	=  <environment_vars>

	  Defines the environment for the postscript, presuming	the New	 envi-
	  ronment syntax.

	  Note	that  if both +PostEnv and +PostEnvironment are	specified, the
	  +PostEnvironment value is used and the +PostEnv value	is ignored.

       If any of the prescript or postscript values are	not enclosed in	double
       quotes, they are	silently ignored.

       Below is	an example of the use of starter pre and post scripts:

       +PreCmd = "my_pre"
       +PreArgs	= "pre\"arg1 prea'rg2"
       +PreEnv = "one=1;two=\"2\""
       +PostCmd	= "my_post"
       +PostArguments =	"post\"arg1 'post''ar g2'"
       +PostEnvironment	= "one=1 two=\"2\""

       For  this example  PreArgs generates a first argument of	 pre"a1" and a
       second argument of  pre'a2 .  PostArguments generates a first  argument
       of  post"a1 and a second	argument of  post'a 2 .

       MACROS AND COMMENTS

       In addition to commands,	the submit description file can	contain	macros
       and comments.

       Macros

	  Parameterless	macros in the form  of	 $(macro_name:default  initial
	  value)  may be used anywhere in HTCondor submit description files to
	  provide textual substitution at submit time. Macros can  be  defined
	  by lines in the form of

	       <macro_name> = <string>

	  Two  pre-defined  macros are supplied	by the submit description file
	  parser. The  $(Cluster) or  $(ClusterId) macro supplies the value of
	  the  ClusterId job ClassAd attribute,	and the	 $(Process) or	$(Pro-
	  cId) macro supplies the value	of the	ProcId job ClassAd  attribute.
	  These	 macros	are intended to	aid in the specification of input/out-
	  put files, arguments,	etc., for clusters with	lots of	 jobs,	and/or
	  could	be used	to supply an HTCondor process with its own cluster and
	  process numbers on the command line.

	  The  $(Node) macro is	defined	for parallel universe jobs, and	is es-
	  pecially  relevant  for  MPI	applications. It is a unique value as-
	  signed for the duration of the job that essentially  identifies  the
	  machine  (slot)  on  which  a	 program is executing. Values assigned
	  start	at 0 and increase monotonically. The values  are  assigned  as
	  the parallel job is about to start.

	  Recursive  definition	 of  macros is permitted. An example of	a con-
	  struction that works is the following:

       foo = bar
       foo =  snap $(foo)

	  As a result,	foo = snap bar .

	  Note that both left- and right- recursion works, so

       foo = bar
       foo =  $(foo) snap

	  has as its result  foo = bar snap .

	  The construction

       foo = $(foo) bar

	  by itself will not work, as it does not have an initial  base	 case.
	  Mutually recursive constructions such	as:

       B = bar
       C = $(B)
       B = $(C)	boo

	  will not work, and will fill memory with expansions.

	  A  default value may be specified, for use if	the macro has no defi-
	  nition. Consider the example

       D = $(E:24)

	  Where	 E is not defined within the submit description	file, the  de-
	  fault	value 24 is used, resulting in

       D = 24

	  This	is of limited value, as	the scope of macro substitution	is the
	  submit description file. Thus, either	the macro is or	is not defined
	  within  the  submit  description file. If the	macro is defined, then
	  the default value is useless.	If the	macro  is  not	defined,  then
	  there	is no point in using it	in a submit command.

	  To  use  the dollar sign character ( $ ) as a	literal, without macro
	  expansion, use

       $(DOLLAR)

	  In addition to the normal macro, there is also  a  special  kind  of
	  macro	 called	a substitution macro that allows the substitution of a
	  machine ClassAd attribute value defined on the resource machine  it-
	  self	(gotten	 after a match to the machine has been made) into spe-
	  cific	commands within	the submit description file. The  substitution
	  macro	is of the form:

       $$(attribute)

	  As  this form	of the substitution macro is only evaluated within the
	  context of the machine ClassAd, use of  a  scope  resolution	prefix
	  TARGET. or  MY. is not allowed.

	  A  common use	of this	form of	the substitution macro is for the het-
	  erogeneous submission	of an executable:

       executable = povray.$$(OpSys).$$(Arch)

	  Values for the  OpSys	and  Arch attributes are substituted at	 match
	  time	for  any given resource. This example allows HTCondor to auto-
	  matically choose the correct executable for the matched machine.

	  An extension to the syntax of	the substitution macro provides	an al-
	  ternative  string to use if the machine attribute within the substi-
	  tution macro is undefined. The syntax	appears	as:

       $$(attribute:string_if_attribute_undefined)

	  An example using this	extended syntax	provides a path	name to	a  re-
	  quired  input	 file. Since the file can be placed in different loca-
	  tions	on different machines, the file's path name is given as	an ar-
	  gument to the	program.

       arguments = $$(input_file_path:/usr/foo)

	  On  the  machine,  if	the attribute  input_file_path is not defined,
	  then the path	 /usr/foo is used instead.

	  A further extension to the syntax of the substitution	 macro	allows
	  the  evaluation of a ClassAd expression to define the	value. In this
	  form,	the expression may refer to machine  attributes	 by  prefacing
	  them	with  the  TARGET. scope resolution prefix. To place a ClassAd
	  expression into the substitution macro, square brackets are added to
	  delimit the expression. The syntax appears as:

       $$([ClassAd expression])

	  An  example  of a job	that uses this syntax may be one that wants to
	  know how much	memory it can use. The application cannot detect  this
	  itself,  as  it  would potentially use all of	the memory on a	multi-
	  slot machine.	So the job determines the memory per slot, reducing it
	  by  10%  to account for miscellaneous	overhead, and passes this as a
	  command line argument	to the application. In the submit  description
	  file will be

       arguments = --memory $$([TARGET.Memory *	0.9])

	  To insert two	dollar sign characters ( $$ ) as literals into a Clas-
	  sAd string, use

       $$(DOLLARDOLLAR)

	  The environment macro, $ENV, allows the evaluation of	an environment
	  variable  to	be  used in setting a submit description file command.
	  The syntax used is

       $ENV(variable)

	  An example submit description	file command that uses this  function-
	  ality	 evaluates  the	submitter's home directory in order to set the
	  path and file	name of	a log file:

       log = $ENV(HOME)/jobs/logfile

	  The environment variable is evaluated	when  the  submit  description
	  file is processed.

	  The  $RANDOM_CHOICE  macro  allows a random choice to	be made	from a
	  given	list of	parameters at submission time. For an  expression,  if
	  some randomness needs	to be generated, the macro may appear as

	   $RANDOM_CHOICE(0,1,2,3,4,5,6)

	  When evaluated, one of the parameters	values will be chosen.

       Comments

	  Blank	 lines	and  lines beginning with a pound sign ('#') character
	  are ignored by the submit description	file parser.

Exit Status
       condor_submit will exit with a status value of 0	(zero)	upon  success,
       and a non-zero value upon failure.

Examples
	  *  Submit Description	File Example 1:	This example queues three jobs
	  for execution	by HTCondor. The first will be given command line  ar-
	  guments  of  15  and 2000 , and it will write	its standard output to
	  foo.out1 . The second	will be	given command line arguments of	30 and
	  2000	,  and	it will	write its standard output to  foo.out2 . Simi-
	  larly	the third will have arguments of 45 and	6000 , and it will use
	  foo.out3  for	 its  standard	output.	Standard error output (if any)
	  from all three programs will appear in  foo.error .

	     ####################
	     #
	     # submit description file
	     # Example 1: queuing multiple jobs	with differing
	     # command line arguments and output files.
	     #
	     ####################

	     Executable	    = foo
	     Universe	    = vanilla

	     Arguments	    = 15 2000
	     Output  = foo.out0
	     Error   = foo.err0
	     Queue

	     Arguments	    = 30 2000
	     Output  = foo.out1
	     Error   = foo.err1
	     Queue

	     Arguments	    = 45 6000
	     Output  = foo.out2
	     Error   = foo.err2
	     Queue

	  Or you can get the same results as the above submit file by using  a
	  list of arguments with the Queue statement

	     ####################
	     #
	     # submit description file
	     # Example 1b: queuing multiple jobs with differing
	     # command line arguments and output files,	alternate syntax
	     #
	     ####################

	     Executable	    = foo
	     Universe	    = vanilla

	     # generate	different output and error filenames for each process
	     Output  = foo.out$(Process)
	     Error   = foo.err$(Process)

	     Queue Arguments From (
	       15 2000
	       30 2000
	       45 6000
	     )

	  *  Submit  Description  File Example 2: This submit description file
	  example queues 150 runs of program foo which must have been compiled
	  and  linked for an Intel x86 processor running RHEL 3. HTCondor will
	  not attempt to run the processes on machines which have less than 32
	  Megabytes of physical	memory,	and it will run	them on	machines which
	  have at least	64 Megabytes, if such machines are  available.	Stdin,
	  stdout, and stderr will refer	to  in.0 ,  out.0 , and	 err.0 for the
	  first	run of this program (process 0).  Stdin,  stdout,  and	stderr
	  will	refer  to   in.1  ,   out.1 , and  err.1 for process 1,	and so
	  forth. A log file containing entries about where and	when  HTCondor
	  runs,	takes checkpoints, and migrates	processes in this cluster will
	  be written into file	foo.log	.

	     ####################
	     #
	     # Example 2: Show off some	fancy features including
	     # use of pre-defined macros and logging.
	     #
	     ####################

	     Executable	    = foo
	     Universe	    = standard
	     Requirements   = OpSys == "LINUX" && Arch =="INTEL"
	     Rank	    = Memory >=	64
	     Request_Memory = 32 Mb
	     Image_Size	    = 28 Mb

	     Error   = err.$(Process)
	     Input   = in.$(Process)
	     Output  = out.$(Process)
	     Log = foo.log
	     Queue 150

	  * Submit Description	File  Example  3:  This	 example  targets  the
	  /bin/sleep  program to run only on a platform	running	a RHEL 6 oper-
	  ating	system.	The example presumes that the pool  contains  machines
	  running  more	than one version of Linux, and this job	needs the par-
	  ticular operating system to run correctly.

	     ####################
	     #
	     # Example 3: Run on a RedHat 6 machine
	     #
	     ####################
	     Universe	  = vanilla
	     Executable	  = /bin/sleep
	     Arguments	  = 30
	     Requirements = (OpSysAndVer == "RedHat6")

	     Error   = err.$(Process)
	     Input   = in.$(Process)
	     Output  = out.$(Process)
	     Log     = sleep.log
	     Queue

	  * Command Line example: The following	command	uses the  -append  op-
	  tion to add two commands before the job(s) is	queued.	A log file and
	  an error log file are	specified. The submit description file is  un-
	  changed.

       condor_submit -a	"log = out.log"	-a "error = error.log" mysubmitfile

	  Note that each of the	added commands is contained within quote marks
	  because there	are space characters within the	command.

	  *  periodic_remove example: A	job should be removed from the	queue,
	  if the total suspension time of the job is more than half of the run
	  time of the job.

	  Including the	command

	  periodic_remove = CumulativeSuspensionTime >
			    ((RemoteWallClockTime -  CumulativeSuspensionTime)
       / 2.0)

	  in the submit	description file causes	this to	happen.

General	Remarks
	  * For	security reasons, HTCondor will	refuse to run any jobs submit-
	  ted by user root (UID	= 0) or	by a user whose	default	group is group
	  wheel	 (GID  =  0). Jobs submitted by	user root or a user with a de-
	  fault	group of wheel will appear to sit forever in the queue	in  an
	  idle state.

	  *  All  path	names specified	in the submit description file must be
	  less than 256	characters in length, and command line arguments  must
	  be  less  than  4096	characters in length; otherwise, condor_submit
	  gives	a warning message but the jobs will not	execute	properly.

	  * Somewhat understandably, behavior gets bizarre if the  user	 makes
	  the  mistake	of  requesting	multiple HTCondor jobs to write	to the
	  same file, and/or if the user	alters any files that need to  be  ac-
	  cessed  by an	HTCondor job which is still in the queue. For example,
	  the compressing of data or output files before an HTCondor  job  has
	  completed is a common	mistake.

	  *  To	 disable checkpointing for Standard Universe jobs, include the
	  line:

	     +WantCheckpoint = False

	  in the submit	description file before	the queue command(s).

See Also
       HTCondor	User Manual

Author
       Center for High Throughput Computing, University	of Wisconsin-Madison

Copyright
       Copyright (C) 1990-2015 Center for High Throughput Computing,  Computer
       Sciences	 Department, University	of Wisconsin-Madison, Madison, WI. All
       Rights Reserved.	Licensed under the Apache License, Version 2.0.

				     date      just-man-pages/condor_submit(1)

Name | Synopsis | Description | Options | Submit Description File Commands | Exit Status | Examples | General Remarks | See Also | Author | Copyright

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=condor_submit&sektion=1&manpath=FreeBSD+12.1-RELEASE+and+Ports>

home | help