Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
cgroup.conf(5)		   Slurm Configuration File		cgroup.conf(5)

       cgroup.conf - Slurm configuration file for the cgroup support

       cgroup.conf  is	an ASCII file which defines parameters used by Slurm's
       Linux cgroup related plugins.  The file location	 can  be  modified  at
       system  build  time using the DEFAULT_SLURM_CONF	parameter or at	execu-
       tion time by setting the	SLURM_CONF environment variable. The file will
       always be located in the	same directory as the slurm.conf file.

       Parameter  names	are case insensitive.  Any text	following a "#"	in the
       configuration file is treated as	a comment  through  the	 end  of  that
       line.   Changes	to  the	configuration file take	effect upon restart of
       Slurm daemons, daemon receipt of	the SIGHUP signal, or execution	of the
       command "scontrol reconfigure" unless otherwise noted.

       For  general  Slurm  Cgroups  information,  see	the  Cgroups  Guide at

       The following cgroup.conf parameters are	defined	to control the general
       behavior	of Slurm cgroup	plugins.

	      Slurm cgroup plugins require valid and functional	cgroup subsys-
	      tem to be	mounted	under  /sys/fs/cgroup/<subsystem_name>.	  When
	      launched,	 plugins  check	 their	subsystem availability.	If not
	      available, the plugin launch fails unless	CgroupAutomount	is set
	      to yes. In that case, the	plugin will first try to mount the re-
	      quired subsystems.

	      Specify the PATH under which cgroups  should  be	mounted.  This
	      should  be  a  writable  directory  which	 will  contain cgroups
	      mounted one per subsystem. The default PATH is /sys/fs/cgroup.

       The following cgroup.conf parameters are	defined	to control the	behav-
       ior of this particular plugin:

	      Constrain	the job	cgroup kernel memory to	this amount of the al-
	      located memory, specified	in bytes. The AllowedKmemSpace must be
	      between the upper	and lower memory limits, specified by MaxKmem-
	      Percent and MinKmemSpace,	respectively. If AllowedKmemSpace goes
	      beyond  the upper	or lower limit,	it will	be reset to that upper
	      or lower limit, whichever	has been exceeded.

	      Constrain	the job/step cgroup RAM	to this	percentage of the  al-
	      located  memory.	 The  percentage  supplied may be expressed as
	      floating point number, e.g. 101.5.  Sets the cgroup soft	memory
	      limit  at	 the  allocated	memory size and	then sets the job/step
	      hard memory limit	at the (AllowedRAMSpace/100) * allocated  mem-
	      ory. If the job/step exceeds the hard limit, then	it might trig-
	      ger Out Of Memory	(OOM) events (including	oom-kill)  which  will
	      be logged	to kernel log ringbuffer (dmesg	in Linux). Setting Al-
	      lowedRAMSpace above 100 may cause	system	Out  of	 Memory	 (OOM)
	      events  as  it allows job/step to	allocate more memory than con-
	      figured to the nodes.  Reducing configured node available	memory
	      to  avoid	 system	 OOM  events  is  suggested.   Setting Allowe-
	      dRAMSpace	below 100 will result in jobs  receiving  less	memory
	      than  allocated and soft memory limit will set to	the same value
	      as the hard limit.  Also	see  ConstrainRAMSpace.	  The  default
	      value is 100.

	      Constrain	 the  job  cgroup swap space to	this percentage	of the
	      allocated	memory.	 The default value  is	0,  which  means  that
	      RAM+Swap	will  be limited to AllowedRAMSpace. The supplied per-
	      centage may be expressed as a floating point number, e.g.	 50.5.
	      If  the  limit  is  exceeded, the	job steps will be killed and a
	      warning message will be written to  standard  error.   Also  see
	      ConstrainSwapSpace.   NOTE:  Setting  AllowedSwapSpace to	0 does
	      not restrict the Linux kernel from using swap space. To  control
	      how the kernel uses swap space, see MemorySwappiness.

	      If  configured to	"yes" then constrain allowed cores to the sub-
	      set of allocated resources. This functionality makes use of  the
	      cpuset  subsystem.   Due	to  a  bug  fixed in version 1.11.5 of
	      HWLOC, the task/affinity plugin may be required in  addition  to
	      task/cgroup for this to function properly.  The default value is

	      If configured to "yes" then constrain the	job's allowed  devices
	      based on GRES allocated resources. It uses the devices subsystem
	      for that.	 The default value is "no".

	      If configured to "yes" then constrain the	job's Kmem  RAM	 usage
	      in addition to RAM usage.	Only takes effect if ConstrainRAMSpace
	      is set to	"yes". The default value is "no". If set to  yes,  the
	      job's  Kmem limit	will be	set to AllowedKmemSpace	if set;	other-
	      wise, the	job's Kmem limit will be set to	its RAM	 limit.	  Also
	      see AllowedKmemSpace.

	      If  configured  to  "yes"	 then constrain	the job's RAM usage by
	      setting the memory soft limit to the allocated  memory  and  the
	      hard  limit  to the allocated memory * AllowedRAMSpace.  The de-
	      fault value is "no", in which case the job's RAM limit  will  be
	      set  to  its  swap  space	 limit if ConstrainSwapSpace is	set to
	      "yes".  Also  see	 AllowedSwapSpace,  AllowedRAMSpace  and  Con-
	      strainSwapSpace.	NOTE: When enabled, ConstrainRAMSpace can lead
	      to a noticeable decline in per-node job throughout.  Sites  with
	      high-throughput requirements should carefully weigh the tradeoff
	      between per-node throughput, versus potential problems that  can
	      arise   from   unconstrained  memory  usage  on  the  node.  See
	      <>	 for   further

	      If  configured  to "yes" then constrain the job's	swap space us-
	      age.  The	default	value is "no". Note that when set to "yes" and
	      ConstrainRAMSpace	 is  set to "no", AllowedRAMSpace is automati-
	      cally set	to 100%	in order to limit the RAM+Swap amount to  100%
	      of  job's	 requirement  plus  the	percent	of allowed swap	space.
	      This amount is thus set to both RAM and  RAM+Swap	 limits.  This
	      means  that  in that particular case, ConstrainRAMSpace is auto-
	      matically	enabled	with the same limit than the one used to  con-
	      strain swap space.  Also see AllowedSwapSpace.

	      Set an upper bound in percent of total RAM on the	RAM constraint
	      for a job.  This will be the memory constraint applied  to  jobs
	      that  are	not explicitly allocated memory	by Slurm (i.e. Slurm's
	      select plugin is not configured to manage	 memory	 allocations).
	      The  PERCENT  may	be an arbitrary	floating point number. The de-
	      fault value is 100.

	      Set an upper bound (in percent of	total RAM) on  the  amount  of
	      RAM+Swap that may	be used	for a job. This	will be	the swap limit
	      applied to jobs on systems where memory is not being  explicitly
	      allocated	to job.	The PERCENT may	be an arbitrary	floating point
	      number between 0 and 100.	 The default value is 100.

	      Set an upper bound in percent of total Kmem for a	job. The  PER-
	      CENT  may	 be  an	 arbitrary  floating point number. The default
	      value is 100.

	      Configure	the kernel's priority for swapping out anonymous pages
	      (such  as	 program  data)	 verses	 file  cache pages for the job
	      cgroup. Valid values are between 0 and 100, inclusive.  A	 value
	      of 0 prevents the	kernel from swapping out program data. A value
	      of 100 gives equal priorioty  to	swapping  out  file  cache  or
	      anonymous	 pages.	 If not	set, then the kernel's default swappi-
	      ness value will be used. Either ConstrainRAMSpace	or  Constrain-
	      SwapSpace	 must  be set to yes in	order for this parameter to be

	      Set a lower bound	(in MB)	on the memory limits  defined  by  Al-
	      lowedKmemSpace. The default limit	is 30M.

	      Set  a  lower  bound (in MB) on the memory limits	defined	by Al-
	      lowedRAMSpace and	AllowedSwapSpace. This	prevents  accidentally
	      creating	a  memory cgroup with such a low limit that slurmstepd
	      is immediately killed due	to lack	of RAM.	The default  limit  is

	      If  configured to	"yes" then set a default task affinity to bind
	      each step	 task  to  a  subset  of  the  allocated  cores	 using
	      sched_setaffinity.   The default value is	"no".  Note: This fea-
	      ture requires the	Portable Hardware Locality (hwloc) library  to
	      be installed.

       Debian  and  derivatives	 (e.g.	Ubuntu)	usually	exclude	the memory and
       memsw (swap) cgroups by default.	To include them, add the following pa-
       rameters	to the kernel command line: cgroup_enable=memory swapaccount=1

       This  can  usually  be placed in	/etc/default/grub inside the GRUB_CMD-
       LINE_LINUX variable. A command such as update-grub must	be  run	 after
       updating	the file.

       # Slurm cgroup support configuration file

       Copyright (C) 2010-2012 Lawrence	Livermore National Security.  Produced
       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
       Copyright (C) 2010-2016 SchedMD LLC.

       This file is part of Slurm, a resource  management  program.   For  de-
       tails, see <>.

       Slurm  is free software;	you can	redistribute it	and/or modify it under
       the terms of the	GNU General Public License as published	 by  the  Free
       Software	 Foundation;  either version 2 of the License, or (at your op-
       tion) any later version.

       Slurm is	distributed in the hope	that it	will be	 useful,  but  WITHOUT
       ANY  WARRANTY;  without even the	implied	warranty of MERCHANTABILITY or
       FITNESS FOR A PARTICULAR	PURPOSE.  See the GNU General  Public  License
       for more	details.


December 2016		   Slurm Configuration File		cgroup.conf(5)


Want to link to this manual page? Use this URL:

home | help