Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
CHYVES(8)		FreeBSD	System Manager's Manual		     CHYVES(8)

NAME
       chyves -	bhyve(8) front end manager -- version 0.2.0

SYNOPSIS
       o   chyves dataset <pool> {install|upgrade}

       o   chyves dev

       o   chyves {firmware|iso} list

       o   chyves {firmware|iso} import	{http-URL|ftp-URL|local-path}

       o   chyves {firmware|iso} rename	<resource> <new-resource-name>

       o   chyves {firmware|iso} delete	<resource>

       o   chyves <guest> clone	<clonenames>|MG	[-ce|-cu|-ie|-iu] [<pool>]

       o   chyves <guest> console

       o   chyves <guest>|MG|all console {reset|tmux|vnc}

       o   chyves <guest>|MG create [<size>] [<pool>]

       o   chyves <guest>|MG|all delete	[force|keepnet]

       o   chyves <guest> disk add [<size>]

       o   chyves <guest> disk disk{n} {description|notes} <annotation>

       o   chyves <guest> disk delete disk{n}

       o   chyves <guest> disk resize disk{n} <new-size>

       o   chyves <guest> disk list

       o   chyves <guest> get {<property>|all}

       o   chyves <guest>|MG|all reclaim

       o   chyves <guest> rename <new-guest-name>

       o   chyves <guest>|MG|global|defaults|all set <prop1>=<value>...

       o   chyves <guest> snapshot [<@snapshotname>]

       o   chyves <guest> snapshot list

       o   chyves <guest> snapshot delete <@snapshotname>

       o   chyves <guest> snapshot rollback [<@snapshotname>]

       o   chyves <guest>|MG|all start [<iso>]

       o   chyves <guest>|MG|all stop [force]

       o   chyves <guest>|MG|all upgrade

       o   chyves <guest>|MG|all unset <property>

       o   chyves help

       o   chyves info [-zbprvstcudnakwl|-h]

       o   chyves   list   [bridges|clones|defaults|disks|firmware|iso|pools|\
	   properties|<property>]

       o   chyves list global [<pool>|primary]

       o   chyves list {processes|snapshots} [<guest>]

       o   chyves list tap active

       o   chyves network <guest> add {tap|tap{n}|vale{n}}

       o   chyves network <guest> add {tap|tap{n}} [bridge{n}]

       o   chyves network <guest> remove {tap{n}|vale{n}}

       o   chyves network bridge{n} attach {vlan-iface{n}|physical-iface{n}}

       o   chyves network bridge{n} {default|private}

       o   chyves network bridge{n} {join|unjoin} tap{n}

       o   chyves network bridge{n} migrate bridge{n}

       o   chyves upgrade [master|dev|check]

       o   chyves version

SYNTAX NOMENCLATURE:
       The following syntax nomenclature is used throughout this page:

       o   subcommand -	Required text, exactly as shown.

       o   [subcommand]	- Optional text, exactly as shown.

       o   <user-input>	- User supplied	input field. Required when not contain
	   in [	].

       o   {n} - A whole number. Context determines valid range.

       o   ...	-  Repeating  as  many iterations as desired, follows the same
	   preceding syntax pattern.

       o   | - Separates options in a list.

       o   [optional|list] - An	optional field,	 list  of  valid  options.  ""
	   meaning literally no	input is also an option	with these lists.

       o   {require|list} - A required field, list of valid options.

       o   $null - '', meaning literally no input.

       o   [-abcdefg]  -  An optional flag field but must start	with a '-' and
	   followed by any combination,	in any order. Eg. '-gca'

       o   <guest> - chyves guest name

       o   <pool> - ZFS	pool. The word "primary" can be	used  to  specify  the
	   primary pool.

       o   <size>  -  Whole  number  with  a  size  suffix  in	kilobytes (K),
	   megabytes (M), gigabytes (G), or terabytes (T).

       o   <resource> -	ISO or UEFI firmware resource name

       o   MG -	Multiple guests	can be given using commas. The word "all"  can
	   be used to indicate all guests for these commands.

DESCRIPTION
       chyves(8)  relies  on the FreeBSD hypervisor bhyve(8), zfs(8), nmdm(4),
       Bourne shell sh(1), and cu(1) to	start and manage  type	2  virtualized
       guests.	Optionally  grub2-bhyve	 and tmux(1) can be used to expand the
       capabilities of chyves. chyves uses configuration files,	ZFS filesystem
       datasets	 and  volumes  to store	guest, chyves settings,	and chyves re-
       sources in an organized hierarchy. bhyve	uses virtio drivers built into
       the  GENERIC  kernel for	network	virtualization.	The virtio drivers are
       used to paravirtualize I/O for disk and network access.

       For networking, a guest is presented a PCI Ethernet device through  the
       use  of	virtio(4). On FreeBSD guests this means	the vtnet(4) driver is
       used. After a certain commit, Intel e1000 emulation was added as	an al-
       ternative  to  virtio-net.  See guest property 'bhyve_net_type' for de-
       tails. In either	case, host network communication is through  a	tap(3)
       or vale(4) interface to the guest.

       For  storage,  guests are provided block	storage	device(s) via ZFS vol-
       umes from the host by default. ZFS datasets are used to organize	 these
       block  devices and other	resources in a hierarchical structure. See ZFS
       section for more	information.

       Multi-guest [MG]	support	is the ability to specify multiple guest names
       that  are  comma	 separate for a	sub-command. Not all sub-commands have
       this functionality but the ones that do,	allow for rapid	 execution  of
       the same	command	over many guests, this can be incredibly helpful for a
       fleet of	VMs. Commands in the synopsis above with the "guest|MG"	syntax
       indicate	 compatibility	with multi-guest. Using	the word 'all' instead
       of guest	will specify all active	guests on  the	system	for  supported
       commands.

DEPENDENCIES
       bhyve(8)	and chyves(8) will run on a base FreeBSD installation. However
       the required kernel modules are not loaded by default  on  FreeBSD.  By
       default	chyves	will  attempt to load the required kernel modules when
       needed. There are also certain non-FreeBSD-base binaries	 that  enhance
       chyves  capabilities.  These  binaries  are  checked for	and in certain
       cases, exits if the required component is not detected.

       At least	one ZFS	pool is	required. All data is stored on	 one  or  many
       ZFS  pools and depends on ZFS volumes for guest's disks backing.	chyves
       capitalizes on multi-pool configurations. Multiple pool	configurations
       are  common  in	FreeNAS	deployments, separated storage types (eg: SSD,
       HD, NVMe), and OS plus storage configurations.

   Kernel modules:
       o   vmm - Required to run the guests as this allocates the resources.

       o   nmdm(4) - Required for serial console access	to the guests.

       o   if_tap(4) - Required	for network access to the guests.

       o   if_bridge(4)	- Required for network access to the guests.

       o   bridgestp - Required	by if_bridge.

       o   netmap(4) - Required	for VALE support. Not compiled into GENERIC on
	   10.3.

   Enhancing Binaries:
       o   bhyve  Custom  variants  required  for  VALE	 or  UEFI  SUPPORT for
	   pre-11.0-ALPHA* builds. Manually compile from /usr/src.

       o   BHYVE_UEFI_20151002.fd UEFI firmware	required to boot UEFI  guests.
	   In Peter Grehan's public FreeBSD file directory.

       o   BHYVE_UEFI_20160526.fd  UEFI	 firmware required to boot UEFI	guests
	   with	VNC support. In	Peter Grehan's public FreeBSD file directory.

       o   grub2-bhyve Required	for booting non-FreeBSD	or non-UEFI guests. In
	   ports and pkg.

       o   tmux	 Required  when	using the command chyves <guest> console tmux.
	   In ports and	pkg.

       o   vale-ctl Required for VALE networking support. In '/usr/src'.

   Hardware dependencies:
       bhyve(8)	is dependent on	hardware virtualization	CPU features. One fea-
       ture  is	 called	Second Level Address Translation (SLAT), also known as
       nested paging. Intel's implementation is	called	Extended  Page	Tables
       (EPT)  and AMD's	implementation is called Rapid Virtualization Indexing
       (RVI). This CPU feature is referenced as	 'POPCNT'  in  dmesg(8)	 under
       'Features2:' line.

       There  is  an early series of CPUs from Intel (Xeon 5500) that has most
       of the necessary	virtualization capabilities but	lack a	feature	 Intel
       later  implemented  called  unrestricted	 guest ('UG' in	dmesg(8) under
       "VT-x:"). Without this feature bhyve is limited to FreeBSD guests  with
       only  one  CPU  core. AMD has the 'UG' functionality in all of it's RVI
       implementations.

       PCI Pass-through	support	is also	provided through  another  sub-set  of
       hardware	virtualization CPU features called I/O MMU virtualization. In-
       tel calls their implementation 'VT-d' and AMD calls  their  implementa-
       tion  'AMD-Vi'. See the pcidev_{n} property under Guest Properties sec-
       tion for	instructions on	attaching PCI pass-through devices. A check is
       ran  before  starting a guest that has PCI pass-through devices config-
       ured and	will not start the guest if the	host does not have the	neces-
       sary CPU	feature.

ZFS
       As  mentioned in	the DEPENDENCIES section above,	there is a hard	depen-
       dency on	ZFS and	as such	requires at least one ZFS pool.	There are many
       features	that rely on ZFS, such as cloning. The ISO, UEFI firmware, and
       guest resources and properties are stored within	ZFS datasets.

       Single and multiple ZFS pools are supported, there is no	loss in	 func-
       tionality  for  either.	chyves datasets	get mounted to '/chyves/pool'.
       The '/chyves' directory itself is stored	on '/' and does	not belong  to
       any of the chyves datasets. The first pool is assigned the primary pool
       role but	this can be changed afterwards.	The primary pool is  the  only
       pool  containing	the UEFI Firmware resources, ISO resources, the	global
       properties, and the comprehensive logs. The primary pool	can be changed
       with  chyves  dataset <new-prim-pool> promote which moves the resources
       and global configuration	to <new-prim-pool> from	 the  current  primary
       pool.

       See   DATASET  SUB-COMMANDS  section  below  for	 managing  the	chyves
       datasets.

       Example of a multi-pool folder structure:

	   /chyves/prim_pool/guests/
			    /guests/<guest1>/.config/.cfg	       # File
			    /guests/<guest1>/disk{n}		       # Volumes
			    /guests/<guest1>/img		       # Dataset(s)
			    /guests/<guest1>/log		       # Dataset
			    /guests/<guest1>/log/YYYYMM.log	       # Files
			    /guests/<guest2>/.config/.cfg	       # File
			    /guests/<guest2>/disk{n}		       # Datasets
			    /guests/<guest2>/img		       # Dataset(s)
			    /guests/<guest2>/log		       # Dataset
			    /guests/<guest2>/log/YYYYMM.log	       # Files
			    /guests/.defaults			       # File
			    /.config				       # Dataset
			    /.config/global.cfg			       # File
			    /.config/pool.cfg			       # File
					     pool_role=primary	       # Pool property
			    /logs				       # Dataset
			    /logs/YYYYMM.log			       # Files
			    /ISO				       # Datasets
			    /Firmware				       # Datasets
	   /chyves/secd_pool/guests
			    /guests/<guest3>/.config/.cfg	       # File
			    /guests/<guest3>/disk{n}		       # Volumes
			    /guests/<guest3>/img		       # Dataset(s)
			    /guests/<guest3>/log		       # Dataset
			    /guests/<guest3>/log/YYYYMM.log	       # Files
			    /.config				       # Dataset
			    /.config/pool.cfg			       # File
					     pool_role=secondary       # Pool property
	   /chyves/anther_pl/guests
			    /guests/<guest4>/.config/.cfg	       # File
			    /guests/<guest4>/disk{n}		       # Volumes
			    /guests/<guest4>/img		       # Dataset(s)
			    /guests/<guest4>/log		       # Dataset
			    /guests/<guest4>/log/YYYYMM.log	       # Files
			    /.config				       # Dataset
			    /.config/pool.cfg			       # File
					     pool_role=secondary       # Pool property

       Offline dataset:

	   offlinePL/chyves_offline				       # Datasets
	   offlinePL/chyves_offline/.config/pool.cfg		       # File
						    pool_role=offline  # Pool property

       Example of a single pool	dataset	hierarchy:

	     /chyves/zroot/guests
			  /guests/<guest1>/.config/.cfg		       # File
			  /guests/<guest1>/disk{n}		       # Volumes
			  /guests/<guest1>/img			       # Dataset(s)
			  /guests/<guest1>/log			       # Dataset
			  /guests/.defaults			       # File
			  /.config				       # Dataset
			  /.config/global.cfg			       # File
			  /.config/pool.cfg			       # File
					   pool_role=primary	       # Pool property
			  /logs					       # Dataset
			  /ISO					       # Datasets
			  /Firmware				       # Datasets

   GUEST STORAGE
       By default, chyves uses ZFS volumes as guest storage backing. Each vol-
       ume  is named disk{n} and is attached to	the guest via an emulated AHCI
       controller. On all hosts, 8 ahci-hd devices are supported per PCI slot.
       On  certain  hosts, up to 32 ahci devices can be	attached per PCI slot.
       See global property 'consolidate_bhyve_pci_devices' for details.

       There is	limited	support	for raw	file backed guest storage. This	 is  a
       manual  process and there are not any chyves commands that create, mod-
       ify, or delete these raw	image files. All files in  the	'img/'	folder
       and  child folders are attached to the guest on start. To have indepen-
       dent snapshots for each	raw  image,  create  a	child  datasets	 under
       <pool>/chyves/guests/<guest>/img,  this	has an advantage over ZFS vol-
       umes as there is	no refreservation cost associated with the  snapshots.
       The command chyves info reports the raw image files and size. To	create
       a sparse	raw image run:

       truncate	-s 16G /chyves/<pool>/guests/<guest>/img/<image-name>.

LOGS
       All logs	entries	are stored on the  primary  pool  under	 /chyves/<pri-
       mary-pool>/chyves/logs  with  the  dataset's compression	turned on. The
       file are	separated on a monthly basis using a YYYYMM.log	file name. The
       log entries use the ISO 8601 date format	for monitoring tools. There is
       a separate log file on a	per guest basis, however at this time not  all
       guest  events  are  recorded  to	 this  file,  however these events are
       recorded	in the global logs regardless.

	   2016-07-09T21:27:00+0000 - [2] - Mimicking tap50 interface for g10.

       The logs	are dash delimited with	leading	and  trailing  single  spaces.
       The   first   field   indicates	 the   time   in   ISO	 8601  format,
       '2016-07-09T21:27:00+0000' in the example above.	The  second  field  is
       the  'stdout_level',  '2'  in the example above.	The third field	is the
       log event, 'Mimicking tap50 interface for g10.' in the  example	above.
       See the Global Configuration Properties section below for more informa-
       tion on what 'stdout_level' does.

NETWORK
       The network mode	is set by the global setting 'network_design_mode' and
       determines  how	chyves handles network devices for the guests. The two
       modes are 'auto'	and 'system', both modes are discussed below.

       VALE support is experimentally provided as a  proof  of	concept.  VALE
       support	is  not	widely implemented and is feature limited at it's cur-
       rent development	stage. To use VALE requires that  'device  netmap'  be
       compiled	  into	the  kernel  and  that	'vale-ctl'  be	compiled  from
       '/usr/src/tools/tools/netmap' and moved to $PATH	 directory.  VALE  na-
       tively  support em(4), igb(4), ixgbe(4),	and re(4). Other NICs must use
       emulation which may reduce performance. See netmap(4) for more informa-
       tion. VALE interfaces are added to a guest the same way a tap interface
       is added	using: 'chyves network add vale{n}[:{p}]'  where  {n}  is  the
       vale  switch  you want to use and optionally :{p} specifies the port on
       that switch. Specifying a port is not recommended. VALE interfaces  are
       clever  in  that	 the interface is commonly shared between other	guests
       connecting to that 'virtual' VALE switch. Two guests using 'vale0' will
       be connected to the same	virtual	switch.

       Hosts after a certain commit support Intel e1000	emulation as an	alter-
       native  to  using  virtio-net.  This  can  be   utilized	  by   setting
       'bhyve_net_type'	to 'e1000', see	guest property for details.

   AUTO	DESIGN NETWORK MODE
       By  default, chyves is configured to manage the complete	network	design
       layout. chyves keeps track of which physical, tap, and vlan  interfaces
       are  associated	with  which bridges, and which tap and VALE interfaces
       are associated with which guests. Upon starting a guest	these  designs
       are  checked, verified, and then	created	if not existing. To manage the
       network design using this method, use the 'chyves network' commands  to
       create the associations and 'chyves list	bridges' for an	overview.

       On a fresh simple installation, the only	command	needed is 'chyves net-
       work bridge0 attach em0'. This tells chyves to associate	the  interface
       'em0'  with  the	default	bridge.	To change the default bridge, the com-
       mand would be 'chyves network bridge64 default' to  set	'bridge64'  as
       the  default. All tap interfaces	created	for guests are then associated
       with the	bridge set in defaults.

       As indicated above, tap interfaces are attached to a bridge, which  op-
       tionally	can be attached	to a physical or vlan interface. If the	bridge
       is not associated with a	physical or vlan interface, the	bridge must be
       put  into  'private'  mode.  This  is  done  by running 'chyves network
       bridge64	private'. A private bridge allows sensitive traffic to	trans-
       verse  the  bridge  without  the	possibility of traffic being monitored
       from an outside network.	A private bridge can also use  pf(4)  for  NAT
       capabilities but	this is	not built into chyves.

   SYSTEM DESIGN NETWORK MODE
       The  'system' mode assumes the user will	correctly create and configure
       all devices on the host system, including the tap  devices.  No	checks
       are ran to test connectivity and	chyves simply attaches tap or VALE in-
       terfaces	to the guest as	configured in the 'net_ifaces' property.

       Converting from 'system'	to 'auto'  will	 not  automatically  start  to
       work. Each device will need to be manually associated with each other.

PROPERTIES
       There are four types of chyves properties: global configuration proper-
       ties, pool properties, per-guest	properties, and	default	guest  proper-
       ties.  Each  is stored in a file	in the chyves dataset hierarchy. It is
       not recommended to directly edit	the '*.cfg'  files,  this  is  because
       there  are no filters to	stop typos, warn for incorrect property	names,
       or invalid values. The filter is	only applied when running the  'chyves
       guest  set'  command and	not rechecked when pulling the values from the
       files.

       Also certain actions are	triggered when setting certain properties. For
       example,	 setting  the  'template' guest	property will set or unset the
       readonly	flag on	the ZFS	dataset.

       The pool	configuration is  stored  in  a	 file  at  '/chyves/pool/.con-
       fig/pool.cfg'. On the primary pool there	is also	a file in the same di-
       rectory called 'global.cfg', this  contains  the	 global	 settings  for
       chyves.

       The  'dataset_role' setting is used to store the	role of	that pool con-
       taining the chyves datasets. Valid values are primary,  secondary,  and
       offline.	 Only  one 'primary' dataset can be configured per system. The
       primary dataset is the only dataset that	contains the ISO and  firmware
       resources. Only the primary pool	has guest defaults configuration file.
       These defaults are referenced when setting properties for  new  guests.
       The  primary  pool is also where	new guests are created when the	[pool]
       argument	is not declared	with 'chyves create' command. All other	active
       pools  are set to the "secondary" role. Pools set to the	'offline' role
       are left	untouched and is a good	role to	put a local replicated backup.
       The primary pool	is critical to the operation of	chyves,	without	a pri-
       mary pool set, only the 'dataset' sub-command are permitted to run. The
       first pool setup	on the system is the primary pool. This	can be changed
       later with 'chyves dataset pool promote'.

   Pool	Properties
       These properties	keeps track of the version  of	the  dataset  and  the
       role.  With  multiple pools, defined per-pool roles are necessary. This
       is defined through the use of a	dataset	 called	 'pool/chyves/.config'
       where the pool specific properties (pool.cfg) are stored.

       o   dataset_role={primary|secondary|offline}

	   Managed through 'chyves dataset' commands.

	   Vital  in  multi-pool  configurations, each role type has different
	   characteristics as explained	below:

	   primary - Pools in this role	are utilized exclusively for the  fol-
	   lowing  purposes: Store ISO and Firmware resources. Reference point
	   for global properties. New guest are	defaulted  to  be  created  on
	   this	pool unless otherwise specified	as a property for 'chyves cre-
	   ate'. Only one ZFS pool can be in the primary role per  system  and
	   is the only role for	single zpool configurations.

	   secondary  -	 Pools in this role are	utilized for storing other ac-
	   tive	guests not stored on the primary ZFS pool.

	   offline - Pools in this role	are not	utilized,  this	 role  is  in-
	   tended  for backup. Guests on these pools are omitted from all sub-
	   commands. To	further	prevent	action on 'offline' pools, the dataset
	   is renamed to 'chyves_offline' which	chyves will completely ignore.

       o   pool_version={dataset-version}

	   Managed through 'chyves dataset' commands.

	   This	 property  indicates  the  current version of the chyves pool.
	   This	an upgrade mechanism to	ensure	the  dataset  and  it's	 child
	   datasets  contain  the  correct  architecture.  The command 'chyves
	   dataset pool	upgrade' brings	a dataset incrementally	to the	latest
	   version when	there is a discrepancy between the dataset version and
	   the chyves version. A check is ran before execution to  ensure  the
	   version of the pool is compatible with the version of chyves.

   Global Configuration	Properties
       These properties	control	the behavior of	chyves.

       Many of these properties	are not	recommended to be changed directly un-
       less the	scope of their impact is understood or	when  managed  through
       subcommands  such  as 'chyves network'. The preferred management	method
       is given	on the first line after	each property in bold text.

       Usage:

       chyves global set <property>=<value>

       chyves list global [<pool>]

       Properties:

       o   auto_load_kernel_mods={yes|no}

	   User	managed	and settable.

	   When	set to "yes", chyves will load the  kernel  modules  necessary
	   for	guests.	When set to 'no', chyves will simply check if the mod-
	   ules	are loaded and exit with a message if not loaded.

       o   bridge{n}_phy_attach={interface|private}

	   Managed through 'chyves network' commands.

	   This	property contains a physical or	 vlan  interface  name	for  a
	   bridge  to  belong.	When set to 'private' the bridge is not	joined
	   with	a outside network.

	   Example values:

	   bridge0_attach=em0

	   bridge10_phy_attach=vlan52

	   bridge512_phy_attach=private

	   Usage:

	   chyves network bridge{n} attach {<interface>|private}

       o   bridge{n}_tap_members={tap{n}[,tap{n},tap{n}]}

	   Managed through 'chyves network' commands.

	   This	property is a comma separated list  of	the  tap  members  for
	   bridge{n},  where {n} is the	interface number. Valid	values for {n}
	   are 0 to 32768. Guests are automatically added to this property us-
	   ing	the  bridge{n}	that  is  set in 'defaults' under the 'bridge'
	   property.

	   Examples values:

	   bridge0_tap_members=t0

	   bridge70_tap_members=t20,t30

	   A tap can be	moved to another bridge	by first removing the tap from
	   the	original bridge	and adding to the desired bridge. Use the fol-
	   lowing command syntax to do this:

	   chyves network bridge{n} unjoin tap{n}

	   chyves network bridge{n} join tap{n}

       o   check_for_updates={daily|weekly|monthly|always}

	   User	managed	and settable.

	   This	determines how often to	check for updates for  chyves  on  the
	   GitHub  repository.	The keyword 'always' means to check every time
	   chyves is ran.

       o   check_for_updates_last_check=YYYYMMDD

	   Managed internally.

	   This	contain	the date of last check.

       o   check_for_updates_last_check_status={1|0}

	   Managed internally.

	   When	set to '1' an update is	available.

       o   check_for_updates_timeout_seconds={n}

	   User	managed	and settable.

	   This	number is used to set the network timeout for fetch(1)	before
	   giving up.

       o   check_for_updates_unique_id=UUID

	   Managed internally.

	   This	 contains  a  UUID used	for identification. It is set when the
	   primary pool	was created.

       o   console_start_offset={integer}

	   User	managed	and settable.

	   This	is a compatibility mechanism used to offset the	 first	chyves
	   null	 console modem number so that a	collision are less likely with
	   another application or bhyve	front end manager. The initial default
	   is  set  to '50'. After the first guests are	created, then the next
	   highest console number is used to create the	next guest.

       o   consolidate_bhyve_pci_devices={no|yes}

	   User	managed	and settable.

	   The default is set to 'no' as it may	have performance implications,
	   however  this may be	necessary if more than 26 PCI devices are con-
	   nect	to a guest. This property signals the bhyve string  generators
	   to use all PCI functions for	similar	devices.

	   On all hosts, 8 PCI functions can be	used per PCI slot. However, up
	   to 32 ahci-hd devices can be	attached to  a	single	PCI  slot  for
	   hosts running 12-CURRENT after commit r302459, 11-STABLE after com-
	   mit r304422,	or hosts running 10-STABLE after commit	r304420. Mean-
	   ing	in  ideal  conditions up to 208	PCI devices can	be attached on
	   any host or up to 832 devices on supported hosts.

       o   default_clone_flag

	   User	managed	and settable.

	   Flag	to use when cloning a guest and	the clone type flag  is	 omit-
	   ted.	See clone command under	GUEST CLONE SUB-SUB-COMMAND section.

       o   default_info_flags

	   User	managed	and settable.

	   Flags  set here are passed to 'chyves info' when no flags are used.
	   See command 'chyves info' for valid flags.

       o   default_list_flags

	   User	managed	and settable.

	   Flags set here are passed to	'chyves	info' WHEN the command 'chyves
	   list'  is  used.  Both  commands  use the same backend function and
	   'chyves list' is  kept  around  for	legacy	reasons.  See  command
	   'chyves info' for valid flags.

       o   dev_mode={off|on|-xvn}

	   User	managed	and settable, advanced feature.

	   Setting  to	'on' allows for	functions to be	called direct from the
	   command line	using chyves dev. Using	the -xvn flags instead of  the
	   word	 on  will  use	Bourne's set command to	turn on	special	option
	   flags during	chyves dev execution. These flags can be used individ-
	   ually (-x|-v|-n) or combined	(-xv). See sh(1).

       o   eject_iso_on_n_reboot={n}

	   User	managed	and settable.

	   See same property name under	the Guest Properties section below for
	   description of functionality.

       o   log_mode={host|guest|dual}

	   User	managed	and settable.

	   Determines the source of information	to save	in the log files  when
	   'log_to_file'  is  set to 'yes'. When set to	'guest'	only guest ac-
	   tions are recorded. When set	to 'host' only actions	not  involving
	   guests are recorded.	When set to dual, both are recorded.

       o   log_to_file={yes|no}

	   User	managed	and settable.

	   Turns  on  logging to a file. For general logging the file is saved
	   at '/chyves/logs/YYYYMM.log'	and for	guests the file	 is  saved  at
	   '/chyves/guests/guest/log/YYYYMM.log'	     or		    at
	   '/chyves/poolguests/guest/ log/YYYYMM.log'.

       o   network_design_mode={auto|system}

	   User	managed	and settable.

	   The default is 'auto', which	means chyves manages everything	 above
	   the	physical or vlan Ethernet interface, all the way to the	guest.
	   This	is managed through 'chyves  network'  commands.	 When  set  to
	   'system',  chyves  only  attaches the tap or	VALE interfaces	to the
	   guest and leaves it to the user to have configured everything  else
	   (correctly).	See NETWORK section above.

       o   restrict_new_property_names=[off|on]

	   User	managed	and settable, advanced feature.

	   Setting  to	"on"  allows  for  new	properties  to be created with
	   'chyves set'. The default is	set to "off" to	prevent	 creating  new
	   properties  due to typos. Even when set to 'on' this	will not allow
	   a creation of new global or defaults	properties.

       o   stdout_level={0-3}

	   User	managed	and settable.

	   Determines the amount of output to send to the terminal.  The  four
	   numeric  settings  do  the  following. This does not	impact the log
	   files.

	   0 - off - No	output

	   1 - minimal Sub command action shown

	   2 - regular + Each step in the sub command shown

	   3 - verbose ++ Outputs bhyve	and loader command used.

	   4 --	Never print to screen. Used in __return* functions that	 would
	   break even with the value set to '0'.

       o   tap_start_offset={integer}

	   User	managed	and settable.

	   This	 is  a compatibility mechanism used to offset the first	chyves
	   tap interface so that a collision is	less likely with  another  ap-
	   plication or	bhyve front end	manager. The initial default is	set to
	   "50", valid values are from 0 to 32767.

       o   tap_up_by_default={off|on}

	   User	managed	and settable.

	   When	set to "yes", the sysctl: net.link.tap.up_on_open  is  set  to
	   "1"	when  running script. Default is "on" but it is	recommended to
	   set to "off"	and update your	/boot/loader.conf to set this sysctl.

       o   vlan_iface_base_name=vlan_base_name

	   User	managed	and settable.

	   When	using a	non-standard vlan naming nomenclature  this  needs  to
	   reflect  the	 change. The default is	"vlan" when using the standard
	   "vlan{n}" naming nomenclature.

       o   uefi_vnc_port_start_offset={integer}

	   User	managed	and settable.

	   This	is a compatibility mechanism used to offset the	first VNC port
	   used	 by  UEFI  guests. The initial default is set to "5900", valid
	   values are from 1 to	65536.

   Guest Properties
       These are properties used by chyves to specify the properties  to  pass
       to bhyve, grub-bhyve, and chyves	itself when starting a guest. Changing
       these values on a running guest will not	yield  any  change  until  the
       guest is	completely stop	and then started again.

       Usage:

       chyves <guest> get <property>

       chyves <guest> get all

       chyves list properties

       chyves list <property>

       chyves  <guest>|MG|all  set  <property1>=<value>	 [<property2>=<value>]
       [<property3>=<value>]

       chyves  <guest>|MG|all  set  <property1>=<value>	 [<property2>=<value>]
       [<property3>=<value>]	 [<guest2>    [<property1>=<value>]    [<prop-
       erty2>=<value>]]

       o   bargs

	   For advanced	users to pass additional flags directly	to bhyve.  See
	   bhyve(8) for	complete list of flag options. To find out which flags
	   are used when starting a specific guest, turn on global  configura-
	   tion	'stdout_level' to '3' or check the global log file.

       o   bhyveload_flags

	   For	advanced users to pass additional flags	directly to bhyveload.
	   This	is primarily intended to pass '-e' flags for environment vari-
	   ables.  See	bhyveload(8)  for  a complete list of flag options. To
	   find	out which flags	are used when starting a specific guest,  turn
	   on  global  configuration 'stdout_level' to '3' or check the	global
	   log file.

       o   bhyve_disk_type={virtio-blk|ahci-hd}

	   Sets	the method to attach ZFS volumes and disk images to the	guest.
	   Most	guests do not support Virt-IO block devices without installing
	   drivers at installation.

       o   bhyve_net_type={virtio-net|e1000}

	   Sets	the method to attach tap interfaces to the guest. Intel	 e1000
	   emulation is	available on hosts on 12-CURRENT after commit r302504,
	   11-STABLE after commit  r304424,  and  on  10-STABLE	 after	commit
	   r304425.

       o   cpu={n}

	   Assigns {n} number of CPU cores to a	guest, up to 16.

       o   chyves_guest_version=MMmm

	   Managed through 'chyves guest upgrade' command.

	   Used	 to ensure the guest contains latest properties	when moved be-
	   tween hosts on different versions of	chyves.	"MM" signifies the ma-
	   jor version as an integer and "mm" as the minor version as an inte-
	   ger.	Changes	to the major version require an	 upgrade  via  'chyves
	   guest upgrade' because a guest property value format	has changed or
	   a new required property was added. Differences in the minor version
	   do no require an upgrade but	may not	have full functionality.

       o   description="INSERT BRIEF DESCRIPTION"

	   Used	to describe guest. Use double quotes when description contains
	   spaces.

       o   eject_iso_on_n_reboot={n}

	   This	property determines if and when	to  eject  the	optical	 media
	   when	 installing  an	OS. By default this is set to '1' because most
	   *nix	systems	install	on the first boot. Live	CD should be turned to
	   '0' and Windows systems should be set to '2'	or greater.

	   There  is  a	global property	which shares the same name has has the
	   same	functionality. The global property is used if not set  on  the
	   guest.

       o   loader={bhyveload|uefi|grub-bhyve}

	   Tells  which	 loader	to use to boot guest. bhyveload	is for FreeBSD
	   based guests, uefi is used for UEFI based guests, and grub-bhyve is
	   used	for all	other guests.

	   When	setting	a guest	to use the uefi	loader,	some uefi_* properties
	   are populated. One of these properties is  uefi_firmware  which  is
	   required for	guests to start.

       o   net_ifaces=tap{n}|vale{n}[:{p}]

	   Managed   through   'chyves	network'  commands  when  'network_de-
	   sign_mode' is set to	'auto'.

	   User	settable when 'network_design_mode' is set to 'system'.

	   Stores which	tap and	VALE interfaces	are assigned to	a guest. Where
	   {n}	is  a unique interface number. Taps are	process	locked and can
	   not be shared simultaneously	by multiple guests.  Multiple  devices
	   are stored using commas as delimiters.

	   VALE	 interface  names  must	 be vale{n}[:{p}] where	{n} can	be al-
	   pha-numeric up to 9 characters long.	Optionally a port can be spec-
	   ified where :{p} starts with	a separating colon and either a	single
	   alphabetical	character or a number 0-9999.

	   Usage:

	   chyves network <guest> add {tap|tap{n}|vale{s}[:{p}]}

	   chyves network <guest> add {tap|tap{n}} bridge{n}

	   chyves network <guest> remove {tap{n}|vale{s}[:{p}]}

	   See NETWORK SUB-COMMANDS section for	complete list of  commands  to
	   manage this property.

       o   notes="INSERT FURTHER NOTES"

	   Used	to further describe guests. Use	double quotes when description
	   contains spaces.

       o   os=operating-system

	   Used	to determine the properties to use for the  grub.cfg  and  de-
	   vice.map files when the loader property is set to grub-bhyve.

	   Supported  values  are  openbsd60, openbsd59, openbsd58, openbsd57,
	   netbsd, debian, d8lvm,  centos6,  centos7,  arch,  gentoo,  coreos,
	   coreossecure, default, or custom. Any other values are invalid. de-
	   fault is the	generic	catch-all value.

	   When	'coreos' is used, the flag 'coreos.autologin' is used  to  by-
	   pass	 authentication	 for the console. When 'coreossecure' is used,
	   the console requires	a username and password.

	   Using custom	relies on the user to manually create the grub.cfg and
	   device.map	    files	in	 the	  guest	     directory
	   /chyves/<pool>/guests/<guest>.

	   Example:

	   For an OpenBSD guest	"obsd59" located in /chyves/kubota/obsd59/ and
	   an install ISO in /chyves/dozer/ISO/install59.iso/, your files will
	   look	like this:

	   device.map file:

	   (hd0) /dev/zvol/kubota/chyves/obsd59/disk0
	   (cd0) /chyves/dozer/ISO/install59.iso/install59.iso

       grub.cfg	file for installation:

	   kopenbsd -h com0 (cd0)/5.9/amd64/bsd.rd
	   boot

       grub.cfg	file after installation	is complete:

	   kopenbsd -h com0 -r sd0a (hd0,openbsd1)/bsd
	   boot

       The chyves project welcomes any OS submissions to help other users from
       re-discovering leg work done by others.

       o   pcidev_{n}={bhyve-pci-dev}

	   This	 is  an	advance	feature	and assumes you	have read bhyve(8) AND
	   understood it. This property	does NOT have a	complete input verifi-
	   cation suite	or complete host support verification.

	   This	 property  attaches a bhyve PCI	device to the guest. The '{n}'
	   is completely arbitrary number and is only a	matter of the order of
	   attachment.	'{bhyve-pci-dev}'  is  the bhyve PCI device to attach.
	   See bhyve(8)	for a complete list or below for common	uses.

	   chyves attaches each	custom PCI device alone	on the PCI bus,	except
	   for pass-through devices.

	   To create a virtio RNG, the command would be

	   chyves <guest> set pcidev_0=virtio-rnd

	   To  create  a  permanently  attached	 optical  media	device using a
	   chyves ISO resource where "cd.iso" is the  ISO  name,  the  command
	   would be:

	   chyves	<guest>	      set	pcidev_1=ahci-cd,/chyves/<pri-
	   mary-pool>/ISO/cd.iso/cd.iso

	   To create a sparse 16G disk and attach, the commands	would be:

	   truncate -s 16G /chyves/<pool>/guests/<guest>/01.img

	   chyves			  <guest>			   set
	   pcidev_2=ahci-hd,/chyves/<pool>/guests/<guest>/01.img[,block-op-
	   tions]

	   Although not	recommended, a tap interface can be manually  attached
	   using this command:

	   chyves <guest> set pcidev_3=virtio-net,tap10

	   ...using Intel e1000	emulation (only	12-CURRENT hosts):

	   chyves <guest> set pcidev_5=e1000,tap10

	   ...with an associated mac address:

	   chyves <guest> set pcidev_4=virtio-net,tap10,mac=58:9C:FC:00:00:00

	   PCI	pass-through  devices  which  share the	same physical PCI slot
	   number on the host also will	share the same virtual	physical  slot
	   number in the guest.	This is	due to the potential for kernel	panics
	   on certain devices such as Fiber Channel PCI	cards. Keep  in	 mind,
	   only	eight PCI functions can	be assigned to a virtual physical slot
	   in bhyve, additional	devices	are ignored. To	set a PCI pass-through
	   device  use	the  follow  commands, where '6' and '7' are arbitrary
	   values and '9/0/0' and '9/0/1' is the PCI bus/slot/function	device
	   triplet.

	   chyves <guest> set pcidev_6=passthru,9/0/0

	   chyves <guest> set pcidev_7=passthru,9/0/1

	   To get the PCI triplet on the host, run:

	   root@bhost:~	# pciconf -lv |	grep -A4 -E '^ppt'
	   ppt0@pci0:9:0:0: class=0x0c0400 card=0x01461077 chip=0x24321077 rev=0x02 hdr=0x00
	   vendor     =	'QLogic	Corp.'
	   device     =	'ISP2432-based 4Gb Fibre Channel to PCI	Express	HBA'
	   class      =	serial bus
	   subclass   =	Fibre Channel
	   ppt1@pci0:9:0:1: class=0x0c0400 card=0x01461077 chip=0x24321077 rev=0x02 hdr=0x00
	   vendor     =	'QLogic	Corp.'
	   device     =	'ISP2432-based 4Gb Fibre Channel to PCI	Express	HBA'
	   class      =	serial bus
	   subclass   =	Fibre Channel

       Normally	 the device in the example above would use the isp(4) drivers.
       However,	the devices need to be using the ppt drivers. In this context,
       the  ppt	 drivers  tell	the  host  to  ignore the device as it will be
       pass-through to a virtualized guest. To get a PCI device	 to  load  the
       ppt  drivers instead of it's normal drivers, edit the /boot/loader.conf
       file to contain the PCI triplet of each device in  the  pptdevs	direc-
       tive. The following is consistent with our examples above:

	   pptdevs="9/0/0 9/0/1"

       A reboot	is necessary after editing the /boot/loader.conf.

       The '-S'	flag must be appended to the current 'bargs' guest property to
       wire the	guest memory for PCI pass-through to work.

       To remove a pcidev_{n} from a guest run:	 chyves	 <guest>|MG|all	 unset
       <pcidev_{n}>.

       VGA/GPU	Passthrough  has not been reported to work. chyves issue #3 on
       GitHub was opened to document success and failures of bhyve passthrough
       of VGA/GPU devices. Check there for the latest status.

       o   ram={n}[S]

	   {n}	is the number of bytes and [S] is the size denomination	in "K"
	   kilobytes,  "M"  megabytes,	"G"  gigabytes,	 or   "T"   terabytes.
	   Megabytes are assumed if no suffix is given.

       o   rcboot={n}

	   Any	non-zero number	tells chyves to	start the guest	upon host boot
	   when	the 'chyves_enable=YES'	directive is used  in  '/etc/rc.conf'.
	   The integer represents the boot priority where the highest numbered
	   guest is started first. Using '0' disables starting on  host	 boot.
	   If multiple guest share the same priority number, then those	guests
	   are started alphabetically.

       o   revert_to_snapshot=@snapshot-name

	   Describes the snapshot to revert to based on	 the  handling	method
	   specified in	'revert_to_snapshot_method'. If	this property is blank
	   but a method	is specified, then the latest snapshot is used.

       o   revert_to_snapshot_method={both|off|reboot|start}

	   Prior to starting and/or rebooting, a guest can be  reverted	 to  a
	   previous  snapshot state. This is helpful for guests	that get modi-
	   fied	outside	of specification such as  frequently  compromised  web
	   server  or VDI instances. Setting to	'start'	only reverts the guest
	   when	starting the guest, 'reboot' only on reboots, 'both' for  both
	   starts  and	restarts,  and	'off'  to  disable this	feature	on the
	   guest.

	   The snapshot	rollback impacts the entire  dataset  including	 guest
	   properties. The guest properties are	reloaded after the rollback to
	   ensure there	is a consistent	guest state. See the snapshot rollback
	   command  under  the GUEST SNAPSHOT SUB-SUB-COMMANDS section for de-
	   tails on what actions are taken during a rollback.

       o   serial=nmdm{n}

	   Attaches null modem console.	{n} must be a unique  number  and  can
	   not	be  shared between started guests. A unique number is given on
	   creation. Not recommended to	be changed and can  never  be  changed
	   while guest is running.

       o   tap{n}_mac={xx:xx:xx:yy:yy:yy}

	   Where  {n}  is the tap interface number to assign the specified MAC
	   address to be attached to the network adapter inside	the guest  OS.
	   The MAC address is an ASCII string in ethers(5) format.

	   Usage:

	   chyves <guest> set tap10_mac=58:9C:FC:00:00:00

       o   template={yes|no}

	   When	 set  to  'yes'	 chyves	 will not start	the guest. The guest's
	   datasets are	set to readonly	so no changes can be  made  until  set
	   back	to 'no'.

       o   uefi_console_output={vnc|serial}

	   When	set to "serial", a UEFI	guest will use the standard serial in-
	   terface. Guests set to "vnc"	will use a VNC connection on supported
	   hosts.  VNC	output	is experimental	and introduced to bhyve	in May
	   2016. VNC support requires at least 11-CURRENT (r300097) or	later,
	   a  modified	bhyve  binary  with  graphics  support,	 and  the UEFI
	   firmware "BHYVE_UEFI_20160526.fd".

       o   uefi_firmware=firmware-resource-name

	   UEFI	firmware bootrom file used to  boot  UEFI  guests.  These  are
	   available  in Peter Grehan's	public FreeBSD file directory or sysu-
	   tils/uefi-edk2-bhyve. The value must	be the exact name as displayed
	   in 'chyves list firmware'.

       o   uefi_vnc_client={print|freerdp|custom}

	   This	 stores	 the preferred VNC client used to start	a session with
	   the guest properties	pre-populated.

       o   uefi_vnc_client_custom_cmd

	   This	 contains  the	command	 to  start   the   VNC	 client	  when
	   'uefi_vnc_client' is	set to 'custom'.

       o   uefi_vnc_ip=x.x.x.x

	   Specifies  the  IP  for the host to bind-to for VNC connections for
	   the guest. The default is "0.0.0.0" which binds to all IPs  on  the
	   host.

       o   uefi_vnc_mouse_type=ps2|usb3

	   This	 property is only used when the	property "uefi_console_output"
	   is set to "vnc". When set to	"ps2" a	PS/2 mouse is connected	to the
	   guest.  When	set "usb3" a USB 3.0 tablet is emulated	which provides
	   much	better mouse performance but is	only supported on newer	 guest
	   OSes.

       o   uefi_vnc_pause_until_client_connect={on|off}

	   When	 set  to  "on"	the guest will wait to boot until a VNC	client
	   connects. This is helpful for installations requiring a  key	 press
	   within a short timing window.

       o   uefi_vnc_port={n}

	   Specifies  the  port	 for the host to listen	on for VNC connections
	   for the guest. Valid	values are from	1 to 65536.

       o   uefi_vnc_res=widthxheight

	   Specifies the resolution the	VNC client will	connect	with. The fol-
	   lowing resolutions are supported:

	   1920x1200

	   1920x1080

	   1600x1200

	   1600x900

	   1280x1024

	   1280x720

	   1024x768

	   800x600

	   640x480

       o   uuid=uuid

	   Sets	UUID for bhyve instance. Set by	/bin/uuidgen at	creation.

       o   virtio_block_options_disk{n}=[nocache[,direct][,ro][,sector-
	   size=logical[/physical]]]

	   This	is an advance feature and assumes you  have  read  zfs(8)  man
	   page	 AND  completely  understood it. This property does NOT	have a
	   complete input verification suite.

	   This	is a per disk setting used to append  bhyve  virtio-block  op-
	   tions  to  a	 disk. While there is a	very limited use case of these
	   options, they can be	 utilized  but	it  is	not  recommended.  See
	   bhyve(8) for	what these options do.

	   Keep	 in mind that the logical sectorsize is	already	used by	chyves
	   and pulled from the zvol's 'volblocksize'  ZFS  property  when  the
	   virtio_block_options	  is   not   set.   When  the  property	 "vir-
	   tio_block_options_disk{n}" exists, the sectorsize is	no longer  au-
	   tomatically populated to prevent collision.

	   To  set this	property for disk0 to have read	only with a 512	sector
	   size	and a 4096 physical size:

	   chyves    <guest>	 set	 virtio_block_options_disk0=ro,sector-
	   size=512/4096

   Default Guest Properties
       These  properties  are  referenced  when	creating new guests. The guest
       section of properties have the same purpose as described	in Guest Prop-
       erties section above, unless otherwise noted. Below are the initial de-
       fault values on a fresh installation of chyves for reference.

       The some	UEFI properties	are not	assigned until the 'loader'  value  is
       set  to 'uefi'. This is to prevent assigning properties to guests which
       can not utilize those features.

       The UEFI	VNC properties are not assigned	until  the  'uefi_console_out-
       put'  value is set to 'vnc'. This is to prevent unused VNC port numbers
       from being assigned to guests which can not utilize those features.

       Usage:

       chyves defaults set <property>=<value>

       chyves list defaults

       Defaults	and initial values:

       o   bargs

	   "-A -H -P" -	default	for guests running on FreeBSD less than	10.3.

	   "-A -H -P -S" - default for	guests	running	 on  FreeBSD  10.3  or
	   later.

       o   bhyve_net_type=virtio-net

       o   bridge=bridge0

	   Used	to set the bridge to join tap interfaces to for	new guests.

       o   cpu=1

       o   loader=bhyveload

       o   os=default

       o   ram=256M

       o   rcboot=0

       o   revert_to_snapshot=""

       o   revert_to_snapshot_method=off

       o   size=8G

	   Used	during guest and disk creation when the	size property is omit-
	   ted.	No longer a guest property. The	ZFS property 'used' is used to
	   show	the size in 'chyves info -z'.

       o   template=no

       o   uefi_console_output=serial

       o   uefi_firmware="-"

       o   uefi_vnc_client=print

       o   uefi_vnc_ip=0.0.0.0

       o   uefi_vnc_mouse_type=ps2

       o   uefi_vnc_pause_until_client_connect=no

       o   uefi_vnc_res=800x600

       These  default disk properties are direct ZFS values used when creating
       disks for the  guest.  These  properties	 are  not  recommended	to  be
       changed,	 a person is liable to chainsaw	a foot off and then massacre a
       nearby litter of	kitten and or puppies in the process. Heed this	 warn-
       ing!  The  value	 "inherit"  will  inherit  the	value  set  by the ZFS
       (*grand)parents.	See zfs(8) on settable values but remember,  think  of
       the kittens and puppies.	The initial default values are:

       o   disk_volmode=dev

       o   disk_volblocksize=512

       o   disk_dedup=inherit

       o   disk_compression=inherit

       o   disk_primarycache=inherit

       o   disk_secondarycache=inherit

TMUX SUPPORT
       Utilization of tmux is provided under the command 'chyves guest console
       tmux'.

       chyves utilizes tmux by creating	a session called "chyves",  then  cre-
       ates  a	window named after the guest, then splits that window into two
       panes. The left pane is used to manage the guest	and the	right pane  is
       the serial console for the guest. If a window named after the guest al-
       ready exists, then the command is inserted into	the  active  pane  for
       that  guest  window and then the	session	is joined or rather 'attached'
       in tmux's parlance. Each	additional guest has a	window	created	 under
       the same	'chyves' named session.	See tmux(1) for	full details on	use.

       To  use	a  command for tmux, enter: 'Ctrl + b (release-keys) command'.
       Where Ctrl + b is the modifier, (release-keys) is  literally  releasing
       the  Ctrl  and b	keys, and command is the case sensitive	letter or com-
       mand to type in.	Sometimes this is written as  'C-m'  where  'm'	 means
       modifier. In this man page, the abbreviated syntax used is 'C-b'. Below
       are necessary tmux commands for basic use with chyves:

       While in	a tmux session:

       o   C-b n Go to next guest window.

       o   C-b p Go to previous	guest window.

       o   C-b l Go to last guest window.

       o   C-b w List all available guest windows and select which to go to.

       o   C-b (left-arrow) Move cursor	to the pane to the left

       o   C-b (right-arrow) Move cursor to the	pane to	the right

       o   C-b (up-arrow) Move cursor to the pane above

       o   C-b (down-arrow) Move cursor	to the pane below

       o   C-b c Create	a new window

       o   C-b % Split the current pane	vertically

       o   C-b " Split the current pane	horizontally

       o   C-b x Kill current window

       o   C-b d Detach	from tmux session, remains opened in background.

       o   C-b D Detach	other connections from session.	Good if	a  session  is
	   opened  from	 a  smaller resolution client in one location. Discon-
	   necting it will maximum the screen use on the local connection.

       When not	in a tmux session:

       o   tmux	attach -t chyves

	   Reattach to the tmux	session	named 'chyves'.

       o   tmux	at

	   Reattach to the last	tmux session

UEFI SUPPORT
       Support to boot guests via UEFI	is  provided  through  the  use	 of  a
       bootrom	firmware  set  with  the  'uefi_firmware'  property. This boot
       method is set by	changing the 'loader' property to 'uefi'. When this is
       done  for  the  first time on a guest, many other guest UEFI properties
       are populated.

       Graphical console support is provided via UEFI GOP as of	May 2016, this
       replaces	 the serial interface. Support is provided as a	technical pre-
       view as there are still sharp edges. Know issues:  partial  VNC	client
       support	and  mouse  mapping issues when	'uefi_vnc_mouse_type' property
       set to 'ps2'. The following is required for graphical console use:  the
       property	 'uefi_console_output'	must  be  set to 'vnc' on the guest, a
       host on 11-BETA1, and the UEFI firmware 'BHYVE_UEFI_20160526.fd', and a
       VNC client.

       See  Guest Properties section for complete list of related uefi_* prop-
       erties.

   VNC CLIENT SUPPORT
       Support for VNC client launching	is utilized with chyves	 <guest>  con-
       sole  vnc. This either prints the VNC settings or starts	a VNC session.
       Currently FreeRDP is preconfigured or a custom VNC client can  be  con-
       figured with the	'uefi_vnc_client_custom_cmd' property.

COMMANDS
       o   dataset

	   See DATASET SUB-COMMANDS section.

       o   firmware

	   See FIRMWARE	SUB-COMMANDS section.

       o   <guest>

	   See GUEST SUB-COMMANDS sections.

       o   help

	   Prints version information, command valid syntax.

       o   dev

	   chyves dev [<function>|<command>] "param1" ... "param7"

	   Allows  for	a  function to be called direct	from command line when
	   developing for chyves. Requires dev_mode to be set  with  something
	   other  than 'off'. This is globally set on the primary pool on. See
	   Global Configuration	Properties section above.

	   The properties are function indexed and only	seven  parameters  are
	   possible  due  to  the way Bourne indexes parameters. Double	quotes
	   are required	for parameters with strings, such as commands.

       o   info

	   See INFO COMMAND section.

       o   iso

	   See ISO SUB-COMMANDS	section.

       o   list

	   See LIST SUB-COMMANDS section.

       o   upgrade

	   chyves upgrade [master|dev|check]

	   Downloads and installs the latest version from the current  running
	   branch  on GitHub. Optionally the branch can	be specified to	change
	   branches. Changing from dev to master may render an unusable	system
	   if  the  'chyves_guest_version' or 'dataset_version'	has changed in
	   those branches and those changes have not made it to	'master' yet.

	   To run only a version check,	use the	keyword	'check'.

       o   version

	   Prints the current running chyves version, dataset  version,	 guest
	   version, and	branch.

   DATASET SUB-COMMANDS
       o   install

	   chyves dataset <pool> install

	   Creates  ZFS	 datasets  and	properties  on the specified pool. The
	   first pool is always	setup as the primary pool. This	is  the	 first
	   step	to getting chyves running.

       o   upgrade

	   chyves dataset <pool> upgrade

	   Upgrades  dataset version from an old version to the	newest version
	   incrementally. This is done by adding, removing, and	updating prop-
	   erties  which  are  required	by the latest version of chyves. It is
	   recommended to upgrade the primary pool first in case their are new
	   defaults, otherwise the 'chyves guest upgrade' will need to be man-
	   ually ran against the guests	on secondary pools.

   FIRMWARE SUB-COMMANDS
       These sub-commands manage UEFI firmware resources.

       o   import

	   chyves firmware import {URL|<local-path-to-firmware>}

	   Imports a firmware resource into chyves. Either a local  or	remote
	   source  can	be  given.  Remote  sources  can  be  from http	or ftp
	   sources.

       o   rename

	   chyves firmware rename {firmware-name} <desired-name>

	   Rename an firmware resource.

       o   delete

	   chyves firmware delete {firmware-name}

	   Delete an firmware resource.

       o   list

	   chyve firmware list

	   List	available firmware resources.

   GUEST SUB-COMMANDS
       o   clone

	   See GUEST CLONE SUB-SUB-COMMAND section below.

       o   console

	   See GUEST CONSOLE SUB-SUB-COMMANDS section below.

       o   create

	   chyves <guest> create [<size>] [<pool>]

	   Where guest is a uniquely identifying name used  to	reference  the
	   guest,  optionally  [size] is the size of the block device, and op-
	   tionally [pool] is the pool to create the guest  on	in  multi-pool
	   configurations.  If [size] is not supplied, then 'size' in defaults
	   is used. If [pool] is not supplied, then the	primary	pool is	 used.
	   A  tap interface is added and associated with the bridge set	in de-
	   faults. Support for multi-guest.

       o   delete

	   chyves <guest>|MG delete [force|keepnet]

	   Permanently delete a	guest from the host. If	the 'force' keyword is
	   used, then dependent	clones are deleted as well. The	'keepnet' key-
	   word	is used	to keep	the network associations, this is  useful  for
	   clones that share the exact properties.

       o   disk

	   See GUEST DISK SUB-SUB-COMMANDS section below.

       o   get

	   chyves <guest> get {<property>|all}

	   Gets	property for guests. See PROPERTIES section above for more de-
	   tail. Use 'chyves list global' or 'chyves list  defaults'  for  re-
	   spective property types.

       o   reclaim

	   chyves <guest>|MG reclaim

	   Reclaims a guest's VMM resources. This does not delete a guest from
	   the host, it	destroys the guest's VMM resources (bhyvectl  vernacu-
	   lar).

       o   rename

	   chyves <guest> rename <new-guest-name>

	   Renames the guest

       o   set

	   chyves <guest>|MG|global|defaults set [<prop1>=<value>]...

	   Sets	 ZFS properties	for guests. Using the guest name of 'defaults'
	   sets	the default values for newly created guests. Using  the	 guest
	   name	of 'global' sets global	properties.

       o   snapshot

	   See GUEST SNAPSHOT SUB-SUB-COMMANDS section below.

       o   start

	   chyves <guest>|MG start [<iso>]

	   Where  iso is optionally the	name of	an ISO resource, this media is
	   ejected after x amounts of boots as set by global or	guest property
	   'eject_iso_on_n_reboot'. This command starts	bhyveload, grub-bhyve,
	   and UEFI guests.

	   For grub-bhyve guests if a ISO is supplied, it is assumed to	be the
	   installation	ISO for	whichever OS is	set by the 'os'	property. This
	   means the device.map	and the	grub.cfg files generated reflect boot-
	   ing	from  the installation CD. If a	non-booting CD is needed to be
	   attached to a guest using grub-bhyve, then manually adding a	custom
	   PCIDEV device is suggested. See the pcidev{n}_ property under Guest
	   Properties section for instructions.

       o   stop

	   chyves <guest>|MG stop [force]

	   Attempts to gracefully stop	a  guest.  This	 is  effectively  like
	   pressing the	power button on	a computer tower, this means the guest
	   OS determines how to	respond. Running 'poweroff'  (or  the  equiva-
	   lent)  will	shutdown  a guest and the VMM resources	are reclaimed.
	   Using the [force] option forces the guest to	stop by	running	 'kill
	   -9' and then	reclaiming the guest's VMM resources. USE EXTREME CAU-
	   TION	WITH 'force' AND ONLY AS A LAST	RESORT,	THIS CAN CAUSE A  COR-
	   RUPT	OS.

       o   upgrade

	   chyves <guest>|MG upgrade

	   Upgrades  a guest to	the latest chyves guest	version. This add, re-
	   moves, and modifies guest properties	to be compatible with the  the
	   current  version  of	chyves.	See 'chyves_guest_version' guest prop-
	   erty	for details.

       o   unset

	   chyves <guest>|MG|all unset <property>

	   Only	the following properties may be	removed	from guests:

       o   eject_iso_on_n_reboot

       o   pcidev_{n}

       o   virtio_block_options_disk{n}

   GUEST CLONE SUB-SUB-COMMAND
       chyves <guest> clone <clonename>|MG [-ce|-cu|-ie|-iu] [<pool>]

       This clones a guest, there are four cloning methods that	 chyves	 uses.
       The  cloning  method  is	 determined  by	the clone flag after the clone
       name(s).	The first letter of the	flag indicates the clone's  dependency
       on  the	parent's  datasets,  the  second  letter indicates whether new
       unique properties are created for the clone. If the cloning method flag
       is  omitted,  the  default  is  used as defined in global property 'de-
       fault_clone_flag', this is initially set	to '-iu' as it is  the	safest
       and more	performant.

       The [pool] property is used to clone the	guest to a different pool than
       where the guest currently resides. This can only	be done	 for  indepen-
       dent clones.

       Dataset dependency:

       o   -c

	   Creates  a  ZFS  clone  of  the  guest's disks. Initially the clone
	   shares the same blocks as the parent	and only a few KB are consumed
	   to  point  the  clone  dataset  to  the parent's dataset. This also
	   means, the clones must  be  deleted	before	the  parent  guest  is
	   deleted.  See  'chyves list clones' to display the clone hierarchy.
	   The [pool] property is ignored as the clone must reside on the same
	   pool	as it's	parent.

       o   -i

	   Creates an independent clone	which is not dependent on the parent's
	   datasets. Using the [pool] property will clone  the	guest  to  the
	   specified pool, otherwise the guest's pool is used.

       Guest properties:

       o   -e

	   Copies the guest properties exactly as the parent's.	This means the
	   parent can not start	while the clone	is running, or vise-versa  due
	   to the overlapping serial and tap interfaces. Use the 'keepnet' op-
	   tion	when deleting these guests to prevent removing the network as-
	   sociation.

       o   -u

	   Sets	 new a unique: serial interface, a new tap is created for each
	   interface of	the parent under the same bridge, VALE interfaces  are
	   copied  over	 and  ports  removed,  a  new  unique port is used for
	   uefi_vnc_port, and a	new UUID is generated.

	   If the guest's property 'template' is set to	'yes', then the	 prop-
	   erty	 is  temporarily set to	'no' to	create unique properties, then
	   after cloning the guests and	clones have their property  'template'
	   set to 'yes'.

   GUEST CONSOLE SUB-SUB-COMMANDS
       Manage a	guest through a	console	using various methods described	below.
       The primary method for managing guests is through a  serial  interface.
       On supported hosts and UEFI guests, video emulation is possible through
       an internal VNC server attached to a frame  buffer.  See	 UEFI  SUPPORT
       above. All other	guests need to be administered via a serial communica-
       tion device.

       o   $null

	   chyves <guest> console

	   Establishes a serial	connection to a	 guest	using  cu(1)  via  the
	   nmdm(4) device as configure in the 'serial' property. To escape the
	   cu connection, press	tilde (~) twice, then  period  (.),  sometimes
	   hitting  ENTER  before  that	sequence is required. If that fails to
	   exit	the console, try pressing tilde	(~), then press	 Control  +  d
	   (^d). If all	else fails then	use 'chyves guest console reset'.

       o   reset

	   chyves <guest>|MG console reset

	   Resets  the	serial	connection for a guest by using	'kill' against
	   the cu processes associated with the	guest as set by	 the  'serial'
	   property.

       o   tmux

	   chyves <guest>|MG console tmux

	   Creates  a  tmux  session named chyves, then	creates	a window named
	   after the guest, and	then creates a vertically  splits  the	window
	   with	two panes. The right pane is intended for guest	management and
	   the left pane contains the serial interface which is	 started  with
	   'chyves guest console'. See TMUX SUPPORT section above.

       o   vnc

	   chyves <guest>|MG console vnc

	   Requires a UEFI guest that is configured for	VNC use	on a host that
	   supports UEFI graphics support. See UEFI SUPPORT above. Opens a X11
	   VNC	client to the guest using the uefi_vnc_client_*	global proper-
	   ties. See VNC CLIENT	SUPPORT	section	above.

   GUEST DISK SUB-SUB-COMMANDS
       These sub-commands deal with a guest's disk, multi-guest	 is  not  sup-
       ported.

       o   add

	   chyves <guest> disk add [<size>]

	   Adds	a new disk to the guest. The [size] field is optional and will
	   use the guest defaults when not specified.

       o   description

	   chyves <guest> disk disk{n} description <description>

	   Sets	a description property for a disk.

       o   list

	   chyves <guest> disk list

	   Lists the disks attached to a guest and each	size, description, and
	   notes for each disk.

       o   notes

	   chyves <guest> disk disk{n} notes <note>

	   Sets	a notes	property for a disk.

       o   remove

	   chyves <guest> disk remove disk{n}

	   Removes a disk from the guest.

       o   resize

	   chyves <guest> disk resize disk{n} <new-size>

	   Resizes  a  disk. THIS CAN BREAK THINGS ON THE GUEST	IF THE SIZE IS
	   REDUCED.

   GUEST SNAPSHOT SUB-SUB-COMMANDS
       Keep in mind that due to	the way	ZFS volumes work, when taking a	 snap-
       shot  all the bits must be able to be rewritten.	This means if you have
       a 8GB chyves disk with 6GB written to it	and you	take a snapshot, there
       will  be	 14GB  reserved	for the	disk until the snapshot	is deleted. If
       another 2GB is written and another snapshot is taken, then 16GB will be
       consumed	and so forth.

       o   $null

	   chyves <guest> snapshot [@snapshot]

	   Takes a snapshot of a guest and it's	child datasets.	Auto generates
	   a name when no name is given. Must start with an '@'	sign.  Appends
	   '-LIVE-SYSTEM'  to  the  snapshot name when the guest is running to
	   both	auto-generate and user supplied	names.

       o   delete

	   chyves <guest> delete [@snapshot]

	   chyves <guest> delete [@snapshot%[snapshot]]

	   Deletes a snapshot or a range of snapshots. Use the '%'  symbol  to
	   indicate  a	range  of snapshots to be deleted. If a	second name is
	   not specified then then all snapshots are deleted up	to the	speci-
	   fied	snapshot name. If a first name is not specified, then the lat-
	   est top level guest snapshot	 is  destroyed.	 Before	 deletion  the
	   amount of of	space that will	be recovered is	displayed.

       o   rollback

	   chyves <guest> snapshot rollback [@snapshot]

	   Rolls  back	a guest	and it's datasets to a previous	snapshot state
	   including guest properties. When no name is given, the latest snap-
	   shot	 is the	rollback point.	If there are multiple snapshots	in-be-
	   tween the given snapshot and	current	snapshot(s),  those  are  dis-
	   played  before  deletion.  A	guest can not be rollback past a clone
	   snapshot point until	those clones are deleted.

	   When	the 'network_design_mode' is set to 'auto' and there is	a dis-
	   crepancy  in	 the network configuration, the	conflicting tap	inter-
	   faces are added and removed from host configuration.	When a tap in-
	   terface  only  exists  on the current configuration,	the tap	is re-
	   moved. When a tap interface only exists in the snapshot state,  the
	   tap is added. Added tap interfaces are joined to the	default	bridge
	   device as the historic bridge associations are not tracked.

   INFO	COMMAND
       Displays	active guests and their	disks on the system when no flags  are
       used.  The  global  property 'default_info_flags' is used when no flags
       are given, this property	is populated with end user's preferred	flags.
       The  following  flags  are  used	 in combination	to display information
       about the guests	and disks. These flags are used	in combination,	 mean-
       ing  the	correct	syntax is "-rvs" rather	than "-r -v -s". See below for
       a description of	what each flag does:

       o   -z [size] Displays size, for	guests this show what is being used by
	   the dataset.	For disks the size displays the	volume size.

       o   -b [basic] CPU core count, and RAM allocation.

       o   -p [pool] Display the pool for the guest.

       o   -r [role] Display what role the pool	is set to.

       o   -v  [verbose]  Display OS, loader, chyves guest version, net	inter-
	   faces, and console port.

       o   -s [status] Display status of VMM , bhyve instance, and the	rcboot
	   value.

       o   -t [template] Displays if guest is a	template.

       o   -c [clone] Displays clone relationship, if any.

       o   -u  [uefi]  Print  UEFI  console  output type, firmware, VNC	IP and
	   port, mouse type, pause boot	until VNC  connect,  and  VNC  resolu-
	   tion.`

       o   -d [description] Displays description property.

       o   -n [notes] Dispays notes property.

       o   -a [all] Display all	fields above.

       o   -k [no-disks] Do not	display	guests'	disks.

       o   -h [help] Display this message.

       o   -w [truncate-to-width] Truncate output to the width of the terminal

       o   -l [less] Pipes the output to less.

   ISO SUB-COMMANDS
       These sub-commands manage ISO resources.

       o   import

	   chyves iso import {URL|<local-path-to-iso>}

	   Imports  an	ISO  resource  into  chyves.  Either a local or	remote
	   source can be give. Remote sources can be from  http	 or  ftp.  The
	   user	is prompted for	a file hash when a remote source is given. Af-
	   ter the download completes the file hash is compared	with  what  is
	   actually  hashed.  MD5, SHA1, SHA256, and SHA512 are	currently sup-
	   ported. If the hashes mis-match then	the user is  prompted  to  op-
	   tionally delete the file. If	no hash	is supplied, the user is heck-
	   led.

	   Supplied ISO	resources can be compressed with current  support  for
	   files ending	in .xz and .gz.	Upon import these files	are extracted.

       o   rename

	   chyves iso rename {ISO-name}	<desired-name>

	   Rename an ISO resource.

       o   delete

	   chyves iso delete {ISO-name}

	   Delete an ISO resource.

       o   list

	   chyves iso list

	   List	available ISO resources.

   LIST	SUB-COMMANDS
       Displays	information about various components for chyves.

       o   $null

	   Lists  properties  about  the  guests and their disks. User config-
	   urable, uses	same backend as	chyves info and	can be	configured  by
	   setting 'default_info_flags'.

       o   bridges

	   List	 bridges configured for	with 'chyves network' and the attached
	   physical or vlan interfaces.

       o   clones

	   List	dependent clones in a hierarchical tree.

       o   defaults

	   List	default	values newly created guests.

       o   disks

	   List	disks.

       o   firmware

	   Lists the UEFI firmware resources.

       o   global

	   Lists all the global	properties for all the chyves pools.

       o   global [<pool>|primary]

	   Lists all the global	properties for a chyves	pools. The word	 "pri-
	   mary" can be	used instead of	the primary pool's name.

       o   iso

	   Lists  the  ISO  resources.	More  ISO  resource commands under ISO
	   SUB-COMMANDS	section.

       o   pools

	   Lists all the pools and their roles.	Helpful	when  no  guests  have
	   been	created.

       o   processes

	   Lists all the processes *hyve processes.

       o   processes [<guest>]

	   Lists all the processes *hyve processes for a guest.

       o   properties

	   List	all properties on system.

       o   <property>

	   List	 the  property	given for all the guests. See Guest Properties
	   section above for a list

       o   snapshots

	   Lists all of	the top	level snapshots	for all	the guests.  Does  not
	   show	snapshots of disks (they are there, though).

       o   snapshots <guest>

	   Lists  all  of  the	snapshots  for a guest.	Does show snapshots of
	   disks and other child datasets.

       o   tap active

	   List	all the	tap interfaces from /dev/tap*

   NETWORK SUB-COMMANDS
       When global property 'network_design_mode' is set to 'auto', these com-
       mands  are  used	to create and manage network design by storing proper-
       ties about associate network devices and	creates	bridge and tap	inter-
       faces.  The sub-commands	below store properties that are	referenced and
       later used to verify and	(re)created the	network	design when starting a
       guest. The host still requires a	user set configuration of the vlan and
       physical	interfaces. chyves will	handle creating	 and  associating  the
       tap  interfaces with the	appropriate bridge interfaces and also attach-
       ing those configured vlan and physical interfaces to the	bridges	 while
       starting	a guest. Multiple bridges and taps per guest is	supported. See
       NETWORK section above.

       o   add

	   chyves network <guest> add {tap|tap{n}|vale{s}[:{p}]}

	   chyves network <guest> add {tap|tap{n}} bridge{n}

	   Used	to add a tap or	VALE interface to a guest, this	also adds  the
	   tap	interface  to  the default bridge as set in defaults. When the
	   'tap' keyword is used, the next available tap  interface  is	 used.
	   {n}	is  the	tap number, {s}	is the VALE switch number, and option-
	   ally	{p} is the port	on the VALE switch.

       o   attach

	   chyves network bridge{n} attach iface{n}

	   Used	to associate a vlan or physical	interface with a bridge.  Keep
	   in mind that	a physical or vlan interface can only be on one	bridge
	   group. See ifconfig(8) for details.

       o   default

	   chyves network bridge{n} default

	   Used	to set the default bridge to use for newly created guests.

       o   join

	   chyves network bridge{n} join tap{n}

	   Used	to add a tap to	a bridge interface.

       o   migrate

	   chyves network bridge{n} migrate bridge{n}

	   Used	to migrate a set of interfaces from  an	 existing  to  another
	   bridge  interface.  If migrating to an existing bridge the external
	   interface is	overwritten with the interface from the	 bridge	 being
	   migrated from and the tap interface lists are combined.

       o   private

	   chyves network bridge{n} private

	   Used	 to set	the bridge to private mode. This means there is	not an
	   external network connected. This can	be useful when sensitive  data
	   is  being exchanged over a network but not necessarily needing out-
	   side	access such as a SQL database. This could also	be  used  with
	   NAT.	However	NAT support is not build into chyves.

       o   remove

	   chyves network <guest> remove {tap{n}|vale{s}[:{p}]}

	   Used	 to remove a tap or vale interface from	a guest, this also re-
	   moves tap devices from the bridge configuration.

       o   unjoin

	   chyves network bridge{n} remove tap{n}

	   Used	to remove a tap	from a bridge interface.

EXAMPLES
       Install chyves on zpool named zroot:

	   chyves dataset zroot	install

       Configure bridge0 to use	em0 as the outside interface:

	   chyves network bridge0 attach em0

       Fetch FreeBSD install ISO for later:

	   chyves iso import ftp://ftp.example.org/FreeBSD-10.2-RELEASE-amd64.iso

       Create a	new guest named	bguest with the	default	size HDD:

	   chyves bguest create

       Create three new	guests named "gst1", "gst2", and "gst3"	each with a  6
       gigabyte	HDD on a pool called "dev-pool":

	   chyves gst1,gst2,gst3 create	6G dev-pool

       List ISO	and UEFI firmware resources:

	   chyves list iso
	   chyves list firmware

       or

	   chyves iso list
	   chyves firmware list

       Install the FreeBSD guest bguest:

	   chyves bguest install FreeBSD-10.2-RELEASE-amd64.iso

       Console into the	installation:

	   chyves bguest console

       Once  installation  is done, exit console (~~.),	and stop the guest (to
       prevent it from attempting to boot from the installation	media):

	   chyves bguest stop

       Reclaim the VMM resources, sometimes necessary:

	   chyves bguest reclaim

       Now that	the guest is installed,	it can be started like usual:

	   chyves bguest start

       Some guest OS's can be gracefully stopped using:

	   chyves bguest stop

       However some guest OS's need coercing:

	   chyves bguest stop force

       List all	guests:

	   chyves list

       Start all guests:

	   chyves all start

       Stop all	guests:

	   chyves all stop

       Change guest properties by using	set:

	   chyves bguest set ram=512M
	   chyves bguest set cpu=1
	   chyves bguest set os=netbsd

       Set a description or notes to help identify bguest:

	   chyves bguest set description="FreeBSD chyves guest"
	   chyves bguest set description="Runs internal	webserver using	nginx."

       You can also change multiple properties for one guest  in  one  command
       by:

	   chyves bguest set ram=512M cpu=1 os=netbsd loader=grub-bhyve

       Or change multiple guests property in one command by:

	   chyves bguest,bguest2,bguest3 set ram=512M description="b cluster"

       Or change multiple guests and multiple properties in one	command	by:

	   chyves bguest,bguest2,bguest3 set cpu=1 ram=512M bguest4 rcboot=50

       In  the above example this changes the ram to '512M' and	cpu to '1' for
       bguest, bguest2,	and bguest3. Then on bguest4, rcboot priority  is  set
       to '50'.

       Change  all current guests properties in	one command by using the 'all'
       keyword:

	   chyves all set cpu=6	ram=4G

       Change the default properties for future	created	guests:

	   chyves defaults set cpu=4 ram=4G

       List the	default	guest properties:

	   chyves list defaults

       Get a specific guest property for bguest:

	   chyves bguest get ram

       Get all bguest properties:

	   chyves bguest get all

       Display a lot of	information about guests and their disks:

	   chyves info -a

       Get a list of available flags for chyves	info:

	   chyves info -h

       Set your	preferred flags	for chyves info:

	   chyves global set default_info_flags="-zps"

       Set your	preferred flags	for chyves list:

	   chyves global set default_list_flags="-zps"

       List global properties:

	   chyves list global

       Snapshot	snapshot actions:

	   chyves bguest snapshot @beforeupdate
	   chyves list snapshots
	   chyves list snapshots bguest
	   chyves bguest snapshot rollback @beforeupdate

       To delete the default disk0 ZFS volume, create a	sparse 16G disk	image,
       and attach to bguest as a custom	bhyve PCI device:

	   chyves bguest disk delete disk0
	   truncate -s 16G /chyves/dozer/guests/bguest/01.img
	   chyves bguest set pcidev_0=ahci-hd,/chyves/dozer/guests/bguest/01.img

       Create a	true ZFS clone of 'bguest' with	unique properties:

	   chyves bguest clone -cu bguest2

       Create two true ZFS clones of 'bguest' with same	exact properties:

	   chyves bguest clone -ce exactCL1,exactCL2

       Create two independent clones of	'bguest' with unique properties:

	   chyves bguest clone -iu bguest2,bguest3

       Create independent clone	of 'bguest' with same exact properties:

	   chyves bguest clone -ie exactCL1

       Delete 'bguest2'	from system:

	   chyves '`bguest2`' delete

       Delete 'bguest' and dependent clones from system:

	   chyves '`bguest`' delete force

       Delete 'exactCL1' from system but keep network associations:

	   chyves '`exactCL1`' delete keepnet

       Install	and  run a Debian guest	(note LVM installs require "os"	be set
       to "d8lvm") Seems to work with installation disks version  8.0  through
       8.2,  however something was changed in 8.3 that breaks the installation
       disk from working.:

	   chyves debianvm create 8G
	   chyves debianvm set loader=grub-bhyve os=debian
	   chyves debianvm start debian-8.2.0-amd64-i386-netinst.iso
	   chyves debianvm console

       Install and run a Arch Linux guest :

	   chyves archguest create
	   chyves archguest set	loader=grub-bhyve os=arch
	   chyves archguest start archguest archlinux-2015.10.01-dual.iso
	   chyves archguest console

       Install and run a CentOS	or RHEL	 guest,	 (note	version	 6  would  use
       os=centos6):

	   chyves centosguest create
	   chyves centosguest set loader=grub-bhyve os=centos7 ram=392M
	   chyves centosguest start CentOS-7-x86_64-Everything-1511.iso
	   chyves centosguest console

       Install and run a Gentoo	guest:

	   chyves gentooguest create
	   chyves gentooguest set loader=grub-bhyve os=gentoo
	   chyves gentooguest start install-amd64-minimal-20160414.iso
	   chyves gentooguest console

       Install	and  run  an unlisted custom grub-bhyve	guest, see os property
       for details:

	   chyves customgst create
	   chyves customgst set	loader=grub-bhyve os=custom
	   <Edit grub.cfg and device.map>
	   chyves customgst start install.iso
	   chyves customgst console
	   <Installation completed>
	   chyves customgst stop	     # This is done to edit grub.cfg
	   <Edit grub.cfg and device.map>
	   chyves customgst start

       Install and run a Windows guest (On a host running FreeBSD 11.0-RC2  or
       later):

	   chyves windowsguest create 32g
	   chyves windowsguest set ram=2G
	   chyves windowsguest set bhyve_net_type=e1000
	   chyves windowsguest set loader=uefi
	   chyves windowsguest set uefi_firmware=<firmware>
	   chyves windowsguest set uefi_console_output=vnc
	   chyves windowsguest set uefi_vnc_pause_until_client_connect=yes
	   chyves windowsguest set eject_iso_on_n_reboot=3
	   chyves windowsguest start <reg_windows_installation.iso>
	   chyves windowsguest get uefi_vnc_ip
	   chyves windowsguest get uefi_vnc_port
	   <Connect using a VNC	client from the	above values.>
	   <Install OS and shutdown when completed.>
	   chyves windowsguest set uefi_vnc_pause_until_client_connect=no
	   chyves windowsguest unset eject_iso_on_n_reboot
	   <Install the	virtio-net drivers for better performance or on	old hosts.>
	   chyves iso import https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso
	   chyves windowsguest start virtio-win-0.1.96.iso
	   <Install virtio-net drivers.	Reboot.>

       Install and run a CoreOS	guest:

	   chyves coreos create	32g
	   chyves coreos set ram=1G loader=grub-bhyve os=coreos
	   chyves coreos start coreos_production_iso_image.iso
	   chyves coreos console
	   <After installation,	set a user name	and password.>
	   chyves coreos set os=coreossecure

AUTHOR
       Justin D	Holcomb	-- justinholcomb.me

SEE ALSO
       bhyve(8),  bhyveload(8),	 bridge(4), cu(1), dmesg(8), ethers(5),	ifcon-
       fig(8), netmap(4), nmdm(4), pf(4),  sh(1),  tap(3),  tmux(1),  vale(4),
       virtio(4), vtnet(4), zfs(8)

chyves			       September 11 2016		     CHYVES(8)

NAME | SYNOPSIS | SYNTAX NOMENCLATURE: | DESCRIPTION | DEPENDENCIES | ZFS | LOGS | NETWORK | PROPERTIES | TMUX SUPPORT | UEFI SUPPORT | COMMANDS | EXAMPLES | AUTHOR | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=chyves&sektion=8&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help