Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
MOGTOOL(1)	      User Contributed Perl Documentation	    MOGTOOL(1)

       mogtool -- Inject/extract data to/from a	MogileFS installation

       WARNING:	this utility is	deprecated! See	MogileFS::Utils

	   $ mogtool [general-opts] <command> [command-opts] <command-args>

	   $ mogtool --trackers= --domain=foo --class=bar	...
	   $ mogtool --conf=foo.conf ...

	   $ mogtool inject thefile.tgz	thefilekey
	   $ mogtool inject --bigfile thebigfile.tgz thefilekey
	   $ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
	   $ mogtool inject --bigfile --gzip mydirectory thedirkey
	   $ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
	   $ mogtool inject --nobigfile	bigcontiguousfile bigcfilekey

	   $ mogtool inject --bigfile --gzip --verify \
			    --description="Description"	\
			    --receipt="," \
			    --concurrent=5 --chunksize=32M \
			    somehugefile thefilekey

	   $ mogtool extract thefilekey	thenewfile.tgz
	   $ mogtool extract thefilekey	-
	   $ mogtool extract --bigfile thedirkey .
	   $ mogtool extract --bigfile --asfile	thedirkey thefile.tgz
	   $ mogtool extract --bigfile thedevkey /dev/hda4

	   $ mogtool delete thekey

	   $ mogtool locate --noverify thekey
	   $ mogtool locate --bigfile thekey

	   $ mogtool list
	   $ mogtool listkey key

	   Turn	on MogileFS debug output.

	   Specify one or more trackers	for your MogileFS installation.	 Note
	   that	you can	specify	preferred IPs to override the default IPs
	   with.  So it	would look something like

	   Set the MogileFS domain to use.

	   Set the class within	the domain to use.  Defaults to	_default.

	   Specify a configuration file	to load	from.

	   Specify a directory to use as a library path.  Right	now, this
	   should be the directory where you expect to find the
	   file, if it's not actually installed.

	   Insert a resource into MogileFS.  See "INJECT OPTIONS" and "INJECT
	   ARGUMENTS" for the rest of how to use the inject mode.

	   Extract a resource from MogileFS.  See "EXTRACT OPTIONS" and
	   "EXTRACT ARGUMENTS" for how to use extract.

	   Delete a resource.  See "DELETE OPTIONS" and	"DELETE	ARGUMENTS".

       locate|lo key
	   List	the paths to the file identified by the	given key.

	   List	all big	files contained	in MogileFS.  No options, no

       listkey|lsk key
	   List	all files which	match the key. Key is just a prefix, and this
	   will	list all keys which match the prefix. So if you	specify	key as
	   "ABC1" then you'll get all keys which start with the	characters

       The following options are used to control the behavior of the injector.

	   If specified, use chunking to break the resource into manageable

	   When	instructed to break files into chunks, the injector will use
	   the specified chunk size as the maximum chunk size.	Defaults to
	   64M.	 You can specify the chunk size	manually and specify the
	   units--defaults to bytes.

	   If specified, mogtool will gzip the data as it's going into
	   MogileFS.  The resource will	be marked as compressed.

	   Note	that you do not	need to	specify	this if	the resource is
	   already gzipped, but	it doesn't hurt.  (We automatically detect
	   that	and mark it as compressed.)

	   If you previously were working on injecting a big file as chunks
	   and the process died, normally mogtool refuses to do	it again.
	   Specify this	option to force	the overwrite of that file.

	   NOTE: Other than in the above case (partial failure), mogtool will
	   not prompt before overwriting an existing file.

	   If on, we do	a full MD5 verification	of every chunk after it	is
	   replicated.	This can take a	while on large files!

	   Specifies a description for this file.  Optional, but assists in
	   reporting and listing the large files in MogileFS.  (This is	also
	   displayed in	any receipts that are created.)

       --receipt=<email	address>[, <email address>]*
	   If specified, emails	a copy of the receipt file to the specified
	   comma-separated email addresses.  Also creates a local filesystem
	   copy	of the receipt file.

	   Specifies the number	of concurrent processes	to run for MogileFS
	   insertion.  If you are noticing mogtool spend most of it's time
	   waiting for children	and not	actually buffering data, you may wish
	   to raise this number.  The default is 1 but we've found 3 or	4 work

	   What	you actually want to inject.  This can be a file, directory,
	   or a	raw partition in the format /dev/X.

	   Please see "USAGE EXAMPLES" for more	information on how to inject
	   these different types of resources and the differences thereof.

       key Specifies the key to	save this file to.  For	big files, the key is
	   actually "_big_N:key" and "key,#" where N is	one of a bunch of
	   things we use and # is the chunk number.

	   Generally, you want this to be descriptive so you remember what it
	   is later and	can identify the file just by looking at the key.

	   If specified, indicates that	this resource was chunked on injection
	   and should be reassembled for extraction.

	   Specifies to	mogtool	that it	should ungzip the output if and	only
	   if it was compressed	when inserted into the MogileFS	system.	 So,
	   if you're extracting	a file that wasn't gzipped to begin with, this
	   doesn't do anything.

	   Useful when extracting something previously inserted	as a
	   directory--this option instructs mogtool to treat the resource as a
	   file	and not	actually run it	through	tar for	decompression.

       key Specifies the key to	get the	file from.

	   What	destination means varies depending on what type	of resource
	   you're extracting.  However,	no matter what,	you can	specify	a
	   single dash (-) to mean STDOUT.

	   Please see the usage	examples for more information on how extract

	   The resource	is a "big file"	and all	chunks should be deleted.

       key Specifies the key of	the file to delete.

	   Verify that the returned paths actually contain the file.  The
	   locate commands defaults to verify, you can disable it with

	   The resource	is a "big file"	and the	locations of the information
	   key should be printed.

       key Specifies the key of	the file to locate

       0   Success during operation.

       1   During the locate, list, or listkey operation, the key was not

       2   Some	fatal error occurred.

       Please note that	all examples assume you	have a default config file
       that contains the tracker and domain to use.  Saves us from having to
       clutter up the command line.

   Small Files (<64MB)
       When it comes to	using small files, mogtool is very, very easy.


	   $ mogtool inject foo.dbm foo.dbm.2004.12

       Injects the file	foo.dbm	into MogileFS under the	key of

	   $ mogtool inject --gzip foo.dbm foo.dbm.2004.12

       Injects the same	file to	the same key, but compresses it	on the fly for


	   $ mogtool extract foo.dbm.2004.12 newfoo.dbm

       Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.

	   $ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm

       Gets the	file and automatically decompresses it,	if and only if it was
       compressed.  So basically, you can turn on gzip in your config file and
       mogtool will do the smart thing each time.

	   $ mogtool extract foo.dbm.2004.12 -

       Print the resource to standard out.  If you want, you can pipe it
       somewhere or redirect to	a file (but why	not just specify the

   Large Files (>64MB)
       Given mogtool's ability to break	files into chunks and later reassemble
       them, inserting large files (even files over the	4GB barrier) is
       relatively easy.


	   $ mogtool inject --bigfile largefile.dat largefile.dat

       As expected, inserts the	file largefile.dat into	the MogileFS system
       under the name largefile.dat.  Not very creative.  Uses the default
       64MB chunks.

	   $ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat

       Specify to use 16MB chunks instead of the default.  Otherwise, the

	   $ mogtool inject --bigfile --chunksize=1000K	--gzip largefile.dat somekey

       Do it again, but	specify	1000KB chunks, gzip automatically, and upload
       it under	a different key	somekey.

	   $ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey

       Same as above, but use 5	children processes for uploading chunks	to
       MogileFS.  This can take	up to 300MB of memory in this example!	(It
       tends to	use about (concurrency + 1) * chunksize	bytes.)

	   $ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
		     --receipt="" --verify --description="A large file" \
		     largefile.dat somekey

       Break this file into 128MB chunks, set a	description, use 3 children to
       upload them, gzip the file as you go, do	a full MD5 verification	of
       every chunk, then email a receipt with all of the MogileFS paths	to me.

       Lots of flexibility with	mogtool.


	   $ mogtool extract --bigfile somekey newfile.dat

       In its basic form, extracts the previously inserted large file and
       saves it	as newfile.dat.

	   $ mogtool extract --bigfile --gzip somekey newfile.dat

       If the file was gzipped on entry, ungzip	it and save the	result.	 If it
       wasn't gzipped, then we just save it.

       Directories are easily injected and extracted with mogtool.  To create
       the data	stream that is inserted	into MogileFS, we use tar.


	   $ mogtool inject --bigfile mydir mykey

       Run mydir through tar and then save it as mykey.

	   $ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey

       Inject, but also	gzip and use multiple injectors.

       Note how	this is	just like injecting a large file.  See injection
       examples	for large files	for more examples.


	   $ mogtool extract --bigfile mykey .

       Extract the previously injected directory mykey to your local

	   $ mogtool extract --bigfile --asfile	mykey foo.tar

       Take the	previously generated tarball and save it as foo.tar.  Simply
       creates the file	instead	of extracting everything inside.

       mogtool has the ability to inject raw partitions	into MogileFS and to
       retrieve	them later and write them back to a partition.	They're
       treated just like directories for the most part,	we just	don't pipe
       things through tar.


	   $ mogtool inject --bigfile /dev/hda3	hda3.backup

       Save a raw copy of your partition /dev/hda3 to the key hda3.backup.

	   $ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup

       Same, but compress on the fly during injection.


	   $ mogtool extract --bigfile hda3.backup /dev/hda4

       Extract the partition at	hda3.backup to the partition /dev/hda4.
       WARNING:	mogtool	won't ask for confirmation, make sure you don't
       mistype partition numbers!

   Deleting a Resource
       WARNING:	Please make sure you're	specifying the right parameter,	as
       delete does not prompt for confirmation of the request!

	   $ mogtool delete thekey

       Delete a	normal file.

	   $ mogtool delete --bigfile thekey

       Delete a	chunked	file--this deletes all chunks and the receipt, so the
       file is gone.

   Listing Big Files
	   $ mogtool list backup

       Lists all large files stored in MogileFS.  It is	not possible to	list
       all normal files	at this	time.

   Listing Files Matching a Key
	   $ mogtool listkey abc1

       Lists all files in MogileFS whose keys start with the characters

       Instead of adding a ton of options to the command line every time,
       mogtool enables you to create a default configuration file that it will
       read all	of the options from.  It searches two locations	for a default
       configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf.
       (Alternately, you can specify --conf=whatever as	an option on the
       command line.)

       The file	can consist of any number of the following items:

	   trackers =,
	   domain = mogiledomain
	   class = fileclass
	   lib = /home/foo/lib
	   gzip	= 1
	   big = 1
	   overwrite = 1
	   chunksize = 32M
	   receipt =,
	   verify = 1
	   concurrent =	3

       None?  Send me any you find!  :)

       --concurrent for	extract
	   It would be nice to have concurrent extraction going	on.

       recover mode
	   If the receipt file is ever corrupt in MogileFS it would be useful
	   to recover a	file given just	a receipt.  It would have the same
	   arguments as	the extract mode, except use a receipt file as the
	   data	source.

       partition size verification
	   We can easily get the partition size	when we	save one to MogileFS,
	   so we should	use that information to	determine during extraction if
	   a target partition is going to be big enough.

       on the fly gzip extraction
	   Right now we	can gzip on an injection, but we should	support	doing
	   decompression on the	fly coming out of MogileFS.

       make list take a	prefix
	   If you can specify a	prefix,	that makes things easier for finding
	   small files that are	stored in MogileFS.

       more information	on list
	   Have	list load up the info file and parse it	for information	about
	   each	of the big files being stored.	Maybe have this	as an option
	   (-l).  (This	means the reading and parsing of info files should be
	   abstracted into a function.)

       Mark Smith <> - most of the implementation and

       Brad Fitzpatrick	<> - concepts and	rough draft.

       Robin H.	Johnson	<> - locate function.

       Copyright (c) 2002-2004 Danga Interactive. All rights reserved.

       Hey! The	above document had some	coding errors, which are explained

       Around line 289:
	   Expected text after =item, not a number

       Around line 293:
	   Expected text after =item, not a number

perl v5.32.1			  2011-01-05			    MOGTOOL(1)


Want to link to this manual page? Use this URL:

home | help