Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
Sync(3)		      User Contributed Perl Documentation	       Sync(3)

NAME
	 MLDBM::Sync - safe concurrent access to MLDBM databases

SYNOPSIS
	 use MLDBM::Sync;			# this gets the	default, SDBM_File
	 use MLDBM qw(DB_File Storable);	# use Storable for serializing
	 use MLDBM qw(MLDBM::Sync::SDBM_File);	# use extended SDBM_File, handles values > 1024	bytes
	 use Fcntl qw(:DEFAULT);		# import symbols O_CREAT & O_RDWR for use with DBMs

	 # NORMAL PROTECTED read/write with implicit locks per i/o request
	 my $sync_dbm_obj = tie	%cache,	'MLDBM::Sync' [..other DBM args..] or die $!;
	 $cache{"AAAA"}	= "BBBB";
	 my $value = $cache{"AAAA"};

	 # SERIALIZED PROTECTED	read/write with	explicit lock for both i/o requests
	 my $sync_dbm_obj = tie	%cache,	'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $sync_dbm_obj->Lock;
	 $cache{"AAAA"}	= "BBBB";
	 my $value = $cache{"AAAA"};
	 $sync_dbm_obj->UnLock;

	 # SERIALIZED PROTECTED	READ access with explicit read lock for	both reads
	 $sync_dbm_obj->ReadLock;
	 my @keys = keys %cache;
	 my $value = $cache{'AAAA'};
	 $sync_dbm_obj->UnLock;

	 # MEMORY CACHE	LAYER with Tie::Cache
	 $sync_dbm_obj->SyncCacheSize('100K');

	 # KEY CHECKSUMS, for lookups on MD5 checksums on large	keys
	 my $sync_dbm_obj = tie	%cache,	'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $sync_dbm_obj->SyncKeysChecksum(1);
	 my $large_key = "KEY" x 10000;
	 $sync{$large_key} = "LARGE";
	 my $value = $sync{$large_key};

DESCRIPTION
       This module wraps around	the MLDBM interface, by	handling concurrent
       access to MLDBM databases with file locking, and	flushes	i/o explicity
       per lock/unlock.	 The new [Read]Lock()/UnLock() API can be used to
       serialize requests logically and	improve	performance for	bundled	reads
       & writes.

	 my $sync_dbm_obj = tie	%cache,	'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;

	 # Write locked	critical section
	 $sync_dbm_obj->Lock;
	   ... all accesses to DBM LOCK_EX protected, and go to	same tied file handles
	   $cache{'KEY'} = 'VALUE';
	 $sync_dbm_obj->UnLock;

	 # Read	locked critical	section
	 $sync_dbm_obj->ReadLock;
	   ... all read	accesses to DBM	LOCK_SH	protected, and go to same tied files
	   ... WARNING,	cannot write to	DBM in ReadLock() section, will	die()
	   ... WARNING,	my $v =	$cache{'KEY'}{'SUBKEY'}	will trigger a write so	not safe
	   ...	 to use	in ReadLock() section
	   my $value = $cache{'KEY'};
	 $sync_dbm_obj->UnLock;

	 # Normal access OK too, without explicity locking
	 $cache{'KEY'} = 'VALUE';
	 my $value = $cache{'KEY'};

       MLDBM continues to serve	as the underlying OO layer that	serializes
       complex data structures to be stored in the databases.  See the MLDBM
       BUGS section for	important limitations.

       MLDBM::Sync also	provides built in RAM caching with Tie::Cache md5 key
       checksum	functionality.

INSTALL
       Like any	other CPAN module, either use CPAN.pm, or perl -MCPAN "-e"
       shell, or get the file MLDBM-Sync-x.xx.tar.gz, unzip, untar and:

	 perl Makefile.PL
	 make
	 make test
	 make install

LOCKING
       The MLDBM::Sync wrapper protects	MLDBM databases	by locking and
       unlocking around	read and write requests	to the databases.  Also
       necessary is for	each new lock to tie() to the database internally,
       untie()ing when unlocking.  This	flushes	any i/o	for the	dbm to the
       operating system, and allows for	concurrent read/write access to	the
       databases.

       Without any extra effort	from the developer, an existing	MLDBM database
       will benefit from MLDBM::sync.

	 my $dbm_obj = tie %dbm, ...;
	 $dbm{"key"} = "value";

       As a write or STORE operation, the above	will automatically cause the
       following:

	 $dbm_obj->Lock; # also	ties
	 $dbm{"key"} = "value";
	 $dbm_obj->UnLock; # also unties

       Just so,	a read or FETCH	operation like:

	 my $value = $dbm{"key"};

       will really trigger:

	 $dbm_obj->ReadLock; # also ties
	 my $value = $dbm{"key"};
	 $dbm_obj->Lock; # also	unties

       However,	these lock operations are expensive because of the underlying
       tie()/untie() that occurs for i/o flushing, so when bundling reads &
       writes, a developer may explicitly use this API for greater
       performance:

	 # tie once to database, write 100 times
	 $dbm_obj->Lock;
	 for (1..100) {
	   $dbm{$_} = $_ * 100;
	   ...
	 }
	 $dbm_obj->UnLock;

	 # only	tie once to database, and read 100 times
	 $dbm_obj->ReadLock;
	 for(1..100) {
	   my $value = $dbm{$_};
	   ...
	 }
	 $dbm_obj->UnLock;

CACHING
       I built MLDBM::Sync to serve as a fast and robust caching layer for use
       in multi-process	environments like mod_perl.  In	order to provide an
       additional speed	boost when caching static data,	I have added an	RAM
       caching layer with Tie::Cache, which regulates the size of the memory
       used with its MaxBytes setting.

       To activate this	caching, just:

	 my $dbm = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $dbm->SyncCacheSize(100000);  # 100000	bytes max memory used
	 $dbm->SyncCacheSize('100K');  # 100 Kbytes max	memory used
	 $dbm->SyncCacheSize('1M');    # 1 Megabyte max	memory used

       The ./bench/bench_sync.pl, run like "bench_sync.pl "-c""	will run the
       tests with caching turned on creating a benchmark with 50% cache	hits.

       One run without caching was:

	=== INSERT OF 50 BYTE RECORDS ===
	 Time for 100 writes + 100 reads for  SDBM_File			 0.16 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.17 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  GDBM_File			 3.37 seconds	  17980	bytes
	 Time for 100 writes + 100 reads for  DB_File			 4.45 seconds	  20480	bytes

       And with	caching, with 50% cache	hits:

	=== INSERT OF 50 BYTE RECORDS ===
	 Time for 100 writes + 100 reads for  SDBM_File			 0.11 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.11 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  GDBM_File			 2.49 seconds	  17980	bytes
	 Time for 100 writes + 100 reads for  DB_File			 2.55 seconds	  20480	bytes

       Even for	SDBM_File, this	speedup	is near	33%.

KEYS CHECKSUM
       A common	operation on database lookups is checksumming the key, prior
       to the lookup, because the key could be very large, and all one really
       wants is	the data it maps too.  To enable this functionality
       automatically with MLDBM::Sync, just:

	 my $sync_dbm_obj = tie	%cache,	'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $sync_dbm_obj->SyncKeysChecksum(1);

	!! WARNING: keys() & each() do not work	on these databases
	!! as of v.03, so the developer	will not be fooled into	thinking
	!! the stored key values are meaningful	to the calling application
	!! and will die() if called.
	!!
	!! This	behavior could be relaxed in the future.

       An example of this might	be to cache a XSLT conversion, which are
       typically very expensive.  You have the XML data	and the	XSLT data, so
       all you do is:

	 # $xml_data, $xsl_data	are strings
	 my $xslt_output;
	 unless	($xslt_output =	$cache{$xml_data.'&&&&'.$xsl_data}) {
	   ... do XSLT conversion here for $xslt_output	...
	   $cache{$xml_data.'&&&&'.xsl_data} = $xslt_output;
	 }

       What you	save by	doing this is having to	create HUGE keys to lookup on,
       which no	DBM is likely to do efficiently.  This is the same method that
       File::Cache uses	internally to hash its file lookups in its
       directories.

New MLDBM::Sync::SDBM_File
       SDBM_File, the default used for MLDBM and therefore MLDBM::Sync has a
       limit of	1024 bytes for the size	of a record.

       SDBM_File is also an order of magnitude faster for small	records	to use
       with MLDBM::Sync, than DB_File or GDBM_File, because the	tie()/untie()
       to the dbm is much faster.  Therefore, bundled with MLDBM::Sync release
       is a MLDBM::Sync::SDBM_File layer which works around this 1024 byte
       limit.  To use, just:

	 use MLDBM qw(MLDBM::Sync::SDBM_File);

       It works	by breaking up up the STORE() values into small	128 byte
       segments, and spreading those segments across many records, creating a
       virtual record layer.  It also uses Compress::Zlib to compress STORED
       data, reducing the number of these 128 byte records. In benchmarks, 128
       byte record segments seemed to be a sweet spot for space/time
       efficiency, as SDBM_File	created	very bloated *.pag files for 128+ byte
       records.

BENCHMARKS
       In the distribution ./bench directory is	a bench_sync.pl	script that
       can benchmark using the various DBMs with MLDBM::Sync.

       The MLDBM::Sync::SDBM_File DBM is special because is uses SDBM_File for
       fast small inserts, but slows down linearly with	the size of the	data
       being inserted and read.

       The results for a dual PIII-450 linux 2.4.7, with a ext3	file system
       blocksize 4096 mounted async on a RAID-1	2xIDE 7200 RPM disk were as
       follows:

	=== INSERT OF 50 BYTE RECORDS ===
	 Time for 100 writes + 100 reads for  SDBM_File			 0.16 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.19 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  GDBM_File			 1.09 seconds	  18066	bytes
	 Time for 100 writes + 100 reads for  DB_File			 0.67 seconds	  12288	bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 0.31 seconds	  13192	bytes

	=== INSERT OF 500 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.52 seconds	 110592	bytes
	 Time for 100 writes + 100 reads for  GDBM_File			 1.20 seconds	  63472	bytes
	 Time for 100 writes + 100 reads for  DB_File			 0.66 seconds	  86016	bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 0.32 seconds	  58192	bytes

	=== INSERT OF 5000 BYTE	RECORDS	===
	(skipping test for SDBM_File 100 byte limit)
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 1.41 seconds	1163264	bytes
	 Time for 100 writes + 100 reads for  GDBM_File			 1.38 seconds	 832400	bytes
	 Time for 100 writes + 100 reads for  DB_File			 1.21 seconds	 831488	bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 0.58 seconds	 508192	bytes

	=== INSERT OF 20000 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	(skipping test for MLDBM::Sync db size > 1M)
	 Time for 100 writes + 100 reads for  GDBM_File			 2.23 seconds	2063912	bytes
	 Time for 100 writes + 100 reads for  DB_File			 1.89 seconds	2060288	bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 1.26 seconds	2008192	bytes

	=== INSERT OF 50000 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	(skipping test for MLDBM::Sync db size > 1M)
	 Time for 100 writes + 100 reads for  GDBM_File			 3.66 seconds	5337944	bytes
	 Time for 100 writes + 100 reads for  DB_File			 3.64 seconds	5337088	bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 2.80 seconds	5008192	bytes

AUTHORS
       Copyright (c) 2001-2002 Joshua Chamas, Chamas Enterprises Inc.  All
       rights reserved.	 Sponsored by development on NodeWorks
       http://www.nodeworks.com	and Apache::ASP	http://www.apache-asp.org

       This program is free software; you can redistribute it and/or modify it
       under the same terms as Perl itself.

SEE ALSO
	MLDBM(3), SDBM_File(3),	DB_File(3), GDBM_File(3)

perl v5.32.1			  2002-07-03			       Sync(3)

NAME | SYNOPSIS | DESCRIPTION | INSTALL | LOCKING | CACHING | KEYS CHECKSUM | New MLDBM::Sync::SDBM_File | BENCHMARKS | AUTHORS | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=MLDBM::Sync&sektion=3&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help