Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
Statistics::DescriptivUser Contributed Perl DocumentStatistics::Descriptive(3)

       Statistics::Descriptive - Module	of basic descriptive statistical

	 use Statistics::Descriptive;
	 $stat = Statistics::Descriptive::Full->new();
	 $stat->add_data(1,2,3,4); $mean = $stat->mean();
	 $var  = $stat->variance();
	 $tm   = $stat->trimmed_mean(.25);
	 $Statistics::Descriptive::Tolerance = 1e-10;

       This module provides basic functions used in descriptive	statistics.
       It has an object	oriented design	and supports two different types of
       data storage and	calculation objects: sparse and	full. With the sparse
       method, none of the data	is stored and only a few statistical measures
       are available. Using the	full method, the entire	data set is retained
       and additional functions	are available.

       Whenever	a division by zero may occur, the denominator is checked to be
       greater than the	value $Statistics::Descriptive::Tolerance, which
       defaults	to 0.0.	You may	want to	change this value to some small
       positive	value such as 1e-24 in order to	obtain error messages in case
       of very small denominators.

       Many of the methods (both Sparse	and Full) cache	values so that
       subsequent calls	with the same arguments	are faster.

   Sparse Methods
       $stat = Statistics::Descriptive::Sparse->new();
	    Create a new sparse	statistics object.

	    Effectively	the same as

	      my $class	= ref($stat);
	      undef $stat;
	      $stat = new $class;

	    except more	efficient.

	    Adds data to the statistics	variable. The cached statistical
	    values are updated automatically.

	    Returns the	number of data items.

	    Returns the	mean of	the data.

	    Returns the	sum of the data.

	    Returns the	variance of the	data.  Division	by n-1 is used.

	    Returns the	standard deviation of the data.	Division by n-1	is

	    Returns the	minimum	value of the data set.

	    Returns the	index of the minimum value of the data set.

	    Returns the	maximum	value of the data set.

	    Returns the	index of the maximum value of the data set.

	    Returns the	sample range (max - min) of the	data set.

   Full	Methods
       Similar to the Sparse Methods above, any	Full Method that is called
       caches the current result so that it doesn't have to be recalculated.
       In some cases, several values can be cached at the same time.

       $stat = Statistics::Descriptive::Full->new();
	    Create a new statistics object that	inherits from
	    Statistics::Descriptive::Sparse so that it contains	all the
	    methods described above.

	    Adds data to the statistics	variable.  All of the sparse
	    statistical	values are updated and cached.	Cached values from
	    Full methods are deleted since they	are no longer valid.

	    Note:  Calling add_data with an empty array	will delete all	of
	    your Full method cached values!  Cached values for the sparse
	    methods are	not changed

       $stat->add_data_with_samples([{1	=> 10},	{2 => 20}, {3 => 30},]);
	    Add	data to	the statistics variable	and set	the number of samples
	    each value has been	built with. The	data is	the key	of each
	    element of the input array ref, while the value is the number of
	    samples: [{data1 =>	smaples1}, {data2 => samples2},	...]

	    Returns a copy of the data array.

	    Returns a copy of the data array without outliers. The number
	    minimum of samples to apply	the outlier filtering is
	    $Statistics::Descriptive::Min_samples_number, 4 by default.

	    A function to detect outliers need to be defined (see
	    "set_outlier_filter"), otherwise the function will return an undef

	    The	filtering will act only	on the most extreme value of the data
	    set	(i.e.: value with the highest absolute standard	deviation from
	    the	mean).

	    If there is	the need to remove more	than one outlier, the
	    filtering need to be re-run	for the	next most extreme value	with
	    the	initial	outlier	removed.

	    This is not	always needed since the	test (for example Grubb's
	    test) usually can only detect the most exreme value. If there is
	    more than one extreme case in a set, then the standard deviation
	    will be high enough	to make	neither	case an	outlier.

	    Set	the function to	filter out the outlier.

	    $code_ref is the reference to the subroutine implemeting the
	    filtering function.

	    Returns "undef" for	invalid	values of $code_ref (i.e.: not defined
	    or not a code reference), 1	otherwise.

	    o	Example	#1: Undefined code reference

		    my $stat = Statistics::Descriptive::Full->new();
		    $stat->add_data(1, 2, 3, 4,	5);

		    print $stat->set_outlier_filter(); # => undef

	    o	Example	#2: Valid code reference

		    sub	outlier_filter { return	$_[1] >	1; }

		    my $stat = Statistics::Descriptive::Full->new();
		    $stat->add_data( 1,	1, 1, 100, 1, );

		    print $stat->set_outlier_filter( \&outlier_filter ); # => 1
		    my @filtered_data =	$stat->get_data_without_outliers();
		    # @filtered_data is	(1, 1, 1, 1)

		In this	example	the series is really simple and	the outlier
		filter function	as well.  For more complex series the outlier
		filter function	might be more complex (see Grubbs' test	for

		The outlier filter function will receive as first parameter
		the Statistics::Descriptive::Full object, as second the	value
		of the candidate outlier. Having the object in the function
		might be useful	for complex filters where statistics property
		are needed (again see Grubbs' test for outlier).

       $stat->set_smoother({ method => 'exponential', coeff => 0, });
	    Set	the method used	to smooth the data and the smoothing
	    coefficient.  See "Statistics::Smoother" for more details.

	    Returns a copy of the smoothed data	array.

	    The	smoothing method and coefficient need to be defined (see
	    "set_smoother"), otherwise the function will return	an undef

	    Sort the stored data and update the	mindex and maxdex methods.
	    This method	uses perl's internal sort.

	    If called with a non-zero argument,	this method sets a flag	that
	    says the data is already sorted and	need not be sorted again.
	    Since some of the methods in this class require sorted data, this
	    saves some time.  If you supply sorted data	to the object, call
	    this method	to prevent the data from being sorted again. The flag
	    is cleared whenever	add_data is called.  Calling the method
	    without an argument	returns	the value of the flag.

	    Returns the	skewness of the	data.  A value of zero is no skew,
	    negative is	a left skewed tail, positive is	a right	skewed tail.
	    This is consistent with Excel.

	    Returns the	kurtosis of the	data.  Positive	is peaked, negative is

       $x = $stat->percentile(25);
       ($x, $index) = $stat->percentile(25);
	    Sorts the data and returns the value that corresponds to the
	    percentile as defined in RFC2330:

	    o	For example, given the 6 measurements:

		-2, 7, 7, 4, 18, -5

		Then F(-8) = 0,	F(-5) =	1/6, F(-5.0001)	= 0, F(-4.999) = 1/6,
		F(7) = 5/6, F(18) = 1, F(239) =	1.

		Note that we can recover the different measured	values and how
		many times each	occurred from F(x) -- no information regarding
		the range in values is lost.  Summarizing measurements using
		histograms, on the other hand, in general loses	information
		about the different values observed, so	the EDF	is preferred.

		Using either the EDF or	a histogram, however, we do lose
		information regarding the order	in which the values were
		observed.  Whether this	loss is	potentially significant	will
		depend on the metric being measured.

		We will	use the	term "percentile" to refer to the smallest
		value of x for which F(x) >= a given percentage.  So the 50th
		percentile of the example above	is 4, since F(4) = 3/6 = 50%;
		the 25th percentile is -2, since F(-5) = 1/6 < 25%, and	F(-2)
		= 2/6 >= 25%; the 100th	percentile is 18; and the 0th
		percentile is -infinity, as is the 15th	percentile, which for
		ease of	handling and backward compatibility is returned	as
		undef()	by the function.

		Care must be taken when	using percentiles to summarize a
		sample,	because	they can lend an unwarranted appearance	of
		more precision than is really available.  Any such summary
		must include the sample	size N,	because	any percentile
		difference finer than 1/N is below the resolution of the

	    (Taken from: RFC2330 - Framework for IP Performance	Metrics,
	    Section 11.3.  Defining Statistical	Distributions.	RFC2330	is
	    available from: <> .)

	    If the percentile method is	called in a list context then it will
	    also return	the index of the percentile.

       $x = $stat->quantile($Type);
	    Sorts the data and returns estimates of underlying distribution
	    quantiles based on one or two order	statistics from	the supplied

	    This method	use the	same algorithm as Excel	and R language
	    (quantile type 7).

	    The	generic	function quantile produces sample quantiles
	    corresponding to the given probabilities.

	    $Type is an	integer	value between 0	to 4 :

	      0	=> zero	quartile (Q0) :	minimal	value
	      1	=> first quartile (Q1) : lower quartile	= lowest cut off (25%) of data = 25th percentile
	      2	=> second quartile (Q2)	: median = it cuts data	set in half = 50th percentile
	      3	=> third quartile (Q3) : upper quartile	= highest cut off (25%)	of data, or lowest 75% = 75th percentile
	      4	=> fourth quartile (Q4)	: maximal value

	    Exemple :

	      my @data = (1..10);
	      my $stat = Statistics::Descriptive::Full->new();
	      print $stat->quantile(0);	# => 1
	      print $stat->quantile(1);	# => 3.25
	      print $stat->quantile(2);	# => 5.5
	      print $stat->quantile(3);	# => 7.75
	      print $stat->quantile(4);	# => 10

	    Sorts the data and returns the median value	of the data.

	    Returns the	harmonic mean of the data.  Since the mean is
	    undefined if any of	the data are zero or if	the sum	of the
	    reciprocals	is zero, it will return	undef for both of those	cases.

	    Returns the	geometric mean of the data.

       my $mode	= $stat->mode();
	    Returns the	mode of	the data. The mode is the most commonly
	    occuring datum.  See
	    <> . If all
	    values occur only once, then mode()	will return undef.

	    "trimmed_mean(ltrim)" returns the mean with	a fraction "ltrim" of
	    entries at each end	dropped. "trimmed_mean(ltrim,utrim)" returns
	    the	mean after a fraction "ltrim" has been removed from the	lower
	    end	of the data and	a fraction "utrim" has been removed from the
	    upper end of the data.  This method	sorts the data before
	    beginning to analyze it.

	    All	calls to trimmed_mean()	are cached so that they	don't have to
	    be calculated a second time.

	    "frequency_distribution_ref($partitions)" slices the data into
	    $partition sets (where $partition is greater than 1) and counts
	    the	number of items	that fall into each partition. It returns a
	    reference to a hash	where the keys are the numerical values	of the
	    partitions used. The minimum value of the data set is not a	key
	    and	the maximum value of the data set is always a key. The number
	    of entries for a particular	partition key are the number of	items
	    which are greater than the previous	partition key and less then or
	    equal to the current partition key.	As an example,

	       $f = $stat->frequency_distribution_ref(2);
	       for (sort {$a <=> $b} keys %$f) {
		  print	"key = $_, count = $f->{$_}\n";


	       key = 2.5, count	= 4
	       key = 4,	count =	3

	    since there	are four items less than or equal to 2.5, and 3	items
	    greater than 2.5 and less than 4.

	    "frequency_distribution_refs(\@bins)" provides the bins that are
	    to be used for the distribution.  This allows for non-uniform
	    distributions as well as trimmed or	sample distributions to	be
	    found.  @bins must be monotonic and	contain	at least one element.
	    Note that unless the set of	bins contains the range	that the total
	    counts returned will be less than the sample size.

	    Calling "frequency_distribution_ref()" with	no arguments returns
	    the	last distribution calculated, if such exists.

       my %hash	= $stat->frequency_distribution($partitions);
       my %hash	= $stat->frequency_distribution(\@bins);
       my %hash	= $stat->frequency_distribution();
	    Same as "frequency_distribution_ref()" except that returns the
	    hash clobbered into	the return list. Kept for compatibility
	    reasons with previous versions of Statistics::Descriptive and
	    using it is	discouraged.

	    "least_squares_fit()" performs a least squares fit on the data,
	    assuming a domain of @x or a default of 1..$stat->count().	It
	    returns an array of	four elements "($q, $m,	$r, $rms)" where

	    "$q	and $m"
		satisfy	the equation C($y = $m*$x + $q).

	    $r	is the Pearson linear correlation cofficient.

		is the root-mean-square	error.

	    If case of error or	division by zero, the empty list is returned.

	    The	array that is returned can be "coerced"	into a hash structure
	    by doing the following:

	      my %hash = ();
	      @hash{'q', 'm', 'r', 'err'} = $stat->least_squares_fit();

	    Because calling "least_squares_fit()" with no arguments defaults
	    to using the current range,	there is no caching of the results.

       I read my email frequently, but since adopting this module I've added 2
       children	and 1 dog to my	family,	so please be patient about my response
       times.  When reporting errors, please include the following to help me

       o   Your	version	of perl.  This can be obtained by typing perl "-v" at
	   the command line.

       o   Which version of Statistics::Descriptive you're using.  As you can
	   see below, I	do make	mistakes.  Unfortunately for me, right now
	   there are thousands of CD's with the	version	of this	module with
	   the bugs in it.  Fortunately	for you, I'm a very patient module

       o   Details about what the error	is.  Try to narrow down	the scope of
	   the problem and send	me code	that I can run to verify and track it

       Current maintainer:

       Shlomi Fish, <> , ""


       Colin Kuskie

       My email	address	can be found at under Who's	Who or
       at: .

       Fabio Ponciroli & Adzuna	Ltd. team (outliers handling)

       RFC2330,	Framework for IP Performance Metrics

       The Art of Computer Programming,	Volume 2, Donald Knuth.

       Handbook	of Mathematica Functions, Milton Abramowitz and	Irene Stegun.

       Probability and Statistics for Engineering and the Sciences, Jay

       Copyright (c) 1997,1998 Colin Kuskie. All rights	reserved.  This
       program is free software; you can redistribute it and/or	modify it
       under the same terms as Perl itself.

       Copyright (c) 1998 Andrea Spinelli. All rights reserved.	 This program
       is free software; you can redistribute it and/or	modify it under	the
       same terms as Perl itself.

       Copyright (c) 1994,1995 Jason Kastner. All rights reserved.  This
       program is free software; you can redistribute it and/or	modify it
       under the same terms as Perl itself.

       This program is free software; you can redistribute it and/or modify it
       under the same terms as Perl itself.

perl v5.24.1			  2015-06-19	    Statistics::Descriptive(3)


Want to link to this manual page? Use this URL:

home | help