Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
HTML::TableExtract(3) User Contributed Perl DocumentationHTML::TableExtract(3)

NAME
       HTML::TableExtract - Perl module	for extracting the content contained
       in tables within	an HTML	document, either as text or encoded element
       trees.

SYNOPSIS
	# Matched tables are returned as table objects;	tables can be matched
	# using	column headers,	depth, count within a depth, table tag
	# attributes, or some combination of the four.

	# Example: Using column	header information.
	# Assume an HTML document with tables that have	"Date",	"Price", and
	# "Cost" somewhere in a	row. The columns beneath those headings	are
	# what you want	to extract. They will be returned in the same order as
	# you specified	the headers since 'automap' is enabled by default.

	use HTML::TableExtract;
	my $te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
	$te->parse($html_string);

	# Examine all matching tables
	foreach	my $ts ($te->tables) {
	  print	"Table (", join(',', $ts->coords), "):\n";
	  foreach my $row ($ts->rows) {
	     print join(',', @$row), "\n";
	  }
	}

	# Shorthand...top level	rows() method assumes the first	table found in
	# the document if no arguments are supplied.
	foreach	my $row	($te->rows) {
	   print join(',', @$row), "\n";
	}

	# Example: Using depth and count information.
	# Every	table in the document has a unique depth and count tuple, so
	# when both are	specified it is	a unique table.	Depth and count	both
	# begin	with 0,	so in this case	we are looking for a table (depth 2)
	# within a table (depth	1) within a table (depth 0, which is the top
	# level	HTML document).	In addition, it	must be	the third (count 2)
	# such instance	of a table at that depth.

	my $te = HTML::TableExtract->new( depth	=> 2, count => 2 );
	$te->parse_file($html_file);
	foreach	my $ts ($te->tables) {
	   print "Table	found at ", join(',', $ts->coords), ":\n";
	   foreach my $row ($ts->rows) {
	      print "	", join(',', @$row), "\n";
	   }
	}

	# Example: Using table tag attributes.
	# If multiple attributes are specified,	all must be present and	equal
	# for match to occur.

	my $te = HTML::TableExtract->new( attribs => { border => 1 } );
	$te->parse($html_string);
	foreach	my $ts ($te->tables) {
	  print	"Table with border=1 found at ", join(',', $ts->coords), ":\n";
	  foreach my $row ($ts->rows) {
	     print "   ", join(',', @$row), "\n";
	  }
	}

	# Example: Extracting as an HTML::Element tree structure
	# Rather than extracting raw text, the html can	be converted into a
	# tree of element objects. The HTML document is	composed of
	# HTML::Element	objects	and the	tables are HTML::ElementTable
	# structures. Using this, the contents of tables within	a document can
	# be edited in-place.

	use HTML::TableExtract qw(tree);
	my $te = HTML::TableExtract->new( headers => qw(Fee Fie	Foe Fum) );
	$te->parse_file($html_file);
	my $table = $te->first_table_found;
	my $table_tree = $table->tree;
	$table_tree->cell(4,4)->replace_content('Golden	Goose');
	my $table_html = $table_tree->as_HTML;
	my $table_text = $table_tree->as_text;
	my $document_tree = $te->tree;
	my $document_html = $document_tree->as_HTML;

DESCRIPTION
       HTML::TableExtract is a subclass	of HTML::Parser	that serves to extract
       the information from tables of interest contained within	an HTML
       document. The information from each extracted table is stored in	table
       objects.	Tables can be extracted	as text, HTML, or HTML::ElementTable
       structures (for in-place	editing	or manipulation).

       There are currently four	constraints available to specify which tables
       you would like to extract from a	document: Headers, Depth, Count, and
       Attributes.

       Headers,	the most flexible and adaptive of the techniques, involves
       specifying text in an array that	you expect to appear above the data in
       the tables of interest. Once all	headers	have been located in a row of
       that table, all further cells beneath the columns that matched your
       headers are extracted. All other	columns	are ignored: think of it as
       vertical	slices through a table.	In addition, TableExtract
       automatically rearranges	each row in the	same order as the headers you
       provided. If you	would like to disable this, set	automap	to 0 during
       object creation,	and instead rely on the	column_map() method to find
       out the order in	which the headers were found. Furthermore,
       TableExtract will automatically compensate for cell span	issues so that
       columns are really the same columns as you would	visually see in	a
       browser.	This behavior can be disabled by setting the gridmap parameter
       to 0. HTML is stripped from the entire textual content of a cell	before
       header matches are attempted -- unless the keep_html parameter was
       enabled.

       Depth and Count are more	specific ways to specify tables	in relation to
       one another. Depth represents how deeply	a table	resides	in other
       tables. The depth of a top-level	table in the document is 0. A table
       within a	top-level table	has a depth of 1, and so on. Each depth	can be
       thought of as a layer; tables sharing the same depth are	on the same
       layer. Within each of these layers, Count represents the	order in which
       a table was seen	at that	depth, starting	with 0.	Providing both a depth
       and a count will	uniquely specify a table within	a document.

       Attributes match	based on the attributes	of the html <table> tag, for
       example,	border widths or background color.

       Each of the Headers, Depth, Count, and Attributes specifications	are
       cumulative in their effect on the overall extraction.  For instance, if
       you specify only	a Depth, then you get all tables at that depth (note
       that these could	very well reside in separate higher- level tables
       throughout the document since depth extends across tables).  If you
       specify only a Count, then the tables at	that Count from	all depths are
       returned	(i.e., the nth occurrence of a table at	each depth). If	you
       only specify Headers, then you get all tables in	the document
       containing those	column headers.	If you have specified multiple
       constraints of Headers, Depth, Count, and Attributes, then each
       constraint has veto power over whether a	particular table is extracted.

       If no Headers, Depth, Count, or Attributes are specified, then all
       tables match.

       When extracting only text from tables, the text is decoded with
       HTML::Entities by default; this can be disabled by setting the decode
       parameter to 0.

   Extraction Modes
       The default mode	of extraction for HTML::TableExtract is	raw text or
       HTML. In	this mode, embedded tables are completely decoupled from one
       another.	In this	case, HTML::TableExtract is a subclass of
       HTML::Parser:

	 use HTML::TableExtract;

       Alternatively, tables can be extracted as HTML::ElementTable
       structures, which are in	turn embedded in an HTML::Element tree
       representing the	entire HTML document. Embedded tables are not
       decoupled from one another since	this tree structure must be
       maintained. In this case, HTML::TableExtract is a subclass of
       HTML::TreeBuilder (itself a subclass of HTML:::Parser):

	 use HTML::TableExtract	qw(tree);

       In either case, the basic interface for HTML::TableExtract and the
       resulting table objects remains the same	-- all that changes is what
       you can do with the resulting data.

       HTML::TableExtract is a subclass	of HTML::Parser, and as	such inherits
       all of its basic	methods	such as	"parse()" and "parse_file()". During
       scans, "start()", "end()", and "text()" are utilized. Feel free to
       override	them, but if you do not	eventually invoke them in the SUPER
       class with some content,	results	are not	guaranteed.

   Advice
       The main	point of this module was to provide a flexible method of
       extracting tabular information from HTML	documents without relying to
       heavily on the document layout. For that	reason,	I suggest using
       Headers whenever	possible -- that way, you are anchoring	your
       extraction on what the document is trying to communicate	rather than
       some feature of the HTML	comprising the document	(other than the	fact
       that the	data is	contained in a table).

METHODS
       The following are the top-level methods of the HTML::TableExtract
       object. Tables that have	matched	a query	are actually returned as
       separate	objects	of type	HTML::TableExtract::Table. These table objects
       have their own methods, documented further below.

   CONSTRUCTOR
       new()
	   Return a new	HTML::TableExtract object. Valid attributes are:

	   headers
	       Passed as an array reference, headers specify strings of
	       interest	at the top of columns within targeted tables. They can
	       be either strings or regular expressions	(qr//).	If they	are
	       strings,	they will eventually be	passed through a non-anchored,
	       case-insensitive	regular	expression, so regexp special
	       characters are allowed.

	       The table row containing	the headers is not returned, unless
	       "keep_headers" was specified or you are extracting into an
	       element tree. In	either case the	header row can be accessed via
	       the hrow() method from within the table object.

	       Columns that are	not beneath one	of the provided	headers	will
	       be ignored unless "slice_columns" was set to 0. Columns will,
	       by default, be rearranged into the same order as	the headers
	       you provide (see	the automap parameter for more information)
	       unless "slice_columns" is 0.

	       Additionally, by	default	columns	are considered what you	would
	       see visually beneath that header	when the table is rendered in
	       a browser.  See the "gridmap" parameter for more	information.

	       HTML within a header is stripped	before the match is attempted,
	       unless the "keep_html" parameter	was specified and
	       "strip_html_on_match" is	false.

	   depth
	       Specify how embedded in other tables your tables	of interest
	       should be.  Top-level tables in the HTML	document have a	depth
	       of 0, tables within top-level tables have a depth of 1, and so
	       on.

	   count
	       Specify which table within each depth you are interested	in,
	       beginning with 0.

	   attribs
	       Passed as a hash	reference, attribs specify attributes of
	       interest	within the HTML	<table>	tag itself.

	   automap
	       Automatically applies the ordering reported by column_map() to
	       the rows	returned by rows(). This only makes a difference if
	       you have	specified Headers and they turn	out to be in a
	       different order in the table than what you specified. Automap
	       will rearrange the columns in the same order as the headers
	       appear. To get the original ordering, you will need to take
	       another slice of	each row using column_map(). automap is
	       enabled by default.

	   slice_columns
	       Enabled by default, this	option controls	whether	vertical
	       slices are returned from	under headers that match. When
	       disabled, all columns of	the matching table are retained,
	       regardles of whether they had a matching	header above them.
	       Disabling this also disables "automap".

	   keep_headers
	       Disabled	by default, and	only applicable	when header
	       constraints have	been specified,	"keep_headers" will retain the
	       matching	header row as the first	row of table data when
	       enabled.	This option has	no effect if extracting	into an
	       element tree structure. In any case, the	header row is
	       accessible from the table method	"hrow()".

	   gridmap
	       Controls	whether	the table contents are returned	as a grid or a
	       tree.  ROWSPAN and COLSPAN issues are compensated for, and
	       columns really are columns. Empty phantom cells are created
	       where they would	have been obscured by ROWSPAN or COLSPAN
	       settings. This really becomes an	issue when extracting columns
	       beneath headers.	Enabled	by default.

	   subtables
	       Extract all tables embedded within matched tables.

	   decode
	       Automatically decode retrieved text with
	       HTML::Entities::decode_entities(). Enabled by default. Has no
	       effect if "keep_html" was specified or if extracting into an
	       element tree structure.

	   br_translate
	       Translate <br> tags into	newlines. Sometimes the	remaining text
	       can be hard to parse if the <br>	tag is simply dropped. Enabled
	       by default. Has no effect if keep_html is enabled or if
	       extracting into an element tree structure.

	   keep_html
	       Return the raw HTML contained in	the cell, rather than just the
	       visible text. Embedded tables are not retained in the HTML
	       extracted from a	cell. Patterns for header matches must take
	       into account HTML in the	string if this option is enabled. This
	       option has no effect if extracting into an elment tree
	       structure.

	   strip_html_on_match
	       When "keep_html"	is enabled, HTML is stripped by	default	during
	       attempts	at matching header strings (so if
	       "strip_html_on_match" is	not enabled and	"keep_html" is,	you
	       would have to include potential HTML tags in the	regexp for
	       header matches).	Stripped header	tags are replaced with an
	       empty string, e.g. 'hot d<em>og</em>' would become 'hot dog'
	       before attempting a match.

	   error_handle
	       Filehandle where	error messages are printed. STDERR by default.

	   debug
	       Prints some debugging information to STDERR, more for higher
	       values.	If "error_handle" was provided,	messages are printed
	       there rather than STDERR.

   REGULAR METHODS
       The following methods are invoked directly from an HTML::TableExtract
       object.

       depths()
	   Returns all depths that contained matched tables in the document.

       counts($depth)
	   For a particular depth, returns all counts that contained matched
	   tables.

       table($depth, $count)
	   For a particular depth and count, return the	table object for the
	   table found,	if any.

       tables()
	   Return table	objects	for all	tables that matched. Returns an	empty
	   list	if no tables matched.

       first_table_found()
	   Return the table state object for the first table matched in	the
	   document. Returns undef if no tables	were matched.

       current_table()
	   Returns the current table object while parsing the HTML. Only
	   useful if you're messing around with	overriding HTML::Parser
	   methods.

       tree()
	   If the module was invoked in	tree extraction	mode, returns a
	   reference to	the top	node of	the HTML::Element tree structure for
	   the entire document (which includes,	ultimately, all	tables within
	   the document).

       tables_report([$show_content, $col_sep])
	   Return a string summarizing extracted tables, along with their
	   depth and count. Optionally takes a $show_content flag which	will
	   dump	the extracted contents of each table as	well with columns
	   separated by	$col_sep. Default $col_sep is ':'.

       tables_dump([$show_content, $col_sep])
	   Same	as "tables_report()" except dump the information to STDOUT.

       start
       end
       text
	   These are the hooks into HTML::Parser. If you want to subclass this
	   module and have things work,	you must at some point call these with
	   content.

   DEPRECATED METHODS
       Tables used to be called	'table states'.	Accordingly, the following
       methods still work but have been	deprecated:

       table_state()
	   Is now table()

       table_states()
	   Is now tables()

       first_table_state_found()
	   Is now first_table_found()

   TABLE METHODS
       The following methods are invoked from an HTML::TableExtract::Table
       object, such as those returned from the "tables()" method.

       rows()
	   Return all rows within a matched table. Each	row returned is	a
	   reference to	an array containing the	text, HTML, or reference to
	   the HTML::Element object of each cell depending the mode of
	   extraction. Tables with rowspan or colspan attributes will have
	   some	cells containing undef.	 Returns a list	or a reference to an
	   array depending on context.

       columns()
	   Return all columns within a matched table. Each column returned is
	   a reference to an array containing the text,	HTML, or reference to
	   HTML::Element object	of each	cell depending on the mode of
	   extraction.	Tables with rowspan or colspan attributes will have
	   some	cells containing undef.

       row($row)
	   Return a particular row from	within a matched table either as a
	   list	or an array reference, depending on context.

       column($col)
	   Return a particular column from within a matched table as a list or
	   an array reference, depending on context.

       cell($row,$col)
	   Return a particular item from within	a matched table, whether it be
	   the text, HTML, or reference	to the HTML::Element object of that
	   cell, depending on the mode of extraction. If the cell was covered
	   due to rowspan or colspan effects, will return undef.

       space($row,$col)
	   The same as cell(), except in cases where the given coordinates
	   were	covered	due to rowspan or colspan issues, in which case	the
	   content of the covering cell	is returned rather than	undef.

       depth()
	   Return the depth at which this table	was found.

       count()
	   Return the count for	this table within the depth it was found.

       coords()
	   Return depth	and count in a list.

       tree()
	   If the module was invoked in	tree extraction	mode, this accessor
	   provides a reference	to the HTML::ElementTable structure
	   encompassing	the table.

       hrow()
	   Returns the header row as a list when headers were specified	as a
	   constraint. If "keep_headers" was specified initially, this is
	   equivalent to the first row returned	by the "rows()"	method.

       column_map()
	   Return the order (via indices) in which the provided	headers	were
	   found.  These indices can be	used as	slices on rows to either order
	   the rows in the same	order as headers or restore the	rows to	their
	   natural order, depending on whether the rows	have been pre-adjusted
	   using the automap parameter.

       lineage()
	   Returns the path of matched tables that led to matching this	table.
	   The path is a list of array refs containing depth, count, row, and
	   column values for each ancestor table involved. Note	that
	   corresponding table objects will not	exist for ancestral tables
	   that	did not	match specified	constraints.

NOTES ON TREE EXTRACTION MODE
       As mentioned above, HTML::TableExtract can be invoked in	'tree' mode
       where the resulting HTML	and extracted tables are encoded in
       HTML::Element tree structures:

	 use HTML::TableExtract	'tree';

       There are a number of things to take note of while using	this mode. The
       entire HTML document is encoded into an HTML::Element tree. Each	table
       is part of this structure, but nevertheless is tracked separately via
       an HTML::ElementTable structure,	which is a specialized form of
       HTML::Element tree.

       The HTML::ElementTable objects are accessible by	invoking the tree()
       method from within each table object returned by	HTML::TableExtract.
       The HTML::ElementTable objects have their own row(), col(), and cell()
       methods (among others). These are not to	be confused with the row() and
       column()	methods	provided by the	HTML::TableExtract::Table objects.

       For example, the	row() method from HTML::ElementTable will provide a
       reference to a 'glob' of	all the	elements in that row. Actions (such as
       setting attributes) performed on	that row reference will	affect all
       elements	within that row. On the	other hand, the	row() method from the
       HTML::TableExtract::Table object	will return an array (either by
       reference or list, depending on context)	of the contents	of each	cell
       within the row. In tree mode, the content is represented	by individual
       references to each cell -- these	are references to the same
       HTML::Element objects that reside in the	HTML::Element tree.

       The cell() methods provided in both cases will therefore	return
       references to the same object. The exception to this is when a 'cell'
       in the table grid was originally	'covered' due to rowspan or colspan
       issues -- in this case the cell content will be undef. Likewise,	the
       row() or	column() methods from HTML::TableExtract::Table	objects	will
       return arrays potentially containing a mixture of object	references and
       undefs.	If you're going	to be doing lots of manipulation of the	table
       elements, it might be more efficient to access them via the methods
       provided	by the HTML::ElementTable object instead. See
       HTML::ElementTable for more information on how to manipulate those
       objects.

       An alternative to the cell() method in HTML::TableExtract::Table	is the
       space() method. It is largely similar to	cell(),	except when given
       coordinates of a	cell that was covered due to rowspan or	colspan
       effects,	it will	return the contents of the cell	that was covering that
       space rather than undef.	So if, for example, cell (0,0) had a rowspan
       of 2 and	colspan	of 2, cell(1,1)	would return undef and space(1,1)
       would return the	same content as	cell(0,0) or space(0,0).

REQUIRES
       HTML::Parser(3),	HTML::Entities(3)

OPTIONALLY REQUIRES
       HTML::TreeBuilder(3), HTML::ElementTable(3)

AUTHOR
       Matthew P. Sisk,	<sisk@mojotoad.com>

COPYRIGHT
       Copyright (c) 2000-2017 Matthew P. Sisk.	 All rights reserved. All
       wrongs revenged.	This program is	free software; you can redistribute it
       and/or modify it	under the same terms as	Perl itself.

SEE ALSO
       HTML::Parser(3),	HTML::TreeBuilder(3), HTML::ElementTable(3), perl(1).

perl v5.24.1			  2017-05-25		 HTML::TableExtract(3)

NAME | SYNOPSIS | DESCRIPTION | METHODS | NOTES ON TREE EXTRACTION MODE | REQUIRES | OPTIONALLY REQUIRES | AUTHOR | COPYRIGHT | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=HTML::TableExtract&sektion=3&manpath=FreeBSD+12.0-RELEASE+and+Ports>

home | help