Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
WWW::Search(3)	      User Contributed Perl Documentation	WWW::Search(3)

NAME
       WWW::Search - Virtual base class	for WWW	searches

SYNOPSIS
	 use WWW::Search;
	 my $sEngine = 'AltaVista';
	 my $oSearch = new WWW::Search($sEngine);

DESCRIPTION
       This class is the parent	for all	access methods supported by the
       "WWW::Search" library.  This library implements a Perl API to web-based
       search engines.

       See README for a	list of	search engines currently supported, and	for a
       lot of interesting high-level information about this distribution.

       Search results can be limited, and there	is a pause between each
       request to avoid	overloading either the client or the server.

   Sample program
       Here is a sample	program:

	 my $sQuery = 'Columbus	Ohio sushi restaurant';
	 my $oSearch = new WWW::Search('AltaVista');
	 $oSearch->native_query(WWW::Search::escape_query($sQuery));
	 $oSearch->login($sUser, $sPassword);
	 while (my $oResult = $oSearch->next_result())
	   {
	   print $oResult->url,	"\n";
	   } # while
	 $oSearch->logout;

       Results are objects of type "WWW::SearchResult" (see WWW::SearchResult
       for details).  Note that	different backends support different result
       fields.	All backends are required to support title and url.

SEE ALSO
       For specific search engines, see	WWW::Search::TheEngineName (replacing
       TheEngineName with a particular search engine).

       For details about the results of	a search, see WWW::SearchResult.

METHODS	AND FUNCTIONS FOR SEARCHERS
       new To create a new WWW::Search,	call

	       $oSearch	= new WWW::Search('SearchEngineName');

	   where SearchEngineName is replaced with a particular	search engine.
	   For example:

	       $oSearch	= new WWW::Search('Yahoo');

	   If no search	engine is specified, a default (currently
	   'Null::Empty') will be chosen for you.

       version
	   Returns the value of	the $VERSION variable of the backend engine,
	   or $WWW::Search::VERSION if the backend does	not contain $VERSION.

       maintainer
	   Returns the value of	the $MAINTAINER	variable of the	backend
	   engine, or $WWW::Search::MAINTAINER if the backend does not contain
	   $MAINTAINER.

       installed_engines
	   Returns a list of the names of all installed	backends.  We can not
	   tell	if they	are up-to-date or working, though.

	     use WWW::Search;
	     my	@asEngines = sort &WWW::Search::installed_engines();
	     local $" =	', ';
	     print (" +	These WWW::Search backends are installed: @asEngines\n");
	     # Choose a	backend	at random (yes,	this is	rather silly):
	     my	$oSearch = WWW::Search->new($asEngines[rand(scalar(@asEngines))]);

       native_query
	   Specify a query (and	optional options) to the current search
	   object.  Previous query (if any) and	its cached results (if any)
	   will	be thrown away.	 The option values and the query must be
	   escaped; call WWW::Search::escape_query() to	escape a string.  The
	   search process is not actually begun	until "results()" or
	   "next_result()" is called (lazy!), so native_query does not return
	   anything.

	   Example:

	     $oSearch->native_query('search-engine-specific+escaped+query+string',
				   { option1 =>	'able',	option2	=> 'baker' } );

	   The hash of options following the query string is optional.	The
	   query string	is backend-specific.  There are	two kinds of options:
	   options specific to the backend, and	generic	options	applicable to
	   multiple backends.

	   Generic options all begin with 'search_'.  Currently	a few are
	   supported:

	   search_url
	       Specifies the base URL for the search engine.

	   search_debug
	       Enables backend debugging.  The default is 0 (no	debugging).

	   search_parse_debug
	       Enables backend parser debugging.  The default is 0 (no
	       debugging).

	   search_to_file FILE
	       Causes the search results to be saved in	a set of files
	       prefixed	by FILE.  (Used	internally by the test-suite, not
	       intended	for general use.)

	   search_from_file FILE
	       Reads a search from a set of files prefixed by FILE.  (Used
	       internally by the test-suite, not intended for general use.)

	   Some	backends may not implement these generic options, but any
	   which do implement them must	provide	these semantics.

	   Backend-specific options are	described in the documentation for
	   each	backend.  In most cases	the options and	their values are
	   packed together to create the query portion of the final URL.

	   Details about how the search	string and option hash are interpreted
	   might be found in the search-engine-specific	manual pages
	   (WWW::Search::SearchEngineName).

       gui_query
	   Specify a query to the current search object; the query will	be
	   performed with the engine's default options,	as if it were typed by
	   a user in a browser window.

	   Same	arguments as "native_query()" above.

	   Currently, this feature is supported	by only	a few backends;
	   consult the documentation for each backend to see if	it is
	   implemented.

       cookie_jar
	   Call	this method (anytime before asking for results)	if you want to
	   communicate cookie data with	the search engine.  Takes one
	   argument, either a filename or an HTTP::Cookies object.  If you
	   give	a filename, WWW::Search	will attempt to	read/store cookies
	   there (by passing the filename to HTTP::Cookies::new).

	     $oSearch->cookie_jar('/tmp/my_cookies');

	   If you give an HTTP::Cookies	object,	it is up to you	to save	the
	   cookies if/when you wish.

	     use HTTP::Cookies;
	     my	$oJar =	HTTP::Cookies->new(...);
	     $oSearch->cookie_jar($oJar);

	   If you pass in no arguments,	the cookie jar (if any)	is returned.

	     my	$oJar =	$oSearch->cookie_jar;
	     unless (ref $oJar)	{ print	"No jar" };

       date_from
	   Set/get the start date for limiting the query by a date range.  See
	   the documentation for each backend to find out if date ranges are
	   supported.

       date_to
	   Set/get the end date	for limiting the query by a date range.	 See
	   the documentation for each backend to find out if date ranges are
	   supported.

       env_proxy
	   Enable loading proxy	settings from environment variables.  The
	   proxy URL will be read from $ENV{http_proxy}.  The username for
	   authentication will be read from $ENV{http_proxy_user}.  The
	   password for	authentication will be read from $ENV{http_proxy_pwd}.

	   If you don't	want to	put passwords in the environment, one solution
	   would be to subclass	LWP::UserAgent and use
	   $ENV{WWW_SEARCH_USERAGENT} instead (see user_agent below).

	   env_proxy() must be called before the first retrieval is attempted.

	   Example:

	     $ENV{http_proxy	 } = 'http://my.proxy.com:80';
	     $ENV{http_proxy_user} = 'bugsbun';
	     $ENV{http_proxy_pwd } = 'c4rr0t5';
	     $oSearch->env_proxy('yes');  # Turn on with any true value
	     ...
	     $oSearch->env_proxy(0);  #	Turn off with zero
	     ...
	     if	($oSearch->env_proxy)  # Test

       http_proxy
	   Set up an HTTP proxy	(for connections from behind a firewall).

	   Takes the same arguments as LWP::UserAgent::proxy().

	   This	routine	should be called before	calling	any of the result
	   functions (any method with "result" in its name).

	   Example:

	     # Turn on and set address:
	     $oSearch->http_proxy(['http','ftp'] => 'http://proxy:8080');
	     # Turn off:
	     $oSearch->http_proxy('');

       http_proxy_user,	http_proxy_pwd
	   Set/get HTTP	proxy authentication data.

	   These routines set/get username and password	used in	proxy
	   authentication.  Authentication is attempted	only if	all three
	   items (proxy	URL, username and password) have been set.

	   Example:

	       $oSearch->http_proxy_user("myuser");
	       $oSearch->http_proxy_pwd("mypassword");
	       $oSearch->http_proxy_user(undef);   # Example for no authentication

	       $username = $oSearch->http_proxy_user();

       maximum_to_retrieve
	   Set the maximum number of hits to return.  Queries resulting	in
	   more	than this many hits will return	the first hits,	up to this
	   limit.  Although this specifies a maximum limit, search engines
	   might return	less than this number.

	   Defaults to 500.

	   Example:
	       $max = $oSearch->maximum_to_retrieve(100);

	   You can also	spell this method "maximum_to_return".

       maximum_to_return
	   Synonym for maximum_to_retrieve

       timeout
	   The maximum length of time any portion of the query should take, in
	   seconds.

	   Defaults to 60.

	   Example:
	       $oSearch->timeout(120);

       login
	   Backends which need to login	to the search engine should implement
	   this	function.  Takes two arguments,	user and password.  Return
	   nonzero if login was	successful.  Return undef or 0 if login
	   failed.

       logout
	   Backends which need to logout from the search engine	should
	   implement this function.

       approximate_result_count
	   Some	backends indicate how many results they	have found, e.g. with
	   a number shown at the top of	the search results page.  Note that
	   there is no corresponding method that returns the actual count of
	   results; that's because results are normally	retrieved in batches
	   (i.e. pages)	and at any given time there's no way to	know how big
	   the final list of results will be.  NEW: if request has not been
	   made	to the search provider,	this method will return	undef (used to
	   return zero).  NEW: if the results page does	not explicitly
	   indicate the	result count, this method will return undef (used to
	   return zero).

       approximate_hit_count
	   This	is an alias for	approximate_result_count().

       results
	   Return all the results of a query as	an array of WWW::SearchResult
	   objects.

	   Note: This might take a while, because a web	backend	will keep
	   asking the search engine for	"next page of results" over and	over
	   until there are no more next	pages, and THEN	return from this
	   function.

	   If an error occurs at any time during query processing, it will be
	   indicated in	the response().

	   Example:

	       @results	= $oSearch->results();
	       # Go have a cup of coffee while the previous line executes...
	       foreach $oResult	(@results)
		 {
		 print $oResult->url(),	"\n";
		 } # foreach

       next_result
	   Call	this method repeatedly to return each result of	a query	as a
	   WWW::SearchResult object.  Example:

	       while ($oResult = $oSearch->next_result())
		 {
		 print $oResult->url(),	"\n";
		 } # while

	   When	there are no more results, or if an error occurs,
	   next_result() will return undef.

	   If an error occurs at any time during query processing, it will be
	   indicated in	the response().

       seek_result($offset)
	   Set which result should be returned next time "next_result()" is
	   called.  Results are	zero-indexed.

	   The only guaranteed valid offset is 0, which	will replay the
	   results from	the beginning.	In particular, seeking past the	end of
	   the current cached results probably will not	do what	you might
	   think it should.

	   Results are cached, so this does not	re-issue the query or cause IO
	   (unless you go off the end of the results).	To re-do the query,
	   create a new	search object.

	   Example:

	       $oSearch->seek_result(0);

       response
	   Returns an HTTP::Response object which resulted from	the most-
	   recently-sent query.	 Errors	can be detected	like this:

	       if (! $oSearch->response->is_success)
		 {
		 print STDERR "Error:  " . $oSearch->response->as_string() . "\n";
		 } # if

	   Note	to backend authors: even if the	backend	does not involve the
	   web,	it should return an HTTP::Response object.

       submit
	   This	method can be used to submit URLs to the search	engines	for
	   indexing.  Consult the documentation	for each backend to find out
	   if it is implemented	there, and if so what the arguments are.

	   Returns an HTTP::Response object describing the result of the
	   submission request.	Consult	the documentation for each backend to
	   find	out the	meaning	of the response.

       opaque
	   This	function provides an application a place to store one opaque
	   data	element	(or many, via a	Perl reference).  This facility	is
	   useful to (for example), maintain client-specific information in
	   each	active query when you have multiple concurrent queries.

       escape_query
	   Escape a query.  Before queries are sent to the internet, special
	   characters must be escaped so that a	proper URL can be formed.
	   This	is like	escaping a URL,	but all	non-alphanumeric characters
	   are escaped and and spaces are converted to "+"s.

	   Example:

	       $escaped	= WWW::Search::escape_query('+hi +mom');
	       # $escaped is now '%2Bhi+%2Bmom'

	   See also "unescape_query()".	 NOTE that this	is not a method, it is
	   a plain function.

       unescape_query
	   Unescape a query.  See "escape_query()" for details.

	   Example:

	       $unescaped = WWW::Search::unescape_query('%22hi+mom%22');
	       # $unescaped eq q{"hi mom"}

	   NOTE	that this is not a method, it is a plain function.

       strip_tags
	   Given a string, returns a copy of that string with HTML tags
	   removed.  This should be used by each backend as they insert	the
	   title and description values	into the search	results	objects.

	   NOTE	that this is not a method, it is a plain function.

       is_http_proxy
	   Returns true	if proxy information is	available.

METHODS	AND FUNCTIONS FOR BACKEND PROGRAMMERS
       reset_search
	   Resets internal data	structures to start over with a	new search (on
	   the same engine).

       is_http_proxy_auth_data
	   Returns true	if all authentication data (proxy URL, username, and
	   password) are available.

       agent_name($sName)
	   If your search engine rejects certain browser, you can trick	it
	   into	thinking you're	any browser type you want.  See	below under
	   user_agent().

       agent_email($sName)
       user_agent($NON_ROBOT)
	   This	internal routine creates a user-agent for derived classes that
	   query the web.  If any non-false argument is	given, a normal
	   LWP::UserAgent (rather than a LWP::RobotUA) is used.

	   Returns the user-agent object.

	   If a	backend	needs the low-level LWP::UserAgent or LWP::RobotUA to
	   have	a particular name, $oSearch->agent_name() and possibly
	   $oSearch->agent_email() should be called to set the desired values
	   *before* calling $oSearch->user_agent().

	   If the environment variable WWW_SEARCH_USERAGENT has	a value, it
	   will	be used	as the class for a new user agent object.  This	class
	   should be a subclass	of LWP::UserAgent.  For	example,

	     $ENV{WWW_SEARCH_USERAGENT}	= 'My::Own::UserAgent';
	     # If this env.var.	has no value,
	     # LWP::UserAgent or LWP::RobotUA will be used.
	     $oSearch =	new WWW::Search('MyBackend');
	     $oSearch->agent_name('MySpider');
	     if	($iBackendWebsiteRequiresNonRobot)
	       {
	       $oSearch->user_agent('non-robot');
	       }
	     else
	       {
	       $oSearch->agent_email('me@here.com');
	       $oSearch->user_agent();
	       }

	   Backends should use robot-style user-agents whenever	possible.

       http_referer
	   Get / set the value of the HTTP_REFERER variable for	this search
	   object.  Some search	engines	might only accept requests that
	   originated at some specific previous	page.  This method lets
	   backend authors "fake" the previous page.  Call this	method before
	   calling http_request.

	     $oSearch->http_referer('http://prev.engine.com/wherever/setup.html');
	     $oResponse	= $oSearch->http_request('GET',	$url);

       http_method
	   Get / set the method	to be used for the HTTP	request.  Must be
	   either 'GET'	or 'POST'.  Call this method before calling
	   http_request.  (Normally you	would set this during
	   _native_setup_search().)  The default is 'GET'.

	     $oSearch->http_method('POST');

       http_request($method, $url)
	   Submit the HTTP request to the world, and return the	response.
	   Similar to LWP::UserAgent::request.	Handles	cookies, follows
	   redirects, etc.  Requires that http_referer already be set up, if
	   needed.

       next_url
	   Get or set the URL for the next backend request.  This can be used
	   to save the WWW::Search state between sessions (e.g.	if you are
	   showing pages of results to the user	in a web browser).  Before
	   closing down	a session, save	the value of next_url:

	     ...
	     $oSearch->maximum_to_return(10);
	     while ($oSearch->next_result) { ... }
	     my	$urlSave = $oSearch->next_url;

	   Then, when you start	up the next session (e.g. after	the user
	   clicks your "next" button), restore this value before calling for
	   the results:

	     $oSearch->native_query(...);
	     $oSearch->next_url($urlSave);
	     $oSearch->maximum_to_return(20);
	     while ($oSearch->next_result) { ... }

	   WARNING: It is entirely up to you to	keep your interface in sync
	   with	the number of hits per page being returned from	the backend.
	   And,	we make	no guarantees whether this method will work for	any
	   given backend.  (Their caching scheme might not enable you to jump
	   into	the middle of a	list of	search results,	for example.)

       split_lines
	   This	internal routine splits	data (typically	the result of the web
	   page	retrieval) into	lines in a way that is OS independent.	If the
	   first argument is a reference to an array, that array is taken to
	   be a	list of	possible delimiters for	this split.  For example,
	   Yahoo.pm uses <p> and <dd><li> as "line" delimiters for
	   convenience.

       generic_option
	   This	internal routine checks	if an option is	generic	or backend
	   specific.  Currently	all generic options begin with 'search_'.
	   This	routine	is not a method.

       _native_setup_search
	   Do some backend-specific initialization.  It	will be	called with
	   the same arguments as native_query().

       setup_search
	   This	internal routine does generic Search setup.  It	calls
	   "_native_setup_search()" to do backend-specific setup.

       need_to_delay
	   A backend should override this method in order to dictate whether
	   user_agent_delay() needs to be called before	the next HTTP request
	   is sent.  Return any	perlish	true or	zero value.

       user_agent_delay
	   According to	what need_to_delay() returns, user_agent_delay() will
	   be called between requests to remote	servers	to avoid overloading
	   them	with many back-to-back requests.

       absurl
	   An internal routine to convert a relative URL into a	absolute URL.
	   It takes two	arguments, the 'base' url (usually the search engine
	   CGI URL) and	the URL	to be converted.  Returns a URI	object.

       retrieve_some
	   An internal routine to interface with "_native_retrieve_some()".
	   Checks for overflow.

       _native_retrieve_some
	   Fetch the next page of results from the web engine, parse the
	   results, and	prepare	for the	next page of results.

	   If a	backend	defines	this method, it	is in total control of the WWW
	   fetch, parsing, and preparing for the next page of results.	See
	   the WWW::Search::AltaVista module for example usage of the
	   _native_retrieve_some method.

	   An easier way to achieve this in a backend is to inherit
	   _native_retrieve_some from WWW::Search, and do only the HTML
	   parsing.  Simply define a method _parse_tree	which takes one
	   argument, an	HTML::TreeBuilder object, and returns an integer, the
	   number of results found on this page.  See the WWW::Search::Yahoo
	   module for example usage of the _parse_tree method.

	   A backend should, in	general, define	either _parse_tree() or
	   _native_retrieve_some(), but	not both.

	   Additional features of the default _native_retrieve_some method:

	   Sets	$self->{_prev_url} to the URL of the page just retrieved.

	   Calls $self->preprocess_results_page() on the raw HTML of the page.

	   Then, parses	the page with an HTML::TreeBuilder object and passes
	   that	populated object to $self->_parse_tree().

	   Additional notes on using the _parse_tree method:

	   The built-in	HTML::TreeBuilder object used to parse the page	has
	   store_comments turned ON.  If a backend needs to use	a subclassed
	   or modified HTML::TreeBuilder object, the backend should set
	   $self->{'_treebuilder'} to that object before any results are
	   retrieved.  The best	place to do this is at the end of
	   _native_setup_search.

	     my	$oTree = new myTreeBuilder;
	     $oTree->store_pis(1);  # for example
	     $self->{'_treebuilder'} = $oTree;

	   When	_parse_tree() is called, the $self->next_url is	cleared.
	   During parsing, the backend should set $self->next_url to the
	   appropriate URL for the next	page of	results.  (If _parse_tree()
	   does	not set	the value, the search will end after parsing this page
	   of results.)

	   When	_parse_tree() is called, the URL for the page being parsed can
	   be found in $self->{_prev_url}.

       result_as_HTML
	   Given a WWW::SearchResult object, formats it	human-readable with
	   HTML.

       preprocess_results_page
	   A filter on the raw HTML of the results page.  This allows the
	   backend to alter the	HTML before it is parsed, such as to correct
	   for known problems, HTML that can not be parsed correctly, etc.

	   Takes one argument, a string	(the HTML webpage); returns one	string
	   (the	same HTML, modified).

	   This	method is called from within _native_retrieve_some (above)
	   before the HTML of the page is parsed.

	   See the WWW::Search::Ebay distribution 2.07 or higher for example
	   usage.

       test_cases (DEPRECATED)
	   Deprecated.

	   Returns the value of	the $TEST_CASES	variable of the	backend
	   engine.

       hash_to_cgi_string (DEPRECATED)
	   Given a reference to	a hash of string => string, constructs a CGI
	   parameter string that looks like 'key1=value1&key2=value2'.

	   If the value	is undef, the key will not be added to the string.

	   At one time,	for testing purposes, we asked backends	to use this
	   function rather than	piecing	the URL	together by hand, to ensure
	   that	URLs are identical across platforms and	software versions.
	   But this is no longer necessary.

	   Example:

	       $self->{_options} = {
				    'opt3' => 'val3',
				    'search_url' => 'http://www.deja.com/dnquery.xp',
				    'opt1' => 'val1',
				    'QRY' => $native_query,
				    'opt2' => 'val2',
				   };
	       $self->{_next_url} = $self->{_options}{'search_url'} .'?'.
				    $self->hash_to_cgi_string($self->{_options});

IMPLEMENTING NEW BACKENDS
       "WWW::Search" supports backends to separate search engines.  Each
       backend is implemented as a subclass of "WWW::Search".
       WWW::Search::Yahoo provides a good sample backend.

       A backend must have the routine "_native_setup_search()".  A backend
       must have the routine "_native_retrieve_some()" or "_parse_tree()".

       "_native_setup_search()"	is invoked before the search.  It is passed a
       single argument:	the escaped, native version of the query.

       "_native_retrieve_some()" is the	core of	a backend.  It will be called
       periodically to fetch URLs.  It should retrieve several hits from the
       search service and add them to the cache.  It should return the number
       of hits found, or undef when there are no more hits.

       Internally, "_native_retrieve_some()" typically sends an	HTTP request
       to the search service, parses the HTML, extracts	the links and
       descriptions, then saves	the URL	for the	next page of results.  See the
       code for	the "WWW::Search::AltaVista" module for	an example.

       Alternatively, a	backend	can define the method "_parse_tree()" instead
       of "_native_retrieve_some()".  See the "WWW::Search::Ebay" module for a
       good example.

       If you implement	a new backend, please let the authors know.

BUGS AND DESIRED FEATURES
       The bugs	are there for you to find (some	people call them Easter	Eggs).

       Desired features:

       A portable query	language.
	   A portable language would easily allow you to move queries easily
	   between different search engines.  A	query abstraction is non-
	   trivial and unfortunately will not be done any time soon by the
	   current maintainer.	If you want to take a shot at it, please let
	   me know.

AUTHOR
       John Heidemann <johnh@isi.edu> Maintained by Martin Thurn,
       "mthurn@cpan.org", <http://www.sandcrawler.com/SWB/cpan-modules.html>.

COPYRIGHT
       Copyright (c) 1996 University of	Southern California.  All rights
       reserved.

       Redistribution and use in source	and binary forms are permitted
       provided	that the above copyright notice	and this paragraph are
       duplicated in all such forms and	that any documentation,	advertising
       materials, and other materials related to such distribution and use
       acknowledge that	the software was developed by the University of
       Southern	California, Information	Sciences Institute.  The name of the
       University may not be used to endorse or	promote	products derived from
       this software without specific prior written permission.

       THIS SOFTWARE IS	PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
       WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
       MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

perl v5.32.1			  2020-04-03			WWW::Search(3)

NAME | SYNOPSIS | DESCRIPTION | SEE ALSO | METHODS AND FUNCTIONS FOR SEARCHERS | METHODS AND FUNCTIONS FOR BACKEND PROGRAMMERS | IMPLEMENTING NEW BACKENDS | BUGS AND DESIRED FEATURES | AUTHOR | COPYRIGHT

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=WWW::Search&sektion=3&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help