Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
AUTOSEARCH(1)	      User Contributed Perl Documentation	 AUTOSEARCH(1)

       AutoSearch -- a web-search tracking application

       AutoSearch [--stats] [--verbose]	-n "Query Name"	-s "query string"
       --engine	engine [--mail] [--options "opt=val"]...
       [--filter "filter"] [--host host] [--port port] [--userid bbunny
       --password c4rr0t5] [--ignore_channels KABC,KCBS,KNBC] qid

       AutoSearch --VERSION AutoSearch --help AutoSearch --man

       AutoSearch performs a web-based search and puts the results set in
       qid/index.html.	Subsequent searches (i.e., the second form above)
       AutoSearch determine what changes (if any) occured to the results sent
       since the last run.  These incremental changes are recorded in

       AutoSearch is amenable to be run	as a cron job because all the input
       parameters are saved in the web pages.  AutoSearch can act as a
       automated query agent for a particular search.  The output files	are
       designed	to be a	set of web pages to easily display the results set
       with a web browser.


	   AutoSearch -n 'LSAM Replication'
	       -s '"lsam replication"'
	       -e AltaVista

       This query (which should	be all on one line) creates a directory
       replication_query and fills it with the fascinating output of the
       AltaVista query on "lsam	replication", with pages titled	``LSAM
       Replication''.  (Note the quoting:  the single quotes in	'"lsam
       replication"' are for the shell,	the double quotes are for AltaVista to
       search for the phrase rather than the separate words.)

       A more complicated example:

	   AutoSearch -n 'External Links to LSAM'
	       -s '( or'
	       -e AltaVista::AdvancedWeb
	       -o coolness=hot

       This query does an advanced AltaVista search and	specifies the
       (hypothetical) ``coolness'' option to the search	engine.

	   The query identifer specifies the directory in which	all the	files
	   that	relate to this query and search	results	will live.  It can be
	   an absolute path, or	a relative path	from cwd.  If the directory
	   does	not exist, it will be created and a new	search started.

	   Show	search statistics: the query string, number of hits, number of
	   filtered hits, filter string, number	of suspended (deleted) hits,
	   previous set	size, current set size,	etc.

       "-v" or "--verbose"
	   Verbose: output additional messages and warnings.

       "-n" or "--qn" or "--queryname"
	   Specify the query name.  The	query name is used as a	heading	in the
	   web pages, therefore	it should be a 'nice' looking version of the
	   query string.

       "-s" or "--qs" or "--querystring"
	   Specify the query string.  The query	string is the character	string
	   which will be submitted to the search engine.  You may include
	   special characters to group or to qualify the search.

       "-e" or "--engine"
	   Specify the search engine.  The query string	will be	submitted to
	   the user specified search engine.

	   In many cases there are specialized versions	of search engines.
	   For example,	AltaVista::AdvancedWeb and AltaVista::News allow more
	   powerful and	Usenet searches.  See AltaVista	or the man page	for
	   your	search engine for details about	specialized variations.

	   In addition to all the normal file maintenance, print all new URLs
	   to STDOUT, one per line.

       "-o" or "--options"
	   Specify the query options.  The query options will be submitted to
	   the user search engine with the query string.  This feature permits
	   modification	of the query string for	a specific search engine or
	   option.  More than one query	option may be specified.

	   Example: "-o	what=news" causes AltaVista to search Usenet.
	   Although this works,	the preferred mechanism	in this	case would be
	   "-e AltaVista::News"	or "-e AltaVista::AdvancedNews".  Options are
	   intended for	internal or expert use.

       "-f" or "--uf" or "--urlfilter"
	   This	option specifies a regular expression which will be compared
	   against the URLs of any results; if they match the case-insensitive
	   regular expression, they will be removed from the hit set.

	   Example: "-f	'.*\.isi\.edu'"	avoids all of ISI's web	pages.

       "--cleanup i"
	   Delete all traces of	query results from more	than i days ago.  If
	   --cleanup is	given, all other options other than the	qid will be

	   Reconstruct the complete command line (AutoSearch and all its
	   arguments) that was used to create the query	results.  Command line
	   will	be shown on STDERR.  If	--cmdline is given, all	other options
	   other than the qid will be ignored.

       "--mail user@address" or	"-m user@address"
	   After search	is complete, send email	to that	user, listing the NEW
	   results.  Email is HTML format.  Requires the Email::Send and
	   related modules.  If	you send email through an SMTP server, you
	   must	set environment	variable SMTPSERVER to your server name	or IP
	   address.  If	your SMTP server requires password, you	must set
	   environment variables SMTPUSERNAME and SMTPPASSWORD.	 If you	send
	   email via sendmail, you should set environment variable SENDMAIL if
	   the sendmail	executable is not in the path.

       "--emailfrom user@address"
	   If your outgoing mail server	rejects	email from certain users, you
	   can use this	argument to set	the From: header.

       "--userid bbunny"
	   If the search engine	requires a login/password (e.g.
	   Ebay::Completed), use this.

       "--password Carr0t5"
	   If the search engine	requires a login/password (e.g.	Ebay::Mature),
	   use this.

       AutoSearch submits a query to a search engine, produces HTML pages that
       reflect the set of 'hits' (filtered search results) returned by the
       search engine, and tracks these results over time.  The URL and title
       are displayed in	the qid/index.html, the	URL, the title,	and
       description are displayed in the	'weekly' files.

       To organize these results, each search result is	placed in a query
       information directory (qid).  The directory becomes the search results
       'handle', an easy way to	track a	set of results.	 Thus a	qid of
       "/usr/local/htdocs/lsam/autosearch/load_balancing" might	locate the
       results on your web server at

       Inside the qid directory	you will find files relating to	this query.
       The primary file	is index.html, which reflects the latest search
       results.	 Every not-filtered hit	for every search is stored in
       index.html.  When a hit is no longer found by the search	engine it a
       removed from index.html.	 As new	results	for a search are returned from
       the search engine they are placed in index.html.

       At the bottom of	index.html, there is a heading "Weekly Search
       Results", which is updated each time the	search is submitted (see
       "AUTOMATED SEARCHING").	The list of search runs	is stored in reverse
       chronological order.  Runs which	provide	no new information are
       identified with

	       No Unique Results found for search on <date>

       Runs which contain changes are identified by

	       Web search results for search on	<date>

       which will be linked a page detailing the changes from that run.

       Detailed	search results are noted in weekly files.  These files are
       named YYYYMMDD.html and are stored in the qid directory.	 The weekly
       files include THE URL, title, and a the description (if available).
       The title is a link to the original web page.

       On UNIX-like systems, cron(1) may be used to establish periodic
       searches	and the	web pages will be maintained by	AutoSearch.  To
       establish the first search, use the first example under SYNOPSIS.  You
       must specify the	qid, query name	and query string.  If any of the items
       are missing, you	will be	interactively prompted for the missing

       Once the	first search is	complete you can re-run	the search with	the
       second form under SYNOPSIS.

       A cron entry like:

	   0 3 * * 1 /nfs/u1/wls/ /www/div7/lsam/autosearch/caching

       might be	used to	run the	search each Monday at 3:00 AM.	The query name
       and query string	may be repeated; but they will not be used.  This
       means that with a cron line like:

	   0 3 * * 1 /nfs/u1/wls/ /www/div7/lsam/autosearch/caching -n caching -s caching

       a whole new search series can be	originated by

	   rm -r /www/div7/lsam/autosearch/caching

       However,	the only reason	to start a new search series would be to throw
       away the	old weekly files.

       We don't	recommend running searches more	than once per day, but if so
       the per-run files will be updated in-place.  Any	changes	are added to
       the page	with a comment that "Recently Added:"; and deletions are
       indicated with "Recently	Suspended:."

       The basic format	of these two pages is simple and customizable.	One
       requirement is that the basic structure remain unchanged.  HTML
       comments	are used to identify sections of the document.	Almost
       everything can be changed except	for the	strings	which identify the
       section starts and ends.

       Noteworthy tags and their meaning:

		       The text	contained within this tag is placed at the top
		       of the output page.  If the text	contains AutoSearch
		       WEB Searching, then the query name will replace it.  If
		       the text	does not contain this magic string and it is
		       the first ever search, the user will be asked for a
		       query name.

		       The text	contained between the braces is	the query
		       string.	This is	how AutoSearch maintains the query
		       string.	You may	edit this string to change the query
		       string; but only	in qid/index.html.  The	text ask user
		       is special and will force AutoSearch to request the
		       search string from the user.

		       The text	contained between the braces is	the search
		       engine.	Other engines supported	are HotBot and Lycos.
		       You may edit this string	to change the engine used; but
		       only in qid/index.html.	The text ask user is special
		       and will	force AutoSearch to to request the search
		       string from the user.

		       The text	contained between the braces specifies a query
		       options.	 Multiple occurrencs of	this command are
		       allowed to specify multiple options.

		       The text	contained between the braces is	the URL
		       filter.	This is	how AutoSearch maintains the filter.
		       Again you may edit this string to change	the query
		       string; but only	in qid/index.html.  The	text ask user
		       is special and will force AutoSearch to ask the user
		       (STDIN) for the query string.  When setting up the
		       first search, you must edit first_index.html, not
		       qid/index.html.	The URL	filter is a standard perl5
		       regular expression.  URLs which do not match will be

		       The text	contained within this tag is placed at the
		       bottom of the output page.  This	is a good place	to put
		       navigation, page	owner information, etc.

       The remainder of	the tags fall into a triplet of	~Heading, ~Template,
       and ~, where ~ is Summary, Weekly, Appended, and	Suspended. The sub-
       sections	appear in the order given.  To produce a section AutoSearch
       outputs the heading, the	template, the section, n copies	of the
       formatted data, and an /section.	 The tags and their function are:

       ~Heading	       The heading tag identifies the heading for a section of
		       the output file.	 The SummaryHeading is for the summary
		       portion,	etc.  The section may be empty (e.g.,
		       Suspended) and thus no heading is output.

       ~Template       The template tag	identifies how each item is to be
		       formatted.  Simple text replacement is used to change
		       the template into the actual output text.  The text to
		       be replaced is noted in ALLCAPS.

       ~	       This tag	is used	to locate the section (Summary,
		       Weekly, etc.).  This section represents the actual
		       n-items of data.

       You can edit these values in the	qid/index.html page of an existing
       search.	The file first_index.html (in the directory above qid) will be
       used as a default template for new queries.

       Examples	of these files can be seen in the pages	under
       "", or in the output generated
       by a new	AutoSearch.

       first_index.html	   optional file to determine the default format of
			   the index.html file of a new	query.

       first_date.html	   optional file to determine the default format of
			   the YYYYMMDD.html file for a	new query.

       qid/index.html	   (automatically created) latest search results, and
			   reverse chronological list of periodic searches.

       qid/date.html	   file	used as	a template for the YYYYMMDD.html

       qid/YYYYMMDD.html   (automatically created) summary of changes for a
			   particular date (AKA	'Weekly' file).

       Optional	files first_index.html and first_date.html are used for	the
       initial search as a template for	qid/index.html and date.html,
       respectively.  If either	of these files does not	exist; a default-
       default template	is stored within the AutoSearch	source.	 The intention
       of these	two files is to	permit a user to establish a framework for a
       group of	search sets which have a common	format.	 By leaving the
       default query name and query string alone, they will be overridden by
       command line inputs.

       For the library,	see WWW::Search, for the perl regular expressions, see

       Wm. L. Scheding

       AutoSearch is a re-implementation of an earlier version written by
       Kedar Jog.

       Copyright (C) 1996-1997 University of Southern California.  All rights

       Redistribution and use in source	and binary forms are permitted
       provided	that the above copyright notice	and this paragraph are
       duplicated in all such forms and	that any documentation,	advertising
       materials, and other materials related to such distribution and use
       acknowledge that	the software was developed by the University of
       Southern	California, Information	Sciences Institute.  The name of the
       University may not be used to endorse or	promote	products derived from
       this software without specific prior written permission.


       These are good ideas that people	have suggested.

       URL validation.
	   Validate the	status of each URL (with HTTP HEAD requests) and
	   indicate this status	in the output.

	   It should be	possible to merge the results of searches from two
	   search-engines.  If this merger were	done as	a new search engine,
	   this	operation would	be transparent to AutoSearch.

       None known at this time;	please inform the maintainer
       if any crop up.

perl v5.32.0			  2020-04-03			 AUTOSEARCH(1)


Want to link to this manual page? Use this URL:

home | help