Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
Gungho::Component::RobUserlContributed Perl DoGungho::Component::RobotRules(3)

NAME
       Gungho::Component::RobotRules - Respect robots.txt

SYNOPSIS
	 ---
	 components:
	   - RobotRules

METHODS
   setup
   setup_robot_rules_parser
   setup_robot_rules_storage
   handle_response
   send_request
   allowed($request)
       Returns 1 if request is allowed to be fetched, 0	if not.	-1 and -2 are
       returned	when there is a	pending	request	to fetch /robots.txt

   dispatch_pending_robots_txt
       Dispatches requests that	were pending because of	a missing robots.txt
       entry

   push_pending_robots_txt
       Pushes a	request	in the wait queue for a	robots.txt

   parse_robot_rules
       Parses the robot	rule and stores	it

perl v5.32.1			  2007-11-29  Gungho::Component::RobotRules(3)

NAME | SYNOPSIS | METHODS

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=Gungho::Component::RobotRules&sektion=3&manpath=FreeBSD+13.0-RELEASE+and+Ports>

home | help