Server: appserver-7f0f8755-nginx-15961cad18524ec5a9db05f2a6a7e440
Current directory: /usr/lib/python2.7
Software: nginx/1.27.5
Shell Command
Create a new file
Upload file
File: robotparser.pyc
� ӫ[c @ s} d Z d d l Z d d l Z d g Z d d d � � YZ d d d � � YZ d d d � � YZ d e j f d � � YZ d S( s& robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt i����Nt RobotFileParserc B sb e Z d Z d d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z RS( ss This class provides a set of methods to read, parse and answer questions about a single robots.txt file. t c C s> g | _ d | _ t | _ t | _ | j | � d | _ d S( Ni ( t entriest Nonet default_entryt Falset disallow_allt allow_allt set_urlt last_checked( t selft url( ( s! /usr/lib/python2.7/robotparser.pyt __init__ s c C s | j S( s� Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. ( R ( R ( ( s! /usr/lib/python2.7/robotparser.pyt mtime! s c C s d d l } | j � | _ d S( sY Sets the time the robots.txt file was last fetched to the current time. i����N( t timeR ( R R ( ( s! /usr/lib/python2.7/robotparser.pyt modified* s c C s/ | | _ t j | � d d !\ | _ | _ d S( s, Sets the URL referring to a robots.txt file.i i N( R t urlparset hostt path( R R ( ( s! /usr/lib/python2.7/robotparser.pyR 2 s c C s� t � } | j | j � } g | D] } | j � ^ q" } | j � | j | _ | j d k rk t | _ nO | j d k r� | j d k r� t | _ n% | j d k r� | r� | j | � n d S( s4 Reads the robots.txt URL and feeds it to the parser.i� i� i� i� i� N( i� i� ( t URLopenert openR t stript closet errcodet TrueR R t parse( R t openert ft linet lines( ( s! /usr/lib/python2.7/robotparser.pyt read7 s c C sA d | j k r- | j d k r= | | _ q= n | j j | � d S( Nt *( t useragentsR R R t append( R t entry( ( s! /usr/lib/python2.7/robotparser.pyt _add_entryE s c C s& d } d } t � } | j � x�| D]�} | d 7} | s� | d k rZ t � } d } q� | d k r� | j | � t � } d } q� n | j d � } | d k r� | | } n | j � } | s� q&