grthtrhthjhtyjytjytkergtrhtrjytjerhrfh3 \€"ã@s\dZddlZddlZddlZdgZejddƒZGdd„dƒZGdd„dƒZ Gd d „d ƒZ dS) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt éNÚRobotFileParserÚ RequestRatezrequests secondsc@sjeZdZdZddd„Zdd„Zdd„Zd d „Zd d „Zd d„Z dd„Z dd„Z dd„Z dd„Z dd„ZdS)rzs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. ÚcCs,g|_d|_d|_d|_|j|ƒd|_dS)NFr)ÚentriesÚ default_entryÚ disallow_allÚ allow_allÚset_urlÚ last_checked)ÚselfÚurl©r ú*/usr/lib64/python3.6/urllib/robotparser.pyÚ__init__s  zRobotFileParser.__init__cCs|jS)z·Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r )r r r rÚmtime$szRobotFileParser.mtimecCsddl}|jƒ|_dS)zYSets the time the robots.txt file was last fetched to the current time. rN)Útimer )r rr r rÚmodified-szRobotFileParser.modifiedcCs&||_tjj|ƒdd…\|_|_dS)z,Sets the URL referring to a robots.txt file.ééN)r ÚurllibÚparseÚurlparseÚhostÚpath)r r r r rr 5szRobotFileParser.set_urlcCs†ytjj|jƒ}WnRtjjk rd}z2|jdkr:d|_n|jdkrT|jdkrTd|_WYdd}~XnX|j ƒ}|j |j dƒj ƒƒdS) z4Reads the robots.txt URL and feeds it to the parser.é‘é“Ti�iôNzutf-8)rr) rZrequestZurlopenr ÚerrorZ HTTPErrorÚcoderrÚreadrÚdecodeÚ splitlines)r ÚfÚerrÚrawr r rr:s zRobotFileParser.readcCs,d|jkr|jdkr(||_n |jj|ƒdS)NÚ*)Ú useragentsrrÚappend)r Úentryr r rÚ _add_entryGs  zRobotFileParser._add_entrycCs6d}tƒ}|jƒ�x|D�]þ}|sT|dkr8tƒ}d}n|dkrT|j|ƒtƒ}d}|jdƒ}|dkrr|d|…}|jƒ}|s€q|jddƒ}t|ƒdkr|djƒjƒ|d<tj j |djƒƒ|d<|ddk�r|dkrê|j|ƒtƒ}|j j |dƒd}q|ddk�r4|dk�r|j j t|dd ƒƒd}q|dd k�rh|dk�r|j j t|dd ƒƒd}q|dd k�r¦|dk�r|djƒjƒ�r t|dƒ|_d}q|dd kr|dkr|djdƒ}t|ƒdk�r|djƒjƒ�r|djƒjƒ�rtt|dƒt|dƒƒ|_d}qW|dk�r2|j|ƒdS)z”Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. rréú#Nú:z user-agentZdisallowFZallowTz crawl-delayz request-rateú/)ÚEntryrr(ÚfindÚstripÚsplitÚlenÚlowerrrÚunquoter%r&Ú rulelinesÚRuleLineÚisdigitÚintÚdelayrÚreq_rate)r ÚlinesÚstater'ÚlineÚiZnumbersr r rrPsd             zRobotFileParser.parsecCs |jr dS|jrdS|jsdStjjtjj|ƒƒ}tjjdd|j|j |j |j fƒ}tjj |ƒ}|sfd}x"|j D]}|j|ƒrn|j|ƒSqnW|jrœ|jj|ƒSdS)z=using the parsed robots.txt decide if useragent can fetch urlFTrr,)rrr rrrr3Ú urlunparserZparamsZqueryZfragmentÚquoterÚ applies_toÚ allowancer)r Ú useragentr Z parsed_urlr'r r rÚ can_fetch“s$    zRobotFileParser.can_fetchcCs4|jƒs dSx|jD]}|j|ƒr|jSqW|jjS)N)rrr@r8r)r rBr'r r rÚ crawl_delay°s    zRobotFileParser.crawl_delaycCs4|jƒs dSx|jD]}|j|ƒr|jSqW|jjS)N)rrr@r9r)r rBr'r r rÚ request_rate¸s    zRobotFileParser.request_ratecCs0|j}|jdk r||jg}djtt|ƒƒdS)NÚ )rrÚjoinÚmapÚstr)r rr r rÚ__str__Às  zRobotFileParser.__str__N)r)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrrr rr(rrCrDrErJr r r rrs    Cc@s(eZdZdZdd„Zdd„Zdd„ZdS) r5zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.cCs>|dkr| rd}tjjtjj|ƒƒ}tjj|ƒ|_||_dS)NrT)rrr>rr?rrA)r rrAr r rrÊs zRuleLine.__init__cCs|jdkp|j|jƒS)Nr$)rÚ startswith)r Úfilenamer r rr@ÒszRuleLine.applies_tocCs|jr dndd|jS)NZAllowZDisallowz: )rAr)r r r rrJÕszRuleLine.__str__N)rKrLrMrNrr@rJr r r rr5Çsr5c@s0eZdZdZdd„Zdd„Zdd„Zdd „Zd S) r-z?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_d|_d|_dS)N)r%r4r8r9)r r r rrÛszEntry.__init__cCs�g}x|jD]}|jd|›�ƒq W|jdk r@|jd|j›�ƒ|jdk rj|j}|jd|j›d|j›�ƒ|jtt|j ƒƒ|jdƒdj |ƒS)Nz User-agent: z Crawl-delay: zRequest-rate: r,rrF) r%r&r8r9ZrequestsZsecondsÚextendrHrIr4rG)r ZretÚagentZrater r rrJás    z Entry.__str__cCsF|jdƒdjƒ}x.|jD]$}|dkr*dS|jƒ}||krdSqWdS)z2check if this entry applies to the specified agentr,rr$TF)r0r2r%)r rBrRr r rr@îs zEntry.applies_tocCs$x|jD]}|j|ƒr|jSqWdS)zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r4r@rA)r rPr<r r rrAûs   zEntry.allowanceN)rKrLrMrNrrJr@rAr r r rr-Ùs   r-) rNÚ collectionsZ urllib.parserZurllib.requestÚ__all__Ú namedtuplerrr5r-r r r rÚ s 2