File manager

�
�V~gc@s}dZd
dl
Z
d
dlZdg
Zddd��YZddd��YZdd
d��YZd	ejf
d
��YZdS(s&
 robotparser.py

    Copyright (C) 2000  Bastian Kleineidam

    You can choose between two licenses when using this package:
    1) GNU GPLv2
    2) PSF license for Python 2.2

    The robots.txt Exclusion Protocol is implemented as specified in
    http://www.robotstxt.org/norobots-rfc.txt

i����NtRobotFileParsercBsbeZ
dZd
d�
Zd�Zd�Zd�Zd�Zd�Zd�Z	d	�Z
d
�ZRS(ss This class provides a set of methods to read, parse and answer
    questions about a single robots.txt file.

    tcCs>g|_d|_t|_t|_|j|
�

d
|_dS(Ni(tentriestNonet
default_entrytFalsetdisallow_allt	allow_alltset_urltlast_checked(tselfturl((s0/opt/alt/python27/lib64/python2.7/robotparser.pyt__init__s
	
	
	
	


c


Cs|jS(
s�Returns the time the robots.txt file was last fetched.

        This is useful for long-running web spiders that need to
        check for new robots.txt files periodically.

        (
R	(
R
((s0/opt/alt/python27/lib64/python2.7/robotparser.pytmtime!sc
Csd
dl}
|
j�|_
dS(sYSets the time the robots.txt file was last fetched to the
        current time.

        i����N(ttimeR	(R
R((s0/opt/alt/python27/lib64/python2.7/robotparser.pytmodified*s
cCs/|
|_t
j
|
�
d
d!\|_|_dS(s,Sets the URL referring to a robots.txt file.i
iN(Rturlparsethosttpath(R
R((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR2s	
c
Cs�t�}
|
j
|j�
}g|D]}|j�^q"}|j�
|
j|_|jdkrkt|_nO|jdkr�|jdkr�t|_n%|jdkr�|r�|j	|�

ndS(s4Reads the robots.txt URL and feeds it to the parser.i�
i�
i�
i�
i�N(i�
i�
(
t	URLopenertopenRtstriptcloseterrcodetTrueRRtparse(R
topenert
ftlinetlines((s0/opt/alt/python27/lib64/python2.7/robotparser.pytread7s	










cCsAd
|
jkr-|j
dkr=|
|_
q=n|jj|
�

dS(Nt
*(t
useragentsRRRtappend(R
tentry((s0/opt/alt/python27/lib64/python2.7/robotparser.pyt
_add_entryEs
cCs&d
}d
}t�}|j
�
x�
|
D]�
}|d7}|s�|dkrZt�}d
}q�|dkr�|j|�

t�}d
}q�n|jd�
}|d
kr�|| }n|j�}|s�q&n|jdd�}t|�
dkr&|d
j�j�|d
<tj	|dj��
|d<|d
dkru
|dkrX
|j|�

t�}n|j
j|d�

d}q|d
dkr�
|d
kr�
|jjt
|dt��

d}q�
q|d
dkr|d
kr�
|jjt
|dt��

d}q�
qq&q&W|dkr"|j|�

nd	S(
s�parse the input lines from a robots.txt file.
           We allow that a user-agent: line is not preceded by
           one or more blank lines.ii
it
#t
:s
user-agenttdisallowtallowN(tEntryRR#tfindRtsplittlentlowerturllibtunquoteR R!t	rulelinestRuleLineRR(R
Rtstatet
linenumberR"Rt
i((s0/opt/alt/python27/lib64/python2.7/robotparser.pyRNsP

	







	
	



	

















	









cCs�|jr
t
S|jrtS|js't
Stjtj|�
�
}tjd
d
|j	|j
|j|jf�
}tj
|�
}|s�d}nx-|jD]"}|j|
�
r�|j|�
Sq�W|jr�|jj|�
StS(s=using the parsed robots.txt decide if useragent can fetch urlR
t
/(RRRRR	RR-R.t
urlunparseRtparamstquerytfragmenttquoteRt
applies_tot	allowanceR(R
t	useragentRt
parsed_urlR"((s0/opt/alt/python27/lib64/python2.7/robotparser.pyt	can_fetch�s$	

	
	





	


	
c
CsE|j}
|j
dk	r+|
|j
g
}
nd
jtt|
��
d
S(Ns

(RRRtjointmaptstr(R
R((s0/opt/alt/python27/lib64/python2.7/robotparser.pyt__str__�s
	


(t__name__t
__module__t__doc__RR
RRRR#RR>RB(((s0/opt/alt/python27/lib64/python2.7/robotparser.pyRs								4	 R0c
Bs)eZ
dZd
�Zd�Zd�ZRS(soA rule line is a single "Allow:" (allowance==True) or "Disallow:"
       (allowance==False) followed by a path.cCsS|
d
kr|rt}nt
jt
j
|
�
�
}
tj|
�
|_||_dS(NR
(RRR5R-R9RR;(R
RR;((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR�s

	


cCs|jd
kp|
j
|j�
S(NR(Rt
startswith(R
tfilename((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR:�s
c

Cs|jrd
pdd|j
S(NtAllowtDisallows: (R;R(
R
((s0/opt/alt/python27/lib64/python2.7/robotparser.pyRB�s
(RCRDRERR:RB(((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR0�s
		R(c
Bs2eZ
dZd
�Zd�Zd�Zd�ZRS(s?An entry has one or more user-agents and zero or more rulelinesc

Csg|_g|_
dS(
N(R R/(
R
((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR�s
	
c
Csjg}
x'|jD]}|
j
d
|dg�

qWx*|jD]}|
j
t|�
dg�

q:Wdj|
�
S(NsUser-agent: s

R
(R textendR/RAR?(R
trettagentR((s0/opt/alt/python27/lib64/python2.7/robotparser.pyRB�s





cCs]|
jd
�
dj
�}
x=|jD]2}|dkr9tS|j
�}||
kr#tSq#WtS(s2check if this entry applies to the specified agentR4iR(R*R,R RR(R
R<RL((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR:�s





cCs.x'|jD]}|j
|
�
r
|jSq
WtS(
sZPreconditions:
        - our agent applies to this entry
        - filename is URL decoded(R/R:R;R(R
RGR((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR;�s


(RCRDRERRBR:R;(((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR(�s


			
Rc
Bs#eZ
d�Zd
�Zd�ZRS(c
Gs tj
j||
�

d
|_dS(Ni�(R-tFancyURLopenerRR(R
targs((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR�s

cCsd
S(N(NN(
R(R
Rtrealm((s0/opt/alt/python27/lib64/python2.7/robotparser.pytprompt_user_passwd�scCs(||_t
jj||
||||�S(
N(RR-RMthttp_error_default(R
RtfpRterrmsgtheaders((s0/opt/alt/python27/lib64/python2.7/robotparser.pyRQ�s
	

(RCRDRRPRQ(((s0/opt/alt/python27/lib64/python2.7/robotparser.pyR�s
		((((	RERR-t__all__RR0R(RMR(((s0/opt/alt/python27/lib64/python2.7/robotparser.pyt<module>s

	�$