/usr/local/lib64/python3.6/site-packages/pyarrow/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/pyarrow/__pycache__/hdfs.cpython-36.pyc (7055B)
3
8%Egg @ s d dl Z d dlZd dlZd dlZd dlmZmZ d dlmZ d dl j
Z
G dd de
j eZ dd Z dd Z
d
d Zd d
ZdddZdddZdS ) N)
implements _DEPR_MSG)
FileSystemc s e Zd ZdZd ddZdd Zd
d Zeej fd d
Z eej
fddZ
eej d fdd Z fddZ eej
fddZ
eej fddZd fdd Zdd Z ZS ) HadoopFileSystema
DEPRECATED: FileSystem interface for HDFS cluster.
See pyarrow.hdfs.connect for full connection details
.. deprecated:: 2.0
``pyarrow.hdfs.HadoopFileSystem`` is deprecated,
please use ``pyarrow.fs.HadoopFileSystem`` instead.
defaultr Nlibhdfsc C s> t jtjdddtdd |dkr(t | j||||| d S )Nzhdfs.HadoopFileSystemz2.0.0zfs.HadoopFileSystem )
stacklevelr )warningswarnr format
FutureWarning_maybe_set_hadoop_classpath_connect)selfhostportuser kerb_ticketZdriver
extra_conf r 8/usr/local/lib64/python3.6/site-packages/pyarrow/hdfs.py__init__( s
zHadoopFileSystem.__init__c C s t | j| j| j| j| jffS )N)r r r r r r )r r r r
__reduce__3 s zHadoopFileSystem.__reduce__c C s dS )zR
Return True if this is a Unix-style file store with directories.
Tr )r r r r _isfilestore7 s z HadoopFileSystem._isfilestorec s t j|S )N)superisdir)r path) __class__r r r = s zHadoopFileSystem.isdirc s t j|S )N)r isfile)r r )r r r r A s zHadoopFileSystem.isfileFc s t j||S )N)r delete)r r recursive)r r r r E s zHadoopFileSystem.deletec s t j|S )a
Create directory in HDFS.
Parameters
----------
path : str
Directory path to create, including any parent directories.
Notes
-----
libhdfs does not support create_parents=False, so we ignore this here
)r mkdir)r r kwargs)r r r r" I s
zHadoopFileSystem.mkdirc s t j||S )N)r rename)r r new_path)r r r r$ X s zHadoopFileSystem.renamec s t j|S )N)r exists)r r )r r r r& \ s zHadoopFileSystem.existsc s t j||S )ap
Retrieve directory contents and metadata, if requested.
Parameters
----------
path : str
HDFS path to retrieve contents of.
detail : bool, default False
If False, only return list of paths.
Returns
-------
result : list of dicts (detail=True) or strings (detail=False)
)r ls)r r detail)r r r r' ` s zHadoopFileSystem.lsc c sR | j |dd}t||\}}|||fV x$|D ] }| j| j||E dH q.W dS )a
Directory tree generator for HDFS, like os.walk.
Parameters
----------
top_path : str
Root directory for tree traversal.
Returns
-------
Generator yielding 3-tuple (dirpath, dirnames, filename)
T)r( N)r' _libhdfs_walk_files_dirswalk
_path_join)r top_pathcontents directoriesfilesdirnamer r r r* q s
zHadoopFileSystem.walk)r r NNr N)F)F)__name__
__module__ __qualname____doc__r r r r r r r r r" r$ r& r' r*
__classcell__r r )r r r s
r c C st dd l } | jdtjjddr"d S dtjkrXtjdkr>t }q`djtjd }t |}nt d}|j
d tjd< d S )
Nr zhadoop-common[^/]+.jarZ CLASSPATH HADOOP_HOMEwin32z
{}/bin/hadoopZhadoopzutf-8) researchosenvirongetsysplatform_derive_hadoop_classpathr _hadoop_classpath_globdecode)r9 classpath
hadoop_binr r r r s
r c C s dd l } ddtjd ddf}| j|| jd}| jd|j| jd
}| jd|jd}dtjkrdtjd n tjd d }|d jd| S )Nr findz-Lr7 z-namez*.jar)stdoutxargsecho)stdinrF tr' '':')rI ZHADOOP_CONF_DIRz /etc/hadoop:zutf-8)rG rH )rJ rK rL )
subprocessr; r< PopenPIPErF check_outputencode)rN Z find_argsrE Z
xargs_echoZjarsZ hadoop_confr r r r@ s
r@ c C s dd l }| ddf}|j|S )Nr rC z--glob)rN rQ )rD rN Zhadoop_classpath_argsr r r rA s
rA c C sR g }g }x@|D ]8}t j|d d }|d dkr<|j| q|j| qW ||fS )Nname kindfile) posixpathsplitappend)r, r- r/ r. cZ
scrubbed_namer r r r) s
r) r c C s, t jtjdddtdd t| ||||dS )a
DEPRECATED: Connect to an HDFS cluster.
All parameters are optional and should only be set if the defaults need
to be overridden.
Authentication should be automatic if the HDFS cluster uses Kerberos.
However, if a username is specified, then the ticket cache will likely
be required.
.. deprecated:: 2.0
``pyarrow.hdfs.connect`` is deprecated,
please use ``pyarrow.fs.HadoopFileSystem`` instead.
Parameters
----------
host : NameNode. Set to "default" for fs.defaultFS from core-site.xml.
port : NameNode's port. Set to 0 for default or logical (HA) nodes.
user : Username when connecting to HDFS; None implies login user.
kerb_ticket : Path to Kerberos ticket cache.
extra_conf : dict, default None
extra Key/Value pairs for config; Will override any
hdfs-site.xml properties
Notes
-----
The first time you call this method, it will take longer than usual due
to JNI spin-up time.
Returns
-------
filesystem : HadoopFileSystem
z hdfs.connectz2.0.0zfs.HadoopFileSystemr )r )r r r r r )r
r r r r
r )r r r r r r r r connect s #
r[ c C s4 t j " t jd t| ||||d}W d Q R X |S )Nignore)r r r r r )r
catch_warnings simplefilterr )r r r r r fsr r r r s
r )r r NNN)r r NNN)r; rW r> r
Z pyarrow.utilr r Zpyarrow.filesystemr Zpyarrow._hdfsioZ_hdfsior r r@ rA r) r[ r r r r r
s
i
,