/usr/local/lib64/python3.6/site-packages/pyarrow/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/pyarrow/__pycache__/feather.cpython-36.pyc (7739B)
3
9%Eg*$ @ s d dl Z d dlmZ d dlmZmZmZmZ d dljZ d dl
m Z d dl m
Z
d dlmZ dd ZG d d
d
Zd d Zd
ddhZdddZdddZdddZdS ) N) _pandas_api)CodecTable
concat_tablesschema)_feather) FeatherError)Versionc C s t jtdk rtdd S )Nz0.17.0z!feather requires pandas >= 0.17.0)r Z
loose_versionr ImportError r r ;/usr/local/lib64/python3.6/site-packages/pyarrow/feather.py_check_pandas_version s r
c @ s6 e Zd ZdZd ddZd
ddZdd Zdd
d ZdS )FeatherDataseta
Encapsulates details of reading a list of Feather files.
Parameters
----------
path_or_paths : List[str]
A list of file names
validate_schema : bool, default True
Check that individual file schemas are all the same / compatible
Tc C s || _ || _d S )N)pathsvalidate_schema)selfZ
path_or_pathsr r r r __init__/ s zFeatherDataset.__init__Nc C sn t | jd |d}|g| _|j| _x@| jdd D ].}t ||d}| jrT| j|| | jj| q2W t| jS )a,
Read multiple feather files as a single pyarrow.Table
Parameters
----------
columns : List[str]
Names of columns to read from the file
Returns
-------
pyarrow.Table
Content of the file as a table (of columns)
r )columns N)
read_tabler Z_tablesr r validate_schemasappendr )r r Z_filpathtabler r r r 3 s zFeatherDataset.read_tablec C s( | j j|j s$tdj|| j |j d S )Nz-Schema in {!s} was different.
{!s}
vs
{!s})r equals
ValueErrorformat)r Zpiecer r r r r L s z FeatherDataset.validate_schemasc C s t | j|dj|dS )a
Read multiple Parquet files as a single pandas DataFrame
Parameters
----------
columns : List[str]
Names of columns to read from the file
use_threads : bool, default True
Use multiple threads when converting to pandas
Returns
-------
pandas.DataFrame
Content of the file as a pandas DataFrame (of columns)
)r ) use_threads)r
r to_pandas)r r r r r r read_pandasS s zFeatherDataset.read_pandas)T)N)NT)__name__
__module__ __qualname____doc__r r r r r r r r r # s
r c C sN |j dkrd S |jtj tj fkr4tdj| ntdj| t|jd S )Nr zqColumn '{}' exceeds 2GB maximum capacity of a Feather binary column. This restriction may be lifted in the futurezkColumn '{}' of type {} was chunked on conversion to Arrow and cannot be currently written to Feather format)Z
num_chunkstypeextbinarystringr r str)namecolr r r check_chunked_overflowh s
r+ lz4ZzstdZ uncompressed c
C sh t jr(t t jr(t| t jjr(| j } t j| rvt j
| dd}|dkrzx0t |j j
D ]\}}|| } t|| qVW n| }|dkrt|jtt|jkrtd|dk rtd|dk rtdn8|dkrtjdrd }n |dk r|tkrtd
j|tytj||||||d W nL tk
rb t|tr\ytj | W n tj k
rZ Y nX Y nX dS ) a
Write a pandas.DataFrame to Feather format.
Parameters
----------
df : pandas.DataFrame or pyarrow.Table
Data to write out as Feather format.
dest : str
Local destination path.
compression : string, default None
Can be one of {"zstd", "lz4", "uncompressed"}. The default of None uses
LZ4 for V2 files if it is available, otherwise uncompressed.
compression_level : int, default None
Use a compression level particular to the chosen compressor. If None
use the default compression level
chunksize : int, default None
For V2 files, the internal maximum size of Arrow RecordBatch chunks
when writing the Arrow IPC file format. None means use the default,
which is currently 64K
version : int, default 2
Feather file version. Version 2 is the current. Version 1 is the more
limited legacy format
F)Zpreserve_indexr z'cannot serialize duplicate column namesNz2Feather V1 files do not support compression optionz0Feather V1 files do not support chunksize optionZ lz4_framer, z1compression="{}" not supported, must be one of {}) compressioncompression_level chunksizeversion) r Z have_pandasr
Z
has_sparse
isinstancepdZSparseDataFrameZto_denseZ
is_data_framer Z from_pandas enumerater namesr+ lenZ column_namessetr r Z is_available_FEATHER_SUPPORTED_CODECSr r
write_feather Exceptionr( osremoveerror)
dfdestr. r/ r0 r1 r ir) r* r r r r9 z sJ
r9 Tc C s t t| ||dj|dS )a
Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use
feather.read_table.
Parameters
----------
source : str file path, or file-like object
columns : sequence, optional
Only read a specific set of columns. If not provided, all columns are
read.
use_threads : bool, default True
Whether to parallelize reading using multiple threads. If false the
restriction is only used in the conversion to Pandas and not in the
reading from Feather format.
memory_map : boolean, default True
Use memory mapping when opening file on disk
Returns
-------
df : pandas.DataFrame
)r
memory_map)r )r
r r )sourcer r rA r r r read_feather s rC c C s t j| |d}|dkr |j S dd |D }ttdd |rJ|j|}n
s z read_table..c S s | t kS )N)int)tr r r s z read_table..c S s | t kS )N)r( )rH r r r rI s c S s g | ]
}|j qS r )r )rD rH r r r rF s z s
E
J