/usr/local/lib64/python3.6/site-packages/tokenizers/tools/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-36.pyc (11513B)
3
Eg9 @ s d dl Z d dlZd dlZd dlmZmZmZmZmZm Z m
Z
d dl m Z d dlmZ d dl
mZmZ e jjeZe jjedZeeZej ZW dQ R X G dd dZee Zeee Z G d d
d
e
Z G d d d Z G d
d dZ G dd dZ efee! e!dddZ"dS ) N)ListOptionalTupleDictCallableAny
NamedTuple)Template)r ) TokenizerEncodingzvisualizer-styles.cssc @ s, e Zd ZU eeeeeedddZdS )
Annotation)startendlabelc C s || _ || _|| _d S )N)r
r r )selfr
r r r G/usr/local/lib64/python3.6/site-packages/tokenizers/tools/visualizer.py__init__ s zAnnotation.__init__N) __name__
__module__ __qualname__intr
r r strr r r r r r s
r c @ s e Zd ZU ee ee dS ) CharStateKeyN)r r r r r token_ixanno_ixr r r r r s
r c @ sD e Zd ZU ee dd Zedd Zedd Z e
ddd Z d
S ) CharStatec C s || _ d | _g | _d S )N)char_ixr tokens)r r r r r r ' s zCharState.__init__c C s t | jdkr| jd S d S )Nr )lenr )r r r r r - s zCharState.token_ixc C s t | jdkS )zJ
BPE tokenizers can output more than one token for a char
)r r )r r r r
is_multitoken1 s zCharState.is_multitoken)returnc C s t | j| jdS )N)r r )r r r )r r r r
partition_key8 s zCharState.partition_keyN) r r r r r r r propertyr r! r r# r r r r r $ s
r c @ s e Zd ZdS )AlignedN)r r r r r r r r% ? s r% c @ s e Zd ZdZejdejdZdee e
e e ge
f dddZg dfeee
e e
e d d
d Zeeeeef d d
dZeee eedddZeeeeedddZeeeedddZeeeeee dddZ dS )EncodingVisualizera
Build an EncodingVisualizer
Args:
tokenizer (:class:`~tokenizers.Tokenizer`):
A tokenizer instance
default_to_notebook (:obj:`bool`):
Whether to render html output in a notebook by default
annotation_converter (:obj:`Callable`, `optional`):
An optional (lambda) function that takes an annotation in any format and returns
an Annotation object
z(.{1})?(unk|oov)(.{1})?)flagsTN) tokenizerdefault_to_notebookannotation_converterc C sZ |rDyddl m}m} W n* tk
rB } ztdW Y d d }~X nX || _|| _|| _d S )Nr )displayHTMLzWe couldn't import IPython utils for html display.
Are you running in a notebook?
You can also pass `default_to_notebook=False` to get back raw HTML
)IPython.core.displayr+ r, ImportError Exceptionr( r) annotation_coverter)r r( r) r* r+ r, er r r r V s zEncodingVisualizer.__init__)text annotationsr) r" c
C s | j }|dk r|}|rVyddlm}m} W n* tk
rT } ztdW Y dd}~X nX | jdk rptt| j|}| j j
|}t j |||} |r||| n| S dS )a
Build a visualization of the given text
Args:
text (:obj:`str`):
The text to tokenize
annotations (:obj:`List[Annotation]`, `optional`):
An optional list of annotations of the text. The can either be an annotation class
or anything else if you instantiated the visualizer with a converter function
default_to_notebook (:obj:`bool`, `optional`, defaults to `False`):
If True, will render the html in a notebook. Otherwise returns an html string.
Returns:
The HTML string if default_to_notebook is False, otherwise (default) returns None and
renders the HTML in the notebook
Nr )r+ r, zeWe couldn't import IPython utils for html display.
Are you running in a notebook?)
r) r- r+ r, r. r/ r0 listmapr( encoder& _EncodingVisualizer__make_html)
r r2 r3 r) Zfinal_default_to_notebookr+ r, r1 encodinghtmlr r r __call__l s
zEncodingVisualizer.__call__)r3 r" c C s t | dkri S ttdd | }t |}td| }|dk rBd}d}d}d}i }x4t|D ](}d | d
| d | d ||< ||7 }q\W |S )
a
Generates a color palette for all the labels in a given set of annotations
Args:
annotations (:obj:`Annotation`):
A list of annotations
Returns:
:obj:`dict`: A dictionary mapping labels to colors in HSL format
r c S s | j S )N)r )xr r r
s z;EncodingVisualizer.calculate_label_colors.. @
zhsl(,z%,%)r setr5 r sorted) r3 labelsZ
num_labelsZh_stepslhcolorsr r r r calculate_label_colors s
z)EncodingVisualizer.calculate_label_colors)consecutive_chars_listr2 r8 c C s2 | d }|j dkr*|j|j }d| dS | d }|j }|j d }||| }g } i }
|jdk r| jd |jrz| jd |jd r| jd n
| jd
tjj|j|j dk r| jd |j|j |
d < n
| jd
ddj| d} d} x*|
j D ] \}
}| d|
d| d7 } qW d| d| d| dS )a
Converts a list of "consecutive chars" into a single HTML element.
Chars are consecutive if they fall under the same word, token and annotation.
The CharState class is a named tuple with a "partition_key" method that makes it easy to
compare if two chars are consecutive.
Args:
consecutive_chars_list (:obj:`List[CharState]`):
A list of CharStates that have been grouped together
text (:obj:`str`):
The original text being processed
encoding (:class:`~tokenizers.Encoding`):
The encoding returned from the tokenizer
Returns:
:obj:`str`: The HTML span for a set of consecutive chars
r Nz(r tokenz multi-token z odd-tokenz
even-tokenz
special-tokenZstokz non-tokenzclass=" " z data-z="zz)
r r r appendr! r& unk_token_regexsearchjoinitems)rL r2 r8 firstZstokenlastr
r Z span_textZ css_classesZ
data_itemscssdatakeyvalr r r consecutive_chars_to_html s6
z,EncodingVisualizer.consecutive_chars_to_html)r2 r8 r3 r" c C s\ t j| ||}|d g}|d j}g }t j|}|d j}|d k rp|| } | j}
||
} |jd| d|
d x|dd D ]} | j}||kr|jt j|| |d | g}|d k r|jd |d k r|| } | j}
||
} |jd| d|
d |}| j |d j kr|j| q~|jt j|| |d | g}q~W |jt j|| |d t|}
|
S )Nr z&r )r2 r8 z) r& %_EncodingVisualizer__make_char_statesr rK r rS r^ r# HTMLBody)r2 r8 r3 char_statesZcurrent_consecutive_charsZ prev_anno_ixZspansZlabel_colors_dictZ cur_anno_ixannor colorcsresr r r Z __make_html sX
z EncodingVisualizer.__make_html)r2 r3 r" c C sH dgt | }x4t|D ](\}}x t|j|jD ] }|||< q0W qW |S )a
Args:
text (:obj:`str`):
The raw text we want to align to
annotations (:obj:`AnnotationList`):
A (possibly empty) list of annotations
Returns:
A list of length len(text) whose entry at index i is None if there is no annotation on
charachter i or k, the index of the annotation that covers index i where k is with
respect to the list of annotations
N)r enumerateranger
r )r2 r3 annotation_mapr air r r Z__make_anno_mapC s
z"EncodingVisualizer.__make_anno_mapc
C s t j| |}dd tt| D }xTt|jD ]F\}}|j|}|dk r.|\}} x"t|| D ]}
||
jj| q\W q.W x t|D ]\} } | || _qW |S )a
For each character in the original text, we emit a tuple representing it's "state":
* which token_ix it corresponds to
* which word_ix it corresponds to
* which annotation_ix it corresponds to
Args:
text (:obj:`str`):
The raw text we want to align to
annotations (:obj:`List[Annotation]`):
A (possibly empty) list of annotations
encoding: (:class:`~tokenizers.Encoding`):
The encoding returned from the tokenizer
Returns:
:obj:`List[CharState]`: A list of CharStates, indicating for each char in the text what
it's state is
c S s g | ] }t |qS r )r ).0r r r r
s s z9EncodingVisualizer.__make_char_states..N) r& "_EncodingVisualizer__make_anno_maprg r rf r Ztoken_to_charsrS r )
r2 r8 r3 rh ra r rM offsetsr
r rj r r r r r Z__make_char_statesX s
z%EncodingVisualizer.__make_char_states)TN) r r r __doc__recompile
IGNORECASErT r
boolr r r r r r AnnotationListr: staticmethodr rK r r r r^ r7 PartialIntListrm r_ r r r r r& C s( "( CCr& )childrenr" c C s dj | }d| d| dS )a[
Generates the full html with css from a list of html spans
Args:
children (:obj:`List[str]`):
A list of strings, assumed to be html elements
css_styles (:obj:`str`, `optional`):
Optional alternative implementation of the css
Returns:
:obj:`str`: An HTML string with style markup
rQ z?
z4
)rV )rw Z
css_stylesZ
children_textr r r r` s
r` )#os itertoolsrp typingr r r r r r r stringr Z
tokenizersr
r pathdirname__file__rV Z css_filenameopenfreadrZ r rt r rv r r r% r& r r` r r r r s( $
?