> Spans > Visualizing

Python For Data Science

Accessing spans If you're in a Jupyter notebook, use displacy.render otherwise,

use displacy.serve to start a web server and show the visualization in your browser.

spaCy Cheat Sheet Span indices are exclusive. So doc[2:4] is a span starting at token 2, up to – but not including! – token 4.
>>> doc = nlp("This is a text")

>>> from spacy import displacy

>>> span = doc[2:4]

Visualize dependencies
Learn spaCy online at www.DataCamp.com >>>
'a
span.text

text'
>>> doc = nlp("This is a sentence")

>>> displacy.render(doc, style="dep")

Creating a span manually

spaCy >>> from spacy.tokens import Span #Import the Span object

>>> doc = nlp("I live in New York") #Create a Doc object

>>> span = Span(doc, 3, 5, label="GPE") #Span for "New York" with label GPE (geopolitical)

>>> span.text

spaCy is a free, open-source library for advanced Natural Language 'New York’

processing (NLP) in Python. It's designed specifically for production use and Visualize named entities
helps you build applications that process and "understand" large volumes
>>> doc = nlp("Larry Page founded Google")

of text. Documentation: spacy.io

>>> $ pip install spacy

> Linguistic features >>> displacy.render(doc, style="ent")

>>> import spacy Attributes return label IDs. For string labels, use the attributes with an underscore. For example, token.pos_ .

Part-of-speech tags Predicted by Statistical model

> Statistical models >>> doc = nlp("This is a text.")

> Word vectors and similarity

>>> [token.pos_ for token in doc] #Coarse-grained part-of-speech tags

Download statistical models ['DET', 'VERB', 'DET', 'NOUN', 'PUNCT']

To use word vectors, you need to install the larger models ending in md or lg , for example en_core_web_lg .
>>> [token.tag_ for token in doc] #Fine-grained part-of-speech tags

['DT', 'VBZ', 'DT', 'NN', '.']

Predict part-of-speech tags, dependency labels, named entities

and more. See here for available models: spacy.io/models Comparing similarity
>>> $ python -m spacy download en_core_web_sm Syntactic dependencies Predicted by Statistical model
>>> doc1 = nlp("I like cats")

>>> doc2 = nlp("I like dogs")

Check that your installed models are up to date >>> doc = nlp("This is a text.")

>>> [token.dep_ for token in doc] #Dependency labels

>>>
>>>
doc1.similarity(doc2) #Compare 2 documents

doc1[2].similarity(doc2[2]) #Compare 2 tokens

['nsubj', 'ROOT', 'det', 'attr', 'punct']

>>> doc1[0].similarity(doc2[1:3]) # Comparetokens and spans
>>> $ python -m spacy validate >>> [token.head.text for token in doc] #Syntactic head token (governor)

['is', 'is', 'text', 'is', 'is']

Accessing word vectors
Loading statistical models
Named entities Predicted by Statistical model
>>> doc = nlp("I like cats") #Vector as a numpy array

>>> import spacy

>>> doc[2].vector #The L2 norm of the token's vector

>>> nlp = spacy.load("en_core_web_sm") # Load the installed model "en_core_web_sm" >>> doc = nlp("Larry Page founded Google")
>>> doc[2].vector_norm
>>> [(ent.text, ent.label_) for ent in doc.ents] #Text and label of named entity span

[('Larry Page', 'PERSON'), ('Google', 'ORG')]

> Documents and tokens > Syntax iterators

Processing text
> Pipeline components
Sentences Ususally needs the dependency parser
Functions that take a Doc object, modify it and return it.
Processing text with the nlp object returns a Doc object that holds all
>>> doc = nlp("This a sentence. This is another one.")

information about the tokens, their linguistic features and their relationships >>> [sent.text for sent in doc.sents] #doc.sents is a generator that yields sentence spans

['This is a sentence.', 'This is another one.']

>>> doc = nlp("This is a text")

Accessing token attributes Base noun phrases Needs the tagger and parser

>>> doc = nlp("This is a text")

Pipeline information >>> doc = nlp("I have a red car")

#doc.noun_chunks is a generator that yields spans

>>>[token.text for token in doc] #Token texts

>>> nlp = spacy.load("en_core_web_sm")

>>> [chunk.text for chunk in doc.noun_chunks]

['This', 'is', 'a', 'text']

>>> nlp.pipe_names
['I', 'a red car']
['tagger', 'parser', 'ner']

>>> nlp.pipeline

[('tagger', <spacy.pipeline.Tagger>),

> Label explanations ('parser', <spacy.pipeline.DependencyParser>),

('ner', <spacy.pipeline.EntityRecognizer>)]

>>> spacy.explain("RB")

'adverb'

Custom components
>>> spacy.explain("GPE")
Learn Data Skills Online at
'Countries, cities, states' def custom_component(doc): #Function that modifies the doc and returns it

print("Do something to the doc here!")

www.DataCamp.com
return doc

nlp.add_pipe(custom_component, first=True) #Add the component first in the pipeline

Components can be added first , last (default), or before or after an existing component.
> Extension attributes > Rule-based matching > Glossary
Custom attributes that are registered on the global Doc, Token and Span classes and become available as ._ .
>>> from spacy.tokens import Doc, Token, Span

Using the matcher Tokenization

>>> doc = nlp("The sky over New York is blue")
# Matcher is initialized with the shared vocab
Segmenting text into words, punctuation etc
>>> from spacy.matcher import Matcher

Attribute extensions With default value # Each dict represents one token and its attributes

>>> matcher = Matcher(nlp.vocab)

Lemmatization
# Add with ID, optional callback and pattern(s)

# Register custom attribute on Token class

>>> pattern = [{"LOWER": "new"}, {"LOWER": "york"}]

>>> Token.set_extension("is_color", default=False)

>>> matcher.add("CITIES", None, pattern)
Assigning the base forms of words, for example:

# Overwrite extension attribute with default value

# Match by calling the matcher on a Doc object

doc[6]._.is_color = True "was" → "be" or "rats" → "rat".

>>> doc = nlp("I live in New York")

>>> matches = matcher(doc)

# Matches are (match_id, start, end) tuples

Property extensions With getter and setter >>> for match_id, start, end in matches:
Sentence Boundary Detection
# Get the matched span by slicing the Doc

span = doc[start:end]

# Register custom attribute on Doc class

print(span.text)
Finding and segmenting individual sentences.
>>> get_reversed = lambda doc: doc.text[::-1]
'New York'
>>> Doc.set_extension("reversed", getter=get_reversed)

# Compute value of extension attribute with getter

Part-of-speech (POS) Tagging
>>> doc._.reversed

'eulb si kroY weN revo yks ehT'

Token patterns
Assigning word types to tokens like verb or noun.
# "love cats", "loving cats", "loved cats"

Method extensions Callable Method >>> pattern1 = [{"LEMMA": "love"}, {"LOWER": "cats"}]

# "10 people", "twenty people"

>>> pattern2 = [{"LIKE_NUM": True}, {"TEXT": "people"}]

Dependency Parsing
# Register custom attribute on Span class
# "book", "a cat", "the sea" (noun + optional article)

>>> has_label = lambda span, label: span.label_ == label

>>> pattern3 = [{"POS": "DET", "OP": "?"}, {"POS": "NOUN"}]
>>> Span.set_extension("has_label", method=has_label)
Assigning syntactic dependency labels,

# Compute value of extension attribute with method

describing the relations between individual

>>> doc[3:5].has_label("GPE")

True
Operators and quantifiers tokens, like subject or object.
Can be added to a token dict as the "OP" key
Named Entity Recognition (NER)
! Negate pattern and match exactly 0 times

Labeling named "real-world" objects,

? Make pattern optional and match 0 or 1 times

like persons, companies or locations.
+ Require pattern to match 1 or more times

Allow pattern to match 0 or more time

Text Classification
*

Assigning categories or labels to a whole

document, or parts of a document.

Statistical model
Process for making predictions based on examples.

Training
Updating a statistical model with new examples.

Learn Data Skills Online at

www.DataCamp.com