Home
Workbench
LX-Tokenizer

File Processing

Notebook

Web Service

File processing

Input format: Input files must be in .txt FORMAT with UTF-8 ENCODING and contain PORTUGUESE TEXT. Input files and folders can also be compressed to the .zip format.

Privacy: The input file you upload and the respective output files will be automatically deleted from our computer after being processed and the result downloaded by you. No copies of your files will be retained after your use of this service.

Email address validation

The size of your input file is large and its processing may take some time.

To receive by email an URL from which to download your processed file, please copy the code displayed below into the field "Subject:" of an email message (with the message body empty) and send it to request@portulanclarin.net

To proceed, please send an email to request@portulanclarin.net with the following code in the "Subject" field:

To:	`request@portulanclarin.net`
Subject:

The communication with the server cannot be established. Please try again later.

We are sorry but an unexpected error has occurred. Please try again later.

The code has expired. Please click the button below to get a new code.

For enhanced security, a new code has to be validated. Please click the button below to get a new code.

Privacy: After we reply to you with the URL for download, your email address is automatically deleted from our records.

Designing your own experiment with a Jupyter Notebook

A Jupyter notebook (hereafter just notebook, for short) is a type of document that contains executable code interspersed with visualizations of code execution results and narrative text.

Below we provide an example notebook which you may use as a starting point for designing your own experiments using language resources offered by PORTULAN CLARIN.

Pre-requisites

To execute this notebook, you need an access key you can obtain by clicking the button below. A key is valid for 31 days. It allows to submit a total of 1 billion characters by means of requests with no more 200000 characters each. It allows to enter 100,000 requests, at a rate of no more than 200 requests per hour.

For other usage regimes, you should contact the helpdesk.

The input data sent to any PORTULAN CLARIN web service and the respective output will be automatically deleted from our computers after being processed. However, when running a notebook on an external service, such as the ones suggested below, you should take their data privacy policies into consideration.

Running the notebook

You have three options to run the notebook presented below:

Run on Binder — The Binder Project is funded by a 501c3 non-profit organization and is described in detail in the following paper:
Jupyter et al., "Binder 2.0 - Reproducible, Interactive, Sharable Environments for Science at Scale."
Proceedings of the 17th Python in Science Conference. 2018. doi://10.25080/Majora-4af1f417-011
Run on Google Colab — Google Colaboratory is a free-to-use product from Google Research.
Download the notebook from our public Github repository and run it on your computer.
This is a more advanced option, which requires you to install Python 3 and Jupyter on your computer. For anyone without prior experience setting up a Python development environment, we strongly recommend one of the two options above.

This is only a preview of the notebook. To run it, please choose one of the following options:

Run on Binder Run on Google Colab Download from Github

Using LX-Tokenizer to tokenize a text from the BDCamões corpus¶

This is an example notebook that illustrates how you can use the LX-Tokenizer web service to tokenize a sample text from the BDCamões corpus (the full corpus is available from the PORTULAN CLARIN repository).

Before you run this example, replace access_key_goes_here by your webservice access key, below:

In [1]:

LXTOKENIZER_WS_API_KEY = 'access_key_goes_here'
LXTOKENIZER_WS_API_URL = 'https://portulanclarin.net/workbench/lx-tokenizer/api/'

Importing required Python modules¶

The next cell will take care of installing the requests package, if not already installed, and make IT available to use in this notebook.

In [2]:

try:
    import requests
except:
    !pip3 install requests
    import requests
try:
    import matplotlib.pyplot as plt
except:
    !pip3 install matplotlib
    import matplotlib.pyplot as plt

Wrapping the complexities of the JSON-RPC API in a simple, easy to use function¶

The WSException class defined below, will be used later to identify errors from the webservice.

In [3]:

class WSException(Exception):
    'Webservice Exception'
    def __init__(self, errordata):
        "errordata is a dict returned by the webservice with details about the error"
        super().__init__(self)
        assert isinstance(errordata, dict)
        self.message = errordata["message"]
        # see https://json-rpc.readthedocs.io/en/latest/exceptions.html for more info
        # about JSON-RPC error codes
        if -32099 <= errordata["code"] <= -32000:  # Server Error
            if errordata["data"]["type"] == "WebServiceException":
                self.message += f": {errordata['data']['message']}"
            else:
                self.message += f": {errordata['data']!r}"
    def __str__(self):
        return self.message

The next function invokes the LX-Tokenizer webservice through its public JSON-RPC API.

In [4]:

def tokenize(text, format):
    '''
    Arguments
        text: a string with a maximum of 4000 characters, Portuguese text, with
             the input to be processed
        format: either 'CINTIL' or 'JSON'

    Returns a string with the output according to specification in
       https://portulanclarin.net/workbench/lx-tokenizer/
    
    Raises a WSException if an error occurs.
    '''

    request_data = {
        'method': 'tokenize',
        'jsonrpc': '2.0',
        'id': 0,
        'params': {
            'text': text,
            'format': format,
            'key': LXTOKENIZER_WS_API_KEY,
        },
    }
    request = requests.post(LXTOKENIZER_WS_API_URL, json=request_data)
    response_data = request.json()
    if "error" in response_data:
        raise WSException(response_data["error"])
    else:
        return response_data["result"]

The JSON output format¶

The JSON format (which we obtain by passing format="JSON" into the tokenize function) is the most convenient when we need to further process the annotations, because each abstraction is mapped directly into a Python native object (lists, dicts, strings, etc) as follows:

The returned object is a list, where each element corresponds to a paragraph of the given text;
In turn, each paragraph is a list where each element represents a sentence;
Each sentence is a list where each element represents a token;
Each token is a dict where each key-value pair is an attribute of the token.

In [5]:

text = '''Esta frase serve para testar o funcionamento do tokenizador. Esta outra
frase faz o mesmo.'''
tokenized_text = tokenize(text, format="JSON")
for pnum, paragraph in enumerate(tokenized_text, start=1): # enumerate paragraphs in text, starting at 1
    print(f"paragraph {pnum}:")
    for snum, sentence in enumerate(paragraph, start=1): # enumerate sentences in paragraph, starting at 1
        print(f"  sentence {snum}:")
        for tnum, token in enumerate(sentence, start=1): # enumerate tokens in sentence, starting at 1
            print(f"    token {tnum}: {token!r}")  # print a token representation

paragraph 1:
  sentence 1:
    token 1: {'form': 'Esta', 'space': 'LR'}
    token 2: {'form': 'frase', 'space': 'LR'}
    token 3: {'form': 'serve', 'space': 'LR'}
    token 4: {'form': 'para', 'space': 'LR'}
    token 5: {'form': 'testar', 'space': 'LR'}
    token 6: {'form': 'o', 'space': 'LR'}
    token 7: {'form': 'funcionamento', 'space': 'LR'}
    token 8: {'form': 'de_', 'space': 'L', 'raw': 'do'}
    token 9: {'form': 'o', 'space': 'R'}
    token 10: {'form': 'tokenizador', 'space': 'L'}
    token 11: {'form': '.', 'space': 'R'}
  sentence 2:
    token 1: {'form': 'Esta', 'space': 'LR'}
    token 2: {'form': 'outra', 'space': 'LR'}
    token 3: {'form': 'frase', 'space': 'LR'}
    token 4: {'form': 'faz', 'space': 'LR'}
    token 5: {'form': 'o', 'space': 'LR'}
    token 6: {'form': 'mesmo', 'space': 'L'}
    token 7: {'form': '.', 'space': 'R'}

Downloading and preparing our working text¶

In the next code cell, we will download a copy of the book "Viagens na minha terra" and prepare it to be used as our working text.

In [6]:

# A plain text version of this book is available from our Gitbub repository:
sample_text_url = "https://github.com/portulanclarin/jupyter-notebooks/raw/main/sample-data/viagensnaminhaterra.txt"

req = requests.get(sample_text_url)
sample_text_lines = req.text.splitlines()

num_lines = len(sample_text_lines)
print(f"The downloaded text contains {num_lines} lines")

# discard whitespace at beginning and end of each line:
sample_text_lines = [line.strip() for line in sample_text_lines]

# discard empty lines
sample_text_lines = [line for line in sample_text_lines if line]

# how many lines do we have left?
num_lines = len(sample_text_lines)
print(f"After discarding empty lines we are left with {num_lines} non-empty lines")

The downloaded text contains 2509 lines
After discarding empty lines we are left with 2205 non-empty lines

Tokenizing with the LX-Tokenizer web service¶

There is a limit on the number of web service requests per hour that can be made in association with any given key. Thus, we should send as much text as possible in each request while also conforming with the 200000 characters per request limit.

To this end, the following function slices our text into chunks smaller than 200K:

In [7]:

def slice_into_chunks(lines, max_chunk_size=200000):
    chunk, chunk_size = [], 0
    for lnum, line in enumerate(lines, start=1):
        if (chunk_size + len(line)) <= max_chunk_size:
            chunk.append(line)
            chunk_size += len(line) + 1
            # the + 1 above is for the newline character terminating each line
        else:
            yield "\n".join(chunk)
            if len(line) > max_chunk_size:
                print(f"line {lnum} is longer than 4000 characters; truncating")
                line = line[:4000]
            chunk, chunk_size = [line], len(line) + 1
    if chunk:
        yield "\n".join(chunk)

Next, we will apply slice_into_chunks to the sample text to get the chunks to be annotated.

In [8]:

chunks = list(slice_into_chunks(sample_text_lines))
tokenized_text = [] # tokenized paragraphs will be stored here
chunks_processed = 0  # this variable keeps track of which chunks have been processed already
print(f"There are {len(chunks)} chunks to be tokenized")

There are 2 chunks to be tokenized

Next, we will invoke tokenize on each chunk. If we get an exception while tokenizing a chunk:

check the exception message to determine what was the cause;
if the maximum number of requests per hour has been exceeded, then wait some time before retrying;
if a temporary error occurred in the webservice, try again later.

In any case, as long as the notebook is not shutdown or restarted, the text that has been tokenized thus far is not lost, and re-running the following cell will pick up from the point where the exception occurred.

In [9]:

for cnum, chunk in enumerate(chunks[chunks_processed:], start=chunks_processed+1):
    try:
        tokenized_text.extend(tokenize(chunk, format="JSON"))
        chunks_processed = cnum
        # print one dot for each tokenized chunk to get some progress feedback
        print(".", end="", flush=True)
    except Exception as exc:
        chunk_preview = chunk[:100] + "[...]" if len(chunk) > 100 else chunk
        print(
            f"\nError: tokenization of chunk {cnum} failed ({exc}); chunk contents:\n\n{chunk_preview}\n\n"
        )
        break

..

Let's create a sentence length histogram¶

In [10]:

%matplotlib inline
plt.rcParams.update({'figure.figsize':(20,5)})

x = [len(sentence) for paragraph in tokenized_text for sentence in paragraph]
plt.hist(x, bins=100)
plt.gca().set(title='Frequency Histogram', ylabel='Frequency');
# To learn more about matplotlib visit https://matplotlib.org/

Getting the status of a webservice access key¶

In [11]:

def get_key_status():
    '''Returns a string with the detailed status of the webservice access key'''
    
    request_data = {
        'method': 'key_status',
        'jsonrpc': '2.0',
        'id': 0,
        'params': {
            'key': LXTOKENIZER_WS_API_KEY,
        },
    }
    request = requests.post(LXTOKENIZER_WS_API_URL, json=request_data)
    response_data = request.json()
    if "error" in response_data:
        raise WSException(response_data["error"])
    else:
        return response_data["result"]

In [12]:

get_key_status()

Out[12]:

{'requests_remaining': 99999988,
 'chars_remaining': 999206637,
 'expiry': '2030-01-10T00:00+00:00'}

Instructions to use this web service

The web service for this application is available at https://portulanclarin.net/workbench/lx-tokenizer/api/.

Below you find an example of how to use this web service with Python 3.

This example resorts to the requests package. To install this package, run this command in the command line: pip3 install requests.

To use this web service, you need an access key you can obtain by clicking in the button below. A key is valid for 31 days. It allows to submit a total of 1 billion characters by means of requests with no more 200000 characters each. It allows to enter 100,000 requests, at a rate of no more than 200 requests per hour.

For other usage regimes, you should contact the helpdesk.

The input data and the respective output will be automatically deleted from our computer after being processed.

No copies will be retained after your use of this service.

import json
import requests  # to install this library, enter in your command line:
                 #  pip3 install requests

# This is a simple example to illustrate how you can use the LX-Tokenizer web service

# Requires: key is a string with your access key
# Requires: text is a string, UTF-8, with a maximum 200000 characters, Portuguese text, with
#           the input to be processed
# Requires: format is a string, indicating the output format, which can be either
#           'CINTIL' or 'JSON'

# Ensures: output according to specification in https://portulanclarin.net/workbench/lx-tokenizer/
# Ensures: dict with number of requests and characters input so far with the access key, and
#          its date of expiry

key = 'access_key_goes_here' # before you run this example, replace access_key_goes_here by
                             # your access key

# this string can be replaced by your input
text = '''Dentro deste parágrafo, há vários casos especiais para a separação de palavras na
ortografia do português.
Tu também deste este exemplo: dar-se-lho-ia.
E prà frente é que é pelo caminho certo, etc.'''

# To read input text from a file, uncomment this block
#inputFile = open("myInputFileName", "r", encoding="utf-8") # replace myInputFileName by
                                                            # the name of your file
#text = inputFile.read()
#inputFile.close()

format = 'CINTIL'  # other possible value is 'JSON'

# Processing:

url = "https://portulanclarin.net/workbench/lx-tokenizer/api/"
request_data = {
    'method': 'tokenize',
    'jsonrpc': '2.0',
    'id': 0,
    'params': {
        'text': text,
        'format': format,
        'key': key,
    },
}
request = requests.post(url, json=request_data)
response_data = request.json()
if "error" in response_data:
    print("Error:", response_data["error"])
else:
    print("Result:")
    print(response_data["result"])


# To write output in a file, uncomment this block
#outputFile = open("myOutputFileName","w", encoding="utf-8") # replace myOutputFileName by
                                                             # the name of your file
#output = response_data["result"]
#outputFile.write(output)
#outputFile.close()


# Getting acess key status:

request_data = {
    'method': 'key_status',
    'jsonrpc': '2.0',
    'id': 0,
    'params': {
        'key': key,
    },
}
request = requests.post(url, json=request_data)
response_data = request.json()
if "error" in response_data:
    print("Error:", response_data["error"])
else:
    print("Key status:")
    print(json.dumps(response_data["result"], indent=4))

Access key for the web service

This is your access key for this web service.

The following access key for this web service is already associated with .

This key is valid until and can be used to process requests or characters.

An email message has been sent into your address with the information above.

Email address validation

To receive by email your access key for this webservice, please copy the code displayed below into the field "Subject" of an email message (with the message body empty) and send it to request@portulanclarin.net

To proceed, please send an email to request@portulanclarin.net with the following code in the "Subject" field:

To:	`request@portulanclarin.net`
Subject:

The communication with the server cannot be established. Please try again later.

We are sorry but an unexpected error has occurred. Please try again later.

The code has expired. Please click the button below to get a new code.

For enhanced security, a new code has to be validated. Please click the button below to get a new code.

Privacy: When your access key expires, your email address is automatically deleted from our records.

LX-Tokenizer's documentation

LX-Tokenizer

LX-Tokenizer is a freely available online service for tokenizing Portuguese text. It was developed and is mantained by the NLX-Natural Language and Speech Group at the University of Lisbon, Department of Informatics.

You may also be interested to use our LX Sentence Splitter, LX-Tagger, or LX-Suite online services for delimiting sentences, part-of-speech tagging and sub-syntactic analysis of Portuguese.

Features and evaluation

The LX-Tokenizer service is composed by two processing tools:

LX Sentence Splitter:
Marks sentence boundaries with <s>…</s>, and paragraph boundaries with <p>…</p>.
Unwraps sentences split over different lines.
A f-score of 99.94% was obtained when testing on a 12,000 sentence corpus accurately hand tagged with respect to sentence and paragraph boundaries.
LX-Tokenizer:

Segments text into lexically relevant tokens, using whitespace as the separator. Note that, in these examples, the | (vertical bar) symbol is used to mark the token boundaries more clearly.

um exemplo → |um|exemplo|

Expands contractions. Note that the first element of an expanded contraction is marked with an _ (underscore) symbol:

do → |de_|o|

Marks spacing around punctuation or symbols. The \* and the */ symbols indicate a space to the left and a space to the right, respectively:

um, dois e três → |um|,*/|dois|e|três|

5.3 → |5|.|3|

1. 2 → |1|.*/|2|

8 . 6 → |8|\*.*/|6|

Detaches clitic pronouns from the verb. The detached pronoun is marked with a - (hyphen) symbol. When in mesoclisis, a -CL- mark is used to signal the original position of the detached clitic. Additionally, possible vocalic alterations of the verb form are marked with a # (hash) symbol:

dá-se-lho → |dá|-se|-lhe_|-o|

afirmar-se-ia → |afirmar-CL-ia|-se|

vê-las → |vê#|-las|

This tool also handles ambiguous strings. These are words that, depending on their particular occurrence, can be tokenized in different ways. For instance:

deste → |deste| when occurring as a Verb

deste → |de|este| when occurring as a contraction (Preposition + Demonstrative)

This tool achieves a f-score of 99.72%.

These tools work in a pipeline scheme, where each tool takes as input the output of the previous tool.

Authorship

LX-Tokenizer was developed by António Branco and João Silva at the NLX—Natural Language and Speech Group at the University of Lisbon, Department of Informatics.

Acknowledgments

The development of a state-of-the-art, complete suite of shallow processing tools for Portuguese was supported by FCT-Fundação para a Ciência e Tecnologia under the contract POSI/PLP/47058/2002 for the project TagShare and the contract POSI/PLP/61490/2004 for the project QueXting, and the European Commission under the contract FP6/STREP/27391 for the project LT4eL.

Publications

Irrespective of the most recent version of this tool you may use, when mentioning it, please cite this reference:

Silva, João, António Branco, Sérgio Castro and Ruben Reis, 2010, "Out-of-the-Box Robust Parsing of Portuguese". In Proceedings of the 9th International Conference on the Computational Processing of Portuguese (PROPOR2010), Lecture Notes in Artificial Intelligence, 6001, Berlin, Springer, pp. 75–85.

Contact us

Why LX-Tokenizer?

LX because LX is the "code" name Lisboners like to use to refer to their hometown.

License

The complete text of this license is here.