LX-Tagger documentation

LX-Tagger

LX-Tagger is a freely available online service for the part-of-speech tagging of Portuguese. It was developed and is mantained by the NLX-Natural Language and Speech Group at the University of Lisbon, Department of Informatics.

You may also be interested to use our LX-Suite online service, which adds sub-syntactic analysis.

Features and evaluation

The LX-Tagger service is composed by a set of shallow processing tools:

• LX Sentence Splitter:
  Marks sentence boundaries with <s>…</s>, and paragraph boundaries with <p>…</p>.
  Unwraps sentences split over different lines.

  A f-score of 99.94% was obtained when testing on a 12,000 sentence corpus accurately hand tagged with respect to sentence and paragraph boundaries.

• LX-Tokenizer:
  

  Segments text into lexically relevant tokens, using whitespace as the separator. Note that, in these examples, the | (vertical bar) symbol is used to mark the token boundaries more clearly.

  um exemplo → |um|exemplo|

  Expands contractions. Note that the first element of an expanded contraction is marked with an _ (underscore) symbol:

  do → |de_|o|

  Marks spacing around punctuation or symbols. The \* and the */ symbols indicate a space to the left and a space to the right, respectively:

  um, dois e três → |um|,*/|dois|e|três|

  5.3 → |5|.|3|

  1. 2 → |1|.*/|2|

  8 . 6 → |8|\*.*/|6|

  Detaches clitic pronouns from the verb. The detached pronoun is marked with a - (hyphen) symbol. When in mesoclisis, a -CL- mark is used to signal the original position of the detached clitic. Additionally, possible vocalic alterations of the verb form are marked with a # (hash) symbol:

  dá-se-lho → |dá|-se|-lhe_|-o|

  afirmar-se-ia → |afirmar-CL-ia|-se|

  vê-las → |vê#|-las|

  This tool also handles ambiguous strings. These are words that, depending on their particular occurrence, can be tokenized in different ways. For instance:

  deste → |deste| when occurring as a Verb

  deste → |de|este| when occurring as a contraction (Preposition + Demonstrative)

  This tool achieves a f-score of 99.72%.

• LX-Tagger:
  

  Assigns a single morpho-syntactic tag, from the tagset, to every token. The tag is attached to the token by using a / (slash) symbol as separator:

  um exemplo → um/IA exemplo/CN

  Each individual token in multi-token expressions gets the tag of that expression prefixed by "L" and followed by the number of its position within the expression:

  de maneira a que → de/LCJ1 maneira/LCJ2 a/LCJ3 que/LCJ4

  This tagger was developed using the TnT software on a manually annotated 260k token corpus. An accuracy of 96.87% was obtained under 10-fold cross-evaluation.

These tools work in a pipeline scheme, where each tool takes as input the output of the previous tool.

Authorship

LX-Tagger was developed by António Branco and João Silva at the NLX—Natural Language and Speech Group at the University of Lisbon, Department of Informatics.

Acknowledgments

The development of this state-of-the-art tagger for Portuguese was supported by FCT-Fundação para a Ciência e Tecnologia under the contract POSI/PLP/47058/2002 for the project TagShare and the contract POSI/PLP/61490/2004 for the project QueXting, and the European Commission under the contract FP6/STREP/27391 for the project LT4eL.

This project was developed in cooperation with CLUL—Centro de Linguística da Universidade de Lisboa. The training and test corpora prepared for the development of this service evolved from a corpus provided by CLUL.

The part-of-speech tagger underlying this service was developed with Thorsten Brants' TnT software with his written permission.

References

Irrespective of the most recent version of this tool you may use, when mentioning it, please cite this reference:

• Branco, António and João Silva, 2004, "Evaluating Solutions for the Rapid Development of State-of-the-Art POS Taggers for Portuguese" . In Proceedings, LREC2004 - 4th International Conference on Language Resources and Evaluation, Lisbon, 26-28 May 2004, pp.507-510.

Contact us

Contact us using the following email address: 'nlxgroup' concatenated with 'at' concatenated with 'di.fc.ul.pt'.

Tagset

Why LX-Tagger?

LX because LX is the "code" name Lisboners like to use to refer to their hometown.