edu.mit.csail.brill
Class PennTokenizer

java.lang.Object
  |
  +--edu.mit.csail.brill.PennTokenizer

public final class PennTokenizer

extends Object

Tokenizer that conforms to the Penn Treebank conventions for tokenization.

Author:

Jimmy Lin

Constructor Summary

PennTokenizer()

Method Summary

static String

tokenize(String str) Tokenizes according to the Penn Treebank conventions.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PennTokenizer

public PennTokenizer()

Method Detail

tokenize

public static String tokenize(String str)

Tokenizes according to the Penn Treebank conventions.