TinySegmenter ============= âTinySegmenter in Pythonâ is a Python port_ of TinySegmenter_ (which is an extremely compact (23KB) Japanese tokenizer originally written in JavaScript by Mr. Taku Kudo. It works on Python 2.5 or above. .. _port: http://lilyx.net/tinysegmenter-in-python/ .. _TinySegmenter: http://chasen.org/~taku/software/TinySegmenter/ Authors ------- See the original authors in ``AUTHORS`` file. Installation ------------ See ``INSTALL`` file. Usage ----- Example code for direct usage:: > import tinysegmenter > segmenter = tinysegmenter.TinySegmenter() > print ' | '.join(segmenter.tokenize(u"ç§ã®ååã¯ä¸éã§ã")) ç§ | ã® | åå | 㯠| ä¸é | ã§ã âTinySegmenter in Pythonââs interface is compatible with NLTKâs TokenizerI, although the distribution file below does not directly depend on NLTK. If youâd like to use it as a tokenizer in NLTK, you have to modify the first few lines of the code as below (so you can't use the pypi repository version for now, if you wish to do this. Get the sources.):: import nltk import re from nltk.tokenize.api import * class TinySegmenter(TokenizerI):