machine learning - Python NLTK pos_tag not returning the correct part-of-speech tag -

having this:

text = word_tokenize("the quick brown fox jumps on lazy dog") 

and running:

nltk.pos_tag(text) 

i get:

[('the', 'dt'), ('quick', 'nn'), ('brown', 'nn'), ('fox', 'nn'), ('jumps', 'nns'), ('over', 'in'), ('the', 'dt'), ('lazy', 'nn'), ('dog', 'nn')] 

this incorrect. tags quick brown lazy in sentence should be:

('quick', 'jj'), ('brown', 'jj') , ('lazy', 'jj') 

testing through online tool gives same result; quick, brown , fox should adjectives not nouns.

in short:

nltk not perfect. in fact, no model perfect.

note:

as of nltk version 3.1, default pos_tag function no longer old maxent english pickle.

it perceptron tagger @honnibal's implementation, see nltk.tag.pos_tag

>>> import inspect >>> print inspect.getsource(pos_tag) def pos_tag(tokens, tagset=none):     tagger = perceptrontagger()     return _pos_tag(tokens, tagset, tagger)  

still it's better not perfect:

>>> nltk import pos_tag >>> pos_tag("the quick brown fox jumps on lazy dog".split()) [('the', 'dt'), ('quick', 'jj'), ('brown', 'nn'), ('fox', 'nn'), ('jumps', 'vbz'), ('over', 'in'), ('the', 'dt'), ('lazy', 'jj'), ('dog', 'nn')] 

at point, if wants tl;dr solutions, see https://github.com/alvations/nltk_cli

---

in long:

try using other tagger (see https://github.com/nltk/nltk/tree/develop/nltk/tag) , e.g.:

• hunpos
• stanford pos
• senna

using default maxent pos tagger nltk, i.e. nltk.pos_tag:

>>> nltk import word_tokenize, pos_tag >>> text = "the quick brown fox jumps on lazy dog" >>> pos_tag(word_tokenize(text)) [('the', 'dt'), ('quick', 'nn'), ('brown', 'nn'), ('fox', 'nn'), ('jumps', 'nns'), ('over', 'in'), ('the', 'dt'), ('lazy', 'nn'), ('dog', 'nn')] 

using stanford pos tagger:

$ cd ~ $ wget http://nlp.stanford.edu/software/stanford-postagger-2015-04-20.zip $ unzip stanford-postagger-2015-04-20.zip $ mv stanford-postagger-2015-04-20 stanford-postagger $ python >>> os.path import expanduser >>> home = expanduser("~") >>> nltk.tag.stanford import postagger >>> _path_to_model = home + '/stanford-postagger/models/english-bidirectional-distsim.tagger' >>> _path_to_jar = home + '/stanford-postagger/stanford-postagger.jar' >>> st = postagger(path_to_model=_path_to_model, path_to_jar=_path_to_jar) >>> text = "the quick brown fox jumps on lazy dog" >>> st.tag(text.split()) [(u'the', u'dt'), (u'quick', u'jj'), (u'brown', u'jj'), (u'fox', u'nn'), (u'jumps', u'vbz'), (u'over', u'in'), (u'the', u'dt'), (u'lazy', u'jj'), (u'dog', u'nn')] 

using hunpos (note: default encoding iso-8859-1 not utf8):

$ cd ~ $ wget https://hunpos.googlecode.com/files/hunpos-1.0-linux.tgz $ tar zxvf hunpos-1.0-linux.tgz $ wget https://hunpos.googlecode.com/files/en_wsj.model.gz $ gzip -d en_wsj.model.gz  $ mv en_wsj.model hunpos-1.0-linux/ $ python >>> os.path import expanduser >>> home = expanduser("~") >>> nltk.tag.hunpos import hunpostagger >>> _path_to_bin = home + '/hunpos-1.0-linux/hunpos-tag' >>> _path_to_model = home + '/hunpos-1.0-linux/en_wsj.model' >>> ht = hunpostagger(path_to_model=_path_to_model, path_to_bin=_path_to_bin) >>> text = "the quick brown fox jumps on lazy dog" >>> ht.tag(text.split()) [('the', 'dt'), ('quick', 'jj'), ('brown', 'jj'), ('fox', 'nn'), ('jumps', 'nns'), ('over', 'in'), ('the', 'dt'), ('lazy', 'jj'), ('dog', 'nn')] 

using senna (make sure you've latest version of nltk, there changes made api):

$ cd ~ $ wget http://ronan.collobert.com/senna/senna-v3.0.tgz $ tar zxvf senna-v3.0.tgz $ python >>> os.path import expanduser >>> home = expanduser("~") >>> nltk.tag.senna import sennatagger >>> st = sennatagger(home+'/senna') >>> text = "the quick brown fox jumps on lazy dog" >>> st.tag(text.split()) [('the', u'dt'), ('quick', u'jj'), ('brown', u'jj'), ('fox', u'nn'), ('jumps', u'vbz'), ('over', u'in'), ('the', u'dt'), ('lazy', u'jj'), ('dog', u'nn')] 

---

or try building better pos tagger:

• ngram tagger: http://streamhacker.com/2008/11/03/part-of-speech-tagging-with-nltk-part-1/
• affix/regex tagger: http://streamhacker.com/2008/11/10/part-of-speech-tagging-with-nltk-part-2/
• build own brill (read code it's pretty fun tagger, http://www.nltk.org/_modules/nltk/tag/brill.html), see http://streamhacker.com/2008/12/03/part-of-speech-tagging-with-nltk-part-3/
• perceptron tagger: https://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/
• lda tagger: http://scm.io/blog/hack/2015/02/lda-intentions/

---

complains pos_tag accuracy on stackoverflow include:

• pos tagging - nltk thinks noun adjective
• python nltk pos tagger not behaving expected
• how obtain better results using nltk pos tag
• pos_tag in nltk not tag sentences correctly

issues nltk hunpos include:

• how tag textfiles hunpos in nltk?
• does know how configure hunpos wrapper class on nltk?

issues nltk , stanford pos tagger include:

• trouble importing stanford pos tagger nltk
• java command fails in nltk stanford pos tagger
• error using stanford pos tagger in nltk python
• how improve speed stanford nlp tagger , nltk
• nltk stanford pos tagger error : java command failed
• instantiating , using stanfordtagger within nltk
• running stanford pos tagger in nltk leads "not valid win32 application" on windows