首页 > 编程语言 > 详细

[Javascript Natural] Break up language strings into parts using Natural

时间:2016-11-08 07:38:30      阅读:190      评论:0      收藏:0      [点我收藏+]

A part of Natural Language Processing (NLP) is processing text by “tokenizing” language strings. This means we can break up a string of text into parts by word, sentence, etc. In this lesson, we will use the natural library to tokenize a string. First, we will break the string into words using WordTokenizerWordPunctTokenizer, and TreebankWordTokenizer. Then we will break the string into sentences using RegexpTokenizer.

 

var natural = require(natural),
  tokenizer = new natural.WordTokenizer();
console.log(tokenizer.tokenize("your dog has fleas."));
// [ ‘your‘, ‘dog‘, ‘has‘, ‘fleas‘ ] 

 

tokenizer = new natural.TreebankWordTokenizer();
console.log(tokenizer.tokenize("my dog hasn‘t any fleas."));
// [ ‘my‘, ‘dog‘, ‘has‘, ‘n\‘t‘, ‘any‘, ‘fleas‘, ‘.‘ ] 
 
tokenizer = new natural.RegexpTokenizer({pattern: /\-/});
console.log(tokenizer.tokenize("flea-dog"));
// [ ‘flea‘, ‘dog‘ ] 
 
tokenizer = new natural.WordPunctTokenizer();
console.log(tokenizer.tokenize("my dog hasn‘t any fleas."));
// [ ‘my‘,  ‘dog‘,  ‘hasn‘,  ‘\‘‘,  ‘t‘,  ‘any‘,  ‘fleas‘,  ‘.‘ ] 

 

[Javascript Natural] Break up language strings into parts using Natural

原文:http://www.cnblogs.com/Answer1215/p/6041263.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!