hadoop上执行mahout的bayes分类算法

沙诺 posted @ 2013年7月17日 16:09 in 云计算 with tags hadoop mahout 分类算法 , 4256 阅读

这两天做了一个hadoop上跑的分类算法——贝叶斯分类。下面介绍一下实验的运行过程。。

1,获取数据集:http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz(做分类实验通常用的数据集)

2,解压数据:我的位置:/home/XXXXXX/hadoop/mahout/mahout-distribution-0.6/examples/bin/work

3,预处理训练数据集并需要把txtfile转换成sequenceFile(mahout处理的文件必须是sequenceFile格式的)。命令:mahout org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups -p /home/XXXXXX/hadoop/mahout/mahout-distribution-0.6/examples/bin/work/20news-bydate-train -o /home/XXXXXX/hadoop/mahout/mahout-distribution-0.6/examples/bin/work/bayes-train-input -a org.apache.mahout.vectorizer.DefaultAnalyzer -c UTF-8

4,将work下的bayes-train-input放到hadoop的分布式文件系统上的 20news-input。命令:hadoop dfs -put /home/XXXXXX/hadoop/mahout/mahout-distribution-0.6/examples/bin/work/bayes-train-input 20news-input

5,用处理好的训练数据集进行训练得出分类模型即中间结果。命令:mahout trainclassifier -i 20news-input -o newsmodel -type bayes -ng 3 -source hdfs

查看分类模型的内容:命令:hadoop fs -lsr /user/hadoop/newsmodel;还可以导出到本地的txt格式查看:命令:mahout seqdumper -s /user/XXXXXX/newsmodel/trainer-tfIdf//trainer-tfIdf/part-00000 -o /home/XXXXXX/hadoop/out/part-1

插入一张图片,不然显得太单调了:

训练得出分类模型的mapreduce过程

6,用模型测试。命令:mahout testclassifier -m newsmodel -d 20news-input -type bayes -ng 3 -source hdfs -method mapreduce

用模型测试时还有点小错误,先这样写上,等测试成功了然后再纠正此处的错误。。见谅

Avatar_small
cleaning company in 说:
2019年9月09日 23:22

Any cleaning building contractors will maintain administration together with organization within the cleaning efforts also, you don’t really need to actively supervise it regularly to make sure that a steady operation. As you outsource an individual's cleaning products, it becomes an absolute background service that could be constantly being employed behind any scene you can sell organic the perfect cleaning provider possible.

Avatar_small
I sell pittsburgh ho 说:
2019年11月10日 23:16

Help your house be look extra spacious by ridding yourself of any avoidable junk (hire space if you have to). You'll see a significant difference in the best way your closet look, and in your garage, hall and lavatory. Buyers prefer to feel for instance they're obtaining sufficient space or room,


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter