通过mapreduce程序统计旅游订单(wordcount升级版)
本文将结合一个实际的MapReduce程序案例,探讨如何通过分析旅游产品的预订数据来揭示消费者的偏好。
程序概览
首先,让我们来看一下这个MapReduce程序的核心代码。这个程序的目的是处理一个包含旅游产品预订信息的文本文件,并统计每个产品特性的出现次数。Map阶段的代码如下:
java
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
if (key.get() > 0) { // 跳过表头
String line = value.toString();
String[] fields = line.split("\t");
if (fields.length > 1 && !fields[1].isEmpty()) {
String[] arrstr = Arrays.copyOfRange(fields, 8, fields.length - 1);
for(String str:arrstr){
if(StringUtils.isNotBlank(str)){
word.set(str);
context.write(word, new IntWritable(1));
}
}
}
}
}
}
Reduce阶段的代码如下:
java
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
全部代码
java
package org.example;
import java.io.IOException;
import java.util.Arrays;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
public class KeyWord{
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
// private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
if (key.get() > 0) { // 跳过表头
String line = value.toString();
String[] fields = line.split("\t");
if (fields.length > 1 && !fields[1].isEmpty()) {
String[] arrstr = Arrays.copyOfRange(fields, 8, fields.length - 1);
for(String str:arrstr){
if(StringUtils.isNotBlank(str)){
word.set(str);
context.write(word, new IntWritable(1));
}
}
// int a;
// if(StringUtils.isNotBlank(fields[4])){
// a = Integer.parseInt(fields[4]);
// }else{
// a=0;
// }
}
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public void keyWorsds() throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Word Count on Second Field");
job.setJarByClass(KeyWord.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(job, new Path("/Users/shareit/ds_task_am/wordcount/src/main/resources/mapreduce数据(1).txt"));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path("/Users/shareit/ds_task_am/wordcount/producttotalhuman"));
job.waitForCompletion(true);
}
}
结论
通过MapReduce程序对旅游产品预订数据的分析,我们能够洞察到消费者的偏好和行为模式。这些信息对于旅游企业来说是宝贵的,可以帮助他们更好地定位市场,设计符合消费者需求的产品,并最终提高客户满意度和市场份额。随着数据分析技术的不断进步,旅游行业将能够更加精准地满足消费者的需求,推动行业的持续发展。
如有遇到问题可以找小编沟通交流哦。另外小编帮忙辅导大课作业,学生毕设,代写各种mapreduce程序等。不限于python,java,大数据,模型训练等。