MapReduce是啥
通俗解释
比如计算一副牌有多少张，最简单的方法是自己一张一张数。
但MapReduce思想是把牌分给大家，大家一起数，数完加起来。
分而治之
应用在计算任务可以水平切分，而不是相互依赖
比如需要a事件完成才能执行b事件（可以理解为上下游关系）
Map和Reduce是什么呢
Map（映射）：分配给集群机器，对每个目标应用同一操作
Reduce（归纳）：整合部分结果
- file分为多个spilt，交给多个Mapper Task处理
- 处理完根据键值对进行shuffle，保证同一个key的放到一起
- 整合完交给reducer Task
- 写入hdfs
例子！！！！
文本
the weather is good today is good good weather is good today has good weather
通过split拆分
Split-0: [0, "the weather is good] Split-1: [1, "today is good"] Split-2: [2, "good weather is good"] Split-3: [3, "today has good weather]
Mapper映射
Mapper-0: ["the", 1], ["weather", 1], ["is", 1], ["good", 1] Mapper-1: ["today", 1], ["is", 1], ["good", 1] Mapper-2: ["good", 2], ["weather", 1], ["is", 1] Mapper-3: [today", 1], ["has", 1], ["good", 1], ["weater", 1]
shuffle
["good", {1, 1, 2, 1}]
["has", {1}]
["is", {1, 1, 1}]
["the", {1}]
["today", {1, 2}]
["weater", {1,1}]reducer
Reducer-0: ["good", 5] Reducer-1: ["has", 1] Reducer-2: ["is", 3] Reducer-3: ["the", 1] Reducer-4: ["today", 2] Reducer-5: ["weather", 3]
