| 12345678910111213141516 |
- Do assignment 0. Totally missed the first 5 mins cause it was about that
- map, reduce, combine, partition - how to explain any problem with these functions
- think of it as a design pattern
- you can make things run faster by avoiding communication between nodes
- use combiners to avoid this!
- in the algorithm, spill files (disk storage holding mapper output) are sorted!
- spills are taken and merged together (how many spills build up before they start merging can be controlled)
- during this time, combiners are run
- when reducers take the segments of the spills that match its key from each mapper, it needs to combine again
- so, sort/combine is done again here
- you can put your combine stuffs into the mapper, and that is fine. you don't need to "emit" each iteration of your loop over inputs, you can emit as many times as you want whenever you want
- a "mapper" and a "reducer" are objects which have setup, "mapping/reducing" (called once per key/value pair), and cleanup methods
- you can override the "mapper" class to make your own cleanup and map/reduce functions
- this allows preserving state/storing intermediary values
- this is better than using combiners since these operate during mapping (in mem) vs combiners that operate on spills on disk
- IMPORTANT: Combiners are NOT guaranteed to run. so don't rely on them to provide necessary functionality
|
