Assignment 5
The goal of this assignment is to learn to write a more complicated MapReduce program for processing big data sets in parallel using MapReduce framework.
Create a MapReduce program called MovieRatings that will print all the average ratings for a movie based on user feedback
The program should work on MovieLens data set, which can be downloaded from http://www.grouplens.org/node/73. Download the 100k data (ml-100zip) set into your Downloads folder. Extract the ml-100k.zip file into ~/Downloads folder itself. For this particular assignment, the program will use the u.data and u.item files under the extracted folder. Technically speaking, you really do not have to do this step if you had already done assignment 3. Because this was data preparatory step for assignment 3.
On running the MovieRatings MapReduce program, it should process the data file, which contains a list of movie ratings by users and also process u.item file, which has movie related information. In the u.data file, each user has rated at least 20 movies. Users and items are numbered consecutively from 1. The u.data file is randomly ordered and the fields are tab separated with list of fields being user id, item id, rating and timestamp. The u.item field contains sveral pieces of information about each movie. The u.item file is pipe(|) separated. Unfortunately, the text of the README file is incorrect in terms of field separator for u.item, but it has the correct information about the fields and their order. Please look at the actual file and also the README file.
The program should take 2 The first argument should be the directory where the movielens user data set
(u.data) and movie information (u.item) files are placed (called /movie-and-ratings in hdfs), and the second argument is the name of directory where the results will be placed in HDFS (called /movie-rating-result).
The result of the program should be a list of all movies rated with the following information per line -- movie id, movie title, release date, IMDB URL, average rating, total number of unique users rated, total number of ratings for the movie
Use 2 reducers by setting the number of reducers to 2.
Write a combiner for efficiency, if possible
Define a custom counter that has a running count of total records processed in map, total number of unique movies in
Before you begin, please split the input file into 5 files called udata, u2.data, u3.data, u4.data and u5.data, each having 20000 records. Use the Unix split command. Reference: https://kb.iu.edu/d/afar
All the input files - 5 user ratings file and the movie information file should be gzip compressed before processing.
Reducer output files should be
What to turn in?
Jar file that can be run at command line
A zip of NetBeans or Eclipse project
A sample run with output as a Word I have given you a template for assignment 4 solution. Follow the guidelines, fill the document and submit.
CS 340 Milestone One Guidelines and Rubric Overview: For this assignment, you will implement the fundamental operations of create, read, update,
Retail Transaction Programming Project Project Requirements: Develop a program to emulate a purchase transaction at a retail store. This
7COM1028 Secure Systems Programming Referral Coursework: Secure
Create a GUI program that:Accepts the following from a user:Item NameItem QuantityItem PriceAllows the user to create a file to store the sales receip
CS 340 Final Project Guidelines and Rubric Overview The final project will encompass developing a web service using a software stack and impleme
