Python Programming
For this assignment the Cloudera Virtual Machine provided for this module must be used. All processing
INSTRUCTIONS TO CANDIDATES
ANSWER ALL QUESTIONS
Equipment and Facilities to be Used
For this assignment the Cloudera Virtual Machine provided for this module must be used. All processing
must be done via scripts and code, and these must be stored and included in submission. Terminal
commands must be stored in shell scripts, language specific code has to be stored in separate files (for
example, SQL code must be stored in SQL scripts and python code must be stored in Python scripts).
The solution has to be implemented using both SQL and Python.
Workload
For the successful completion of this assignment, a total of 80 hours should be budgeted.
Task
You will be given a dataset and a set of problem statements. Where possible (you will need to carefully
explain any reasons for not supplying both solutions), you are required implement the solution in both
SQL (using either Hive or Impala), and Spark (using pyspark or spark-shell).
General instructions
You will follow a typical data analysis process:
1. Load / ingest the data to be analysed
2. Prepare / clean the data
3. Analyse the data
4. Visualise results / generate report
For steps 1, 2 and 3 you will use the virtual machine (and the software installed on it) that has been
provided as part of this module. The data necessary for this assignment will be provided in a MySQL
dump format which you will need to copy onto the virtual machine and start working with it from there.
The virtual machine has a MySQL server running and you will need to load the data into the MySQL
server. From there you will be required to use Sqoop to get the data into Hadoop.
For the cleansing, preparation and analysis you will implement the solution twice (where possible). First
in SQL using either Hive or Impala and then in Spark using either pyspark or spark-shell.
For the visualisation of the results you are free to use any tool that fulfils the requirements, which can be
tools you have learned about such as Python’s matplotlib, SAS or Qlik, or any other free open source tool
you may find suitable.
5. As well as the official hashtags, each tweet may be labelled with other hashtags. Restricting the data
to ‘during-game’ tweets, list the top 10 most common non official hashtags over the whole dataset
with their frequencies.
6. Draw the graph of the progress of one of the games (the game you choose should have a complete set
of tweets for the entire duration of the game). It may be useful to summarize the tweet frequencies
in 1 minute intervals.
(it's on mysql only) creating an email marketing I believe it's easy.
Related Questions
. The fundamental operations of create, read, update, and delete (CRUD) in either Python or Java
CS 340 Milestone One Guidelines and Rubric
Overview: For this assignment, you will implement the fundamental operations of create, read, update,
. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class
Retail Transaction Programming Project
Project Requirements:
Develop a program to emulate a purchase transaction at a retail store. This
. The following program contains five errors. Identify the errors and fix them
7COM1028
Secure Systems Programming
Referral Coursework: Secure
. Accepts the following from a user: Item Name Item Quantity Item Price Allows the user to create a file to store the sales receipt contents
Create a GUI program that:Accepts the following from a user:Item NameItem QuantityItem PriceAllows the user to create a file to store the sales receip
. The final project will encompass developing a web service using a software stack and implementing an industry-standard interface. Regardless of whether you choose to pursue application development goals as a pure developer or as a software engineer
CS 340 Final Project Guidelines and Rubric
Overview The final project will encompass developing a web service using a software stack and impleme
