The aim of this assignment is to develop a content-based image retrieval pipeline. In contrast to manual feature extraction techniques, this assignment will be based on the autoencoder deep learning architecture to automatically extract features from the dataset. An autoencoder is an unsupervised deep learning model, consisting of an encoder and a decoder network. As shown in Figure 1, the encoder decreases the dimensionality of the input up to the layer with the fewest neurons, called latent space. The decoder then tries to reconstruct the input from this low-dimensional representation. This way, the latent space forms a bottleneck, which forces the autoencoder to learn a compression representation of the data.
Figure 1: The abstract architecture of an autoencoder. The encoder compresses the input to a low-dimensional representation in the latent space, the decoder tries to reconstruct the output from the latent activation.
• You have to form groups of 2-3 persons. Please get in touch with one of the supervisors if you cannot find a partner.
• You should finish the project in two weeks, and present your solution in the next exercise which will be on Thur., 24th June 2021 in the regular slot. After that you will have one more week to improve your implementation and tune your pipeline based on the feedback from the presentation. The final submission deadline is Sun., 4th July 2021, 23:55.
• The practical assignment will be introduced in the exercise slot on Thur., 17th June 2021. Each task will be described in detail and you can ask questions on things that are unclear. Please make sure to take a close look at the assignment and get started on it until then.
• You may only use standard functionalities in your code, unless exter- nal libraries are explicitly allowed in the exercise description. Such external packages are marked as pip:<package name>. If you are un- sure about if you can use some library, get in touch with the supervisors. We can only support programs written in Python. You have to provide additional documentation on where to find your code and how it works. Missing or incomplete documentation will lead to point reduction.
• For this practical assignment you can achieve 24 points plus 12 bonus points for additional work. The bonus task is optional; the achieved points are added to your overall exercise points without affecting the de- nominator.
• Please upload your results to ILIAS as a ZIP file, containing the trained model as HDF5 file, the Python code, as well as a PDF file with the report of each task (results, images, descriptions). Upload only one file per group in the form of an ILIAS group submission.
• Make sure to read the full assignment sheet before starting with your work. You might have to iterate the model building, training and evaluation steps until you reach a satisfying performance. For example, after creating an initial architecture for your autoencoder (Section 3.2), you should train the model (Section 3.3), and check the intermediate results (Section 4). From these check you will most likely detect issues, which bring you back to the model building process. Therefore, the tasks are not strictly sequential. Instead, they serve as an outline for the general procedure.
In this exercise we will create and train an autoencoder to find a low-dimensional representation of the dataset images. For this, you can either use the Tensor- Flow (pip:tensorflow) or the PyTorch (pip:pytorch) deep learning framework. The following explanations are based on TensorFlow. However, there are many learning resources and blog posts for both frameworks available on the internet.
We will use the CIFAR10 dataset. It consists of small images of 32 32 pixels from 10 different classes. Due to its simplicity, the CIFAR10 dataset will keep the training times for our network relatively low, while still being sufficient to showcase the full image retrieval pipeline.
1. Getting the dataset – You can either use the example CIFAR10 dataset of Keras or download the dataset from the University of Toronto.
2. Loading the dataset – Load the dataset into Numpy (pip:numpy) ar- rays. We recommend organizing the data as shown in Table 1.
3. Loading the dataset – Convert the pixel values from uint8 numbers in the range [0, 255] to float32 numbers in the range [0, 1].
4. Checking the results – Verify that the data shows the following prop- erties:
• The value range of the image arrays should be [0, 1].
• The dtype of the image arrays should be float32.
• The shape of the arrays should be as shown in Table 1.
Variable Description Shape
x_train The images of the training dataset. (50000, 32, 32, 3)
y_train The labels of the training dataset. (50000, 1)
x_test The images of the testing dataset. (10000, 32, 32, 3)
y_test The labels of the testing dataset. (10000, 1)
Table 1: Organizing the training and testing datasets as Numpy arrays.
What to submit? Submit the Python code for this exercise, as well as a short description of your approach and the results in the PDF document.
Search for resources on how to build an autoencoder in the deep learning frame- work of your choice (TensorFlow or PyTorch). Use the same image as input and output of the model. You can start with a simple architecture. As soon as you have a working model (i.e., the network trains) you can further improve the complexity of your architecture. As the final result, you should present a con- volutional deep neural network, i.e., a network including multiple convolutional layers for feature extraction in the encoder. The latent layer should feature an output capacity of 10 neurons, resulting in a ten-dimensional feature encoding for each image. The following hints should help you in the process of building the autoencoder.
• Start with a simple architecture, e.g., two dense layers for both encoder and decoder. When training goes well, increase the complexity of the architecture step by step.
• To later encode the features, you will need to execute only the encoder part of your network. Therefore, it will help to directly modularize the autoencoder accordingly. In Keras (as part of TensorFlow 2.x) this can be done by creating three models: the encoder model, the decoder model, and the full autoencoder as the composition of the encoder and decoder models.
• After each modification to your model, train it for a few epochs to detect possible issues (see Section 3.3).
• If the model trains well, check the reconstruction results and analyze the distributions of the latent activations (see Section 4). This will help you to refine the hyperparameter settings (layer sizes, parameter for convolutions, or activation functions) of your model.
What to submit? Provide a description on how you model evolved in the iterative model building and refinement process in the PDF file. Add the Python code of the final model. Also submit a visual representation of the final models architecture (e.g., by using TensorFlow’s model_to_dot function) in the PDF file.
Training a model can be a time-consuming task. To monitor the training pro- cess, log training metrics (loss and accuracy) using a monitoring system like TensorBoard. In Keras, this can easily done by using the TensorBoard callback. Please include screenshots of the plots visualizing the metrics for the final train- ing of your model, as shown in Figure 2. The following hints should help you in minimizing the effort for training.
• Do not fully train a model that has obvious issues or is not finished yet. After starting the training, observe loss and accuracy metrics over the first few epochs using a monitoring system like TensorBoard. Figure 2 shows, how a healthy training process looks like. Be sure to check both training and testing accuracy and loss. If the training accuracy still increases, but the testing accuracy drops, this is a sign of overfitting.
• Save data whenever possible. There is nothing more annoying than hav- ing trained a model for hours to realize that the result was never saved. Keras makes model logging easy by providing callback mechanisms that are executed at certain steps during training. Using a ModelCheckpoint you can easily save the full model configuration, including trained weights and model architecture.
CS 340 Milestone One Guidelines and Rubric Overview: For this assignment, you will implement the fundamental operations of create, read, update,
Retail Transaction Programming Project Project Requirements: Develop a program to emulate a purchase transaction at a retail store. This
7COM1028 Secure Systems Programming Referral Coursework: Secure
Create a GUI program that:Accepts the following from a user:Item NameItem QuantityItem PriceAllows the user to create a file to store the sales receip
CS 340 Final Project Guidelines and Rubric Overview The final project will encompass developing a web service using a software stack and impleme
