Build, Train and Evaluate a Deep Neural Network model Using Image Captioning

Requirement

Prepare one python notebook (recommended- use Google Colab) to build, train and

evaluate model (TensorFlow or TensorFlow.Keras library recommended) on the two

datasets given below. Read the instructions carefully.

Question: Image Captioning : Image Captioning is the process of generating textual description of an image. It uses both Natural Language Processing and Computer Vision to generate the captions. The dataset will be in the form [image → captions]. The dataset consists of input images and their corresponding output captions.

Encoder

The Convolutional Neural Network(CNN) can be thought of as an encoder. The input image is given to CNN to extract the features. The last hidden state of the CNN is connected to the Decoder.

Decoder

The Decoder is a Recurrent Neural Network(RNN) which does language modelling up to the word level. The first time step receives the encoded output from the encoder and also the <START> vector.

1. Import Libraries/Dataset (0 mark) 

2. Import the required libraries

3. Check the GPU available (recommended- use free GPU provided by Google

4. Colab).

2. Data Processing

1. Read the pickle file

(https://drive.google.com/file/d/1QG2_tAFLroc4ATSOLRiEAwYTYTsL_xVc/view?

usp=sharing) and convert the data into the correct format which could be used

for ML model. 

Pickle file contains the image id and the text associated with the image.

Eg: '319847657_2c40e14113.jpg#0\tA girl in a purple shirt hold a pillow .

Each image can have multiple captions.

319847657_2c40e14113.jpg -&gt; image name

#0 -> Caption ID

\t  -> separator between Image name and Image Caption

A girl in a purple shirt hold a pillow . -> Image Caption

Corresponding image wrt image name can be found in the image dataset folder.

Image dataset Folder : https://drive.google.com/file/d/1-

mPKMpphaKqtT26ZzbR5hCHGedkNyAf1/view?usp=sharing

1. Plot at least two samples and their captions (use matplotlib/seaborn/any other

library). 

2. Bring the train and test data in the required format. 

Model Building

4. Use Pretrained Mobilenet model trained on ImageNet dataset (available publicly on google) for image feature extraction.

5. Create 3 layered GRU layer model and other relevant layers for image caption generation.

6. Add L2 regularization to all the GRU layers. 

7. Add one layer of dropout at the appropriate position and give reasons. 

8. Choose the appropriate activation function for all the layers. 

9. Print the model summary.

Model Compilation

5. Compile the model with the appropriate loss function. 

6. Use an appropriate optimizer. Give reasons for the choice of learning rate and its value.

Model Training

8. Train the model for an appropriate number of epochs. Print the train and validation loss for each epoch. Use the appropriate batch size. 

9. Plot the loss and accuracy history graphs for both train and validation set. Print the total time taken for training.

Model Evaluation

11. Take a random image from google and generate caption for that image.

Q2)

Let us define a sequence parity function as a function that takes in a sequence of binary

inputs and returns a sequence indicating the number of 1’s in the input so far; specifically, if at time t the 1’s in the input so far is even it returns 1, and 0 if it is odd. For example, given input sequence [0, 1, 0, 1, 1, 0], the parity sequence is [0, 0, 0, 1, 0, 0].

Implement the minimal vanilla recurrent neural network to learn the parity function.

Explain your rationale using a state transition diagram and parameters of the network.

Evaluation Process -

1. Task Response and Task Completion- All the models should be logically sound and have decent accuracy (models with random guessing, frozen and incorrect accuracy, exploding gradients, etc. will lead to deduction of marks. Please do a sanity check of your model and results before submission).

2. There are a lot of subparts, so answer each completely and correctly, as no partial marks will be awarded for partially correct subparts.

3. Implementation- The model layers, parameters, hyperparameters, evaluation metrics, etc. should be properly implemented.

4. Notebooks without output will not be considered for evaluation.

5. Request to upload an HTML version of the jupyter notebook as well with all the outputs and comments.

Additional Tips -

1. Code organization- Please organize your code with correct line spacing and indentation, and add comments to make your code more readable.

2. Try to give explanations or cite references wherever required.

3. Use other combinations of hyperparameters to improve model accuracy.

If you need any programming assignment help in Machine Learning Homework, Machine Learning project or Machine Learning Assignement or need solution of above problem then we are ready to help you.

Send your request at realcode4you@gmail.com and get instant help with an affordable price.

We are always focus to delivered unique or without plagiarism code which is written by our highly educated professional which provide well structured code within your given time frame.

If you are looking other programming language help like C, C++, Java, Python, PHP, Asp.Net, NodeJs, ReactJs, etc. with the different types of databases like MySQL, MongoDB, SQL Server, Oracle, etc. then also contact us.