Homework 7

This homework focuses on fully convolutional networks. You’ll use a new dataset and design a model with upconvolution layers. Try to keep your model compact (less than 100k parameters) during prototyping, as it makes training faster. Your final model can be a bit larger (500k+).

starter code data

Fully convolutional networks

Design a network containing convolutional layers followed by up-convolutional layers. You can chose the kernel size, stride and number of output channels as you like. You may have one final convolutional layer after your up-convolutions (which you will need, for example, if you are using skip connections).

Once you have specified the network and training code, train it.

Skip connections

You might find it helpful to add skip connections from earlier conv layer to the upconvolutional ones. You can either add or concatenate the feature maps, depending on the number of channels.

Class imbalance

One of the biggest challenges you’ll face is a huge class imbalance in the training set. The most prominent class (background) makes up almost 70% of all pixels, tiles make up another 38%, and all remaining 4 classes make up less than 2% combined. A vanilla segmentation system will just ignore those 2%, which is suboptimal. In order to combat the class imbalance you can weight the log-likelihood loss differently for different labels.

Getting Started

We provide you with starter code that loads the image dataset and the corresponding labels from a training and validation set.

The code will measure classification accuracy as you train the model. We also provide an optional tensorboard interface.

  1. Define your model in models.py and modify the training code in train.py.
  2. Train your model. 
     python3 -m homework.train
  3. Optionally, you can use tensorboard to visualize your training loss and accuracy. 
     python3 -m homework.train -l myRun

    and in another terminal tensorboard --logdir myRun, where myRun is the log directory. Pro-tip: You can run tensorboard on the parent directory of many logs to visualize them all.

  4. Test your model by visualizing the predicted masks. 
     python3 -m homework.test -i 2
  5. To evaluate your code against grader, execute: 
     python3 -m grader homework
  6. Create the submission file 
     python3 -m homework.bundle

Input example

input

Output example

output

Grading

We will have a linear grading scheme depending on your validation and testing IoU score. The thresholds for validation is as following:

20% IoU (intersection over union): 0 points
40% IoU (intersection over union): 100 points
Linear scaling score from 20%-40%.

Extra credit

45+% IoU: + 10 points
50+% IoU: + 10 points
55+% IoU: + 10 points

Intersection over union is defined as \(IoU(A,B)=\frac{|A\cap B|}{|A\cup B|} \), which is \(A\) and \(B\) are the predicted and ground-truth pixel classes.

Visualizing your network

We provide you with a visualization tool to see how your model predicts the masks in test.py.

viz

Relevant operations