Coding assignments
All coding assignments are team assignments. Five student work on one assignment (and codebase). We will grade you as a group (with exceptions for individuals that do not contribute).
In the coding assignment, you will
- implement the core idea of five papers on a standard architecture and task/dataset
- evaluate your implementation on 3 different datasets
- compare the relative performance, stability (to hyper-parameters) across all datasets
- draw conclusions for which methods work best (or if there is no clear winner)
In class, you will present your findings by presenting your repo to the rest of class.
Timeline
- Sun midnight: Complete code due
What should the code contain
What tasks and datasets are allowed
Most main-stream vision and language tasks (tasks people write respectable research papers on). You may pick two main-stream datasets and a toy dataset if you want to highlight something particularly interesting on a toy dataset.
What datasets are NOT allowed (this list may grow as class continues):
- MNIST (or any variants like FashionMNIST)
How to submit your code
Create a repo in the github team. Your repo should contain.
readme.md: A writeup with your conclusions- All the code to reproduce your experiments
- Additional notebooks with ablations and visualizations
Grading criterion
We will grade you (linearly) on four criteria
- Correctness (are all methods implemented correctly)
- Basic evaluation (did you evaluate all methods on 3 datasets)
- Ablations and additional experiments (how many extra experiments did you perform to compare different methods)
- Readme and writeup of conclusion
Each criterion has a weight of 5pt with a bias of -10pt (for a max total of 10pt). We will use your presentation as additional input for grading.
Policy on using open-source codes
It is allowed to use open-source implementations but you need to cite the source properly. E.g., if you start with some existing codes, please first commit the original file and then modify in the following commits.