Coursework 4

Notes on report structure

Structure

There is no mandatory structure, but you should make sure that whatever you do is logical. Start with an Introduction and end with a Conclusion, with a list of References at the end. There is no need for an abstract, a table of contents, or a list of figures. Three possible ways to structure your report would be: (1) experiment-by-experiment in which you present the methods, experimental setup, results, and discussion for each experiment in its own section; (2) overall methods section, followed by experiments, followed by results and discussion, each section covering all experients; or (3) overall methods section followed by sections per experiments for the experimental process, results, and discussion.

Introduction

The introduction should place your work in context, giving the overall motivation for the work, and clearly outlining the research questions you are investigating. This should include a concise description of the task and the data (be precise, eg state the size of the training/dev/test sets), linking the research questions to the task, and perhaps motivating them with reference to your previous coursework and to other papers in the literature. You get extra credit for this section if you are proposing interesting/innovated research hypotheses or if your introduction provides great insight into the task or data, with references to the literature.

Methods

When you present your experimental methods you need to clearly motivate what you have done, perhaps with references to the literature. You need to be precise in specifying all the hyperparameters used (someone reading the paper needs to be able to replicate your experiments), and you should explain why you have chosen particular hyperparameter settings. Extra credit for a wide range of experiments and hypotheses tested - this does not mean you have to focus on very deep or computationally expensive architectures: it means that you get credit for carefully and thoroughly carrying out and reporting experiments to investigate your research questions. This section should also discuss how you implemented your work in TensorFlow, giving details for design choices and showing TF implementation choices, mostly just by referencing TF functions (rather than including code).

You should also make sure you concisely describe the baseline system(s) you developed in CW3, and how they are used in this work (they might just be used as a baseline accuracy to compare against, or you may be more directly further developing those systems).

Results and Discussion

Your experimental results should be presented clearly and concisely, followed by interpretation and discussion of results and conclusions you can draw from the experiments. You need to present your results in a way that makes it easy for a reader to understand what they main. You should facilitate comparisons either using graphs with multiple curves or (if appropriate, e.g. for accuracies) a results table. You need to avoid many plots spread over several pages, poorly labelled graphs, graphs which should be comparable but which use different axis scales. A good presentation will enable the reader to compare trends in the same graph - each graph should summarise the results relating to a particular hypothesis or (sub)question.

Minimally you need to show the learning curves (error vs epoch) for training and dev, and the test set accuracies. You should use your CW3 baseline system as a reference you compare against. Extra credit would be given for results which give other analyses beyond just the learning curves and accuracies/errors.

Your discussion should interpret the results, both in terms of summarising the outcomes of a particular test/hypothesis, and attempting to relate to the underlying models. A good report would have a detailed analysis, and would show good understanding of why particular results are observed, perhaps with reference to the literature.

Conclusions

You should draw conclusions from each experiment, related to the research question which motivated it. Your should state the conclusions clearly and concisely. It is also good if the conclusion from one experiment influence what you do in later experiments - your aim is to learn from your experiments. Extra credit if you relate your findings to what has been reported in the literature.

If you are using an experiment-by-experiment structure, it is OK to have conclusions for an experiment in "its" section, but you should also have an overall conclusions section at the end. If there is an overall conclusion across the research questions, then discuss it, and also you might reprise what you conclude in terms of your "best" system (based on your research questions and constraints).

A good conclusions section would also include a further work discussion, building on work done so far, and referencing the literature where appropriate.

FAQ

Q: How should we choose what research questions to address in coursework 4?
A: There is no rule here, but in general it is better to be "deep and narrow" rather than "broad and shallow". Focusing on specific research questions should help you motivate specific experiments from which you can hopefully draw conclusions. It will also help to make your report more coherent.
Q: Do we have to do experiments on both the large label and small label datasets (e.g. both CIFAR-10 and CIFAR-100)?
A: It is completely OK to focus on just one of the datasets; this is dependent on your research question. For example, if you are looking at approaches to address overfitting and limited training data per class, then it may make sense to largely focus on CIFAR-100. However, if you focus on CIFAR-100 for example, please do take one or two interesting models and run them on CIFAR-10. (and similar for MSD).
Q: Is the focus on getting the best possible accuracy?
A: No. Of course, accuracy is an important measure, but the main aim is to run a series of experiments using comparable architectures and hyper-parameter settings. Given time and compute constraints, you will probably need to choose a setup which does not give the best possible performance. At the end, if you have time, you may wish to run a more elaborate model to see what accuracy it achieves. You should also think about what analyses you will perform in addition to dev/test set accuracy, which can give further insight to your investigation.
Q: Now that Tensorflow 1 is officially out, can we use it for cw4? Are there any plans for setting up a second miniconda environment on scutter?
A: You can use tensorflow 1.0 but we don't support it this year.
Q: is it OK to use tf.layers module?
A: Yes. Note that tf.layers in v1.0 is under tf.contrib.layers in v0.12.
Q: What level of verbosity do you want results in - if, say, I've run a grid search over a couple of variables and want to present results, tabulating these is going to give you a couple of pages of error and accuracy tables, graphs. Do you want these presented in-line, in an appendix or not at all? I recall you giving general feedback that graph spam wasn't wanted, particularly learning curves?
A: Your aim is to communicate the results clearly, concisely, and correctly. For example, think about how you can display the results as graphs - you can have multiple curves on a one graph, but do make sure that the scale is such that the relevant differences are visible.
Q: Is there a page limit?
A: There is no page limit, but as before try to keep it concise and to the point.

This page maintained by Steve Renals.
Last updated: 2017/03/07 10:28:40UTC

Home : Teaching : Courses : Mlp