Informatics Research Methodologies
Paper Reviews
Here are the links to the materials you will need for this
assessment:
Reviewing some research papers is one of the three assessed
practical exercises for this module. Four research papers have been
selected, from which you should choose at least two. A review form
has been provided with a similar structure to a referee's form for a
conference or journal. Its purpose is to ensure that you have read the
paper and have understood it. The review process tests whether you
have identified the main scientific contribution(s) of the paper and
whether the research reported succeeded in its aims. These reviews
constitute 30% of the assessment
of the module. Between 10 and 20 hours are allocated, in total, to
reading the research papers and completing the forms.
The paper reviews are designed to help you develop the following skills:
- Analytical reading. You should read each
paper with a specific series of questions in mind: what are the
claims made for this work? what kind of claims are they? what is
the evidence for these claims? is the evidence sufficient? This
will help you analyse and critically assess the paper.
- Differential reading Reading a paper from front to
back at the same steady speed is usually not optimal. Identify
the critical parts of the paper: where the claims are made, where
the evidence is presented, the key idea, etc, and read these
parts in depth and maybe several times. Multiple passes are
sometimes useful, e.g. to get the general gist of a complex
argument on the first pass and understand it in more detail on
subsequent passes. It may only be necessary to skim other parts,
e.g. background material with which you are already familiar,
unless the presentation of this material is an issue. You may
want to place yourself mentally in the situation of other readers
with a different background to yourself, in order to assess how
well the presentation will work for them. In particular, you
should place yourself in the position of the typical reader of
the journal, conference proceedings, etc where the paper was
published, for instance, when assessing whether enough technical
detail has been given.
- Succinct Presentation You must express
your opinion of the paper briefly but clearly.
Referee forms, such as that provided by the ERA system, ask you
to assess the relevance of the paper. You should assess this with
respect to the outlet in which the paper was published. The
outlet, e.g. the journal, conference proceedings, etc, is
specified in the citation of the paper. Most journals will have a
web page in which the coverage of the journal is specified.
Similarly, most conferences will have a web page on which there
is a Call for Papers, specifying the coverage of the
conference. You can find these web pages via your favourite
search engine. Those for conferences may have disappeared, but
you might find a Call for Papers for a more recent conference in
the series. Failing this, try to work out the outlet's coverage
from its title.
ERA has a generic list of topics: Computer Science, Artificial
Intelligence and Cognitive Science. Ignore these, and specify in
the comments box any areas of the outlet's coverage that you find
relevant to the paper.
For each of the research papers there is a deadline for
submission. Your review should be submitted electronically, using
the
submit irm <paper number> <filename>
by this deadline. <paper number> should be a number from 1 to 4, depending on which of the four papers has been reviewed. <filename> should be a pdf file. Please do not put spaces in filenames; these cause problems for the automatic file printing command. Use hyphen or underline instead of space.
Each review will be marked and returned to you with
feedback and a mark within approximately two weeks of the deadline. Note
that the mark awarded is provisional and must be confirmed by the
Board of Examiners. Only the best two scores will be used to
calculate the final mark for this part of the module. You can
submit as many reviews as you like, but are strongly advised to
submit at least two.
Alan Bundy
