Filling in the "Why I chose this paper" field

For each member of research active staff submitted to RAE 2008 we will submit a copy of form RA2. In particular, this form will contain details of the staff member's four (or fewer in some cases) research outputs. For each research output there will be associated a free-form field entitled "Other relevant details" in RAE-speak. Since this name does not make it very clear what this field is for, we have called it "Why I chose this paper" and will also refer to it as the "text box".

This field is very important because it will constitute the starting point for the Panel's assessment of each output. So submitted staff must take especial care in filling it in. To propose wording for this field, fill in the "Why I chose this paper" of the Informatics@Edinburgh Publications Portal. To update the field, use the update form. Note that this will only give you access to the papers on which you are first author. If you are not the first author, ask him/her to make the updates for you. If all else fails, send the updated wording to Tamise Totterdell.

The purpose of this web page is to advise you how to fill in this field.

How the RAE View this Field

The role of this field is described in the Computer Science and Informatics Sub-Panel Criteria and Working Methods document as follows:

• "22. To help in the assessment of research quality the sub-panel expects departments to provide additional information (in no more than 100 words, or up to 300 words in exceptional cases) in the 'Other relevant details' field for each output listed in RA2. This should address the three assessment criteria of originality, rigour and significance and should include:
  • a statement summarising the research contribution of the output
  • evidence of academic or other impact.
• 23. The sub-panel does not expect the 'Other relevant details' field to be used for any other purpose except in exceptional circumstances.

Some Tips on Filling in the Field

Please fill in the form in the 3rd person.

You should start your entry with a brief description of the content of the publication. You should also comment on the quality of the outlet. But this is not enough. The most important word in the text above is impact. The rating of your output will be determined principally by the impact of the research that it describes, i.e. whether it made a difference. What constitutes evidence of significant impact will vary widely depending on the area of research you work in and the nature of the research results achieved. There follows a non-exhaustive list of the kinds of thing you might mention, depending on your particular circumstances. I welcome suggestions of things that I have missed and might add to this list for the benefit of others.

• You should summarise the achievements of your research in a form accessible to a computing expert who is not a specialist in your area. Try to make it clear why this is significant, original and rigorous work within the methodology and state of the art of your research area. Outline the impact this work has had or is expected to have.
• Acceptance in a high quality journal or conference would indicate that the referees considered this work to be of high quality. If you know the impact factor of the outlet, then you could include this. Similarly, you could include the acceptance ratio of a conference. A best paper award or shortlisting would indicate exceptional quality and should definitely be mentioned.
• Frequent citation of a publication would indicate that it has been widely read and built on. If you know the citation index or the number of citations of the publication, then you could include this. If you can point to a growing body of work that builds on your work then this is well worth mentioning.
• The application of a piece of research would indicate its practical impact. For instance, a product might been developed using the results of the research or the research might have been applied to achieve results in another field. Patents can be mentioned, but do not in themselves prove successful practical application. Successful spin-out and start-up companies arising from the research should certainly be mentioned, as should take-up by an existing company.
• Widespread take-up of software arising from the research indicates significant impact. If you have download figures then mention these. The School of Informatics Software Download Database (ISDD) has been designed to be linked to RAE 2008 relevant software and keeps a record of downloads. See the FAQ for further information.
• Interest in your work can be indicated by invitations to give seminars at other labs and/or to give invited talks at conferences.
• Interest can also be indicated if research collaborations have grown out of it, especially international or industrial ones. For instance, you might have been included in a project proposal consortium on the basis of your expertise and advances in some sub-field.
• If your work has helped define a new area of research, which is now routinely included in calls for papers, for instance, or has led to a popular new workshop or conference series, then this would indicate significant impact, especially if the interest goes well beyond your own group, its ex-members and previous collaborators.
• Media interest should be treated with caution, since it can be driven by factors that are irrelevant to the quality of your work. However, in some cicumstances it could be very relevant. For instance, if your work has changed the way the public think about an issue then this would represent significant impact.
• Restrict yourself to evidence-based claims, i.e. don't claim that the work is "world-class", "seminal", "ground-breaking", etc. Rather provide evidence of its impact, significance, originality, rigour, etcs that would lead the panel to conclude these things for themselves. In particular, do not suggest a star ranking for your output. This is for the panel to decide and you will annoy them if you try to preempt this decision. [Confusingly, there is also a separate field where you are asked to estimate the star ranking, but this is for internal purposes and will not form part of our submission.]

Remember that this field will be read by panel members who are not experts in your research area. Try to make it as non-technical as possible, e.g. avoid acronyms and other jargon. Don't assume prior knowledge beyond that of a computing undergraduate.

Examples of Filling in the Field

• "The book explains the new generics features of Java to practicing developers. O'Reilly is a leading publisher of technical books. I chose this publication to underline the impact of my earlier research on generics for Java, which is now in widespread industrial use. "A brilliant exposition of generics. By far the best book on the topic, it provides a crystal-clear tutorial that starts with the basics and ends leaving the reader with a deep understanding of both the use and design of generics." -- Gilad Bracha, Java Generics Lead, Sun Microsystems"
• "This paper is the first ever to show that compiler analysis can effectively control issue queue utilisation. Furthermore it shows that a critical-path analysis scheme with just-in-time adaptation of queue size can reduce power consumption to a greater extent than any existing prior scheme. Furthermore it can achieve this with less loss of performance with the additional benefit of no hardware overhead. This was published at one of the best architecture conferences who are traditionally highly sceptical of compiler schemes. The acceptance rate at HPCA was just 15 %. A paper launching an entirely new approach to power management in superscalar architecture."
• "This is the first paper ever to show a completely automatic parallelization approach for pointer based c programs on local address space multi-core embedded DSP systems. Several innovations are described including a novel pointer conversion technique to overcome alias problems and a novel use of our rank-modification transformation framework to increase the dimensionality of array data. This work showed outstanding performance results - average 3.78 on a 4 processor machine. A breakthrough paper. Highest ranked at PACT - 1st out of 162 submissions - 18% acceptance rate.
• "This is a new text book of 700 pages in full colour with 430 exercises all with worked solutions. There will be a companion text with Matlab software and a supporting web site. The goal is for the book to become widely adopted as the leading text in the machine learning field. Its predecessor ("Neural Networks for Pattern Recognition", Oxford University Press, 1995) has sold 22,000 copies, and it is hoped that the new book will have an even larger impact. "
• "This paper received the 2005 GECCO Best Paper award for A-Life, Evolutionary Robotics, and Adaptive Behavior. It shows how the spontaneous neural system activity found in developing animals can inspire a new type of architecture for learning systems, based on generating training patterns for a system that later learns from real data. The new architecture requires far fewer training examples to reach a given level of performance on a handwritten digit recognition task, and suggests that this approach is an effective way of constructing a complex system."
• "Message sequence charts are a commonly used notation in software design. They are part of the UML notation, and many software design tools support them. In this paper, which is the journal version of a paper at ICSE'00, we developed a formal framework and efficient algorithms for detecting the realizability of a family of MSCs as communicating state machines, and for inferring new MSCs if the behaviors they specify are not realizable, and synthesizing the state machines if they are. Our methods from this paper are now patented by us under a U.S. Patent received in 2004, and this work is now heavily cited in the MSC literature in software engineering. "
• "This book is my biggest single output in 10 years. It presents SDRT: a theory of discourse interpretation. The book has been very well received, and prompted a Linguistics Professor at Paris VII to organise a conference based on SDRT last year. According to google.scholar, it has now received over 100 citations, in spite of being published only 2 years ago."
• "Technical paper in one of highest quality publications in AI and one particularly focussed on communication to an audience beyond AI people. Describes work on CoAX Demonstration in 2001. CoAX was a multinational collaborative project over period 2000-2002. 30 participating organisations from 4 countries including Stanford, MIT, CMU, Boeing, Lockheed Martin, BBN, QinetiQ. Production of related software, presentations, demonstrations, DVD, web-site, conferences, invited presentations and publications involving both technology and application orientated papers. Austin Tate was CoAX Project Director."

Citation Indices and Impact Factors

The citation index of a paper measures how many times it has been cited by someone other than the author. The impact factor of a journal is based on the average citation indices for the papers published in that journal. Impact factors of top outlets in Informatics hover around 1. The top score, at around 10, is from ACM Surveys. Top conferences in an area can score higher than the top journals. The acknowledged main source of these figures is the ISI Web of Science. However, it unfortunately does not cover all Informatics outlets, especially conferences. You can get free access to this source via the Library or MyEd, provided you are registered with EASE or Athens. Citeseer used to be good for Computing outlets, but has been moribund for several years, so is no longer of much use. The new kid on the block is Google Scholar, but I have been unable to track down these kinds of measures on it. Let me know if you have and where to find them. The Publish or Perish software provides a nice interface to the Google citation information.