Feedback on Reviews of Lester et al Paper.

The key criticism I was hoping you would find is that to establish the
"Persona Effect" it was necessary to have a control condition in which
the same advice was provided, but *without* an agent. Most of you
identified this criticism, but some wrongly classified it as
minor. The absence of this agent-free condition meant that the
experiment only compared various uses of agents and could not
establish that using agents was better than not using them. What the
experiments *did* (more or less) establish was that the learning
effectiveness and experience was improved if the agents gave more
advice, but this was not the claim explicitly made in the paper.

There were other minor criticisms with the paper, which some of you identified.

1. The muted agent had high scores for whether the agent's advice was
   useful and believable, even though it gave no advice. This called
   the experimental set-up into question.

2. Some of the questionnaire questions were convoluted and one wonders
   whether such young students properly understood them.

3. The comparison to related work was a bit thin.

4. There was insufficient detail about the raw data and statistical
   analysis, especially on the improved performance on post-tests over
   pre-tests.

5. In particular, the possibility that it was time on task that
   determined the enhanced learning was not rejected as an
   explanation.

I felt that some of your criticisms were unfair or inaccurate.

1. Projects of this type are inevitably limited in terms of the number
   of and variation across subjects. Provided the conclusions are
   properly qualified, it is still legitimate to draw conclusions from
   studies limited to 100 12-year old subjects each spending an hour
   on the task. In fact, 100 subjects is quite high compared to
   many similar studies.

2. This was not an exploratory study to discover a hypothesis. I think
   the context makes clear that the experimenters designed an
   experiment to test the "Persona Effect": a hypothesis they had in mind
   from the start.

3. The research lies somewhere between Cognitive Science and
   Artificial Intelligence. It is not really Computer Science.

4. It is unfair to accuse the paper of unoriginality just because
   other systems have used agents or analysed the benefits of feedback
   in educational systems. The focus of this paper was to show the
   "Persona Effect", and in this it was original.

5. It is a feature of work of this kind that things that are
   "obviously true" turn out to be false. So it should not be
   criticised for investigating obvious things.

6. Although the techniques of a learning environment and an animated
   agent were combined into a system, this was not the main focus of
   the work and not very original in itself. The system was needed to provide
   the set-up for the experiment.

7. The readers of a paper like this can be legitimately assumed to
   have certain background knowledge about the computational
   techniques used to build the system and statistically analysis
   techniques used to analyse the results. So it is not fair to
   strongly criticise the authors for not giving an elementary
   introduction to them.

8. This paper was from a conference proceedings, in which paper
   lengths are strictly limited. So the authors are forced to omit
   detail they would have preferred to include. So one mustn't have
   too high expectations about the description of detail. On the other
   hand, it is legitimate to criticise a luck of judgement on what to
   include and omit.