University Homepage School Homepage School Contacts School Search

Hc1 2010--2011: Assessed Assignment 1
evaluating conversations 

Issued: Thursday 20th January, Deadline: 12 noon, Monday 7th February

The Loebner Prize Competition in Artificial Intelligence was established in 1990 by Hugh Loebner and the Cambridge (Massachusetts) Center for BehavioralStudies. It is awarded annually to the designer of the computer system that best succeeds in passing a variant of the Turing Test. In 2008 Elbot by Fred Roberts and Artificial Solutions  won 1st Place.
See http://www.loebner.net/Prizef/2008_Contest/loebner-prize-2008.html  for more background information.

Your task is to look at the transcripts of the conversations between various judges and a human, and the judges and Elbot, and comment on them.
Note that in the transcipts, “local sent” refers to the (human) judge;  Left “remote sent” to the (hidden) human, and Right “remote sent” to Elbot.

Here are examples of the transcripts of the dialogues, as a pdf file to be downloaded.

A. Select three sections of dialogue that illustrate good examples of 'human-like conversation' between the judge (local sent) and the program (right hand side, remote sent). Justify your selection by saying for each what they did well and why you selected this example.

B. Select three sections of dialogue that illustrate examples of where there is a failure to produce good 'human-like' conversation between the judge and the program. Note that these examples could come from Elbot (right hand side) or from the Judge (local sent) or the hidden human (left hand side, remote sent). Justify your selection by saying for each where the failure occurred, whose (Elbot, judge, or hidden human) failure it was,  and why you chose this example.

C. Imagine you were a judge. Give five questions you might ask and explain why you think these would be good questions. Speculate on how you think the programs might answer these questions. Please do this BEFORE trying it out on Elbot.

D. Go to the Elbot site and try out your questions on Elbot. Record the answers and comment on their appropriateness and failings.

E. What are the relative strengths and weaknesses of Elbot? Illustrate your answer with reference to  examples from the transcripts and your interaction.

Assessments should be submitted to the Informatics Teaching Office (ITO) in Appleton Tower, Room 4.02, Level 4. You must submit a single typed copy of each assignment---please make sure you staple the pages together. Please make sure that the assignments are clearly marked with your full name, matriculation number, the course title, your tutorial group, and the assignment number. Please note the policy on late assignments.

---