Feedback on paper of Oren, Norman and Preece.


KIND OF CONTRIBUTION

1. The paper describes a new technique: a possibilistic-based, subjective-logic
   framework for argumentation under uncertainty.

2. It establishes a property of this technique, namely the computational
   complexity of its reasoning mechanism.

4. It suggests an application of this technique, namely to contract monitoring
   and enforcement, and provides experimental evidence that this technique can
   be so applied.


HYPOTHESES AND THEIR EVALUATION

Various claims are made in the paper, but only two seem to be backed up by any
evaluation.

1. "Reasoning about the state of a literal in this framework can be done in
    polynomial time".

	Two complexity results are stated without proof:

	   + "The unoptimised version presented here runs in exponential time
	      (O(2n)) for n arguments in the agent’s knowledge base)"

	   + "It is trivial to show that the algorithm presented in Figure 2
	      runs in O(n) time where n is the number of graph edges"

Since neither of these results shows "polynomial time", I admit to some
confusion as to the evaluation of this claim. 

2. "[The framework] has applications as a powerful mechanism for contract
    monitoring in complex domain".

   An example application to this domain is shown in section 4.

Examples of claims that do not seem to be evaluated and, therefore, one assumes
are not meant to be serious contributions of the paper are:

* "The dialogues that emerge bear some similarity to the dialogues that occur
   when humans argue about contracts ..."

* "... our approach is highly suited to complex, partially observable domains
   with fallible sensors where determining environment state cannot be done for
   free."

Unless 'contract monitoring' is meant to be such a domain.


FLAWS

* Neither claim 1 nor 2 are adequately evaluated. For claim 1, two theorems are
  stated without proof and, in any case, neither theorem corresponds to the
  claim made. For claim 2, one worked example is weak evidence for the claim,
  especially as there is no analysis of what properties of the technique make it
  especially suitable for the application.

* A significant claim is missing, namely that the formulae defining the
  subjective logic, e.g., in figure 1 and section 3.1, are correct. Ditto the
  'dialogue game' in Figure 2 and section 3.2.  There is some discussion of a
  semantics for this logic in section 5, but the standard semantics, due to
  Dung, is claimed to be difficult to apply and finding a suitable semantics is
  left for further work. In the absence of a formal semantics, one might hope
  for at least an informal justification for the various formulae and opinion
  propagating mechanisms, but mostly these are just presented as a fait
  accompli.

* It is claimed that the approach is "based on Dempster-Schafer theory", but
  this claim is never established. If it had been, perhaps it would have
  provided a basis for establish the correctness of the method.

* It is claimed in section 1 that "We conclude the paper by examining related
  research". I struggled to find this examination. There are just throwaway
  remarks scattered throughout the paper, but no in-depth discussion. Such a
  discussion is required, for instance, to defend the background assumptions of
  the paper that "[no argumentation frameworks] are designed to represent
  possibilistic knowledge, making them unsuitable for many real world
  domains". 


YOUR REVIEWS

     Generally, good reviews. Here are some of my most common criticisms. 

* This work does not model a natural system, as some of you claimed. Perhaps if
  a similarity with human dialogues had been demonstated then one could claim
  this, but not as the paper stands. Any claim about a correspondence to the
  human mind/brain ought to be evaluated using experimental psychology or
  neuroscience. 

* It's unclear whether a system was built, as some of you assumed. Note that the
  'framework' is a purely logical construct and there is no indication that it
  was implemented as a computer system. I read the worked example in section 4
  as a manual simulation of the way such a system might work.

* Few people identified as a claim that of polynominal complexity. Even fewer
  criticised the lack of a proof of this claim.

* Few people identified the poverty of the experimental evidence for claim 2.

* No-one highlighted the lack of a formal semantics for the logic or the lack of
  a correctness proof for the formulae and reasoning mechanism.

* Some people were overly critical. The exponential time was only due to the
  heuristic, which could be relaxed so this problem was solvable. Also, the
  inability to deal with cyclic arguments (looping) did not seem too serious a
  problem, given that it was not the main focus of the work and solutions might
  perhaps be adapted from other argumentation frameworks. All research has
  limitations and researchers should be encouraged to identify them, so there is
  a danger in being over-critical when they do so.