Feedback on Elkan Paper

Hypotheses: Elkan uses his abstract to specify clearly 4 hypotheses he
is making in his paper:

    1. That fuzzy logic collapses to 2 values.
    2. That hardly anyone appears to have used fuzzy logic in an expert system. 
    3. That fuzzy logic *has* been successfully used in controllers.
    4. That the technical limitations of fuzzy logic will become more evident, 
       even in controllers as they are scaled up. 

Most people identified #1, but lots of you either missed all or some
of the others, or misinterpreted them.

Evidence: Elkan clearly specifies on the top of p2 what kind of
evidence he intends to advance to support these hypotheses, namely
theoretical (a theorem) and experimental (a survey of the literature
and his own personal experience). 

    * There was some misunderstanding about this evidence, which was
      compounded if you had misinterpreted the hypothesis it was
      intended to support.

    * Theorem 1 was to support hypothesis 1. Note that clause 4 of
      definition 1 is not usually part of the definition of fuzzy
      logic (see for instance the definition in Wikipedia:
      http://en.wikipedia.org/wiki/Fuzzy_logic). Without it, the
      theorem does not go through. With it, fuzzy logic has worse
      problems, ie it is inconsistent (try evaluating the logically
      equivalent expressions in theorem 1 when t(B)=1/3 and
      t(A)=1/4). I did not really expect you to spot this, since it
      requires knowledge beyond what you can find in the paper,
      although someone did spot it.
    
    * Hypothesis #2 was supported by the failure to find any papers on
      this topic after a thorough search of the literature. This
      search was not a literature survey in the conventional sense, so
      evaluating the paper as a survey was unfair, although Elkan
      contributes to this misunderstanding by line 2, p2. This use of
      negative experimental evidence is unusual and subtle, and a lot
      of people misunderstood what he was trying to do here. Several
      people, however, spotted that he missed some examples of fuzzy
      logic, expert systems, e.g. CADIAG-2, which were around in 1993.

    * Hypothesis #3 was supported by finding lots of uses of fuzzy
      controllers in the literature, although he does not give much
      detail.

    * Hypothesis #4 is not supported much at all, except by informal
      argument based on his previous analysis of the limitations of
      fuzzy logic.

Flaws: Apart from those listed above, the main flaws in the paper are
gaps in Elkan's argument.

    * He claims that fuzzy logic succeeds with controllers but fails
      with expert systems because the latter require rule chaining
      during inference, whereas the former do not. However, there is
      insufficient analysis of the systems surveyed to be sure that
      this is the reason. [For instance, does CADIAG-2 chain rules
      together? If not, then it is not really a counter-example.]
      Several people spotted this.

    * He does not explain why theorem 1 ensures that long chains of
      reasoning will result in inconsistent conclusions, as he
      claims. No one mentioned this.

    * Elkan identifies 5 aspects of the architectures of fuzzy
      controllers and explains that 4 of these make possible credit
      assignment and, hence, adaption. The 5th aspect, that fuzzy
      operators are used, does not contribute to credit assignment. We
      are meant to conclude that the fuzzy operators could be replaced
      by those from some alternative uncertainty calculus, without
      detriment to the controllers. However, this assumes that credit
      assignment is the only interesting process going on in these
      controllers. Maybe there is some other critical process for
      which fuzzy operators are vital. No one spotted this. He does
      mention some non-fuzzy controllers, but no detail is given, so
      it's hard to know what to conclude. Some people did spot this.

    * I assume the heart of the argument about hypothesis #4, is that
      scaled-up controllers will also want to chain rules together,
      for which fuzzy logic will be inadequate. However, these was no
      evidence that controllers have any such chaining requirement ---
      indeed, what evidence is presented points the other way. No one
      raised this.
      
Other points: Here are a few other generic points I noticed when marking the reviews. 

    * Several people chose "indentifies and motivates a new problem"
      in the latex form, but without stating clearly what this new
      problem was. It's not obvious. You need to justify all such
      choices.

    * Several people failed to recognise that the literature
      survey provided experimental evidence, in particular the
      *absence* of certain kinds of paper is evidence. 

    * "One step ahead of the pack" is not a good summary of the
      originality of this paper. There was no pack; Elkan was pretty
      much a lone voice. This is partly a fault of ERA in not provided
      a richer array of choices, but you can always cut and paste in
      your choice.  Note that a paper can be original without being
      significant. These are often confused and conflated.

    * This paper does not propose a new technique or algorithm, nor
      improve one, nor build a system. Nor is it a literature survey
      in the conventional sense.