Feedback on Reviews of Reed and Lenat Paper.

HYPOTHESIS: The closest the authors come to stating a hypothesis is that the
expressiveness of CycL is both necessary and sufficient for the ontology
incorporation task. However, this hypothesis is not further explicitly discussed
and no explicit evidence is advanced to support it. It did not seem to me to fit
the implicit theme of the paper very well. In particular, it is not addressed in
the conclusion, where you might expect to find some summary of degree to which
the hypothesis has been established.

They might have made a hypothesis around the adequacy of the (semi-)automated
tools for assisting subject matter experts, but they had insufficient discussion
of these tools to make it the main theme of the paper.

The main focus of the paper is an overview of various projects to integrate
third party ontologies into Cyc, including an account of both positive and
negative experiences. Therefore, they might have advanced a hypothesis of the
form:

	It is possible to incorporate third party ontologies into Cyc in a
	reasonably faithful way but the following difficulties might be
	anticipated:   ...

followed by a list of the principal difficulties encountered, e.g. need for
(semi-)automated tools to assist with size of task, problems of tracking a
moving target if source ontology is under revision, lack of exact match of terms
between the source and target ontologies, etc. They might also have added some
advantages of Cyc compared to other target ontologies. It's a pity that such an
explicit summary of the main lessons was not given in the conclusion, for
instance.

EVIDENCE: The evidence was experimental arising from the practical experiences
with incorporating the various ontologies. There was no explicit return to the
hypothesis about the expressiveness of CycL, or attempt to use the experiences
to defend it.  It would also have been useful if more explicit linking of
similar experiences across different ontology incorporation tasks had been
given, so some generality in the lessons was demonstrated.

FLAWS: We have listed the three most serious flaws already: no explicit evidence
for stated hypothesis; implicit hypothesis unstated; no generality identified in
the evidence.

* Ontologies are always a compromise between expressivity of the knowledge
represented and efficiency of the inferences drawn from it. One might expect
efficiency to have been compromised by incorporating a relatively small ontology
within a huge one and perhaps moving from a decidable logic to an undecidable
one. This critical issue was not mentioned, i.e. having incorporated the third
party ontologies, could Cyc feasibly perform inferences over them?

* Moreover, no evidence is given that the incorporations were *correct*. Drawing
inferences from them (or failing to, e.g. failing to prove inconsistencies)
might have provided supporting evidence, but was omitted. 

* Only a small part of each source ontology was mapped. However, I'm not sure
this flaw was as significant as many of you claimed. The sub-ontologies mapped
might have been big enough to provide the evidence they needed. 

* The abstract and beginning of the introduction are identical. This is not good
practice, but is very common. I don't think it is a very serious flaw. 


COMMON ERRORS:

* Many of you omitted "Experimental or theoretical evidence supporting a
hypothesis" as a contribution of the paper, but most of the paper was a report
of experimental evidence. Admittedly, it was not clear what hypothesis this
evidence was supporting, but on balance I would see this as the main
contribution. 

* Several of you claimed that this work "builds a system". The main system in
question is Cyc, which was already in existence, but was merely enhanced by this
work. I would hesitate to describe the newly added micro-theories as "systems". 

* Several of you claimed that this work "Identifies and motivates a new
problem", namely this as the bottleneck to ontology growth. But this cannot be
described as "new". Others of you did find some genuinely new problems though.