Feedback on paper of Grimnes et al.


FLAWS

This paper does make claims -- rather too many of them, in fact. However, none
of these claims is properly evaluated. In fact, the results of the system are
only mentioned in passing, to illustrate its workings. The description of how
the system works is also rather superficial. Not even what are claimed to be
novel technologies are described in any detail. It also turns out that some of
the novel technologies, e.g. semantic profiling, are only in an early stage of
development. Nor is there any discussion of related work. The lack of any
description of the Agentcities project is a particularly glaring omission.

In mitigation, I should point out that the paper is clearly from a conference,
CIA 2003, where there would have been strict space limits. Note that it is
commonplace to publish the proceedings of conferences in the Springer LNAI and
LNCS series. However, when space is tight it is better to limit the number of
claims and then use the space to evaluate just these claims -- maybe mentioning
other potential claims in future work.


HYPOTHESES

Page 138 of the paper lists 4 contributions, which might be interpreted as or
converted into hypotheses or claims. These are:

	1. The 'power' of the system comes from Semantic Web standards, as
	opposed to the FIPA (agent) standards.

	2. Weak RDF ontologies are appropriate for this kind of application.

	3. The system embodies several component reuse technologies.

	4. The system introduces several novel technologies, e.g. RDF Query by
	Example (QbEx).

Page 149 has a further claim:

	5. A loosely-coupled user-friendly interface component has been provided
	via the BlueJADE gateway agent.

In the discussion in section 6, it is asserted that these claims have been shown
to hold. However, very little evidence is advanced in this or earlier sections
to substantiate these claims. 

What kind of evidence might we have expected to find?

	1. This system is based on a mixture of SW and FIPA standards. Ideally,
	two systems, each based on different standards, should be compared. The
	notion of 'power' needs to be made more precise, so that it would be
	clear what evaluation might reveal a difference in 'power'. At least, we
	might expect more discussion of the impact the different standards had
	within the system and how these contributed to
	the 'power'.

	2. Again, ideally we'd like to see a comparison of two systems differing
	only in their ontology languages. At least, we might see more discussion
	of the role of the ontology language.

	3. Presumably a claim could be made about the properties of these
	component reuse technologies. Prior to this we'd at least like to see
	evidence that they *could* be reused, e.g. by re-deploying them in some
	other application. More discussion of these technologies would also have
	been useful.

	4. Similarly, a claim could be made about the properties of these novel
	technologies. We'd expect to see a description of them. Their novelty
	should have been established by a discussion of related work.

	5. We'd like to see evidence that this interface *was* user friendly,
	for instance, the results of a standard questionnaire to users.
 

YOUR REVIEWS

Generally, the reviews of this paper were very good. Everyone got the general
message that the claims were not evaluated properly. No-one spotted all the
potential claims, but since these were implicit and vague, that is to be
expected. 

As I warned, people using the ERA system were tempted to just tick the boxes and not
give adequate justification for their conclusions, leading to poor reviews. That
this is not a necessary consequence of using ERA was demonstrated by some people
who *did* give adequate justification.