Feedback on paper of Gold et al.


METHODOLOGY

The main aim of this paper is to identify a new problem with service-oriented
approaches to software development: the necessity of but difficulty in
understanding the software. To do this it takes a single hypothetical case study
and steps through this to identify different aspects of the software
understanding problem. Although some possible solutions to these problem are
proposed, the discussion is based on some questionable assumptions, which render
the discussion unconvincing.

There is no program, no theory and only 'thought' experiments. The methodology
is akin to the conceptual analysis used in philosophy, sociology or
linguistics. To assess it we have to see how thorough and convincing the
analysis is. Are there important aspects of the case study that have not been
adequately discussed? Is the case study really representative of the full range
of commonly occurring situations. Are the arguments compelling?


HYPOTHESES

With this methodology, it's hard to see what the hypotheses are. However, in the
first paragraph there is a statement that new problems have been identified:

1. "While [service-oriented approaches] address some aspects of the problem [of
software inflexibility], however, understanding the software still poses some
difficulty."

which we can take to be the principal hypothesis, because this is what the
subsequent evaluation focuses on confirming. Then, the penultimate section
claims to suggest some "potential solutions":

2. "We see several possible solutions to some of the problems the software
engineer faces when trying to comprehend service-oriented software."

The problem is then broken down into subproblems and potential solutions to each
are discussed:

1a. "[There are] knowledge boundaries between organizations"

1b. The supply network is only partially visible.

1c. "An overall view [of state information] could prove difficult to achieve,
however, because of the supply network's partial visibility and the limited flow
of information between service providers due to confidentiality concerns."

1d. "Service-oriented software presents an inherent uncertainty because of its
distributed and negotiated nature."

1e. "Service-oriented software's construction means traditional understanding
support tools won~t work."



FLAWS

The underlying assumption of the analysis is that it is the customer that is
responsible for understanding why the service was not delivered as contracted.
Most of the problems then arise because the customer is denied access to the
details of the system, partly due to its complexity, but mostly because these
details are commercially confidential. Most of the proposed solutions then
discuss various ways in which the customer might persuade the service providers
to release this confidential information anyway.

This all seems very unrealistic. Surely the onus of understanding why a contract
has not been delivered is primarily on the service provider, who will not expect
to be paid until the contract is met. This obligation is recursive, with the
customer complaining to their immediate supplier, then that supplier
complaining, as appropriate, to their sub-contractors, and so on. Thus each
sub-contractor must debug their product: to meet their contractual obligations;
to improve their product; and because they are the only party with access to the
details of how their product works. It is not realistic for them to release this
confidential information to their immediate customers (and, therefore, potential
commercial rivals) up the supply chain, and expect these customers to solve the
problem for them. Any 'solution' based on such unrealistic assumptions is bound
to be unconvincing.

Another problem with the analysis is that hypothesis 1 hints at a contrast
between the problems of software understanding confronting users of
service-oriented software compared with those of traditional software. However,
there is no significant analysis of the differences in the paper.

There is no discussion of related work. Perhaps no one else has done a similar
analysis, but there is no explicit claim of novelty either.

The paper is very unclear about what the main claims or hypotheses are. The
reader is left to try to infer these, and it is very easy to focus on an issue
that the authors did not intend to be a mainstream contribution.



YOUR REVIEWS


You identified a wide variety of different potential hypotheses, many of which
were only minor or were just scene-setting background remarks. This is not
surprising, given the authors' failure to identify their intended
hypotheses. Usually, in such a situation, one should focus on the evaluation as
a pointer to the intended hypotheses. This is clearly the fictional case study
and the subsequent identification of problems and their possible
solutions. Other hypotheses were often given as much or even more attention,
somewhat diluting the impact of the review.

Only two people identified that the whole analysis was based on some
questionable assumptions, and one of them gave no details.