Informatics Report Series


Report   

EDI-INF-RR-1250


Related Pages

Report (by Number) Index
Report (by Date) Index
Author Index
Institute Index

Home
Title:Eliminating The Middleman: Peer-to-Peer Dataflow
Authors: Adam Barker ; Jon Weissman ; Jano Van Hemert
Date:Jun 2008
Publication Title:The 17th IEEE International Symposium on High Performance Distributed Computing (HPDC 2008)
Publisher:ACM/IEEE
Publication Type:Conference Paper Publication Status:Pre-print
Abstract:
Efficiently executing large-scale, data-intensive workflows such as Montage must take into account the volume and pattern of communication. When orchestrating data-centric workflows, centralised servers common to standard workflow systems can become a bottleneck to performance. However, standards-based workflow systems that rely on centralisation, e.g., Web service based frameworks, have many other benefits such as a wide user base and sustained support. This paper presents and evaluates a light-weight hybrid architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Furthermore our architecture is standards compliment, flexible and is a non-disruptive solution; service definitions do not have to be altered prior to enactment. Our architecture could be realised within any existing workflow framework, in this paper, we focus on a Web service based framework. Taking inspiration from Montage, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2--4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using our hybrid architecture increases as the complexity of a workflow grows.
Links To Paper
1st Link
Bibtex format
@InProceedings{EDI-INF-RR-1250,
author = { Adam Barker and Jon Weissman and Jano Van Hemert },
title = {Eliminating The Middleman: Peer-to-Peer Dataflow},
book title = {The 17th IEEE International Symposium on High Performance Distributed Computing (HPDC 2008)},
publisher = {ACM/IEEE},
year = 2008,
month = {Jun},
url = {http://www.adambarker.org/hpdc2008.pdf},
}


Home : Publications : Report 

Please mail <reports@inf.ed.ac.uk> with any changes or corrections.
Unless explicitly stated otherwise, all material is copyright The University of Edinburgh