A Benchmark for web services

Abstract

Even though there are several benchmarks developed to measure the performance of Web service frameworks, the general tendency of them is to simulate only theoretical scenarios such as streaming homogeneous data structures like arrays. On the other hand, the computer industry has an established culture of developing performance benchmarks imitating not only imaginary situations but also real world scenarios. This dissertation discusses whether it is quite necessary to test the performance of web service frameworks against such benchmarks that closely reproduce real world situations. This discussion is based on results obtained by running two benchmarks (namely one replicating 12 different real world scenarios that are optimum candidates for web service applications and another only simulating a theoretical situation) and concludes that the real world type Benchmark represents a reasonable subset of actual scenarios because the ranking of the leading Web services frameworks is consistent with Industry wide opinions [22] while statistically reaffirming the significance of using real world type benchmarks. Additionally, this dissertation identifies complexity of the SOAP messages involved in Web service transactions and size of the payloads those messages are carrying as two major factors that affect the round trip time of the SOAP messages and reveals that a framework that is good at handling complex SOAP messages may not deal with the messages that are carrying larger payloads equally well and provides statistical proof for that.