Following 6 scenarios were taken into consideration for this benchmark;
1) SCENARIO-1 : Direct Service call to backend service (which is hosted at axis2 server) from the client.
2) SCENARIO-2 : Service Call to the backend service through a Pass through proxy (Using message realy)
3) SCENARIO-3 : Service call to the backend service through the XSLT transformation proxy (with xalan XSLT processor)
3) SCENARIO-3 : Service call to the backend service through the XSLT transformation proxy (with xalan XSLT processor)
4) SCENARIO-4 : Service call to the backend service through the XSLT transformation proxy (with saxon XSLT processor)
5) SCENARIO-5 : Service call to the backend service through the XSLT transformation proxy (with saxon XSLT processor) - with patched XSLT mediator which is doing XML stream transformation (using message relay)
6) SCENARIO-6 : Service call to the backend service through the XSLT transformation proxy (with saxon enterprise edition XSLT processor) - with patched XSLT mediator which is doing XML stream transformation (using message relay)
Following are the configurations of the machines which were used in the performance benchmark. For more information refer the attached document named hardware-configuration-of-machines.txt
Hardware/Software configuration of the machines used
1) Configuration of the machine(32 bit) which was running the client (load testing tool used was java-bench):
OS : Debian GNU/Linux 6.0
CPU : Intel(R) Pentium(R) 4 CPU 2.80GHz
RAM : 1 GB
2) Configuration of the machine(64 bit) which was running the axis2 server which is hosting the service:
OS : Ubuntu 11.04
CPU : Genuine Intel(R) CPU T2400 @ 1.83GHz
RAM : 3 GB
3) Configuration of the machine(64 bit) which was running the WSO2 ESB:
OS : Debian GNU/Linux 6.0
CPU : Pentium(R) Dual-Core CPU E5400 @ 2.70GHz
RAM : 4 GB
Results/Observation
The time per request were measured for the XSLT transformations with the ESB and calculated the latency numbers. The message sizes used were 5k, 10k, 50k, 100k, 200k, 300k, 400k, 500k. The time per request were measured for the number of requests 10000, 11000, 12000, 13000, 14000, 15000. During the benchmarking session, I got the numbers for xalan as well as saxon XSLT processors. Since the primary concern of this benchmark was on latency, the concurrency level used was one.
i) Time per request
| Request Size | Direct request (ms) | ESB -Pass through(Message relay) - Time per request(ms) | ESB (xalan) – Time per request (ms) | ESB (saxon) – Time per request (ms) | ESB patched (saxon) -– Time per request (ms) | ESB Patched (Saxon EE) Timer per request (ms) |
| 5k | 2.932 | 5.087 | 11.458 | 9.899 | 7.486 | 7.413 |
| 10k | 4.564 | 6.332 | 16.73 | 15.387 | 11.145 | 10.473 |
| 50k | 17.881 | 20.984 | 46.328 | 42.481 | 40.834 | 34.598 |
| 100k | 34.691 | 37.339 | 83.554 | 79.325 | 71.151 | 62.992 |
| 200k | 67.66 | 70.771 | 155.233 | 146.885 | 128.672 | 119.886 |
| 300k | 100.661 | 103.945 | 233.79 | 213.84 | 186.985 | 177.184 |
| 400k | 133.125 | 137.898 | 295.675 | 283.95 | 245.029 | 238.081 |
| 500k | 166.008 | 169.460 | 364.027 | 356.182 | 302.779 | 297.676 |
ii) Latency
The latency was calculated by the following equation.
Latency = (Time per request through ESB - Direct request)
| Request Size | ESB(Message relay) latency (ms) | ESB (xalan) latency (ms) | ESB (saxon) latency (ms) | ESB patched (saxon) latency (ms) | ESB patched (saxon-EE) latency (ms) |
| 5k | 2.155 | 8.526 | 6.967 | 4.554 | 4.481 |
| 10k | 1.768 | 12.166 | 10.823 | 6.581 | 5.909 |
| 50k | 3.103 | 28.447 | 24.6 | 22.953 | 16.717 |
| 100k | 2.648 | 48.863 | 44.634 | 36.46 | 28.301 |
| 200k | 3.111 | 87.573 | 79.225 | 61.012 | 52.226 |
| 300k | 3.284 | 133.129 | 113.179 | 86.324 | 76.523 |
| 400k | 4.773 | 162.55 | 150.825 | 111.904 | 104.956 |
| 500k | 3.452 | 198.019 | 190.174 | 136.771 | 131.668 |
iii) Overhead
The overhead percentage introduced by the ESB is calculated by the following equation.
ESB Overhead = (Time per request through ESB - Direct request time)/(Direct request time) * 100 %
| Request Size | ESB(Message relay) overhead(%) | ESB (xalan) overhead percentage (%) | ESB (saxon) overhead percentage (%) | ESB patched (saxon) overhead percentage (%) | ESB patched (saxon-EE) overhead percentage (%) |
| 5k | 73.49931787176 | 290.7912687585 | 237.619372442 | 155.3206002729 | 152.830832196453 |
| 10k | 38.737949167397 | 266.5644171779 | 237.1384750219 | 144.1936897458 | 129.469763365469 |
| 50k | 17.353615569599 | 159.0906548851 | 137.5761981992 | 128.3653039539 | 93.490296963257 |
| 100k | 7.6331036868352 | 140.8520942031 | 128.661612522 | 105.0993052953 | 81.580236949065 |
| 200k | 4.5979899497488 | 129.4309784215 | 117.0928170263 | 90.1744014189 | 77.188885604493 |
| 300k | 3.2624353026495 | 132.2547957998 | 112.4357993662 | 85.7571452698 | 76.020504465483 |
| 400k | 3.585352112676 | 122.103286385 | 113.2957746479 | 84.059342723 | 78.840187793427 |
| 500k | 2.0794178593803 | 119.2828056479 | 114.5571297769 | 82.3881981591 | 79.314249915667 |
In case of scenarios 3 & 4, the only difference is the underlying XSLT processor used by the ESB is changed from the xalan to saxon. When comparing the performance numbers of xalan and saxon; saxon is showing a little performance improvement. Therefore, there is no significant gain with the use of saxon as opposed to xalan (which is the default XSLT processor shipped with the ESB).
In case of scenario 5, I developed a new mediator to do XSLT transformation by passing in the stream directly to the XSLT processor with the use of Message Relay.
Compared to scenarios 3 & 4, scenario 5 is showing a significant performance improvement.
The difference between scenario 5 and 6 is that the underlying saxon XSLT processor is changed from the community edition to enterprise edition. Compared to scenario 5, scenario 6 is showing a performance improvement.
Throughout this post I have talked about a transformation mediator which is supporting streaming. This will be available with the next release of the WSO2 ESB. This meadiator's name will be relayTransformer.
I will write a new post on this new mediator on some other time.
NOTE: This performance figures were calculated, inorder for me to get an idea about the performance gain of this new mediator. These figures were not calculated in an ideal environment. Therefore, if you try to compare these numbers with a performance benchmark that you might have done, it might not be comparable.
References
[1] - http://wso2.org/products/download/esb/java/4.0.0/wso2esb-4.0.0.zip
[2] - http://heshans.blogspot.com/search/label/ESB
[3] - http://wso2.org/project/esb/java/4.0.0/docs/samples/message_mediation_samples.html#Sample8
No comments:
Post a Comment