Liferay Portal 6 1 Performance Whitepaper
Liferay Portal 6 1 Performance Whitepaper
Liferay Portal 6 1 Performance Whitepaper
Performance
Benchmark Study of
Liferay Portal 6.1 Enterprise Edition
LIFERAY ENGINEERING
Table of Contents
Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Test Scenario. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Benchmark Configuration and Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Environment Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Benchmark Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Transaction Centric Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Collaboration Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Content and Document Management Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
WWW.LIFERAY.COM
LIFERAY ENGINEERING
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Test Scenario
The document utilizes the following conventions when discussing test cases and results:
Virtual Users Simulated users concurrently transacting on the portal system. Transactions vary depending upon the test cases
Total Users Total number of users in the portal database that could be used as part of a test
Each portal deployment is unique in its requirements and performance characteristics. Liferay collaborated with clients across a broad
spectrum of industries to determine the scenarios that best modeled product use cases. Based on this feedback, Liferay decided to
classify the test cases into three categories:
Transaction centric scenarios
Apply to financial, insurance, and ecommerce deployments where a large number of users will login and perform transaction like
online banking (bill payments, etc), online insurance applications, airline and hotel booking, and etc
Frequent authenticated access with longer average user session times
Collaboration centric scenarios
Apply to corporate intranets looking to leverage shared document repositories with other social collaboration tools like blogs,
wikis, and forums
Apply to Facebook-like social networks and developer communities
Mostly authenticated access; roughly 5:1 ratio between read and write transactions
Content and document management scenarios
Apply to corporate intranets and customers looking to manage and share documents
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Application
Tier
Web Tier
Application
Server
Apache Web
Server
Database
Tier
Database
Server
Application
Server
Hardware platforms:
1. Web Server
1 x Intel Core 2 Duo E6405 2.13GHz CPU, 2MB L2 cache (2 cores total)
4GB memory, 1 x 146GB 7.2k RPM IDE
2. Application Server
2 x Intel Core 2 Quad X5677 3.46GHz CPU, 12MB L2 cache (8 cores and 16 threads)
16GB memory, 2 x 146GB 10k RPM SCSI
3. Database Tier
2 x Intel Core 2 Quad X5677 3.46GHz CPU, 12MB L2 cache (8 cores and 16 threads)
16GB memory, 4 x 146GB 15k RPM SCSI
Network:
Gigabit network between all servers and test clients
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Software:
Liferay Portal 6.1 Enterprise Edition
Sun Java 6 (1.6.0_31)
Tomcat 7.0.26
CentOS 5.5 64-bit Linux
MySQL 5.5 Community Server
Apache HTTPD Server 2.2
Grinder 3 load test client with Liferay customizations
Methodology
Liferay utilized the Grinder load testing tool and its distributed load injectors. In all test scenarios, the injectors ramped up users at a rate
of one user every 100 milliseconds until achieving the desired virtual user load.
The benchmark data was gathered after an initial ramp up time of 5 minutes to initialize all application elements and warm up all injectors.
As part of data gathering, the following statistics were gathered:
OS level statistics on web, application, and database servers (includes CPU, context switches, IO performance)
JVM garbage collection information via Visual VM and garbage collector logs
Average transaction times, standard deviations, and throughput from the Grinder console
A single application server was used to determine maximum throughput. Once the maximum throughput was reached on a single server,
Liferay added a second application server to prove the linear scalability hypothesis: that doubling the available application server hardware
will double the maximum number of virtual user supported by the system.
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Benchmark Results
Transaction Centric Scenarios
Isolated Login
The first of two transaction centric scenario focuses on the login process of Liferay Portal. The login and permission retrieval process is
one of the most resource intensive processes within the portal. At login, the portal must retrieve user and security information from the
database and calculate authorizations.
We first examine Liferays performance with simple content portlets on the page. These portlets are extremely fast, lending average
rendering times of less than 10ms.
Table 1 illustrates the performance observed during this test. The mean time for login remains less than 200ms as we approach the
performance inflection point. At 23000 virtual users, we have a mean time () of 121ms and 95% of the logins (2) around 663ms.
The optimal performance point with relatively small standard deviation occurs somewhere between 23000 virtual users.
At 26000 virtual users, we exceed the established performance budget of this test (i.e. sub 1 second login times). However, performance
has not degraded excessively. At 29000, the system breaches into unacceptable performance territory. Thus, the performance inflection
point for login is roughly between 27000 and 29000 virtual users.
Virtual Users
DURATION (min)
(ms)
(ms)
2 (ms)
Throughput (TPS)
14000
30
38.8
18.8
76.4
399
77
18000
30
111
193
189
511
78
21000
30
114
243
278
567
85
23000
30
121
271
663
652
85
26000
30
199
443
1085
734
87
27000
30
210
412
1034
764
88
29000
30
2320
1530
5380
548
88
Login Time
6000
5000
4000
3000
Login Time
2000
1000
0
14000
18000
21000
23000
26000
27000
29000
Concurrent Users
Figure 2: Mean Login Time
WWW.LIFERAY.COM
LIFERAY ENGINEERING
In terms of throughput, the portal appears to have an optimal throughput of roughly 730 transactions per second.
Throughput
900
Transactions / sec
800
700
600
500
400
300
Throughput
200
100
0
14000
18000
21000
23000
26000
27000
29000
Concurrent Users
Figure 3: Isolated Login Throughput
Upon maxing out a single application server, a second portal application server was deployed. The benchmark results showed that
Liferay Portal was able to breach 54,000 virtual users using two application servers. At 54,000 users across two application servers,
the performance characteristics remained identical to those gathered with 27000 users on a single application server.
Login with Legacy Simulator
This test scenario helps demonstrate the impact of adding a portlet that will sleep for 2 seconds. The 2 seconds simulate the impact of
integration with systems like Salesforce.com or interacting with a companys enterprise service bus. The hypothesis is that individual
portlet performance will have impacts on the overall performance of the portal solution.
The statistics indicate a decrease in the maximum number of concurrent users prior to reaching the optimum performance point.
In this scenario, the portal reaches optimal throughput and performance at roughly 6300 virtual users, 20700 users less than the
previouslogin scenario. At the inflection point, we see that 95% (2) of the combined login and homepage transactions consume
2.6s with a mean time of 2.2s.
Virtual Users
DURATION (min)
Throughput
(TPS)
CPU (%)
3000
30
2050
28.1
2106.2
76.7
23
4200
30
2070
47.5
2165
107
33
5400
30
2100
115
2330
138
47
6300
30
2200
243
2686
160
60
6400
30
2650
2660
7970
158
61
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Login Time
Mean TXN Time (ms)
10000
8000
6000
Login Time
4000
2000
0
3000
4200
5400
6300
6400
Concurrent Users
Figure 4: Legacy Login
Figure 4 illustrates Liferay Portal approaching its optimal performance just above the 6300 virtual users threshold.
As with the first scenario, a second portal application server was deployed upon determining the inflection point. The benchmark results
showed that Liferay Portal was able to breach 12600 virtual users using two application servers. At 12600 users, the transaction times
remained similar to the times gathered on a single application server.
This test confirms that individual portlets will have an impact on the performance of the overall portal solution. Slower portlet transactions
will decrease the maximum concurrent user load each physical server may support.
Collaboration Scenarios
Message Boards
Message boards represents one of the foundational elements around social collaboration. The message board test cases demonstrates
the full range of capabilities for the Liferay message board, simulating how an end user may utilize the features. In Table 4 and 5, we see
the breakdown for each individual transaction within the test, including login, browsing, and posting.
In almost every case, 95% of the transactions remain under 2s when we have roughly 5800 virtual users. At 6200 users, we see that the
system has begun to exceed the performance inflection point.
Virtual Users
DURATION
(min)
LOGIN TIME
(ms)
LOGIN TIME
(ms)
BROWSE
CATEGORY
(ms)
BROWSE
CATEGORY
(ms)
BROWSE
THREAD
(ms)
BROWSE
THREAD
(ms)
BROWSE
POSTS
(ms)
BROWSE
POSTS
(ms)
3400
30
35.4
10.6
120
14.2
63.9
31.7
164
25.7
4200
30
37.7
12.3
133
19.1
69.8
34.9
182
31.2
5000
30
41.8
16.7
150
28.1
78.2
40.1
202
39.4
5800
30
61
32.2
201
58.6
104
60.8
250
61.7
6200
30
84
80.5
257
117
138
97.6
311
111
6400
30
156
220
322
176
178
154
377
177
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Virtual Users
POST THREAD
(ms)
POST THREAD
(ms)
REPLY THREAD
(ms)
REPLY THREAD
(ms)
TOTAL
(ms)
TOTAL
(ms)
TOTAL
2(ms)
CPU (%)
3400
86.3
11.4
98.4
17.1
568
110.7
789.4
36
4200
92.5
14.8
107
20.9
622
133.2
133.2
48
5000
104
22.6
122
30.1
698
177
1052
59
5800
141
52.1
161
57.6
918
323
1564
75
6200
193
118
218
126
1201
650.1
2501.2
76
6400
273
226
300
229
1606
1504
4614
78
350
300
Login Time
250
Browse Cat
200
Browse Thread
150
Browse Posts
100
Post Thread
50
Reply Thread
0
3400
4200
5000
5800
6200
6400
Concurrent Users
Figure 5: Collaboration Performance
As with previous tests, Liferay confirmed that the maximum user threshold doubled when doubling the number of physical servers.
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Blogging
Blogging is another cornerstone for social collaboration. As with the message board test cases, we attempt to simulate real end user
behavior of browsing, reading, and contributing to blogs. While the blogging components in Liferay reuse some of the components of
the Message Boards, we do see somewhat different performance due to the reduced complexity of the Blogs features (e.g. no nested
categories and thus reduced entitlement validation).
As shown in Tables 6 and 7, the statistics point to a performance inflection point of roughly 6000 virtual users. At this load, we observed
total mean transaction times () at 540.8ms with 95% of all transactions consuming roughly 1.0s. Individual transactions are substantially
lower. For instance, to post comments on a blog and to post a new blog entry, the statistics report 95% of the transaction at about 250ms
and 319ms respectively.
Virtual
Users
DURATION
(min)
LOGIN TIME
(ms)
LOGIN TIME
(ms)
VIEW
SUMMARIES
(ms)
VIEW
SUMMARIES
(ms)
VIEW
ENTRY
(ms)
VIEW
ENTRY
(ms)
POST NEW
ENTRY
(ms)
POST NEW
ENTRY
(ms)
5200
30
38.8
18.2
65.9
23.6
82
22.2
119
30.2
5600
30
44.6
23.1
74.7
30
91.6
27.1
134
40.9
6000
30
57.1
39.5
96.7
52.2
108
43.9
168
75.7
6400
30
107
140
146
122
147
96.1
222
161
TOTAL (ms)
TOTAL (ms)
TOTAL 2(ms)
CPU (%)
5200
105
26.4
410.7
111.4
633.5
64
5600
117
32.8
461.9
153.9
769.7
70
6000
141
54.9
540.8
266.2
1073.2
77
6400
187
136
809
655.1
2119.2
77
WWW.LIFERAY.COM
LIFERAY ENGINEERING
2000
1500
1000
0
5200
5600
6000
6400
Concurrent Users
Figure 6: 95% Transaction Time for Blogging Test Case
Figure 6 depicts the total mean transaction time as the system approaches the optimal performance point. From the Table 7, we see
total mean transaction time moving to 2.1s at 6400 users, from 1.0s at 6000 virtual users. While for many customers, 2.1s for an entire
test case of 5 transactions is perfectly acceptable performance, the underlying performance numbers indicate the system has begun
exhibiting queuing behaviors.
Content and Document Management Scenarios
Liferay provides rich capabilities for both web content management and document management. The product supports document
management features via the document and media gallery. The gallery is backed by a full featured content repository that supports
multi-level workflow approvals, custom document metadata definitions, and social collaboration features (e.g. ratings, comments, etc).
The performance test cases demonstrate the typical usage scenarios with users browsing for files, viewing file details (e.g. metadata,
comments, ratings), download the file, and finally uploading new files. The testing environment removes potential network bottlenecks
by providing fast network connections between clients downloading files and the document repository (1Gbps).
As shown in Table 8, overall transaction times for browsing, viewing, uploading, and downloading documents remain sub second across
most transactions. At the performance inflection point of 5400 users, 95% of file downloads occurred in 316ms for a 100KB document.
Document upload times for a 100KB document with 5400 virtual users remains under 500ms, coming in at 406ms for 95% of the users.
10
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Virtual
Users
DURATION
(min)
BROWSE
FOLDER
(ms)
BROWSE
FOLDER
(ms)
VIEW FILE
DETAILS
(ms)
VIEW FILE
DETAILS
(ms)
DOWNLOAD
FILE
(ms)
DOWNLOAD
FILE
(ms)
UPLOAD
FILE
(ms)
UPLOAD
FILE
(ms)
3500
30
227
59.8
68.5
59.2
28.8
79.4
125
35.3
4000
30
245
30.5
71
14.4
31.6
83.6
131
26.9
4600
30
274
43.6
80.1
19.4
36.2
101
150
40.7
5200
30
318
79.1
95.4
32.4
41.9
14.9
175
50.7
5400
30
412
148
124
61.3
58.1
129
222
92
5500
30
769
487
302
334
196
311
446
378
800
700
600
Browse Folder
500
400
300
Download File
200
Upload File
100
0
3500
4000
4600
5200
5400
5500
Concurrent Users
Figure 7 - Document Repository Mean Time
11
WWW.LIFERAY.COM
LIFERAY ENGINEERING
Summary
Liferay Engineering, in collaboration with various clients and partners, commissioned this benchmark study to demonstrate the
performance and scalability of Liferay Portal and to provide statistics for future capacity planning.
Based on the results of this study, Liferay determined that the Liferay Portal platform provides an extremely scalable and high performance
environment for building an infrastructure portal, a collaboration portal, a content portal, and any combination of these capabilities.
With its immense flexibility and proven performance and scalability, Liferay believes the Liferay Portal platform is uniquely positioned to
help bring Web 2.0 capabilities to the enterprise.
Due to the many performance enhancements introduced in the enterprise edition, the benchmarks apply to Liferay Portal 6.1 EE and
not to 6.1 CE. This approach ensures that Liferays EE subscription customers realize the benefits of the engineering teams testing
immediately while also providing similar benefits to Liferays open source community in a future community edition release.
Acknowledgements
Liferay would like to thank the Liferay customer network for their contributions in helping develop performance test cases.
Liferay would also like to thank the Liferay Portal Open Source Community for their important contributions in performing
independent benchmarking and testing.
12
WWW.LIFERAY.COM
LIFERAY, INC. is a provider of leading enterprise open source portal and collaboration
software products, used by major enterprises worldwide, including Allianz, AutoZone, Cisco
Systems, Lufthansa Flight Training, The French Ministry of Defense, and the United Nations.
Liferay, Inc. offers professional services, technical support, custom development and
professional training to ensure successful deployment in the most demanding IT environments.
2012, Liferay, Inc. All rights reserved.
041612