I got home tonight (by now last night), and realized that there might be production users out there that would not have the time or resources to do the same kind of testing, and decided to build and configure a brand new 2.5.x version of Sakai onto a brand new server and document my changes that affected performance. I will, over the next few weeks, post as much information as I can get my fingers to type up, on different issues regarding performance tuning of Sakai.
Tonight though, we only start by setting up an environment that will cater for our monitoring needs...
So I started with a clean checkout of 2.5.x, read up a bit about the requirements on hte installation docs, and settled on a tomcat 5.2.26. There was a note in the installation that required no higher than 5.2.23, but further investigation gave me the idea that the problems were fixed for 5.2.26.
I installed the latest version 5 JDK, set up some environment variables, and I was off...
I did come to realize that there were way too many different references to JDK memory settings in the installation documentation, but luckily my entire exercise revolves around the monitoring and tweaking of these values, so no matter...
During my time at Unisa, we realized that we were not handling the connection pooling parameters correctly, as it seemed we either had too many, or too few, and no clear way of monitoring their usage during load. This sparked a search and rescue mission that ended up with the best tool for inspecting tomcat and its processes I have ever seen: Lambda Probe (http://www.lambdaprobe.org).
Extract from Lambda Probe website: "Lambda Probe (formerly known as Tomcat Probe) - the ultimate tool for monitoring and management of Apache Tomcat instance in real time. Lambda Probe will help you to visualise real time information about Apache Tomcat instance via easy to use and friendly web interface."
Installation was a breeze: I had to copy the Lambda probe.war into the webapps of my newly extracted tomcat, had to make sure that the tomcat-users.xml had a user belonging to the manager role, and had to add the following to the JAVA_OPTS environment variable before starting Tomcat: -Dcom.sun.management.jmxremote
Unfortunately though, Lambda Probe can only report on database connection pools that are published via JNDI, and since Sakai uses Spring to directly create its dbcp datasources, Lambda was not able to report on the Sakai pools. Luckily the problem also presented the easy solution; since it was Spring that was injecting the datasource into Sakai, it later struck me that it should be simple enough to let Spring query JNDI and simply pull the datasource from there, in stead of configuring it in the Spring bean configuration (components.xml in the db subproject).
Before we go too far, let us first make sure we have everything we need:
- Tomcat was installed and could run successfully
- Adding probe.war from Lambda Probe and restarting tomcat worked 100%
- Testing the probe application worked fine, although it needed the extra "-Dcom.sun.management.jmxremote" parameters. I added it to the JAVA_OPTS environment, restarted and everything worked even better.
- I checked out Sakai 2.5.x, made sure Maven was working, and compiled and deployed Sakai.
- I created an empty Sakai database in MySQL, and set up the correct parameters in sakai.properties (which simply preps the components.xml in the db project, so we will still need to go change this).
- Started up Sakai and checked the logs for database creation and all the spring component instantiations etc.
Looking at the Lambda interface again, I could clearly see a difference in the amount of applications, number of threads and memory usage after the Sakai deploy.
Lambda was however still not showing any database pool resources or their usage, so here we go with the JNDI configuration:
- Inside the Tomcat server.xml there is a GlobalNamingResources section. Mimic your database settings with the following Resource tag:
<globalnamingresources>
...
<Resource name="sakaidb" auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3306/sakai_2_5_x?useUnicode=true&characterEncoding=UTF-8&relaxAutoCommit=true"
username="sakaiuser" password="sakaipassword" maxActive="10" maxIdle="10"
maxWait="5000"/>
...
</globalnamingresources> - In the Tomcat context.xml file, we need to add a resource link to the global resource inside the context definition:
<context>
...
<ResourceLink name="sakaidb"
global="sakaidb"
type="javax.sql.DataSource"/>
...
</context> - Comment out any properties ending in "@javax.sql.BaseDataSource" in your sakai.properties, as this will no longer be needed. You will notice that I did not add a validation query to the connection defined in server.xml above. Please see the Tomcat documentation for details of how to configure further parameters on your datasource.
- In the Sakai source tree, modify components.xml in db/db-impl/pack/src/webapp/WEB-INF and delete the entire
... element and all of its children. Replace it with:
<bean id="javax.sql.BaseDataSource"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName">
<value>java:comp/env/sakaidb</value>
</property>
</bean> - In the Sakai source tree, modify components.xml in
warehouse/warehouse-impl/pack/src/webapp/WEB-INF and delete all properties for the org.sakaiproject.warehouse.service.DataWarehouseManager.dataSource bean. - Rebuild your sakai: mvn clean install sakai:deploy
- Restart Tomcat, and keep a very good eye on the log files. If all goes well, you should still have a working Sakai, but you would now be able to monitor your pool usage via the /probe web application
