Apache Tomcat (Production Setup)

Question

Good morning, I’m doing a project and using Tomcat 8 as my server. The point is, I have a Linux server to host my application, but I would like to know if I need to do any different configuration when using Tomcat in production.

Any security related modifications or server usage required? My server is hosted in SP and has 16gb ram. Is something necessary so that it can make use of the full capacity of the server? or even modify some of the machine’s jvm settings when we are using it for production projects?

My project makes use of Vraptor 4 and Hibernate as a priority. If anyone can provide some tips.

Thank you.

Answer 1

TL;DR

There is no definitive answer, but certainly the production configurations may differ between development and production, for example.

The problem is that these settings vary from system to system and many are actually not made directly in Tomcat, but in the environment, Java or the application itself.

I will try to describe some items that are actually basic but that can serve as a start

Memoriam

Perform performance and load tests by analyzing system memory using some profiler like jvisualvm that come in the JDK.

Determine the memory used by after startup and after some time of use.

Set JVM initial memory parameters to something close to the average memory consumed, so the application boot will be faster because Java won’t need to keep allocating memory multiple times.

Try to simulate a large load in the system and see how it behaves. Of course this varies depending on the context, so try to guess how many users you need to meet at peak times.

For this, consider the heaviest routines in the system. Sometimes a system has a routine that, when executed, consumes a lot of resources. Ideally, such a routine will run in some kind of queue, otherwise two or three users can run it simultaneously and crash the system.

After analyzing the system in stress, define the maximum memory so that it is possible to meet the demand of users, but always taking care not to put too much memory and so end up using disk paging, taking into account all other processes such as databases, for example.

I always use the JVM parameter -XX:+HeapDumpOnOutOfMemoryError. If an OOEM (Out Of Memory Error) occurs it will generate an image of the memory and you can analyze relatively easily which routine or object is leaking or which memory.

Warming and health-check

On the production server it is always interesting to create a routine run after application startup to warm up the instance and check if it has been properly initialized.

The warming up (Warming) It is important to ensure that system performance will remain the same after a reboot, for example. Works like an elevator knob that auto presses three times automatically to ensure the elevator always comes faster.

Joking aside, heating and checking should ensure that the caches and objects used in the main routines of the system are properly initialized without problems.

Coding and locality

Be sure to standardize the encoding and the locale both the system, the JVM and the Tomcat, preferably for UTF-8.

Tomcat has the characteristic to interpret requests without an encoding set to be ISO-8859-1. But there is an attribute URIEncoding which can be added to the server.xml where you can set the default of your system.

Logs

Obviously, the log level of Tomcat should be much lower than in development.

While in dev, the level debug can be the standard, in production we use info or even Warning. Every recording in the log file is time consumed. One expert said he has seen logs being responsible for 50% of an application’s resource consumption.

It is important to find a balance between record and performance and preferably have some mechanism to switch the level of logs dynamically and be able to track some functionality when problems occur.

Rotate logs and avoid writing to very large and fragmented files.

Caching and Scheduling

In development, it is generally used some simplified mechanism to simulate caching and scheduling tasks.

For production, if there is a need to cache data using some tool or library, such as ehcache or memcached, can be added to the respective configuration in Tomcat for proper use.

The same goes for Quartz, for example. A production server must have its configuration parameterized according to the needs of users, which may be different from a development environment.

Static resources

Depending on system demand, you may want to put CSS, Javascript, images, fonts, and other static resources on a separate web server or CDN, thus releasing Tomcat to process what matters.

Alternatively, use a proxy to cache Varnish Cache. After a proper configuration, it will cache the static resources on first access and subsequently serve them to the user without accessing Tomcat.

Also consider minifying, concatenating, and compressing the files. Scripts and Styles can have unneeded characters removed (minification), can be concatenated (joining multiple files into one) and various other contents can be compressed (gzip) for faster streaming to the browser.

Security

Normal system requests, if it is safe, will hardly pose any risk to the server.

However, Tomcat management tools that allow monitoring and managing the server must be protected. To tell you the truth, I’ve never seen this done directly on Tomcat while these tools are exported on the web.

What is usually done is to use a proxy or firewall to block any external access that is not directed specifically to your application in HTTP request format.

All access to management tools must be done through a secure connection, for example, using an SSH terminal and/or a VPN connection.

The default settings should be safe enough for most cases, but someone might have changed something without realizing the risk, so it’s good to look at all the safety considerations in documentation and see if there’s any need to change anything.

Database

Regardless of whether you use Tomcat to connect to the bank or do so in your own application, use a pool connections is essential for performance rather than creating and closing connections for each access to the bank.

In addition, it is important to monitor the use of connections to detect leaks. It is extremely common for inattentive programmers to forget to close resources in a block finally, for example and sometimes a connection is not returned to pool. Soon the system starts to slow down, because with fewer connections new requests begin to wait in a queue to access the database. Soon the system will be completely locked, not by a large processing, but because all connections with the bank are blocked and new requests to the server are eternally waiting or until some time out.

Just like on memory case, monitor the use of connections during requests. If a request uses too many connections, it’s time to refactor the system to make better use of them. Otherwise set a limit according to the maximum expected user load at peak times.

Also consider duration of timeouts. Often we put low values, for example, 30 seconds, because we think that no routine should take longer than this and if there is some leakage of connections the system recovers in 30 seconds. But there’s always that routine that takes relatively much longer than the others and we’re forced to increase that amount to, say, five minutes or so. There is no magic.

It is important to remember that the bank access performance is not only linked to the execution of a query, but one should consider the time to go through the results (ResultSet).

Once a colleague and I were able to decrease the response time of a query screen from, say, 30 minutes to 3. The order of magnitude was the most caused by a poor configuration of how the JDBC driver collected the results from the bank. He was making this record the record. We made a tuning using various tests and we came to the conclusion that transmitting records in blocks from 300 to 400 records had the optimal performance for that query. More than that the gain was minimal and the memory consumption exacerbated.

Limits and timeouts

File uploads should be limited according to the system’s need. It makes no sense, for example, to allow 1 upload Gigabyte if the system only allows uploads of small avatars.

Time-consuming requests are a sign of a problem, but as I mentioned above about long database connections, some specific routines may require a longer timeout.

Some examples of how to configure this you can see here.

Other

Make sure the APR is installed and working. This is a library of native components that improves Tomcat performance.

I hope you are not using Jsps but some template engine like Velocity. But, in case you are, first my condolences and, without second, ensure that the WAR Jsps that are deployed in production are pre-compiled into archives class. This avoids wasting time compiling Jsps during user request. And don’t forget that Jsps should be compiled using the exact version of the application server.

Always try to use HTTPS and keep a valid certificate and not just for login. All modern systems support 100% HTTPS.

If you have a lot of competition, you may need to increase the amount of threads in the server request fulfillment pool. This article says that you can increase proportionally to the number of cores, but honestly never saw the need to put a very high value.

Considerations

I know most things weren’t exactly directed to Tomcat, but I did my best to align my server-specific knowledge and general knowledge of Java web applications.

I did this because in the vast majority of cases the burden of performance is on application and security over the system/abient.