Solr I/O increases over time

Question

I’m running about eight servers with solr Servers (version 3.5) behind a Load Balancer. All servers are identical and LB is set to weight by the number of connections. The servers have about 4 million documents and receive a constant stream of requests. There is virtually no writing of new documents. When Solr starts, it works very well. But after some time running, it starts to take longer to respond and the I/O level goes crazy, reaching 100%. See the New Relic chart:

If the server behaves well initially, why does it start to fail after a while? And if I re-start Solr, I/O goes back to being low for a while, increasing again later.

Original question in English

Answer 1

The answer to this question is linked with the content of that post.

What happens in this case is that the searches of this system depend on a lot of reading of the indexes of solr. Since indexes are on disk, I/O is high. To optimize frequent disk access, Linux caches the most widely read disk areas in memory. It uses the memory not occupied by applications for this cache. When the memory gets full it again has to read from the disk. For this reason, when the solr starts, the JVM takes up less memory space and so the space left over is sufficient for the index cache.

(The problem occurred on a server with 15Gb of RAM and an index solr with 20Gb)

The solution to the problem is simply to increase the amount of machine memory so that the whole index fits into it. This way, there will be virtually no I/O when reading the index.