There is a Jboss EAP 6.4 application server.
Our application sometimes loads the server 100%. The work of users becomes impossible.
Almost always this happens during the synchronization of the application with 1C via a web service.
It seems that the application eats up resources after receiving the data: we get a large list of data, we start to write it to the database - everything hangs. But there are inconsistencies. For example, sometimes there is little data, and the problem still appears. Sometimes, on the contrary, there is a lot of data, but everything works like a clock.
Used visualvm to examine server status. But almost to no avail.
I make a snapshot, filter by our package, sort by total time - at the top of the list are the usual methods of getting data from the database, which usually work out relatively quickly.
I was thinking about locking the database, but so far it seems unlikely too.
The important point is that data is received and processed in 1 method, and the processor has 8 cores. That is, it turns out 1 thread eats up 8 cores , which in general looks impossible. If someone has ideas on what to do in this situation, it would be very cool .. I enclose data from visual vm collected at the time of loading the CPU 100%
1 answer
Such symptoms may be due to problems with memory consumption, which leads to an excessive load on the GC ( Garbage collection ) - the memory runs out, the GC starts and "eats" 100% of the CPU . If it was not possible to free up a lot of memory, then soon the GC will start again. Configure logging of GC runs and monitor them. It is possible that by the time synchronization starts, the memory has already been “eaten” by other processes, and it is necessary to optimize them, and not synchronization.
If the web service is SOAP , then there is a high overhead for processing the XML , a lot of memory and CPU required: if possible, try switching to REST/JSON . Check synchronization processing algorithms, look for the ability to create fewer objects. If synchronization receives data on changes since the previous synchronization, try launching it more often to reduce the amount of data transferred at a time.
Experiment with the size of the memory available to the application (JVM parameter Xmx ), for an industrial server usually indicate at least 2-4 GB. Do not rush to immediately allocate too much memory, because in this case, although the GC is likely to be launched less frequently, but its running time may increase significantly - there may come the so-called stop the world , during which the application becomes practically inoperable. See also server memory - it is possible that the RAM has run out and the dumping of its contents to disk begins.
If the Java version Java not the latest, try upgrading, because GC algorithms are improved from version to version. As far as I know in the 8th version there were significant improvements. There are also many GC settings.
Tips on JVM settings are an extreme case: first of all, deal with your software, because in the absolute majority of cases, all performance problems are in the application software, not in the JVM .
