There is a small application on rails (4.2.6), using nginx + puma (3.4.0) as the web server. Debian 8.1 64bit server. I noticed one strange thing, with each request used by puma processes, the amount of memory increases, and never decreases (during the absence of any requests). This leads to the fact that the RAM on the server (512 MB) ends, and the postgres service (9.4) stops. I tried a different number of workers, threads, work in cluster mode and in normal mode - the result is always the same. After a few heavy requests the RAM ends. Is it really the way it should be? What are the ways to eliminate this problem? According to the information I found, Puma is one of the best choices to avoid the problem of slow clients and long requests. The current config is:

#!/usr/bin/env puma ... threads 2,4 ... workers 1 preload_app! on_restart do puts 'Refreshing Gemfile' ENV["BUNDLE_GEMFILE"] = "/home/.../current/Gemfile" end on_worker_boot do ActiveSupport.on_load(:active_record) do ActiveRecord::Base.establish_connection end end
  • 512MB for RoR applications (taking into account the need of the operating system) is unfortunately not enough. Need at least 1GB. - cheops
  • @cheops, but after all, this memory is more than enough for me if I make usual requests (the majority of requests are small, with a response within 500ms along with the render). And only with a bold request (render ~ 12k records in json with the processing of each record), which is rendered for about 10 seconds, the memory starts to grow, and after the completion of the request is not released. Ie made one request, the memory is still left, the server is alive. We made a couple more - the memory is over, the server is up - Marsel.V
  • That is, the whole question for now is why after the completion of the request, the used memory does not decrease back to the previous value - Marsel.V

1 answer 1

Described in the answer is really not quite true .

GC can free up memory, but the rate of release can be less than the growth rate. Outside of synthetic tests usually is.

The answer is subject to processing taking into account this fact.

Yes, this is due to the peculiarities of GC work in Ruby itself. He keeps his own pool of memory, increasing it if necessary, but never reducing it . And there is a reason for this.

Objects “swept up” by the garbage collector free memory from the point of view of the interpreter, but not from the point of view of the OS. Therefore, just the fact that the memory is occupied by the process does not mean that it really has something valuable for the program , most likely it’s just a “stock” in which the interpreter will place new objects on its own, without pulling the OS allocator.

Accordingly, at each moment of time the process will occupy the maximum of what he needed for the entire time of his life. The size of the pool will be with a small margin equal to the peak memory consumption.

The trouble with large objects is that they require large sequential memory areas. And if such a block is not in the pool, then ... the pool is still increased by such a value that a large object fits!

This problem could be partially defeated, using the compaction garbage collector, when the GC “shifts” smaller objects closer to each other in the process, thereby forming larger consecutive areas of free memory. The same could allow to give large "tails" of unallocated memory back to the OS. But it is great complicates the work of S-shnyh extensions that remember where the object was, directly at the address in memory. You can not just tell them "I moved that object, keep in mind."

You can do a lot of things.

  • Half a gigabyte for an entire rail application and data warehouses is ... very little. It makes sense to add, if not physical RAM, then at least swap.
  • Tighten the garbage collector to a more careful expansion of the pool and frequent trips. This is unlikely to help, but it may slightly delay the inevitable. This is a rather extensive and dangerous topic that requires thorough stress testing for every change.
  • (The richest in adventures) Really reduce peak memory consumption by giving big answers in pieces so that the GC has time to clean up what has already been sent to the client. There is ActionController::Live , with which you can write the answer to the stream in pieces , without loading all the original data for it into memory.