How malloc memory eats

No, there will be nothing from the series “Ahhh, I did malloc (new), and forgot to do free (delete)!”
There will be something sophisticated here: we will cut off pieces of memory a little bit, hide them in a secluded place ... And when the operating system will pay the ransom and say “Enough!”, we will try to return everything back. It would seem that the simplest operation of allocating and freeing memory - nothing portends trouble.
Those who are interested in how to destroy the memory - I ask for habrakat

A little background

On duty, you have to work a lot with large memory buffers (imagine an image of 5000x40000 pixels). Sometimes (due to fragmentation) it is not possible to allocate a continuous piece of memory for everything. Therefore, a certain memory manager was written, which allocated how much there is, possibly in several pieces. Naturally, the memory manager must both allocate and delete. Then the following interesting thing was discovered: Task Manager after release shows the level of memory usage the same as before block allocation. However, no new memory block in the program can be allocated. The use of virtual memory analysis tools (VMMap from Mark Rusinovich) shows that the memory remains busy despite its freeing in the code and despite the TM readings.

Analysis

Let's quickly write some program that allocates and frees memory. Something like that, akin to “Hello, World!”:

intmain(void){
  constint blockCount = 1024;
  constint blockSize = 1024*1024;
  char **buf;
  printf("Hit something...\n");
  getchar();
  buf = (char**)malloc(blockCount*sizeof(char*));
  for (int i=0; i<blockCount; i++)
  {
    buf[i] = (char*)malloc(blockSize*sizeof(char));
  }
  printf("Memory allocated\n");
  printf("Hit something...\n");
  getchar();
  for (int i=0; i<blockCount; i++)
  {
    free(buf[i]);
  }
  free(buf);
  printf("Hit something...\n");
  printf("Memory freed\n");
  getchar();
  return0;
}

With simple calculations, you can make sure that the program must allocate 1 GB of memory, and then free everything. After starting and checking, all memory is freed. Hmm, the system seems to be blackmailing. However, we cut large pieces.

Now take and slightly correct the source code:

constint blockSize = 520133//К примеру...;

In this case, we get that the memory was allocated, but not freed:
Before “Memory freed”:

After “Memory freed”: The


inquiring mind of the programmer did not stop there! I started looking for a threshold at which such an effect occurs. After a short binary selection, it turned out that with a size equal to

• 520168 bytes and above - release goes fine
• 520167 bytes and below - we have the described problem

Looking ahead, I’ll say that I couldn’t explain this threshold value in any way. It is not even divisible by 1024!

Possible explanation

After a long vigil over Google and studying the forums, I came to the following conclusions.
It turns out that after allocating memory using the malloc / new functions in the event that a small chunk is allocated, the memory is not freed by the free / delete functions, but passes from the committed bit to the reserved bit. And if we access this memory immediately after deletion (apparently within the framework of one heap), then it can be re-allocated. However, when we try to allocate memory from another class (or a static function), we get an exception - not enough memory. Apparently, when allocating memory from a static function, memory is allocated not in the same heap as with normal allocation from within the application class.
As a result, after creating a large block of memory (from small pieces), we run out of memory and cannot continue to allocate for ourselves a little more, even a little bit! memory.

Wrong decision

Using the VirtualAlloc / VirtualFree ( MSDN ) functions solves this problem, the memory is completely returned to the process after use (the MEM_RELEASE key), however, when using VirtualAlloc, there is a strong fragmentation of the memory, and somewhere around 800 MB of memory is not available for use, because the maximum free block size is 28Kb. Classic malloc works better in this regard, as there is some defragmenter.

Final decision

I found a third-party implementation of malloc and free (which, as it turns out, is widely known in narrow circles), which has the classic drawback of defragmenting memory, but freeing up memory completely after use. Plus it also works much faster.
For the curious and thirsty there is a link

Remarks

It was not possible to repeat the problem under * NIX OS (Ubuntu, Debian, CentOS))
Under Windows, the problem was reproduced on Windows Server 2003 x64, Windows 7 x64, Windows XP x32.
Do not directly trust the long-tested functions right away, they can catch a catch.

UPD: MS VS 2010 is used to compile on Windows