Kdjykuj

import numpy as np



array = np.zeros((210000, 210000)) # default numpy.float64

array.nbytes

When I run the above code on my 8GB memory MacBook with macOS, no error occurs. But running the same code on a 16GB memory PC with Windows 10, or a 12GB memory Ubuntu laptop, or even on a 128GB memory Linux supercomputer, the Python interpreter will raise a MemoryError. All the test environments have 64-bit Python 3.6 or 3.7 installed.

You are most likely using Mac OS X Mavericks or newer, so 10.9 or up. From that version onwards, MacOS uses virtual memory compression, where memory requirements that exceed your physical memory are not only redirected to memory pages on disk, but those pages are compressed to save space.

For your ndarray, you may have requested ~332GB of memory, but it's all a contiguous sequence of NUL bytes at the moment, and that compresses really, really well:

That's a screenshot from the Activity Monitor tool, with the process details of my Python process where I replicated your test (use the (I) icon on the toolbar to open it); this is from the Memory tab, where you can see that the Real Memory Size column is only 9.3 MB used, against a Virtual Memory Size of 332.71GB.

Once you start setting other values for those indices, you'll quickly see the memory stats increase to gigabytes instead of megabytes:

while True:

    index = tuple(np.random.randint(array.shape[0], size=2))

    array[index] = np.random.uniform(-10 ** -307, 10 ** 307)

or you can push the limit further by assigning to every index (in batches, so you can watch the memory grow):

array = array.reshape((-1,))

for i in range(0, array.shape[0], 10**5):

    array[i:i + 10**5] = np.random.uniform(-10 ** -307, 10 ** 307, 10**5)

The process is eventually terminated; my Macbook Pro doesn't have enough swap space to store hard-to-compress gigabytes of random data:

>>> array = array.reshape((-1,))

>>> for i in range(0, array.shape[0], 10**5):

...     array[i:i + 10**5] = np.random.uniform(-10 ** -307, 10 ** 307, 10**5)

...

Killed: 9

You could argue that MacOS is being too trusting, letting programs request that much memory without bounds, but with memory compression, memory limits are much more fluid. Your np.zeros() array does fit your system, after all. Even though you probably don't actually have the swap space to store the uncompressed data, compressed it all fits fine so MacOS allows it and terminates processes that then take advantage of the generosity.

If you don't want this to happen, use resource.setrlimit() to set limits on RLIMIT_STACK to, say 2 ** 14, at which point the OS will segfault Python when it exceeds the limits.

@Martijn Pieters' answer is on the right track, but not quite right: this has nothing to do with memory compression, but instead it has to do with virtual memory.

For example, try running the following code on your machine:

arrays = [np.zeros((21000, 21000)) for _ in range(0, 10000)]

This code allocates 32TiB of memory, but you won't get an error (at least I didn't, on Linux). If I check htop, I see the following:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command

31362 user       20   0 32.1T 69216 12712 S  0.0  0.4  0:00.22 python

This because the OS is perfectly willing to overcommit on virtual memory. It won't actually assign pages to physical memory until it needs to. The way it works is:

• 
  calloc asks the OS for some memory to use


• the OS looks in the process's page tables, and finds a chunk of memory that it's willing to assign. This is fast operation, the OS just stores the memory address range in an internal data structure.


• the program writes to one of the addresses.


• the OS receives a page fault, at which point it looks and actually assigns the page to physical memory. A page is usually a few KiB in size.


• the OS passes control back to the program, which proceeds without noticing the interruption.

I have no idea why creating a single huge array doesn't work on Linux or Windows, but I'd expect it to have more to do with the platform's implementation of libc and the limits imposed there than the operating system.

For fun, try running arrays = [np.ones((21000, 21000)) for _ in range(0, 10000)]. You'll definitely get an out of memory error, even on MacOs or Linux with swap compression. Yes, certain OSes can compress RAM, but they can't compress it to the level that you wouldn't run out of memory.