Q259 : Memory Swapping During Matrix Inversion

Question
I am running a large MALZ case, and it seems to be stuck at the matrix inversion step. What is happening?

---

Answer
This problem can occur on systems that do not have sufficient RAM to hold the entire computation matrix of MALZ in memory. This memory shortage causes constant disk "swapping".

The problem generally occurs only towards the end of the matrix inversion process since, at that point, the matrix inversion algorithm needs to access many widely scattered matrix elements in rapid succession. When the matrix is too large to fit entirely into memory, this forces the operating system to load parts of the matrix from disk. Of course, memory swapping occurs very often during the normal execution of a program without making the program stall. What is unique with matrix inversion is that the program must carry out a very large number of operations (as many as 250 billions) and typically very few operations are carried out before the algorithm needs to access a new memory location. As a result, the program spends most of its time loading memory pages from the disk. When this happens, very little CPU time is spent in MALZ; most of it is spent in the memory manager!

This problem is not due to the specific matrix inversion algorithm used in MALZ: all algorithms behave in the same way. There are only two ways to get rid of this problem:
1. Install more RAM.
2. Reduce the number of conductor segments in the network.

Often, it is possible to use this last option without affecting the computation accuracy significantly.

---