Figure 1 shows the statistics of the jobs to produce the covariance matrix.
Figure 1: Memory and time per column of the covariance matrix, as well as the distribution of the jobs per hours.
When running on holybicep01, the covariance matrix at \(N_{\rm side} = 128\) needs:
For the 04 mask, it takes 8 or 9s per column, estimated size: 7GB
For 04b mask, takes 50s per column, estimated size: 282GB
For 04c mask, takes 5 or 6s per column, estimated size: 5.66GB
For 04d mask, 3s per column, estimated size: 1.44GB
The estimated size are for a sparse matrix. For the final dense matrix, these end up being a factor of 2 smaller matrices.
Solving the eigensystem for the 04c mask took 120 to 210 minutes, and 35GB on a single node.
Memory scaling.
To estimate the scaling of the eigensystem solving, we generate symmetric \(N \times N\) matrices for varying N values.
In figure 2, we report the scaling as well as a linear fit.
We get a factor of 12 in memory wrt the covariance matrix size, which is not fully understood at this point.
Figure 2: Scaling of the memory with covmat size.
Using this fit, we can predict that
for nside=128, we have 19456 observed pixels, the covmat should then represents then 2888 Mb, and solving the eigenproblem would need 40282.9 Mb = 39.3 Gb, which is close to the 35Gb we experimentally found.
for nside=256, we have 76270 observed pixels, the covmat should then represents then 44381 Mb, and solving the eigenproblem would need 547800.2 Mb = 534 Gb.
