ML searches with LT on 06 sims

(C. Umiltà)
Updated on 2020-09-28

Update on Sep 28, 2020: The bias issue whown in this posting has been resolved. updated results can be found here .

Comparison with previous study

In a previous study I run ML searches on DC02 sims with and without using the lensing template. Quoting from my old posting "(...) the mean of the simulations does not correspond to the exvals of the model: this could be the reason of the observed bias in the recovered r. For this reason in all simulations I subtract to the simulations the mean bias, i.e., the difference between the expectation value and the mean of the sims. The bias in r is largely removed by subtracting the mean of the sims."

The old script is the same I used with DC02 sims. The new script is adapted from the script 'make_s4_simset_XXp00_driver.m'. At the beginning I though differences came from the change in script, but it does not seem to be the case. You can only see small differences for r=0.003.

Technical note: We only use 100 sims for both DC02 and DC06. The lensing template used is the 'ideal' one. In the figure, I use the same binning for the 'broad' distributions, but I do not constrain the binning for the thin orange ones. The final pager with spectra for DC02 maps and the ideal LT can be found here, while the final pager with spectra for DC06 maps and these LT can be found here.

I run the same ML searches on DC 06. I noticed a few things:

• The DC 06 sims also present some bias, as you can see by clicking 'standard' and '06.00';
• If we remove this bias with standard mean subtraction as last time, the bias on the runs with LT vanishes, but the bias of the distribution without LT increases. You can see the purple line in the DC06 sims moving away from the true value when subtracting the mean. You can also see it in the third coluimn of the table below. However the shift is only a fraction of the width of the distribution;
• For DC02 we have a final file without the LT template (1733) and a final file with the LT template (4331). If I run ML searches on 4331 final file excluding the LT I would expect to get the same results I get from 1733. The distributions are very similar but not exactly the same (that is the difference between the purple and blue histograms). Not sure this is a true issue. For DC06 we don't have any file without the LT so I can't compare.

The table below summarizes mean and 1-$$\sigma$$ values for the histograms (reporting values only for the new script). Values are multiplied by 1000 for readability.

1733 (DC02 no LT) 4331 (DC02 with LT, LT ignored) 4326 (DC06 with LT, LT ignored) 4331 (DC02 with LT) 4326 (DC06 with LT)
standard, r=0 $$-0.9 \pm 2.7$$ $$-0.9 \pm 2.8$$ $$-0.3 \pm 3.2$$ $$-1.49 \pm 0.28$$ $$-0.850 \pm 0.070$$
with mean subtraction, r=0 $$0.0 \pm 3.0$$ $$0.2 \pm 2.8$$ $$0.5 \pm 3.3$$ $$-0.03 \pm 0.30$$ $$-0.017 \pm 0.075$$
standard, r=0.003 $$1.9 \pm 2.2$$ $$1.8 \pm 2.2$$ $$3.2 \pm 3.3$$ $$1.29 \pm 0.44$$ $$1.52 \pm 0.25$$
with mean subtraction, r=0.003 $$2.8 \pm 2.2$$ $$2.9 \pm 2.2$$ $$4.1 \pm 3.4$$ $$2.99 \pm 0.47$$ $$3.03 \pm 0.33$$

Overall we see that the bias is still present in DC06, but it is slightly reduced. Using the ideal LT the distributions are also slightly narrower in DC06 than DC02. In contrast, if we run the ML search without the LT, we see a slightly more biased and broader distribution, with or without mean subtraction.

Update

In the figure below, I reproduce Clem's post figure on bpwf and add the variables that I use in the ML script. I compare here dc06 and dc02. This is to check if we see some difference. The 'Spec' click compares simulation mean and expectation values for both Clem's script in blue/cyan and mine in red/magenta. The 'Ratio' click shows ratios between the mean of sims and expvals. The last click shows the factor I actually subtract to each sim that ends up reducing the bias (mean of sims - expvals). In the 'Ratio' and 'Mean subtraction factor' clicks the ylims are fixed and equal for all plots on the grid.

These figures do not show any particular difference in the data I used wrt the data Clem used. So the bias has to arise from somewhere else. In an email exchange, Clem suggested that " [...] it is something to do with the forcing of the mean of the LT spectra values to the exp vals - where the corresponding non-LT spectra are allowed to have their natural deviations from the exp vals". And a possible soluton would be to force the mean for all BB spectra to be exactly as the expvals. Doing this seems in fact to remove the bias, as shown in Figure 3.