In the Example_LLH_setup_fixed_dirs.ipynb there is a block that states:

from do_manage import im_dist

this gives an error since there is no file in the package with this name. Instead there is a file called do_manage2. Should the line instead be:

from do_manage2 import im_dist

Thanks!

Added in some updates I've done in the private repo.

do_manage2.py -

• Made it so that long ssh cmds are split up (there's a limit to length)
• added a --q arg to pass to sub_jobs, which is needed to submit jobs to the virtual queue
• create started directories to put files in that say if each seed has been started for in and out of FoV LLH jobs

do_llh_inFoV4realtime2.py -

• Made it so that each job, when it starts processing a seed, will create a file saying it has started
• Skips and seed that has already been started
• Each job, after it has finished its own seeds will check the other jobs' seeds to see if they have been started, and if not, will run them

do_llh_oFoV4realtime2.py -

• Made it so that each job, when it starts processing a seed, will create a file saying it has started
• Skips and seed that has already been started
• Each job, after it has finished its own seeds will check the other jobs' seeds to see if they have been started, and if not, will run them

do_rates_mle_InOutFoV2.py -

• Made it so that it doesn't crash if there's no partial coding image to use

pyscript_template_rhel7.pbs -

• Added arg pmem, which give memory per processor and lets the multi-core bkg job have more memory

Added model, LLH, and response optimizations. Changed LLH in FoV script to use these updates. Everything is python 3 compatible and tested in python 3.10. Individual parts have been testes and a whole analysis has been run with no errors.
See summary of changes below.

Changes in LLH.py

• Added a LLH_webins2 class with updates to LLH_webins
• Works the same as LLH_webins, but expects the error to be error^2 instead of error
• Has added support for models that return counts instead of rates (less multiplications)
• Caches data dpis and other data products selected for certain time bins in a dictionary so they don't have to be remade
• Added new pois_norm_conv_n*2 functions that take error^2 and are optimized to do less logs and exps

Changes in models.py

• Made it so Source_Model_InOutFoV caches the normalized photon fluxes for each set of spectral params so it doesn't recalculate it every time set_flux_params is called.
• New model, Sig_Bkg_Model added.
• It takes an already made bkg model (any model is fine, that keeps all of it's parameters fixed), and a Source_Model_InOutFoV model.
• Model only has one parameter, "A" that's used to update the signal DPIs in the Source_Model_InOutFoV model while keeping all other internal parameters fixed, bypassing calling any of the bkg models and keeping the bkg DPIs cached.
• Also has new functions that return count DPIs instead of rate DPIs, that LLH_webins2 uses and saves time by not doing time exposure every LLH eval.
• Also has new functions to get error DPIs squared, saves time by not doing the square root. LLH_webins2 support these new functions.

Changes in do_llh_inFoV4realtime2.py

• Added the new model, Sig_Bkg_Model and LLH object, LLH_webins2.
• Changed how parameters and set and updated to reflect the new model used.

Changes in response.py

• removed 2 unnecessary additions of large arrays in calc_tot_resp_dpis.

I have updated the jupyter and python files to be compatible with python3. I have also changed some imports to also be compatible with python 3 (while retaining compatibility with python2).

When running the Example_Reading_Results.ipynb, I run into an error in the 13th cell where the function get_rate_res_fnames() is not finding any files with 'rates' in the name. It is searching in the F646018360/ directory and there are none of these files in the repo. Are they supposed to be there or included with the Zenodo data files?

I have tested the code and made sure that it runs and verified that the jupyter notebooks produce the same output.

The code still needs to be tested on a high performance computing environment and have some hardcoded paths changed.

Example_LLH_setup_fixed_dirs_py3_pip.pdf Example_Reading_Results_py3_pip.pdf

Example_LLH_setup_fixed_dirs_py2_no_debug.pdf Example_Reading_Results_py2_no_debug.pdf

Listening for alerts, downloading data, should all be removed from the NITRATES repo and developed in a separate repo. This repo (Conductor or Orchestrator or...) should import NITRATES code (after https://github.com/Swift-BAT/NITRATES/pull/7 is merged) and run the analyses.

Currently many cronjobs in the data_scraping folder are used to constantly download all the data. This is overkill and also often breaks. We should instead use Jamie's API to find and download the data, most likely inside of do_data_setup.py . This would also make it much easier to run NITRATES elsewhere.

The new model, Sig_Bkg_Model and new LLH object LLH_webins2 that were made and merged here #9 , were only applied to the in FoV analysis in script do_llh_inFoV4realtime2.py . To apply it to the out of FoV analysis similar changes will need to be made to the script do_llh_outFoV4realtime2.py. The changes being using LLH_webins2 instead of LLH_webins and using Sig_Bkg_Model instead of the usual compound model to combine the signal and bkg models. Along with the new way to set the bkg and signal parameters.

I have tested the code and made sure that it runs and verified that the jupyter notebooks produce the same output.

The code still needs to be tested on a high performance computing environment and have some hardcoded paths changed.

Example_LLH_setup_fixed_dirs_py3_pip.pdf Example_Reading_Results_py3_pip.pdf

Example_LLH_setup_fixed_dirs_py2_no_debug.pdf Example_Reading_Results_py2_no_debug.pdf

There are a few things that we need to think about in order to start implementing automated testing. These are:

1. What are the things that we need to test? We definitely want to test the llh calculation and the bkg estimation, what else?
2. Can we create any codes that are self-contained to test the points in item 1?
3. What GRB do we want to make the default test case that we test everything against?