Stimulator pipeline code

[dixilo]

This PR adds pipline code for the stimulator.
The make_stm_cal.py is the main script for this pipeline, where it makes HDF5 output files as output products of stimulator's gain and time constant analysis using stimulator.py.
Small modification on the stimulator.py is intended to fix the assignment so that concatenation of the AxisManager works properly.
Modification on pyproject.toml should be needed since stimulator.py uses ruptures package.

[ykyohei]

Thanks for the PR! I have some comments and inlince comments.

1. To make this metadata easier to use, we need to make additional ManifestDb for the CMB or other calibration observations.
   I think update_bgmap_delay is good example.
   

   sotodlib/sotodlib/site_pipeline/update_bgmap_delay.py

   Lines 1156 to 1171 in cce97f0

   	def _main(
   	cfg: BgmapDelayCfg,
   	executor: Union["MPICommExecutor", "ProcessPoolExecutor"],
   	as_completed_callable: Callable) -> None:
   	"""
   	Run update function. This is done in two steps:
   	First, we process bgmap data for each ufm, and make its db.
   	We will process it in parallel over channels.
   	Next, we make another db to map observation and the bgmap.
   	We will process this in serial since it should be quick.
   	"""
   	run_update_bgmap(cfg, executor, as_completed_callable)
   	run_update_obs(cfg)
   	return

2. readout_delay and stm_gain_with_tau_correction are also important calibration parameters that users will use, but it's not included in "stm_gain" and "stm_timeconstant" archives now.

3. I'm not sure if we need to make "stm_gain" and "stm_timeconstant" archives. For example, we can use "load_fields" to load only stm_gain, stm_tau, readout_delay etc from stm_cal archive.
   

   sotodlib/sotodlib/core/metadata/loader.py

   Lines 850 to 857 in cce97f0

   	``load_fields`` (list of str or None)
   	List of fields to load. For AxisManager metadata this may
   	include entire child AxisManagers, or fields within them
   	using "." for hierarchical addressing. For ResultSet metadata,
   	subitems of this list can be specified as dictionaries
   	(original_fieldname -> new_fieldname) which allow the user to
   	change the fieldname of the subitem. Default is None, which
   	means to load all fields. Wildcards are not supported.

4. Could you also make (draft) PR of config files for this pipeline in site-pipeline-configs repository?
   https://github.com/simonsobs/site-pipeline-configs/tree/main/lat

[dixilo]

Hi Kyohei,

Thanks for the review comments!
I made a modification according to your suggestions.

1. ManifestDb structure
   I made additional helper functions and database operation codes so that ManifestDbs will have information associating regular observatoins and stimulator calibration.
   The code structure was changed from the update_bgmap_delay (especially I didn't used codes like get_cal_obsids) since it calls DBs massively. Updating DBs at the end of _main function still requires ~5 min (measured with tiger with 1 CPU).

2. readout_delay and stm_gain_with_tau_correction
   ManifestDBs for these values are also prepared. As I will mention later there are no longer separate h5 files.

3. Archve structure
   I made a modification so that it would no longer create separate archives.
   Since users can get associated stimulator's obs_id from ManifestDb and load the necessary section of the HDF5 file in the curernt Db structure, it would be necessary.

4. Yes I made a PR for the pipeline config as well:
   https://github.com/simonsobs/site-pipeline-configs/pull/282

[ykyohei]

Thanks for all the changes. I suggest to reduce the number of manifest dbs from 4 to 2.

• stm_gain.sqlite
• stm_time_constant.sqlite
• stm_gain_with_tau_correction.sqlite -> drop
• stm_readout_delay.sqlite -> drop

There are two reasons.
stimulator time constant and readout delay should always exist together.
I also think observation with time_constant tag is supposed to produce both gain and time_constant, because the gain will be calculated with lowest chopping frequency part of data. I should confirm to @YudaiSeino.

sotodlib/sotodlib/stimulator/stimulator.py

Lines 122 to 126 in cce97f0

	valid_gain = True
	if aman.stm_cal.chopping_freqs.shape[0] <= 1:
	valid_timeconstant = False
	else:
	valid_timeconstant = True

Then the context file will be

- db: '.../lat_stimulator_cal_260913m/stm_gain.sqlite'
    label: stm_gain
    load_fields:
        - stm_gain
- db: '.../lat_stimulator_cal_260913m/stm_time_constant.sqlite'
    label: stm_tau
    load_fields:
        - stm_tau
        - readout_delay
        - stm_gain_with_tau_correction

For the update_obs_corresp, if I understand correctly this runs over all the observations everytime, is this intentional? Do we only need to run new obsevations because this runs after the stimulator data processing?