First Simulation Delivery
6 months after FDR we have to deliver our first simulated data to ESO. The exact date for this is uncertain, we set April 27 2022 for ourselves, as that is the FDR2 board report + 6 months. We are in pretty good shape to achieve that with ScopeSIM (v1.4) and MicadoWISE as they currently are.
I (HB) have made this an issue in MicadoWISE because that makes it easy divide this up into smaller issues. It is labeled with "To review" because then it shows up in the kanban board and is therefore reviewed every week by me.
(@ycao and @kdleschinski I tag you in this issue, I'll send a follow-up email as well.)
Plan
Proposed plan by HB, detailed below:
- Limit our scope to data required for Standard Imaging pipeline skeleton.
- Use MicadoWISE to simulate the data, through a script in the
simulations
directory. - (Optionally) store the data in the MicadoWISE archive.
The reason to simulate through MicadoWISE is because:
- This creates the headers compliant with our design and archive (see #53 (closed) to get that in ScopeSIM directly),
- Allows full reproducibility (or almost, see #211 ),
- Makes it easy to script everything,
- Allows direct ingestion into the archive.
In the (near) future it should be possible to simulate the data directly through ScopeSIM as well; in my opinion this is not necessary for our April 26th delivery.
Scope
We should limit our scope to what is both useful and achievable. Ultimately it should probably be @ycao who has the final say in what we deliver.
Within Scope
Proposal by HB for what is within scope of our first data delivery:
- Standard Imaging Raw
- DARK_RAW (ScopeSIM Template source: empty_sky)
- FLAT_RAW (ScopeSIM Template source: flatlamp)
- SCIENCE_RAW (ScopeSIM Template source: cluster)
- Astrometric Imaging
- Nothing
- High Contrast Imaging
- Nothing
We can already simulate all this right now.
For now we should simulate the effects as they are required for the skeleton, as in we don't care about the same things that scientists care about. E.g. we don't care about the Atmospheric Dispersion, because there is nothing that we do in the pipeline with the ADC (currently, see #49 (closed)).
Unsure about Scope
Data items that HB is unsure about whether we should or should not simulate them:
- Standard Imaging Raw
- ILLUM_RAW (doable if we use the same setup as for SCIENCE_RAW)
- NONLIN_RAW (doable if we use the same setup as for FLAT_RAW)
- STDFIELD_RAW (doable if we use the same setup as for SCIENCE_RAW)
- Intermediate / Processed data from Standard Imaging Pipeline:
- It is unclear to me (HB) what is required of us w.r.t. delivering intermediate and final data products.
- Output from other pipelines (Astrometric Imaging) that is used in Standard Imaging (so required to run skeleton):
- NONLINEARITY_IMG (this is quite a complicated structure that we have not properly defined yet)
- ILLUMCORR_HDR (might be doable)
- DISTORTIONWAM_HDR and DISTORTIONELT_HDR (I expect our design of these will change based on the work of the Astrometry Working Group)
- External Data (that is required to run the pipeline, but should somehow be tied to the simulations):
- REF_ASTROM_CAT
- REF_PHOTOM_CAT
My proposal is to not simulate any of them now and add what we can.
E.g. we can probably add ILLUM_RAW and STDFIELD_RAW by just simulating the same cluster as for SCIENCE_RAW. That would technically work to test our skeleton, but doesn't make sense from a science perspective.
I (HB) am not so sure about the *_HDR data items (FITS files with all information in the headers, no pixels/tables) because we have not properly defined what these should look like. I mean, there is a data structure defined in the DRLD, but these will certainly change once we start implementing it. OTOH, it should be pretty easy to create these data items so maybe I should.
I (HB) am even less sure about the *_CAT data items (FITS files with tables, no pixels), because a) we have not very well defined what these contents should be, and b) I have no idea how to ensure that the contents would actually match the simulations. (Maybe it should be the other way around, @ycao and @buddel define there reference catalogs and use those as input to ScopeSIM when simulating STDFIELD_RAW and ILLUM_RAW.)
Out of Scope
Proposal by HB for what is out of scope for our first data delivery:
- Standard Imaging Processed
- Persistence correction frames
- Astrometric Imaging
- Everything
- High Contrast Imaging
- Everything
Process / Actions
Proposed process to use and associated actions:
(I've added names to this, but I think it should be a collaborative effort.)
-
@ycao decides on the exact scope (maybe by editing the simulation script directly?):
- Which data to simulate and process?
- How many, which DIT and NDIT etc (e.g. for flats and darks)?
- What dither pattern (e.g. for the SCIENCE_RAW)?
- What sources / effects.
-
@buddel:
- Create the simulation script.
- Run the simulations and store in the archive.
-
@kdleschinski
- Fix anything that is broken / not supported in ScopeSIM (nothing essential in v0.1.4 to run the skeleton I think).
Note that at some point HB should get out of the loop, so we can use this first-simulated-data-delivery as a project to facilitate this handover.