prototype astrometric imaging distortion calibration Critical Algorithm

Handy links

Context

Imaging: the distortion calibration algorithm is adopted as a Critical Algorithm for FDR. AI-HB+GVK: the prototyped algorithms are:

determine the >=3rd order polynomial distortion map starting from a catalog of pinhole PSF positions (x,y,dx,dy). A pair of such catalogs is derived from the Cold and Warm Astrometric Masks observations. The source extraction procedure itself is not part of the prototyping. However, the catalog contains realistic centroiding errors (i.e., measurement uncertainties and systematic errors in the dx,dy) as a function of position in the Focal Plane Array.
Determine the 2nd+3rd order polynomial distortion map starting from a catalog of stellar PSFs. Such a catalog would be extracted from the science exposure on sky. Again, source extraction procedure not part of software prototyping, but catalog will contain realistic centroiding errors.
CA in slides

Roadmap

DONE: read Riechert+18 and describe the astrometric model of MICADO in DRLD: 5.7.4 Astrometric Calibration Imaging
DONE: get MASCADO operational: the code used for that paper and since evolved
DONE: initialise a Jupyter Notebook preferably in MASCADO fork at OmegaCEN gitlab that will eventually do the following:
DONE: use the simulated dataset of pinhole and star positions in the WAM plane and their distorted positions on the detector as supplied in NominalDistortion_HRI_100x100.txt
DONE: create REF_WAM_CAT: the catalog with the WAM pinhole positions in mm.
DONE: create WAM_CAT: the catalog of observed pinhole positions on one detector.
DONE: execute mascado_analyze to recover the applied distortion.
DONE: make plot of residual distortion for the pinhole positions
DONE HB+GVK: answer the open questions at the top of mascado_fullcircle.ipynb
- Are the predicted and real coordinates indeed on the detector plane?: YES
- Can they therefore be easily converted to pixel coordinates and back?: YES: "Predicted X/Y" and "Real X/Y" are in mm and one pixel is 15micron and 4mas in Low Resolution Imager and 1.5mas in High Resolution Imager.
- Are the "Predicted X/Y" in effect the true coordinates that we want to end up with after the correction?: YES
DONE: understand why affine transformation is a 3x3 matrix (had expected 2x2). It is the usual 2x2 matrix plus two locations where the offset of the coordinate system compared to (0,0) in ZEMAX is.
DONE: HB: add plot of residual distortion for the star positions (e.g., reusing code that generates at least the top-right plot in output of mascado_analyze
DONE: HB: request J-Uwe FDR version of WAM pinhole positions, FDR version of expected distortion pattern and request expected measurement noise on the centroid positions on the detector.
DONE: HB: double check inside MASCADO code that our description of the distortion model in DRLD section 5 (so this pairwise multiplications of Legendre polynomials) is really what is used inside MASCADO as distortion model. #116 (comment 26943)
HB+GVK+J-Uwe: do estimate of accuracy of recovered distortion
GVK: rewrite text so that it explains the coordinates are actually pixels, because there is a platescale factor.
GVK: question some details to ensure there are no bugs in the algorithm.
GVK: write in bullets/pseudo-code procedure:

how to repeat it for the 36 rotator angles
to deal with multiple detectors that have unknown rotation and linear translation (perhaps first execute 9 independent affine transformations that put all detectors on a common single detector projection plane?)
deal with optical part (A) as described in DRLD Section "Astrometric Calibration for Imaging"

DONE HB: the notebook is our report in Section 13 of the DRLD: please check if we can easily export the notebook to latex+figures that can then be uploaded to overleaf. If not we will scan it and include it as figures in that section including a link to where it can be viewed in github or gitlab. Quite easy and looks good: https://gitlab.astro-wise.org/micado/datareductionlibrarydesigndrs/-/merge_requests/92
GVK: Restructure the notebook so that it also becomes the report that ESO expects in Section 13 of the DRLD.
- Make two main Sections that tackle each a main question:
  - "Do we have an algorithm to accurately determine the MICADO astrometric distortions ?"
  - "Can it be implemented compliant with ESO specifications in the MICADO Data Reduction Library?"
- Use same symbols for quantities in notebook and DRLD mathematical description
- For processing steps in the notebook give the number of the equation in the DRLD mathematical description
HB: FDR2 delivery of critical algorithm via a short release note that instructs ESO prototype reviewer (I expect Yves) to clone https://github.com/JuHeart/mascado repository, identify the rerunnable notebook. The report is the export as latex of the notebook (option for that in JupyterNotebook web-based notebook environment to be tested for that). This latex is inserted in Section 13 of the DRLD.

Gijs-notes-to-self:

read all distortion calibration relevant articles:
email thread with JUwe on distortion calibration prototyping
Column explanation for Example.txt, using as obtained here from Andre Bone.
- i and j, are just the indices of each point.
- X/Y-Field are the field (in degrees) that originated those points
- R-Field: radius of said field (sqrt(field_x^2+field_y^2)).
- Predicted X/Y: the original points in the grid (in mm, Zemax assumes a constant plate-scale across the detector), where the rays would hit if there were no distortion.
- Real X/Y: the positions where the rays actually hit the detector (blue points in the image).
- Distortion: finally, the tenth column is the value of the distortion in percentage.
Jupyter notebook with prototype distortion calibration

Edited Jul 20, 2021 by Hugo Buddelmeijer