This file contains an image simulator for the Euclid VISible instrument.
The approximate sequence of events in the simulator is as follows:
Read in a configuration file, which defines for example, detector characteristics (bias, dark and readout noise, gain, plate scale and pixel scale, oversampling factor, exposure time etc.).
Read in another file containing charge trap definitions (for CTI modelling).
Read in a file defining the cosmic rays (trail lengths and cumulative distributions).
Read in CCD offset information, displace the image, and modify the output file name to contain the CCD and quadrant information (note that VIS has a focal plane of 6 x 6 detectors).
Read in a source list and determine the number of different object types.
Read in a file which assigns data to a given object index.
Load the PSF model (a 2D map with a given over sampling or field dependent maps).
Generate a finemap (oversampled image) for each object type. If an object is a 2D image then calculate the shape tensor to be used for size scaling. Each type of an object is then placed onto its own finely sampled finemap.
Loop over the number of exposures to co-add and for each object in the object catalog:
- determine the number of electrons an object should have by scaling the object’s magnitude with the given zeropoint and exposure time.
- determine whether the object lands on to the detector or not and if it is a star or an extended source (i.e. a galaxy).
- if object is extended determine the size (using a size-magnitude relation) and scale counts, convolve with the PSF, and finally overlay onto the detector according to its position.
- if object is a star, scale counts according to the derived scaling (first step), and finally overlay onto the detector according to its position.
- add a ghost of image of the object (scaled to the peak pixel of the object) [optional].
Apply calibration unit flux to mimic flat field exposures [optional].
Apply a multiplicative flat-field map to emulate pixel-to-pixel non-uniformity [optional].
Add a charge injection line (horizontal and/or vertical) [optional].
Add cosmic ray tracks onto the CCD with random positions but known distribution [optional].
Apply detector charge bleeding in column direction [optional].
Add constant dark current and background light from Zodiacal light [optional].
Include spatially uniform scattered light to the pixel grid [optional].
Add photon (Poisson) noise [optional]
Add cosmetic defects from an input file [optional].
Add pre- and overscan regions in the serial direction [optional].
Apply the CDM03 radiation damage model [optional].
Apply CCD273 non-linearity model to the pixel data [optional].
Add readout noise selected from a Gaussian distribution [optional].
Convert from electrons to ADUs using a given gain factor.
Add a given bias level and discretise the counts (the output is going to be in 16bit unsigned integers).
Finally the simulated image is converted to a FITS file, a WCS is assigned and the output is saved to the current working directory.
Warning
The code is still work in progress and new features are being added. The code has been tested, but nevertheless bugs may be lurking in corners, so please report any weird or inconsistent simulations to the author.
This script depends on the following packages:
| requires: | PyFITS (tested with 3.0.6 and 3.3) |
|---|---|
| requires: | NumPy (tested with 1.6.1, 1.7.1, 1.8.0, 1.9.1, and 1.9.2) |
| requires: | numexpr (tested with 2.0.1 and 2.3.1) |
| requires: | SciPy (tested with 0.10.1, 0.12, 0.13, 0.14, and 0.15.1) |
| requires: | vissim-python package |
Note
This class is not Python 3 compatible. For example, xrange does not exist in Python 3 (but is used here for speed and memory consumption improvements). In addition, at least the string formatting should be changed if moved to Python 3.x.
Note
CUDA acceleration requires an NVIDIA GPU that supports CUDA and PyFFT and PyCUDA packages. Note that the CUDA acceleration does not speed up the computations unless oversampled PSF is being used. If > 2GB of memory is available on the GPU, speed up up to a factor of 50 is possible.
Before trying to run the code, please make sure that you have compiled the cdm03bidir.f90 Fortran code using f2py (f2py -c -m cdm03bidir cdm03bidir.f90) and the the .so is present in the CTI folder. For testing, please run the unittest as follows:
python simulator.py -t
This will run the unittest and compare the result to a previously calculated image. Please inspect the standard output for results.
Running the test will produce an image representing VIS lower left (0th) quadrant of the CCD (x, y) = (0, 0). Because noise and cosmic rays are randomised one cannot directly compare the science outputs but we must rely on the outputs that are free from random effects. The data subdirectory contains a file named “nonoisenocrQ0_00_00testscience.fits”, which is the comparison image without any noise or cosmic rays.
A minimal benchmarking has been performed using the TESTSCIENCE1X section of the test.config input file:
Galaxy: 26753/26753 magnitude=26.710577 intscale=185.424955025 FWHM=0.117285374813 arc sec
7091 objects were place on the detector
real 0m53.545s
user 0m51.249s
sys 0m2.228s
These numbers have been obtained with my desktop (3.4 GHz Intel Core i7 with 32GB 1600MHz DDR3) with 64-bit Python 2.7.3 installation. Further speed testing can be performed using the cProfile module as follows:
python -m cProfile -o vissim.profile simulator.py -c data/test.config -s TESTSCIENCE3X
and then analysing the results with e.g. snakeviz or RunSnakeRun.
The result above was obtained with nominally sampled PSF, however, that is only good for testing purposes. If instead one uses say four times over sampled PSF (TESTSCIENCE4x) then the execution time may increases substantially. This is mostly due to the fact that convolution becomes rather expensive when done in the finely sampled PSF domain. If the four times oversampled case is run on CPU using SciPy.signal.fftconvolve for the convolution the run time is:
real 22m48.456s
user 21m58.730s
sys 0m50.171s
Instead, if we use an NVIDIA GPU for the convolution (and code that has not been optimised), the run time is:
real 12m7.745s
user 11m55.047s
sys 0m9.535s
| version: | 1.4 |
|---|
Version and change logs:
0.1: pre-development backbone.
0.4: first version with most pieces together.
0.5: this version has all the basic features present, but not fully tested.
0.6: implemented pre/overscan, fixed a bug when an object was getting close to the upper right corner of an
image it was not overlaid correctly. Included multiplicative flat fielding effect (pixel non-uniformity).
0.7: implemented bleeding.
0.8: cleaned up the code and improved documentation. Fixed a bug related to checking if object falls on the CCD.
Improved the information that is being written to the FITS header.
0.9: fixed a problem with the CTI model swapping Q1 with Q2. Fixed a bug that caused the pre- and overscan to
be identical for each quadrant even though Q1 and 3 needs the regions to be mirrored.
1.0: First release. The code can now take an over sampled PSF and use that for convolutions. Implemented a WCS
to the header.
1.05: included an option to add flux from the calibration unit to allow flat field exposures to be generated.
Now scaled the number of cosmic rays with the exposure time so that 10s flats have an appropriate number
of cosmic ray tracks.
1.06: changed how stars are laid down on the CCD. Now the PSF is interpolated to a new coordinate grid in the
oversampled frame after which it is downsampled to the CCD grid. This should increase the centroiding
accuracy.
1.07: included an option to apply non-linearity model. Cleaned the documentation.
1.08: optimised some of the operations with numexpr (only a minor improvement).
1.1: Fixed a bug related to adding the system readout noise. In previous versions the readout noise was
being underestimated due to the fact that it was included as a variance not standard deviation.
1.2: Included a spatially uniform scattered light. Changed how the image pixel values are rounded before
deriving the Poisson noise. Included focal plane CCD gaps. Included a unittest.
1.21: included an option to exclude cosmic background; separated dark current from background.
1.25: changed to a bidirectional CDM03 model. This allows different CTI parameters to be used in parallel
and serial directions.
1.26: an option to include ghosts from the dichroic. The ghost model is simple and does not take into account
the fact that the ghost depends on the focal plane position. Fixed an issue with image coordinates
(zero indexing). Now input catalogue values agree with DS9 peak pixel locations.
1.27: Convolution can now be performed using a GPU using CUDA if the hardware is available. Convolution mode
is now controlled using a single parameter. Change from 'full' to 'same' as full provides no valid information
over 'same'. In principle the 'valid' mode would give all valid information, but in practise it leads to
truncated convolved galaxy images if the image and the kernel are of similar size.
1.28: Moved the cosmic ray event generation to a separate class for easier management. Updated the code to
generate more realistic looking cosmic rays. Included a charge diffusion smoothing to the cosmic rays
to mimic the spreading of charge within the CCD. This is closer to reality, but probably still inaccurate
given geometric arguments (charge diffusion kernels are measured using light coming from the backside of
the CCD, while cosmic rays can come from any direction and penetrate to any depth).
1.29: Fixed a bug in the object pixel coordinates for simulations other than the 0, 0 CCD. The FPA gaps
were incorrectly taken into account (forcing the objects to be about 100 pixels of per gap).
1.30: now nocti files contain ADC offset and readnoise, the same as the true output if CTI is simulated.self.information['mode']
1.31: now a single FOLDER variable at the beginning of the program that should be set to
point to the location of the vissim-python. Modified the ghost function, a fixed offset from the source, but
more suitable for the correct input model.
1.32: option to fix the random number generator seed using -f flag.
1.4: included an option to simulate an image with the shutter open leading to readout trails.
Todo
| author: | Sami-Matias Niemi |
|---|---|
| contact: | s.niemi@icloud.com |
Euclid Visible Instrument Image Simulator
The image that is being build is in:
self.image
| Parameters: | opts (OptionParser instance) – OptionParser instance |
|---|
Add either horizontal or vertical charge injection line to the image.
Add cosmic rays to the arrays based on a power-law intensity distribution for tracks. Cosmic ray properties (such as location and angle) are chosen from random Uniform distribution. For details, see the documentation for the cosmicrays class in the support package.
Include flux from the calibration source.
Add objects from the object list to the CCD image (self.image).
Scale the object’s brightness in electrons and size using the input catalog magnitude. The size-magnitude scaling relation is taken to be the equation B1 from Miller et al. 2012 (1210.8201v1; Appendix “prior distributions”). The spread is estimated from Figure 1 to be around 0”.1 (1 sigma). A random draw from a Gaussian distribution with spread of 0”.1 arc sec is performed so that galaxies of the same brightness would not be exactly the same size.
Warning
If random Gaussian dispersion is added to the scale-magnitude relation, then one cannot simulate several dithers. The random dispersion can be turned off by setting random=no in the configuration file so that dithers can be simulated and co-added correctly.
Add objects from the object list and associated ghost images to the CCD image (self.image).
Scale the object’s brightness in electrons and size using the input catalog magnitude. The size-magnitude scaling relation is taken to be the equation B1 from Miller et al. 2012 (1210.8201v1; Appendix “prior distributions”). The spread is estimated from Figure 1 to be around 0”.1 (1 sigma). A random draw from a Gaussian distribution with spread of 0”.1 arc sec is performed so that galaxies of the same brightness would not be exactly the same size.
Warning
If random Gaussian dispersion is added to the scale-magnitude relation, then one cannot simulate several dithers. The random dispersion can be turned off by setting random=no in the configuration file so that dithers can be simulated and co-added correctly.
Add pre- and overscan regions to the self.image. These areas are added only in the serial direction. Because the 1st and 3rd quadrant are read out in to a different serial direction than the nominal orientation, in these images the regions are mirrored.
The size of prescan and overscan regions are defined by the prescanx and overscanx keywords, respectively.
Add readout trails resulting from reading out the shutter open.
Quadrants assumed to be numbered: 2 3 0 1
Adds a bias level to the image being constructed.
The value of bias is read from the configure file and stored in the information dictionary (key bias).
Apply bleeding along the CCD columns if the number of electrons in a pixel exceeds the full-well capacity.
Bleeding is modelled in the parallel direction only, because the CCD273s are assumed not to bleed in serial direction.
| Returns: | None |
|---|
Apply cosmetic defects described in the input file.
Warning
This method does not work if the input file has exactly one line.
Apply dark the cosmic background. Scales the background with the exposure time.
Additionally saves the image without noise to a FITS file.
Apply dark current. Scales the dark with the exposure time.
Additionally saves the image without noise to a FITS file.
Applies multiplicative flat field to emulate pixel-to-pixel non-uniformity.
Because the pixel-to-pixel non-uniformity effect (i.e. multiplicative) flat fielding takes place before CTI and other effects, the flat field file must be the same size as the pixels that see the sky. Thus, in case of a single quadrant (x, y) = (2048, 2066).
Applies a CCD273 non-linearity model to the image being constructed.
Add Poisson noise to the image.
Applies CDM03 radiation model to the image being constructed.
See also
Class :CDM03
Applies readout noise to the image being constructed.
The noise is drawn from a Normal (Gaussian) distribution with average=0.0 and std=readout noise.
Adds spatially uniform scattered light to the image.
Configures the simulator with input information and creates and empty array to which the final image will be build on.
Converts a floating point image array (self.image) to an integer array with max values defined by the argument max.
| Parameters: | max (float) – maximum value the the integer array may contain [default 65k] |
|---|---|
| Returns: | None |
Convert from electrons to ADUs using the value read from the configuration file.
Generates finely sampled images of the input data.
Tests if the object falls on the detector area being simulated.
| Parameters: | object (list) – object to be placed to the self.image being simulated. |
|---|---|
| Returns: | whether the object falls on the detector or not |
| Return type: | bool |
Overlay data from a source object onto the self.image if image = True, else t self.ghost
| Parameters: |
|
|---|
Processes configuration information and save the information to a dictionary self.information.
The configuration file may look as follows:
[TEST]
quadrant = 0
CCDx = 0
CCDy = 0
CCDxgap = 1.643
CCDygap = 8.116
xsize = 2048
ysize = 2066
prescanx = 50
ovrscanx = 20
fullwellcapacity = 200000
dark = 0.001
readout = 4.5
bias = 1000.0
cosmic_bkgd = 0.182758225257
e_ADU = 3.1
injection = 150000.0
magzero = 15182880871.225231
exposures = 1
exptime = 565.0
readouttime = 78.0
rdose = 8.0e9
RA = 145.95
DEC = -38.16
sourcelist = data/source_test.dat
PSFfile = data/interpolated_psf.fits
parallelTrapfile = data/cdm_euclid_parallel.dat
serialTrapfile = data/cdm_euclid_serial.dat
cosmeticsFile = data/cosmetics.dat
flatfieldfile = data/VISFlatField2percent.fits
output = test.fits
addSources = yes
noise = yes
cosmetics = no
chargeInjectionx = no
chargeInjectiony = no
radiationDamage = yes
cosmicRays = yes
overscans = yes
bleeding = yes
flatfieldM = yes
random = yes
background = yes
ghosts = no
shutterOpen = no
For explanation of each field, see /data/test.config. Note that if an input field does not exist, then the values are taken from the default instrument model as described in support.VISinstrumentModel.VISinformation(). Any of the defaults can be overwritten by providing a config file with a correct field name.
Reads the config file information using configParser and sets up a logger.
Reads in the cosmic ray track information from two input files.
Stores the information to a dictionary called cr.
Reads object list using numpy.loadtxt, determines the number of object types, and finds the file that corresponds to a given object type.
The input catalog is assumed to contain the following columns:
- x coordinate
- y coordinate
- apparent magnitude of the object
- type of the object [0=star, number=type defined in the objects.dat]
- rotation [0 for stars, [0, 360] for galaxies]
This method also displaces the object coordinates based on the quadrant and the CCD to be simulated.
Note
If even a single object type does not have a corresponding input then this method forces the program to exit.
Reads in a PSF from a FITS file.
Note
at the moment this method supports only a single PSF file.
Create a single simulated image of a quadrant defined by the configuration file. Will do all steps defined in the config file sequentially.
| Returns: | None |
|---|
Smooths a given image with a gaussian kernel with widths given as sigmas. This smoothing can be used to mimic charge diffusion within the CCD.
The default values are from Table 8-2 of CCD_273_Euclid_secification_1.0.130812.pdf converted to sigmas (FWHM / (2sqrt(2ln2)) and rounded up to the second decimal.
Note
This method should not be called for the full image if the charge spreading has already been taken into account in the system PSF to avoid double counting.
| Parameters: |
|
|---|---|
| Returns: | smoothed image array |
| Return type: | ndarray |
Writes out a simple FITS file.
| Parameters: |
|
|---|---|
| Returns: | None |
Writes out a FITS file using PyFITS and converts the image array to 16bit unsigned integer as appropriate for VIS.
Updates header with the input values and flags used during simulation.
This script provides a class that can be used to generate objects such as galaxies using IRAF.
| requires: | PyRAF |
|---|---|
| requires: | PyFITS |
| requires: | NumPy |
| author: | Sami-Matias Niemi |
| contact: | smn2@mssl.ucl.ac.uk |
| version: | 0.1 |
Generates an image frame with stars and galaxies using IRAF’s artdata.
Add object(s) from inputlist to the output image.
| Parameters: | inputlist (str) – name of the input list |
|---|
Generates an ascii file with uniform random x and y positions. The magnitudes of galaxies are taken from an isotropic and homogeneous power-law distribution.
The output ascii file contains the following columns: xc yc magnitude model radius ar pa <save>
| Parameters: |
|
|---|
Generates an ascii file with uniform random x and y positions. The magnitudes of stars are taken from an isotropic and homogeneous power-law distribution.
The output ascii file contains the following columns: xc yc magnitude
| Parameters: |
|
|---|
For some reason mkobjects sometimes adds crazy values to an image. This method tries to remove those values and set them to more reasonable ones. The values > 65k are set to the median of the image.
| Parameters: | filename (str) – name of the input file to modify [default = self.settings[‘output’]] |
|---|---|
| Returns: | None |
Run all methods sequentially.