Detector sampling of optical/IR spectra: how many pixels per FWHM?

TL;DR

This paper investigates how the number of pixels per FWHM in optical/IR spectra affects measurement accuracy, noise, and resolution, emphasizing the importance of appropriate sampling for reliable spectral analysis.

Contribution

It provides a detailed simulation-based analysis of pixellation effects on spectral resolution, noise, and bias, extending the understanding of optimal sampling beyond the Nyquist limit.

Findings

Coarse sampling increases wavelength noise by 10-20% at 2 pixels/FWHM.

Pixellation significantly impacts the ability to resolve close spectral features.

Sampling frequency should be tailored to the specific instrumental Line Spread Function (LSF).

Abstract

Most optical and IR spectra are now acquired using detectors with finite-width pixels in a square array. This paper examines the effects of such pixellation, using computed simulations to illustrate the effects which most concern the astronomer end-user. Coarse sampling increases the random noise errors in wavelength by typically 10 - 20% at 2 pixels/FWHM, but with wide variation depending on the functional form of the instrumental Line Spread Function (LSF) and on the pixel phase. Line widths are even more strongly affected at low sampling frequencies. However, the noise in fitted peak amplitudes is minimally affected. Pixellation has a substantial but complex effect on the ability to see a relative minimum between two closely-spaced peaks (or relative maximum between two absorption lines). The consistent scale of resolving power presented by Robertson (2013) is extended to cover pixellated spectra. The systematic bias errors in wavelength introduced by pixellation are examined. While they may be negligible for smooth well-sampled symmetric LSFs, they are very sensitive to asymmetry and high spatial frequency substructure. The Modulation Transfer Function for sampled data is shown to give a useful indication of the extent of improperly sampled signal in an LSF. The common maxim that 2 pixels/FWHM is the Nyquist limit is incorrect and most LSFs will exhibit some aliasing at this sample frequency. While 2 pixels/FWHM is often an acceptable minimum for moderate signal/noise work, it is preferable to carry out simulations for any actual or proposed LSF to find the effects of sampling frequency. Where end-users have a choice of sampling frequencies, through on-chip binning and/or spectrograph configurations, the instrument user manual should include an examination of their effects. (Abridged)