MetadataTranslator#

class astro_metadata_translator.MetadataTranslator(header, filename=None)#

Bases: object

Per-instrument metadata translation support.

Parameters:

header (dict-like) – Representation of an instrument header that can be manipulated as if it was a dict.
filename (str or ResourcePathExpression, optional) – Name of the file whose header is being translated. For some datasets with missing header information this can sometimes allow for some fixups in translations. It is usually used for error reporting and logging.

Attributes Summary

`all_properties`	All the valid properties for this translator including extensions.
`default_resource_package`	Module name to use to locate the correction resources.
`default_resource_root`	Default package resource path root to use to locate header correction files within the `default_resource_package` package.
`default_search_path`	Default search path to use to locate header correction files.
`extensions`	`PropertyDefinition`)
`name`	The declared name of the translator.
`supported_instrument`	Name of instrument understood by this translation class.
`translators`	All registered metadata translation classes.

Methods Summary

`are_keys_ok`(keywords)	Are the supplied keys all present and defined?.
`can_translate`(header[, filename])	Indicate whether this translation class can translate the supplied header.
`can_translate_with_options`(header, options)	Determine if a header can be translated with different criteria.
`cards_used`()	Cards used during metadata extraction.
`defined_in_this_class`(name)	Report if the specified class attribute is defined specifically in this class.
`determine_translatable_headers`(filename[, ...])	Given a file return all the headers usable for metadata translation.
`determine_translator`(header[, filename])	Determine a translation class by examining the header.
`fix_header`(header, instrument, obsid[, filename])	Apply global fixes to a supplied header.
`is_key_ok`(keyword)	Return `True` if the value associated with the named keyword is present in this header and defined.
`is_keyword_defined`(header, keyword)	Return `True` if the value associated with the named keyword is present in the supplied header and defined.
`observing_date_to_observing_day`(...)	Return the YYYYMMDD integer corresponding to the observing day.
`observing_date_to_offset`(observing_date)	Calculate the observing day offset to apply for a given observation.
`quantity_from_card`(keywords, unit[, ...])	Calculate a Astropy Quantity from a header card and a unit.
`resource_root`()	Return package resource to use to locate correction resources within an installed package.
`search_paths`()	Search paths to use when searching for header fix up correction files.
`to_altaz_begin`()	Return value of altaz_begin from headers.
`to_altaz_end`()	Return the AltAz for the end of the observation.
`to_boresight_airmass`()	Return value of boresight_airmass from headers.
`to_boresight_rotation_angle`()	Return value of boresight_rotation_angle from headers.
`to_boresight_rotation_coord`()	Return value of boresight_rotation_coord from headers.
`to_can_see_sky`()	Return whether the observation can see the sky or not.
`to_dark_time`()	Return value of dark_time from headers.
`to_datetime_begin`()	Return value of datetime_begin from headers.
`to_datetime_end`()	Return value of datetime_end from headers.
`to_detector_exposure_id`()	Return value of detector_exposure_id from headers.
`to_detector_group`()	Return value of detector_group from headers.
`to_detector_name`()	Return value of detector_name from headers.
`to_detector_num`()	Return value of detector_num from headers.
`to_detector_serial`()	Return value of detector_serial from headers.
`to_detector_unique_name`()	Return a unique name for the detector.
`to_exposure_group`()	Return the group label associated with this exposure.
`to_exposure_id`()	Return value of exposure_id from headers.
`to_exposure_time`()	Return value of exposure_time from headers.
`to_exposure_time_requested`()	Return the requested exposure time in seconds.
`to_focus_z`()	Return a default defocal distance of 0.0 mm if there is no keyword for defocal distance in the header.
`to_group_counter_end`()	Return the observation counter of the observation that ends this group.
`to_group_counter_start`()	Return the observation counter of the observation that began this group.
`to_has_simulated_content`()	Return a boolean indicating whether any part of the observation was simulated.
`to_instrument`()	Return value of instrument from headers.
`to_location`()	Return value of location from headers.
`to_object`()	Return value of object from headers.
`to_observation_counter`()	Return an integer corresponding to how this observation relates to other observations.
`to_observation_id`()	Return value of observation_id from headers.
`to_observation_reason`()	Return the reason this observation was taken.
`to_observation_type`()	Return value of observation_type from headers.
`to_observing_day`()	Return the YYYYMMDD integer corresponding to the observing day.
`to_observing_day_offset`()	Return the offset required to calculate observing day.
`to_physical_filter`()	Return value of physical_filter from headers.
`to_pressure`()	Return value of pressure from headers.
`to_relative_humidity`()	Return value of relative_humidity from headers.
`to_science_program`()	Return value of science_program from headers.
`to_telescope`()	Return value of telescope from headers.
`to_temperature`()	Return value of temperature from headers.
`to_tracking_radec`()	Return value of tracking_radec from headers.
`to_visit_id`()	Return value of visit_id from headers.
`translator_version`()	Return the version string for this translator class.
`validate_value`(value, default[, minimum, ...])	Validate the supplied value, returning a new value if out of range.

Attributes Documentation

all_properties: dict[str, PropertyDefinition] = {}#: All the valid properties for this translator including extensions.

default_resource_package = 'astro_metadata_translator'#: Module name to use to locate the correction resources.

default_resource_root: str | None = None#: Default package resource path root to use to locate header correction files within the default_resource_package package.

default_search_path: Sequence[str] | None = None#: Default search path to use to locate header correction files.

extensions: dict[str, PropertyDefinition] = {}#

PropertyDefinition)

Some instruments have important properties beyond the standard set; this is the place to declare that they exist, and they will be treated in the same way as the standard set, except that their names will everywhere be prefixed with ext_.

Each property is indexed by name (str), with a corresponding PropertyDefinition.

Type:: Extension properties (str

name: str | None = None#: The declared name of the translator.

supported_instrument: str | None = None#: Name of instrument understood by this translation class.

translators: dict[str, type[MetadataTranslator]] = {'DECam': <class 'astro_metadata_translator.translators.decam.DecamTranslator'>, 'HSC': <class 'astro_metadata_translator.translators.hsc.HscTranslator'>, 'MegaPrime': <class 'astro_metadata_translator.translators.megaprime.MegaPrimeTranslator'>, 'SDSS': <class 'astro_metadata_translator.translators.sdss.SdssTranslator'>, 'SuprimeCam': <class 'astro_metadata_translator.translators.suprimecam.SuprimeCamTranslator'>}#: All registered metadata translation classes.

Methods Documentation

are_keys_ok(keywords)#

Are the supplied keys all present and defined?.

Parameters:: keywords (iterable of str) – Keywords to test.
Returns:: all_ok – True if all supplied keys are present and defined.
Return type:: bool

abstract classmethod can_translate(header, filename=None)#

Indicate whether this translation class can translate the supplied header.

Parameters:

header (dict-like) – Header to convert to standardized form.
filename (str, optional) – Name of file being translated.

Returns:

can – True if the header is recognized by this class. False otherwise.

Return type:

bool

classmethod can_translate_with_options(header, options, filename=None)#

Determine if a header can be translated with different criteria.

Parameters:

header (dict-like) – Header to convert to standardized form.
options (dict) – Headers to try to determine whether this header can be translated by this class. If a card is found it will be compared with the expected value and will return that comparison. Each card will be tried in turn until one is found.
filename (str, optional) – Name of file being translated.

Returns:

can – True if the header is recognized by this class. False otherwise.

Return type:

bool

Notes

Intended to be used from within can_translate implementations for specific translators. Is not intended to be called directly from determine_translator.

cards_used()#

Cards used during metadata extraction.

Returns:: used – Cards used when extracting metadata.
Return type:: frozenset of str

classmethod defined_in_this_class(name)#

Report if the specified class attribute is defined specifically in this class.

Parameters:: name (str) – Name of the attribute to test.
Returns:: in_class – True if there is a attribute of that name defined in this specific subclass. False if the method is not defined in this specific subclass but is defined in a parent class. Returns None if the attribute is not defined anywhere in the class hierarchy (which can happen if translators have typos in their mapping tables).
Return type:: bool

Notes

Retrieves the attribute associated with the given name. Then looks in all the parent classes to determine whether that attribute comes from a parent class or from the current class. Attributes are compared using id().

classmethod determine_translatable_headers(filename, primary=None)#

Given a file return all the headers usable for metadata translation.

This method can optionally be given a header from the file. This header will generally be the primary header or a merge of the first two headers.

In the base class implementation it is assumed that this supplied header is the only useful header for metadata translation and it will be returned unchanged if given. This can avoid unnecessarily re-opening the file and re-reading the header when the content is already known.

If no header is supplied, a header will be read from the supplied file using read_basic_metadata_from_file, allowing it to merge the primary and secondary header of a multi-extension FITS file. Subclasses can read the header from the data file using whatever technique is best for that instrument.

Subclasses can return multiple headers and ignore the externally supplied header. They can also merge it with another header and return a new derived header if that is required by the particular data file. There is no requirement for the supplied header to be used.

Parameters:

filename (str or lsst.resources.ResourcePathExpression) – Path to a file in a format understood by this translator.
primary (dict-like, optional) – The primary header obtained by the caller. This is sometimes already known, for example if a system is trying to bootstrap without already knowing what data is in the file. For many instruments where the primary header is the only relevant header, the primary header will be returned with no further action.

Yields:

headers (iterator of dict-like) – A header usable for metadata translation. For this base implementation it will be either the supplied primary header or a header read from the file. This implementation will only ever yield a single header.

Return type:

Iterator[MutableMapping[str, Any]]

Notes

Each translator class can have code specifically tailored to its own file format. It is important not to call this method with an incorrect translator class. The normal paradigm is for the caller to have read the first header and then called determine_translator() on the result to work out which translator class to then call to obtain the real headers to be used for translation.

classmethod determine_translator(header, filename=None)#

Determine a translation class by examining the header.

Parameters:

header (dict-like) – Representation of a header.
filename (str, optional) – Name of file being translated.

Returns:

translator – Translation class that knows how to extract metadata from the supplied header.

Return type:

type [MetadataTranslator]

Raises:

ValueError – None of the registered translation classes understood the supplied header.

classmethod fix_header(header, instrument, obsid, filename=None)#

Apply global fixes to a supplied header.

Parameters:

header (dict) – The header to correct. Correction is in place.
instrument (str) – The name of the instrument.
obsid (str) – Unique observation identifier associated with this header. Will always be provided.
filename (str, optional) – Filename associated with this header. May not be set since headers can be fixed independently of any filename being known.

Returns:

modified – True if a correction was applied.

Return type:

bool

Notes

This method is intended to support major discrepancies in headers such as:

Periods of time where headers are known to be incorrect in some way that can be fixed either by deriving the correct value from the existing value or understanding the that correction is static for the given time. This requires that the date header is known.
The presence of a certain value is always wrong and should be corrected with a new static value regardless of date.

It is assumed that one off problems with headers have been applied before this method is called using the per-obsid correction system.

Usually called from astro_metadata_translator.fix_header.

For log messages, do not assume that the filename will be present. Always write log messages to fall back on using the obsid if filename is None.

is_key_ok(keyword)#

Return True if the value associated with the named keyword is present in this header and defined.

Parameters:: keyword (str) – Keyword to check against header.
Returns:: is_ok – True if the header is present and not-None. False otherwise.
Return type:: bool

static is_keyword_defined(header, keyword)#

Return True if the value associated with the named keyword is present in the supplied header and defined.

Parameters:

header (dict-lik) – Header to use as reference.
keyword (str) – Keyword to check against header.

Returns:

is_defined – True if the header is present and not-None. False otherwise.

Return type:

bool

classmethod observing_date_to_observing_day(observing_date, offset)#

Return the YYYYMMDD integer corresponding to the observing day.

The offset is subtracted from the time of observation before calculating the year, month and day.

Parameters:

observing_date (astropy.time.Time) – The observation date.
offset (astropy.time.TimeDelta | numbers.Real | None) – The offset to subtract from the observing date when calculating the observing day. If a plain number is given it is taken to be in units of seconds. If None no offset is applied.

Returns:

day – The observing day as an integer of form YYYYMMDD.

Return type:

int

Notes

For example, if the offset is +12 hours both 2023-07-06T13:00 and 2023-07-07T11:00 will return an observing day of 20230706 because the observing day goes from 2023-07-06T12:00 to 2023-07-07T12:00.

classmethod observing_date_to_offset(observing_date)#

Calculate the observing day offset to apply for a given observation.

In some cases the definition of the observing day offset has changed during the lifetime of the instrument. For example lab data might have a different offset to that when the instrument is on the telescope.

Parameters:: observing_date (astropy.time.Time) – The observation date.
Returns:: offset – The offset to apply when calculating the observing day for a specific time of observation. None implies the offset is not known for that date.
Return type:: astropy.time.TimeDelta or None

quantity_from_card(keywords, unit, default=None, minimum=None, maximum=None, checker=None)#

Calculate a Astropy Quantity from a header card and a unit.

Parameters:

keywords (str or list of str) – Keyword to use from header. If a list each keyword will be tried in turn until one matches.
unit (astropy.units.UnitBase) – Unit of the item in the header.
default (float, optional) – Default value to use if the header value is invalid. Assumed to be in the same units as the value expected in the header. If None, no default value is used.
minimum (float, optional) – Minimum possible valid value, optional. If the calculated value is below this value, the default value will be used.
maximum (float, optional) – Maximum possible valid value, optional. If the calculated value is above this value, the default value will be used.
checker (Callable, optional) – Callback function to be used by the translator method in case the keyword is not present. Function will be executed as if it is a method of the translator class. Running without raising an exception will allow the default to be used. Should usually raise KeyError.

Returns:

q – Quantity representing the header value.

Return type:

astropy.units.Quantity

Raises:

KeyError – The supplied header key is not present.

resource_root()#

Return package resource to use to locate correction resources within an installed package.

Return type:

tuple[str | None, str | None]

Returns:

resource_package (str) – Package resource name. None if no package resource are to be used.
resource_root (str) – The name of the resource root. None if no package resources are to be used.

search_paths()#

Search paths to use when searching for header fix up correction files.

Returns:: paths – Directory paths to search. Can be an empty list if no special directories are defined.
Return type:: list

Notes

Uses the classes default_search_path property if defined.

abstract to_altaz_begin()#

Return value of altaz_begin from headers.

Telescope boresight azimuth and elevation at start of observation.

Returns:: The translated property.
Return type:: astropy.coordinates.builtin_frames.altaz.AltAz
Parameters:: self (MetadataTranslator)

to_altaz_end()#

Return the AltAz for the end of the observation.

Base class implementation returns None. Subclasses should override if the value is known.

Returns:: altaz – The AltAz for the end of the observation.
Return type:: astropy.coordinates.AltAz or None

abstract to_boresight_airmass()#

Return value of boresight_airmass from headers.

Airmass of the boresight of the telescope.

Returns:: The translated property.
Return type:: float
Parameters:: self (MetadataTranslator)

abstract to_boresight_rotation_angle()#

Return value of boresight_rotation_angle from headers.

Angle of the instrument in boresight_rotation_coord frame.

Returns:: The translated property.
Return type:: astropy.coordinates.angles.core.Angle
Parameters:: self (MetadataTranslator)

abstract to_boresight_rotation_coord()#

Return value of boresight_rotation_coord from headers.

Coordinate frame of the instrument rotation angle (options: sky, unknown).

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

to_can_see_sky()#

Return whether the observation can see the sky or not.

Returns:: can_see_sky – True if the detector is receiving photons from the sky. False if the sky is not visible to the detector. None if the metadata translator does not know one way or the other.
Return type:: bool or None

Notes

The base class translator uses a simple heuristic of returning True if the observation type is “science” or “object” and False if the observation type is “bias” or “dark”. For all other cases it will return None.

abstract to_dark_time()#

Return value of dark_time from headers.

Duration of the exposure with shutter closed (seconds).

Returns:: The translated property.
Return type:: astropy.units.quantity.Quantity
Parameters:: self (MetadataTranslator)

abstract to_datetime_begin()#

Return value of datetime_begin from headers.

Time of the start of the observation.

Returns:: The translated property.
Return type:: astropy.time.core.Time
Parameters:: self (MetadataTranslator)

abstract to_datetime_end()#

Return value of datetime_end from headers.

Time of the end of the observation.

Returns:: The translated property.
Return type:: astropy.time.core.Time
Parameters:: self (MetadataTranslator)

abstract to_detector_exposure_id()#

Return value of detector_exposure_id from headers.

Unique integer identifier for this detector in this exposure.

Returns:: The translated property.
Return type:: int
Parameters:: self (MetadataTranslator)

abstract to_detector_group()#

Return value of detector_group from headers.

Collection name of which this detector is a part. Can be None if there are no detector groupings.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

abstract to_detector_name()#

Return value of detector_name from headers.

Name of the detector within the instrument (might not be unique if there are detector groups).

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

abstract to_detector_num()#

Return value of detector_num from headers.

Unique (for instrument) integer identifier for the sensor.

Returns:: The translated property.
Return type:: int
Parameters:: self (MetadataTranslator)

abstract to_detector_serial()#

Return value of detector_serial from headers.

Serial number/string associated with this detector.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

to_detector_unique_name()#

Return a unique name for the detector.

Base class implementation attempts to combine detector_name with detector_group. Group is only used if not None.

Can be over-ridden by specialist translator class.

Returns:: name – detector_group``_``detector_name if detector_group is defined, else the detector_name is assumed to be unique. If neither return a valid value an exception is raised.
Return type:: str
Raises:: NotImplementedError – Raised if neither detector_name nor detector_group is defined.

to_exposure_group()#

Return the group label associated with this exposure.

Base class implementation returns the exposure_id in string form. A subclass may do something different.

Returns:: name – The exposure_id converted to a string.
Return type:: str

abstract to_exposure_id()#

Return value of exposure_id from headers.

Unique (with instrument) integer identifier for this observation.

Returns:: The translated property.
Return type:: int
Parameters:: self (MetadataTranslator)

abstract to_exposure_time()#

Return value of exposure_time from headers.

Actual duration of the exposure (seconds).

Returns:: The translated property.
Return type:: astropy.units.quantity.Quantity
Parameters:: self (MetadataTranslator)

to_exposure_time_requested()#

Return the requested exposure time in seconds.

Base class implementations returns the same value as exposure_time. This information may not be available for all instruments.

Returns:: exptime – The recorded exposure time in seconds.
Return type:: astropy.units.Quantity

to_focus_z()#

Return a default defocal distance of 0.0 mm if there is no keyword for defocal distance in the header. The default keyword for defocal distance is FOCUSZ.

Returns:: focus_z – The defocal distance from header or the 0.0mm default.
Return type:: astropy.units.Quantity

to_group_counter_end()#

Return the observation counter of the observation that ends this group.

The definition of the relevant group is up to the metadata translator. It can be the last observation in the exposure_group or the last observation in the visit, but must be derivable from the metadata of this observation. It is of course possible that the last observation in the group does not exist if a sequence of observations was not completed.

Returns:: counter – The observation counter for the end of the relevant group. Default implementation always returns the observation counter of this observation.
Return type:: int

to_group_counter_start()#

Return the observation counter of the observation that began this group.

The definition of the relevant group is up to the metadata translator. It can be the first observation in the exposure_group or the first observation in the visit, but must be derivable from the metadata of this observation.

Returns:: counter – The observation counter for the start of the relevant group. Default implementation always returns the observation counter of this observation.
Return type:: int

to_has_simulated_content()#

Return a boolean indicating whether any part of the observation was simulated.

Returns:: is_simulated – True if this exposure has simulated content. This can be if some parts of the metadata or data were simulated. Default implementation always returns False.
Return type:: bool

abstract to_instrument()#

Return value of instrument from headers.

The instrument used to observe the exposure.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

abstract to_location()#

Return value of location from headers.

Location of the observatory.

Returns:: The translated property.
Return type:: astropy.coordinates.earth.EarthLocation
Parameters:: self (MetadataTranslator)

abstract to_object()#

Return value of object from headers.

Object of interest or field name.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

to_observation_counter()#

Return an integer corresponding to how this observation relates to other observations.

Returns:: sequence – The observation counter. Always 0 for this implementation.
Return type:: int

Notes

Base class implementation returns 0 to indicate that it is not known how an observatory will define a counter. Some observatories may not use the concept, others may use a counter that increases for every observation taken for that instrument, and others may define it to be a counter within an observing day.

abstract to_observation_id()#

Return value of observation_id from headers.

Label uniquely identifying this observation (can be related to ‘exposure_id’).

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

to_observation_reason()#

Return the reason this observation was taken.

Base class implementation returns the science if the observation_type is science, else unknown. A subclass may do something different.

Returns:: name – The reason for this observation.
Return type:: str

abstract to_observation_type()#

Return value of observation_type from headers.

Type of observation (currently: science, dark, flat, bias, focus).

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

to_observing_day()#

Return the YYYYMMDD integer corresponding to the observing day.

Returns:: day – The observing day as an integer of form YYYYMMDD. If the header is broken and is unable to obtain a date of observation, 0 is returned and the assumption is made that the problem will be caught elsewhere.
Return type:: int

Notes

Base class implementation uses the TAI date of the start of the observation corrected by the observing day offset. If that offset is None no offset will be applied.

The offset is subtracted from the time of observation before calculating the year, month and day.

For example, if the offset is +12 hours both 2023-07-06T13:00 and 2023-07-07T11:00 will return an observing day of 20230706 because the observing day goes from 2023-07-06T12:00 to 2023-07-07T12:00.

to_observing_day_offset()#

Return the offset required to calculate observing day.

Base class implementation returns None.

Returns:: offset – The offset to apply. Returns None if the offset is not defined.
Return type:: astropy.time.TimeDelta or None

Notes

This offset must be subtracted from a time of observation to calculate the observing day. This offset must be added to the YYYYMMDDT00:00 observing day to calculate the time span coverage of the observing day.

abstract to_physical_filter()#

Return value of physical_filter from headers.

The bandpass filter used for this observation.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

abstract to_pressure()#

Return value of pressure from headers.

Atmospheric pressure outside the dome.

Returns:: The translated property.
Return type:: astropy.units.quantity.Quantity
Parameters:: self (MetadataTranslator)

abstract to_relative_humidity()#

Return value of relative_humidity from headers.

Relative humidity outside the dome.

Returns:: The translated property.
Return type:: float
Parameters:: self (MetadataTranslator)

abstract to_science_program()#

Return value of science_program from headers.

Observing program (survey or proposal) identifier.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

abstract to_telescope()#

Return value of telescope from headers.

Full name of the telescope.

Returns:: The translated property.
Return type:: str
Parameters:: self (MetadataTranslator)

abstract to_temperature()#

Return value of temperature from headers.

Temperature outside the dome.

Returns:: The translated property.
Return type:: astropy.units.quantity.Quantity
Parameters:: self (MetadataTranslator)

abstract to_tracking_radec()#

Return value of tracking_radec from headers.

Requested RA/Dec to track.

Returns:: The translated property.
Return type:: astropy.coordinates.sky_coordinate.SkyCoord
Parameters:: self (MetadataTranslator)

abstract to_visit_id()#

Return value of visit_id from headers.

ID of the Visit this Exposure is associated with.

Science observations should essentially always be associated with a visit, but calibration observations may not be.

Returns:: The translated property.
Return type:: int
Parameters:: self (MetadataTranslator)

classmethod translator_version()#

Return the version string for this translator class.

Returns:: version – String identifying the version of this translator.
Return type:: str

Notes

Assumes that the version is available from the __version__ variable in the parent module. If this is not the case a translator should subclass this method.

static validate_value(value, default, minimum=None, maximum=None)#

Validate the supplied value, returning a new value if out of range.

Parameters:

value (float) – Value to be validated.
default (float) – Default value to use if supplied value is invalid or out of range. Assumed to be in the same units as the value expected in the header.
minimum (float) – Minimum possible valid value, optional. If the calculated value is below this value, the default value will be used.
maximum (float) – Maximum possible valid value, optional. If the calculated value is above this value, the default value will be used.

Returns:

value – Either the supplied value, or a default value.

Return type:

float