IFS¶

Ingesting IFS data products into Data Central allows for population of the Single Object Viewer, from which users are able to download and explore (file type dependent) those data products. The Schema Browser will also include an entry for each IFS data product type. The minimum requirements for ingesting IFS data products are described below.

Note

Please Note: this is a living document, additional requirements for various data product visualizations (2D maps etc) will be set, and updated, in the near future. Although Data Central will take every effort to minimize changes to the format of the meta-data required, in order to provide survey teams with interactive visualizations and deliver the best possible user experience, we may from time-to-time require changes to these requirements.

For ingested surveys, where possible, we will auto-generate any meta data required (from fits headers), but teams should be aware that this document will likely contain additional/updated requirements on a ~6 monthly timescale in order to be compatible with the latest Data Central release.

Important

Prerequisite: You’ll need to have provided an input catalogue of sources, as per the Catalogues documentation, following all sections up to and including Source Catalogue Identification.

Note

Remember that the documentation mentioned here is the static, paper-like documentation, the documentation on Documentent Central is entirely separate.

Data Model¶

Data Central’s ingestion process will map your data onto the Data Central data model format. IFS data are organised hierarchically, as per:

Survey
└── DataRelease
    └── Schema:IFS
        └── Facility
            └── Data Product

To explore the data model further, visit the IFS section of the Schema Browser to explore the relationships between facilities and data products.

Directory Structure¶

To ingest IFS data products, you will provide two folders, one containing the data products themselves, and one containing the metadata. You’ll also need to have provided an input catalogue of sources, as per the Catalogues documentation, following all sections up to and including Source Catalogue Identification.

Data¶

The ifs directory should contain the fits files themselves.

data
└── <survey>
    └── <datarelease>
        └── ifs
            └── <source_name>
                ├── product1.fits
                └── product2.gz
            └── <source_name>
                ├── product3.fits
                └── product4.fits

Attention

<survey> and <datarelease> should be replaced with the values you chose in Getting Started, e.g., sami and dr2

A good rule of thumb is to keep your files succinct and with as few extensions as possible (i.e, do not pack tens of extensions into your IFS cubes, separate into different files). This makes mapping the correct file (by data type) to a browser visualizer simpler. Currently we support 2D map visualizations in the browser, have a look at a SAMI DR2 object in the Single Object Viewer.

Metadata¶

The following file structure should be adopted. A top-level <survey> directory should contain a single directory per <datarelease>. Both directories should have metadata files described below which will populate the Schema Browser.

metadata
└── <survey>
    ├── <survey>_survey_meta.txt
    └── <datarelease>
        ├── <survey>_<data_release>_data_release_meta.txt
        └── ifs/
            ├── <survey>_<datarelease>_facility_meta.txt
            ├── <survey>_<datarelease>_group_meta.txt
            ├── <survey>_<datarelease>_product_ingestion.txt
            ├── <survey>_<datarelease>_product_meta.txt
            └── docs/

The metadata/catalogues/ directory will contain a minimum of 4 metadata files, plus a docs/ directory if you have supplied additional documentation for a particular facility/product.

Metadata Files¶

Attention

Metadata files are always pipe-delimited, and have the extension .txt

<survey>_<datarelease>_facility_meta.txt¶

This file describes the facilities you would like to register, and will be used to populate the Schema Browser and SOV. Provide the following a single pipe-delimited .txt file containing an entry (row) for each facility:

name	pretty_name	description \| documentation
sami	SAMI	Brief SAMI instrument description \| sami_instrument_doc.txt

Please name this file: <survey>_<datarelease>_facility_meta.txt e.g., sami_dr2_facility_meta.txt

<survey>_<datarelease>_facility_meta.txt

This file should contain the following columns

name(required=True, type=char, max_limit=100)¶

Facility name. Use only alphanumeric characters. This must be unique per data release.

pretty_name(required=True, type=char, max_limit=100)¶: A human-readable version of the facility name. This can contain any characters (up to the character limit).

description(required=True, type=char, max_limit=1000)¶: A succinct paragraph describing the facility.

documentation(required=True, type=char, max_limit=1000)¶: If you would like formatted text to appear in the schema browser, please supply the name of the file containing html-formatted text (see Formatting for more info). Note, this is typically for 2-3 paragraphs of information. Detailed documentation should be written into a Document Central article. If you do not wish to supply documentation for a particular row, leave this entry blank.

<survey>_<datarelease>_product_meta.txt¶

This file describes the types of data products you would like to register, and will be used to populate the Schema Browser and SOV. Provide the following a single pipe-delimited .txt file containing an entry (row) for each product:

facility_name	name	description	documentation	group_name	version	contact
sami	spectral_cube-blue	Fully reduced and flux calibrated SAMI cube (blue)	spec_cube_blue.html	cube	1.2	John Smith <john.smith@institute.org>
sami	sfr_default_1-comp	1-component star formation rate map from default cube	sfr.html	sfr	1.2	John Smith <john.smith@institute.org>

Please name this file: <survey>_<datarelease>_product_meta.txt e.g., sami_dr2_product_meta.txt

<survey>_<datarelease>_product_meta.txt

This file should contain the following columns

name(required=True, type=char, max_limit=100)

Data product name. Choose from the following:

name	pretty_name	vis_type
cube_blue	Flux Cube: Blue	3d_cube
adaptive_blue	Adaptively-binned Flux Cube: Blue	3d_cube
annular_blue	Annularly-binned Flux Cube: Blue	3d_cube
sectors_blue	Sectors-binned Flux Cube: Blue	3d_cube
cube_red	Flux Cube: Red	3d_cube
adaptive_red	Adaptively-binned Flux Cube: Red	3d_cube
annular_red	Annularly-binned Flux Cube: Red	3d_cube
sectors_red	Sectors-binned Flux Cube: Red	3d_cube
spectrum_3-kpc_blue	3kpc Aperture Spectrum: Blue	1d_spectrum
spectrum_re_blue	Re Elliptical Aperture Spectrum: Blue	1d_spectrum
spectrum_1-4-arcsec_blue	1”.4 Aperture Spectrum: Blue	1d_spectrum
spectrum_2-arcsec_blue	2” Aperture Spectrum: Blue	1d_spectrum
spectrum_3-arcsec_blue	3” Aperture Spectrum: Blue	1d_spectrum
spectrum_4-arcsec_blue	4” Aperture Spectrum: Blue	1d_spectrum
spectrum_3-kpc_red	3kpc Aperture Spectrum: Red	1d_spectrum
spectrum_re_red	Re Elliptical Aperture Spectrum: Red	1d_spectrum
spectrum_1-4-arcsec_red	1”.4 Aperture Spectrum: Red	1d_spectrum
spectrum_2-arcsec_red	2” Aperture Spectrum: Red	1d_spectrum
spectrum_3-arcsec_red	3” Aperture Spectrum: Red	1d_spectrum
spectrum_4-arcsec_red	4” Aperture Spectrum: Red	1d_spectrum
stellar-velocity_default_two-moment	Stellar Velocity Map	2d_map
stellar-velocity-dispersion_default_two-moment	Stellar Velocity Dispersion Map	2d_map
stellar-velocity_adaptive_two-moment	Stellar Velocity Map from Adaptively-binned Cube	2d_map
stellar-velocity-dispersion_adaptive_two-moment	Stellar Velocity Dispersion Map from Adaptively-binned Cube	2d_map
stellar-velocity_annular_two-moment	Stellar Velocity Map from Annularly-binned Cube	2d_map
stellar-velocity-dispersion_annular_two-moment	Stellar Velocity Dispersion Map from Annularly-binned Cube	2d_map
stellar-velocity_sectors_two-moment	Stellar Velocity Map from Sectors-binned Cube	2d_map
stellar-velocity-dispersion_sectors_two-moment	Stellar Velocity Dispersion Map from Sectors-binned Cube	2d_map
gas-velocity_default_1-comp	1-component Ionised Gas Velocity Map	2d_map
gas-vdisp_default_1-comp	1-component Ionised Gas Velocity Dispersion Map	2d_map
OII3728_default_1-comp	1-component Line Emission Map: [OII] (3726Å+3729Å)	2d_map
Halpha_default_1-comp	1-component Line Emission Map: Hα	2d_map
Hbeta_default_1-comp	1-component Line Emission Map: Hβ	2d_map
OIII5007_default_1-comp	1-component Line Emission Map: [OIII] (5007Å)	2d_map
OI6300_default_1-comp	1-component Line Emission Map: [OI] (6300Å)	2d_map
NII6583_default_1-comp	1-component Line Emission Map: [NII] (6583Å)	2d_map
SII6716_default_1-comp	1-component Line Emission Map: [SII] (6716Å)	2d_map
SII6731_default_1-comp	1-component Line Emission Map: [SII] (6731Å)	2d_map
gas-velocity_default_recom-comp	Recommended-component Ionised Gas Velocity Map	2d_map
gas-vdisp_default_recom-comp	Recommended-component Ionised Gas Velocity Dispersion Map	2d_map
OII3728_default_recom-comp	Recommended-component Line Emission Map: [OII] (3726Å+3729Å)	2d_map
Halpha_default_recom-comp	Recommended-component Line Emission Map: Hα	2d_map
Hbeta_default_recom-comp	Recommended-component Line Emission Map: Hβ	2d_map
OIII5007_default_recom-comp	Recommended-component Line Emission Map: [OIII] (5007Å)	2d_map
OI6300_default_recom-comp	Recommended-component Line Emission Map: [OI] (6300Å)	2d_map
NII6583_default_recom-comp	Recommended-component Line Emission Map: [NII] (6583Å)	2d_map
SII6716_default_recom-comp	Recommended-component Line Emission Map: [SII] (6716Å)	2d_map
SII6731_default_recom-comp	Recommended-component Line Emission Map: [SII] (6731Å)	2d_map
gas-velocity_adaptive_1-comp	1-component Ionised Gas Velocity Map from Adaptively-binned Cube	2d_map
gas-vdisp_adaptive_1-comp	1-component Ionised Gas Velocity Dispersion Map from Adaptively-binned Cube	2d_map
OII3728_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: [OII] (3726Å+3729Å)	2d_map
Halpha_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: Hα	2d_map
Hbeta_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: Hβ	2d_map
OIII5007_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: [OIII] (5007Å)	2d_map
OI6300_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: [OI] (6300Å)	2d_map
NII6583_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: [NII] (6583Å)	2d_map
SII6716_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: [SII] (6716Å)	2d_map
SII6731_adaptive_1-comp	1-component Line Emission Map from Adaptively-binned Cube: [SII] (6731Å)	2d_map
gas-velocity_annular_2-comp	2-component Ionised Gas Velocity Map from Annularly-binned Cube	2d_map
gas-vdisp_annular_2-comp	2-component Ionised Gas Velocity Dispersion Map from Annularly-binned Cube	2d_map
OII3728_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: [OII] (3726Å+3729Å)	2d_map
Halpha_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: Hα	2d_map
Hbeta_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: Hβ	2d_map
OIII5007_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: [OIII] (5007Å)	2d_map
OI6300_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: [OI] (6300Å)	2d_map
NII6583_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: [NII] (6583Å)	2d_map
SII6716_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: [SII] (6716Å)	2d_map
SII6731_annular_2-comp	2-component Line Emission Map from Annularly-binned Cube: [SII] (6731Å)	2d_map
gas-velocity_sectors_1-comp	1-component Ionised Gas Velocity Map from Sectors-binned Cube	2d_map
gas-vdisp_sectors_1-comp	1-component Ionised Gas Velocity Dispersion Map from Sectors-binned Cube	2d_map
OII3728_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: [OII] (3726Å+3729Å)	2d_map
Halpha_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: Hα	2d_map
Hbeta_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: Hβ	2d_map
OIII5007_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: [OIII] (5007Å)	2d_map
OI6300_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: [OI] (6300Å)	2d_map
NII6583_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: [NII] (6583Å)	2d_map
SII6716_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: [SII] (6716Å)	2d_map
SII6731_sectors_1-comp	1-component Line Emission Map from Sectors-binned Cube: [SII] (6731Å)	2d_map
gas-velocity_adaptive_recom-comp	Recommended-component Ionised Gas Velocity Map from Adaptively-binned Cube	2d_map
gas-vdisp_adaptive_recom-comp	Recommended-component Ionised Gas Velocity Dispersion Map from Adaptively-binned Cube	2d_map
OII3728_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: [OII] (3726Å+3729Å)	2d_map
Halpha_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: Hα	2d_map
Hbeta_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: Hβ	2d_map
OIII5007_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: [OIII] (5007Å)	2d_map
OI6300_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: [OI] (6300Å)	2d_map
NII6583_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: [NII] (6583Å)	2d_map
SII6716_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: [SII] (6716Å)	2d_map
SII6731_adaptive_recom-comp	Recommended-component Line Emission Map from Adaptively-binned Cube: [SII] (6731Å)	2d_map
extinct-corr_default_1-comp	Extinction correction map from 1-component Hα/Hβ flux ratio	2d_map
sfr-mask_default_1-comp	Star formation mask map from 1-component emission line ratios	2d_map
sfr_default_1-comp	Star formation rate map from 1-component Hα flux	2d_map
sfr-dens_default_1-comp	Star formation rate surface density from 1-component Hα flux	2d_map
extinct-corr_default_recom-comp	Extinction correction map from recommended-component Hα/Hβ flux ratio	2d_map
sfr-mask_default_recom-comp	Star formation mask map from recommended-component emission line ratios	2d_map
sfr_default_recom-comp	Star formation rate map from recommended-component Hα flux	2d_map
sfr-dens_default_recom-comp	Star formation rate surface density from recommended-component Hα flux	2d_map
gas-velocity_sectors_recom-comp	Recommended-component Ionised Gas Velocity Map from Sectors-binned Cube	2d_map
gas-vdisp_sectors_recom-comp	Recommended-component Ionised Gas Velocity Dispersion Map from Sectors-binned Cube	2d_map
OII3728_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: [OII] (3726Å+3729Å)	2d_map
Halpha_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: Hα	2d_map
Hbeta_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: Hβ	2d_map
OIII5007_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: [OIII] (5007Å)	2d_map
OI6300_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: [OI] (6300Å)	2d_map
NII6583_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: [NII] (6583Å)	2d_map
SII6716_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: [SII] (6716Å)	2d_map
SII6731_sectors_recom-comp	Recommended-component Line Emission Map from Sectors-binned Cube: [SII] (6731Å)	2d_map
extinct-corr_adaptive_1-comp	Extinction correction map from 1-component adaptive-binned Hα/Hβ flux ratio	2d_map
sfr-mask_adaptive_1-comp	Star formation mask map from 1-component adaptive-binned emission line ratios	2d_map
sfr_adaptive_1-comp	Star formation rate map from 1-component adaptive-binned Hα flux	2d_map
sfr-dens_adaptive_1-comp	Star formation rate surface density from 1-component adaptive-binned Hα flux	2d_map
extinct-corr_adaptive_recom-comp	Extinction correction map from recommended-component adaptive-binned Hα/Hβ flux ratio	2d_map
sfr-mask_adaptive_recom-comp	Star formation mask map from recommended-component adaptive-binned emission line ratios	2d_map
sfr_adaptive_recom-comp	Star formation rate map from recommended-component adaptive-binned Hα flux	2d_map
sfr-dens_adaptive_recom-comp	Star formation rate surface density from recommended-component adaptive-binned Hα flux	2d_map
extinct-corr_sectors_1-comp	Extinction correction map from 1-component sectors-binned Hα/Hβ flux ratio	2d_map
sfr-mask_sectors_1-comp	Star formation mask map from 1-component sectors-binned emission line ratios	2d_map
sfr_sectors_1-comp	Star formation rate map from 1-component sectors-binned Hα flux	2d_map
sfr-dens_sectors_1-comp	Star formation rate surface density from 1-component sectors-binned Hα flux	2d_map
extinct-corr_sectors_recom-comp	Extinction correction map from recommended-component sectors-binned Hα/Hβ flux ratio	2d_map
sfr-mask_sectors_recom-comp	Star formation mask map from recommended-component sectors-binned emission line ratios	2d_map
sfr_sectors_recom-comp	Star formation rate map from recommended-component sectors-binned Hα flux	2d_map
sfr-dens_sectors_recom-comp	Star formation rate surface density from recommended-component sectors-binned Hα flux	2d_map
extinct-corr_annular_2-comp	Extinction correction map from 2-component annular-binned Hα/Hβ flux ratio	2d_map
sfr-mask_annular_2-comp	Star formation mask map from 2-component annular-binned emission line ratios	2d_map
sfr_annular_2-comp	Star formation rate map from 2-component annular-binned Hα flux	2d_map
sfr-dens_annular_2-comp	Star formation rate surface density from 2-component annular-binned Hα flux	2d_map

If you have additional product names that aren’t covered by this list, please let us know as soon as possible.

facility_name(required=True, type=char, max_limit=100)¶: The name of the facility (must match a facility name from the <survey>_<datarelease>_facility_meta.txt file)

description(required=True, type=char, max_limit=1000): A succinct paragraph describing the product.

documentation(required=True, type=char, max_limit=1000): If you would like formatted text to appear in the schema browser, please supply the name of the file containing html-formatted text (see Formatting for more info). Note, this is typically for 2-3 paragraphs of information. Detailed documentation should be written into a Document Central article. If you do not wish to supply documentation for a particular row, leave this entry blank.

group_name(required=True, type=char, max_limit=100)¶: The name of the group (must match a group name from the <survey>_<datarelease>_group_meta.txt file)

version(required=True, type=char, max_limit=100)¶: Product version as defined by the team e.g., v1.8

contact(required=True, type=char, max_limit=500)¶: Format as: John Smith <john.smith@institute.org>

<survey>_<datarelease>_group_meta.txt¶

To aid the user in finding data products, a grouping is applied in the Schema Browser and the SOV (i.e., all IFS-derived spectra can be grouped under ‘Spectra’). This file describes the groups you would like to register, and will be used to populate the Schema Browser and categorize data products in the SOV. Provide the following a single pipe-delimited .txt file containing an entry (row) for each group:

name	pretty_name
cube	Cubes
stelkin	Stellar Kinematics

Please name this file: <survey>_<datarelease>_group_meta.txt e.g., sami_dr2_group_meta.txt

<survey>_<datarelease>_group_meta.txt

This file should contain the following columns

name(required=True, type=char, max_limit=100): Group name. Use only alphanumeric characters. This must be unique per data release.

pretty_name(required=True, type=char, max_limit=100): A human-readable version of the group name. This can contain any characters (up to the character limit).

<survey>_<datarelease>_product_ingestion.txt¶

The final file you will need to provide links each data product with the relevant source from your input catalogue (as mentioned above, you’ll need to have provided an input catalogue of sources, as per the Catalogues documentation, following all sections up to and including Source Catalogue Identification). This file should include an entry (row) for every file you wish to ingest into Data Central.

facility_name	data_product_name	file_name	rel_file_path	source_name
sami	OIII5007_annular_2-comp	99428_OIII5007_annular_2-comp.fits	99428/99428_OIII5007_annular_2-comp.fits	99428

Please name this file: <survey_dr>_product_ingestion.txt e.g., sami_dr2_product_ingestion.txt

<survey>_<datarelease>_product_ingestion.txt

This file should contain the following columns

facility_name(required=True, type=char, max_limit=100): The name of the facility (must match a facility name from the <survey>_<datarelease>_facility_meta.txt file)

data_product_name(required=True, type=char, max_limit=100)¶: The name of the data product (must match a facility name from the <survey>_<datarelease>_product_meta.txt file)

file_name(required=True, type=char, max_limit=100)¶: The filename of the data product you’ll be providing

rel_file_path(required=True, type=char, max_limit=100)¶: The relative path of the file. e.g., <source_name>/product1.fits. See Directory structure.

source_name(required=True, type=char, max_limit=100)¶: The source name as provided in your source catalogue (see Source Catalogue Identification).

is_best(required=False, type=bool, default=True)¶: You may associate multiple IFS with a single product, and define ONE of those products as the best (add a description of this in your <survey>_<datarelease>_product_meta.txt file or data release information on Document Central. The best product for a source will be highlighted in the SOV as such.