ARI's Gaia Services

Table of contents

Gaia

Which Gaia data are currently available?

The Gaia data will be provided little by little as different catalog releases. Below, you can see how many releases are scheduled and what is their content.

Estimated period Available data
1st Release 14th September 2016 - 12:30 CEST
  • Positions:
    ra, ra_error, dec, dec_error
  • G mean magnitudes:
    phot_g_mean_mag
  • Proper motions & Parallaxes from Tycho-2 for corresponding Gaia sources (source: TGAS):
    pmra, pmra_error, pmdec, pmdec_error
2nd Release April 2018
  • Median radial velocities for sources brighter than 12 mag (RVS)
  • Blue & red photometries (fluxes and magnitudes for all sources)
  • Proper motions
  • Parallaxes
  • Epoch astrometry for a pre-selected list of > 10,000 asteroids
  • Photometry for a sample of variable stars
3rd Release ??
  • Orbital parameters
4th Release ??
Final Release 2022

Statistics

Where can I find statistics about table and columns for every available catalog?

All available catalogs are described, column by column, in the page  Data & Statistics. If you do not find (anymore) a column or table, it means it is not available (anymore).

"All-In-Once"

You can find statistical information in an all-in-once image like in the figure below.

Below this figure on the  Data & Statistics page, you have two buttons allowing to display and download more statistics. By doing so, a modal dialog opens and present you the following information.

Javascript must be enabled to see these additional statistics! If it is not enabled, nothing will happen when hovering the "all-in-once" figure.

Basic statistics

"Basic statistics" means here the following information: count, minimum, maximum, mean, standard deviation and quartiles (Q1, median and Q3).

Below is an example of how these statistics are reported in the statistics view:

Histogram

The histogram is available under two formats: image (PNG) and table (CSV). This latter lets you reproduce it or access the counts for each histogram bin. See how...

Sky-map

The sky-map is divided into two parts: the map itself on the left and a legend on the right. This latter indicates you the mapping between the pixel values and the pixel colors. The histogram on the right of the color map shows the ratio of displayed pixels for each range of colors.

The sky-map is available under two formats: image (PNG) and table (FITS). Thanks to the FITS file, you can get the raw value of each pixel and/or reproduce the map with different settings (color map, normalisation, projection, ...). See how...

How to send and display a column's histogram in TOPCAT?

  1. Open TOPCAT.

  2. Send the histogram to TOPCAT.

    On the statistics view of the column of interest, click on Send to TOPCAT.

    If you are not yet connected to SAMP, a dialog will popup. Just click on Yes.

    TOPCAT should have now received the table. If so, it should appear in the table list under a name generated with the table and column name (e.g. for the column vtmag of Tycho2, it would be tycho2.vtmag).

    If sending the table with SAMP does not work, download the CSV file of the histogram: button Download CSV on the statistics view of the column of interest. Then, in TOPCAT, load the table: FileLoad Table.

  3. Plot the histogram.

    Now, ensure the table you sent is selected and click on the Histogram Plot button (or alternatively, go into the menu GraphicsHistogram Plot):

    In the bottom part of the opened window, set X = minRange and Weight = count, like shown below.

    Congratulations, you have reproduce the column histogram in TOPCAT!

    You can now zoom in/out, change the bin size down to a certain limit, have a cumulative histogram, normalize the count values, flip axes, put axes in log, select a subset of bins, ...

How the sky-maps are computed?

All sky-maps are based on Healpix. The resolution is adapted in function of the size of the catalog in order to obtain as less "empty pixels" (i.e. pixels with no information) as possible. The order/resolution of the Healpix tessellation is generally indicated in the header of the FITS file.

Pixel values

In density maps, the value of a pixel is the number of sources contained in the corresponding sky area.
In column maps, it is the average of this column value for all sources contained in this sky area.

Image or FITS?

The images aim to provide a quick preview of the map. A linear normalization is performed on all pixel values, and the limit values of the color map are adjusted so that excluding outliers (determined with an histogram of 100 bins ; bins representing less than 5E-4% of the total number of pixels are considered as outliers). On the right part of the image, the legend indicates you the mapping between the pixel values and the pixel colors. The histogram on the right of the color map shows the ratio of displayed pixels (including outliers) for each range of colors.

The FITS files are Healpix maps. Since it is a table, you can display it in TOPCAT, but it would be actually more interesting in Aladin if you want to project the map on the sky, to play with the color map or to get a specific pixel value.

How to send and display a sky-map in Aladin?

  1. Open Aladin.

  2. Send the Healpix map to Aladin.

    On the statistics view of the sky-map of interest, click on Send to Aladin.

    If you are not yet connected to SAMP, a dialog will popup. Just click on Yes.

    Aladin should have now received and already displayed the map. If so, it should appear in the plan list (right panel) under a name generated with the table and column name (e.g. for the column e_vtmag of Tycho2, it would be tycho2.e_vtmag).

    Congratulations, you have displayed your sky-map in Aladin!

    You can now move on the sky, zoom in-out, change the coordinate system, change the color map and the normalisation (see button of the tool bar pixel), change the projection (see button of the tool bar prop), ...

    By default, the Send to Aladin action also change the color map in Aladin to Rainbow ; otherwise the default color map is Gray (i.e. shades of gray).

    If sending the table with SAMP does not work, download the FITS file of the sky-map: button Download FITS on the statistics view of the column of interest. Then, in Aladin, load the table: FileOpen, and then go in the tab File. You can also drag & drop the file from a file browser directly in the Aladin view area.

Single Source Search

What is a Single Source Search service?

This Web-Service let's you query the Gaia catalogue in order to get only one source. You can either provide a Gaia source ID, a pair of coordinates or an object name (as known by Simbad or NED). It is also possible to provide a CSV-like file listing sources to fetch.

When an object name or a position is given, only the closest Gaia source is returned. However, as described below, it is possible to get more Gaia sources close to the specified position with a special parameter.

This service always return all information available in the Gaia catalog. It is not possible to choose the column you want to get.

Which tables are accessible in the Single Source Search service?

Only two tables can be targeted by this service: gaia_source and tgas_source.

How to use the Single Source Search Service?

The Single Source Search Service is not a VO standard. It is a Web-Service that can be access easily from a website, by command line or any other tool able to perform HTTP GET/POST requests.

ARI Web-Site

The easiest way to use this service is through the ARI's Gaia web-page: http://gaia.ari.uni-heidelberg.de/singlesource.html. All you have to do is fill in the form and click on what you want to do:

  • View: to display the result below on the same web-page. The closest source will be shown entirely as a table while the other candidates will be plotted on an Aladin Lite view.
  • Download as...: to download the result in the format of your choice.
  • Send to...: to send the result using SAMP to a VO client such as TOPCAT.

You may notice that in addition of all available information about a matching Gaia source, its angular distance from the target position is returned. This value is merely labeled distance and is expressed in degrees. You can see it either between brackets in the title of the left panel for the closest source or in the measurement table of the Aladin Lite view when you click on another Gaia source candidate. Of course, this value is also available when you download/send the query result.

HTTP GET/POST requests

As said above, you can simply query this Web-Service by command line or from a standalone application by sending HTTP requests with appropriate parameters to http://gaia.ari.uni-heidelberg.de/singlesource/search for Gaia or http://gaia.ari.uni-heidelberg.de/singlesource/tgas for TGAS.

Both HTTP-GET and HTTP_POST methods are supported with exactly the same parameters. All the allowed parameters are described just below:

Example Description
General parameters
f votable The format in which you want to get the result.
Three values are allowed: csv, json and votable.
If this parameter is omitted or if another value is provided, the result will be returned using the CSV format.
Search by ID
id 381540121205828224 The Gaia source ID of the source you want the information.
Search by Object Name
(the closest Gaia source will be returned)
obj rr lyrae The name of the target object.
This name is resolved using the CDS Sesame Name Resolver.
v whatever If this parameter is provided (whatever is its value), the Single Source Service will returned the closest position but also the 20 other closest Gaia sources.
Search by Position
(the closest Gaia source will be returned)
ra 247.35191542
16 29 24.4597
16:29:24.4597
The Right Ascension (ICRS, epoch=2015) of the target position. It must be either expressed in degrees or written with the HMS notation.
dec -26.432002612
-26 25 55.2094
-26:25:55.2094
The Declination (ICRS, epoch=2015) of the target position. It must be either expressed in degrees or written with the DMS notation.
v whatever If this parameter is provided (whatever is its value), the Single Source Service will returned the closest position but also the 20 other closest Gaia sources.
Search by File
positions

This research mode can be performed only with an HTTP-POST request with encoding=multipart/form-data.

Each line must designate a source that you want to find in the Gaia catalogue. A line entry can be one of the following: an object name, a Gaia source ID or a pair of ICRS coordinates (RA and DEC values must be expressed in degrees or using a sexagesimal notation (HMS for RA and DMS for DEC) and must be separated by a comma).

The returned document will always contain as many rows as the input file. When no match is found for an input source, an empty row is returned.

If an object name is wrong or a pair of coordinates is incorrect, the whole request will be canceled and an error message will be returned ; the number of the incorrect line will be indicated.

A row is always starting with the two following columns:

  • input_source: the input row (i.e. object name or coordinates).
  • distance: the angular distance (in degrees) between the input target and the matched Gaia/TGAS source. This column is NULL (or empty) when the input target was a Gaia source ID.
Multiple researches in a a single HTTP request?

It is NOT possible to research more than one source in a single HTTP request, except by uploading a CSV file (parameter positions)!

If parameters of different research modes (i.e. by id, by position, object name or file) are set in the same HTTP request, the following priority rule will be applied: positions > id > obj > ra & dec.

Besides, if the same parameter is provided several times, only the last value will be taken into account.

Examples of Single Source Search queries in command line
# Research for the Gaia source 381540121205828224
wget -O mySource.csv 'http://gaia.ari.uni-heidelberg.de/singlesource/search?id=381540121205828224'

# Research of the closest TGAS source around rr lyrae (result in JSON)
wget -O mySource.json 'http://gaia.ari.uni-heidelberg.de/singlesource/tgas?obj=rr+lyrae&f=json'

# Research of the closest Gaia source around a specific position (result in VOTable with more candidates)
wget -O mySources.vot 'http://gaia.ari.uni-heidelberg.de/singlesource/search?ra=12.3&dec=88.26&f=votable&v=true'

# Research by file (result in VOTable)
curl -F "positions=@pos.csv" -F "f=votable" 'http://gaia.ari.uni-heidelberg.de/singlesource/search'

Which output formats are available?

Three output formats are proposed:

  • CSV
  • JSON
  • VOTable

You can choose the format you want using the parameter f (allowed values: csv, json and votable). If no such parameter is set or if the given format name is unknown/unsupported, the default format will be CSV.

Only the VOTable format provides metadata (description, unit, UCD) for each output column.

Cone Search

What is a Cone Search service?

A Cone Search service (originally called Simple Cone Search service) is a web service following the standard defined by the IVOA. A such service is able to execute cone search requests (i.e. research of sources around a given position in the sky and with a given radius) on a specified catalog.
See the IVOA standard

This service has a fixed behavior and standard parameters. Thus, whoever is providing a such service in the world, a Cone Search compatible client is already able to use it

Which tables are accessible in the Cone Search service?

Only two tables can be targeted by this service: gaia_source and tgas_source.

How to use the Cone Search service?

Some clients
  • This specific Cone Search service comes with a web interface available at this page: Cone Search.
  • Besides, any Cone Search service is accessible using a Cone Search client. A well-known client for this purpose is TOPCAT.
    To use it, click on the menu item VOCone Search. In the opened window, you can either search in the registry for this service or write/copy-paste the following URL in the bottom part: http://gaia.ari.uni-heidelberg.de/cone/search for Gaia or http://gaia.ari.uni-heidelberg.de/cone/tgas for TGAS.
Parameters

Minimum three parameters must be provided:

  • RA: right ascension in degrees between [0;360] (ICRS,J2015)
  • DEC: declination in degrees between [-90;+90] (ICRS,J2015)
  • SR: radius of the cone in degrees between [0;180]

An additional parameter is possible: VERB. It lets you choose the amount of columns you want to get. Only three values are possible: 1 (minimum), 2 (normal) and 3 (maximum). If none is specified by the user, the default value is 2. The real meaning of these values depends in function of the service/catalog. In the case of this service: 1 is for [source_id, ra, dec, parallax, pmra, pmdec, phot_g_mean_mag], 2 is to get the same columns + their errors, and 3 is to get all available columns.

Cone Search in command line?
# Cone search around ( 12.3 ; 88.26 ) with a radius of 1°
wget -O myResultFile.vot 'http://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1'

# Same example with CURL
curl -o myResultFile.vot 'http://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1'

# Same example but with ALL available columns
wget -O myResultFile.vot 'http://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1&VERB=3'
# or
curl -o myResultFile.vot 'http://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1&VERB=3'

Which output formats are available?

Only one output format is possible in a Simple Cone Search Service: VOTable.

Why are my results never bigger than 10,000,000 rows?

Considering the size of the whole Gaia catalog, this Cone Search service is limiting the result files to 10 million rows. This limit may change (bigger or smaller) in the future.

If you are not really interested by all sources in the specified cone, you should use the TAP service in order to add more constraints on your research.

I get so many/few columns. How to get less/more?

By default, this Cone Search service returns the following columns:

  • source_id
  • ra
  • ra_error
  • dec
  • dec_error
  • parallax
  • parallax_error
  • pmra
  • pmra_error
  • pmdec
  • pmdec_error
  • phot_g_mean_mag

Thanks to the optional parameter VERB you can choose the amount of columns you want to get. Only three values are possible: 1 (minimum), 2 (normal) and 3 (maximum). If none is specified by the user, the default value is 2. If you use the value 1 you will get less column (i.e. same columns without the errors). On the contrary, the value 3 will let you get all available columns.

Be careful when using VERB=3 that your search radius stays reasonable. Otherwise, the number of matching rows multipled by the number of all columns (≥ 50 for the Gaia catalog) can result into a huge result file.

TAP

What is a TAP service?

A TAP (Table Access Protocol) service is a web service following the standard defined by the IVOA. A such service is able to any query expressed in ADQL on one or more tables.
See the IVOA standard

ADQL (Astronomical Data Query Language) is a query language derived from SQL, language broadly used by DataBase Management Systems. However, only SELECT-type queries are possible. In addition of the most common SQL functions and features, ADQL provides a way to select sources by specifying sky regions.
See the IVOA standard

This service, as the Cone Search service, has a fixed behavior and standard parameters. Thus, whoever is providing a such service in the world, a TAP compatible client is already able to use it

Which catalogs are accessible in the TAP service?

All catalogs and their columns available for TAP requests are listed on the page Data & Statistics. If you do not find (anymore) a column or table, it means it is not available (anymore).

How to use the TAP service?

Some clients
  • This specific TAP service comes with a web interface available at this page: TAP.
  • Besides, any Cone Search service is accessible using a Cone Search client. A well-known client for this purpose is TOPCAT.
    To use it, click on the menu item VOTable Access Protocol (TAP) Query. In the opened window, you can either search in the registry for this service or write/copy-paste the following URL in the bottom part: http://gaia.ari.uni-heidelberg.de/tap.
  • Another compatible VO client is TAPHandle. In the top part of the website, you have an input field in which, as with TOPCAT, you can either search in the registry or write the above URL to connect to this TAP service.
Execution modes

TAP comes with two execution modes: synchronous and asynchronous. In the first mode, you interact like with the Cone Search service: you send parameters (here: an ADQL query) and you get the result file in response.

In asynchronous mode, the interaction with the service is....well, asynchronous. Basically, you send your query but instead of getting the result file, you are redirected toward a description of the corresponding query execution job on the service side. Among all information provided in this description document, you will find a job ID, an execution state/phase (e.g. EXECUTING, COMPLETED or ERROR) and a link toward the result whenever the query has finished successfully. Once your query sent and the job description got, you can come again to the same description document (using the URL on which you have been redirected) and get the result whenever you want before the automatic job destruction.

In brief, the asynchronous mode aims to be used for long queries whereas the synchronous mode is more limited in term of duration. Besides, if the service is temporarily too busy, it may reject queries in synchronous mode while in asynchronous mode, queries will be just queued for a later execution.

Anyway, do not bother too much with this execution modes and particularly with how the asynchronous one is working. Compatible VO clients or the web form provided on this website are there to hide to you any kind of complexity

TAP in command line?
# SYNCHRONOUS QUERY - with WGET
wget -O myResultFile.vot \
--post-data="REQUEST=doQuery&LANG=ADQL&QUERY=SELECT TOP 10 * FROM gaia_source" \
'http://gaia.ari.uni-heidelberg.de/tap/sync'

# Same example (but output in CSV) with CURL
curl -o myResultFile.csv \
-d "REQUEST=doQuery&LANG=ADQL&FORMAT=CSV" --data-urlencode "QUERY=SELECT TOP 10 * FROM gaia_source" \
'http://gaia.ari.uni-heidelberg.de/tap/sync'

############################################

# ASYNCHRONOUS QUERY - with WGET
# 1. start the query execution and get the job description URL
joburl=`wget -O jobDescription.xml \
--post-data="REQUEST=doQuery&LANG=ADQL&FORMAT=votable/td&PHASE=RUN&QUERY=SELECT TO 10 * FROM gaia_source" \
'http://gaia.ari.uni-heidelberg.de/tap/async' \
&& grep "<jobId>.*</jobId>" jobDescription.xml | sed "s/.*>\(.*\)<.*/http:\/\/gaia.ari.uni-heidelberg.de\/tap\/async\/\1/"`

# 2. command to know when the job is finished:
wget -O jobPhase.xml "${joburl}/phase" && more jobPhase.xml

# 3. when the returned string is COMPLETED, ERROR or ABORTED,
the job is finished.
# 3a. If COMPLETED, you can get the result with the following
wget -O myResultFile.vot "${joburl}/results/result"
# 3b. If ERROR, you can get the error details like this:
wget -O myErrorFile.vot "${joburl}/error"

How to write an ADQL query?

The best and most accurate documentation you can find about ADQL is of course the standard document of the IVOA: http://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf.

However, some services or people of the VO are already providing cheat sheets and tutorials that may help you to discover this language. Here are some of them:

Tutorials Cheat sheets
A dedicated ADQL help may come later on this page or directly on the TAP web-form!

How to perform a crossmatch with my own table(s)?

A TAP service may come with an UPLOAD feature allowing a user to upload its own table (obligatory a VOTable file) on the service in order to perform crossmatch with the TAP tables. An additional HTTP parameter specifying the table file(s) to upload and its (their) name must be provided. Once done, you just have to write in the ADQL query TAP_UPLOAD.{myTable} (where {myTable} is the name you specified when uploading your table) to make a reference to it.

This TAP service supports this feature. Though the web interface is not yet adapted for such request, you can use it with any TAP client like TOPCAT.

In this latter, any table visible in the table list can be uploaded on the target TAP service. You do not have anything else to do than just mentionning the table as TAP_UPLOAD.t{tableIndex}. This {tableIndex} prefixes the table name in the table list:

Here is an example of ADQL query making a positionnal crossmatch between gaia_source and this table indexed 2 in TOPCAT:

SELECT target.*, src.*
FROM TAP_UPLOAD.t2 src JOIN gaia_source target
  ON CONTAINS(POINT('', target.ra, target.dec), CIRCLE('', src.ra, src.dec, 1./60)) = 1
Table order

In order to faster your query, always put the smallest table in first position in a JOIN close.

Similarly, you should really pay attention on the JOIN condition ; in particular, in case of a positional crossmatch, the POINT is always the sources you want to get, and the CIRCLE is always the input positions.

Circle radius

An expression like 5/60 will always return an integer (so, here: 0). In order to avoid this undesired behavior, merely add a dot after one of the numbers. For instance: 5./60, 5/60. or 5./60..

Which output formats are available?

The protocol TAP defines VOTable as the mandatory format, but allows services to propose more. In order to let the clients/users know the possible output formats, each TAP service must declare them in an endpoint named capabilities. There, you will also find all other kinds of information about the TAP service: output formats, limits and additional ADQL functions.

TAP Capabilities

Why can I not download the whole Gaia catalog with ADQL?

Considering the volume of some catalogs (or of some special query result), this TAP service is limiting the result files to a set number of rows. This limit is declared following the TAP protocol on the standard endpoint capabilities:

TAP Capabilities

When the result of a query is truncated, an OVERFLOW flag is appended to the VOTable document.

Why has my query been automatically aborted?

This situation happens when the user sends an ABORT or DELETE request on the job, or when the maximum execution duration or the destruction time of the job has been reached. In this last case, you should not be able to see the job anymore.

Both of the limits mentionned above are declared following the TAP protocol on the standard endpoint capabilities:

TAP Capabilities

How to get ADQL examples in TOPCAT?

A TAP service may provide some examples on a web-page usually located at {TAP_root-URL}/examples. Though it is a web-page, and so in HTML, it is using a particular syntax allowing programs to parse it and retrieve the list of examples, their name, description and ADQL query. Thus, a smart tool can display this list to TAP users so that they can choose to execute one of them.

TOPCAT is one of these smart VO programs. You can see below how to display Service-Provided examples, when a TAP Service provides some.

More examples?

Even if the TAP Service does not provide any example, TOPCAT generates some for the selected table. These examples are listed in different sub-lists in function of what they do:

Basic
Examples to get all the columns or the principal ones for the first rows. But also to perform a BOX or CONE search.
UPLOAD
To perform a simple crossmatch between the selected TAP table and the table selected in the TOPCAT tables list. A table must have been loaded in TOPCAT so that this sub-list appears.
TAP_SCHEMA
Examples of manipulation of the TAP metadata (i.e. list and description of published schema, tables and columns).

Slides

I have question(s)/problem(s)/suggestion(s). Who can I contact?

For any questions/comments/suggestions you can send an email to Grégory Mantelet - gmantele@ari.uni-heidelberg.de

If you have experienced a problem with a service or the website, please provide enough information in order to help me reproducing it.