ARI's Gaia Services

Which Gaia data are currently available?

The Gaia data will be provided little by little as different catalogue releases. Below, you can see how many releases are scheduled and what is their content.

Where can I find statistics about table and columns for every available catalogue?

Any available table or column is described in the page  Data & Statistics. If you can not find a column or table, it means it is not available.

Several statistics about a table and its columns are computed and provided by the Gaia Team of ARI, and are available on the same page.

Table - Density map

For a table, the only "statistical" product is a density map. You can see a small snapshot of it on the left of a table name.

Once the description of a table is extended, a button appears. By clicking on it, you will see a full resolution density map with a colormap's histogram. On the dialog window, you will also be able to download the image either as an image file or as an Healpix map, loadable in Aladin for a more interactive navigation. (see How the sky maps are computed? and How to send and display a sky map in Aladin? for more details)

Column

The description of table always lists its columns. Besides its name description, unit and datatype, statistics are provided for every individual numerical columns. All available columns statistics can easily be displayed by clicking to the appropriate button on the last column.

Column - Basic statistics

Basic statistics (min, max, mean, standard deviation, quartiles) are provided along with a sum-up image. Below is a description of this figure.

Column - Histogram

The histogram is computed with all non-null values in the whole table for the specified column.

By clicking on the histogram image, you can display it at an higher resolution.

An image of it is displayed (and can be downloaded) but you can also download it as a CSV table. Thanks to this CSV, you could reproduce or use this histogram in tools like TOPCAT. (see How to send and display a column's histogram in TOPCAT?)

Column - Sky-map

A sky map represents the mean value (non-NULL values excluded) of the specified column in every Healpix cells in the sky.

By clicking on the sky map image, you can display it at an higher resolution.

The provided sky map image is divided into two parts: the map itself on the left and a legend on the right. This latter indicates you the mapping between the pixel values and the pixel colors. The histogram on the right of the color map shows the ratio of displayed pixels for each range of colors.

As for the histogram, the sky map can be downloaded as an image (PNG) but also as a table (FITS ; same format as a table's density map described above). Thanks to the FITS file, you can get the raw value of each pixel and/or reproduce the map with different settings (color map, normalisation, projection, ...). (see How the sky maps are computed? and How to send and display a sky map in Aladin? for more details)

What is a Single Source Search service?

This Web-Service let's you query the Gaia catalogue in order to get only one source. You can either provide a Gaia source ID, a pair of coordinates or an object name (as known by Simbad or NED). It is also possible to provide a CSV-like file listing sources to fetch.

When an object name or a position is given, only the closest Gaia source is returned. However, as described below, it is possible to get more Gaia sources close to the specified position with a special parameter.

This service always returns all information available in the Gaia catalogue. It is not possible to choose the columns you want to get.

Which tables are accessible in the Single Source Search service?

Only two tables can be targeted by this service: gaia_source (for each existing release) and tgas_source.

How to use the Single Source Search Service?

The Single Source Search Service is not a VO standard. It is a web service that can be access easily from a website, by command line or any other tool able to perform HTTP GET/POST requests.

The easiest way to use this service is through the ARI's Gaia webpage: https://gaia.ari.uni-heidelberg.de/singlesource.html. All you have to do is fill in the form and click on what you want to do:

• View: to display the result below on the same web-page. The closest source will be shown entirely as a table while the other candidates will be plotted on an Aladin Lite view.
• Download as...: to download the result in the format of your choice.
• Send to...: to send the result using SAMP to a VO client such as TOPCAT.

You may notice that in addition of all available information about a matching Gaia source, its angular distance from the target position is returned. This value is merely labeled distance and is expressed in degrees. You can see it either between brackets in the title of the left panel for the closest source or in the measurement table of the Aladin Lite view when you click on another Gaia source candidate. Of course, this value is also available when you download/send the query result.

As said above, you can simply query this Web-Service by command line or from a standalone application by sending HTTP requests with appropriate parameters to https://gaia.ari.uni-heidelberg.de/singlesource/search for Gaia or https://gaia.ari.uni-heidelberg.de/singlesource/tgas for TGAS.

Both HTTP GET and POST methods are supported with exactly the same parameters. All the allowed parameters are described just below:

# Research for the Gaia source 381540121205828224
wget -O mySource.csv 'https://gaia.ari.uni-heidelberg.de/singlesource/search?id=381540121205828224'

# Research of the closest TGAS source around rr lyrae (result in JSON)
wget -O mySource.json 'https://gaia.ari.uni-heidelberg.de/singlesource/tgas?obj=rr+lyrae&f=json'

# Research of the closest Gaia source around a specific position (result in VOTable with more candidates)
wget -O mySources.vot 'https://gaia.ari.uni-heidelberg.de/singlesource/search?ra=12.3&dec=88.26&f=votable&v=true'

# Research by file (result in VOTable)
curl -F "positions=@pos.csv" -F "f=votable" 'https://gaia.ari.uni-heidelberg.de/singlesource/search'

Which output formats are available?

Three output formats are proposed:

• CSV
• JSON
• VOTable

You can choose the format you want using the parameter f (allowed values: csv, json and votable). If no such parameter is set or if the given format name is unknown/unsupported, the default format will be CSV.

Only the VOTable format provides metadata (description, unit, UCD) for each output column.

How to get a web link toward a Gaia source?

By appending to the URL of the web-form of the Single Source Service the hash tag #id= (or #tgas_id= for TGAS), the normal web-form will be automatically filled with the given source ID and the request will be sent. Consequently, the page displays quite immediately all information about a Gaia (or TGAS) source.

For instance: https://gaia.ari.uni-heidelberg.de/singlesource.html#id=2125982595039626368.

What is a Cone Search service?

A Cone Search service (originally called Simple Cone Search service) is a web service following the standard defined by the IVOA. A such service is able to execute cone search requests (i.e. research of sources around a given position in the sky and with a given radius) on a specified catalogue.
See the IVOA standard

This service has a fixed behavior and standard parameters. Thus, whoever is providing a such service in the world, a Cone Search compatible client is already able to use it

Which tables are accessible in the Cone Search service?

Only two tables can be targeted by this service: gaia_source (for each data release) and tgas_source.

How to use the Cone Search service?

• This specific Cone Search service comes with a web interface available at this page: Cone Search.
• Besides, any Cone Search service is accessible using a Cone Search client. A well-known client for this purpose is TOPCAT
  To use it, click on the menu item VOCone Search. In the opened window, you can either search in the registry for this service or write/copy-paste the following URL in the bottom part: https://gaia.ari.uni-heidelberg.de/cone/search for Gaia or https://gaia.ari.uni-heidelberg.de/cone/tgas for TGAS.
• You can also use PyVO if you are working with Python scripts.

Minimum three parameters must be provided:

• RA: right ascension in degrees between [0;360] (ICRS,J2015)
• DEC: declination in degrees between [-90;+90] (ICRS,J2015)
• SR: radius of the cone in degrees between [0;180]

An additional parameter is possible: VERB. It lets you choose the amount of columns you want to get. Only three values are possible: 1 (minimum), 2 (normal) and 3 (maximum). If none is specified by the user, the default value is 2. The real meaning of these values depends in function of the service/catalogue. In the case of this service: 1 is for [source_id, ra, dec, parallax, pmra, pmdec, phot_g_mean_mag], 2 is to get the same columns + their errors, and 3 is to get all available columns.

# Cone search around (12.3, 88.26) with a radius of 1°
wget -O myResultFile.vot 'https://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1'

# Same example with CURL
curl -o myResultFile.vot 'https://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1'

# Same example but with ALL available columns
wget -O myResultFile.vot 'https://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1&VERB=3'
# or
curl -o myResultFile.vot 'https://gaia.ari.uni-heidelberg.de/cone/search?RA=12.3&DEC=88.26&SR=1&VERB=3'

Which output formats are available?

Only one output format is possible in a Simple Cone Search Service: VOTable.

Why are my results never bigger than 10,000,000 rows?

Considering the size of the whole Gaia catalogue, this Cone Search service is limiting the result files to 10 million rows. This limit may change (bigger or smaller) in the future.

If you are not really interested by all sources in the specified cone, you should use the TAP service in order to add more constraints on your research.

I get so many/few columns. How to get less/more?

By default, this Cone Search service returns the following columns:

• source_id
• ra
• ra_error
• dec
• dec_error
• parallax
• parallax_error
• pmra
• pmra_error
• pmdec
• pmdec_error
• phot_g_mean_mag

Thanks to the optional parameter VERB you can choose the amount of columns you want to get. Only three values are possible: 1 (minimum), 2 (normal) and 3 (maximum). If none is specified by the user, the default value is 2. If you use the value 1 you will get less columns (i.e. same columns without the errors). On the contrary, the value 3 will let you get all available columns.

Be careful when using VERB=3 that your search radius stays reasonable. Otherwise, the number of matching rows multipled by the number of all columns (≥ 50 for the Gaia catalogue) can result into a huge result file.

What is a TAP service?

A TAP (Table Access Protocol) service is a web service following the standard defined by the IVOA. A such service is able to any query expressed in ADQL on one or more tables.
See the IVOA standard

ADQL (Astronomical Data Query Language) is a query language derived from SQL, language broadly used by DataBase Management Systems. However, only SELECT-type queries are possible. In addition of the most common SQL functions and features, ADQL provides a way to select sources by specifying sky regions.
See the IVOA standard

This service, as the Cone Search service, has a fixed behavior and standard parameters. Thus, whoever is providing a such service in the world, a TAP compatible client is already able to use it

Which catalogues are accessible in the TAP service?

All catalogues and their columns available for TAP requests are listed on the page Data & Statistics. If you do not find (anymore) a column or table, it means it is not available (anymore).

How to use the TAP service?

• This specific TAP service comes with a web interface available at this page: TAP.
• Besides, any TAP service is accessible using a TAP client. A well-known client for this purpose is TOPCAT.
  To use it, click on the menu item VOTable Access Protocol (TAP) Query. In the opened window, you can either search in the registry for this service or write/copy-paste the following URL in the bottom part: https://gaia.ari.uni-heidelberg.de/tap.
• Another compatible VO client is TAPHandle. In the top part of the website, you have an input field in which, as with TOPCAT, you can either search in the registry or write the above URL to connect to this TAP service.
• You can also use PyVO if you are working with Python scripts.

TAP comes with two execution modes: synchronous and asynchronous. In the first mode, you interact like with the Cone Search service: you send parameters (here: an ADQL query) and you get the result file in response.

In asynchronous mode, the interaction with the service is... well, asynchronous. Basically, you send your query but instead of getting the result file, you are redirected toward a description of the corresponding query execution job on the service side. Among all information provided in this description document, you will find a job ID, an execution state/phase (e.g. EXECUTING, COMPLETED or ERROR) and a link toward the result whenever the query has finished successfully. Once your query sent and the job description got, you can come again to the same description document (using the URL on which you have been redirected) and get the result whenever you want before the automatic job destruction.

In brief, the asynchronous mode aims to be used for long queries whereas the synchronous mode is more limited in term of duration. Besides, if the service is temporarily too busy, it may reject queries in synchronous mode while in asynchronous mode, queries will be just queued for a later execution.

Anyway, do not bother too much with this execution modes and particularly with how the asynchronous one is working. Compatible VO clients or the web form provided on this website are there to hide to you any kind of complexity

# SYNCHRONOUS QUERY - with WGET
wget -O myResultFile.vot \
--post-data="REQUEST=doQuery&LANG=ADQL&QUERY=SELECT TOP 10 * FROM gaiadr3.gaia_source" \
'https://gaia.ari.uni-heidelberg.de/tap/sync'

# Same example (but output in CSV) with CURL
curl -o myResultFile.csv \
-d "REQUEST=doQuery&LANG=ADQL&FORMAT=CSV" --data-urlencode "QUERY=SELECT TOP 10 * FROM gaiadr3.gaia_source" \
'https://gaia.ari.uni-heidelberg.de/tap/sync'

############################################

# ASYNCHRONOUS QUERY - with WGET
# 1. start the query execution and get the job description URL
joburl=`wget -O jobDescription.xml \
--post-data="REQUEST=doQuery&LANG=ADQL&FORMAT=votable/td&PHASE=RUN&QUERY=SELECT TOP 10 * FROM gaiadr3.gaia_source" \
'https://gaia.ari.uni-heidelberg.de/tap/async' \
&& grep "<jobId>.*</jobId>" jobDescription.xml | sed "s/.*>\(.*\)<.*/https:\/\/gaia.ari.uni-heidelberg.de\/tap\/async\/\1/"`

# 2. command to know when the job is finished:
wget -O jobPhase.xml "${joburl}/phase" && more jobPhase.xml

# 3. when the returned string is COMPLETED, ERROR or ABORTED,
the job is finished.
# 3a. If COMPLETED, you can get the result with the following
wget -O myResultFile.vot "${joburl}/results/result"
# 3b. If ERROR, you can get the error details like this:
wget -O myErrorFile.vot "${joburl}/error"

How to write an ADQL query?

A simple ADQL query has the next structure:

SELECT [TOP <NUMBER>] (* / <COLUMNS>) FROM <TABLE> [WHERE <CONDITIONS>] [ORDER BY <COLUMN> ASC/DESC]

Where the parenthesis mean either/or and the brackets mean that they are optional, but important for data selection. As a breakdown:

• TOP <NUMBER>: Number of rows you want to retrieve
• <COLUMNS>: Comma separated list of columns
• *: Wildcard for the column list, meaning it would pick all. Thus, use with caution when querying big tables.
• FROM <TABLE> table where the information resides.
• WHERE <CONDITIONS> the conditions the data need to fill, with logical opeators (AND/OR) mixing relational operations (such as equality, bigger/lower than,...) between columns and values mixing. For instance: WHERE parallax_over_error < 0.1 AND ruwe < 1.4
• ORDER BY <COLUMN> ASC/DESC refers to the order of the rerieved data by one or various columns (comma-separated)

For example:

SELECT TOP 10 ra, dec, ruwe FROM gaiadr3.gaia_source WHERE ruwe < 1.4 ORDER BY ruwe ASC

Means: select the right ascension, declination and re-normalised unit weight error for the first ten sources of the gaiadr3.gaia_source where the ruwe is lower than 1.4 and order them in ascending order by that column.

But they can become quite complicated depending on your needs. As such, the best and most accurate documentation you can find about ADQL is of course the standard document of the IVOA: https://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf.

However, some services or people of the VO are already providing cheat sheets and tutorials that may help you to discover this language. Here are some of them:

• General short course by GAVO
• Same course but adapted to Gaia
• An Evernote introduction in 2 parts

• Provided by GAVO
• Provided by VizieR
• Provided by Simbad

How to perform a crossmatch with my own table(s)?

A TAP service may come with an UPLOAD feature allowing a user to upload its own table (obligatory a VOTable file) on the service in order to perform crossmatch with the TAP tables. An additional HTTP parameter specifying the table file(s) to upload and its (their) name must be provided. Once done, you just have to write in the ADQL query TAP_UPLOAD.{myTable} (where {myTable} is the name you specified when uploading your table) to make a reference to it.

This TAP service supports this feature. Though the web interface is not yet adapted for such request, you can use it with any TAP client like TOPCAT.

In this latter, any table visible in the table list can be uploaded on the target TAP service. You do not have anything else to do than just mentionning the table as TAP_UPLOAD.t{tableIndex}. This {tableIndex} prefixes the table name in the table list:

Here is an example of ADQL query making a positionnal crossmatch between gaia_source and this table indexed 2 in TOPCAT:

SELECT target.*, src.*
FROM TAP_UPLOAD.t2 src JOIN gaia_source target
  ON CONTAINS(POINT('', target.ra, target.dec), CIRCLE('', src.ra, src.dec, 1./60)) = 1

Which output formats are available?

The protocol TAP defines VOTable as the mandatory format, but allows services to propose more. In order to let the clients/users know the possible output formats, each TAP service must declare them in an endpoint named capabilities. There, you will also find all other kinds of information about the TAP service: output formats, limits and additional ADQL functions.

Why can I not download the whole Gaia catalogue with ADQL?

Considering the volume of some catalogues (or of some special query result), this TAP service is limiting the result files to a set number of rows. This limit is declared following the TAP protocol on the standard endpoint capabilities:

When the result of a query is truncated, an OVERFLOW flag is appended to the VOTable document.

Why has my query been automatically aborted?

This situation happens when the user sends an ABORT or DELETE request on the job, or when the maximum execution duration or the destruction time of the job has been reached. In this last case, you should not be able to see the job anymore.

Both of the limits mentionned above are declared following the TAP protocol on the standard endpoint capabilities:

How to get ADQL examples in TOPCAT?

A TAP service may provide some examples on a web-page usually located at {TAP_root-URL}/examples. Though it is a web-page, and so in HTML, it is using a particular syntax allowing programs to parse it and retrieve the list of examples, their name, description and ADQL query. Thus, a smart tool can display this list to TAP users so that they can choose to execute one of them.

TOPCAT is one of these smart VO programs. You can see below how to display Service-Provided examples, when a TAP Service provides some.