Star catalog formats

Star catalogs can be loaded from well-known formats (VOTable, CSV, etc.) using the STIL data loader, or they can use a binary format tailor-made for Gaia Sky. In general, the binary format loads much faster and is more compact. That’s why we use it for our big level-of-detail star catalogs based on Gaia data.

This section discusses the level-of-detail (LOD) datasets (from Gaia DR2 on) where not all data fits into the CPU memory (RAM) and especially the GPU memory (VRAM).

In order to solve the issue, Gaia Sky implements a LOD structure based on the spatial distribution of stars into an octree. The culling of the octree is determined using a draw distance setting, called ${\theta}$ . ${\theta}$ is actually the minimum visual solid angle (as seen from the camera) of an octant for it to be observed and its stars to be rendered. Larger ${\theta}$ values lead to less octants being observed, and smaller ${\theta}$ values lead to more octants being observed.

Balancing the loading of data depends on several parameters:

The maximum java heap memory (set to 4 Gb by default), let’s call it maxheap.
The available graphics memory (VRAM, video ram). It depends on your graphics card. Let’s call it VRAM.
The draw distance setting , ${\theta}$ .
The maximum number of loaded stars, ${\nu}$ . This is in the configuration file ($GS_CONFIG/config.yaml) under the key scene::octree::maxstars. The default value balances the maximum heap memory space and the default data set.

So basically, a low ${\theta}$ (below 50-60 degrees) means lots of observed octants and lots of stars. Setting ${\theta}$ very low causes Gaia Sky to try to load lots of data, eventually overflowing the heap space and creating an OutOfMemoryError. To mitigate that, one can also increase the maximum heap space.

Finally, there is the maximum number of loaded stars, ${\nu}$ . This is a number is set according to the maxheap setting. When the number of loaded stars is larger than ${\theta}$ , the loaded octants that have been unobserved for the longest time will be unloaded and their memory structures will be freed (both in GPU and CPU). This poses a problem if the draw distance setting is set so that the observed octants at a single moment contain more stars than than ${\theta}$ . That is why high values for ${\theta}$ are recommended. Usually, values between 60 and 80 are fine, depending on the dataset and the machine.

${\theta}$: Draw distance, minimum visual solid angle for octants to be rendered
${\nu}$: Maximum number of stars in memory at a given time

Binary format specification

Gaia catalogs contain typically hundreds of millions of stars. They are too large to fit in your neighbor’s consumer GPU. In order to be able to represent such catalogs in real time, Gaia Sky implements a level-of-detail algorithm backed by an octree. The data format of all level-of-detail catalogs is a custom binary format to make it more compact and fast to load. This binary format can, however, also be used for smaller star catalogs. This section contains its specification.

There are two types of files: the metadata (metadata.bin) and the particle files (particles_xxxxxxx.bin). The metadata file contains all the nodes of the octree (called octants). Each octant points to a particle file, containing its particles. The number in the particle file name is the identifier of the octant. Additionally, the particle files can also be used for standalone smaller star catalogs.

The distance units are internal units.

Metadata file

The metadata reader is implemented here. The metadata file contains the information of the octants of the octree. The metadata format has currently two possible versions, 0 and 1, which are automatically detected by Gaia Sky.

Version 0

Version 0 (legacy) does not contain its version number in the file itself. Instead, if the first four bytes interpreted as an integer are zero or positive, version 0 is assumed. The format is the following.

1 single-precision integer (32-bit) – number of octants in the file
For each octant:
- 1 single-precision integer (32-bit) – Page ID - ID of current octant
- 3 single-precision float (32-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 1 single-precision float (32-bit) – Octant half-size in X
- 1 single-precision float (32-bit) – Octant half-size in Y
- 1 single-precision float (32-bit) – Octant half-size in Z
- 8 single-precision integer (32-bit * 8) – IDs of the 8 children (-1 if no child)
- 1 single-precision integer (32-bit) – Level of octant (depth)
- 1 single-precision integer (32-bit) – Cumulative number of stars in this node and its descendants
- 1 single-precision integer (32-bit) – Number of stars in this node
- 1 single-precision integer (32-bit) – Number of children nodes

Version 1

Since 3.0.4

Version 1 was introduced in Gaia Sky 3.0.4, and starts with a negative integer in the first four bytes, typically -1. Then comes the version number. The main difference with the legacy version is that the page IDs are encoded with a 64-bit integer instead of 32.

1 single-precision integer (32-bit) – special token number -1, signaling the presence of a version number
1 single-precision integer (32-bit) – version number (1 in this case)
1 single-precision integer (32-bit) – number of octants in the file
For each octant:
- 1 double-precision integer (64-bit) – Page ID - ID of current octant
- 3 single-precision float (32-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 1 single-precision float (32-bit) – Octant half-size in X
- 1 single-precision float (32-bit) – Octant half-size in Y
- 1 single-precision float (32-bit) – Octant half-size in Z
- 8 double-precision integer (64-bit * 8) – IDs of the 8 children (-1 if no child)
- 1 single-precision integer (32-bit) – Level of octant (depth)
- 1 single-precision integer (32-bit) – Cumulative number of stars in this node and its descendants
- 1 single-precision integer (32-bit) – Number of stars in this node
- 1 single-precision integer (32-bit) – Number of children nodes

Star particle files

A particle file contains the information of a number of stars. These can be the stars belonging to a particular octant in a LOD octree, or all the stars in a particular star catalog.

The class in charge of loading and writing binary star particle files is the BinaryDataProvider.

The binary readers/writers are implemented in the following files:

Version 0 was used in DR2, version 1 was used mainly in the first batch of eDR3. Version 2 is used in the second batch of eDR3 and future DRs. Versions 0 and 1 are not annotated, so they are detected using the file name. Starting from version 2, the version number is in the file header, using a special token (negative integer).

Version 0

The version 0 is specified below. It contains a header with the number of stars and then a bunch of data for each star. It contains a 3-integer set which is the Tycho identifier, mainly for compatibility with TGAS.

1 single-precision integer (32-bit) – number of stars in the file
For each star:
- 3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 3 double-precision floats (64-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
- 3 double-precision floats (64-bit * 3) – mualpha, mudelta, radvel - proper motion
- 4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
- 1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
- 3 single-precision integer (32-bit * 3) – Tycho identifiers
- 1 double-precision integer (64-bit) – Gaia SourceID
- 1 single-precision integer (32-bit) – namelen -> Length of name
- namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16

Version 1

Version 1 is the same as version 0 but without the Tycho identifiers.

1 single-precision integer (32-bit) – number of stars in the file
For each star:
- 3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 3 double-precision floats (64-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
- 3 double-precision floats (64-bit * 3) – mualpha, mudelta, radvel - proper motion
- 4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
- 1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
- 1 double-precision integer (64-bit) – Gaia SourceID
- 1 single-precision integer (32-bit) – namelen -> Length of name
- namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16

Since 3.0.2

Version 2

This version is much more compact, and it uses smaller data types when possible. The header contains a token integer (-1) marking the following version number, plus the number of stars.

1 single-precision integer (32-bit) – special token number -1, signaling the presence of a version number
1 single-precision integer (32-bit) – version number (2 in this case)
1 single-precision integer (32-bit) – number of stars in the file
For each star:
- 3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 3 single-precision floats (32-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
- 3 single-precision floats (32-bit * 3) – mualpha, mudelta, radvel - proper motion
- 4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
- 1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
- 1 double-precision integer (64-bit) – Gaia SourceID
- 1 single-precision integer (32-bit) – namelen -> Length of name
- namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16

The RGB color of stars uses 8 bits per channel in RGBA, and is encoded into a single float using the libgdx Color class.

Some discussion on memory issues and the streaming loader can be found here.

LOD catalog processing

All LOD catalogs are based on one of the Gaia data releases (DR2, DR3, etc.), and they also include the brighter stars from the Hipparcos catalog. The official Gaia-Hipparcos crossmatch is used to identify stars that are contained in both catalogs. In this case, the parallax is taken from the source that has the smaller parallax error. The rest of de data is taken from the Gaia catalog, but some attributes are merged (for instance, the final star contains both the HIP number and the Gaia source id).

The LOD catalogs are generated using a program written in Rust, called gaiasky-catgen. The source code can be found in this repository. In the LOD generation process, each star is processed individually. The catalog is filtered according to the input parameters, and some corrections are applied to star attributes.

Distances

In most catalogs, distances are derived from parallaxes, using the formula

$d[pc] = 1000 / pllx [mas].$

All parallaxes are zero-point corrected as instructed in the official DR documentation before being converted to distances. Sometimes, some parallaxes are negative. In this case, Gaia Sky opts for keeping the star and assigning it a default parallax of 0.04 mas, which corresponds to 25 kpc instead of discarding it.

However, some catalogs use distances determined elsewhere by different methods and injected into the generation process as additional columns. This is the case for the geometric (Bayesian) distances and the photometric distances catalogs.

Magnitude/color corrections

Extinction and reddening factors are applied to star magnitudes and colors, respectively.

When the extinction value $A_g$ is present in the catalog or in an additional column, it is applied directly to the magnitude. Otherwise, we default to the following analytical extinction,

$A_g = \text{min}(3.2, \frac{150}{|\text{sin}(b)|} * 5.9e-4),$

where $b$ is the galactic longitude of the star.

Similarly, we apply the reddening value $E_{BP-RP}$ when it is in the catalog or in an additional column. Otherwise, we fall back to the following analytical determination, based on the extinction:

$E_{BP-RP} = \text{min}(1.6, A_g * 2.9e-4).$