Level-of-detail catalogs

This section discusses the level-of-detail (LOD) datasets (from Gaia DR2 on) where not all data fits into the CPU memory (RAM) and especially the GPU memory (VRAM).

In order to solve the issue, Gaia Sky implements a LOD structure based on the spatial distribution of stars into an octree. The culling of the octree is determined using a draw distance setting, called ${\theta}$ . ${\theta}$ is actually the minimum visual solid angle (as seen from the camera) of an octant for it to be observed and its stars to be rendered. Larger ${\theta}$ values lead to less octants being observed, and smaller ${\theta}$ values lead to more octants being observed.

Balancing the loading of data depends on several parameters:

The maximum java heap memory (set to 4 Gb by default), let’s call it maxheap.
The available graphics memory (VRAM, video ram). It depends on your graphics card. Let’s call it VRAM.
The draw distance setting , ${\theta}$ .
The maximum number of loaded stars, ${\nu}$ . This is in the configuration file ($GS_CONFIG/global.properties) under the key scene.octree.maxstars. The default value balances the maximum heap memory space and the default data set.

So basically, a low ${\theta}$ (below 50-60 degrees) means lots of observed octants and lots of stars. Setting ${\theta}$ very low causes Gaia Sky to try to load lots of data, eventually overflowing the heap space and creating an OutOfMemoryError. To mitigate that, one can also increase the maximum heap space.

Finally, there is the maximum number of loaded stars, ${\nu}$ . This is a number is set according to the maxheap setting. When the number of loaded stars is larger than ${\theta}$ , the loaded octants that have been unobserved for the longest time will be unloaded and their memory structures will be freed (both in GPU and CPU). This poses a problem if the draw distance setting is set so that the observed octants at a single moment contain more stars than than ${\theta}$ . That is why high values for ${\theta}$ are recommended. Usually, values between 60 and 80 are fine, depending on the dataset and the machine.

${\theta}$: Draw distance, minimum visual solid angle for octants to be rendered
${\nu}$: Maximum number of stars in memory at a given time

Data format

Gaia catalogs contain typically hundreds of millions of stars. They are too large to fit in your neighbor’s consumer GPU. In order to be able to represent such catalogs in real time, Gaia Sky implements a level-of-detail algorithm backed by an octree. The data format of level-of-detail catalogs is a custom binary format to make it more compact and fast to load.

There are two types of files: the metadata (metadata.bin) and the particle files (particles_xxxxxxx.bin). The metadata file contains all the nodes of the octree (called octants). Each octant points to a particle file, containing its particles. The number in the particle file name is the identifier of the octant.

The distance units are internal units.

Metadata file

The metadata reader is implemented here. The metadata file contains the information of the octants of the octree. The metadata format has currently two possible versions, 0 and 1, which are automatically detected by Gaia Sky.

Version 0

Version 0 (legacy) does not contain its version number in the file itself. Instead, if the first four bytes interpreted as an integer are zero or positive, version 0 is assumed. The format is the following.

1 single-precision integer (32-bit) – number of octants in the file
For each octant:
- 1 single-precision integer (32-bit) – Page ID - ID of current octant
- 3 single-precision float (32-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 1 single-precision float (32-bit) – Octant half-size in X
- 1 single-precision float (32-bit) – Octant half-size in Y
- 1 single-precision float (32-bit) – Octant half-size in Z
- 8 single-precision integer (32-bit * 8) – IDs of the 8 children (-1 if no child)
- 1 single-precision integer (32-bit) – Level of octant (depth)
- 1 single-precision integer (32-bit) – Cumulative number of stars in this node and its descendants
- 1 single-precision integer (32-bit) – Number of stars in this node
- 1 single-precision integer (32-bit) – Number of children nodes

Version 1

Since 3.0.4

Version 1 was introduced in Gaia Sky 3.0.4, and starts with a negative integer in the first four bytes, typically -1. Then comes the version number. The main difference with the legacy version is that the page IDs are encoded with a 64-bit integer instead of 32.

1 single-precision integer (32-bit) – special token number -1, signaling the presence of a version number
1 single-precision integer (32-bit) – version number (1 in this case)
1 single-precision integer (32-bit) – number of octants in the file
For each octant:
- 1 double-precision integer (64-bit) – Page ID - ID of current octant
- 3 single-precision float (32-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 1 single-precision float (32-bit) – Octant half-size in X
- 1 single-precision float (32-bit) – Octant half-size in Y
- 1 single-precision float (32-bit) – Octant half-size in Z
- 8 double-precision integer (64-bit * 8) – IDs of the 8 children (-1 if no child)
- 1 single-precision integer (32-bit) – Level of octant (depth)
- 1 single-precision integer (32-bit) – Cumulative number of stars in this node and its descendants
- 1 single-precision integer (32-bit) – Number of stars in this node
- 1 single-precision integer (32-bit) – Number of children nodes

Particle files

The binary readers/writers are implemented in the following files:

Version 0 was used in DR2, version 1 was used mainly in the first batch of eDR3. Version 2 is used in the second batch of eDR3 and future DRs. Versions 0 and 1 are not annotated, so they are detected using the file name. Starting from version 2, the version number is in the file header, using a special token (negative integer).

Version 0

The version 0 is specified below. It contains a header with the number of stars and then a bunch of data for each star. It contains a 3-integer set which is the Tycho identifier, mainly for compatibility with TGAS.

1 single-precision integer (32-bit) – number of stars in the file
For each star:
- 3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 3 double-precision floats (64-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
- 3 double-precision floats (64-bit * 3) – mualpha, mudelta, radvel - proper motion
- 4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
- 1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
- 3 single-precision integer (32-bit * 3) – Tycho identifiers
- 1 double-precision integer (64-bit) – Gaia SourceID
- 1 single-precision integer (32-bit) – namelen -> Length of name
- namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16

Version 1

Version 1 is the same as version 0 but without the Tycho identifiers.

1 single-precision integer (32-bit) – number of stars in the file
For each star:
- 3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 3 double-precision floats (64-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
- 3 double-precision floats (64-bit * 3) – mualpha, mudelta, radvel - proper motion
- 4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
- 1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
- 1 double-precision integer (64-bit) – Gaia SourceID
- 1 single-precision integer (32-bit) – namelen -> Length of name
- namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16

Since 3.0.2

Version 2

This version is much more compact, and it uses smaller data types when possible. The header contains a token integer (-1) marking the following version number, plus the number of stars.

1 single-precision integer (32-bit) – special token number -1, signaling the presence of a version number
1 single-precision integer (32-bit) – version number (2 in this case)
1 single-precision integer (32-bit) – number of stars in the file
For each star:
- 3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
- 3 single-precision floats (32-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
- 3 single-precision floats (32-bit * 3) – mualpha, mudelta, radvel - proper motion
- 4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
- 1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
- 1 double-precision integer (64-bit) – Gaia SourceID
- 1 single-precision integer (32-bit) – namelen -> Length of name
- namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16

The RGB color of stars uses 8 bits per channel in RGBA, and is encoded into a single float using the libgdx Color class.

Some discussion on memory issues and the streaming loader can be found here.