Level-of-detail catalogs
This section discusses the level-of-detail (LOD) datasets (from Gaia DR2 on) where not all data fits into the CPU memory (RAM) and especially the GPU memory (VRAM).
In order to solve the issue, Gaia Sky implements a LOD structure based on the spatial distribution of stars into an octree. The culling of the octree is determined using a draw distance setting, called
Balancing the loading of data depends on several parameters:
The maximum java heap memory (set to 4 Gb by default), let’s call it maxheap.
The available graphics memory (VRAM, video ram). It depends on your graphics card. Let’s call it VRAM.
The draw distance setting ,
.The maximum number of loaded stars,
. This is in the configuration file ($GS_CONFIG/global.properties
) under the keyscene.octree.maxstars
. The default value balances the maximum heap memory space and the default data set.
So basically, a low OutOfMemoryError
. To mitigate that, one can also increase the maximum heap space.
Finally, there is the maximum number of loaded stars,
Draw distance, minimum visual solid angle for octants to be rendered
Maximum number of stars in memory at a given time
Data format
Gaia catalogs contain typically hundreds of millions of stars. They are too large to fit in your neighbor’s consumer GPU. In order to be able to represent such catalogs in real time, Gaia Sky implements a level-of-detail algorithm backed by an octree. The data format of level-of-detail catalogs is a custom binary format to make it more compact and fast to load.
There are two types of files: the metadata (metadata.bin
) and the particle files (particles_xxxxxxx.bin
). The metadata file contains all the nodes of the octree (called octants). Each octant points to a particle file, containing its particles. The number in the particle file name is the identifier of the octant.
The distance units are internal units.
Metadata file
The metadata reader is implemented here. The metadata file contains the information of the octants of the octree. The metadata format has currently two possible versions, 0 and 1, which are automatically detected by Gaia Sky.
Version 0
Version 0 (legacy) does not contain its version number in the file itself. Instead, if the first four bytes interpreted as an integer are zero or positive, version 0 is assumed. The format is the following.
1 single-precision integer (32-bit) – number of octants in the file
- For each octant:
1 single-precision integer (32-bit) – Page ID - ID of current octant
3 single-precision float (32-bit * 3) – X, Y, Z cartesian coordinates in internal units
1 single-precision float (32-bit) – Octant half-size in X
1 single-precision float (32-bit) – Octant half-size in Y
1 single-precision float (32-bit) – Octant half-size in Z
8 single-precision integer (32-bit * 8) – IDs of the 8 children (-1 if no child)
1 single-precision integer (32-bit) – Level of octant (depth)
1 single-precision integer (32-bit) – Cumulative number of stars in this node and its descendants
1 single-precision integer (32-bit) – Number of stars in this node
1 single-precision integer (32-bit) – Number of children nodes
Version 1
Version 1 was introduced in Gaia Sky 3.0.4
, and starts with a negative integer in the first four bytes, typically -1. Then comes the version number. The main difference with the legacy version is that the page IDs are encoded with a 64-bit integer instead of 32.
1 single-precision integer (32-bit) – special token number
-1
, signaling the presence of a version number1 single-precision integer (32-bit) – version number (
1
in this case)1 single-precision integer (32-bit) – number of octants in the file
- For each octant:
1 double-precision integer (64-bit) – Page ID - ID of current octant
3 single-precision float (32-bit * 3) – X, Y, Z cartesian coordinates in internal units
1 single-precision float (32-bit) – Octant half-size in X
1 single-precision float (32-bit) – Octant half-size in Y
1 single-precision float (32-bit) – Octant half-size in Z
8 double-precision integer (64-bit * 8) – IDs of the 8 children (-1 if no child)
1 single-precision integer (32-bit) – Level of octant (depth)
1 single-precision integer (32-bit) – Cumulative number of stars in this node and its descendants
1 single-precision integer (32-bit) – Number of stars in this node
1 single-precision integer (32-bit) – Number of children nodes
Particle files
The binary readers/writers are implemented in the following files:
Version 0 was used in DR2, version 1 was used mainly in the first batch of eDR3. Version 2 is used in the second batch of eDR3 and future DRs. Versions 0 and 1 are not annotated, so they are detected using the file name. Starting from version 2, the version number is in the file header, using a special token (negative integer).
Version 0
The version 0 is specified below. It contains a header with the number of stars and then a bunch of data for each star. It contains a 3-integer set which is the Tycho identifier, mainly for compatibility with TGAS.
1 single-precision integer (32-bit) – number of stars in the file
- For each star:
3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
3 double-precision floats (64-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
3 double-precision floats (64-bit * 3) – mualpha, mudelta, radvel - proper motion
4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
3 single-precision integer (32-bit * 3) – Tycho identifiers
1 double-precision integer (64-bit) – Gaia SourceID
1 single-precision integer (32-bit) – namelen -> Length of name
namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16
Version 1
Version 1 is the same as version 0 but without the Tycho identifiers.
1 single-precision integer (32-bit) – number of stars in the file
- For each star:
3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
3 double-precision floats (64-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
3 double-precision floats (64-bit * 3) – mualpha, mudelta, radvel - proper motion
4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
1 double-precision integer (64-bit) – Gaia SourceID
1 single-precision integer (32-bit) – namelen -> Length of name
namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16
Version 2
This version is much more compact, and it uses smaller data types when possible. The header contains a token integer (-1) marking the following version number, plus the number of stars.
1 single-precision integer (32-bit) – special token number
-1
, signaling the presence of a version number1 single-precision integer (32-bit) – version number (
2
in this case)1 single-precision integer (32-bit) – number of stars in the file
- For each star:
3 double-precision floats (64-bit * 3) – X, Y, Z cartesian coordinates in internal units
3 single-precision floats (32-bit * 3) – Vx, Vy, Vz - cartesian velocity vector in internal units per year
3 single-precision floats (32-bit * 3) – mualpha, mudelta, radvel - proper motion
4 single-precision floats (32-bit * 4) – appmag, absmag, color, size - Magnitudes, colors (encoded), and size (a derived quantity, for rendering)
1 single-precision integer (32-bit) – HIP number (if any, otherwise negative)
1 double-precision integer (64-bit) – Gaia SourceID
1 single-precision integer (32-bit) – namelen -> Length of name
namelen * char (16-bit * namelen) – Characters of the star name, where each character is encoded with UTF-16
The RGB color of stars uses 8 bits per channel in RGBA, and is encoded into a single float using the libgdx Color class.
Some discussion on memory issues and the streaming loader can be found here.