Data format

Gaia Sky needs to first load datasets in order to display data. Dataset files contain objects, which are organized by Gaia Sky into a scenegraph. A scenegraph is a tree that contains objects and orders them hierarchically depending on their geometrical and spatial relations.

Since 3.3.1

All datasets are partially or totally described in a JSON format. Each dataset lives in its own directory in the data location (referred to as $data/, see folders), and must contain a description in the file dataset.json. If a dataset does not have this file in its directory, it won’t be recognized by Gaia Sky.

Below is an example of the contents of the data location.

$data/
├─ default-data/
│  ├─ dataset.json
│  └─ ...
├─ catalog-hipparcos/
│  ├─ dataset.json
│  └─ ...
├─ catalog-whitedwarfs-dr2/
│  ├─ dataset.json
│  └─ ...
└─ .../

Where are the data files defined?

Gaia Sky implements a very flexible and open data loading mechanism. The data files to be loaded are defined in a couple of keys in the config.yaml configuration file, which is usually located in the $GS_CONFIG folder (see folders). The keys are the following (double-colon indicates nesting):

data::dataFiles – an array containing the list of enabled JSON data files. Each file should be a relative path from the data directory with the prefix $data/. For instance, the default dataset, containing the Solar System and some necessary objects for Gaia Sky to run, is specified in the array as $data/default-data/dataset.json.

$data/[dataset-name]/dataset.json example

Dataset descriptor files contain the metadata of a catalog (name, description, version, etc.) and a pointer to the actual data. Below is a made-up file file dataset.json which describes my super-awesome dataset.

{
"key" : "dataset-key-without-spaces",
"name": "Dataset name",
"version": 1,
"mingsversion" : 30106,
"type": "catalog-gaia",
"description": "The description here.\nCan contain line breaks.",
"releasenotes" : "- What changed since the last version?\n- List here."
"link": "https://arxiv.org/abs/1805.00425",
"check" : "dataset-descriptor.json"
"size": 368633,
"nobjects": 1365,
"check": "$data/my-dataset/dataset.json",
"files": [ "$data/my-dataset" ],
"data": [{
    "loader": "gaiasky.data.JsonLoader",
    "files": [ "$data/my-dataset/particles-particles.json" ]
}]
}

Notice that the data (data::files) points to another JSON file, which contains some additional info about how to load the data, and a pointer to the actual data files. Here it is:

{
"objects": [{
    "name": "My dataset",
    "position": [0.0, 0.0, 0.0],
    "ct": "Stars",
    "fadeout": [1.0e5, 0.5e8],
    "parent": "Universe",
    "impl": "gaiasky.scenegraph.StarGroup",
    "provider": "gaiasky.data.group.STILDataProvider",
    "datafile": "$data/my-dataset/catalog/dataset.vot"
    }]
}

As you can see, the STILDataProvider is the one in charge of loading the data form this dataset, which resides in a VOTable file, dataset.vot.

default-data/dataset.json example file

This is an example of what the default data pack contains. The dataset descriptor file loads different files using different loaders.

{
"data" : [
    {
        "loader": "gaiasky.data.JsonLoader",
        "files": [ "$data/default-data/planets-normal.json",
                    "$data/default-data/moons-normal.json",
                    "$data/default-data/satellites.json",
                    "$data/default-data/asteroids.json",
                    "$data/default-data/orbits_planet.json",
                    "$data/default-data/orbits_moon.json",
                    "$data/default-data/orbits_asteroid.json",
                    "$data/default-data/orbits_satellite.json",
                    "$data/default-data/extra-low.json",
                    "$data/default-data/locations.json",
                    "$data/default-data/locations_earth.json",
                    "$data/default-data/locations_moon.json"]
    },
    {
        "loader": "gaiasky.data.GeoJsonLoader",
        "files": [ "$data/default-data/countries/countries.geo.json" ]
    }]
}

The dataset.json file contains an array, "data", which is a list of pairs containing [loader: files] correspondences. Each "loader" contains the classes that will load the list of files under the corresponding "files" property. The main loader, the JsonLoader, expects JSON files as inputs. Each of these files must have an attribute called "objects", which is an array containing the metadata on the objects to load.

Data loaders

The files are sent to the Scene Graph JSON Loader, which iterates on each loader-files pair in each file, instantiates the loader and uses it to load the files. All loaders need to adhere to a contract, defined in the interface ISceneLoader (here). The loadData() method of each loader must return a list of objects, which is then added to a global list containing all the previously loaded files. At the end, we have a list with all the objects in the scene. This list is passed on to the Scene Graph instance, which constructs the scene graph tree structure which will contains the object model.

As we said, each loader will load a different kind of data; the JSONLoader (here) loads JSON files containing planets, satellites, orbits, star catalogs, etc. The STILDataProvider (here) loads VOTables, FITS, CSV and other files through the STIL library, GeoJsonLoader (here) loads geographic data, and so on.

Catalog formats

Catalogs refer to datasets which are essentially particle-based (stars, galaxies, etc.). There are several off-the-shelf options to get catalog data in various formats into Gaia Sky. The most important are VOTable, FITS and CSV. They are all handled by the STIL data provider.

Let’s see an example of the definition of one such catalog in the Oort cloud:

{
    "name" : "Oort cloud",
    "position" : [0.0, 0.0, 0.0],
    "color" : [0.9, 0.9, 0.9, 0.8],
    "size" : 2.0,
    "labelcolor" : [0.3, 0.6, 1.0, 1.0],
    "labelposition" : [0.0484814, 0.0, 0.0484814],
    "ct" : "Others",

    "fadein" : [0.0004, 0.004],
    "fadeout" : [0.1, 15.0],

    "profiledecay" : 1.0,

    "parent" : "Universe",
    "impl" : "gaiasky.scenegraph.ParticleGroup",

    "provider" : "gaiasky.data.PointDataProvider",
    "factor" :  149.597871,
    "datafile" : "$data/oort-cloud/oortcloud/oort_10000particles.dat"
}

Let’s go over the attributes:

Star catalogs

Star catalogs are special because, additionally to positional information, they contain extra properties such as proper motions, magnitudes, colors and more. All of these are important to be able to render stars faithfully.

The easiest way to load star catalogs is by loading them from VOTable files. Let’s see how these catalogs can be defined in Gaia Sky. For example, the new Hipparcos reduction:

{
  "key": "catalog-hipparcos",
  "name" : "Hipparcos (new reduction)",
  "version" : 4,
  "mingsversion" : 30301,
  "type" : "catalog-star",
  "description" : "Hipparcos new reduction (van Leeuwen, 2007) with curated star names.",
  "releasenotes" : "- Add type in catalog descriptor file.\n- Update to new data format.",
  "link" : "http://adsabs.harvard.edu/abs/2007ASSL..350.....V",
  "size" : 5433174,
  "nobjects" : 177955,
  "check": "$data/catalog-hipparcos/dataset.json",
  "files" : [ "$data/catalog-hipparcos" ],
  "data" : [
  {
    "loader": "gaiasky.data.JsonLoader",
    "files": [ "$data/catalog-hipparcos/particles-hip.json" ]
  }]
}

The file particles-hip.json contains a single object with the actual pointer to the VOTable data, and some additional metadata such as the color of labels, a description of the catalog or the data provider:

{
  "objects" : [
  {
    "name" : "Hipparcos (new red.)",
    "position" : [0.0, 0.0, 0.0],
    "color" : [1.0, 1.0, 1.0, 0.25],
    "size" : 6.0,
    "labelcolor" : [1.0, 1.0, 1.0, 1.0],
    "labelposition" : [0.0, -5.0e7, -4.0e8],
    "ct" : "Stars",

    "fadeout" : [21.0e2, 0.5e5],

    "profiledecay" : 1.0,

    "parent" : "Universe",
    "impl" : "gaiasky.scenegraph.StarGroup",

    "cataloginfo" : {
      "name" : "Hipparcos",
      "description" : "Hipparcos new reduction (van Leeuwen, 2007). 117995 stars.",
      "type" : "INTERNAL"
    },

    "provider" : "gaiasky.data.group.STILDataProvider",
    "datafile" : "$data/catalog-hipparcos/catalog/hipparcos/hipparcos.vot"
  }
]}

Regular star catalogs

Gaia Sky supports all formats supported by the STIL library. Since the data held by the formats supported by STIL is not of a unique nature, this catalog loader makes a series of assumptions. More information can be found in STIL data provider.

Particularly, it is possible to directly load a VOTable, CSV, FITS or ASCII file into Gaia Sky using the Open file icon at the bottom of the control panel.

Level-of-detail star catalogs

Gaia Sky uses level-of-detail structures to represent catalogs with hundreds of millions of stars. This broad and deep topic is covered in its own section:

Level-of-detail: Octree.

JSON data format

Most of the entities and celestial bodies that are not stars in the Gaia Sky scene are defined in a series of JSON files and are loaded using the JsonLoader (here). The format is very flexible and loosely matches the underneath object model, which is a scene graph tree.

An example about defining an extrasolar system with a couple of stars orbiting each other and a couple of planets can be found here.

Objects vs Updates

Every JSON file that contains objects must have a named array as the only top-level object in the file. Depending on the name of this array, two things can happen:

objects – when the array is named objects, it contains new objects to load.
updates – when the array is named updates, it contains updates to pre-existing objects.

Objects in updates arrays are kept and applied at the end of the loading stage, when all objects in objects array have been loaded. They are matched by name.

So far, only the following objects and attributes can be updated:

material – material and all its sub-attributes. In particular all, regular textures, cubemaps and virtual textures: - diffuse, diffuseCubemap, diffuseSVT. - specular, specularCubemap, specularSVT. - normal, normalCubemap, normalSVT. - height, heightCubemap, heightSVT. - emissive, emissiveCubemap, emissiveSVT. - metallic, metallicCubemap, metallicSVT. - roughness, roughnessCubemap, roughnessSVT.
cloud – describes the cloud layer. Can also have a virtual texture. - diffuse, diffuseCubemap, diffuseSVT.
atmosphere – all its direct attributes.
rotation – all its direct attributes.

See the virtual textures section for some examples.

Main objects

All objects within the objects or updates array in the JSON files are main objects, and must have at least the following 5 properties:

name: The name of the object. You can specify multiple names in a string array by using the names key.
color: The color of the object. This will translate to the line color in orbits, to the color of the point for planets when they are far away and to the color of the grid in grids.
ct – The ComponentType (see here). This is basically a string that will be matched to the entity type in ComponentType enum. Valid component types are Stars, Planets, Moons, Satellites, Atmospheres, Constellations, etc.
impl – The package and class name of the implementing class.
parent: The name of the parent entity.

Additionally, different types of entities accept different additional parameters which are matched to the model using reflection. Here are some examples of these parameters:

size – The size of the entity, usually the radius in Km.
appmag – The apparent magnitude.
absmag – The absolute magnitude.

Below is an example of a simple entity, the equatorial grid:

{
    "name" : "Equatorial grid",
    "color" : [1.0, 0.0, 0.0, 0.5],
    "size" : 1.2e12,
    "ct" : "Equatorial",

    "parent" : "Universe",
    "impl" : "gaiasky.scenegraph.SphericalGrid"
}

Model objects

Planets, moons, asteroids, etc. all use the model object Planet (here). This provides a series of utilities that make their JSON specifications look similar.

Coordinates

Within the coordinates object one specifies how to get the positional data of the entity given a time. This object contains a reference to the implementation class (which must implement IBodyCoordinates here) and the necessary parameters to initialize it. There are currently a bunch of implementations that can be of use:

OrbitLintCoordinates – The coordinates of the object are linearly interpolated using the data of its orbit, which is defined in a separated entity. See the [[Orbits|Non-particle-data-loading#orbits]] section for more info. The name of the orbit entity must be given. For instance, the Hygieia moon uses orbit coordinates.

{
    "coordinates" : {
        "impl" : "gaiasky.util.coord.OrbitLintCoordinates",
        "orbitname" : "Hygieia orbit"
    }
}

StaticCoordinates – For entities that never move. A position, which may be given in cartesian coordinates (position attribute) or spherical equatorial coordinates (equatorial attribute, gets right ascension and declination in degrees, and a radius in parsecs):

{
    "coordinates" : {
        "impl" : "gaiasky.util.coord.StaticCoordinates",
        "position" : [-2.169e17, -1.257e17, -1.898e16]
    }
}

AbstractVSOP87 – Used for the major planets, these coordinates

implement the VSOP87 algorithms. Only the implementation is needed. For instance, the Earth uses these coordinates.

{
    "coordinates" : {
        "impl" : "gaiasky.util.coord.vsop87.EarthVSOP87"
    }
}

GaiaCoordinates – Special coordinates for Gaia.
MoonAACoordinates – Special coordinates for the moon using the algorithm described in the book Astronomical Algorithms by Jean Meeus.

Rotation

The rotation object describes, as you may imagine, the rigid rotation of the body in question. A rotation is described by the following parameters:

period – The rotation period in hours.
axialtilt – The axial tilt is the angle between the equatorial plane of the body and its orbital plane. In degrees.
inclination – The inclination is the angle between the orbital plane and the ecliptic. In degrees.
ascendingnode – The ascending node in degrees.
meridianangle – The meridian angle in degrees.

For instance, the rotation of Mars:

{
    "rotation": {
        "period" : 24.622962156,
        "axialtilt" : 25.19,
        "inclination" : 1.850,
        "ascendingnode" : 47.68143,
        "meridianangle" : 176.630
    }
}

Model

This section describes the format to specify models, but omits the procedural generation attributes. These are documented in the procedural generation section.

The model object describes the model which must be used to represent the entity. Models can have two origins:

They may come from a 3D model file. In this case, you just need to specify the file.

{
    "model": {
      "args" : [true],
      "model" : "$data/default-data/models/gaia/gaia.g3db"
    }
}

They may be generated on the fly. In this case, you need to specify the type of model, a series of parameters and the material.

{
    "model": {
      "args" : [true],
      "type" : "sphere",
      "params" : {
        "quality" : 180,
        "diameter" : 1.0,
        "flip" : false
        },
      "material" : {
        "diffuse" : "$data/default-data/tex/base/earth-day*.jpg",
        "diffuseCubemap" : "$data/default-data/tex/cubemap/earth-day*",
        "specular" : "$data/default-data/tex/base/earth-specular*.jpg",
        "normal" : "$data/default-data/tex/base/earth-normal*.jpg",
        "emissive" : "$data/default-data/tex/base/earth-night*.jpg",
        "hieght" : "$data/default-data/tex/base/earth-height*.jpg",
        "heightScale" : 8.12,
        "reflection" : [ 1.0, 1.0, 0.0 ]
    }
}

type – the type of model. Possible values are sphere, disc, cylinder and ring.
params – parameters of the model. This depends on the type. The quality is the number of both horizontal and vertical divisions. The diameter is the diameter of the model and flip indicates whether the normals should be flipped to face outwards. The ring type also accepts innerradius and outerradius.
material – properties of the material, such as textures, reflections, elevation, etc.
- diffuse – the diffuse texture to use.
- diffuseCubemap – the location of the 6 sides of the diffuse cubemap to use. Takes precedence over diffuse. The sides must be images with the _bk.jpg, _ft.jpg, _up.jpg, _dn.jpg, _rt.jpg, _lf.jpg suffixes. The file formats can be JPG or PNG. More information on this can be found in the cubemaps section.
- specular – the specular map to produce specular reflections. This attribute also accepts a specular index or a specular color (RGB). More than one can be specified.
- normal – normal map to produce extra detail in the lighting.
- emissive – emissive texture. For planets, this acts as the night texture, which is applied to the part of the model in the shade. This attribute also accepts an emissive color (RGB) and an emissive index.
- height – height map which will be represented with tessellation or parallax mapping (see graphics configuration) and whose scale is defined in heightScale (in Km).
- reflection – specifies an index or a color. If this is present, the default skymap will be used to generate reflections on the surface of the material. Hint: look up the Reflections object in Gaia Sky. It is defined in satellites.json.

Clouds

This defines the clouds layer. It can be procedurally generated or described with textures. Here we deal only with the textures mode. Let’s see:

"cloud"         : {
    "size" : 6395.0,
    "cloud" : "$data/default-data/tex/base/earth-cloud*.jpg",

    "params" : {
        "quality" : 200,
        "diameter" : 2.0,
        "flip" : false
    }
}

Contains the size of the cloud model, its parameters (quality, diameter, etc.) and the clouds texture. The clouds are combined with the planet using the equation:

$\bar{C}_{result} = \bar{C}_{source} \times \bar{F}_{source} + \bar{C}_{destination} \times \bar{F}_{destination}$

where $\bar{F}_{source}$ is 1, and $\bar{F}_{destination}$ is $1 - \bar{C}_{source}$ .

Atmospheric scattering parameters

Planet atmospheres can also be defined using this object. The atmosphere object gets a number of physical quantities that are fed in the atmospheric scattering algorithm (Sean O’Neil, GPU Gems).

{
    "atmosphere" : {
        "size" : 6600.0,
        "wavelengths" : [0.650, 0.570, 0.475],
        "m_Kr" : 0.0025,
        "m_Km" : 0.0015,
        "m_eSun" : 1.0,
        "fogdensity" : 2.5,
        "fogcolor" : [1.0, 0.7, 0.6],

        "params" : {
            "quality" : 180,
            "diameter" : 2.0,
            "flip" : true
        }
    }
}

The parameters are the following:

size—radius of the sphere model used for the atmosphere, in km.
wavelengths—the values of $1\over{\lambda^4}$ for the red, green and blue channels. These are the Rayleigh scattering rates of different light wavelengths.
Kr—Rayleigh scattering constant.
Km—Mie scattering constant.
eSun—the brightness of the illuminating star.
fog density—density of the simulated fog when inside the atmosphere.
fog color—the color of the fog.

Mesh objects

Gaia Sky supports Galaxy-size arbitrary meshes. These are usually used to represent iso-density surfaces for stars, dust or HII regions, among others. Mesh objects have all the regular attributes of model bodies (name, description, color, size, etc.). Additionally, we offer three shading modes for meshes:

additive—renders the mesh with transparency via additive blending. The DR2 hot star and HII density meshes use this shading mode.
dust—renders an opaque mesh. An opacity value is computed for the edges ( $V \cdot N$ ), where $V$ is the pixel view vector from the camera and $N$ is the normal vector at that pixel. Opacity is rendered using dithering to avoid sorting issues.
regular—renders the mesh with the regular, general-purpose per-pixel lighting shader.

In order to specify the shading mode, a new top-level attribute "shading" : "additive|dust|regular" must be used:

{
    "name" : "DR3 star density",
    "description" : "Star density iso-surface based on DR3 data",
    "color" : [0.95, 0.2, 0.2, 0.75],
    "size" :  3.0856775814913705E7,
    "labelcolor" : [0.95, 0.1, 0.1, 1.0],
    "shading" : "regular",
    "labelposition" : [1000.0, 0.0, 0.0],
    "ct" : "Meshes",

    "fadeout" : [60000.0, 90000.0],

    "parent" : "Universe",
    "impl" : "gaiasky.scenegraph.MeshObject",

    "transformName" : "galacticToEquatorialF",

    "model"         : {
        "args" : [true],
        "staticlight" : true,
        "model" : "$data/mesh-dr3-stardenstiy/meshes/dr3/star_density.obj"
    }
}

Orbits

When we talk about orbits in this context we talk about orbit lines. In the Gaia Sky orbit lines may be created from two different sources. The sources are used by a class implementing the IOrbitDataProvider (here) interface, which is also specified in their orbit object.

An orbit data file. In this case, the orbit data provider is OrbitFileDataProvider.
The orbital elements, where the orbit data provider is OrbitalParametersProvider.

If the orbit is sampled it comes from an orbit data file. In the Gaia Sky the orbits of all major planets are sampled, as well as the orbit of Gaia. For instance, the orbit of Venus.

{
    "name" : "Venus orbit",
    "color" : [1.0, 1.0, 1.0, 0.55],
    "ct" : "Orbits",

    "parent" : "Sol",
    "impl" : "gaiasky.scenegraph.Orbit",
    "provider" : "gaiasky.data.orbit.OrbitFileDataProvider",

    "orbit" : {
        "source" : "$data/default-data/orb.VENUS.dat",
    }
}

If the orbit is defined with its orbital elements, the elements need to be specified in the orbit object. For example, the orbit of Phobos.

{
    "name" : "Phobos orbit",
    "color" : [0.7, 0.7, 1.0, 0.4],
    "ct" : "Orbits",

    "parent" : "Mars",
    "impl" : "gaiasky.scenegraph.Orbit",
    "provider" : "gaiasky.data.orbit.OrbitalParametersProvider",

    "orbit" : {
        "period" : 0.31891023,
        "epoch" : 2455198,
        "semimajoraxis" : 9377.2,
        "eccentricity" : 0.0151,
        "inclination" : 1.082,
        "ascendingnode" : 16.946,
        "argofpericenter" : 157.116,
        "meananomaly" : 241.138,
        "mu" : 9.9e18
    }
}

The orbit object is represented with the orbital elements:

period – in days.
epoch – in Julian days.
semimajoraxis – in km.
eccentricity – no units.
inclination – in degrees.
ascendingnode – in degrees.
argofpericenter – in degrees.
meananomaly – in degrees.
mu – $G*M$ of central body (gravitational constant). Defaults to the Sun’s. This will be anyway automatically recomputed from the period ( $T$ ) and the semi-major axis ( $a$ ), if set, as $\mu=4 * \pi^{2} a^{3} / T^{2}$ .

Note that if the epoch is fully defined, the argofpericenter is not needed.

The orbital elements of extrasolar systems are typically given in a special reference system. In this reference system, the reference plane is the plane whose normal is the line of sight vector from the Sun to the planet or star for whom the orbit is defined. The reference direction is the direction from the object to the north celestial pole projected on the reference plane. In order to apply such a transformation automatically, Gaia Sky provides an additional attribute "model" to objects of type gaiasky.scenegraph.Orbit. When this attribute has the value "extasolar_system", the abovementioned transformation is applied automatically. The default value of "model" is "default". Below is an example:

{
   "name": "J0805+4812 star orbit",
   "color": [
     1.0,
     0.0,
     1.0,
     1.0
   ],
   "ct": [
     "Orbits",
     "Stars"
   ],
   "parent": "J0805+4812 Center",
   "impl": "gaiasky.scenegraph.Orbit",
   "provider": "gaiasky.data.orbit.OrbitalParametersProvider",
   "model": "extrasolar_system",
   "newmethod": true,
   "orbit": {
     "period": 735.9078666506588,
     "epoch": 2457397.4170103897,
     "semimajoraxis": 48464416.969729796,
     "eccentricity": 0.4203524474493615,
     "inclination": 107.89617887219426,
     "ascendingnode": 175.09066812003118,
     "argofpericenter": 326.7130552945523,
     "meananomaly": 0.0,
     "mu": 9.9998e+21
   }
 }

Grids and other special objects

There are a last family of objects which do not fall in any of the previous categories. These are grids and other objects such as the Milky Way (inner and outer parts). These objects usually have a special implementation and specific parameters, so they are a good example of how to implement new objects.

{
    "name" : "Galactic grid",
    "color" : [0.3, 0.5, 1.0, 0.5],
    "size" : 1.4e12,
    "ct" : "Galactic",
    "transformName" : "equatorialToGalactic",

    "parent" : "Universe",
    "impl" : "gaiasky.scenegraph.Grid"
}

For example, the grids accept a parameter transformName, which specifies the geometric transform to use. In the case of the galactic grid, we need to use the equatorialToGalactic transform to have the grid correctly positioned in the celestial sphere.

Creating your own catalog loaders

If you want to load your data files into Gaia Sky, chances are that the STIL data provider can already do it.

If you still need to create your own loader, keep reading.

In order to create a loader for your catalog, one only needs to provide an implementation to the ISceneLoader (here) interface.

public interface ISceneLoader {
  public List<Entity> loadData() throws FileNotFoundException;
  public void initialize(String[] files, Scene scene) throws RuntimeException;
}

The main method to implement is List<Entity> loadData() (here), which must return a list of Entity objects.

But how do we know which file to load? You need to create a catalog-*.json file, add your loader there and create the properties you desire. Usually, there is a property called files which contains a list of files to load. Once you’ve done that, implement the initialize(String[], Scene) (here) method knowing that all the properties defined in the catalog-*.json file with your catalogue loader as a prefix will be passed in the Properties p object without prefix.

Also, you will need to connect this new catalog file with the Gaia Sky configuration so that it is loaded at startup. To do so, locate your config.yaml file (usually under $GS_CONFIG, see folders) and add your new file to the property data::catalogFiles.

Add your implementing jar file to the classpath (usually putting it in the lib/ folder should do the trick) and you are good to go.

If you need examples, take a look at existing loaders such as the OctreeLoader (here) to see how it works.

Loading data using scripts

Data can also be loaded at any time from a Python script. See the scripting section for more info.