Data format#
To display data, Gaia Sky first needs to load datasets. Dataset files contain objects, which Gaia Sky organizes into a scenegraph. A scenegraph is a tree that contains objects and orders them hierarchically depending on their geometrical and spatial relations.
All datasets are partially or totally described in a JSON format. Each dataset lives in its own directory in the data location (referred to as $data/, see folders), and must contain a description in the file dataset.json. If a dataset does not have this file in its directory, it won’t be recognized by Gaia Sky.
Below is an example of the contents of the data location.
$data/
├─ default-data/
│ ├─ dataset.json
│ └─ ...
├─ catalog-hipparcos/
│ ├─ dataset.json
│ └─ ...
├─ catalog-whitedwarfs-dr2/
│ ├─ dataset.json
│ └─ ...
└─ .../
The following sections describe the different parts of the data format.
Where are the data files defined?#
Gaia Sky implements a very flexible and open data loading mechanism. The data files to be loaded are defined in a couple of keys in the config.yaml configuration file, which is usually located in the $GS_CONFIG folder (see folders). The keys are the following (double-colon indicates nesting):
data::dataFiles– an array containing the list of enabled JSON data files. Each file should be a relative path from the data directory with the prefix$data/. For instance, the default dataset, containing the Solar System and some necessary objects for Gaia Sky to run, is specified in the array as$data/default-data/dataset.json.
$data/[dataset-name]/dataset.json example#
Dataset descriptor files contain the metadata of a catalog (name, description, version, etc.) and a pointer to the actual data. Below is a made-up file, dataset.json which describes my super-awesome dataset.
{
"key" : "dataset-key-without-spaces",
"name": "Dataset name",
"version": 1,
"mingsversion" : 30106,
"type": "catalog-gaia",
"description": "The description here.\nCan contain line breaks.",
"releasenotes" : "- What changed since the last version?\n- List here.",
"link": "https://arxiv.org/abs/1805.00425",
"size": 368633,
"nobjects": 1365,
"check": "$data/my-dataset/dataset.json",
"files": [ "$data/my-dataset" ],
"data": [{
"loader": "gaiasky.data.JsonLoader",
"files": [ "$data/my-dataset/particles-particles.json" ]
}]
}
Notice that the data (data::files) points to another JSON file, which contains some additional info about how to load the data, and a pointer to the actual data file particles-particles.json. Here it is:
{
"objects": [{
"name": "My dataset",
"position": [0.0, 0.0, 0.0],
"componentType": "Stars",
"fadeout": [1.0e5, 0.5e8],
"parent": "Universe",
"archetype": "StarGroup",
"provider": "gaiasky.data.group.STILDataProvider",
"dataFile": "$data/my-dataset/catalog/dataset.vot"
}]
}
As you can see, the STILDataProvider is the one in charge of loading the data for this dataset, which resides in a VOTable file, dataset.vot.
default-data/dataset.json example file#
This is an example of what the default data pack contains. The dataset descriptor file loads different files using different loaders.
{
"data" : [
{
"loader": "gaiasky.data.JsonLoader",
"files": [ "$data/default-data/planets-normal.json",
"$data/default-data/moons-normal.json",
"$data/default-data/satellites.json",
"$data/default-data/asteroids.json",
"$data/default-data/orbits_planet.json",
"$data/default-data/orbits_moon.json",
"$data/default-data/orbits_asteroid.json",
"$data/default-data/orbits_satellite.json",
"$data/default-data/extra-low.json",
"$data/default-data/locations.json",
"$data/default-data/locations_earth.json",
"$data/default-data/locations_moon.json"]
},
{
"loader": "gaiasky.data.GeoJsonLoader",
"files": [ "$data/default-data/countries/countries.geo.json" ]
}]
}
The dataset.json file contains an array, "data", which is a list of pairs
containing [loader: files] correspondences. Each "loader" contains the classes that will load the list of files under the
corresponding "files" property. The main loader, the JsonLoader, expects JSON files as inputs. Each of these files must have an attribute called "objects", which is an array containing the metadata on the objects to load.
Data loaders#
The files are sent to the Scene Graph JSON Loader, which iterates on each loader-files pair
in each file, instantiates the loader and uses it to load the files. All loaders need to adhere
to a contract, defined in the interface ISceneLoader (ISceneLoader).
The loadData() method of each loader must return a list of objects, which is then added to a global list containing all the previously loaded objects. At the end, we have a list with all the objects in the scene. This list is passed on to the Scene Graph instance, which constructs the scene graph tree structure that will contain the object model.
As we said, each loader will load a different kind of data; the
JSONLoader (JsonLoader)
loads JSON files containing planets, satellites, orbits, star catalogs, etc. The STILDataProvider (STILDataProvider) loads VOTables, FITS, CSV and other files through the STIL library, GeoJsonLoader (GeoJsonLoader) loads geographic data, and so on.
Creating your own catalog loaders#
If you want to load your data files into Gaia Sky, chances are that the STIL data provider can already do it.
If you still need to create your own loader, keep reading.
In order to create a loader for your catalog, one only needs to provide an implementation to the ISceneLoader interface (core/src/gaiasky/data/api/ISceneLoader.java).
public interface ISceneLoader {
void initialize(String[] files, Scene scene) throws RuntimeException;
void initialize(String[] files, String dsLocation, Scene scene) throws RuntimeException;
void initialize(DataSource ds, Scene scene);
Array<Entity> loadData() throws FileNotFoundException;
void setKey(String key);
void setName(String name);
void setDescription(String description);
void setParams(Map<String, Object> params);
void setIndex(Map<String, Entity> index);
}
The main method to implement is Array<Entity> loadData() in core/src/gaiasky/data/api/ISceneLoader.java, which must return an array of Ashley Entity objects (libGDX Array, not java.util.List).
But how do we know which file to load? You need to create a dataset.json descriptor file, add your loader there and create the properties you desire. Usually, there is a property called files which contains a list of files to load. Once you’ve done that, implement one of the initialize overloads – (String[], Scene) when the data comes from files, (String[], String dsLocation, Scene) when the dataset location is also needed, or (DataSource, Scene) when the data is streamed from a remote source – knowing that all the properties defined in the dataset.json file with your catalog loader as a prefix are passed to you, without the prefix, through the setParams(Map<String, Object>) method.
The remaining setters (setKey, setName, setDescription, setIndex) are called by Gaia Sky with the corresponding metadata from the dataset descriptor, so your loader gets the dataset key, name and description for free, plus the shared entity index built by other loaders.
Also, you will need to connect this new catalog file with the Gaia Sky configuration so that it is loaded at startup. To do so, locate your config.yaml file (usually under $GS_CONFIG, see folders) and add your new file to the property data::dataFiles. You can also drop your new catalog into a subdirectory in the data directory and enable it using the dataset manager in Gaia Sky.
Add your implementing jar file to the classpath (usually putting it in the lib/ folder should do the trick) and you are good to go.
You can use existing loaders as examples, such as the OctreeLoader (OctreeLoader)
to see how it works.
Loading data using scripts#
Data can also be loaded at any time from a Python script. See the scripting section for more info.