Exploring Stellar Wonders: Unraveling the Enigma with GridDB¶

Introduction¶

All of us have grown up in fascination of the sky and stars at night. The mere sight of a vast, twinkling canvas above us ignites a sense of wonder and curiosity. As we gaze at the celestial tapestry, our imaginations soar, pondering the mysteries of the universe. The stars, like ancient storytellers, hold secrets and tales of cosmic proportions. They have inspired countless myths, guided explorers across vast oceans, and propelled humanity's quest for knowledge and understanding. From the timeless beauty of the Milky Way to the fiery brilliance of distant supernovae, the sky beckons us to delve deeper into its mind-boggling wonders. It is in this awe-inspiring realm that GridDB, a powerful analytical tool, emerges as a guiding light, enabling us to unlock the hidden patterns and insights woven within the intricate fabric of the stars. With GridDB, we embark on a captivating journey of exploration, unraveling the enigmatic nature of the universe and painting a vivid portrait of our cosmic cousins. Let's get started!

About the Dataset¶

The Dataset is obtained from the web portal Astronomy Nexus which is owned by astronomy enthusiast and stargazer David Nash. For the purpose of this analysis, the dataset HYG 3.0 has been downloaded. Please refer to the Creative Commons Attribution-ShareAlike license of the dataset here.

Instructions to Download the Dataset¶

The dataset is in the gz format. Use a tool such as 7-Zip File Manager for this. Below are the instructions to extract the dataset from the compressed file.

Select the .gz file (hygdata_v3.csv.gz) and click the 'Extract' button.

In the screen that follows, click on 'Extract'

The file gets extracted inside a folder with the file name 'hygdata_v3.csv'.

Understanding the Dataset Abbreviations¶

The abbreviated column names represent the different properties or attributes associated with each star in the dataset.

• id: Star ID
• hip: Hipparcos ID
• hd: Henry Draper Catalog ID
• hr: Harvard Revised Catalog ID
• gl: Gliese Catalog ID
• bf: Bayer/Flamsteed designation
• proper: Proper name of the star
• ra: Right Ascension (RA) in decimal degrees or hours
• dec: Declination (Dec) in decimal degrees
• dist: Distance to the star
• pmra: Proper motion in right ascension
• pmdec: Proper motion in declination
• rv: Radial velocity
• mag: Apparent magnitude
• absmag: Absolute magnitude
• spect: Spectral type
• ci: Color index
• x, y, z: Cartesian coordinates
• vx, vy, vz: Cartesian velocities
• rarad, decrad: Right Ascension and Declination in radians
• pmrarad, pmdecrad: Proper motion in right ascension and declination in radians
• bayer: Bayer designation
• flam: Flamsteed number
• con: Constellation abbreviation
• comp: Component identifier for multiple stars
• comp_primary: Primary component identifier for multiple stars
• base: Base temperature
• lum: Luminosity
• var: Variable star designation
• var_min, var_max: Minimum and maximum magnitudes for variable stars

Importing Libraries¶

About GridDB¶

GridDB is a reliable option to store and analyze astronomical datasets. GridDB's features, such as its high performance, scalability, and real-time analytics capabilities, make it well-suited for handling the large and complex datasets encountered in astronomy. Astronomical data often requires complex queries to extract meaningful information. GridDB supports SQL-like queries, enabling researchers and astronomers to perform various operations such as filtering, aggregation, and joining of multiple astronomical datasets and analyzing them for powerful insights. Moreover, GridDB's in-memory computing architecture, combined with its disk-based storage, enables fast data processing and low-latency access to data.

In this case, we use Python's requests package to connect to GridDB's WebAPI (GridDB Cloud) to create the data container, load data into the container, and write queries against the data.

Reading the Dataset into a Dataframe¶

We will use this dataframe to load data into the container later on.

Setting up the Authorization and Request¶

To connect to the GridDB cloud's WebAPI, the Request that is constructed should have the username and password encoded to base64 standards. To determine the base64 encoded credentials for your username and password, a tool such as https://www.base64encode.org/ can be used. Refer to the resource listed here to learn more about the different entities involved in creating a request - https://griddb.net/en/blog/using-griddb-cloud-version-1-2/. The basic authentication format in GridDB is 'username:password' where the case-sensitive username and password should be used for encoding.

Construct the base URL based on your GridDB cluster you'd like to connect to. Ensure that you replace the placeholders in the base_url below with the correct values that correspond to your GridDB instance

Creating the Data Container¶

GridDB supports two types of containers namely collections and time-series containers. While collections are general-purpose data containers that store data records with arbitrary schema, time series containers are specifically designed for handling data where each record is associated with a timestamp. In this case, Collections are ideal as they are suitable for storing diverse types of data. Refer to this official resource to learn more on GridDB containers and their characteristics.

An important pre-requisite to creating a container is to determine the data types of each data column you have. GridDB supports both primitive and complex datatypes. Among the primitive datatypes, GridDB supports Integer types, Boolean, String, Floating Point types and Time variables. Among the complex datatypes, GridDB supports Arrays, blob storage and Geospatial data storage. GridDB also has specialized datatypes that support TIMESTAMP-based data storage and retrieval optimizations and options to store Enumerated data types. Refer to this resource to learn more.

Below is a template to create a container object. For Collection containers, the rowkey can is usually set to False whereas for Timeseries containers, the rowkey has to be set to True. When we set 'rowkey' to True, we are effectively telling GridDB that the container has a primary key column. Timeseries containers require the timestamp variable to be the primary key. While Collection Containers can either have a rowkey of True or False, when the rowkey is set to False, GriDB creates a Collection Container by default.

The choice between using a row key or not depends on the requirements of your application and the access patterns you anticipate. If you need to perform frequent key-based lookups or retrieve individual records based on their unique identifiers, setting the row key to "true" would be appropriate. On the other hand, if your application primarily relies on querying and filtering records based on other field values, setting the row key to "false" is suitable.

data_obj = {
"container_name": "Container_Name",
"container_type": "Container Type",
"rowkey": False,
"columns": [
{
"name": "Name of column 1",
"type": "Datatype of column 1"
},
{ "name": "Name of column 2",
"type": "Datatype of column 2"
},
{ "name": "Name of column 3",
"type": "Datatype of column 3"
},

Rules for naming Containers and columns within a container object¶

Note that while creating column names, special characters such as '(', '/' and spaces should be avoided in column names and container names. An underscore ( _ ) can however be used. If there are such characters in your container declaration, the request fails with the error message '{"version":"v2","errorCode":145007,"errorMessage":"[145007:JC_ILLEGAL_SYMBOL_CHARACTER] Illegal character found (character=\" \"(\\u0020), component=base, containerName=\"Astronomy Dataset\", index=9)"}'. Here is a comprehensive list of error codes for GridDB - http://www.toshiba-sol.co.jp/en/pro/griddb/docs-en/v4_3/GridDB_ErrorCodes.html

A pythonic way to create a container¶

Tapping into Python's capabilities, the below code will iterate over the input_variables list and create a column dictionary for each variable. The "name" field of each column for the 'Collection' container object will be populated with the variable from the list 'input_variables'. Similarly, the 'type' field of each column for the 'Collection' container object will be populated with the variable from the list 'data_types'.

The zip() function is used to iterate over both lists simultaneously, allowing you to access the corresponding data type for each input variable. The updated data_obj will contain the dynamically generated column names based on the input variables.

Row Registration (Data Loading)¶

In GridDB, the process of loading data into a container is referred to as "Row Registration" because it involves registering or adding rows to the container. When you register a row, you are essentially inserting a new record into the container, specifying the values for each column in that row. This terminology aligns with the concept of a database table, where rows represent individual records or entities, and registering a row signifies the act of adding a new record to the table. Refer to this resource to learn more about what happens in the backend while using Row Registration. https://www.toshiba-sol.co.jp/en/pro/griddb/docs-en/v4_6/GridDB_Web_API_Reference.html#row-registration

The good thing about using Python in conjunction with GridDB is that we can use both inbuilt functions in Python and packages such as Json and Pandas to manipulate data easily and effectively. In this case, we use the 'to_json' function of the 'json' module.

The dataframe 'hygdata_v3.csv' (Refer to the section 'Reading the Dataset into a Dataframe') is used to load data into a container after a few transformations. The below lines of code convert the data from the DataFrame to JSON format (using the to_json function from the 'json' library). The JSON object is then sent in a PUT request to the GridDB WebAPI to register the rows in the specified container.

Batch Loading properties of GridDB¶

GridDB supports batch loading. If you are using the SDK, here is a useful resource - https://griddb.net/en/blog/griddb-optimization-with-multi-put-and-query/. If you use the WebAPI, then you can just use a simple for loop to load your subsets of data into your GridDB container. There are two major advantages to using batch loading to register rows into your GridDB container. One is that large datasets can be loaded seamlessly. The second is that you can effectively troubleshoot issues with your data easily.

Unveiling the Celestial Mysteries: Exploring the Captivating Astronomy Dataset¶

GridDB provides a query language called "GridDB Query Language", which is similar to SQL syntax. GQL allows you to perform various operations on GridDB databases, such as querying data, filtering results, aggregating data, and more. While GQL shares similarities with SQL, it also has its own unique features and syntax tailored specifically for GridDB's distributed architecture and data model. For Timeseries containers, GridDB provides specialized query language called TQL (TimeSeries Query Language) to support time-based aggregations, time-based selections and to set sliding windows. In this case, we stick to the SQL capabilities of GridDB as we have created a Collection Container. But if you have other use cases that are time-based, check out this TQL Reference . To refer to the SQL reference, visit this page.

GridDB supports conditional expressions like 'Case When'. Refer to this resource to learn more.

What does the distribution of the Top 20,000 stars look like?¶

Insight(s):

• When we look at the celestial co-ordinates of the Top 20,000 stars, we get the above distribution.
• The brightest star in this case has a luminosity of 400M. The next brightest seems to be having a luminosity of 250M.
• Most stars seem to be between 0 and 50M luminosity.

What are some known stars and their spectral classification and magnitude?¶

GridDB supports NULL handling and special handling. Refer to this resource to learn more.