Spherical Panoramas for Astrophysical Data Visualization

TL;DR

This paper introduces a method for creating immersive 360-degree spherical panoramas of astrophysical data using Blender and Google Spatial Media, enhancing data exploration and visualization.

Contribution

The paper presents a novel technique combining Blender and Google Spatial Media to generate immersive spherical panoramas for astrophysical data visualization.

Findings

Effective visualization of various astronomical data types

Enhanced user immersion in data exploration

Potential for improved astrophysical data analysis

Abstract

Data immersion has advantages in astrophysical visualization. Complex multi-dimensional data and phase spaces can be explored in a seamless and interactive viewing environment. Putting the user in the data is a first step toward immersive data analysis. We present a technique for creating 360 degree spherical panoramas with astrophysical data. The three-dimensional software package Blender and the Google Spatial Media module are used together to immerse users in data exploration. Several examples employing these methods exhibit how the technique works using different types of astronomical data.

Tables

Table 1. Table 1: Rendering Engine Comparison

Engine Name	Support Notes
Render Default	All mesh material modes supported.
Cycles	Halo materials for catalogs not supported. Benefits include using node compositor.
Game Engine	Physics-based simulation and particle generator.

Equations

$$x=\varphi$$

$$y=\theta$$

$$x=\varphi$$

$$y=\frac{180^{\circ }}{\pi }\frac{\sin \theta }{\lambda }$$

$$x=2\gamma \cos \theta \sin \frac{\varphi }{2}$$

$$y=\gamma \sin \theta$$

$$\gamma =\frac{180^{\circ }}{\pi }\sqrt{\frac{2}{1+\cos \theta \cos (\varphi /2)}}$$

$$\left[\begin{array}{c}w{x}^{\prime }\\ w{y}^{\prime }\\ w\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

$${\mathrm{p}}^{\prime }=\mathrm{H}\mathrm{p}$$

Full text

Accepted for publication in the journal PASP Special Focus Issue Techniques and Methods for Astrophysical Data Visualization.

Data Analysis and Techniques

1 Introduction

Data presentation in astronomy has expanded beyond the traditional two-dimensional plots to include multi-variable analysis, immersive three-dimensional (3D) models and exploratory analysis on a multitude of hardware platforms. Many forms of astronomical data can benefit from alternative forms of data exploration. Data cubes can be rotated in 3D, fly-throughs of planetary landscapes can be explored, and all-sky maps can be examined. Exploratory 3D presentations give users a different perspective on their data. Traditional desktop and laptop configurations as well as hand-held mobile devices can be used as the hardware mechanism for these data explorations. The techniques can be employed in both research-level visualization and can deliver easy access to materials for education and public outreach (Kent, 2013).

Panoramic imaging in the physical domain is accomplished with a tripod mounted camera to encompass a larger field of view (FOV) than is visible through the lens. Multiple images are obtained while attempting to minimize parallax errors between images (Derrien & Konolige, 2000). In some cases, four to six ultra wide-field cameras are used to record video in every possible direction. Software can then merge these images and video encompassing a FOV into a cylindrical or spherical panorama (Ozawa et al., 2012). Computer hardware and screen resolution along with network bandwidth are now sufficient to allow 360*∘* spherical panoramic video: a constant stream of panoramic images obtained with multiple wide-field cameras that gives a user control over the view while the video advances. Natural movement to view the scene can be accomplished with accelerometer-enabled hand-held devices or with desktop browsers.

We present a methodology overview for creating still and video spherical panoramas using astronomical data. Section 2 gives the reader a brief overview of rendering with Blender and the Google Spatial Media module. In section 3 we describe the math and techniques behind building a spherical panorama. Section 4 outlines examples utilizing the libraries and techniques described in this paper. Section 5 summarizes the techniques.

2 Panoramic Rendering

Traditional data presentation revolves around graphs, charts, tables, and other forms of two-dimensional media. They have a critical role in research publications. Complex or multi-dimensional data can benefit from a more advanced data display mechanism. The ability to immerse users in their data has a number of technical challenges that can be accommodated with recent advances in both hardware and software. Data can be presented within a 3D space, including data cube volume rendering, planetary surface maps, and astronomical catalogs (Hassan & Fluke, 2011; Kent, 2013). This paper introduces astronomers to spherical panoramas as a data display mechanism. It is advantageous to use immersive spherical panoramas as they allow easy and natural control of the viewing space, intuitive and minimalist user interfaces, and cross-platform support across a range of hardware and display software.

Projection algorithms in astronomical FITS images have been well curated and documented (Hanisch et al., 2001). The growing size of CCD cameras required new drizzling algorithms and reprojection models that paved the way for mosaicking and fast, large, wide-field projection and coordinate transformation algorithms (Fruchter & Hook, 2002; Makovoz, 2004). Montage is a recent example of image mosaicking and projection code that has been very successful on this front (Berriman et al., 2007; Jacob et al., 2009). While this paper focuses on adding value to existing images and data, it is important to acknowledge the algorithms that go into making the products from large projects. Astronomical surveys produce wide-field imaging science-ready data products and catalogs (Abazajian et al., 2009; Skrutskie et al., 2006; Kent et al., 2008). Wide-field radio surveys require special algorithms where mosaicking is done in the interferometric uv–plane (Tasse et al., 2012; Offringa et al., 2014) as opposed to the imaging plane.

With wide-field mosaics across multi-wavelength regimes increasing in size, resolution, sensitivity and detail, it is useful to create immersive spherical panoramas where the user is inside the data. Panoramic photography and imaging play an important role in interactive user interfaces, data analysis, and 3D reconstruction (Peleg et al., 2001; Gledhill et al., 2003). More recently, stereoscopic vision, virtual reality, and augmented reality applications can combine to supplement data viewing in geographic and mapping software (Azuma, 1997; Azuma et al., 2001; Azuma, 2004; Wither et al., 2011). This has applications in haptic feedback devices, 3D caves, and other forms of virtual reality (Sanchez-Vives & Slater, 2005). Displaying 3D data on traditional two-dimensional (2D) displays (a computer or a tablet) requires certain considerations when projecting images (Gilbert et al., 2012). Further applications include stereo vision in 3D environments with headgear apparatus (Gasparini & Bertolino, 2013). We focus here on the simplest use of easy to obtain hardware: desktops, laptops, and mobile devices. Used in conjunction, the software package Blender combined with the Google Spatial Media Library make a powerful combination for a researcher to animate and display their data for other users.

We outline spherical panoramas described in this paper that can fit into one of three scenarios (Figure 1). An example from each of these scenarios will be described in section 4.

• •

Static spherical panoramas. This type is best exemplified by a user stationary in the middle of the celestial sphere. All-sky maps work well with this scenario. It requires a single frame and can be executed with short render times.

• •

Single pass fly-throughs. Imagine this scenario as moving through a 3D dataset on a predetermined track. As the user moves along the track at a fixed speed, they are free to look about in any direction.

• •

Orbit flyovers. The scenario is geared toward data where one item is in focus, such as a particular area or object in a simulation or map feature.

2.1 Blender

Blender supports a wide range of 3D visualization and animation scenarios111https://www.blender.org/. It is maintained by the Blender Foundation as open-source software. The software has been adapted for use in a number of astronomy applications with easily accessible primers and tutorials (Kent, 2013, 2015; Taylor, 2015; Naiman, 2016; Gárate, 2017). The novice user is referred to those works and the references therein when using Blender for the first time with their data. Blender versions 2.64a and later will work with the examples and techniques outlined in this paper.

Three-dimensional graphic suites like Blender require a workflow of data selection, modeling, and animation. Other software packages referred to in Kent (2013) including Lightwave 3D, Maya, and 3D Studio Max are popular in the graphics industry. While the internal working models and graphical user interfaces may differ between packages, Blender supports data import from all of those packages. As with any data reduction or graphical display package and depending the complexity of data, it is best to begin learning the package with a simple dataset such that the developed data model can be built upon for reuse. The user’s proficiency can quickly improve with the tutorials and examples provided at the end of this paper. To aid in the ease of scripting, the Python application program interface (API) allows interfacing with popular scientific astrophysics libraries (Astropy Collaboration et al., 2013) such that a variety of astronomical data (ASCII catalogs, FITS images, etc.) can be brought into the environment. We choose to use Blender because its built-in engine capabilities allow for fast panoramic rendering of a 3D scene. In addition, the new Cycles rendering engine allows for a node-based compositor control of many of Blender’s graphical user interfaces (GUIs) and features (Table 1). The camera controls in Blender give a 360${}^{\circ}\times$180*∘* FOV that can be used as a still image in a static spherical panorama (Figure 2).

2.2 Google Spatial Media

The Google Spatial Media module is a collection of specifications and tools for spherical 360*∘* video and spatial audio. The module can take rendered output from Blender and generate spherical panorama video. This is accomplished with a metadata injector that adds information on the file type and FOV to the header. Audio can also be included in the metadata, but it will not be discussed in this work. The module source code is available from a GitHub repository222https://github.com/google/spatial-media. The module is available as a stand-alone executable for Windows and Mac OS or as a Python module for Linux/Windows/Mac OS. By adding to the video metadata header, the output can then be exported to YouTube333http://www.youtube.com. Within YouTube the player becomes aware of the data content and gives the user a set of controls with which to navigate the environment. Users can use a mouse to click and drag or move their tablet or mobile device naturally to view the scene. The preprocessing by Blender to create a full spherical view covering 4$\pi$ steradians is critical before running it through this module. It is assumed that the metadata injection carried out by the module is being applied to to a full spherical panoramic view.

The social media website Facebook also has the capability of supporting 360*∘* video, but the metadata injection must be inserted by the user if the video has not been already created by a spherical camera system. This creates a complication that is solved for the user by using the Google Spatial Media module with the added benefit of the Python interface. Other 3D VR/AR (virtual reality/augmented reality) applications such as Oculus Rift or Vive require rather steep hardware requirements. The scheme presented in this paper is aimed at a researcher utilizing hardware already available as well as open-source software for the processing.

2.3 Hardware and Software Requirements

The Google software interface coupled with Blender and YouTube allows for an easy implementation of a spherical panorama video on a variety of hardware platforms. The results can be viewed with a mouse-desktop/laptop setup with the Chrome444http://chrome.google.com or Firefox browser555http://http://www.mozilla.org/firefox. Tablets and mobile devices with three-axis accelerometers are where this technique’s forte really lies. Using this technique on such devices allows for an immersive data discovery session. A user can move within the dataset as the video plays or be located in a static position (Figure 3). The device itself can be moved or the user can navigate with their hands. If the camera is moved during a spherical panorama video, users can track a particular object by pointing their device in different directions. User device rotation is the controllable element in this scheme. No translation of the mobile device will have an effect on the viewer’s perspective. The technique works equally well on both iOS and Android mobile operating systems.

3 Methods

Images can be combined and projected from an equirectangular, cylindrical equal area or Hammer-Aitoff projection. The first two may be necessary in certain cases where data is not available to encompass the entire sphere of visibility. An equirectangular projection conversion is simply defined with Cartesian coordinates x,y, and angles $\phi,\theta$ as

[TABLE]

[TABLE]

A cylindrical equal area projection can be defined as

[TABLE]

[TABLE]

where $\phi$ is the azimuthal angle and $\lambda$ is the radius of the projected cylinder. We will assume a scaling of $\lambda=1$ in this work’s examples for simplicity (known as a Lambert’s equal area projection). A simple cylindrical panorama setup with a Lambert’s equal area projection is shown in Figure 4.

For all-sky maps, image reprojection is the most computationally intensive part of image processing. Tools such as Montage assist with this (Makovoz & Khan, 2005; Jacob et al., 2009). All-sky maps are readily available from a large number of astronomical surveys. The Hammer-Aitoff to spherical formalism is given as

[TABLE]

[TABLE]

where

[TABLE]

These maps and projection definitions are well defined by Calabretta & Greisen (2002). These transformations will be employed in section 4.

Once a FITS image map is generated, it can be imported into Blender by way of AstroPy and its associated FITS manipulation modules. Planetary surface maps and 3D catalogs can be brought into the Blender environment as described in Kent (2013, 2015). Here we focus on the methods and geometry of virtually filming these data for spherical panoramas.

We can consider the method of virtually photographing our scene with data in a similar fashion to the way a camera is set up to image in a panoramic mode (Figure 5). While the internal Blender mechanics automate this process with the built-in camera object, it is useful to understand some of the underlying mathematics. The projective mapping between two planes is given by

[TABLE]

which is summarized as

[TABLE]

where H is the homography matrix that transforms p to our plane p′**. The formalism for panoramic homography is reviewed in Hartley & Zisserman (2004). When projecting images for viewing, it is important to understand that we are not rotating the FOV about the focal plane, but around the nodal point for the exit pupil that sits in front of the plane (Figure 6). The nodal point for a physical camera is the correct pivot point in panoramic imaging and allows us to minimize parallax errors  (Zhang & Liu, 2014). A virtual camera in a 3D scene can be thought of as a point from which rays originate constrained by a FOV. The rays in a virtual camera’s 360*∘* FOV are unconstrained (see Glassner (1989) for a comprehensive review on ray tracing). Stereo imaging is not within the scope of this paper, but it will utilize two origin points for the rays separated by an inter-pupillary distance, also refereed to as a stereo baseline (Peleg et al., 2001).

With this information in mind, the broad workflow for creating our examples will work as follows. Each example will give details specific to that given scenario and will utilize the following seven steps. The seventh stage will be omitted from the examples as it is uniform to all three, where we use the Google Spatial Media module.

Import data. Maps, textures, and catalogs are brought into the environment. Scripts for these processes are available in Kent (2013). 2. 2.

Set uv-mapping. For astronomical images and maps that need to be projected into the 3D environment, uv-unwrapping can be utilized (Figure 7) using equations 5–6. 3. 3.

Configure data scene. The data scene configuration includes any illumination, texture, and material objects that are needed in the visualization. 4. 4.

Set camera path. The path is configured with a Bézier curve if required. The camera can be moved freely as determined by the user or restricted to initially focus on a particular area or object of the data. 5. 5.

Configure camera. The camera settings can be configured to maximize the FOV depending on how much coverage is yielded by the data (for instance, only a partial set of data in an all-sky map). This requires a panoramic camera selection in either the default rendering engine or the new Cycles engine. 6. 6.

Render Scene. The scene should be rendered at 4K or higher since the spherical panorama image must cover 4$\pi$ steradians (Silva et al., 2016). The best output resolution should be used at 3840 $\times$ 2160 pixels. 7. 7.

Configure metadata. The rendered video can be imported into the Google Spatial Media module as a stand-alone binary or as part of a Python script.

4 Examples

The examples described in this section follow data import and manipulation procedures outlined in Kent (2013, 2015). All data manipulation examples start with the default file that opens with Blender upon startup; no changes need to be made to the default configuration. Most workflows in these examples require a number of mouse/keyboard binding combinations.666References for keyboard entry and model manipulation can be found at http://www.cv.nrao.edu/$∼$bkent/blender/tutorials.html

The first example uses a static view inside a celestial sphere. The second and third examples both utilize a moving camera with two different data sets, an astronomical catalog where the camera flies through with a linear path and a 3D planetary surface with a circular path where the camera can focus on a particular geographic feature.

4.1 All-sky maps

All-sky map data lend themselves very well to the spherical panorama paradigm. While any astronomical FITS image can be used when properly placed on the celestial sphere, wider sky coverage produces a more pronounced effect. All-sky maps cover a complete 4$\pi$ steradians FOV of spatially well-sampled data over a wide range of wavelengths. For this example, we use a static camera view where the user does not translate. The user is essentially placed inside the center of a celestial sphere and can rotate to view everything around them (Figure 8). Data from Skyview can be used, obtaining images centered on the galactic center ($l=0^{\circ},b=0^{\circ}$) in a Cartesian or Hammer-Aitoff projection (McGlynn & Scollick, 1994; Mellinger, 2009). Oversampling the image in a 2:1 or 16:9 aspect ratio with a 360${}^{\circ}\times$180*∘* FOV is best; the end product will be viewable on different resolutions with various hardware.

The workflow for this example is as follows:

Importing data can be accomplished with astropy.io.fits.open or with a high-resolution JPEG, PNG, or TIFF image. Images can be added via the UV/Image Editor (see the lower panel of Figure 7). The data scene requires that a uv sphere mesh be added to the scene by pressing the key combination SHIFT-A. 2. 2.

Pressing the TAB key puts the main viewer in Edit mode.

The A key followed by the U key is used to map the image onto the mesh model. 3. 3.

A material and texture can be added to the mesh sphere with an Emission shading value of 1.0, Image set to the loaded map and Coordinates set to UV. 4. 4.

The camera will be static in this example, rotated to ($x_{\theta},y_{\theta},z_{\theta}$) = ($0^{\circ},0^{\circ},0^{\circ}$) facing the galactic center at ($l=0^{\circ},b=0^{\circ}$). 5. 5.

The Cycles engine can be used to our advantage here, setting the FOV to 180*∘*. 6. 6.

The resolution output should be set to 3840 $\times$ 2160 pixels, the highest resolution currently available on YouTube777https://support.google.com/youtube/answer/6375112. The video output should be set to the MPEG with MPEG-4 encoding and a bitrate of 8000 kilobits per second.

4.2 Catalog fly-through

In this example, we tour a galaxy catalog (Tully et al., 2009) where the user is on a predefined moving track through 3D space. This automated constant speed translation is coupled with the user controlled rotation as the video visualization plays. Translating schemes like this are good for scenarios in which various aspects of the data need to be given a closer look. It also may be the case that the dataset covers a large parameter space, thus traversing a particular axis is advantageous.

Data can be imported from ASCII text files, XML or binary tables using standard Python modules. 2. 2.

3D catalogs are best rendered as a single mesh with disconnected vertices in Edit mode. 3. 3.

Halo Materials are best used with a Emission shading value of 1.0 to create point sources for each catalog object. 4. 4.

A linear path is set up such that the camera will traverse 75% of the full width at half maximum (FWHM) of the width of the catalog’s radial distribution. This allows reasonably uniform coverage of the viewing sphere as the user moves through the 3D catalog space. 5. 5.

Single Halo materials are not supported in the Cycles engine; we use the standard Render engine provided with Blender. 6. 6.

The resolution output should be set to 3840 $\times$ 2160 pixels (Figure 9). The video output should be set to the MPEG with MPEG-4 encoding and a bitrate of 6000 to 8000 kilobits per second, depending on the length of the translation. A trade off can be made between a longer video and a slightly smaller bitrate.

4.3 Planetary Terrain Surface

With this scenario, we can render a panoramic view of the Martian shield volcano Olympus Mons. The camera path will encircle the geographic feature, allowing the user to pan the camera in any direction during the animation. The data discussed in Kent (2015) and Smith et al. (1999) can be used as an example, with layers used to build the terrain maps888http://astrogeology.usgs.gov/tools/map (Figure 10). As the camera revolves around the volcano, the altitude is changed along a predetermined track of revolution. Any kind of solid 3D model viewed externally is well suited to this visualization scenario.

Loading planetary surface images requires both satellite imaging as well as some form of altitude ranging. These can be layered as a displacement modifier and texture for uv-mapping. 2. 2.

Surface data can be rendered as as a subdivided surface plane in Edit mode with a deformation modifier. 3. 3.

A regular surface mesh material with an Emission shading value of 1.0 works best in this example. 4. 4.

A Bézier circle can be used as a path animation combined with an Empty mesh (null vector) that will allow the camera to point toward the surface during the video. The camera is attached to this animated path. The time frame of the path animation can determine the length of the visualization. 5. 5.

The user can choose between Cycles and Render; either will work well in the scenario. 6. 6.

The resolution output should be set to 3840 $\times$ 2160 pixels. The video output should be set to the MPEG with MPEG-4 encoding and a bitrate of 6000 to 8000 kilobits per second.

5 Summary

We have introduced a method for creating 360*∘* spherical panoramas with astrophysical data. The technique employs the 3D software package Blender and the Google Spatial Media module. The outlined principles of panoramic projection allow us to use these tools in concert for immersive data exploration. By moving within a full spherical view, a user can interact with astronomical images and 3D catalogs, models, and maps. The interaction works equally well in a traditional browser environment as well as physical movement with mobile devices. We provide examples for three scenarios that a user could build upon: a static celestial sphere, a linear fly-through, and a revolving orbit focusing on a single area or object.

Demonstration videos as well as sample files, Python scripts, and basic tutorials of Blender principles are available at http://www.cv.nrao.edu/$∼$bkent/blender

We gratefully acknowledge the careful reading by Tom Bridgman and Kel Elkins in improving the content and clarity of this work for the astronomical community.