DATA PROCESSING and VISUALIZATION

 

Workshop2.2: Multispectral Cognification Brief – Data Processing and Visualisation

 

 

 

Point Cloud Generation

The photogrammetry process consisted of obtaining a big quantity of photographs of the environment we wanted to work at, to create a point cloud as a light digital version of the space.

We used two different software to generate the pointcloud of the environment, a free open source (Meshroom) and a closed one with license Metashape. 

Photogrammetry + point clouds 

with Meshroom

With meshroom the workflow was really smooth and user friendly to understand every phase of the process. Nevertheless, many of the pictures from the slot database would not work for similar features, darkness or low contrast

First we imported the photographs on the software, then the matching features of the images will be extracted to generate the pointcloud on the ‘Structure from motion’ step.

A further step to get the mesh of the model, would need to calculate depths of the camera distance to every point in the cloud on the following steps of the process.

This is the point cloud cleaned using Open3d to remove the overlaying and noisy points on from the cloud, explained in the last step.

Merge with Thermal Camera Point Cloud

The goal of this part was to repeat the process with the pictures taken with the thermal camera and include them as a branch of the workflow held by the software as a second texture on the last step.

The images below show from left to right the original RGB photograph, the thermal image also in RGB mode with colors transferred to infrared color scale, and finally the 1 channel (grayscale) thermal image.

We found a problem on the raw thermal camera images, as they became cropped (shown below), missing a lot of information from the surrounding of the image, making it impossible to overlap and find matching features from the pictures.

Multispectral photogrammetry + point clouds with Agisoft Metashape

In parallel to Meshroom, we also tried to create a thermal point cloud using AgiSoft Metashape.  

Two workflows were explored:

Workflow A: create two point clouds; one from RBG images and one from thermal images.  Combine point clouds to create a multispectral point cloud.

Workflow B: create point cloud from RGB images, texture this point cloud with thermal images to create multispectral point cloud. 

Workflow A

A point cloud was created from RGB images in Metashape by adding photos, aligning the photos, building a dense cloud from the depth map, building a mesh and finally building texture. By importing point confidence when building the dense cloud, it was also possible to visualize it on a basis of point confidence. 

Making the point cloud from thermal images was not as straightforward.  In our first attempt, we added 1,200, each 16-bit and of a resolution of 160 x 120. Only 3 of the 1200 were able to be aligned by Metashape.

Next, we added a different set of thermal images, taken on a different camera (the FLIR 96), though of the same location.  There were 820 of these images, each of resolution 640 x 480.  Unlike the set before, they were 8-bit images, but the alignment process was still very poor: only 5 of the 820 images could be aligned by Metashape. 

On reflection, it’s not surprising that attempting to align the thermal images was problematic, as by the nature of their subject thermal images capture gradients of heat, rather than hard edges of solid / air.  It is these edges and boundaries that assist the software in aligning the images to make a point cloud. Also, thermal images are usually very low resolution, as was this stock – whereas Metashape recommends that images for processing are at least 3-5 megapixels. 

Workflow B

Texturing a point cloud with a different set of images requires the replacement images to be identical in a) size and b) name.  

We prototyped this workflow with a set of only 20 RGB images, which were aligned and a (sparse!) point cloud, depth cloud, and then mesh constructed from them. The corresponding thermal images were renamed by hand, and the path changed to texture the point cloud from the thermal images, instead of the initial RGB images.  Metashape returned an error: ‘image size mismatch’.  Resolving the size mismatch was decided to be unfeasible in this workshop, as the thermal images were not only of a different resolution, but also a slightly different frame, and therefore could not be overlaid onto the RGB images without more time problem-solving than we had available. 

Instead, we proved the workflow by creating our own alternative set of images to replace the thermal images. 

A point cloud was created from 193 images using the add photos / align photos / build dense cloud / build mesh.

Python scripts were written to invert the colours of the RGB images, apply to all images, and save in a separate directory to the parent images.  We now had two sets of images with the same size and same names, but with differences between them.  We successfully changed the path of the RGB point cloud images to these inverted images, and textured the point cloud to create an inverted colour point cloud in lieu of our multispectral point cloud. 

RGB point cloud with texture built from inverted RGB images

 

Point Cloud Processing

Global Registration + ICP (Iterative Closest Point)

Merging point clouds via rough and tight alignment

In this process the partially overlapping point clouds are aligned:

First rough alignment through Global Registration in Open3D 

Second tight aligned using ICP in Open3D

     

Point Cloud cleaning

Downsample –

 

Using Open3d the assembled point cloud is downsampled to remove outlying and overlapping points.

Segmentation – 

Segmentation based on color:

import open3d as o3d

import numpy as np

def read_pointcloud(path):

def segment_colours(point cloud):

ind = np.where(colors[:, 1] > 0.1)

red_cluster = pc.select_by_index(ind_list)

def save_pointcloud(pc, name):

def main():

AEC Applications

 

Existing Buildings

  • determine overall scope of renovation
  • identifying problem areas to be addressed in design
  • provide benchmark data for design performance

New Buildings

  • building envelope evaluation & commissioning
  • construction monitoring
  • pre-occupancy evaluation
  • building systems commissioning

Spatial Planning

For buildings being repurposed, acquiring comprehensive multispectral data could help design more energy-efficient layouts.

Designing-in the most appropriate layouts informed by thermal data would result in permanent savings throughout the new life of the building.

Thermal Comfort

Temperature differences during a day or a year affect the thermal comfort of spaces.

Mapping the thermal changes in time might help obtain ideal temperatures in a space by

 Identify opportunities for targeted design interventions:

  • Solar shades
  • Vegetation
  • Envelope remediation

Detecting Architectural Elements

Architectural elements such as lighting, doors and windows can be detected and mapped through thermal data. 

In addition to ?dentif?cation through edges, thermal bridges or energy losses can be identified at these borders and targeted. 

 

Alberto Martínez López

Andrea Lizette Nájera Rodríguez

Christopher Booth

Grace Boyle

?pek Attaro?lu

Shamanth Thenkan Siddaramu