This research provides a  proof of concept for a novel method of performing a regression task using convolutional neural networks to predict the solar irradiation value (kwh/m²) from images. 

Through synthetically generated data using ladybug for grasshopper in rhinoceros, around 30 datasets have been compiled for training two types of machine learning models – 1) TensorFlow/Keras CNN regression model and 2) ResNet50 pretrained model and tested on synthetic and real images. 

Image processing techniques using python’s library “openCV” have been applied on the images in order to measure the impact on the prediction, within each model.  

5 tests have been executed, iterating results from initially 20% accuracy till the final tests 97% accuracy for synthetic data prediction and 93% accuracy for real image prediction.  

This research aims to 

  • simplify the design-simulation-evaluation process, bringing radiation performance based design closer to the architecture practice and enabling more solar irradiation aware designs.
  • make a contribution to cooling down cities +  increase passive measures in architecture. 
  • contribute to a paradigm shift of ornaments in architecture from decoration to passive performance contributor. 

To create a technique  that  helps to provide an intuitive insight to climate  resilient design .

 

PROBLEM STATEMENT  

RELEVANCE

by 2050 about 70 % of the world’s population will live in cities. whilst global temperatures are rising towards +4°c, building facades are a main contributors to the urban heat island effect within cities. Increasing passive measures such as shading, reflectivity, thermal massing, adiabatic cooling can make strong contributions using our world resources more responsibly and make our cities more thermally resilient. 

 

HISTORIC RELEVANCE OF THE ORNAMENT

 

RELEVANCE OF THE ORNAMENT

 

RESEARCH OBJECTIVE

Partner thesis between daniyal tariq + clemens russ. commencing our seminar work of AIA: DATA ENCODING + ML from faculty: gabriella rossi, assistant:hesham shawqy 

Question: 

  • is it possible to predict the solar irradiation values (kWh/m²) of facades in the built environment from pictures with solar irradiation values as labels?

Hypothesis: 

  • Through pretraining a dataset via artificially generated and as built façade patterns, we are assuming that the high diversity of the dataset will be able to predict the solar irradiation values of a real building.  

 

STATE OF THE ART

 

DESIGN SPACE

The objective of our research was to simplify the design-simulation-evaluation process, bringing radiation performance based design closer to the architecture practice and encourage designers to a more solar irradiation aware designs practice. 

Initially the experiment was designed to be based on datasets of numerical values from image to geometry translation. applying the learnings from our “data encoding” seminar from the MaCAD at IaaC and use features such as orientation, solar irradiation per surface,  solar irradiation total, total surface area, facade noise ( displacements in x,y,z) for the prediction. (refer to 2) 

As this is the most computationally expensive experiment design version, a pivot towards the use of convolutional neural networks for running a regression task based on images has seemed to be a more useful approach.  

  • Least computationally heavy, 
  • Broad global computer vision research that it can benefit from, 
  • Future applications for the AEC field, in example: applying the research to google streetview Images and expand the experiment to whole city streets, understanding the cooling effects of the ornaments on whole streets, can be expanded further easily.

DATASETS

SOLAR IRRADIATION ANALYSIS – SETUP

 

DIVERSITY – MOVEMENT OF BRICKS

GROUND TRUTH DATA COLLECTION

Ground Truth

 

General: 

For the predictions a train_test_split of train_size=0.8, shuffle=True, random_state=2 was applied. additionally 10 ground truth images were investigated. all 10 wall samples stated before, have been measured, remodeled in rhino, simulated in ladybug for grasshopper, stored in CSV and compared to the prediction values. 

    • 5 x real images: taken from southern facades in vienna at around 2 p.m. in august + September under sunny conditions.  
    • 3 x laboratory images: images from real built walls M 1:10 in vienna at around 1 p.m. in  September, under sunny conditions.  due to time constraints the test setup was built on a southern terrace, with a children brick “playtoy” measure 1:10. 
    • 3 x laboratory images: images from real built wall. For this test we build a 1m x 1m test wall using standard bricks without mortar in Karachi , Pakistan  in open ground . the images were taken  at around 2pm in September  under sunny conditions 
  • 2 x real images from the internet: images that provide diversity in ornaments, that could hardly be found at a ground level in Vienna, under sunny conditions. 
  • 16 x rendered real walls: all these wall samples stated above, were rendered in the same manner, as the dataset, to get an understanding how the predictions would behave on real situations, but in a scene that it might understand. (refer to section “dataset” for the main parameters such as sun, lat, long, values) furthermore, 6 additional images from another dataset batch were tested. 

SYNTHETIC GROUND TRUTH

CONSTRAINTS 

In the process of iterating through the results , geometrical constraints have been identified, that might impact the correlation between the size of the wall, the wall surface area and the final simulated and therefore prediction values. as ladybugs output value “totalRadiation” gives a value in kwh/m² that is dependent on the meshsurface areas of the “gridsize”, walls that where simulated as solid geometries, with a higher amount of bricks, showed worse results, although the self shading effects of the façade was obviously better.

  • full geometries: full walls, built from solid “brick” meshes, with mortar joints.
  • single façade faces: the top surfaces, this means a hollowed version of the brick walls, without backsides

 

The single facade faces showed the least bias in the values, therefore was identified as acceptable ground truth (GT) for our experiment.

Furthermore, 2 kind of values have been tested for the prediction.

  • radiation result (LB): “The amount of radiation in kWh/m2 falling on the input test _geometry at each of the test points.
  • total radiation (LB): ” The total radiation in kWh falling on the input test _geometry. This is computed through a mass addition of results at each of the test points in kWh/m2 multiplied by the area of the face that the test point is representing.”

Examining of the data showed, that the “radiation results” are not dependent on the surface area around the “testpoints” of the “gridresolution”, the distribution of the values didn’t correlate well with the well with the dataset, therefore the prediction was around 8% inaccurate for the initial dataset + the test images. (not GT)

Whilst, the “total radiation” showed a well distributed dataset of values, which highly correlated with the final “dataset 4” by giving results of 2% inaccuracy.

 

IMAGE PROCESSING

 

 

Image processing general: 

OpenCV in python has been used align the rendered images and the real facade images to an equal ground. for this we’ve tested and compared the prediction results using histogram equalization and binary thresholding: 

  • histogram equalization (HIST)
  • contrast limited adaptive histogram equalization (CLAHE)
  • global thresholding v127 (BI-GLOBAL)
  • adaptive mean thresholding (ADAME)
  • adaptive gaussian thresholding (ADAGA)2
  • otsu’s binarization + gaussian blur. (GA-OTSU)

Both directions have been tested, training on processed images and the predictions on all the types listed above. (refer to section predictions + results)

DATASET –IMAGE PROCESSING

GROUND TRUTH – IMAGE PROCESSING

MODEL-CONVOLUTIONAL NEURAL NETWORK

The following two models have been examined for our experiment: 

  • Model_1: CNN regression model (TensorFlow/Keras):

For the prediction model, a kaggle example for predicting ages from images was used.  although the predictions in the example showed a 30 % inaccuracy, it was a good ground to start and expand from.

  • RES: ResNet50 pretrained model (TensorFlow/Keras api):

“ResNet-50 is a convolutional neural network that is 50 layers deep. ResNet, short for Residual Networks is a classic neural network used as a backbone for many computer vision tasks” 

MEASURING CRITERIA

For the prediction root mean squared error “RMSE” + Null/Baseline Model Test RMSE have been used to verify our predictions in two different ways. for all tests the lost function has been plotted. multiple datasets have been tested, only the most significant outcomes are shown in thesis project. 

PREDICTION

Dataset 1- AVALANCHE TEST: 
  • Tests showed a poor prediction rate of around 20% accuracy.

 Dataset 2- tests: 

  • Tests started to increase to learning rate tremendously to 98% accuracy.
  • Especially gray scaling and value scaling to integers showed positive results.
  • Kernel sizes did not seem to impact the predictions as much as we expected.

Dataset 3: 

  • Although datasets 1-3 were tested under imprecise conditions – 10 a.m. vienna, sunny, june, – the model understood the correlation between shadow and solar irradiation rate still well enough to predict around 98 %.
  • within this test setting the avalanche test setting was closed and predictions on real scenarios were examined.
  • resnet 50 – pretrained model implemented, showing 98% accuracy from the beginning.
  • increase of epochs from “early stopping”, patience 5, was tested until 100 epochs, but didn’t not show a tremendous difference in prediction rate.
  • dataset 1-3 were generated as “full brick geometries”

Dataset 4- final tests “VIENNA”

  • The final tests were examined under all learnings taken from the previous tests.
  •  predictions on synthetic data showed an accuracy from 96-98%,
  • real images without image processing from 75-93%
  • processed images from 65-93%.
  •  dataset distribution – solar irradiation value distribution was one of the biggest levers, for the predictions.
  • RESNET50- backbone and MODEL1 -tested on all test
  • dataset 4 was generated as “single facade faces” (refer to section ground truth – geometrical constraints discussion)

Dataset 5- final tests “KARACHI”

 

  • The final tests were added complexity to the model, by using real images of “laboratory” built walls 1x1m in karachi by daniyal.
  • datasets 4 + 5 were trained and predictions run on images GT 11,12,13 – KARACHI.
  • the outcomes were equal to our viennese datasets.
  • this means in conclusion, the model is able to understand and predict correlation between shadow + “totalRadiation” – solar irradiation values.
  • in case of expanding the model to other countries, the model needs to be trained on further data in order to predict the correct values for the lat. + long.

RESULTS

MEASURING CRITERIA

All prediction results have been compiled in an excel file . accuracies are calculated as stated below:

For the prediction root mean squared error “RMSE” + Null/Baseline Model Test RMSE have been used to verify our predictions in two different ways.

accuracies are calculated as per example:

 

RESULTS

Rendered images: 

Rendered image predictions showed not surprisingly after the intense examination of the model predictions average results around 95% for our real facades, rendered similar as per dataset.

Real images: 

The “real images” have been taken with “pixel 4a” smartphone, without any additional filters besides the google camera software auto filters (like auto focus, auto contrast , auto brightness). interestingly the RESNET 50 was able to predict “real images” from the initial dataset (no image processing applied) with an avg. accuracy around 94 %, due to its abilities to understand complexity + depths in images.

the flat model1 started predictions from 76%  from the initial dataset (no image processing applied) accuracy towards 86% from the initial dataset (with histogram equalization HIST applied)

 

Processed images: 

All image processings have been applied through pythons library openCV.

For the flat “model1” the GT datasets that showed the most “equalized” image data, performed the best by far. therefore image processing treatments

  • contrast limited adaptive histogram equalization (CLAHE) in combination with global thresholding (GLOBAL),
  • adaptive mean thresholding (ADAME).
  • adaptive gaussian thresholding (ADAGA), otsu’s binarization + gaussian blur. (GA-OTSU)

Performed in a range of 89% – 93% “average accuracy”. the the fittest champion of all testing showed to be the adaptive mean thresholding (ADAME), adaptive gaussian thresholding (ADAGA). not obvious the eye, all trainings from different treatments performed more or less “the best” on these two GT image processing ADAME + ADAGA.

For the complex “RESNET50” the GT datasets that showed the most diversity in information for instance depth in grayscale, performed the best by far. therefore image processing treatments like

  • initial dataset with no image processing.
  • histogram equalization (HIST)

Performed in a range of 92% – 94% “average accuracy”.

not surprisingly, the more the binary thresholding was applied, the worse the prediction rate of the resnet50 became. however, still outperforming the flat network “model1” with the worst prediction value around 80%.

 

 

RESULTS SUMMARY

 

CONCLUSION +DISCUSSION 

Conclusions: 

  • adaptive mean, adaptive gaussian thresholding and histogram equalization are a useful treatments for value prediction from images through a flat  network model.  
  • pretrained backbones hold a huge potential for this kind predictions
  • synthetic data has to be treated very carefully, especially for the GT

Discussion: 

  • computer vision holds an interesting and huge potential for the future of the AEC Tech sector. 
  • synthetic data is it too simple to generate huge amounts of data and to bias your results ?

Next steps: 

  • test with real GT data – infrared images + temperature sensors. 
  • test with new backbone “EfficientNetV2L” 
  • predict other values
  • create an online app for the prediction to empower others

experiments

 

ORNAMENT + CRIME II :predicting solar irradiation for self-shading facades through images is a project of IAAC, institute for Advanced Architecture of Catalonia developed in the Master of Advanced Computation in Architecture and Design 21/22

Student: Clemens Russ , Mohammad Daniyal Tariq  // Thesis Advisor: Gabriella Rossi