CONCRETE PUNK
CONCRETE PUNK is a project developed during the Studio II Seminar of the MRAC-21/22-program. This project aimed at redefining computer vision and machine learning applications in construction industry. Through this project we intend to create a automated/semi-automated system that scans and identifies cracks in existing concrete elements using novel computer vision, machine learning techniques further to which a Ultrasonic pulse velocity test performed using robotic system over the targeted areas detected by machine learning and as a result structural state of the element can be determined and healed using biological methods. Its a work in progress project.
CONTEXT
It started with first site visit at Les Tres Xemeneies, which is an old thermal power plant. Intention of the visit to brainstorm potentials about this building and the area around. Constructed in early 1970’s, this ‘new’ thermal power plant was going to meet the needs of growing electricity demand of its time. But it’s bad reputation for pollution brought it to shutdown in 2011.
During explorations about this power plant we found noticeable damages to its structure which triggered the interest in the use of concrete worldwide and its impact on the environment. Inspired from the term “SOLARPUNK” , exploration started with a vision to heal and extend the life of existing concrete and minimize/avoid pouring new concrete.
Facts– Out of 600 million tons of waste in U.S, 85% is concrete and asphalt. >70% gets recycled, 90% in EU, but still more than 100 million tons gets disposed.
STATE OF THE ART
Detection
Standard way to analyze concrete is by using destructive and non destructive methods. Non destructive methods are quick and cause no harm or extraction of samples from the existing concrete.
Ultrasonic pulse velocity test is an in-situ nondestructive test to check the quality of concrete. In this test, the strength and quality of concrete or rock is assessed by measuring the velocity of an ultrasonic pulse passing through a concrete structure. This test is conducted by passing a pulse of ultrasonic through concrete to be tested and measuring the time taken by pulse to get through the structure. Higher velocities indicate good quality and continuity of the material, while slower velocities may indicate concrete with many cracks or voids. UPV can be performed in 3 different ways.
Concrete Degradation
What causes cracks in concrete? There are 3 main causes:
Major problem in concrete is cracking and corrosion of rebar. It is very crucial to detect the problems in concrete element and heal them proactively. Below image shows the relation between the increasing cost of concrete repair vs time. If problem in concrete are addressed at early stage, the cost of repair can be reduced and the life of concrete can be extended. 
This time of early stage can also be termed as “Golden Time”.
Healing
This leads us to looking at the state of the art on methods of healing concrete. A more popular method uses epoxy to fill in the cracks and voids to regain the structural strength. it’s been used by several industries at large scale.
Epoxy has huge environmental and health impacts. some of the health Problems that Epoxy resins cause:
- Dermatitis
- Severe Irritation and Chemical Burns
- Respiratory Irritation
- Asthma
- Allergic Reactions
RESEARCH HYPOTHESIS
First step, manual scanning of target structure using Lidar or photogrammetry drone/robot to generate a mapping trajectory.
Next step, to deploy the robot and initiate the custom trained crack detection model to detect cracks and rusts.The machine learning model segments each image it captures during detection The detections are categorized in 3 main categories:
i. Structural crack
ii. Non Structural crack
iii. Rust
Once the model detects a crack or rust, it classifies them in one of the above mentioned categories and segments them with a specific color for each category and maps it in the 3d model along with the segmentation color.
Further, a robot equipped with UPV device approaches each crack location and perform UPV test. The reading of each crack is linked with respective locations and based on the UPV readings the categorization is re-performed which as a result give a correct evaluation of the health of the concrete structure.
RESEARCH METHODOLOGY
Project implementation has been in multiple stages:
i. Document
Multiple images taken for training the machine learning model.
ii. Choose detection –
How to detect? Out of the many available multiple object detection algorithms we chose Mask RCNN for object detection and segmentation. It takes image as an input and give a detected bounding box and a color segmentation over the target object. It makes a good fit for this project.
What to detect? Concrete cracks, rust are the target detection’s and are organized in following categories:
iii, Detection and segmentation-
a. Creating the training set– All the images documented as explained in earlier part of this blog are used for training the machine learning model. Multiple training’s with different image set were perform in order to achieve a trained model with accurate detection.
First training was performed with photogrammetry image set.It contains 168 images. 
The output from using this set of images in the model training resulted in a poor detection’s and hence, a new dataset was recorded. Detailed explanation about problems faced is explained in later part of this blog.
Second training was performed from scratch by using the new dataset which contains images with closeup of the targeted problems which is crucial in order to train the model to understand specific areas of interest. This new set contains 225 images which are mix of images showing rust, structural cracks and non structural cracks(based of visual identification)
b. Labeling – Next step is labeling each image manual using a labeling tool such as makesens.ai. Labeling is a process of highlighting each problem/area of interest with a polygon.
In the end, it gives a JSON file with all the polygon coordinates in the each image which is used by machine learning model to learn.
c. Training – First authentic release version of MRCNN is used for this project with some custom changes and additional codes to suit the project.After the dataset labeling is completed, it is arranged in the format as required by the MRCNN model architecture.
Above image shows the dataset arrangement, changes required to be made in the codes. Post changes, a pre trained COCO dataset weight file(H5 file) is used to perform transfer learning and start the training.
d. Validation – After training completion a H5 weight file is generated which is the final trained model and is used to perform detection of cracks and rusts. Prior to performing detection on actual scenarios, model is first evaluated by some random dataset or part of training dataset images to check the performance. We used some images from the training dataset for validation.
Above are the results from the validation process. Left image, validation output from first training set model which had high imprecision’s. Right image, validation results from second training set model.
Second trained model is not accurate but has satisfactory output.
e. Model in action – Latest trained model can be used on any concrete surfaces for detection. 5 locations selected to put model in action
Model configured to give following outputs:
- resized normal image from each frame.
- segmented images which has colored segmentation performed by the Machine learning model over each frames.
- masked version of the segmented image (background image remove and with segmentation only)
All the above mentioned 3 types of outputs are useful in the next step of the project.
iv. Mapping –
The original images are processed in a photogrammetry software(Agisoft Metashape) which are further replaced by masked images and segmented images to generate a dense point cloud with detection area colored similar to segmented images from machine learning model. Below is the method and workflow of our execution.
Below images are dense cloud generated from detection by the trained mask_RCNN model (left) and edge detection from openCV (right).
This set reflected different consistency of detection, implying the necessity to design the navigation path of the camera (on drone or a ground robot) depending on the limitations of the detection. Accounted in the limitation include surrounding noise, color contrast between cracks and rust against the original color, scale of cracks.
LIVE ACQUISITION
We are aiming for live acquisition of data, essentially combining the three stages of our methodology (documentation, detection, and mapping). We used the Astra orbbec to combine: machine learning detection – depth image – point cloud output
Astra is a depth camera which has 2 camera streams: color and depth. We used the color stream (uint8) to run the detection. After generating the mask on the detected image, we make a list of pixel IDs. These pixel IDs will extract depth values (uint16) for the segmented areas only. Once extracted, we plot the points in 3D space using Open3D.
The next would be use the plotted 3d space to deploy robot for performing UPV testing and healing.
UPV
For the first prototype, a peizo and piston is been used. Piston is triggered over one side of the crack and peizo takes the reading on the other side of the crack. Time take by the sound to travel from piston to the peizo is recorded using a custom code. Velocity of the sound can be calculated using
(v = x / t) speed = distance / time
First steps involve manual reading but the idea is to have a robot equipped with UPV device which can perform autonomous navigation and take UPV readings.
UPV readings of each location(crack and rust) are mapped over the dense point cloud. Based on the UPV reading crack locations are recolored. UPV readings provides a better understanding of the structural health of the concrete, analysis of crack depth.
FURTHER DEVELOPMENT
1. Reducing the detection process from three flights to one automated flight.
2. Placing sensor boxes to collect environmental data for better healing.
3. Material Check-up- This workflow that we developed to heal concrete can be applied to many materials. Helping these materials to achieve their lifespan and perhaps extend it.
4. Platform- Via a platform clients could upload the model of their buildings, where a detection would run and give information about the situation of the building.The Platform could offer:
i. Crack detection
i. Detection of missing elements
iii. Approximate calculation of cost of repair
CONCRETE PUNK // STUDIO II is a project of IAAC, Institute for Advanced Architecture of Catalonia developed at Master in Robotics and Advanced Construction seminar in 2021/2022 by:
Students: Beril Serbes, Yeo Jeong Kim, ?pek Attaro?lu, Arpan Mathe
Faculty: Aldo Sollazzo, Keith Kaseman
Faculty Assistant: Oriol Arroyo, Maria Espina