A research group at the University of Klagenfurt has designed a real-time capable drone based on object-relative navigation using artificial intelligence. Also on board: a USB3 Vision industrial camera from the uEye LE family from IDS Imaging Development Systems GmbH.
Inspection of critical infrastructure using intelligent drones
Case Study from | IDS Imaging Development Systems GmbH
The inspection of critical infrastructures such as energy plants, bridges or industrial complexes is essential to ensure their safety, reliability and long-term functionality. Traditional inspection methods always require the use of people in areas that are difficult to access or risky. Autonomous mobile robots offer great potential for making inspections more efficient, safer and more accurate. Uncrewed aerial vehicles (UAVs) such as drones in particular have become established as promising platforms, as they can be used flexibly and can even reach areas that are difficult to access from the air. One of the biggest challenges here is to navigate the drone precisely relative to the objects to be inspected in order to reliably capture high-resolution image data or other sensor data. A research group at the University of Klagenfurt has designed a real-time capable drone based on object-relative navigation using artificial intelligence. Also on board: a USB3 Vision industrial camera from the uEye LE family from IDS Imaging Development Systems GmbH.
As part of the research project, which was funded by the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK), the drone has to autonomously recognize what is a power pole and what is an insulator on the power pole. It is supposed to fly around the insulator at a distance of three meters and take pictures. "Precise localization is important such that the camera recordings can be compared across multiple inspection flights," explains Thomas Georg Jantos, PhD student and member of the Control of Networked Systems research group at the University of Klagenfurt. The prerequisite for this is that object-relative navigation must be able to extract so-called semantic information about the objects in question from the raw sensory data captured by the camera. Semantic information makes raw data, in this case the camera images, "understandable" and makes it possible not only to capture the environment, but also to correctly identify and localize relevant objects.
In this case, this means that an image pixel is not only understood as an independent color value (e.g. RGB value), but as part of an object, e.g. an isolator. In contrast to the classic GNNS (Global Navigation Satellite System), this approach not only provides a position in space, but also a precise relative position and orientation with respect to the object to be inspected (e.g. "Drone is located 1.5m to the left of the upper insulator").
The key requirement is that image processing and data interpretation must be latency-free so that the drone can adapt its navigation and interaction to the specific conditions and requirements of the inspection task in real time.
Thomas Jantos with the inspection drone - Photo: aau/Müller
Semantic information through intelligent image processing
A USB3 uEye LE serves as the quadcopter's navigation camera
How much AI fits into a small quadcopter?
The camera, on the other hand, delivers perfect basic data straight away, as the tests in the university's own drone hall show. When selecting a suitable camera model, it was not just a question of meeting the requirements in terms of speed, size, protection class and, last but not least, price. "The camera's capabilities are essential for the inspection system's innovative AI-based navigation algorithm," says Thomas Jantos. He opted for the U3-3276LE C-HQ model, a space-saving and cost-effective project camera from the uEye LE family. The integrated Sony Pregius IMX265 sensor is probably the best CMOS image sensor in the 3 MP class and enables a resolution of 3.19 megapixels (2064 x 1544 px) with a frame rate of up to 58.0 fps. The integrated 1/1.8" global shutter, which does not produce any 'distorted' images at these short exposure times compared to a rolling shutter, is decisive for the performance of the sensor. "To ensure a safe and robust inspection flight, high image quality and frame rates are essential," explains Thomas Jantos. As a navigation camera, the uEye LE provides the embedded AI with the comprehensive image data that the on-board computer needs to calculate the relative position and orientation with respect to the object to be inspected. Based on this information, the drone is able to correct its pose in real time.
The IDS camera is connected to the on-board computer via a USB3 interface. "With the help of the IDS peak SDK, we can integrate the camera and its functionalities very easily into the ROS (Robot Operating System) and therefore into our drone," explains Thomas Jantos. In addition, IDS peak enables efficient raw image processing and simple adjustment of parameters such as auto exposure, auto white balancing, auto gain and image downsampling.
To ensure a high level of autonomy, control, mission management, safety monitoring and data recording, the researchers use the source-available CNS Flight Stack on the on-board computer. The CNS Flight Stack includes software modules for navigation, sensor fusion and control algorithms and allows the autonomous execution of reproducible and adjustable missions. "The modularity of the CNS Flight Stack and the ROS interfaces enable us to seamlessly integrate our sensors and the AI-based 'state estimator' for position detection into the entire stack and thus realize autonomous UAV flights. The functionality of our approach is being investigated and developed using the example of an inspection flight around a power pole in the drone hall at the University of Klagenfurt," explains Thomas Jantos.
Information about CNS Flight Stack
Information about the drone hall
Visualisation of the flight path of an inspection flight around an electricity pole model with three insulators in the research laboratory at the University of Klagenfurt
Precise, autonomous alignment through sensor fusion
The camera captures raw images at 50 fps and an image size of 1280 x 960px. "This is the maximum frame rate that we can achieve with our AI model on the drone's onboard computer," explains Thomas Jantos. When the camera is started, an automatic white balance and gain adjustment are carried out once, while the automatic exposure control remains switched off. The EKF compares the prediction and measurement and corrects the estimate accordingly. This ensures that the drone remains stable and can maintain its position autonomously with high precision.
Electricity pole with insulators in the drone hall at the University of Klagenfurt is used for test flights
Outlook
With corresponding features, IDS cameras are helping to set a new standard in the autonomous inspection of critical infrastructures in this promising research approach, which significantly increases safety, efficiency and data quality.
Client
Camera
uEye LE - the cost-effective, space-saving project camera
Model used:
USB3 Vision Industrial camera U3-3276LE Rev.1.2
Camera family: uEye LE
About IDS Imaging Development Systems GmbH
IDS Imaging Development Systems GmbH is a leading manufacturer of industrial cameras and pioneer in industrial image processing. The owner-managed company develops modular concepts of powerful and versatile USB, GigE and 3D camera as well as models with Artificial Intelligence (AI). The almost unlimited range of applications covers multiple non-industrial and industrial sectors of equipment, plant and mechanical engineering. The AI image processing platform IDS NXT is extremely versatile and opens up new areas of application where classic rule-based image processing reaches its limits. With visionpier, IDS operates an online marketplace that brings together suppliers of ready-made image processing solutions and interested end customers in a targeted manner.
Since its foundation in 1997 as a two-man company, IDS has developed into an independent, ISO and environmentally certified family business with around 350 employees. The headquarters in Obersulm, Germany, is both a development and production site. With branches and representative offices in the USA, Japan, South Korea, the UK, France and the Netherlands, the technology company is also globally represented.
The content & opinions in this article are the author’s and do not necessarily represent the views of RoboticsTomorrow
IDS Imaging Development Systems Inc.
World-class image processing and industrial cameras "Made in Germany". Machine vision systems from IDS are powerful and easy to use. IDS is a leading provider of area scan cameras with USB and GigE interfaces, 3D industrial cameras and industrial cameras with artificial intelligence. Industrial monitoring cameras with streaming and event recording complete the portfolio. One of IDS's key strengths is customized solutions. An experienced project team of hardware and software developers makes almost anything technically possible to meet individual specifications â€" from custom design and PCB electronics to specific connector configurations. Whether in an industrial or non-industrial setting: IDS cameras and sensors assist companies worldwide in optimizing processes, ensuring quality, driving research, conserving raw materials, and serving people. They provide reliability, efficiency and flexibility for your application.
Other Articles
Warning: Illegal string offset 'post_title' in /raid/home/robotics/staging/smarty/templates_c/5734ba46ff408fa62ff141da9d993db832fff2ae_0.file_content_article.tpl.php on line 120
A
Warning: Illegal string offset 'post_excerpt' in /raid/home/robotics/staging/smarty/templates_c/5734ba46ff408fa62ff141da9d993db832fff2ae_0.file_content_article.tpl.php on line 123
A
More about IDS Imaging Development Systems Inc.
Comments (0)
This post does not have any comments. Be the first to leave a comment below.
Featured Product
