Background
Flight test measurements are typically acquired through a combination of adding discrete aeromechanical sensors to the structure and tapping into the communication busses between flight control computers. The amount and type of measurement sources depends on a combination of the maturity of the aircraft design under test and what specific test objectives need to be achieved. The growth of storage capacity for flight test data combined with falling costs of video cameras driven by other industries has led to a significant increase in the amount of video acquired from cockpit displays, control surfaces, and actuators. While many other industries have leveraged video cameras with machine vision techniques to monitor and control processes, the flight test industry has largely used the various video sources either as a secondary data source that is only inspected manually when measurements from other sensors conflict, or by manually acquiring measurements from the video through human inspection.
The objectives of this investigation were focused on determining if the addition of calibration techniques to standard image processing approaches can acquire flight test measurements from video of cockpit gauges with sufficient accuracy to be used in certain phases of flight test. The two key objectives were:
- Determine calibration-driven image processing approaches suitable to augment standard machine vision for extracting measurements from video of cockpit gauges
- Measure the accuracy and suitability of these techniques for use in flight test systems
Approach
We began by implementing various machine vision and image processing algorithms on our existing cockpit video testbed. We implemented the video processing using OpenCV which is an open-source computer vision library which facilitates both rapid development and multi-platform deployment. We captured video from the cockpit displays of our flight simulator to use as test inputs for our algorithm exploration and refinement. The flight simulator produces a data stream that corresponds to the data shown on the gauge displays. We used this as a truth source to assess the accuracy of detection and successively experimented with image processing and calibration techniques to improve measurement accuracy.
Accomplishments
This research project was successful in exploring the possibilities of acquiring numeric measurements, roll and pitch measurements, and light indicator states. This internal research project laid the groundwork of algorithms that could be expanded to future customer systems and was also successful in demonstrating which logic and available open source functions will or will not be useful, as well as helped to create direction and focus for future investigation. We were able to establish that calibration-driven image processing approaches were sufficient for a portion of the types of measurements that would be of interest to flight test applications. We also expect that augmenting these approaches with some adaptive learning approaches may achieve sufficient results for the measurement types that image processing only could not achieve.