Embedded vision components are ever-popular and being incorporated into a plethora of applications. What all these applications have in common is the need to pack more functionality into tight spaces.
Often, it is advantageous for these systems to make decisions on the edge. To enable such systems, including the ability to prototype quickly, Teledyne FLIR has introduced the Quartet Embedded Solution for TX2 module. This customised carrier board enables easy integration of up to 4 x USB3 machine vision cameras at full bandwidth.
Quartet Embedded Solution for TX2 module
It includes the NVidia Jetson deep learning hardware accelerator and comes pre-integrated with Teledyne FLIR’s Spinnaker SDK. Often, it is also very advantageous for these systems to make decision on the edge especially in inspection, mobile robotics, traffic systems, and various types of unmanned vehicles.
To highlight what the Quartet with Spinnaker SDK pre-installed can enable, Teledyne FLIR has explained steps taken in developing an ITS (traffic systems) inspired prototype, running four simultaneous applications, three of which use deep learning for:
- Application 1: Licence plate recognition using deep learning.
- Application 2: Vehicle type categorisation using deep learning.
- Application 3: Vehicle colour classification using deep learning.
- Application 4: See through windshield (past reflection & glare).
Hardware & Software Components
SOM for processing: New Teledyne FLIR Quartet carrier board for TX2 includes:
- 4x TF38 connectors with dedicated USB3 controllers.
- Nvidia Jetson TX2 module.
- Pre-installed with Teledyne FLIR’s powerful and easy-to-use Spinnaker SDK, to ensure plug and play compatibility with Teledyne FLIR Blackfly S board-level cameras.
- Nvidia Jetson deep learning hardware accelerator allows for complete decision-making systems on a single compact board.
Cameras & Cables:
- 3x standard Teledyne FLIR Blackfly S USB3 board-level cameras utilising the same rich feature set as the cased version applied to the latest CMOS sensors and for seamless integration with Quartet.
- 1x custom camera: Blackfly S USB3 board level camera with Sony IMX250MZR polarised sensor.
- Cables: TF38 FPC cables allowing power and data to be transmitted over a single cable to save on space.
Lighting: LED lights to provide sufficient illumination to avoid motion blur for the licence plates.
Licence plate recognition using deep learning
Teledyne FLIR deployed an off-the-shelf Licence Plate Detection (LPDNet) deep learning model from Nvidia
For licence plate recognition, Teledyne FLIR deployed an off-the-shelf Licence Plate Detection (LPDNet) deep learning model from Nvidia, in order to detect the location of the licence plates. To recognise the letter & numbers, they used the Tesseract open-source OCR engine. The camera is a Blackfly S board level 8.9 MP colour camera (BFS-U3-88S6C-BD) with the Sony IMX267 sensor.
Teledyne FLIR limited the region of interest for licence plate detection, in order to speed up performance and applied tracking to improve the robustness. The output includes bounding boxes of the licence plates together with the corresponding licence plate characters.
Vehicle type categorisation using deep learning
For vehicle type categorisation, using transfer learning, Teledyne FLIR trained their own deep learning object detection model for the three toy cars used, namely SUV, sedan and truck. They collected approximately 300 training images of the setup, taken at various distances and angles. The camera is a Blackfly S board level 5 MP colour camera (BFS-U3-51S5C-BD) with the Sony IMX250 sensor.
Teledyne FLIR annotated the bounding boxes of the toy cars, which took approximately 3 hours. They also performed transfer learning to train their own SSD MobileNet object detection model, which took around half a day on an Nvidia GTX1080 Ti GPU. With the GPU hardware accelerator, the Jetson TX2 module can perform deep learning inference efficiently and output bounding boxes of the cars together with the corresponding vehicle types.
Vehicle colour classification using deep learning
For vehicle colour classification, Teledyne FLIR ran the same deep learning object detection model as above, to detect the cars, followed by image analysis on the bounding boxes to classify its colour.
The output includes bounding boxes of the cars together with the corresponding vehicle colours. The camera is a Blackfly S board level 3 MP colour camera (BFS-U3-32S4C-BD) with the Sony IMX252 sensor.
See-through windshield (past reflection & glare)
Glare reduction is critical for traffic-related applications, such as seeing through a windshield to monitor HOV lanes
Glare reduction is critical for traffic-related applications, such as seeing through a windshield to monitor HOV lanes, check for seatbelt compliance and even check for using their phones, while driving. For this purpose, Teledyne FLIR made a custom camera, by combining a Blackfly S USB3 board level camera with the 5MP polarisation Sony IMX250MZR sensor.
This board-level polarisation camera is not a standard product, but Teledyne FLIR is able to swap in different sensors easily to offer custom camera options to showcase its glare removal functionality. They simply streamed the camera images via Teledyne FLIR’s SpinView GUI, which offers various ‘Polarisation Algorithm’ options, such as quad mode, glare reduction mode, to show the glare reduction on a stationary toy car.
Overall system optimisation
While each of the four prototypes worked well independently, Teledyne FLIR noticed that the overall performance was quite poor, when all the deep learning models were running simultaneously.
Nvidia’s TensorRT SDK provides a deep learning inference optimiser and runtime for Nvidia hardware, such as the Jetson TX2 module. They optimised their deep learning models using the TensorRT SDK, resulting in around 10x performance improvement.
On the hardware side, Teledyne FLIR attached a heat sink onto the TX2 module
On the hardware side, Teledyne FLIR attached a heat sink onto the TX2 module, in order to avoid overheating, as it was quite hot with all the applications running. At the end, they managed to achieve good frame rates with all four applications running together - 14 fps for vehicle type identification, 9 fps for vehicle colour classification, 4 FPS for automatic number plate recognition and 8 FPS for the polarisation camera.
Plug-and-play compatibility
Teledyne FLIR developed this prototype within a relatively short period of time, thanks to the ease of use and reliability of the Quartet Embedded Solution and Blackfly S board-level cameras.
The TX2 module with pre-installed Spinnaker SDK ensures plug-and-play compatibility with all the Blackfly S board-level cameras, which can stream reliably at full USB3 bandwidth via the TF38 connection.
Nvidia provides many tools to facilitate development and optimisation on the TX2 module. The Quartet is now available for purchase online on the official company website, as well as through their offices and global distributor network.