D435i IMU Bug On ARM64: Pipeline Start Order Matters

by Alex Johnson 53 views

Introduction

In this comprehensive article, we delve into a peculiar bug encountered with the Intel RealSense D435i camera's IMU (Inertial Measurement Unit) pipeline on ARM64-based systems, specifically the RK3566 running Ubuntu 20.04. This issue manifests as the IMU pipeline failing to start unless the pipeline initialization order is reversed. This behavior is inconsistent with Windows, where the pipeline start order does not affect the IMU functionality. This article aims to provide a detailed account of the problem, the steps to reproduce it, and potential insights into the underlying causes.

The Intel RealSense D435i is a versatile depth camera known for its ability to capture both depth and inertial data. It's widely used in robotics, computer vision, and various other applications. However, when deploying this camera on embedded systems like the RK3566, certain platform-specific issues can arise. This article specifically addresses a pipeline initialization issue that affects the IMU functionality on ARM64 systems. We will explore the environmental setup, the steps to reproduce the bug, the expected and actual behavior, and the minimal reproducible code. Additionally, we will discuss potential causes and request guidance on debugging and workarounds. Understanding this issue is crucial for developers working with RealSense cameras on ARM64 platforms to ensure reliable and consistent performance. By examining the specifics of this bug, we hope to shed light on broader compatibility challenges that can occur when integrating advanced sensors with embedded systems.

Environment

To provide context, here's the environment setup where the issue was observed:

  • Camera: Intel RealSense D435i
  • Firmware: 5.15.1
  • OS: Ubuntu 20.04.6 LTS
  • Kernel: 4.19.232
  • Architecture: ARM64 (aarch64)
  • Board: LubanCat RK3566
  • CPU: 4× Cortex-A55 (1.8GHz)
  • Python: Python 3.8.10
  • librealsense: Built from source
  • pyrealsense2: Built from source

Both librealsense and pyrealsense2 were manually compiled on the device to ensure compatibility and optimal performance. This setup is commonly used in embedded systems and robotics applications where the D435i camera is integrated for depth sensing and inertial measurements. The specific versions of the software components are critical in understanding the bug, as it may be related to certain versions of the libraries or the kernel. The manual compilation of librealsense and pyrealsense2 allows for customization and optimization for the ARM64 architecture, but it also introduces the possibility of configuration issues. Therefore, documenting the environment in detail is crucial for reproducing the bug and finding a solution.

Summary of the Issue

The core issue arises when using two separate pipelines:

  1. Pipeline A: Color + Depth
  2. Pipeline B: IMU (Accelerometer + Gyroscope)

On Windows, everything functions as expected, but on ARM64 (RK3566), a peculiar behavior emerges:

  • If Pipeline A (color+depth) is started first, followed by Pipeline B (IMU), the IMU pipeline fails to start, reporting "No device connected."
  • Reversing the start order (IMU first, then color+depth) resolves the issue, and both pipelines function correctly.

Furthermore:

  • The IMU pipeline operates normally when started independently.
  • The issue is specific to scenarios involving two pipelines and is exclusive to ARM64 platforms.

This behavior suggests a potential conflict related to device backend, USB, or pipeline resource management, which is specific to ARM platforms. The inconsistency between Windows and ARM64 environments indicates that the issue is likely tied to the underlying hardware architecture or the way resources are managed on ARM64 systems. The fact that the IMU pipeline works in isolation but fails when combined with the color+depth pipeline highlights a potential resource contention problem. This could be due to how the USB subsystem or the RealSense SDK handles multiple streams on ARM64. Understanding the specific constraints and limitations of the ARM64 platform is essential to diagnosing and resolving this issue. Further investigation may involve examining the USB device enumeration process, the allocation of resources for different streams, and the interaction between the RealSense SDK and the ARM64 kernel.

Steps to Reproduce

To replicate the issue, follow these steps:

  1. Connect the D435i camera (via USB 3) to the RK3566 board.
  2. Create two pipelines:
    • Pipeline A: Color + Depth
    • Pipeline B: Accelerometer + Gyroscope (IMU)
  3. Start the pipelines in the following order:
    • A (color+depth) → B (IMU)
    • Result: The IMU pipeline fails to connect to the device.
  4. Reverse the order:
    • B (IMU) → A (color+depth)
    • Result: Both pipelines run successfully.
  5. Running the IMU pipeline alone always works.

These steps clearly demonstrate the dependence on the pipeline start order on the ARM64 platform. The failure of the IMU pipeline to connect when started after the color+depth pipeline suggests a potential initialization conflict or a resource allocation issue. By reversing the order, the IMU pipeline is able to claim the necessary resources before the color+depth pipeline, which points to a priority or locking mechanism that is not correctly handled on ARM64. The fact that the IMU pipeline works in isolation further reinforces the idea of a resource contention problem when multiple pipelines are active. This issue is critical for applications that require simultaneous depth and inertial data, as it can lead to unreliable or non-functional IMU readings. Developers need to be aware of this behavior and implement workarounds, such as ensuring the IMU pipeline is initialized first, to avoid potential issues. Additional testing and debugging, potentially involving lower-level USB diagnostics, may be necessary to fully understand and resolve this problem.

Expected Behavior

Ideally, both pipelines should initiate successfully, irrespective of the start order. This is the consistent behavior observed on Windows, which serves as the benchmark for expected functionality. The RealSense D435i camera is designed to support simultaneous depth and inertial data streaming, and the SDK should handle the initialization and resource allocation for multiple pipelines in a consistent manner across different platforms. The expectation is that the start order of the pipelines should not influence their ability to connect and operate. This is crucial for ensuring portability and ease of development, as developers should be able to write code that functions correctly regardless of the underlying platform. The discrepancy between the expected behavior and the actual behavior on ARM64 highlights a potential bug in the platform-specific implementation of the RealSense SDK or the USB driver. Understanding and addressing this issue is essential for maintaining the reliability and usability of the RealSense camera in embedded systems and other ARM64-based applications. Further investigation into the resource management and synchronization mechanisms of the SDK on ARM64 is warranted to identify the root cause of this inconsistency.

Actual Behavior

On ARM64, the IMU pipeline fails to start if the color+depth pipeline is initiated first. The error message typically encountered is "Pipeline start failed: No device connected." However, the specific stack trace may vary depending on the exact timing and system state. This error indicates that the IMU stream is unable to establish a connection with the camera, which effectively prevents the IMU data from being captured. The fact that the error only occurs when the color+depth pipeline is started first suggests a potential conflict in device access or resource allocation. The system might be failing to properly enumerate or initialize the IMU device when it is accessed after the color and depth streams have been established. This could be due to various factors, such as USB bandwidth limitations, driver issues, or synchronization problems within the librealsense library. The error message "No device connected" is a common indicator of hardware-level connectivity issues, but in this case, it seems to be a software-related problem triggered by the pipeline start order. To fully diagnose this behavior, it may be necessary to examine the USB device enumeration process, the driver logs, and the internal state of the RealSense SDK during pipeline initialization. Understanding the exact sequence of events that lead to the failure is crucial for developing a robust solution.

Error Example

The error message observed is:

Pipeline start failed: No device connected

(The exact error may differ; full logs can be provided if needed.)

Minimal Reproducible Code (Simplified)

The following Python code snippet demonstrates the issue:

import pyrealsense2 as rs
import sys

ctx = rs.context()
dev = ctx.query_devices()[0]
serial = dev.get_info(rs.camera_info.serial_number)

# IMU pipeline
pipeline_imu = rs.pipeline()
config_imu = rs.config()
config_imu.enable_device(serial)
config_imu.enable_stream(rs.stream.accel)
config_imu.enable_stream(rs.stream.gyro)

# Main pipeline (color + depth)
pipeline = rs.pipeline()
config = rs.config()
config.enable_device(serial)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

# --- Key part: start order matters on ARM64 ---
try:
    # This order fails on ARM64 (IMU cannot connect)
    pipeline.start(config)
    pipeline_imu.start(config_imu)
except Exception as e:
    print("Pipeline start failed:", e)
    sys.exit(1)

print("Both pipelines started successfully")

Running this code results in:

$ python3 test.py
Pipeline start failed: No device connected

However, if the start order is swapped:

pipeline_imu.start(config_imu)
pipeline.start(config)

the output is:

$ python3 test.py
Both pipelines started successfully

This clearly illustrates that the IMU works normally, and the color + depth streams also function correctly when the IMU pipeline is started first. This minimal reproducible code encapsulates the essence of the bug, making it easier for developers to understand and debug the issue. The code sets up two separate pipelines, one for color and depth streams and another for IMU streams. By starting the pipelines in different orders, the code demonstrates the specific conditions under which the bug occurs. The use of try-except blocks allows for graceful handling of the error and provides a clear error message when the IMU pipeline fails to start. This simplified example is a valuable tool for reproducing the bug in different environments and for testing potential fixes. Developers can use this code as a starting point to further investigate the issue and to develop workarounds for their applications.

Additional Notes

  • The USB3 cable and port have been confirmed to be functional.
  • No USB hubs are being used.
  • rs-enumerate-devices detects IMU streams normally.
  • The issue occurs in both single-threaded and multi-threaded tests.
  • The behavior is 100% reproducible on RK3566 but has never appeared on Windows.

These additional notes provide further context and eliminate some potential causes of the issue. The confirmation of a functional USB3 connection and the absence of USB hubs rule out basic connectivity problems. The fact that rs-enumerate-devices can detect IMU streams indicates that the device is being recognized by the system at a lower level. The persistence of the issue in both single-threaded and multi-threaded scenarios suggests that it is not related to threading or synchronization issues within the application code. The 100% reproducibility on RK3566 and the absence of the issue on Windows strongly indicate a platform-specific problem. This could be due to differences in the USB stack, the kernel drivers, or the librealsense implementation on ARM64. The consistency of the behavior makes it easier to debug, as developers can reliably reproduce the issue in the specified environment. These observations narrow down the potential causes and help to focus the investigation on platform-specific aspects of the RealSense SDK and the ARM64 system.

Request

This issue might stem from:

  • The ARM64 backend (rsusb / v4l2 ordering).
  • Resource locking between the motion module and the RGB module.
  • Device claim behavior on embedded Linux platforms.

Any guidance on further debugging or potential workarounds would be greatly appreciated. The specific areas of concern highlight potential areas for investigation. The ARM64 backend, including the rsusb and v4l2 components, handles the communication with the RealSense camera and the video streams. Differences in the implementation or configuration of these components on ARM64 could lead to the observed behavior. Resource locking between the motion module (IMU) and the RGB module (color and depth) could cause conflicts if the modules are not properly synchronized. The device claim behavior, which is the process of allocating and managing access to the camera's resources, might be different on embedded Linux platforms compared to Windows. This could result in the IMU pipeline failing to claim the necessary resources when started after the color+depth pipeline. To further debug this issue, it may be necessary to examine the source code of librealsense, analyze the USB traffic using tools like Wireshark, and investigate the kernel logs for any relevant error messages. Understanding the interaction between these components and the specific characteristics of the ARM64 platform is crucial for finding a solution. Any insights or suggestions from the RealSense community or Intel developers would be invaluable in resolving this issue.

Conclusion

In conclusion, the issue with the D435i IMU pipeline on ARM64 systems presents a significant challenge for developers aiming to utilize the full capabilities of the RealSense camera on embedded platforms. The dependence on pipeline start order to ensure IMU functionality is a critical bug that needs addressing. Through detailed environmental setup, reproducible steps, and minimal code examples, this article has aimed to provide a comprehensive understanding of the problem. The potential causes identified, such as ARM64 backend issues, resource locking, and device claim behavior, offer a starting point for further investigation. Addressing this issue is crucial for ensuring consistent and reliable performance of RealSense cameras on ARM64 platforms.

For more information on Intel RealSense technology, visit the official Intel RealSense website.