RGB-D and IMU wearable capture combines first-person color observations, depth, and inertial measurements. For robot learning, its value is the relationship between these signals: what was visible, how the viewpoint moved, and where surfaces were observed during a task.

The sensor names alone do not establish usable alignment or learning performance. Timing, calibration, depth provenance, and task context determine what a downstream system can infer. This article explains those requirements and relates them to the published ZL Capture V1 and L1 profiles.

What each stream contributes

  • RGB: appearance, texture, object state, and visible hand-object interactions. It cannot reveal a contact that is hidden from the camera.
  • Depth: geometric information associated with an image or viewpoint. Its units, reference frame, valid range, and method of production must be known.
  • IMU: acceleration and angular velocity at the sensor. Combined with visual observations and a suitable estimator, these measurements can support motion estimation.
  • 6DoF head pose: a position and orientation in a defined reference frame. It describes the head or device, not the hands, robot joints, or contact forces.

Keep measured streams distinct from derived outputs. Pose estimates and processed depth need provenance and validity information so a training pipeline can handle tracking loss, uncertainty, or changes in the processing method.

Native depth and synthetic depth need different checks

A depth output should document how it was obtained. Sensor-derived depth still involves processing and can contain missing or erroneous values. Synthetic depth also needs a description of its inputs, method, metric scale, and expected failure cases. A dense-looking map is not proof of accurate geometry.

Test depth against reference distances in the region where the task happens. Include object edges, small parts, movement, and the materials used in the collection program. In RealSense's depth-quality testing guidance, lighting and reflective surfaces are examples of conditions that affect evaluation. Those observations motivate testing; they do not establish the performance of another device.

ZL Capture V1 lists native depth, while L1 lists synthetic depth. The public profiles do not specify the underlying depth algorithms or accuracy. Confirm those details for the proposed configuration before using either output as geometric supervision or evaluation reference.

Synchronization is a relationship between clocks

A camera frame and an IMU sample may have different timestamps, rates, and delivery delays. The pipeline needs to know when each observation occurred in a shared time domain. Host arrival time alone can include transport and buffering delays.

Project Aria's timestamp documentation provides an example of sensors aligned to device time. For any system under evaluation, ask for the equivalent timestamp definitions, clock mapping, and treatment of resets. This is a general engineering reference, not a statement about ZL Capture's implementation.

Consider an illustrative recording at 30 camera frames per second and 200 IMU samples per second. The intervals are about 33.3 ms and 5 ms, so there is no one-to-one pairing of frames and samples. Preserve the original timestamps and define interpolation or integration in the consuming pipeline. The 30 fps value here is an example, not a ZL Capture frame-rate specification.

Timing errors become more visible during motion. A hypothetical 10 ms offset at 100 degrees per second corresponds to one degree of rotation between observations. Use motion representative of the task when measuring alignment, and retain gaps rather than silently presenting missing data as continuous recording.

Calibration connects the sensor coordinate frames

Camera intrinsics describe how image coordinates relate to camera rays. Extrinsics describe the position and orientation of one sensor relative to another. Camera-to-IMU calibration connects visual observations with inertial motion; time calibration estimates their temporal relationship.

The Kalibr multi-IMU calibration documentation treats spatial relationships, time offsets, and IMU model parameters explicitly. For an exported episode, request the calibration used, its version, coordinate-axis definitions, units, and transform direction. Validate the result after configuration changes that could alter sensor relationships.

For RGB-D data, also establish whether depth is registered to an RGB image or remains in its own camera frame. Overlaying the streams requires the appropriate geometry and timing. Agreeing array dimensions alone is insufficient.

From wearable observation to robot-learning data

Imagine a person picking up a part and placing it in a tray. RGB can show object appearance and the sequence of activity. Depth can help describe the visible geometry. IMU and head pose help characterize viewpoint motion. Together, these observations can support representation learning, reconstruction, or action understanding, depending on the model and labels.

A robot policy may additionally require end-effector targets, gripper state, joint actions, or force information. Those are not supplied by head pose alone. They require separate measurement, annotation, estimation, or a validated mapping method. Define that path before treating human demonstrations as policy-training episodes.

Keep task boundaries, outcomes, sensor validity, and processing versions with the data. An episode should allow a researcher to trace a derived observation back to its source and distinguish measured values from inferred labels.

Applying this to ZL Capture

V1 combines dual HD RGB, dual VGA grayscale fisheye, dual 1000 Hz IMUs, native depth, and 6DoF head pose. L1 combines dual HD RGB, a 200 Hz IMU, synthetic depth, and 6DoF head pose in the lightweight product profile. These configurations offer different starting points for the visual and inertial requirements of a collection program.

Zerolaw's ZL Core platform covers cleaning, slicing, synchronization, calibration, and fusion in the processing pipeline. ZL Corpus organizes reusable datasets with episode schemas and provenance. Confirm the required raw streams, derived outputs, and export contract as part of the engagement.

For device selection, use the egocentric hardware buying guide. For additional sensing or integration requirements, bring a representative task and expected outputs to Zerolaw Multimodal Solutions. A useful first evaluation is a complete sample episode checked in your own training workflow.