Multimodality improves accuracy and equity in camera-based health monitoring

This large-scale study evaluated Shen AI’s multimodal approach for contactless measurement of heart rate (HR) and heart rate variability (HRV). By combining remote photoplethysmography (rPPG) with remote ballistocardiography (rBCG), the system dynamically selects the modality with the highest signal quality - a strategy designed to improve accuracy, robustness, and performance across diverse skin tones and environments. Using data from more than 5,000 participants, the report provides one of the most comprehensive validations to date of quality-gated, camera-based cardiovascular monitoring.

‍

Key insights

• High HR accuracy with rPPG alone: rPPG achieved near-identity HR agreement with pulse oximetry (MAE ≈ 0.37 bpm, R = 0.99) across 5,311 participants.
• HRV estimation is more challenging, but multimodal selection helps: rPPG reached MAE ≈ 6 ms for SDNN vs. 36 ms for rBCG; the quality-driven selector improved SDNN accuracy by ~7% overall and ~18% in phototype VI groups.
• Equity benefit for darker skin tones: rPPG error increased with darker phototypes (III–VI), while rBCG remained stable. The selector shifted weight toward rBCG when optical quality dropped, keeping HR error below 1.3 bpm even at Type VI.
• Signal-quality metrics strongly predict accuracy: Both modalities showed a clear, monotonic decrease in error as quality increased; decision-surface analyses validated a stable and interpretable boundary for modality switching

‍

Research summary

The following is an excerpt of the “Multimodal approach by Shen AI using remote photoplethysmography (rPPG) and ballistocardiography (rBCG) for heart rate and heart rate variability prediction” paper.

‍

Remote photoplethysmography (rPPG) and remote ballistocardiography (rBCG) are promising techniques for contactless cardiovascular monitoring. Both modalities can estimate heart rate (HR) and heart rate variability (HRV), but each has specific limitations: rPPG is sensitive to lighting and skin pigmentation, whereas rBCG is less affected by these factors but generally noisier. Shen AI has developed a multimodal, signal-quality–driven approach that dynamically selects or combines modalities based on confidence metrics to optimize performance.

We evaluated this strategy in a large cohort of 5,311 participants spanning a broad demographic spectrum (mean age 53.8 years, 64.7% female). Data acquisition involved simultaneous recording of facial video (rPPG), micro-movement signals (rBCG), and pulse oximetry (ground truth). Each 60-second recording was segmented, and HR and HRV (SDNN) were computed for each modality. Quality scores were assigned to every window, enabling the best-of algorithm to select the modality with higher expected accuracy.

Results showed that rPPG achieved near-identity agreement with the reference for HR (MAE ≈ 0.37 bpm, R = 0.99), while rBCG exhibited higher error overall (MAE ≈ 3.6 bpm, R = 0.82) but markedly improved in high-quality windows. For SDNN, rPPG again outperformed rBCG (MAE ≈ 6 ms vs 36 ms), though variability estimates were inherently more error-prone than HR. Applying the best-of strategy yielded measurable gains: HR error was reduced by ~3% and SDNN by ~7% overall. Stratification by Fitzpatrick skin type highlighted equity-relevant patterns: rPPG error increased progressively with darker phototypes (III–VI), whereas rBCG remained stable. The selector accordingly shifted weight toward rBCG in these subgroups, mitigating disparities. At phototype VI, SDNN accuracy improved by ~18%, while HR error remained below 1.3 bpm.

These findings demonstrate that multimodal, quality-gated selection effectively leverages the complementary strengths of rPPG and rBCG, ensuring robust and equitable contactless measurement of HR and HRV across a large, diverse population. Future work should extend validation to additional HRV endpoints (e.g., RMSSD, frequency-domain measures), unconstrained real-world settings, and clinical cohorts with arrhythmias or impaired perfusion.

Research details

Title: Multimodal approach by Shen AI using remote

photoplethysmography (rPPG) and ballistocardiography (rBCG) for heart rate and heart rate variability prediction
Authors: Przemysław Jaworski, Szymon Sobczak, Anna Drohomirecka, Tymoteusz Okupnik
Published November, 2025

‍

Read the full study here.

‍