Multimodal approach by Shen AI using remote photoplethysmography (rPPG) and ballistocardiography (rBCG) for heart rate and heart rate variability prediction
Shen AI’s multimodal rPPG+rBCG system delivers high-accuracy, equitable heart rate and HRV monitoring across 5,000+ participants, improving SDNN by up to 18% in darker skin tones.
Multimodality improves accuracy and equity in camera-based health monitoring
This large-scale study evaluated Shen AI’s multimodal approach for contactless measurement of heart rate (HR) and heart rate variability (HRV). By combining remote photoplethysmography (rPPG) with remote ballistocardiography (rBCG), the system dynamically selects the modality with the highest signal quality - a strategy designed to improve accuracy, robustness, and performance across diverse skin tones and environments. Using data from more than 5,000 participants, the report provides one of the most comprehensive validations to date of quality-gated, camera-based cardiovascular monitoring.
Key insights
- High HR accuracy with rPPG alone: rPPG achieved near-identity HR agreement with pulse oximetry (MAE ≈ 0.37 bpm, R = 0.99) across 5,311 participants.
- HRV estimation is more challenging, but multimodal selection helps: rPPG reached MAE ≈ 6 ms for SDNN vs. 36 ms for rBCG; the quality-driven selector improved SDNN accuracy by ~7% overall and ~18% in phototype VI groups.
- Equity benefit for darker skin tones: rPPG error increased with darker phototypes (III–VI), while rBCG remained stable. The selector shifted weight toward rBCG when optical quality dropped, keeping HR error below 1.3 bpm even at Type VI.
- Signal-quality metrics strongly predict accuracy: Both modalities showed a clear, monotonic decrease in error as quality increased; decision-surface analyses validated a stable and interpretable boundary for modality switching
Research summary
The following is an excerpt of the “Multimodal approach by Shen AI using remote photoplethysmography (rPPG) and ballistocardiography (rBCG) for heart rate and heart rate variability prediction” paper.
Remote photoplethysmography (rPPG) and remote ballistocardiography (rBCG) are promising techniques for contactless cardiovascular monitoring. Both modalities can estimate heart rate (HR) and heart rate variability (HRV), but each has specific limitations: rPPG is sensitive to lighting and skin pigmentation, whereas rBCG is less affected by these factors but generally noisier. Shen AI has developed a multimodal, signal-quality–driven approach that dynamically selects or combines modalities based on confidence metrics to optimize performance.
We evaluated this strategy in a large cohort of 5,311 participants spanning a broad demographic spectrum (mean age 53.8 years, 64.7% female). Data acquisition involved simultaneous recording of facial video (rPPG), micro-movement signals (rBCG), and pulse oximetry (ground truth). Each 60-second recording was segmented, and HR and HRV (SDNN) were computed for each modality. Quality scores were assigned to every window, enabling the best-of algorithm to select the modality with higher expected accuracy.
Results showed that rPPG achieved near-identity agreement with the reference for HR (MAE ≈ 0.37 bpm, R = 0.99), while rBCG exhibited higher error overall (MAE ≈ 3.6 bpm, R = 0.82) but markedly improved in high-quality windows. For SDNN, rPPG again outperformed rBCG (MAE ≈ 6 ms vs 36 ms), though variability estimates were inherently more error-prone than HR. Applying the best-of strategy yielded measurable gains: HR error was reduced by ~3% and SDNN by ~7% overall. Stratification by Fitzpatrick skin type highlighted equity-relevant patterns: rPPG error increased progressively with darker phototypes (III–VI), whereas rBCG remained stable. The selector accordingly shifted weight toward rBCG in these subgroups, mitigating disparities. At phototype VI, SDNN accuracy improved by ~18%, while HR error remained below 1.3 bpm.
These findings demonstrate that multimodal, quality-gated selection effectively leverages the complementary strengths of rPPG and rBCG, ensuring robust and equitable contactless measurement of HR and HRV across a large, diverse population. Future work should extend validation to additional HRV endpoints (e.g., RMSSD, frequency-domain measures), unconstrained real-world settings, and clinical cohorts with arrhythmias or impaired perfusion.
Research details
Title: Multimodal approach by Shen AI using remote
photoplethysmography (rPPG) and ballistocardiography (rBCG) for heart rate and heart rate variability prediction
Authors: Przemysław Jaworski, Szymon Sobczak, Anna Drohomirecka, Tymoteusz Okupnik
Published November, 2025
More blog posts
Sagittis et eu at elementum, quis in. Proin praesent volutpat egestas sociis sit lorem nunc nunc sit.
Facial video photoplethysmography for measuring average and quasi-instantaneous heart rate
Facial vPPG matched ECG with up to 1-bpm precision - even for 4-second, quasi-instantaneous heart rate readings - showcasing the potential of accurate, contactless monitoring.
Why camera-based monitoring is outpacing wearables in telehealth
What if your biggest remote patient monitoring challenge isn't clinical - it's getting patients to actually use the devices?
Reducing claims costs and improving patient navigation in insurance with AI health monitoring
AI-driven virtual triage helps insurance providers reduce claims costs and improve patient navigation. With instant, device-free health checks for markers like blood pressure and BMI, insurers can assess risk digitally, prevent unnecessary ER visits, and guide policyholders toward the right level of care — efficiently and proactively.