How to evaluate Shen AI’s accuracy?
How to evaluate Shen AI’s accuracy?
Many customers perform their own validation before deploying Shen AI. That's a good practice: it helps you understand how the SDK performs in your product, on your target devices, and with your users.
To get meaningful results, it's important to compare Shen AI against appropriate reference measurements and to keep the testing conditions consistent. Small differences in methodology can have a much bigger impact on the results than the SDK itself.
Start with the right reference
The quality of your evaluation depends on the quality of your reference device.
For heart rate and heart rate variability (HRV), we recommend comparing Shen AI with an ECG-based reference. For blood pressure, use a clinically validated blood pressure monitor.
Consumer wearables such as smartwatches, fitness bands, and smart rings can be useful for everyday health tracking, but they should not be treated as the ground truth when evaluating measurement accuracy. They are estimates themselves, and differences between two estimation methods do not necessarily indicate that one is incorrect.
Compare measurements correctly
Looking only at the final heart rate or HRV value can be misleading.
For heart rate and HRV, the most reliable comparison is based on individual heartbeat intervals rather than summary values. When comparing results, make sure both systems are measuring the same time period. Even a small timing mismatch can make two otherwise accurate measurements appear different.
If you're performing a technical evaluation, your implementation should allow you to export heartbeat interval data so it can be compared with your reference measurements.
Keep the measurement conditions consistent
A fair comparison starts with a consistent measurement environment. For best results:
- measure in a seated position,
- use even, comfortable lighting,
- keep the face fully visible,
- position the camera directly in front of the face,
- remain still, avoid talking, and breathe normally throughout the scan.
Following the same conditions for every participant helps reduce variability that isn't related to the measurement itself.
Take extra care when evaluating blood pressure
Blood pressure is naturally more variable than heart rate. Even under ideal conditions, it changes from moment to moment.
Before taking a reference measurement, allow the participant to rest for at least three minutes. The cuff should already be in place before the measurement begins, since movement during preparation can affect the reading. Whenever possible, perform the Shen AI measurement and the cuff measurement at the same time.
A careful reference measurement is just as important as the Shen AI measurement itself. Blood pressure is naturally variable, and factors such as posture, arm position, movement, stress, or talking can all affect the reading. Following best practices helps create a fair comparison and reduces differences caused by the measurement process rather than the technologies themselves.
Focus on repeatability, not a single measurement
One measurement rarely tells the whole story. When evaluating Shen AI, collect multiple measurements under the same conditions and look for consistent agreement with your reference device. This provides a much more reliable picture of performance than comparing isolated readings.
If you're planning a formal validation study or need guidance on evaluation methodology, our team can help you design an appropriate testing protocol based on your use case.
