Deep Learning Model Matches Expert Agreement in Sleep Arousal Scoring

A fully convolutional neural network trained to detect arousals from polysomnograms (PSGs) achieved higher median agreement with 10 independent expert scorers than the scorers achieved among themselves, according to a study published in Sleep.¹

Study Design Addresses Key Evaluation Gap

Arousals, described as brief wake-like shifts in electroencephalogram (EEG) activity, are central to diagnosing sleep fragmentation in conditions such as obstructive sleep apnea (OSA), insomnia, and periodic limb movement disorder, but manual scoring of these events shows notably poor interrater reliability compared with sleep staging or respiratory event scoring. Prior automated scoring models have typically been validated against a single scorer in datasets drawn from the same institution as the training data, limiting insight into real-world generalizability.

This is not the research group's first attempt at automating a labor-intensive PSG scoring task.² As The American Journal of Managed Care^® (AJMC^®) previously covered, a related deep learning model from the same lab automated sleep staging—rather than arousal detection—in patients with suspected OSA; that model performed accurately overall, though its accuracy declined as OSA severity increased, a limitation the authors of the current study appear to have anticipated by testing across multiple independent scorers and centers rather than a single reference standard.

To address this gap, researchers from the University of Eastern Finland, Kuopio University Hospital, and collaborating centers built a model using an encoder-decoder architecture with channelwise attention and multiscale feature extraction, drawing on EEG, chin electromyogram, and nasal pressure signals.¹ The model was trained on 1847 PSGs from the Multi-Ethnic Study of Atherosclerosis (MESA) and tested in 205 MESA hold-out recordings as well as an independent cohort of 50 PSGs from the Sleep Revolution (SR) project, each annotated separately by 10 expert scorers across 7 sleep centers in Europe and Australia.

Model Outperforms Interscorer Agreement

In the MESA test set, the model reached an event-by-event F1-score of 0.77, a 1-second κ of 0.67, and an arousal index intraclass correlation coefficient (ICC) of 0.83. In the more rigorous, independent SR dataset, the model achieved median event-by-event F1-score, 1-second κ, and arousal index ICC of 0.61, 0.50, and 0.66, respectively, each exceeding the corresponding median values achieved by the 10 manual scorers against one another (0.57, 0.46, and 0.59).

Against the majority consensus of at least 5 of the 10 scorers, model performance was even stronger, with an event-by-event F1-score of 0.74, essentially matching the scorers' own median agreement with that majority standard (0.73). Most arousals flagged by the model (70.8%) were also identified by the scorer majority, and most arousals the model missed (89.1%) had been scored by only a minority of reviewers, suggesting the model's errors track the same ambiguous cases that divide human scorers rather than representing systematic failures.

Consistent with earlier interrater studies in the same cohort, both the model and manual scorers performed best in stage N3 sleep and worst during wakefulness, when arousal-related EEG changes are harder to distinguish from background activity.

EEG Power Patterns Support Physiological Validity

To probe whether the model's decisions reflect genuine neurophysiological signal rather than pattern-matching artifacts, researchers compared EEG spectral power in the theta, alpha, and greater than 16 Hz frequency bands, the same frequencies specified in American Academy of Sleep Medicine arousal-scoring criteria, between arousals the model flagged and those it did not.

Across all sleep stages, band power was significantly higher in model-detected arousals. This pattern held even when arousals were stratified by degree of manual scorer agreement, reinforcing that the model's outputs align with the physiological basis of the scoring rules rather than an artifact of training data.

“As the model’s performance is on par with manual scorers in an independent dataset, this model can reliably be used to score arousals from different sleep recordings in clinical and research environments,“ the authors wrote.

Implications for Practice

By contrast, they noted limitations, including the model's exclusion of limb movement signals, a 1-second scoring resolution that limits precise onset/offset detection, and the absence of validation in pediatric or diverse clinical populations. Still, they concluded that a model performing on par with trained human scorers in an independent, multicenter dataset could reduce the resource burden of manual PSG review and support larger-scale or longitudinal studies of sleep fragmentation that are currently difficult to conduct with manual scoring alone.

The push toward automating sleep-disorder diagnostics extends beyond arousal scoring itself.³ A recent secondary analysis of the STAR trial, also covered by AJMC, found that hypoglossal nerve stimulation was associated with a substantial reduction in hypoxic burden, a separate physiologic metric capturing the cumulative area under the oxygen desaturation curve during respiratory events, among patients with OSA, including some classified as nonresponders by traditional apnea-hypopnea index criteria.

Taken together, these developments point to a broader shift in sleep medicine toward physiologically grounded, automatically derived metrics, whether of arousal burden or hypoxic burden, that may better capture disease severity and treatment response than conventional frequency-based indices alone.

References

1. Pitkänen H, Huttunen R, Tashakori M, et al. Automated deep learning-based arousal detection complies with ten expert scorers in unseen data. Sleep. Published online July 16, 2026. doi:10.1093/sleep/zsag191
2. Gavidia M. Deep learning model found to accurately, automatically detect sleep staging for patients with suspected OSA. AJMC. February 3, 2020. Accessed July 17, 2026. https://www.ajmc.com/view/deep-learning-model-found-to-accurately-automatically-detect-sleep-staging-for-patients-with-suspected-osa
3. Grossi G. Hypoglossal nerve stimulation linked to reduced hypoxic burden in OSA. AJMC. July 7, 2026. Accessed July 17, 2026. https://www.ajmc.com/view/hypoglossal-nerve-stimulation-linked-to-reduced-hypoxic-burden-in-osa