Sleep graphs are inconsistent and have gaps(?) #442
Open
by 6arms1leg opened 5 years ago
·
28 comments
No Milestone
No Assignees
7 Participants
Notifications
Due Date
No due date set.
Dependencies
This issue currently doesn't have any dependencies.
Loading…
Reference in new issue
There is no content yet.
Delete Branch '%!s(MISSING)'
Deleting a branch is permanent. It CANNOT be undone. Continue?
There are three things I noticed about the sleep activity graphs which I think are wrong(?):
I observed the above with my old Mi Band 1.
I guess we should make the labels of deep and light sleep user-configurable (SCNR ;-)).
We had those labels inverted previously, but some people reported that they were wrong and Mi Fit would report them the other way round, so we toggled the meaning, which is still how it is today.
We had a few problems when we ported to a major new version of the charting library, so it might be that there are still some regressions.
It may very well be that there is no activity data for a while though. If you want to share a gadgetbridge log from a day where this happens (together with a screenshot of the charts view), please do. I could check if the band didn't provide any data or if there's something wrong.
As always, many thanks for your support!
Here is a Gadgetbridge log and the corresponding screenshots of the sleep screen in Gadgetbridge (one with the bar graph zoomed to the whole sleep part only and one zoomed to a ~30min gap):
GB_Sleep_Issue-442.zip
The Mi Band 1's sleep detection is awful, as I went to bed at around 02:00am, but I guess that's a problem with the device itself, not Gadgetbridge.
Ah OK, I checked the log and the Mi Band 1 indeed reports an "intensity" of 0.0 for long periods of time that match the gaps of the bar graph in Gadgetbridge. So Gadgetbridge just displays them as gaps in the bar graph but actually takes them into account for the sleep pie chart.
If that's the case, both graphs are consistent – just the representation with gaps in the bar graph is confusing (maybe it would be better to distinguish between "no data" and "0.0 intensity"? BTW, why isn't the latter interpreted as "Not Worn"?)
This renders points two and three of the initial issue post obsolete, with only the first question remaining.
Thanks for the log and screenshots!
The whole recognition of deep sleep and light sleep with a simple band is a bit like black magic (or charlatanry, if you prefer ;-))
Have a look at the wikipedia articles for example
https://en.wikipedia.org/wiki/Rapid_eye_movement_sleep
https://en.wikipedia.org/wiki/Non-rapid_eye_movement_sleep
The former (REM) sleep includes paralysis of the body, so this should correlate with (very) little intensity values.
The latter (NREM) does not include paralysis, and is characterized as "deep sleep", at least in stages 3/4.
This might explain low intensity values during light sleep and higher intensity values during deep sleep.
I'm sure there is room for improvement in how we display sleep and activity. Ideas and suggestions welcome :-)
RE: 0.0 intensity value vs "not worn": there are probably different ways for the band to distinguish them, one of them being the heartrate measurements.
@cpfeiffer
I think in common language when someone is in "deep sleep" he looks like a rock. No one would say he is in deep sleep because he is moving.
To avoid the confusion, if you think some band might be capable of detecting NREM stages 3/4 which I doubt, we shoud use the terms "restful sleep" and "sleep" or something
@ashimokawa agree.
I believe there's more logic involved on the app side than just counting the light sleep/deep sleep epochs:
https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233173323
@ashimokawa definitions.net has "deep sleep" as "Artificially induced hibernation in humans for the purpose of long distance space travel." So that's what I do half of the night? 👽 😝
@cpfeiffer Not sure how detailed the data from different devices are. Do they just have "markers" for deep vs light sleep – or allow for more/all phases? In the latter case, maybe GB should offer a hypnogram instead of that circle?
@ashimokawa My naive understanding of deep sleep is that the deeper your sleep is, the harder it is to get you to wake up, and the more disoriented you are.
German Wikipedia says similar things.
I never said that you'd be in deep sleep when you move 😀
What I meant was that deep sleep does not mean "no movement". I contrast, some other, non-deep-sleep phases are related to "no movement". Hence, the intensity values actually correlate to this.
@rplevka Totally agree. do you want to join us in finding an appropriate algorithm?
@IzzySoft AFAIK there's no further classification. I doubt these could be accurately calculated from such devices anyway.
@cpfeiffer yes, thought so.
Btw: regression. Just updated to 0.14.4, and my sleep graphs are gone again. At least those for the last night; funnily the ones from the night before are still there. Guess I have to wait for 0.14.5 to bring them back (same happend with 0.14.2, where 0.14.3 brought them back)... And yes, I was wearing the watch, and it was connected all day long.
Izzy you should try installing the ocean survey pebble app and trigger a sync (it's enough to enter and leave the ocean survey app).
The fact that pebble sends activity data (or not) and when it's not something we can influence. Same for the GB version it has no influence: it just happens that we release often and things seem to be related.
@danielegobbetti any pointer as where to find that, so I can give it a try? My Google-Fu just picked up links to the SDK (not being a dev, I have no dev env setup and thus cannot compile that), but no installable
.pbwor the like. If nowhere public, a mail is welcome as well :)@cpfeiffer,
sure, I'll do my best
@danielegobbetti regression solved, partly. Last night's data did show up and looks alright, but the night before is rather lost (just showed ~1h deep sleep, nothing else – and I know I've slept more than that). Nevermind, I guess that's why it's still labeled "Alpha", right? 😝
Nevertheless: I'd still welcome that watch app you've mentioned. Step on my toe if it's attached somewhere in the Wiki and I've just been too blind to see it 🙀
@cpfeiffer - regarding the gaps -
isn't there one more (or even more) activity code we don't recognize? perhaps there's one more counting to deep sleep or a light sleep(i've read some articles about light/deep sleep and learned that NREM consists of 3 stages).I've compared the chart with the actual data exported and browsed using sqlite browser and found out the gaps are actually bars of deep sleep with intensity: 0. So it only looks like a gap. Perhaps there could be some minimum height for a bar so it doesn't look like a gap too much? Or, set the color of the whole cell (on entire Y-axis) for that time epoch, same as the official app does (with e.g. white bar in the foreground to visualize the intensity)
Also, I've found that the code assigns 2 activitiy types to the same code (was that intentional?):
https://github.com/Freeyourgadget/Gadgetbridge/issues/450
..looks to me like the ACTIVITY was supposed to be bound to no. 1, not -1
Yes, the bar height is intensity. We actually had a minimum height some time ago... maybe that was lost at some point. I think that the visualization itself could be improved, though. E.g. by giving all bars the full height and adding the intensity as a line chart overlay. Not sure if that would be better, would need to try or have a mockup. Further suggestions welcome :-)
Regarding activity types, good question, I don't have an answer to at the moment. Needs some digging through git logs, which Android Studio is not so good at.
Last night I got a call and had to work starting at 12:30 AM until about 2:30 AM. When I downloaded the data from my MiBand1S this morning, here is what I saw (apparently I was Deep Sleeping through work!):
Deep and Light really are reversed.
@gadgetguy08 Please check your call history. Maybe you just had a bad dream 😜 But yes, in that case there should be some "light sleep" to account for dreaming, right?
No update in more than 2 years: can we consider this solved, and close the issue – or is anything left to do first?
Dark blue lines on my device (mi-band 1s) still indicate activity/motion and correspond to my being awake in bed, but still get designated as 'deep sleep'. So it continues to provide inaccurate results.
Thanks for the update, @gadgetguy08 – and apologies for a solution still not being available. For the records: Using a Pebble time Steel, I can confirm "gaps" in so far that I've never seen time frames marked "not worn" – though of course I cannot wear my PTS while it's charging.
The core team is somewhat "reduced" currently, so unfortunately it won't be fixed overnight 😢
I'd see the issue of "not worn" time periods embedded in sleep periods also strongly related to this issue. My sleep plots look weird, because GB kinda claims that my Mi band gets lost and reattached during night without me waking up, see attached screenshot.
To my understanding it is obvious here that the band was worn and the embedded "notworn" period should be added in the overall sleep time.
I guess the heart rate data gathered during these "not worn" times are just interpolated (which I think should not be done for any period longer than a few minutes).
BTW: using a Mi Band 3 (FW: 2.4.0.32 on HW V0.23.18.49) with Gadgetbridge 0.47.2 from fDroid, running on Lineage 17 (Android 10).
My plot looks somewhat similar to this previously reported issue: #1856
I can't see that from the graph. I think for the same reason the band can't either. How are you wearing the band?
I wear the band relatively tight, but also tried to vary this as users have raised the question here whether this could have an effect. For me it didn't.
For illustrating what I mean with an "obviously impossible not-worn period" in the given plot let me add a screenshot of the same time in the activity plot mode, zooming a bit in:
GB concludes that (in the middle of the night!), the state changes from light sleep to notworn and then to deep sleep. With such a sequency of state transitions at that time of the day, I'd propose to have this changed by means of some plausibility correction.
As mentioned before, the heart rate in that notworn phase is obviously only interpolated, by zooming into the HR plot it looks very much like a polynomial of 3rd order connecting the ends of the neighbouring data periods. Unless GB is doing this somewhere, the band must have done the interpolation itself.
Nonono, that's Mi Band itself.
I don't see any sense adding comments to many issues all around the bugtracker here, there is limited number of people who will read them and respond...
Anyhow, i think i see what @TaaviE sees as well:
Before and after the "not worn" period, there is an activity (indicated as
intensityon the band). I can imagine, that what the band sees is the following:As i mentioned before - they switched to postprocessing the data, because as you can see, the logic can be flawed. There have been attempts to do the same, but it is actually difficult for many reasons.
This is my view, but i can be wrong of course.
The only reasonable solution to me seems to be manual adjustment option. This is a topic for an another single issue.
As the interpolation curve shape is only hard to see in the previous plot, let me give a zoomed example from another day, seeing most of the night as not-worn phases (the sleep plot concluding far-off that I slept less than 4hrs that night), intermitting with sleep phases, and the heart rate plot interval curve looking like cubic polynomial interpolation:
https://codeberg.org/Freeyourgadget/Gadgetbridge/src/branch/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/activities/charts/AbstractChartFragment.java#L620
https://codeberg.org/Freeyourgadget/Gadgetbridge/src/branch/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/activities/charts/AbstractChartFragment.java#L472
I agree that approaches for sorting out these implausible states are worth an issue of its own. Having some hope that using some data smoothing plus restricting the acceptable state transitions might make an improvement possible without any manual editing, but could end up in the UI as just a config checkbox like "use experimental data filtering", I will try to play a bite with the raw data from the exported sqlite database, hopefully arriving at something helpful.