Inaccurate light/deep sleep detection #250

Closed
by bigretromike opened 5 years ago · 28 comments
bigretromike commented 5 years ago (Migrated from github.com)
Owner

Im not sure if this is mi band 1s related or not but...
Im using the GB for 2 days now - with few yey and nay...
But in the sleep data I have mine sleep chart reverse with compare to mi fit that I used before flashing phone.
currently i Have 7:26 h sleep, with 6:25 deep sleep and 1:01 light sleep. This been allways the oposite in mi fit - so im not sure if this is reversed or the data from mi band have reverse standard... I doube that I had good night ;-)

Im not sure if this is mi band 1s related or not but... Im using the GB for 2 days now - with few yey and nay... But in the sleep data I have mine sleep chart reverse with compare to mi fit that I used before flashing phone. currently i Have 7:26 h sleep, with 6:25 deep sleep and 1:01 light sleep. This been allways the oposite in mi fit - so im not sure if this is reversed or the data from mi band have reverse standard... I doube that I had good night ;-)
Owner

see the comment I left in #257

if you have resources related to actimetry algorithms to recognise deep sleep please post them to #232 or our wiki!

see the comment I left in #257 if you have resources related to actimetry algorithms to recognise deep sleep please post them to #232 or our wiki!
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

Like you post, if its inverted it would look ok.
For me its ONLY inverted: 6:39 = 0:54 Light + 5:45 Deep (definityly MY TYPE of Data but inverted) Its looks almost legit as mi-fit app would throw at me, but inverted.

I think you nailed it but need to just invert for mi band 1s !
Im 100% sure of those numbers.

Like you post, if its inverted it would look ok. For me its ONLY inverted: 6:39 = 0:54 Light + 5:45 Deep (definityly MY TYPE of Data but inverted) Its looks almost legit as mi-fit app would throw at me, but inverted. I think you nailed it but need to just invert for mi band 1s ! Im 100% sure of those numbers.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

0.9.0 still have those value inverted

0.9.0 still have those value inverted
Owner

-- deleted a few comments --

-- deleted a few comments --
computerlyrik commented 5 years ago (Migrated from github.com)
Owner

Just updated to FW 4.15.12.10 and GB 0.9.3
Will report tomorrow about sleep data.
With GB 0.9.1 and FW 4.11 sleep data was indeed inverted.

Just updated to FW 4.15.12.10 and GB 0.9.3 Will report tomorrow about sleep data. With GB 0.9.1 and FW 4.11 sleep data was indeed inverted.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

sadly FW update to 04.15.12.10 wont flip those, also authors didnt confirm fliping them on 0.9.3 :-) so we will have to wait till they make enought research to flip those values :/ until then i will lie to myself that I have sooo much deep sleep ;-)

sadly FW update to 04.15.12.10 wont flip those, also authors didnt confirm fliping them on 0.9.3 :-) so we will have to wait till they make enought research to flip those values :/ until then i will lie to myself that I have sooo much deep sleep ;-)
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

In "Mi Band Settings" I enabled "Use Heartrate Sensor to improve sleep ..." and I see hr on sleep graph for few days, but that didn't helped with inverted values.

In "Mi Band Settings" I enabled "Use Heartrate Sensor to improve sleep ..." and I see hr on sleep graph for few days, but that didn't helped with inverted values.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

0.11.1 - the data is still inverted, "Use Heartrate Sensor to improve sleep ..." is enabled all the time. It looks like M1S flip those because I dont see anyone that report those "flips" and probably M1S is less popular her anyway.
Its been some time now... Will this be fixed or should m1s owners move out ?

0.11.1 - the data is still inverted, "Use Heartrate Sensor to improve sleep ..." is enabled all the time. It looks like M1S flip those because I dont see anyone that report those "flips" and probably M1S is less popular her anyway. Its been some time now... Will this be fixed or should m1s owners move out ?
Owner

It will get fixed eventually. If there are other people that can confirm that Gadgetbridge displays the opposite of Mi Fit, then we can indeed simply flip those values.

It will get fixed eventually. If there are other people that can confirm that Gadgetbridge displays the opposite of Mi Fit, then we can indeed simply flip those values.
rplevka commented 5 years ago (Migrated from github.com)
Owner

@cpfeiffer I think I can confirm what @bigretromike says. I've been using GB with Mi Band 1S for 3 days now and here's what it says about my last two nights:

Light Deep Total
42min 4h 49min 5h 31min
44min 4h 12min 4h 56min
@cpfeiffer I think I can confirm what @bigretromike says. I've been using GB with Mi Band 1S for 3 days now and here's what it says about my last two nights: | Light | Deep | Total | | --- | --- | --- | | 42min | 4h 49min | 5h 31min | | 44min | 4h 12min | 4h 56min | ~~~I'm pretty sure they're inverted~~~
rplevka commented 5 years ago (Migrated from github.com)
Owner

There's definitely got to be some post processing in place on the APP side of things. The device seems to send you data for individual 'epochs' (every minute) with labeling them with activity type. However, you can't get the real deep sleep duration just by counting the number of minutes marked as 'deep sleep'. There should be some algorithm that would look for some patterns (e.g. If there were any light sleep epochs in some interval, mark whole interval as Light Sleep).

There's definitely got to be some post processing in place on the APP side of things. The device seems to send you data for individual 'epochs' (every minute) with labeling them with activity type. However, you can't get the real deep sleep duration just by counting the number of minutes marked as 'deep sleep'. There should be some algorithm that would look for some patterns (e.g. If there were any light sleep epochs in some interval, mark whole interval as Light Sleep).
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

To be honest those inverted values are almost tge same as official mi band application, but like i wrote those values are inverted, if we invert those it will look almost as the data from mi fit, so I wouldnt worry much about post processing

On July 17, 2016 11:31:17 AM GMT+02:00, Roman Plevka notifications@github.com wrote:

There's definitely got to be some post processing in place on the APP
side of things. The device seems to send you data for individual
'epochs' (every minute) with labeling it with activity type. However,
you can't get the real deep sleep just by counting the number of
minutes marked as 'deep sleep'. There should be some algorithm that
would look for some patterns (e.g. If there were any light sleep epochs
in some interval, mark whole interval as Light Sleep).


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233173323

Sent from my Android device with K-9 Mail. Please excuse my brevity.

To be honest those inverted values are almost tge same as official mi band application, but like i wrote those values are inverted, if we invert those it will look almost as the data from mi fit, so I wouldnt worry much about post processing On July 17, 2016 11:31:17 AM GMT+02:00, Roman Plevka notifications@github.com wrote: > There's definitely got to be some post processing in place on the APP > side of things. The device seems to send you data for individual > 'epochs' (every minute) with labeling it with activity type. However, > you can't get the real deep sleep just by counting the number of > minutes marked as 'deep sleep'. There should be some algorithm that > would look for some patterns (e.g. If there were any light sleep epochs > in some interval, mark whole interval as Light Sleep). > > --- > > You are receiving this because you were mentioned. > Reply to this email directly or view it on GitHub: > https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233173323 ## Sent from my Android device with K-9 Mail. Please excuse my brevity.
rplevka commented 5 years ago (Migrated from github.com)
Owner

@bigretromike I checked on the values, but aside the activity type there's also an intensity value (intensity of movement) - for those marked as 'deep sleep' the intensity is very low and for those marked as 'light sleep', the intensity is quite high. That's why I'm not sure whether they're just inverted.

@bigretromike I checked on the values, but aside the activity type there's also an intensity value (intensity of movement) - for those marked as 'deep sleep' the intensity is very low and for those marked as 'light sleep', the intensity is quite high. That's why I'm not sure whether they're just inverted.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

Im unsure because I dont debug is as hard but for me its clearly that deep sleep is inverted with light sleep, maybe those values are in % where 100% is deep sleep so the deep sleep hours have bigger counter, that im unsure but im sure that mi fit showed me from day one that my deep sleep is usuly less than hour and gb tell me that my light sleep is only less than 1h but me deep sleep is 5-7 h, which is inaccure as hell.

On July 17, 2016 12:04:40 PM GMT+02:00, Roman Plevka notifications@github.com wrote:

@bigretromike I checked on the values, but aside the activity type
there's also an intensity value (intensity of movement) - for those
marked as 'deep sleep' the intensity is very low and for those marked
as 'light sleep', the intensity is quite high. That's why I'm not sure
whether they're just inverted.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233174470

Sent from my Android device with K-9 Mail. Please excuse my brevity.

Im unsure because I dont debug is as hard but for me its clearly that deep sleep is inverted with light sleep, maybe those values are in % where 100% is deep sleep so the deep sleep hours have bigger counter, that im unsure but im sure that mi fit showed me from day one that my deep sleep is usuly less than hour and gb tell me that my light sleep is only less than 1h but me deep sleep is 5-7 h, which is inaccure as hell. On July 17, 2016 12:04:40 PM GMT+02:00, Roman Plevka notifications@github.com wrote: > @bigretromike I checked on the values, but aside the activity type > there's also an intensity value (intensity of movement) - for those > marked as 'deep sleep' the intensity is very low and for those marked > as 'light sleep', the intensity is quite high. That's why I'm not sure > whether they're just inverted. > > --- > > You are receiving this because you were mentioned. > Reply to this email directly or view it on GitHub: > https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233174470 ## Sent from my Android device with K-9 Mail. Please excuse my brevity.
rplevka commented 5 years ago (Migrated from github.com)
Owner

@bigretromike totally, same here. My point was that simply inverting these values won't be right either. It's just a funny correlation that inverted values are closer to the reality. I believe the most of the 'deep sleep' epochs would be simply converted to 'light sleep' if a correct sleep-cycle detection algorithm would be in place. That'd almost look like an inverted DS/LS but still would be different.
Even if I invert the values now, I still see 1+ hour error comparing to my mi fit measurements.

@bigretromike totally, same here. My point was that simply inverting these values won't be right either. It's just a funny correlation that inverted values are closer to the reality. I believe the most of the 'deep sleep' epochs would be simply converted to 'light sleep' if a correct sleep-cycle detection algorithm would be in place. That'd almost look like an inverted DS/LS but still would be different. Even if I invert the values now, I still see 1+ hour error comparing to my mi fit measurements.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

So i understand you use both app right? Its problem for me because i totaly droped all app and stayed with gb :-) thats why it iritates me that much. But showing friend gb and sayig 'its best fit app and its open source but it invert my data' aint that good for me ;)

On July 17, 2016 12:24:17 PM GMT+02:00, Roman Plevka notifications@github.com wrote:

@bigretromike totally, same here. My point was that simply inverting
these values won't be right either. It's just a funny correlation that
inverted values are closer to the reality. I believe the most of the
'deep sleep' epochs would be simply converted to 'light sleep' if a
correct sleep-cycle detection algorithm would be in place. That'd
almost look like an inverted DS/LS but still would be different.
Even if I invert the values now, I still see 1+ hour error comparing to
my mi fit measurements.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233175160

Sent from my Android device with K-9 Mail. Please excuse my brevity.

So i understand you use both app right? Its problem for me because i totaly droped all app and stayed with gb :-) thats why it iritates me that much. But showing friend gb and sayig 'its best fit app and its open source but it invert my data' aint that good for me ;) On July 17, 2016 12:24:17 PM GMT+02:00, Roman Plevka notifications@github.com wrote: > @bigretromike totally, same here. My point was that simply inverting > these values won't be right either. It's just a funny correlation that > inverted values are closer to the reality. I believe the most of the > 'deep sleep' epochs would be simply converted to 'light sleep' if a > correct sleep-cycle detection algorithm would be in place. That'd > almost look like an inverted DS/LS but still would be different. > Even if I invert the values now, I still see 1+ hour error comparing to > my mi fit measurements. > > --- > > You are receiving this because you were mentioned. > Reply to this email directly or view it on GitHub: > https://github.com/Freeyourgadget/Gadgetbridge/issues/250#issuecomment-233175160 ## Sent from my Android device with K-9 Mail. Please excuse my brevity.
Owner

Hehe. I totally agree that we need some improvements here.

As a quick fix, I can indeed invert those values, so that the values are at least a bit more realistic. Are you willing to work with me on getting a better algorithm? You don't even need to hack Gadgetbridge, know Android or Java. A description of an algorithm how to calculate the amounts from the samples you see would already help.

Hehe. I totally agree that we need some improvements here. As a quick fix, I can indeed invert those values, so that the values are at least a bit more realistic. Are you willing to work with me on getting a better algorithm? You don't even need to hack Gadgetbridge, know Android or Java. A description of an algorithm how to calculate the amounts from the samples you see would already help.
Owner

Oh, and comparisons with other apps, including Mi Fit help as well.

Oh, and comparisons with other apps, including Mi Fit help as well.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

I would have to install mi fit then, but Im fine with helping you. Just say what I need to install, prepare so I will collect data you need. Currently im using GB only, but if I need mi fit or other app just say.

I would have to install mi fit then, but Im fine with helping you. Just say what I need to install, prepare so I will collect data you need. Currently im using GB only, but if I need mi fit or other app just say.
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

Just invert those values 👍

Just invert those values :+1:
Owner

If you don't have Mi Fit, that's fine. Maybe there are others who have it installed and can share comparisons, e.g. screenshots of their data.

If you don't have Mi Fit, that's fine. Maybe there are others who have it installed and can share comparisons, e.g. screenshots of their data.
rplevka commented 5 years ago (Migrated from github.com)
Owner

@cpfeiffer for a 3rd night in a row I've been trying to download my sleep data to GB with "keep data on a band" option checked, but when i fire up Mi Fit to sync the data, it says there's none anymore. Is that a bug in GB or am I doing it wrong?

@cpfeiffer for a 3rd night in a row I've been trying to download my sleep data to GB with "keep data on a band" option checked, but when i fire up Mi Fit to sync the data, it says there's none anymore. Is that a bug in GB or am I doing it wrong?
Owner

@rplevka For me it does work. Didn't test with Mi Fit, but if I have this option enabled, I can sync twice and get the same samples every time.

Maybe Mi Fit already fetched the data? Or is it simply not available in Mi Fit?

@rplevka For me it does work. Didn't test with Mi Fit, but if I have this option enabled, I can sync twice and get the same samples every time. Maybe Mi Fit already fetched the data? Or is it simply not available in Mi Fit?
rplevka commented 5 years ago (Migrated from github.com)
Owner

I can confirm, the data stay on the band, however mifit won't pick them up for some reason

I can confirm, the data stay on the band, however mifit won't pick them up for some reason
gadgetguy08 commented 5 years ago (Migrated from github.com)
Owner

Just got a Mi Band 1S.
FW 4.6.4.22
HR 1.6.1.22
GB 0.13.7

Arm right

Sleep data shows Deep sleep is lighter color and Light sleep darker. But the "activity" bars are taller for the light bars. Also there were lighter bars at one point when I know I was awake, making me wonder if they are reversed for me.

Attemped to post a screen shot with the Android app..

Just got a Mi Band 1S. FW 4.6.4.22 HR 1.6.1.22 GB 0.13.7 Arm right Sleep data shows Deep sleep is lighter color and Light sleep darker. But the "activity" bars are taller for the light bars. Also there were lighter bars at one point when I know I was awake, making me wonder if they are reversed for me. Attemped to post a screen shot with the Android app..
rplevka commented 5 years ago (Migrated from github.com)
Owner

@gadgetguy08 yes. This is simply caused by the fact that guys felt like switching light and deep sleep types gets more appropriate results when it comes to total light and total deep sleep.
(just read the thread above).
I myself beleive, the values were right, but there was some additional postprocessing which turned some sequences of deep sleep to the light sleep etc.

so I also believe, we should not really invert the values

@gadgetguy08 yes. This is simply caused by the fact that guys felt like switching light and deep sleep types gets more appropriate results when it comes to total light and total deep sleep. (just read the thread above). I myself beleive, the values were right, but there was some additional postprocessing which turned some sequences of deep sleep to the light sleep etc. so I also believe, we should not really invert the values
bigretromike commented 5 years ago (Migrated from github.com)
Poster
Owner

for me current values look ok;
sadly its hard for me to check each version as always I end up wiping those data out of both db and band :/

for me current values look ok; sadly its hard for me to check each version as always I end up wiping those data out of both db and band :/
Owner

We're aware that the mobile is not the ideal place to store the data forever. An external storage service is planned and there is also some experimentation code already.

We're aware that the mobile is not the ideal place to store the data forever. An external storage service is planned and there is also some experimentation code already.
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.