Exporting data - meaning of columns #332

Closed
by robin13 opened 6 years ago · 7 comments
robin13 commented 6 years ago (Migrated from github.com)
Owner

I am looking at exporting the data from gadgetbridge to Elasticsearch. I see the database is SQlite with 6 columns:
timestamp, provider, intensity, steps, type, customShort.

Is there somewhere I could find more information on what exactly provider, intensity, type and customShort mean?
Is Intensity some arbitrary number, or does it relate to something?

Thank you for your help! :)

I am looking at exporting the data from gadgetbridge to Elasticsearch. I see the database is SQlite with 6 columns: timestamp, provider, intensity, steps, type, customShort. Is there somewhere I could find more information on what exactly provider, intensity, type and customShort mean? Is Intensity some arbitrary number, or does it relate to something? Thank you for your help! :)
Owner

A broad overview of the available data is available on our wiki, but you are right: the information you requested is not available.

Since we are in the middle of a database refactoring I am also not updating the wiki page, but will answer here.

provider

Depends on the device and on the app the user is using (for pebbles). The values are available in this file

intensity

Depends on the device. For the pebble it was recently documented, we store what they call VMC. For the miband is some kind of value that increases with the intensity of the physical activity. NB: We have no idea if restful sleep is "more intense" than shallow sleep or the opposite.

type

Depends on the provider. For the miband the values are described in this file. For the pebble see the various files ending with "SampleProvider" available in this directory

Since you plan stoing the sample data in elasticsearch, I would like to make you aware of #49 , hoping that you can contribute to that issue.

Please let me know if you have further questions, or feel free to close the issue.

A broad overview of the available data is available [on our wiki](https://github.com/Freeyourgadget/Gadgetbridge/wiki/Activity-Analysis), but you are right: the information you requested is not available. Since we are in the middle of a database [refactoring](https://github.com/Freeyourgadget/Gadgetbridge/tree/db-refactoring) I am also not updating the wiki page, but will answer here. ### provider Depends on the device and on the app the user is using (for pebbles). The values are available in [this file](https://github.com/Freeyourgadget/Gadgetbridge/blob/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/SampleProvider.java) ### intensity Depends on the device. For the pebble it was recently [documented](https://medium.com/pebble-research/opening-the-black-box-publishing-pebbles-activity-tracking-algorithms-bebdb5a7383a), we store what they call VMC. For the miband is some kind of value that increases with the intensity of the physical activity. NB: We have no idea if restful sleep is "more intense" than shallow sleep or the opposite. ### type Depends on the provider. For the miband the values are described in [this file](https://github.com/Freeyourgadget/Gadgetbridge/blob/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/miband/MiBandSampleProvider.java). For the pebble see the various files ending with "SampleProvider" available in [this directory](https://github.com/Freeyourgadget/Gadgetbridge/tree/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/pebble) Since you plan stoing the sample data in elasticsearch, I would like to make you aware of #49 , hoping that you can contribute to that issue. Please let me know if you have further questions, or feel free to close the issue.
robin13 commented 6 years ago (Migrated from github.com)
Poster
Owner

Thank you for the info @danielegobbetti ! :)

I have a Mi 1S and see some more types: 17, 33 and 20

33 was when I was in the shower, but I'm not sure about the others...

Is TYPE_SLIENT meant to be TYPE_SILENT? What does Silent mean?

Thank you for the info @danielegobbetti ! :) I have a Mi 1S and see some more types: 17, 33 and 20 33 was when I was in the shower, but I'm not sure about the others... Is [`TYPE_SLIENT`](https://github.com/Freeyourgadget/Gadgetbridge/blob/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/miband/MiBandSampleProvider.java#L18) meant to be `TYPE_SILENT`? What does Silent mean?
robin13 commented 6 years ago (Migrated from github.com)
Poster
Owner

Thank you for the link to #49 - I'll link my blog post there when I complete the stack to get data into Elasticsearch nicely. :)

Thank you for the link to #49 - I'll link my blog post there when I complete the stack to get data into Elasticsearch nicely. :)
Owner

@robin13 the types you see in the file are the one more-or less known. It's likely new types were added in the official app but we don't know what they mean. It would be great if you could relate those to the activity you were performing at the time. "Washing" could be a nice activity for a wearable to detect :-)

I'm closing this issue now since what we know was shared. Please feel free to open a PR if you discover other values' meanings.

@robin13 the types you see in the file are the one more-or less known. It's likely new types were added in the official app but we don't know what they mean. It would be great if you could relate those to the activity you were performing at the time. "Washing" could be a nice activity for a wearable to detect :-) I'm closing this issue now since what we know was shared. Please feel free to open a PR if you discover other values' meanings.
robin13 commented 6 years ago (Migrated from github.com)
Poster
Owner

Will do! :)

For anybody else finding this thread: for the Mi Band the customShort currently stores the heartrate (and is valid if < 250).

Will do! :) For anybody else finding this thread: for the Mi Band the **customShort** currently stores the **heartrate** (and is valid if **< 250**).
chriscamacho commented 5 years ago (Migrated from github.com)
Owner

hmmm, having device specific data in the export is bad news indeed!

In order to avoid a ghastly train wreck in future I'd really recommend that there is a device independent abstraction - further this format should definitely be specified.

Just looking at the source code (which should be a last resort) I can work out what many of the type id's are, but 16? imagine half a dozen extra devices down the line, and a user having two different devices (they could have separate sensor domains) things could get very messy

hmmm, having device specific data in the export is bad news indeed! In order to avoid a ghastly train wreck in future I'd really recommend that there is a device independent abstraction - further this format should definitely be specified. Just looking at the source code (which should be a last resort) I can work out what many of the type id's are, but 16? imagine half a dozen extra devices down the line, and a user having two different devices (they could have separate sensor domains) things could get very messy
Owner

Let me clarify this: the sqlite db was never meant to be an export format. It's just a quick hack so that people can have a access to their data.

An explicit export format should of course have device independent values, in addition to the device dependent values.

The reason we store the raw values is that we have to reverse engineer them and we don't know them all. In the future, when we have more knowledge, we can process the raw data again.

So you want to work with us on a pepper export, e.g. as CSV for a start, you are welcome. It's not hard, it's just one of the many things that needs to be done.

Keep in mind that the db structure in master is more rich than what we have in the last release, and the new structure is not completely finished. If you need your data now, use the release-0.11.x branch, there we have just the single table with activity samples.

Let me clarify this: the sqlite db was never meant to be an export format. It's just a quick hack so that people can have a access to their data. An explicit export format should of course have device independent values, _in addition_ to the device dependent values. The reason we store the raw values is that we have to reverse engineer them and we don't know them all. In the future, when we have more knowledge, we can process the raw data again. So you want to work with us on a pepper export, e.g. as CSV for a start, you are welcome. It's not hard, it's just one of the many things that needs to be done. Keep in mind that the db structure in master is more rich than what we have in the last release, and the new structure is not completely finished. If you need your data _now_, use the release-0.11.x branch, there we have just the single table with activity samples.
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.