This is a project to expose and visualize Google Watch and Fitbit device data. This provides access to the lowest level of detail data recorded by the device, allowing a much more detailed view of heart rate, resting heart rate, steps, SpO2, and HRV than is possible in the Fitbit app. In addition, this allows the presentation of much longer timeframes of data than is possible in the Fitbit app.
A Fitbit or Google Watch device records SpO2 every minute, and HRV and heart rate every 5 seconds, however, the Fitbit app doesn’t show you that detailed data, instead it shows data at a more summarized / aggregated level of detail. For example:
- SpO2 is presented in more detail as an “estimated oxygen variation” line chart visualization with detailed minute data (unknown but likely aggregated at what appears to be every 5 minutes) but the chart doesn’t show the actual values. The line is presented only as “High” or “Low” oxygen varation for that night’s values.
- Heart rate is also shown in more detail in two places: 1) Sleeping heart rate is presented with the actual values and presented as relative to resting heart rate and 2) Daily heart rate is presented as a smoothed line chart (unknown but likely aggregated at what appears to be every 5 minutes) with the actual values and indicates heart rate zones.
- HRV is only provided as a single nightly average value.
You can get the SpO2, HRV and heart rate data as recorded at the lowest level of detail from two sources:
- The Fitbit Web API link to documentation.
- The Fitbit/Google data export download feature link to documentation.
This post discusses getting data from the Fitbit Web API.
Getting Fitbit Web API data
To automate this I created an fairly complex application that retrieves data from the Fitbit Web API, saves it in a MongoDB database, and then visualizes data in web pages using Node.js, HTML, Javascript, and Plotly.js.
The application code is available in this Github repository: https://github.com/sitrucp/fitbit_data_viz
The application’s code is structured into a frontend/backend structure as follows:
- etl – this is the app backend Python code that connects to and retrieves the Fitbit Web API data as json files and then inserts these into a MongoDB database.
- node – this is app frontend HTML, Node.js and Plotly.js code to connect to the MongoDB database to get and visualize the data. Screenshots of some of the visualizations are provided below.
Technical highlights include:
- PyCherry is used to pop the Fitbit authentication login browser window so the app can be authenticated on the Fitbit API.
- Fitbit authentication saves a token locally as a json file (auth_tokens.json) and includes a refresh token that is used to get new tokens automatically. This enables the authentication to the Fitbit Web API to be automated by saving and refreshing authentication tokens for subsequent connections.
- A response log (response_log.json) records the last API call response date for each API endpoints to allow the app to get any new data since it was last retrieved.
- An activity log (activity_log.json) is used to record retrieval activity including response errors and messages for tracking and troubleshooting purposes.
- Transformation of the data occurs in several places in the app backend and frontend. Transformation occurs before data is loaded into MongoDB using Python, but also occurs as the data is retrieved from MongoDB using Node.js javascript code, and after it is retrieved for use by Plotly.js charts using HTML page Javascript.
Menu of individual detailed visualizations
The app has a main home page that is a table of contents linking to all visualizations available.
There are two main summarized purpose driven reports, a Today report summarizing today’s data along with past 90 days history and a Sleep report the presents the last night’s sleep data.
There are also many categorized interactive detail reports.
Home – table of contents
Today – summary snapshot
Summary of today’s and past 90 day’s activity.
Sleep – previous night’s sleep detail
Set of visualizations of the previous night’s detailed sleep activity as well as SpO2, heart rate and HRV. Standard Fitbit data presentations in the app and on the web do not contain this granular second level detail which is captured by the device and available through the Web API.
