A Year in Data with Python
Mark Koester
SoCal Python Meetup, Nov 19, 2019
A Year in Data
Self-Tracking and Personal Data Analysis with Python
Mark Koester | www.markwk.com SoCal Python Meetup, Nov 19, 2019
Slides and Code: github.com/markwk/python4selftrackers
About Me
My name is Mark Koester , and I am a product manager.
I track a lot of aspects of my life.
Me in 2014/2015.
2015 was a defining year for me…
Step tracking with Fitbit
Time Tracking with RescueTime
Getting Things Done by David Allen
Two Years Later in 2017
My personal example of improved health and Self
I’ve used self-tracking and personal data analysis to change my life.
And I believe technology and personal data can help others.
Talk Outline
What is Quantified Self? How to Track a Life with Technology?
Visualizing a Year (Month, Week, Day) in Data with QS Ledger .
PART 1: How to Measure a Life
While the use of technology is novel and relatively recent, the idea of measuring and ‘watching’ life is not. Socrates famously said the unexamined life is not worth living and Benjamin Franklin kept a journal on his morals and habits.
Quantified Self
(def.) Measuring or documenting something about your self to gain meaning or make improvements.
Related: Self-tracking, Biohacking, Data-driven life…
2007: Quantified Self is neulogism created by Gary Wolf and Kevin Kelly, two writers at Wired Magazine.
2008: Wolf and Kelly founded the company Quantified Self with the aim “to help people get meaning out of their personal data”
Movement
Definitions of QS:
“Quantified Self (QS) is an emerging area of technology that allows consumers to use a variety of digital tools to collect data and learn about their behaviors and habits of everyday life.” (Rocket Fuel Survey, 2014)
“The Quantified Self (QS) refers to a movement in which its participants track the biological, physical, behavioural, and/or environmental aspects of their everyday lives” (Eiben, 2015)
QS is also become something we do with much of technology but without a specific need to call it out.
Why Track a Life?: Benefits of Self-Tracking
Improved Health.
Better Time Management
Augment your memory.
Save and better invest your money
Achieve goals. Support habits. Manage projects
Understand your mood, energy level and stress.
Curiosity? Learn stuff about yourself.
Personal Data is the Future.
Source: https://github.com/markwk/qs_mind_map
Wearables (Apple Watch, Fitbit, Oura): steps, sleep, heart Rate (one in five Americans own a heart rate sensor today)
Mood Tracking Apps, like MoodNotes
Time Tracking (inc computer usage with RescueTime)
Calendar, Projects and Tasks
Strava, RunKeeper and many other sport apps
Media Consumption: TV, music, articles, books…
Weight
Others: Money, Blood, DNS, Microbiome…
Opportunities
In the tracking an data space
Enabling and tracking new data points => Accessibility, new sensors, cheaper testing, new tracking apps, etc.
Deriving insight and meaning from existing data => More data, data accessibility, better data science and machine learning
While questions of privacy remain the most discussed, there remain a number of challenges as well as opportunities in the personal data space. For me, I often think alot about not just data privacy but data accessibility.
My Contributions and Work
ENABLING AND TRACKING NEW DATA POINTS
PodcastTracker.com
PhotoStats.io
BioMarkerTracker.com
DERIVING INSIGHT AND MEANING FROM EXISTING DATA
Quantified Self (QS) Ledger
Writings: www.markwk.com, datadrivenyou.com
Data-Driven Life Tip #1:
Where to start with tracking a life?
Start with a Question or a Goal
TRACK IT!
When it comes to self-tracking, we often think alot about the technology or “solution.” I have X device, wearable or app. Now what can it track? I prefer to start with a question or goal and then figure out how I might track that area. Once I’m tracking it, I can know my baseline and make appropriate targets for changes and improvement.
PART 2: Visualizing a Year in Data
github.com/markwk/qs_ledger
The project has two primary goals:
download all of your personal data from various tracking services (see below for list of integration services) and store locally.
provide the starting point for personal data analysis, data visualization and a personal data dashboard
It also includes examples for finding correlations, machine learning and patterns across your data.
Built with Python 3 and Jupyter Notebooks
Current Integrations (1/4):
Apple Health : fitness and health tracking and data analysis from iPhone or Apple Watch.
AutoSleep : iOS sleep tracking data analysis of sleep per night and rolling averages.
Fitbit : fitness and health tracking and analysis of Steps, Sleep, and Heart Rate from a Fitbit wearable.
GoodReads : book reading tracking and data analysis for GoodReads.
Google Calendar : past events, meetings and times for Google Calendar.
Current Integrations (2/4):
Google Sheets : get data from any Google Sheet which can be useful for pulling data from IFTTT integrations that add data.
Habitica : habit and task tracking with Habitica’s gamified approach to task management.
Instapaper : articles read and highlighted passages from Instapaper.
Kindle Highlights : Parser and Highlight Extract from Kindle clippings, along with a sample data analysis and tool to export highlights to separate markdown files.
Current Integrations (3/4):
Last.fm : music tracking and analysis of music listening history from Last.fm.
Oura : oura ring activity, sleep and wellness data.
RescueTime : track computer usage and analysis of computer activities and time with RescueTime.
Pocket : articles read and read count from Pocket.
Strava : activities downloader (runs, cycling, swimming, etc.) and analysis from Strava.
Current Integrations (4/4):
Todoist : task tracking and analysis of todo’s and tasks completed history from Todoist app.
Toggl : time tracking and analysis of manual timelog entries from Toggl.
WordCounter : (Mac Only) extract wordcounter app history and visualize recent periods of word counts.
Installation
Use local Python 3 setup OR install Anaconda Distribution.
pip or conda install Pandas, NumPy, Matplotlib and Seaborn
For each project refer to its readme or notebook documentation for any specific dependencies.
Code Organization
Each project has:
NAME_downloader - notebook works with service API or raw data to get and process your data.
NAME_data_analysis - notebook slices, dices and visualizes your data into different charts on different time dimensions.
Basic Usage:
Choose a tracking service
Setup integration (with developer keys when necessary)
Download and process your data
Configure and run data analysis and visualization
Deeper Dive into Code and Usage
SEE: github.com/markwk/python4selftrackers
My Year in Numbers:
3,878,369 steps taken and logged.
26 blog posts published on www.markwk.com.
1559.72km kilometers (969.16 miles) run
94 days (of time) on computer according to RescueTime
84 days (of time) on projects according to manual time tracking
2212 completed tasks in Todoist across more than a dozen projects.
2,991 photos taken (Tracked via PhotoStats.io)
Data-Driven Life Tip #2:
Engage with your data.
Too many people expect that having a wearable or tracking tool is enough to create change. But the reality is that if you want tracking to lead to behavior change, you need to engage with your data and your goal too. Use data to bring you a self-reflection and feedback. Use data to reflect on your goals!
Conclusion:
Tips on how to become a data-driven you
What should I track?
Four Essential Areas for Everyone to Track
Health
Time
Goals, Projects and Tasks
Money
How to Track?
A COUPLE RECOMMENDATIONS
Health:
Blood Tests
Sleep
(and maybe Heart Rate Variability)
Time: RescueTime
Money: Mint.com or Personal Capital
Goals, Projects and Tasks: Todoist
My Steps Towards Data-Driven Self-Improvement
Set a Goal
Track It
Research the area.
Make Lifestyle Changes
And track those too, i.e. track your commitment and follow-through
Check-in, Evaluate and Engage with your data
Repeat
Summary
It’s easier than ever to track our lives.
Some personal data can be more significant than others.
Tip #1: Start with a question or goal, then track it.
Tip #2: Engage with your data (and use Python!)
Tip #3: Track your time, get a heath check-up with blood testing, and find a way to quantify your work and projects.
Python can help: data collection, processing, exploring, visualizing and using.
"In God we trust…
…all others bring data."
W. Edwards Deming
Thanks
Slides and Code: github.com/markwk/python4selftrackers
www.markwk.com datadrivenyou.com
Online References
Original Post: http://www.markwk.com/2019/01/year-in-data.html
QS Ledger Code: github.com/markwk/qs_ledger
QS Mind Map github.com/markwk/qs_mind_map
QS Tools: github.com/markwk/awesome-quantified-self
Find me online at www.markwk.com!