A Year in Data with Python
Mark Koester
SoCal Python Meetup, Nov 19, 2019
A Year in Data
Self-Tracking and Personal Data Analysis with Python
Mark Koester | www.markwk.com
SoCal Python Meetup, Nov 19, 2019
Slides and Code:
github.com/markwk/python4selftrackers
About Me
My name is Mark Koester, and I am a product manager.
I track a lot of aspects of my life.
A Personal Example
Me in 2014/2015.
2015 was a defining year for me…
- Step tracking with Fitbit
- Time Tracking with RescueTime
- Getting Things Done by David Allen
Two Years Later in 2017
My personal example of improved health and Self
I’ve used self-tracking and personal data analysis to change my life.
And I believe technology and personal data can help others.
Talk Outline
- What is Quantified Self? How to Track a Life with Technology?
- Visualizing a Year (Month, Week, Day) in Data with QS Ledger.
PART 1: How to Measure a Life
Quantified Self
- (def.) Measuring or documenting something about your self to gain meaning or make improvements.
- Related: Self-tracking, Biohacking, Data-driven life…
Why Track a Life?: Benefits of Self-Tracking
- Improved Health.
- Better Time Management
- Augment your memory.
- Save and better invest your money
- Achieve goals. Support habits. Manage projects
- Understand your mood, energy level and stress.
- Curiosity? Learn stuff about yourself.
- Personal Data is the Future.
Source: https://github.com/markwk/qs_mind_map
Popular Forms of Tracking
- Wearables (Apple Watch, Fitbit, Oura): steps, sleep, heart Rate (one in five Americans own a heart rate sensor today)
- Mood Tracking Apps, like MoodNotes
- Time Tracking (inc computer usage with RescueTime)
- Calendar, Projects and Tasks
- Strava, RunKeeper and many other sport apps
- Media Consumption: TV, music, articles, books…
- Weight
- Others: Money, Blood, DNS, Microbiome…
Tools for Self-tracking, QS, and Lifelogging
github.com/markwk/awesome-quantified-self
Opportunities
In the tracking an data space
- Enabling and tracking new data points => Accessibility, new sensors, cheaper testing, new tracking apps, etc.
- Deriving insight and meaning from existing data => More data, data accessibility, better data science and machine learning
My Contributions and Work
ENABLING AND TRACKING NEW DATA POINTS
- PodcastTracker.com
- PhotoStats.io
- BioMarkerTracker.com
DERIVING INSIGHT AND MEANING FROM EXISTING DATA
- Quantified Self (QS) Ledger
Writings: www.markwk.com, datadrivenyou.com
Data-Driven Life Tip #1:
Where to start with tracking a life?
Start with a Question or a Goal
TRACK IT!
PART 2: Visualizing a Year in Data
Built with Python 3 and Jupyter Notebooks
Current Integrations (1/4):
- Apple Health: fitness and health tracking and data analysis from iPhone or Apple Watch.
- AutoSleep: iOS sleep tracking data analysis of sleep per night and rolling averages.
- Fitbit: fitness and health tracking and analysis of Steps, Sleep, and Heart Rate from a Fitbit wearable.
- GoodReads: book reading tracking and data analysis for GoodReads.
- Google Calendar: past events, meetings and times for Google Calendar.
Current Integrations (2/4):
- Google Sheets: get data from any Google Sheet which can be useful for pulling data from IFTTT integrations that add data.
- Habitica: habit and task tracking with Habitica’s gamified approach to task management.
- Instapaper: articles read and highlighted passages from Instapaper.
- Kindle Highlights: Parser and Highlight Extract from Kindle clippings, along with a sample data analysis and tool to export highlights to separate markdown files.
Current Integrations (3/4):
- Last.fm: music tracking and analysis of music listening history from Last.fm.
- Oura: oura ring activity, sleep and wellness data.
- RescueTime: track computer usage and analysis of computer activities and time with RescueTime.
- Pocket: articles read and read count from Pocket.
- Strava: activities downloader (runs, cycling, swimming, etc.) and analysis from Strava.
Current Integrations (4/4):
- Todoist: task tracking and analysis of todo’s and tasks completed history from Todoist app.
- Toggl: time tracking and analysis of manual timelog entries from Toggl.
- WordCounter: (Mac Only) extract wordcounter app history and visualize recent periods of word counts.
Installation and Usage
Installation
- Use local Python 3 setup OR install Anaconda Distribution.
- pip or conda install Pandas, NumPy, Matplotlib and Seaborn
- For each project refer to its readme or notebook documentation for any specific dependencies.
Code Organization
Each project has:
- NAME_downloader - notebook works with service API or raw data to get and process your data.
- NAME_data_analysis - notebook slices, dices and visualizes your data into different charts on different time dimensions.
Basic Usage:
- Choose a tracking service
- Setup integration (with developer keys when necessary)
- Download and process your data
- Configure and run data analysis and visualization
Deeper Dive into Code and Usage
SEE: github.com/markwk/python4selftrackers
My Year in Data
Coding Walkthrough (with RescueTime)
My Year in Numbers:
- 3,878,369 steps taken and logged.
- 26 blog posts published on www.markwk.com.
- 1559.72km kilometers (969.16 miles) run
- 94 days (of time) on computer according to RescueTime
- 84 days (of time) on projects according to manual time tracking
- 2212 completed tasks in Todoist across more than a dozen projects.
A Year in Media
- 18,657 book pages read accross 60 books, according to GoodReads
- 1785 articles reading via Pocket and Instapaper
- 16,855 songs listened to and “scrobbled” in 2018 using Last.fm.
- My top genres were Synthwave, Electronic and Synthpop. My top track was Cold summer by Le Matos. And my “favorite” artist was Le Cassette.
2,991 photos taken (Tracked via PhotoStats.io)
Data-Driven Life Tip #2:
Engage with your data.
Conclusion:
Tips on how to become a data-driven you
What should I track?
Four Essential Areas for Everyone to Track
- Health
- Time
- Goals, Projects and Tasks
- Money
How to Track?
A COUPLE RECOMMENDATIONS
- Health:
- Blood Tests
- Sleep
- (and maybe Heart Rate Variability)
- Time: RescueTime
- Money: Mint.com or Personal Capital
- Goals, Projects and Tasks: Todoist
My Steps Towards Data-Driven Self-Improvement
- Set a Goal
- Track It
- Research the area.
- Make Lifestyle Changes
- And track those too, i.e. track your commitment and follow-through
- Check-in, Evaluate and Engage with your data
- Repeat
Summary
- It’s easier than ever to track our lives.
- Some personal data can be more significant than others.
- Tip #1: Start with a question or goal, then track it.
- Tip #2: Engage with your data (and use Python!)
- Tip #3: Track your time, get a heath check-up with blood testing, and find a way to quantify your work and projects.
Python can help: data collection, processing, exploring, visualizing and using.
"In God we trust…
…all others bring data."
W. Edwards Deming
Thanks
www.markwk.com
datadrivenyou.com
References
Online References
- Original Post: http://www.markwk.com/2019/01/year-in-data.html
- QS Ledger Code: github.com/markwk/qs_ledger
- QS Mind Map github.com/markwk/qs_mind_map
- QS Tools: github.com/markwk/awesome-quantified-self
Published References
Find me online at www.markwk.com!
A Year in Data with Python
Mark Koester
SoCal Python Meetup, Nov 19, 2019