A Year in Data with Python

Mark Koester

SoCal Python Meetup, Nov 19, 2019

A Year in Data

Self-Tracking and Personal Data Analysis with Python


Mark Koester | www.markwk.com
SoCal Python Meetup, Nov 19, 2019

Slides and Code:

github.com/markwk/python4selftrackers

About Me

My name is Mark Koester, and I am a product manager.

I track a lot of aspects of my life.

A Personal Example

Me in 2014/2015.

2015 was a defining year for me…

  • Step tracking with Fitbit
  • Time Tracking with RescueTime
  • Getting Things Done by David Allen

Two Years Later in 2017

My personal example of improved health and Self

I’ve used self-tracking and personal data analysis to change my life.


And I believe technology and personal data can help others.

Talk Outline

  1. What is Quantified Self? How to Track a Life with Technology?
  2. Visualizing a Year (Month, Week, Day) in Data with QS Ledger.

PART 1: How to Measure a Life

Quantified Self

  • (def.) Measuring or documenting something about your self to gain meaning or make improvements.
  • Related: Self-tracking, Biohacking, Data-driven life…

Why Track a Life?: Benefits of Self-Tracking

  • Improved Health.
  • Better Time Management
  • Augment your memory.
  • Save and better invest your money
  • Achieve goals. Support habits. Manage projects
  • Understand your mood, energy level and stress.
  • Curiosity? Learn stuff about yourself.
  • Personal Data is the Future.

Source: https://github.com/markwk/qs_mind_map

  • Wearables (Apple Watch, Fitbit, Oura): steps, sleep, heart Rate (one in five Americans own a heart rate sensor today)
  • Mood Tracking Apps, like MoodNotes
  • Time Tracking (inc computer usage with RescueTime)
  • Calendar, Projects and Tasks
  • Strava, RunKeeper and many other sport apps
  • Media Consumption: TV, music, articles, books…
  • Weight
  • Others: Money, Blood, DNS, Microbiome…

Tools for Self-tracking, QS, and Lifelogging

github.com/markwk/awesome-quantified-self

Opportunities

In the tracking an data space

  1. Enabling and tracking new data points => Accessibility, new sensors, cheaper testing, new tracking apps, etc.
  2. Deriving insight and meaning from existing data => More data, data accessibility, better data science and machine learning

My Contributions and Work

ENABLING AND TRACKING NEW DATA POINTS
  • PodcastTracker.com
  • PhotoStats.io
  • BioMarkerTracker.com


DERIVING INSIGHT AND MEANING FROM EXISTING DATA
  • Quantified Self (QS) Ledger


Writings: www.markwk.com, datadrivenyou.com

Data-Driven Life Tip #1:

Where to start with tracking a life?


Start with a Question or a Goal

TRACK IT!

PART 2: Visualizing a Year in Data

github.com/markwk/qs_ledger

Built with Python 3 and Jupyter Notebooks

Current Integrations (1/4):

  • Apple Health: fitness and health tracking and data analysis from iPhone or Apple Watch.
  • AutoSleep: iOS sleep tracking data analysis of sleep per night and rolling averages.
  • Fitbit: fitness and health tracking and analysis of Steps, Sleep, and Heart Rate from a Fitbit wearable.
  • GoodReads: book reading tracking and data analysis for GoodReads.
  • Google Calendar: past events, meetings and times for Google Calendar.

Current Integrations (2/4):

  • Google Sheets: get data from any Google Sheet which can be useful for pulling data from IFTTT integrations that add data.
  • Habitica: habit and task tracking with Habitica’s gamified approach to task management.
  • Instapaper: articles read and highlighted passages from Instapaper.
  • Kindle Highlights: Parser and Highlight Extract from Kindle clippings, along with a sample data analysis and tool to export highlights to separate markdown files.

Current Integrations (3/4):

  • Last.fm: music tracking and analysis of music listening history from Last.fm.
  • Oura: oura ring activity, sleep and wellness data.
  • RescueTime: track computer usage and analysis of computer activities and time with RescueTime.
  • Pocket: articles read and read count from Pocket.
  • Strava: activities downloader (runs, cycling, swimming, etc.) and analysis from Strava.

Current Integrations (4/4):

  • Todoist: task tracking and analysis of todo’s and tasks completed history from Todoist app.
  • Toggl: time tracking and analysis of manual timelog entries from Toggl.
  • WordCounter: (Mac Only) extract wordcounter app history and visualize recent periods of word counts.

Installation and Usage

Installation

  • Use local Python 3 setup OR install Anaconda Distribution.
  • pip or conda install Pandas, NumPy, Matplotlib and Seaborn
  • For each project refer to its readme or notebook documentation for any specific dependencies.

Code Organization

Each project has:

  1. NAME_downloader - notebook works with service API or raw data to get and process your data.
  2. NAME_data_analysis - notebook slices, dices and visualizes your data into different charts on different time dimensions.

Basic Usage:

  • Choose a tracking service
  • Setup integration (with developer keys when necessary)
  • Download and process your data
  • Configure and run data analysis and visualization

Deeper Dive into Code and Usage

SEE: github.com/markwk/python4selftrackers

My Year in Data

Coding Walkthrough (with RescueTime)


www.markwk.com/2019/01/year-in-data.html

My Year in Numbers:

  • 3,878,369 steps taken and logged.
  • 26 blog posts published on www.markwk.com.
  • 1559.72km kilometers (969.16 miles) run
  • 94 days (of time) on computer according to RescueTime
  • 84 days (of time) on projects according to manual time tracking
  • 2212 completed tasks in Todoist across more than a dozen projects.

A Year in Media

  • 18,657 book pages read accross 60 books, according to GoodReads
  • 1785 articles reading via Pocket and Instapaper
  • 16,855 songs listened to and “scrobbled” in 2018 using Last.fm.
  • My top genres were Synthwave, Electronic and Synthpop. My top track was Cold summer by Le Matos. And my “favorite” artist was Le Cassette.

2,991 photos taken (Tracked via PhotoStats.io)

Data-Driven Life Tip #2:

Engage with your data.

Conclusion:

Tips on how to become a data-driven you

What should I track?

Four Essential Areas for Everyone to Track

  • Health
  • Time
  • Goals, Projects and Tasks
  • Money

How to Track?

A COUPLE RECOMMENDATIONS

  • Health:
    • Blood Tests
    • Sleep
    • (and maybe Heart Rate Variability)
  • Time: RescueTime
  • Money: Mint.com or Personal Capital
  • Goals, Projects and Tasks: Todoist

My Steps Towards Data-Driven Self-Improvement

  • Set a Goal
  • Track It
  • Research the area.
  • Make Lifestyle Changes
  • And track those too, i.e. track your commitment and follow-through
  • Check-in, Evaluate and Engage with your data
  • Repeat

Summary

  • It’s easier than ever to track our lives.
  • Some personal data can be more significant than others.
  • Tip #1: Start with a question or goal, then track it.
  • Tip #2: Engage with your data (and use Python!)
  • Tip #3: Track your time, get a heath check-up with blood testing, and find a way to quantify your work and projects.

Python can help: data collection, processing, exploring, visualizing and using.

"In God we trust…

…all others bring data."

W. Edwards Deming

Thanks


Slides and Code: github.com/markwk/python4selftrackers


www.markwk.com
datadrivenyou.com

References

Online References

  • Original Post: http://www.markwk.com/2019/01/year-in-data.html
  • QS Ledger Code: github.com/markwk/qs_ledger
  • QS Mind Map github.com/markwk/qs_mind_map
  • QS Tools: github.com/markwk/awesome-quantified-self

Published References

Find me online at www.markwk.com!