tl;dr check it out at http://craigcarlyle.me
A Quick Intro
Several years ago I started on a weight loss and health journey. I was introduced to the quantified self community with Fitbit and MyFitnessPal. After losing over 50 pounds, I was hooked on the idea of personal data tracking.
Fast-forward to the present and I’m tracking more than ever. Not because I believe in optimizing every aspect of my life, but because it helps me balance it. Haven’t exercised in a few days? Time to hit the gym. I’ve been eating 4,000 calories a day this week? I should eat clean for a while. Haven’t meditated or read in a while? Let me grab my Kindle.
Lots of fantastic tracking tools are available for mobile devices (Gyroscope has been my personal favourite), but I wanted something for the desktop/iPad where I could see more data at once over a longer period of time. I couldn’t find anything that did what I wanted, so I built a solution myself. You can see it here.
I’ve set up my data (since January 2019) to appear in a calendar view. Each date will contain the data from the following sources:
Data tracked and imported from external sources
- ❤️ Heart rate & HRV via Apple Health
- ⚖️ Weight and body fat percentage via Apple Health
- 👟 Steps and distance travelled via Apple Health
- 🧘♂️ Meditation sessions via Apple Health
- 🔥 Calories consumed via MyFitnessPal
- 🍱 Macronutrients (carbohydrates, fat, and protein) via MyFitnessPal
- ☕️ Caffeine (coffee, tea, and pre-workout) via MyFitnessPal
- 💊 Supplement intake via MyFitnessPal
- ✅ Number of tasks completed (work and personal) via Todoist
- 🏋 Workouts via MyFitnessPal
- 🧖 Sauna sessions via MyFitnessPal
- 💆♂️ RMT sessions via MyFitnessPal
- 👩⚕️ Chiropractor sessions via MyFitnessPal
- 🛫 Places I'm travelling to via Nomad List
- 🏊♂️ Dives I've completed via ScubaEarth
- 🦉 Duolingo XP via Duolingo
- 👨💻 Productivity via RescueTime
- 🐙 GitHub Contributions via GitHub
- 📖 Reading progress via Goodreads
Data entered manually into a Google Sheet
- 🆙 How long I'm using my standing desk
- 🔬 Thyroid-stimulating hormone (TSH) results
- 🏆 Accomplishments
- 🤒 Sickness
- 🤕 Injuries
- 🌴 Vacation
- 📝 Bucket list
Fetching the data (the technical stuff)
From Apple Health
In iOS 12, Apple added Siri Shortcuts which allows for some powerful scripting. I wrote a script that goes through my Health Samples, formats the data as valid JSON, then saves the file to my server via SSH. Unfortunately at the time of writing, Apple doesn’t allow these shortcuts to be triggered automatically, so I usually run them manually when I arrive at work in the morning.
From MyFitnessPal
I could have leveraged the same Siri Shortcuts to upload my nutritional data, but MyFitnessPal is notorious for having missing or duplicated entries when synced with Apple Health.
MyFitnessPal also has an invite-only API. I’ve been on a waiting list for years, but have never heard anything.
I came across this mfp module which loads a user’s printable diary and scrapes the data from there using cheerio. I’ve modified it to scrape exercise, caffeine consumption, supplements, and RMT sessions. I collect this data every 10 minutes with a Node.js task run with Heroku Scheduler and save it on my server as a JSON file.
Leveraging IFTTT
Conveniently, Google Sheets are importable as JSON objects. IFTTT has plenty of applets that add a new row to a spreadsheet when triggered by an action. A new row is created whenever:
- I mark a task as complete on Todoist
- My reading progress is updated on Goodreads
- A daily productivity summary is created by RescueTime
No APIs, No Problem
Recently I’ve fallen in love with scuba diving. I use ScubaEarth to track my dives, but unfortunately there’s no API. Again using Heroku Scheduler, I launch an instance of Puppeteer to log in to my account and collect the page’s data from my logbook. The data is parsed into a JSON object using cheerio and then uploaded to my server.
Duolingo has an API, but it’s undocumented and (as far as I can tell) doesn’t have exactly what I need. Using the same Puppeteer instance, I go to my duome profile and collect my recent XP. The new XP JSON objects are added to the existing array of XP objects and uploaded as a file to my server.
The Easy Stuff
Pieter Levels of Nomad List is awesome and allows anybody to access their data as JSON. All that’s required is a simple fetch of https://nomadlist.com/@craigcarlyle.json.
Pushing Further
I thought it’d be cool to map out my travels and dives, so I added another page that does that. The data is listed and visualized with Mapbox.
I also thought it’d be cool to share my bucket list. This is just a simple Google Sheet that’s imported as a JSON object.
Final Thoughts
Although it’s far from elegant this has been a fun project to hack away on. Not only have I built an app that I needed in my life, but it helped me get over my imposter syndrome (something we all have). One of my goals was to teach myself React as I’ve mostly been working in AngularJS and vanilla JavaScript/TypeScript for the past 6+ years. Although it’s not perfect, I’m now comfortable working with React.
At any point I could have said “oh, this is too hard", or "there’s no API so it can’t be done”. Finding creative solutions (like using Puppeteer to scrape data) has helped boost my confidence as a developer.
If anyone has any questions, please don’t hesitate to reach out. I’d be happy to answer them. Thanks for reading!
Top comments (31)
Site looks awesome!
Are you planning on open sourcing the project?
Thanks!
Yes, that's the plan. I need to do some cleanup and write some tests first, but I'll be publishing it to my GitHub account.
Awesome 🤩
Kudus to the project!
Great job! Keep us posted! Can't wait to try it myself 🚀
I've been wanting to do something like this forever, but always end up staring at the wall when i think GUI.
This is awesome inspiration, thanks!
Not sure i personally would share my own data so blatantly tho. :)
This is super cool! I've been looking into creating a similar sort of thing, but more focused around goals.
Great idea! Can't wait to see it 😊
It looks awesome! and I love your initiative to build something that you need in your life and at the same time you use it to learn/practice new things and improve your career.
In my case, I am still stuck in the what to build? is it useful? I can not do this, I will take so long time, and so on. Reading your story inspires me to try to do something. Thanks for sharing Craig.
Never heard of rescuetime before. Thanks for the tip.
This is the kind of content I come to DEV for! It's awesome to see so many pieces fit together, and ingenious solutions crafted where existing APIs could not be found. Thanks for sharing!
This is neat. 👍
Congrats for losing 50 pounds and bringing your idea to life. As other wrote, I had a similar idea few years ago: to stay consistent with life goals, I drew a calendar which contains all weeks for 20 years ahead on A4. From one side it kept me motivating with pursuing career goals, travel goals, growth goals, and to remind of all the things I did in difficult times. From one side this kept me motivated mad from another side seeing 20 years of your life on A4 paper was a bit depressing - life passes pretty fast.
From technical perspective gathering all those data for research would be of incredible value for companies in different niches.
What a fun project! great Idea to increase our productivity! 👍👍👍
WOW!! This is so cool!