How I got my Apple Watch to give me sleep data going back to the first day I bought it.

The apple watch has some sleep tracking apps, but they all have the major flaw that you have to switch them on when you go to bed and switch them off when you wake up.

For me, this makes them unfit for purpose – it’s those days when I am too tired to remember to flick the switch that are the ones that really matter to me.  I want something totally automatic.

So that’s the problem.  Now let me show you my a week of my Google Calendar from a few months ago.

 

The green events are when I’m asleep, the purple ones are when I’m cycling, and the pink ones are when I’m using public transport.

That week I was working in Bounds Green on a very tiring project for whitewaterwriters.com. It was so tiring that on one evening I was abed by 8pm.  I was also building up to cycle all that way – as you can see I started by breaking the journey down, then tried to bite off more than I could chew on Thursday and gratefully got the train all the way there on Friday.

Here’s the cool thing – I didn’t collect any of that data on purpose.  I wasn’t tracking my sleep, or my movement, or anything else at this sort of detail. All of those events where automatically generated for me, after the fact.

The cycling appears on the calendar via the wonderful StravaICAL, and the other transport information comes in via some other code I wrote that imports my Oyster records. I’m going to use this post to tell you about how I managed to us my Apple Watch to give me sleep data going back more than a year to the first day  bought the watch.  This post is going to talk about how I did it, how you can do it, and what I learned from the process.

How I got the old information

Like almost everybody I charge my watch overnight. The last thing I do at night is put it on charge, and the first thing I do in the morning is put it on my wrist.

I’d also been wondering about sleep tracking for a while.

I tried to export the ‘time under charge’  log files from the watch  then I’d have a pretty good sleep tracker… Or at least ‘bed’ tracker, which is most of what I want.

Unfortunately I couldn’t get access to that.

I tried to see if there was any way I could regularly export the ‘lock’ and ‘unlock’ events from the watch, because that would be a good proxy for bed as well.

No luck there either.

However, I did discover something I could export.  My heart rate data.  Every ten minutes the watch takes a measurement of my heartrate. If it’s not on my wrist, then there is no measurement.

I used the app QS Access (app store link) to give me the file.  The process is: open app, press the ‘i’ by ‘heart rate’, choose ‘tabulate all samples’ and export the file to Dropbox.   It saved as ‘Heart rate.csv’.

Turns out that the Heart-rate data gives you pretty much everything you need.  Once I wrote the code,  which you can see here (run with ‘./watson.py sleep’, I was able to pull out data like this:

   25/04/17 22:42 to 11:09 (12:27)

    26/04/17 22:10 to 08:57 (10:47)

    27/04/17 22:56 to 07:34 (8:38)

    28/04/17 23:44 to 08:36 (8:52)

    29/04/17 22:37 to 09:13 (10:36)

    30/04/17 23:20 to 09:07 (9:47)

    02/05/17 00:47 to 08:05 (7:18)

    02/05/17 22:31 to 09:41 (11:10)

    03/05/17 23:56 to 08:00 (8:04)

    04/05/17 22:33 to 08:55 (10:22)

1st of May was a real late night – exactly the sort were I would normally have failed to switch on a sleep tracking app.

A little bit more code made it easy to automatically have this appear on my Google Calendar as you’ve seen above (I produce an ICS file and send it to a server, you could simply open it periodically in iCal)

What I’ve learned

The first thing I did when I got that lovely data was check that it was representative.  There were a few days where I’d forgotten my charger on a trip or similar that needed to be taken out. But it was mostly in good repair.

I wanted to check that the old data was fairly accurate so I did two things.

  • First, I’d attempted to use a different app for sleep tracking a few months ago (I only lasted a week) – I dug out that data and compared it to the watch data – it was the same to within a couple of minutes.
  • Secondly, I wore the watch normally for the next watch (making no intentional changes to my sleep habits), but with some awareness that it was now tracking me.  After that month I compared the ‘aware’ data with the ‘unaware data’. The averages and the standard deviations of sleep time, wake time, and length of sleep were pretty much the same so

I was happy to conclude that:

  • The data I have for the last year of my sleep is pretty accurate
  • I make NO unconscious changes to my sleep patterns now I know I’m being monitored.

There data itself was fascinating  – and I’ve made some lifestyle changes to see if I can improve matters – I’ll write another post in future looking at if they work or not.  For now – people who charge their watch like I do now have a way of getting to their data.

One day you die falling

“One day you die falling” turns out to be a unique phase on google.  I’ve misquoted it from the original which was:

“I wonder if it’s like this for mountain climbers, he thought. You climb bigger and bigger mountains and you know that one day one of them is going to be just that bit too steep. But you go on doing it, because it’s so-o good when you breathe the air up there. And you know you’ll die falling.”

There’s a thing I’m trying to articulate, which is:

Me has job to do

Life puts obstacles in the way

Me: I’ll never do this

Me: overcomes obstacles, does job

(later)
Me: has same job to do again

Life bigger obstacles, less time

Me: Never going to happen

Me overcomes obstacles, does job

(later still)

Me has same job to do

Life: lions, tigers, bears, complete lack of resources

Me: Definitely never happening now, NO chance

Me: well, I did say that last time as well

Life stuff is now on fire

Me: FFS

 

 

Simplewriter

I’m increasingly using SimpleWriter to write articles, documents and all sorts of other things.  SimpleWriter was made by XKCD – it gives you a window to write in and any word that is outside the ‘ten hundred’ most common words turns red.

There are three reasons this helps me:

  • It gives me a ‘game’ to play while I’m making something ready for other people to read.
  • It makes my writing clear and easy to access.
  • It helps me fight my ‘urge to seem more clever than you’.

I wanted to show you how different this approach was by giving you the before and after of the last thing I wrote here.  I can see on one side that there is lots of ‘fluff’ and lots of things that are only to show off in language. Everything on the other side looks much clearer.

 

Before After
“Even when Chivalry was not angry, being Skilled by him was like being trampled by a horse. Or ducked in a fast-flowing river, more like. He’d get in a hurry, and barge into you and dump his information and flee.” “Even when Chivalry was not angry, being Skilled by him was like being trampled by a horse. Or ducked in a fast-flowing river, more like. He’d get in a hurry, and barge into you and dump his information and flee.”
Robin Hobb, Assassin’s Apprentice Robin Hobb, Assassin’s Apprentice
I worry that I can be quite a demanding conversationalist, particularly with people I feel are on the same wavelength. I have a real need to be challenged by the subject, either directly (recently, I spent the weekend with a medic who jumps out of helicopters for a living and it was a fascinating thing to learn about) or at meta-level (when you are working you a silly thought experiment with like-minded friends for the joy of playing with ideas). If there is someone I want to help I naturally want to help by giving information. I worry that I can be quite a hard person to talk to. I want to cut to the center of what we are talking about, so I ask questions that are hard to answer, attack other people’s points, and, when I’m talking – I talk quickly and for a long time to try and drive my point home.
Of course, the problem is this: if I’m interested in you, it can feel like a interrogation; if I want to help, it can feel like a brutal lecture. Spending time in Computer Science academia hasn’t helped; interrogation of the assumptions and brutal laying down of the facts is a target rather than a disadvantage. The problem is it can feel like a job interview if I’m interested in you, and if I want to help, it can feel like a lecture. I got away with this for a long time working in a university – It’s far too normal there.
Well, maybe he thought that your speeches were obscurantist policy tracts Well, maybe he thought that your speeches were obscurantist policy tracts
lost in a cul-de-sac of their own internal self-righteousness and groaning from the lost in a cul-de-sac of their own internal self-righteousness and groaning from the
weight of statistics. I’m just speculating. I can’t say for sure. weight of statistics. I’m just speculating. I can’t say for sure.
Will Bailey, West Wing “Arctic Radar” Will Bailey, West Wing “Arctic Radar”
I feel quite bad about having (relatively unintentionally) ridden roughshod over people who might occasionally like to get a few sentences of their own in amongst my ranting. I feel quite bad about having shouted down people who might like to get a few words of their own in between my yelling. I want to turn this ‘attacking’ off. Maybe use it sometimes, but as a choice, rather than ‘normal’.
I’d like to be able to switch this rather dominant approach off. I’d like to still be able to communicate this way, but certainly have it as an option rather than a default setting.
I’d like to be consciously taking less.
So this year one of my goals was to “swallow 100 stories in 2015”, it’s a very simple goal, all I had to do was, when an anacote occurred to me just keep quiet. So last year one of my goals was to “swallow 100 stories in 2015”. All I had to do was keep quiet when there was a chance to tell a story, or a ‘useful’ fact. 100 times.
This was really hard. Really hard. By May I’d managed to avoid telling about 20 stories. It’s been good thing to do in all of those cases, but it’s really really hard. This was really hard. Really hard. By May I’d managed to avoid telling about 20 stories. It’s been good thing to do in all of those cases, but it’s really really hard.
It’s also unclear if it’s more or less mindful. Either you can be about the topic of conversation or the conversation; either you are showing an awareness of the connection between you as people, or you are totally in the moment. This goal turned out to be ‘learn an completely new way of having a conversation’. A way that was took more notice of:
I think I’m mostly in-the-moment: what’s important to me is to think that a conversation is two or more people exploring an idea together. the topic; the other person; the time we had left to talk;what other things we were interested in talking about; how much detail we could go into
(I’d admit that sometimes my mind is on something else entirely, but that’s another thing).
So actually, what this goal turned out to be, was learning an entirely new way of having a conversation – being mindful of both the topic and the other person, of the amount of time we had left to chat and of how deeply we wanted to stretch out ideas. For someone who is used to the idea of conversation as ‘a tool to get the most information both out of me and into the other person and out of the other person and into me’ this is actually something of a departure. My old way was ‘get the most information out of me and into you and out of your and into me’, so this was different.
In fact, writing this four months in, I would now state the goal as “have a more conscious awareness of conversation and demonstrate that awareness by consciously passing up the chance to relay a story.”
I left the draft here and came back to it in November.
It’s now November, and I’ve just about got though my 100 stories untold. Some are refraining from boasting (Not telling a housemate about a BBC interview I had to turn down) some are avoiding an attack (I bit my tongue more than once with people who I felt deserved to be given a bit of a reality check) some are just me letting another person talk more. It’s been a really positive change. I’ve got a long way to go yet. But it’s nice to be taking steps on the road. I did get thought my 100 stories. Some where stopping myself from boasting, some where avoiding an attack, and some where leaving space in the conversation for the other person to break. It was a really good change. I’ve got a long way to go yet and I’m looking forward to taking more steps on the road.

100 stories untold.

“Even when Chivalry was not angry, being Skilled by him was like being trampled by a horse. Or ducked in a fast-flowing river, more like. He’d get in a hurry, and barge into you and dump his information and flee.”
Robin Hobb, Assassin’s Apprentice

I think I can be a hard person to talk to.  I try to cut to the centre of what we are talking about, so I ask questions that are hard to answer, attack other people’s points, and when I’m talking I talk quickly and for a long time to try and drive my point home.

The problem is it can feel like a job interview if I’m interested in you, and if I want to help, it can feel like a lecture.  I got away with this for a long time working in a university – It’s far too normal there.

Well, maybe he thought that your speeches were obscurantist policy tracts
lost in a cul-de-sac of their own internal self-righteousness and groaning from the
weight of statistics. I’m just speculating. I can’t say for sure.
Will Bailey, West Wing “Arctic Radar”

I feel quite bad about having shouted down people who might like to get a few words of their own in between my yelling.  I want to turn this ‘attacking’ off. Maybe use it sometimes, but as a choice, rather than ‘normal’.

So last year one of my goals was to “swallow 100 stories in 2015”.  All I had to do was keep quiet when there was a chance to tell a story, or a ‘useful’ fact.   100 times.

This was really hard. Really hard. By May I’d managed to avoid telling about 20 stories. It’s been good thing to do in all of those cases, but it’s really really hard.

This goal turned out to be ‘learn an completely new way of having a conversation’.  A way that was took more notice of:

  • the topic;
  • the other person
  • the time we had left to talk
  • what other things we were interested in talking about

My old way was ‘get the most information out of me and into you and out of your and into me’, so this was different.

I did get thought my 100 stories in the year.  Some where stopping myself from boasting, some where avoiding an attack, and some where leaving space in the conversation for the other person to direct.   It was a  good change.  I’ve got a long way to go yet and I’m looking forward to taking more steps on the road.

Making your priority list actually work.

After a year of a bad system, I’ve finally worked out how to use priorities on my ‘to do’ list.   I’ll explain what I did in this post, I hope it helps people.

Note – I wrote this post about two months ago, I wanted to keep it quiet so that I could check that everything was still working months later.

About a year ago, I followed the work of people like Randy Paush and started marking things on my ‘to do’ list with how important they were.

These where the markings I used.

  • 0 – not known, might be very very important, must sort
  • 1 – “Send Funding proposal” so anything to do with money and making sure I had a roof over my head.
  • 2 – “Fix Gym Membership” Things to do with my 2016 New Year’s resolutions.
  • 3 – “Review Jane’s Proposal”  tasks that are part of larger projects
  • 4 – “Get birthday present for Steve” tasks that are social in nature – friends and family.
  • 5 – “clean house” all the ‘normal’ things we need to do as humans
  • 6 – “See if you can make a cup holder from wood”  ‘play’ things.

I’ve been mostly ignoring them.  They have an effect (the fear chart counts 1’s as having six times more of an affect on the chart than 6s) But I really don’t have an urge to work on the low numbers first, which is the whole point.

The reason was that they were wrong.  Kind of. You see, those numbers were exactly for someone focused on building a business, or making themselves better. But it turns out those aren’t the things I care about. I care about other things, both for myself, and for other people.

So something that is a lot more important to me than success is integrity.  So I wrote out the markings again with that in mind.  Here’s what I ended up with:

  • 0 – not known, might be very very important, must sort
  • 1 – “Send James the new slides they need” – actions that I have committed to and that a named person needs before they can do something.
  • 2 – “Lookup Best hotel in Paris” –  Like 1, but these are actions that belong to projects that I have committed to and that a named person needs. (in this example I might have committed to “Organise holiday with girlfriend” but I can take my girlfriend to paris and stay in the second best hotel)
  • 3 – “Go running”: things I have said I will do (to at least one person or in public), but only I care about.
  • 4 – “Pack running bag” Like 3, but part of a project.
  • 5 – “Make doctors appointment” actions I have promised myself, or I feel make me the person I like being
  • 6 – “See if you can make a cup holder from wood”  ‘play’ things.

 

The big sign I got that this was going to work was this: as soon as I changed the markings on my old ‘to do’ list, I got a big jolt of guilt from looking at all of the 1’s together.  So guilty that I had to get up and walk around the room – it was as if I had found all the things on my list that were causing me some worry and concentrated them together.After I pulled together the willpower to tackle them, I felt a lot better, and I think that’s probably a sign that the new list is a bit more “aligned with my values”

So the call to action is this – mark your ‘to do’ list with the things that matter to you as you are, not who you think you are, or who you want to be.

That week – 9th Jan 2017

A screenshot of my calendar. Filling out this alt text I'm thinking of ways of making a text based one... I'll have a proper think...

I want to take more time to check in on myself and see how I’m spending my time.  This was going to be a “What have I been up to this week” post but I found I actually had little memory of what happened.   The calendar is pretty useful – most of it (everything that isn’t blue) is added automatically, so it can give a really good idea of what I actually did rather than what I think I did.

 

First cool thing was that I was up in Preston for a few days helping some university students writer this:

 

Screen Shot 2017-01-16 at 09.50.19

 

Which was very cool and exciting. Looked like this a lot:

Screen Shot 2017-01-19 at 15.22.54

While I was up there I did some nice bouldering  up at West View in Preston, which was nice. Far too much time trying to work out a V6 bouldering problem on a moulded wall until it occurred to me that it was on a moulded wall.  Clearly I’ve spent too much time climbing indoors on wood.

Apart from that I’ve done a lot of time coding on The Open Voice Factory – which should be worth a couple of blog posts of their own in the near future.

Went out to support my girlfriend running on the Sunday as well, very proud of her.

Thoughts I had while applying for the Digital Agenda Awards.

I’ve just spent an hour entering the Digital Agenda Awards.

Entering for prizes is a strange thing to do as a nonprofit. You have to shake off the feeling that you should be doing something more important. This gets worse when you notice that the questions you are filling out are very like the questions in the application for money that you should be working on.

One reason I’m entering is for the ego – I really think that the projects I’ve put forward (The Open Voice Factory and White Water Writers) are very strong and deserve to win. More to the point, it would be good if more people knew about them.

The other thing that’s a bit strange about Prizes is that it feels wrong to nominate yourself. The Digital Agenda awards try and have this both ways: you out your contact information, and then fill out a different set of contact information for the person you are nominating, which is also you. This means you are feeling like a fake when you reach the questions, which are:

  • The idea – what is your core proposition?
  • Product or service – describe the digital innovation that you offer
  • Team – tell us about the people working on this
  • Ability to scale – explain how you plan to grow your take-up and reach
  • Impact – explain how your product or service makes a positive social and/or economic change and what evidence you have to support this

… and would be very hard to fill out if I was nominating someone else.

I feel like that the best way of thinking about entering for prizes as a nonprofit is that they are a ‘proposal training’. They are short tests to check you are thinking strategically about what you want to do and how you want to do it. If you start one and haven’t got anything to write, that might be a sign you have to do some long-term thinking…

Stress is fear

“Stress is the achiever word for fear”. I heard this on the Tim Ferris podcast while driving and it really hit home for me.  I’ve set my computer so I now can’t, even write the word – if I try and write “fear” I write “fear” instead (See?).

The reason it hit so hard was because it makes it clear that so often when we say “I’m stressed” (I’ve turned off text-replace for the moment) we are using it as a status thing.  We’d never say “I’m scared” in the same situation – even if it’s true. We think we are coming over as “I have so much depending on me (I am important)” rather than “I have promised things (some to myself) and I’m scared I won’t deliver)

We should start. We should start calling fear fear, the world will be better for it.

DIY charging station

A quick note on how I made a charging station out of a drying rack.

Like everybody, I have many many things that want to be charged. I also have a mess of many spare cables.

I’d had enough of this and had an idea.   I bought a multi-usb charger from Amazon (this one)  and a simple, wooden drying rack(this one).

I built the rack,  and popped the things I wanted to charge in with cables.

 

Screen Shot 2017-01-16 at 11.15.20
I worked out the length the cables should be and taped each (all spares) to the bottom of the rack (The rack also needed reinforcement, but yours might not).

 

Screen Shot 2017-01-16 at 11.15.39
I was left with lots of cables of different sizes.

Screen Shot 2017-01-16 at 11.15.49

 
I used the ‘toilet roll trick’ to make the cables the same length.

Screen Shot 2017-01-16 at 11.16.23

I then added the adapter….Screen Shot 2017-01-16 at 11.15.49

Screen Shot 2017-01-16 at 11.16.34

and slid the new station into the space for it.  Volia. Screen Shot 2017-01-16 at 11.16.54

How I stopped being a Grammar Nazi

It’s been a little over three months since I stopped being an arse about semantics.

I didn’t think I was at the time, I thought I was just helping make sure everyone was understood. If people are talking about things in the correct way then communication is easier and the world gets better… right?  I’ve even written angry things about it.

 

Turns out I was being an arse.  And the way I found out was thought a web comic.  This one:
Full, detailed and wonderful description of this complex image is available at https://www.explainxkcd.com/wiki/index.php/1735:_Fashion_Police_and_Grammar_Police
…and it stopped me stone dead at “Vindictive about things that are often uncomfortably transparent proxies for race or social class”.

I’m sensitive about class issues, and very aware that I should be doing more on issues of race and it was an awful jolt to see it be put that way.    Instantly it was clear that I had become something that I hate, and that the hate was far stronger than any reason I had to keep up the behaviour.

I’ve not ‘corrected’ anyone’s grammar since September, and I’m working hard to avoid anything of the “I think you mean…” variety. I’m trying to check my understanding obviously, I’m trying to say things like “Can I check I understood? ” when there’s doubt, and to shut the hell up when there isn’t any doubt, but old habits die hard.

 

Anyway, this is a public declaration that I’m stopping.  If you catch me doing it, feel free to tell me I’m a liar as well as an arse.

 

Everything I watched on TV in 2016

Netflix and Amazon Instant video both let you see the list of shows you’ve watched (I’ve also watched about two dozen or so DVDs, six trips to movies, two things on the iPlayer and no live TV – but I’m leaving those out for this)

Most of them where fun, but seeing the whole list together seems kind of big. It starts to look like a lot of time watching TV and not much with friends…. I’m making my 2016 TV public (via copy and paste) for two reasons:

  • It will push me to make my 2017 TV list will look less like a list of the friends I didn’t hang out with.
  • It will mean my friends who want to talk to me about TV at least have a starting point.

The missing numbers on the Amazon list are because I took out the things my other half watched without me.

My Netflix in 2016

…. (I only got an account late in the year)

26/12/2016 Peppa Pig: Season 2: “Bubbles / Emily Elephant / Polly’s Holiday / Teddy’s Day Out”
21/12/2016 Jack Ryan: Shadow Recruit
15/12/2016 Power: Season 1: “Whoever He Is”
12/12/2016 Transformers: Age of Extinction
12/12/2016 Mission: Impossible
12/12/2016 Limitless
12/12/2016 House of Cards: Season 4: “Chapter 52”
11/12/2016 House of Cards: Season 4: “Chapter 51”
11/12/2016 House of Cards: Season 4: “Chapter 50”
10/12/2016 House of Cards: Season 4: “Chapter 49”
10/12/2016 House of Cards: Season 4: “Chapter 48”
10/12/2016 House of Cards: Season 4: “Chapter 47”
10/12/2016 House of Cards: Season 4: “Chapter 46”
10/12/2016 House of Cards: Season 4: “Chapter 45”
09/12/2016 House of Cards: Season 4: “Chapter 44”
09/12/2016 House of Cards: Season 4: “Chapter 43”
09/12/2016 House of Cards: Season 4: “Chapter 42”
09/12/2016 House of Cards: Season 4: “Chapter 41”
09/12/2016 Skiptrace
08/12/2016 Iron Man: Rise of Technovore
03/12/2016 House of Cards: Season 4: “Chapter 40”
28/11/2016 Harlock: Space Pirate
26/10/2016 Power: Season 1: “Not Exactly How We Planned”
08/10/2016 Archer: Season 2: “Jeu Monégasque”
08/10/2016 Archer: Season 2: “White Nights”
08/10/2016 Marvel’s Luke Cage: Season 1: “You Know My Steez”
08/10/2016 Marvel’s Luke Cage: Season 1: “Soliloquy of Chaos”
08/10/2016 Marvel’s Luke Cage: Season 1: “Now You’re Mine”
07/10/2016 Marvel’s Luke Cage: Season 1: “Take It Personal”
07/10/2016 Marvel’s Luke Cage: Season 1: “DWYCK”
07/10/2016 Marvel’s Luke Cage: Season 1: “Blowin’ Up the Spot”
07/10/2016 Archer: Season 2: “Swiss Miss”
07/10/2016 Archer: Season 1: “Dial M for Mother”
07/10/2016 Archer: Season 1: “Job Offer”
07/10/2016 Archer: Season 1: “The Rock”
07/10/2016 Archer: Season 1: “Skytanic”
06/10/2016 Archer: Season 1: “Skorpio”
06/10/2016 Archer: Season 1: “Honeypot”
06/10/2016 Archer: Season 1: “Killing Utne”
06/10/2016 Archer: Season 1: “Diversity Hire”
06/10/2016 Marvel’s Luke Cage: Season 1: “Manifest”
06/10/2016 Ghost in the Shell
06/10/2016 Marvel’s Luke Cage: Season 1: “Suckas Need Bodyguards”
06/10/2016 Marvel’s Luke Cage: Season 1: “Just to Get a Rep”
05/10/2016 Marvel’s Luke Cage: Season 1: “Step in the Arena”
04/10/2016 Star Trek Into Darkness
04/10/2016 Marvel’s Luke Cage: Season 1: “Who’s Gonna Take the Weight?”
02/10/2016 Marvel’s Luke Cage: Season 1: “Code of the Streets”
02/10/2016 Marvel’s Luke Cage: Season 1: “Moment of Truth”
29/09/2016 Sicario

 

My Amazon Instant Video in 2016

1 The 100: Season 3
2 Frozen
3 The King’s Speech
4 Mission: Impossible – Rogue Nation
5 New Year’s Eve (2011)
6 Parks and Recreation, Season 7
7 300: Rise of an Empire
8 Star Trek Beyond
9 Parks and Recreation, Season 6
10 The DUFF
11 Thomas and Friends – Season 1
12 Teletubbies (Brand New Series) – Season 1
13 The LEGO Movie
14 Batman – Mask of the Phantasm
15 Limitless, Season 1
16 Startup – Season 1
17 Three Kings
18 Parks and Recreation Season 5
19 Room
20 I Am Legend
21 The Corruptor
22 Parks and Recreation Season 4
23 Safe House
24 Ballers: Season 1
25 Dollhouse – Season 1
26 Arrow Season 3
27 Above the Law
28 Ong Bak 3
29 Parks and Recreation – Season 1
30 Gone with the Wind
31 Mr. Robot – Season 1
32 Mr. Robot – Season 2
33 Black Sails, Season 3
34 Live Die Repeat: Edge of Tomorrow
35 The Dark Knight Rises
36 The Last Boy Scout
37 Extremely Loud and Incredibly Close
38 Forever Young
39 Black Sails: Season 1
40 Black Sails, Season 2
41 The Imitation Game
42 Suite Française
43 Marvel’s Agents of S.H.I.E.L.D. Season 3
44 The Walking Dead Season 5
45 Batman and Mr Freeze – SubZero
46 Fear the Walking Dead Season 1
47 Rush Hour
48 Preacher – Season 1
49 The Man in the High Castle – Season 1
50 Marvel’s Agents of S.H.I.E.L.D. Season 2
51 Generation Iron
52 DCU: Justice League: The Flashpoint Paradox
53 Man Up
54 Pleasantville
55 Frozen [Plus Bonus Features]
56 Frozen [Plus Bonus Features]
57 Maison Close Season 1 (English Subtitled)
58 Mad Men – Season 1
59 Pacific Rim
60 Brave
61 Superman – Unbound
62 The Informant!
63 Bridesmaids
64 Batman: The Dark Knight Returns Part 2
65 Nashville Season 2
66 The Great Gatsby
67 The Monuments Men
68 The Monuments Men
69 National Lampoon’s Loaded Weapon 1
70 National Lampoon’s Loaded Weapon 1
71 Crazy
72 Your Wild Life’s Gonna Get You Down
73 We’ve Got Things To Do
74 Too Far Gone
75 Nashville Season 1
76 Why Don’t You Love Me
77 Beware the Batman Season 1
78 Downton Abbey | Season 1
79 Private Benjamin
80 Parks and Recreation Season 3
81 Parks and Recreation – Season 2
82 Where He Leads Me
83 The Ghost
84 Toy Story
85 Hustle – Episode 3
86 Hustle – Episode 3
87 Poldark Series 1
88 Sherlock Season 2
89 Sherlock Season 1
90 The Great Game
91 Blow
92 Batman – Mystery of the Batwoman

How likely are US Senators and Presidential Candidates to cite their sources?

This page is a list of US Senators (I added Donald Trump and Hillary Clinton) in order of how likely they are to give the source for  figures they use.

The ranking uses the politician’s  Twitter streams. The method is a little bit of a blunt instrument, but it makes some points nicely.

The list works like this:

  • gather recent tweets by the politicians.
  • take those that involve numbers (because we are interested in if they  reference their figures)
  • check if they contain a URL.

So this is good:

Screen Shot 2014-11-23 at 12.42.57

…and this is bad:

Screen Shot 2014-11-23 at 12.42.46

We’re looking at the most important part of information sharing here. If we are to have grown-up conversations we must use information that can be checked. The first step is simply information is saying where your information came from. It makes no claims about the sources that are used – we weight a peer reviewed study as highly as a Yahoo Answers page – the only interest here is if the source is there at all.

You can view the first list here. I checked how good the idea was with humans (and the UK dataset) here and it worked very well. I’m certainly happy that the list is the most simple and visible way of making politicians aware of the information they are putting out.

All of the code is in GitHub, and if you’d like to look at the list a single account you will find it here.

For the first list, someone on reddit said something very important (about the UK version, but it holds):

I think it’s pretty clear that the top 20% of those MPs are doing something right methodologically, compared to the bottom 20%. But I’d be careful of singling out any individual case with such a blunt tool. Especially if all 5 are from the same party

…and you should take that as your general guide. If your representative. is in the bottom half of the table, there is probably something wrong…

 

Ranking Name Twitter Handle Tweets Tweets with Figures Sourced Tweets with Figures/td> Percentage
1 Sen. Jeff Sessions (@SenatorSessions) 1980 111 101 90.99%
2 Senator Thad Cochran (@SenThadCochran) 1649 156 140 89.74%
3 Senator Thom Tillis (@SenThomTillis) 1684 124 109 87.90%
4 Senator Ted Cruz (@SenTedCruz) 3201 246 216 87.80%
5 Joni Ernst (@SenJoniErnst) 1089 36 31 86.11%
6 Senator Joe Donnelly (@SenDonnelly) 3204 379 316 83.38%
7 Senator Roy Blunt (@RoyBlunt) 3241 295 243 82.37%
8 Dean Heller (@SenDeanHeller) 3232 372 302 81.18%
9 Mike Enzi (@SenatorEnzi) 2893 291 235 80.76%
10 Senator Ron Johnson (@SenRonJohnson) 2898 215 170 79.07%
11 John McCain (@SenJohnMcCain) 3235 181 143 79.01%
12 Senator John Hoeven (@SenJohnHoeven) 1593 141 111 78.72%
13 Senator Joe Manchin (@Sen_JoeManchin) 3220 563 443 78.69%
14 Marco Rubio (@marcorubio) 3227 192 151 78.65%
15 Sen. Tammy Baldwin (@SenatorBaldwin) 3224 165 129 78.18%
16 Al Franken (@alfranken) 2762 161 125 77.64%
17 Cory Booker (@CoryBooker) 3224 152 118 77.63%
18 Sen. Lisa Murkowski (@lisamurkowski) 3243 312 241 77.24%
19 Senator Hatch Office (@SenOrrinHatch) 3215 229 176 76.86%
20 Senator Bob Corker (@SenBobCorker) 3045 194 149 76.80%
21 David Perdue (@sendavidperdue) 1867 184 140 76.09%
22 Richard Shelby (@SenShelby) 1055 90 68 75.56%
23 Dr. Rand Paul (@RandPaul) 3237 224 169 75.45%
24 Senator Ben Sasse (@SenSasse) 670 56 42 75.00%
25 Tom Cotton (@SenTomCotton) 3224 131 98 74.81%
26 Rob Portman (@senrobportman) 3210 341 254 74.49%
27 JohnCornyn (@JohnCornyn) 3212 316 230 72.78%
28 Hillary Clinton (@HillaryClinton) 3231 202 147 72.77%
29 Senator Mike Crapo (@MikeCrapo) 3244 215 156 72.56%
30 Jerry Moran (@JerryMoran) 3233 359 260 72.42%
31 Sen. Debbie Stabenow (@SenStabenow) 1721 131 94 71.76%
32 Ed Markey (@SenMarkey) 3228 287 205 71.43%
33 Pat Roberts (@SenPatRoberts) 3198 257 183 71.21%
34 Sen. Susan Collins (@SenatorCollins) 1852 120 85 70.83%
35 Sen. John Barrasso (@SenJohnBarrasso) 3224 466 330 70.82%
36 Senator John Boozman (@JohnBoozman) 2558 231 163 70.56%
37 Senator Angus King (@SenAngusKing) 2301 190 133 70.00%
38 Johnny Isakson (@SenatorIsakson) 2840 266 185 69.55%
39 Senator Dan Coats (@SenDanCoats) 3224 224 155 69.20%
40 Senator Mike Rounds (@SenatorRounds) 1265 128 88 68.75%
41 Senator Mazie Hirono (@maziehirono) 3203 354 243 68.64%
42 David Vitter (@DavidVitter) 3211 295 202 68.47%
43 Senator John Thune (@SenJohnThune) 3245 262 179 68.32%
44 Michael F. Bennet (@SenBennetCO) 1326 157 107 68.15%
45 Mark Kirk (@SenatorKirk) 3238 426 289 67.84%
46 Sheldon Whitehouse (@SenWhitehouse) 3208 229 155 67.69%
47 Sen Dianne Feinstein (@SenFeinstein) 3248 410 276 67.32%
48 Sen. Lamar Alexander (@SenAlexander) 2872 302 202 66.89%
49 Shelley Moore Capito (@SenCapito) 3239 279 186 66.67%
50 Mark Warner (@MarkWarner) 3219 287 189 65.85%
51 Mike Lee (@SenMikeLee) 3239 231 152 65.80%
52 Richard Burr (@SenatorBurr) 2731 225 148 65.78%
53 Senator Roger Wicker (@SenatorWicker) 2774 283 184 65.02%
54 Inhofe Press Office (@InhofePress) 3221 331 215 64.95%
55 Steve Daines (@SteveDaines) 3198 213 138 64.79%
56 Senator Jack Reed (@SenJackReed) 3225 408 263 64.46%
57 Senator Deb Fischer (@SenatorFischer) 2936 233 150 64.38%
58 Senator Pat Toomey (@SenToomey) 3243 348 223 64.08%
59 Sen. McConnell Press (@McConnellPress) 3237 150 96 64.00%
60 Richard Blumenthal (@SenBlumenthal) 3220 213 134 62.91%
61 Tom Udall (@SenatorTomUdall) 2976 772 485 62.82%
62 Kelly Ayotte (@KellyAyotte) 3239 297 185 62.29%
63 Ron Wyden (@RonWyden) 3222 196 122 62.24%
64 Lindsey Graham (@LindseyGrahamSC) 3236 199 122 61.31%
65 Sherrod Brown (@SenSherrodBrown) 2468 286 175 61.19%
66 Martin Heinrich (@MartinHeinrich) 2487 150 91 60.67%
67 Sen. Maria Cantwell (@SenatorCantwell) 2507 252 152 60.32%
68 Senator Dick Durbin (@SenatorDurbin) 3235 328 197 60.06%
69 Sen. Patrick Leahy (@SenatorLeahy) 3243 302 181 59.93%
70 Sen. Heidi Heitkamp (@SenatorHeitkamp) 3231 1213 726 59.85%
71 Tim Scott (@SenatorTimScott) 3199 273 163 59.71%
72 Senator Patty Murray (@PattyMurray) 3233 315 185 58.73%
73 Jeff Flake (@JeffFlake) 1702 87 51 58.62%
74 Senator Gary Peters (@SenGaryPeters) 3196 350 204 58.29%
75 Senator Jon Tester (@SenatorTester) 2210 146 85 58.22%
76 Bill Cassidy (@BillCassidy) 3228 227 132 58.15%
77 Senator Brian Schatz (@SenBrianSchatz) 1106 81 47 58.02%
78 SenDanSullivan (@SenDanSullivan) 1208 288 166 57.64%
79 Senator Tom Carper (@SenatorCarper) 3202 422 243 57.58%
80 Senator Bob Menendez (@SenatorMenendez) 3233 336 191 56.85%
81 Senator Jeff Merkley (@SenJeffMerkley) 3234 214 121 56.54%
82 Sen. James Lankford (@SenatorLankford) 3200 289 162 56.06%
83 Sen. Jeanne Shaheen (@SenatorShaheen) 3242 277 153 55.23%
84 Senator Bob Casey (@SenBobCasey) 3245 296 163 55.07%
85 Cory Gardner (@SenCoryGardner) 3099 284 155 54.58%
86 Senator Chris Coons (@ChrisCoons) 3230 255 138 54.12%
87 Sen. Barbara Boxer (@SenatorBoxer) 2160 291 156 53.61%
88 Senator Ben Cardin (@SenatorCardin) 3209 216 115 53.24%
89 Chuck Schumer (@SenSchumer) 3208 316 160 50.63%
90 Bill Nelson (@SenBillNelson) 572 52 25 48.08%
91 Senator Jim Risch (@SenatorRisch) 449 47 22 46.81%
92 Senator Tim Kaine (@timkaine) 3223 260 121 46.54%
93 McCaskill Office (@McCaskillOffice) 3221 351 155 44.16%
94 Kirsten Gillibrand (@SenGillibrand) 3202 252 103 40.87%
95 Elizabeth Warren (@SenWarren) 1247 87 34 39.08%
96 Amy Klobuchar (@amyklobuchar) 3199 463 171 36.93%
97 Chris Murphy (@ChrisMurphyCT) 3228 426 155 36.38%
98 Barbara Mikulski (@SenatorBarb) 3238 348 120 34.48%
99 Senator Harry Reid (@SenatorReid) 3218 414 131 31.64%
100 Bernie Sanders (@SenSanders) 3228 304 91 29.93%
101 Donald J. Trump (@realDonaldTrump) 3221 180 53 29.44%
102 ChuckGrassley (@ChuckGrassley) 3221 690 147 21.30%