The highest employment rate in our history is not a dry fact, it means more people with the security of a pay packet and a brighter future.

— David Cameron (@David_Cameron) March 18, 2015

Indeed it isn’t a dry fact, dry facts are things you can reference and cite, and you haven’t.  More to the point, the very wonderful and independent fullfact.org have already taken this one apart…

Anyway…

In November, and then again last week I presented automatically generated rankings of how likely MPs are to cite their sources.  Today we’re going to look in much more detail at the ten MPs who *do* cite most of the sources.  We want to know two things:

• which of the MPs are really doing it properly
• how accurate are the automatically generated rankings.

Along the way we’ll discover a couple of other interesting things.

So as I told you in advance, I used my normal script to download all the tweets from all MPs and rank them. The top ten MPs happened to be:

• David Cameron
• Ed Balls
• Ed Vaizey
• Jon Cruddas
• Liz Kendall
• Mark Lazarowicz
• Sadiq Khan
• Steve Baker
• Tessa Jowell
• Vince Cable

(presented alphabetically)

I then took all of the tweets from those ten and anonomised them so that examples like this:

1m vanished from electoral register in just 1 year – why are Tories & Lib Dems so complacent with our democracy? (£) http://t.co/uJ817FvfXm

— Sadiq Khan (@SadiqKhan) February 27, 2015

Became:

“1m vanished from electoral register in just 1 year – why are Party & Partys so complacent with our democracy? (£)”

I replaced all twitter tags with “@someone”, all Party names with “party” and removed links. I also replaced names of major leaders with “Leader” and so on.  We were left with an anonomised corpus that you can see here.

This means you occasionally find you are now looking at tweets like:

“@someone @someone @someone @someone @someone @someone @someone @someone so lovely to meet you all!”

Marvellous.

So – those anonomised tweets (all 1,500 of them) were reviewed by three people (myself and two others) to see if they would be counted as a ‘fact’. To be clear, we mean – “fact that should be referenced because it’s something we might argue about”  For an example this is a fact:

Figs. reveal record high 49m visits made to museums & galleries directly funded by @DCMS last year, up 4% on 2012/13 https://t.co/8XPpT4LiwX

— Ed Vaizey (@edvaizey) February 12, 2015

But this is a diary entry:

In a statutory instrument committee on drug driving endorsements on licences

— Steve Baker MP (@SteveBakerHW) March 9, 2015

Some things are on the edge:

“If the Party win the election, Britain will face the biggest spending cuts of any major advanced economy”

While one could argue that this is presented as a fact, it’s understood by tone that this is a guess (which politician said this is, of course, left as an exercise for the reader).

So the reviews came back (you can see them all in the Google spreadsheet and we accepted majority vote (so if two of us thought it was a fact that needed a reference, then it was a fact that needed a reference). There will doubtless be errors and omissions, and indeed, you can probably find many of both, but we did the best we could, unpaid, and with hours of effort.

How many ‘facts’ are there?

We, as humans found 232 facts in 1,530 tweets.   This means that:

Among 10 of the MPs that were most likely to cite sources, the panel found that about 16% of their tweets needed references.

I suspect this number reflects the general noise of communication.

How effective is the automatic ranking?

One of the things that is an issue with the automatic ranking that you see on the previous page is that it works by a fairly simple heuristic of “If the tweet has a number in it, it’s probably a fact”.  One of the things I wanted to do with today’s post was look into how accurate that was.  So this table looks into the relationship between how ‘facty’ our judges found it things that matched the heuristic:

This is actually biased against the heuristic because a lot of “With 89 days to go until the election” tweets that made it into the 0 vote sample.

It’s clear that containing a number correlates fairly well with being regarded as a fact by our judges. This is a stronger result than I expected and makes me more comfortable with the ranking as a whole.

So how does the top ten shake out? (Or: Just how sick are you of looking at Ed Vaizey’s Tweets)

Ed Vaizey tweets a lot, a very lot. And he’s pretty good at referencing them

Here’s one I like…

The UK is the top #advertising market in Europe, ahead of Germany, and is predicted to top £9bn by 2016 #digitaluk2015

— Ed Vaizey (@edvaizey) February 3, 2015

…although I’m unconvinced that having lots of advertising is a good thing… In any case I quite like Ed’s tweets, although they do have the slight feel that he may have privatised his Twitter and sold it to Sky Broadband.

In any case this is how the world shakes out for the top ten, ranked by percentage:

(The eagle-eyed amongst you will notice that there are slightly fewer tweets here than you would expect – that’s got a lot to do with the person who stole my laptop with the full corpus on it last week)

The thing to notice is: this isn’t that many facts for some of these guys. It’s interesting to consider what the purpose of an MPs twitter is (and, of course, they might use them differently) is it to respond to constituents? Promote debate? Defend their actions? Make arguments to sway people? Or is it to throw mud, distort issues and generally heckle? Almost everything I want an MP to do requires referencing their sources….

Some extra bits…

For those paying close attention, you’ll notice that the top ten MPs in this article isn’t the same as the top ten MPs in the last article. That’s because I download the MPs data in march and then passed (the anonmised) data onto my judges.   Then my laptop got stolen with the corpus on.  I redownloaded the main data and published it last week but it was different data to the data my judges had worked on.  So today’s post is based on the March data and last week’s is based on May data. Hope that clears things up.

Hello everybody!  As I promised weeks ago, I now present the ranking of British Members of Parliament (as was, they currently technically aren’t because we’re in the short campaign) by how much they reference sources on Twitter.  The methodology is here (see the bottom of this post for some clarifications) and the motivation is here.

This is an automatically generated list of MPs, my code counts the number of times that MPs have figures in their tweets and the number of times they are accompanied by a link.

So this is good:

…and this is bad:

Obviously it’s less than perfect scientifically, but if your MP is in the bottom quarter of the list, then perhaps it’s worth asking him/her why…. (if they aren’t on the list at all, it might be because either they lack a Twitter account or because they made less than five tweets that included a number).

So, the list:

In general I think removing MPs with a small number of tweets has been sensible (although they are still available in the github). I think that it’s nice to see that some of the bottom 16 from last time have upped their game a little bit, but there is still a lot more to do.

I’ve delivered this slightly differently than I talked about in previous posts.  I was very clear about my methodology and approach and even announced when I’d collected the corpus for anonomising. The plan was to do a main list, and then do something special with the top ten people.

Unfortunately my laptop was stolen on Tuesday night, and with it the whole of my corpus.  I (or my reviewers) still have the anonmised version that they’d worked very hard on, but the resources I need to match up the anonomised tweets with the real ones have gone.  So that’s rubbish.

So, the full ranking of MPs above was compiled in the last 48-hours. As before you can view the code on github (modified to remove tweets before 2015), as well as each individual corpus. In the next post I’m going to do an in-depth look at the top ten, unfortunately it might be the a slightly different top 10 than given here…

 

1. good science is setting out your method in advance.
2. this is the sort of thing where it’s easy to accuse people of political bias
3. because I’m going to ask for people to help!

However, for the next version I’d like to do something a little more impressive and scientific. Here’s the process I’m going to follow, paying particular attention to the changes from last time.
First: cull the MPs with few tweets. Once I’ve downloaded the fresh data set, I’m going to cull all of those MPs who had fewer than five tweets that included numbers (a worrying thing on it’s own I think, but irrelevant to the main issue), this stops strange things like Glenda Jackson coming very high up because she happened to cite a source in the one tweet she made with a number.
Second: extract the top and bottom 10 tweeters. Culling the MPs will give us a basic ranking. I’m then going to take the top 10 and bottom 10 MPs and subject them to a more rigorous examination. I’m going to anonomise their (full set of) tweets by removing their name and also keywords like ‘Tories’ or ‘Labour’ and scramble the order (I’m also going to remove any links that exist). This anonomised set will go over to a range of volunteer reviewers who will go though and pick out those tweets that represent facts.

Third: Publish

How you can help