October 14, 2017 at 03:15AM

Today I learned: 1) I've been playing the confidence calibration game (http://ift.tt/1dzDceM) for a while now, and I like to think I've gotten decently good at it. I'm somewhat overconfident in some ranges, but my 60%, 80%, and 99% confidence estimations are pretty much spot-on. The *weird* thing I noticed today is that I am rather *overconfident* when I have 50% confidence. Specifically, when given two possible answers, when I have NO IDEA what the right one is and blindly guess, I am right only 34% of the time, with an N of 29 (10 correct, 16 incorrect). Is this significant? The, uh, "obvious" statistical test to use here is to P-value test, which asks how likely it is that I would see data this "weird" if the null hypothesis (that is, that I actually guess with 50% accuracy) were correct. In other words, the p-value quantifies how surprising the data are under a null hypothesis. The lower the value, the more surprising it is. In this case, assuming binomial distribution of answers, I get a two-tailed P-value of about 0.13. A little suspicious, but not very strong evidence. What if we do this the Bayesian way? Hmm. Well, for this we turn to Bayes' Rule. If p (little "p", not "P", which stands for "probability of") is the probability that I get a random guess right, and D is our observation (10 correct guesses out of 29), then we have P(p|D) = P(D|p) * P(p) / P(D) The probability of getting 10/29 guesses right when each guess has probability p of correctness is given by the binomial distribution, which I don't happen to know off the top of my head but the internet does. The probability of getting the data *at all* is the sum (integral) of all the probabilities of getting that data for every possible value of p (so, ∫Binom(D;x) evaluated from x=0 to x=1). The prior probability of p is the tricky bit, as usual. The typical thing to do here would be to assume we have no knowledge of p and give it a flat prior distribution, so it basically goes away (in fact, it does go away -- a uniform distribution on the range (0,1) is 1 everywhere, so it's just multiplying everything through by one). Plug in the Binomial distribution helpfully provided by Wolfram Alpha and we have P(p|D) = (29c10) * p^10 * (1-p)^19 / (29c10) * ∫[x^10 * (1-x)^29]dx where (29c10) means "29 choose 10" and that integral in the denominator is evaluated from x=0 to x=1. Conveniently, the (29c10) bits cancel, so we can take them out. The integral in the denominator is just a number, which happens to evaluate to about 1.7 * 10^-7. If we graph out what's left, we get this*: http://ift.tt/2yjhcJU Take-home points from the distribution: a) The expected value of p is 35%, which makes sense; b) it's *very likely* that I make random guesses at less than 50% accuracy -- about 95% of that curve is below p = 0.5. So, do I guess at worse than random? Eh, I wouldn't count on it. When it comes down to it, I *don't* put a uniform distribution on the prior for how well I guess at random -- in fact, my prior for that is pretty spiky around p = 0.5. But hey, now I have *some* evidence to the contrary. * Actually, something's wrong with this calculation, because it's not integrating to 1 like a good probability distribution should. Can anyone spot the error? I can't. 2) I *know* I knew this one before, but I forgot, and now I learned it again! Did you know the United States has had an Emperor? At least, he thought he was the Emperor. And he got coins minted after him. Anyway, Emperor Joshua Norton was born an Englishman, moved to San Francisco, lost all his money, went a little nuts, and declared himself Emperor. He was grandiose, for sure, but apparently charming and harmless enough that the locals humored him. Coins, as I said, were minted in his name, and his presence was generally honored and applauded, as were his proclamations. He was beloved enough that 30,000 San Franciscans attended his funeral, even though he owned virtually nothing but a few uniforms, hats, walking sticks, fake letters and bonds, and a saber. Another thing I didn't know about Emperor Norton the first time around -- he was actually arrested at one point by a policeman who tried to have him institutionalized. Public outcry was swift, and the police chief ordered him let free, on the grounds "that he had shed no blood; robbed no one; and despoiled no country; which is more than can be said of his fellows in that line." 3) Everyone knows that college tuition is rising rapidly. Did you know that college *spending* is not? Spending per student (though not, I think, spending per degree granted -- it's not quite the same) has been pretty flat over the last decade (I don't think there's good data from before that).