Likelihood and Bayesian Inference

Lecture 10A handout

Author

Name: _________________

Normal distribution for a population parameter

On Monday, in frequentist inference, a population parameter like \(\pi\) was a blank to fill in.

Another way to think about population parameters: we can use a probability distribution to describe blank to fill in about them. Today we will work with a population mean, \(\mu\), and use a normal distribution to describe what we believe about it.

Example. \(\mu\): average hours of sleep of Harvard students on a school night.

  • Alex believes \(\mu \sim N(7, 0.5^2)\): “around 7 hours, fairly sure.”
  • Sam believes \(\mu \sim N(6.5, 1^2)\): “around 6.5 hours, not very sure.”

Two normal curves describing beliefs about mu, the average hours of sleep of Harvard students. Alex's curve, in green and solid, is centered at 7 and narrow, mostly between 6 and 8. Sam's curve, in pink and dashed, is centered at 6.5 and wide, mostly between 4.5 and 8.5.

Who is more certain about \(\mu\)? How can you tell from the plot?

Probability of \(\mu\). According to Alex’s belief, what is the probability that the average hours of sleep of Harvard students is more than 7.5? The plot below shows Alex’s belief, \(N(7, 0.5^2)\), with the area above 7.5 shaded. Write the probability in notation and give its value.

Normal curve for mu with mean 7 and standard deviation 0.5. The area to the right of 7.5 is shaded pink, and the label reads Area = 0.1587.

Your turn. Use the 68-95-99.7 rule.

  1. According to Sam’s belief, \(N(6.5, 1^2)\), what is the probability that \(\mu\) is more than 7.5?
  1. Who thinks it is more plausible that \(\mu\) is less than 4.5 hours? Find both probabilities.
  1. Write down your own belief about \(\mu\) as \(\mu \sim N(\ ?, \ ?^2)\). Sketch it, and compare with your neighbor.

Same notation, different things. Does each normal distribution describe a random observation, a random sample statistic, or a random population parameter? Then use the 68-95-99.7 rule to find the probability in brackets.

Is each normal distribution about a random observation, a random sample statistic, or a random population parameter?
Normal distribution Observation, sample statistic, or parameter? Probability
\(X \sim N(500, 100^2)\): SAT Math score of a randomly selected test taker \([P(X > 600)]\) blank to fill in blank to fill in
\(\pi \sim N(0.6, 0.1^2)\): our belief about the proportion of all Harvard students who drink coffee daily \([P(\pi > 0.5)]\) blank to fill in blank to fill in
\(p \mid \pi = 0.5 \sim N(0.5, 0.125^2)\): proportion of 16 Randomville babies who choose the helper \([P(p > 0.75)]\) blank to fill in blank to fill in
\(X \sim N(70, 10^2)\): resting heart rate of a randomly selected student \([P(X > 80)]\) blank to fill in blank to fill in
\(\mu \sim N(3.7, 0.1^2)\): our belief about the mean GPA of all Harvard students \([P(\mu < 3.6)]\) blank to fill in blank to fill in
\(\bar{X} \sim N(7, 0.2^2)\): mean hours of sleep of a random sample of 36 students, if the mean for all students is 7 \([P(\bar{X} < 6.6)]\) blank to fill in blank to fill in

Probability vs. likelihood

Flashback: lecture 1. We asked: “If we observe H, H, H, H, T on five coin flips, what is the likelihood that this is a fair coin?” Back then we did not have the words population, sample, parameter, or likelihood. Now we do!

Two diagrams side by side. Left: an oval labeled Population with parameter pi, the probability of heads, and an oval labeled Sample with the data H, H, T, H; an arrow on top goes from population to sample, labeled Probability: P(data given pi). Right: the same two ovals, with an arrow on the bottom going from the sample back to the population, labeled Likelihood: L(pi given data).

  • Probability goes from the blank to fill in: if we knew \(\pi\), how probable is the data? \(\quad P(\text{data} \mid \pi)\)
  • Likelihood goes from the blank to fill in: given the data, how well does each value of \(\pi\) explain it? \(\quad L(\pi \mid \text{data})\)

Is this coin fair? A friend hands you a coin. Let \(\pi\) be the probability that it lands heads. You flip it 4 times and get H, H, T, H. Is the coin fair? Or is \(\pi\) something else, like 0.75?

If we knew \(\pi\): \(P(\text{H} \mid \pi) =\) blank to fill in and \(P(\text{T} \mid \pi) =\) blank to fill in.

The flips are independent, so we multiply. Write \(P(\text{H, H, T, H} \mid \pi)\) in terms of \(\pi\):

Two questions.

  • Probability question: If the coin is fair (\(\pi = 0.5\)), how probable is each possible set of 4 flips?
  • Likelihood question: We observed H, H, T, H. How well does each possible value of \(\pi\) (0.5? 0.75? 0.25?) explain these flips?

Probability: fix \(\pi\), vary the data. If the coin is fair, each of the 16 possible sets of 4 flips has probability \(0.5^4 =\) blank to fill in. If the coin were biased toward heads, \(\pi = 0.75\), each set of 4 flips with \(k\) heads has probability \(0.75^k \times 0.25^{4 - k}\):

Two bar charts of the probability of each of the 16 possible sets of four coin flips. Top, pi equals 0.5: all 16 bars have the same height, 0.0625. Bottom, pi equals 0.75: HHHH has the tallest bar, about 0.32; the four sets with three heads, including HHTH, are each about 0.105; TTTT is about 0.004. In both, the bar for HHTH, our observed data, is pink, and the bars add up to 1.

In each plot, the 16 probabilities add up to blank to fill in. Each is a blank to fill in (over the data).

Likelihood: fix the data, vary \(\pi\). We observed H, H, T, H. Complete the table. Hint: \(0.25^3 \approx 0.016\) and \(0.75^3 \approx 0.42\).

Likelihood of three values of \(\pi\) given the observed flips H, H, T, H
\(\pi\) \(L(\pi \mid \text{HHTH}) = \pi^3(1 - \pi)\)
0.25 (biased toward tails) blank to fill in
0.5 (fair) blank to fill in
0.75 (biased toward heads) blank to fill in

How many times more likely is \(\pi = 0.75\) than a fair coin? blank to fill in

Do these three likelihoods add up to 1? blank to fill in This is a blank to fill in (over \(\pi\)). It is not a probability distribution.

Likelihood curve of pi for the data H, H, T, H. The curve is zero at pi = 0, rises to a peak at pi = 0.75, and falls to zero at pi = 1. Points mark pi = 0.25 (0.012), 0.5 (0.0625), and 0.75 (0.105).

The value of \(\pi\) with the highest likelihood, blank to fill in, is the sample proportion \(\frac{3}{4}\). It is called the blank to fill in.

So is the coin fair? \(\pi = 0.75\) has the highest likelihood. Can we conclude that the coin is not fair? What would make you more convinced?

The likelihood function. \(L(\pi \mid \text{data}) = P(\text{data} \mid \pi)\). The numbers are calculated the same way. What is different is blank to fill in.

Comparison of probability and likelihood
Feature Probability Likelihood
Notation \(P(\text{data} \mid \pi)\) \(L(\pi \mid \text{data})\)
What is fixed? blank to fill in blank to fill in
What varies? blank to fill in blank to fill in
Adds up to 1? blank to fill in blank to fill in
Question it answers How probable is this data if \(\pi\) were this value? How well does this value of \(\pi\) explain the data we saw?

More flips, more focused likelihood. What if we flipped the coin 40 times and got 30 heads? Same proportion, \(\frac{30}{40} = 0.75\), more data.

Two likelihood curves of pi, each rescaled so its peak is 1, both peaking at 0.75. The curve for 4 flips (3 heads), green and solid, is wide and is still fairly high at pi = 0.5. The curve for 40 flips (30 heads), pink and dashed, is narrow, spanning roughly 0.6 to 0.88, and is almost zero at pi = 0.5, marked with a dashed line.

What happens to the likelihood with more data?

Your turn. Your friend has a different coin and suspects it lands tails more often. You flip it 5 times: T, H, T, T, T.

  1. Write \(P(\text{T, H, T, T, T} \mid \pi)\) in terms of \(\pi\), the probability of heads.
  1. Calculate the likelihood for \(\pi = 0.5\) (fair) and for \(\pi = 0.25\). Hint: \(0.5^5 \approx 0.031\) and \(0.75^4 \approx 0.32\).
  1. How many times more likely is \(\pi = 0.25\) than a fair coin? blank to fill in
  2. Add the two likelihoods. Should the sum be 1? blank to fill in
  3. Which value of \(\pi\) has the highest likelihood? blank to fill in

What is wrong here? A friend says: “We flipped H, H, T, H. The likelihood of \(\pi = 0.5\) is 0.0625, so there is a 6.25% chance that the coin is fair.”

Bayesian inference

Remember from last week:

  • Frequentist: the probability of an event is blank to fill in, over many, many repetitions.
  • Bayesian: the probability of an event is blank to fill in. When we get new data, we blank to fill in the probability.

On Monday we did frequentist inference: \(\pi\) was a fixed number, and all of our probabilities were about sample statistics, like \(P(\text{data this extreme} \mid \pi = 0.5)\). Today: Bayesian inference, where the parameter is a blank to fill in.

What frequentist and Bayesian inference find
Inference What we find Read it as
Frequentist (Monday) blank to fill in How surprising is the data, if the parameter were this value?
Bayesian (today) blank to fill in How plausible is each value of the parameter, now that we have seen the data?

The Bayesian recipe.

  1. blank to fill in: what you believe about the parameter before seeing the data, as a probability distribution. \(\quad P(\mu)\)
  2. blank to fill in: collect data. How well does each value of the parameter explain the data? \(\quad L(\mu \mid \text{data})\)
  3. blank to fill in: the updated belief after seeing the data. \(\quad P(\mu \mid \text{data})\)

\[\text{posterior} \ \propto \ \text{prior} \times \text{likelihood}\]

\(\propto\) is read “is proportional to.” Values of \(\mu\) that were plausible before and explain the data well become the most plausible after.

How much do college students sleep?

Researchers asked 253 college students to keep a sleep diary for two weeks. For each student, we have their average hours of sleep on weekdays (WeekdaySleep).

Data: SleepStudy in the Lock5Data package. Onyper, S. V., Thacher, P. V., Gilbert, J. W., & Gradess, S. G. (2012). Class start times, sleep, and academic performance in college: A path analysis. Chronobiology International, 29(3), 318–335.

Let \(\mu\) be the average weekday hours of sleep of all college students like the ones in the study. Sleep experts recommend that adults sleep at least 7 hours a night (Watson et al., 2015).

Do you think \(\mu\) is less than 7 hours? Why?

Step 1: prior. Earlier today, Alex described their belief about \(\mu\) as \(\mu \sim N(7, 0.5^2)\). This is Alex’s blank to fill in distribution.

According to Alex’s prior, what is the probability that \(\mu\) is less than 7 hours? blank to fill in

Step 2: data. Suppose the researchers had entered only the first 10 sleep diaries so far.

Dot plot of the weekday sleep of the first 10 students, in hours, 3, 6.09, 6.6, 6.77, 6.8, 7.7, 7.73, 8.25, 9.02, 9.05. Most are between 6 and 9.1 hours; one student slept 3 hours. A dashed vertical line marks the sample mean, 7.1 hours.

Sample mean: \(\bar{x} = 7.10\) hours. Standard error: 0.56 (given to you, like on Monday).

Likelihood of \(\mu\). Just like the coin: the data are blank to fill in, \(\mu\) blank to fill in. How well does each value of \(\mu\) explain \(\bar{x} = 7.10\)? Each curve below is the sampling distribution of \(\bar{x}\) if \(\mu\) were 6, 7, or 8.

Three normal curves for the sample mean, x-bar, each with standard error 0.56, if mu were 6, 7, or 8. A dashed vertical line marks the observed sample mean, 7.10, in all three. The height of the curve at the dashed line is marked with a pink point. The point is highest when mu is 7, a bit lower when mu is 8, and lowest when mu is 6.

The blank to fill in at the observed \(\bar{x}\) is the likelihood of that value of \(\mu\). Which of the three values of \(\mu\) explains the data best? blank to fill in Worst? blank to fill in

Doing this for every possible value of \(\mu\) gives the likelihood curve. The likelihood of \(\mu\) is shaped like a normal curve, centered at blank to fill in, with spread equal to blank to fill in.

Step 3: posterior.

Three curves for mu. Alex's prior is a green dashed curve centered at 7 with standard deviation 0.5. The likelihood is a gray dotted curve centered at 7.10 with spread 0.56. The posterior is a pink solid curve centered at 7.05 with standard deviation 0.37; it is between the other two and taller and narrower than both.

Alex’s posterior: \(\mu \mid \text{data} \sim N(7.05, 0.37^2)\)

The posterior is a compromise. The posterior mean is a blank to fill in of the prior mean and the sample mean:

\[0.55 \times \underbrace{7}_{\text{prior mean}} + 0.45 \times \underbrace{7.10}_{\text{sample mean}} \approx 7.05\]

  • The weights depend on blank to fill in. Alex’s prior (standard deviation 0.5) and the 10 diaries (standard error 0.56) are about equally certain, so they get about equal weight.
  • The posterior is blank to fill in than both the prior and the likelihood: combining two sources of information makes us more certain.

You will not need to calculate the posterior in this course. We will give you its mean and standard deviation.

A different prior. Sam’s prior was \(\mu \sim N(6.5, 1^2)\): “around 6.5 hours, not very sure.” Sam sees the same 10 sleep diaries. Compared to Alex’s posterior, will Sam’s posterior be centered closer to Sam’s prior mean or closer to the sample mean? Why?

Three curves for mu. Sam's prior is a green dashed curve centered at 6.5, short and wide, with standard deviation 1. The likelihood is a gray dotted curve centered at 7.10. The posterior is a pink solid curve centered at 6.96 with standard deviation 0.49, much closer to the likelihood than to Sam's prior.

Sam’s posterior: \(\mu \mid \text{data} \sim N(6.96, 0.49^2)\).

Alex and Sam saw the same data but ended up with different posteriors. Is this a problem? What could make their posteriors closer to each other?

All 253 sleep diaries. Now the researchers have entered all 253 diaries: \(\bar{x} = 7.87\) hours, standard error 0.073.

Alex's prior (green dashed, centered at 7) and Sam's prior (dark green dotted, centered at 6.5) are short and wide. Alex's posterior (pink solid) and Sam's posterior (dark pink long dashes) after all 253 diaries are two tall, narrow curves centered at 7.85 and 7.86 hours; they lie almost on top of each other.

Alex’s and Sam’s priors and posteriors for mu after all 253 sleep diaries
Person Prior Posterior after 253 diaries
Alex \(N(7, 0.5^2)\) \(N(7.85, 0.073^2)\)
Sam \(N(6.5, 1^2)\) \(N(7.86, 0.073^2)\)

What happened to the two posteriors? Why?

Using the posterior

Probability of a parameter. In Bayesian inference, \(\mu\) is a random variable, so we can find probabilities about \(\mu\), given the data, as areas under the posterior. We will use Alex’s posterior after all 253 diaries, \(\mu \mid \text{data} \sim N(7.85, 0.073^2)\).

What is the probability that \(\mu\) is between 7.75 and 7.95 hours? The plot below shows the area. Write the probability in notation and give its value.

Normal curve for mu with mean 7.85 and standard deviation 0.073. The area between 7.75 and 7.95 is shaded pink and covers most of the curve; the label reads Area = 0.8293.

This is a probability about the blank to fill in, the kind of statement we could not make on Monday.

Bayesian hypothesis testing

Do college students like these get enough sleep on weekdays, on average?

  • Null hypothesis \(H_0\): blank to fill in. They get enough sleep.
  • Alternative hypothesis \(H_A\): blank to fill in. They do not get enough sleep.

On Monday, we blank to fill in and asked how surprising the data would be: \(P(\text{data this extreme} \mid H_0)\), the p-value.

In Bayesian hypothesis testing, we do not assume either hypothesis is true. \(\mu\) is a random variable, so blank to fill in:

\[P(H_0 \mid \text{data}) = P(\mu \ge 7 \mid \text{data}) \qquad P(H_A \mid \text{data}) = P(\mu < 7 \mid \text{data})\]

The steps.

  1. State \(H_0\) and \(H_A\).
  2. Find the blank to fill in probability of each hypothesis: area under the prior.
  3. Find the blank to fill in probability of each hypothesis: area under the posterior.
  4. Compare: which hypothesis is more probable, and by how much? \(\frac{P(H_0 \mid \text{data})}{P(H_A \mid \text{data})}\) is how many times as probable \(H_0\) is as \(H_A\).

The two probabilities always add up to blank to fill in: \(\mu\) is either less than 7 or not.

Before the data: prior probabilities. Fill in the “?” in the plot with Alex’s prior probabilities \(P(H_0)\) and \(P(H_A)\). How do you know, without any calculation?

Alex's prior for mu, a normal curve with mean 7 and standard deviation 0.5. A dashed line at 7 splits it in half. The right half, mu at least 7, is shaded green and labeled H0. The left half, mu less than 7, is shaded pink and labeled HA. In the answer key, both are labeled 0.50; in the student version, the probabilities are question marks.

After 10 diaries: posterior probabilities. Alex’s posterior is \(N(7.05, 0.37^2)\). The plot gives \(P(H_A \mid \text{data})\). Use it to find \(P(H_0 \mid \text{data})\).

Alex's posterior after 10 diaries, a normal curve with mean 7.05 and standard deviation 0.37. A dashed line at 7 is just left of the center. The right part, mu at least 7, is shaded green and labeled H0. The left part, mu less than 7, is shaded pink and labeled HA, 0.45. In the answer key, H0 is labeled 0.55; in the student version, it is a question mark.

\(P(H_0 \mid \text{data}) \approx\) blank to fill in \(\qquad P(H_A \mid \text{data}) \approx\) blank to fill in

\(H_0\) is blank to fill in times as probable as \(H_A\). Would you conclude that college students like these get enough sleep, based on these 10 diaries?

After all 253 diaries. Alex’s posterior is \(N(7.85, 0.073^2)\). Use the plot to find \(P(H_0 \mid \text{data})\).

Alex's posterior after all 253 diaries, a tall, narrow normal curve with mean 7.85 and standard deviation 0.073. The dashed line at 7 is far to its left, so the whole curve is shaded green. HA, mu less than 7, is labeled with probability less than 0.0001. In the answer key, H0, mu at least 7, is labeled with probability greater than 0.9999; in the student version, it is a question mark.

Complete the table.

Alex’s prior and posterior probabilities of the null and alternative hypotheses
\(P(H_0)\): \(\mu \ge 7\) \(P(H_A)\): \(\mu < 7\) Which is more probable?
Prior (before any data) blank to fill in blank to fill in blank to fill in
Posterior after 10 diaries blank to fill in blank to fill in blank to fill in
Posterior after 253 diaries blank to fill in blank to fill in blank to fill in

Conclusion after all 253 diaries:

How is this different from Monday?

  • On Monday, we could only blank to fill in \(H_0\). Failing to reject \(H_0\) is not evidence for \(H_0\).
  • Today, both hypotheses get a probability, so the data can give evidence blank to fill in, including \(H_0\).
  • The p-value, \(P(\text{data this extreme} \mid H_0)\), is a probability about the blank to fill in. \(P(H_0 \mid \text{data})\) is a probability about the blank to fill in.

Bayesian statisticians also compare hypotheses with the Bayes factor: how much the data changed the odds of one hypothesis against the other. We will not calculate it in this course.

Credible intervals

A 95% credible interval is the blank to fill in: 2.5% of the area below it and 2.5% above it.

The plot shows Alex’s posterior after all 253 diaries with the middle 95% shaded. What is Alex’s 95% credible interval?

Normal curve for mu with mean 7.85 and standard deviation 0.073. The middle 95 percent of the area is shaded pink, between dashed lines labeled 7.71 and 7.99.

Interpretation:

In Bayesian inference, \(\mu\) is a random variable, so “95% probability that \(\mu\) is in the interval” is exactly what the interval means:

\[P(7.71 < \mu < 7.99 \mid \text{data}) = 0.95\]

Your turn. After the first 10 sleep diaries, Sam’s posterior was \(\mu \mid \text{data} \sim N(6.96, 0.49^2)\).

Two plots of Sam's posterior, a normal curve with mean 6.96 and standard deviation 0.49. On the left, the area to the left of 7 is shaded pink, and the label reads Area = 0.5325. On the right, the middle 95 percent is shaded pink, between dashed lines labeled 6.00 and 7.92.

  1. \(H_0: \mu \ge 7\) and \(H_A: \mu < 7\). Use the left plot to find Sam’s posterior probabilities \(P(H_0 \mid \text{data})\) and \(P(H_A \mid \text{data})\).
  1. Sam’s prior probabilities were \(P(H_0) = 0.31\) and \(P(H_A) = 0.69\). Which hypothesis was more probable before the data, and how many times as probable? What about after the data? What do you conclude?
  1. Use the right plot to write Sam’s 95% credible interval. blank to fill in
  2. Interpret the interval in (c) in context.
  1. Is Sam’s interval wider or narrower than an interval from all 253 diaries? Why?

Frequentist vs. Bayesian

For all 253 diaries:

Frequentist and Bayesian 95% intervals for mu from all 253 sleep diaries
Interval From To
95% confidence interval (frequentist) 7.72 8.01
95% credible interval (Bayesian, Alex’s prior) 7.71 7.99

Almost the same numbers! With a lot of data, the prior matters very little. But they mean different things. Interpret each one.

Complete the table.

Comparison of frequentist and Bayesian inference
Feature Frequentist (Monday) Bayesian (today)
The parameter is blank to fill in blank to fill in
Probability means long-run proportion over many samples how plausible something is, given what we know
Uses a prior? blank to fill in blank to fill in
What we find blank to fill in blank to fill in
Hypothesis testing blank to fill in blank to fill in
Interval blank to fill in blank to fill in
“95% probability that the parameter is in the interval” blank to fill in blank to fill in

Correct or not? Write “correct” or “not correct” and explain.

  1. Alex’s posterior probability that \(\mu\) is less than 7 hours, after all 253 diaries, is less than 0.0001. blank to fill in
  2. In frequentist inference, we can find the probability that \(\mu\) is less than 7 hours. blank to fill in
  3. There is a 95% probability that \(\mu\) is between 7.71 and 7.99, according to Alex’s posterior. blank to fill in
  4. The posterior always ends up exactly at the prior mean. blank to fill in
  5. With more data, the prior has less influence on the posterior. blank to fill in
  6. The likelihood \(L(\mu \mid \text{data})\) is a probability distribution of \(\mu\). blank to fill in
  7. In Bayesian hypothesis testing, we start by assuming that the null hypothesis is true. blank to fill in
  8. After 10 diaries, Alex’s \(P(H_0 \mid \text{data}) \approx 0.55\) and \(P(H_A \mid \text{data}) \approx 0.45\), so the data give strong evidence for \(H_0\). blank to fill in

Scope of the conclusions. Can we use these results to describe how much Harvard students sleep? The students wrote their own sleep diaries. How might this affect the data?

Summary

  • Likelihood \(L(\pi \mid \text{data})\): blank to fill in, does not need to add up to 1. Probability \(P(\text{data} \mid \pi)\): blank to fill in, adds up to 1.
  • Bayesian inference: blank to fill in \(\times\) blank to fill in \(\propto\) blank to fill in. The posterior mean is a compromise between the prior mean and the sample mean, weighted by how certain each one is.
  • Different priors can give different posteriors. With more data, blank to fill in.
  • Bayesian hypothesis testing: no hypothesis is assumed true. Find blank to fill in and conclude in favor of the much more probable one.
  • A 95% credible interval has a blank to fill in of containing \(\mu\).
  • Frequentist: \(P(\text{data} \mid \text{parameter})\). Bayesian: \(P(\text{parameter} \mid \text{data})\). Remember: \(P(A \mid B) \neq P(B \mid A)\)!