12|THE \(t\)-TEST part 2

Overview

Research designs

  • Single-sample \(t\)-test
    • Compare sample against expected population mean based on logic/theory/scale design
  • E.g. ‘common sense’ theory
    • Population average amount of sleep

\(\mu = 8\) hours

Research designs

  • E.g. measure of happiness

What is your current level of happiness?






\(\mu = 3\)

Assumptions

  • Assumptions for single-sample \(t\)-tests
    1. Independence
      • Independent random sampling
      • Values in the sample are independent observations
    2. Normality
      • The population sampled is normally distributed
      • With large samples \((n > 30)\), this assumption can be violated without affecting the validity of the hypothesis test
    3. Homogeneity of variance
      • Variability in the original and treated populations is the same

Effect size

  • Hypothesis test Step 4: Make decision (reject null?)
  • Step 4b: Evaluate effect size
    • Cohen’s \(d\) for single-sample \(t\)-test
    • Original equation included population SD, \(\sigma\)
    • Estimated Cohen’s \(d\) uses sample SD, \(s\)

\(\text{Estimated } d = \dfrac{\text{mean difference}}{\text{sample standard deviation}} = \dfrac{M - \mu}{s}\)

\(\text{For class RT data, } d = \dfrac{322.59 - 284}{45.31} = 0.85\)

Effect size: \(r^2\)

  • Proportion of all variability in the data attributable to treatment effect
  • Simplifying assumption: Treatment adds or subtracts a constant to each score
  • E.g. 1 point on a scale of 1 to 5
  • \(r^2\) separates that variability due to treatment from natural variability between scores

\(r^2 = \dfrac{SS_{treatment}}{SS_{total}}\)

\(r^2\)

  1. Calculate sum of squared deviations from sample \(M\)
    • Variability excluding treatment effect
    • \(SS_{without \ treatment}\)
  2. Calculate \(SS\) from \(H_0\) \(\mu\)
    • This is total variability
    • \(SS_{total}\)
  3. Substract \(SS_{without \ treatment}\) from \(SS_{total}\) to find \(SS_{treatment}\)
    • Variability attributable to treatment effect

\[\begin{align} r^2 = \dfrac{SS_{treatment}}{SS_{total}} &= \dfrac{SS_{total} - SS_{without \ treatment}}{SS_{total}} \\ &= \dfrac{10-6}{10} = 0.4 \end{align}\]

\(r^2\)

  • If we already calculated \(t\)…

\(r^2 = \dfrac{t^2}{t^2 + df}\)

  • Works for any kind of \(t\)-test
    • Single / related / independent-samples
  • Interpreting \(r^2\)
    • \(r^2 = 0.01\): small effect
    • \(r^2 = 0.09\): medium effect
    • \(r^2 = 0.25\): large effect

Reporting results

Given the average reaction time for the population of \(\mu = 284 ms\), according to humanbenchmark.com, a two-tailed single-sample \(t\)-test suggests that BC1101 students have significantly different reaction times \((M = 322.59\); \(SD = 45.31)\) than the general population; \(t(22) =\) \(4.08\), \(p < .05\), \(d = 0.85\).

Confidence intervals

  • Complementary to significance & effect size
  • Quantifies precision of sample estimate
  • Comprised of:
    • The point estimate
      • Our best guess of the population parameter
    • Margin of error
      • A range either side of point estimate
      • Indicates the amount of uncertainty surrounding estimate of population mean
      • Based on desired ‘confidence’, i.e. range of the distribution
      • E.g. 95%, 99%, 80%, etc…

Calculating CI boundaries

  • So far, we have been specifying \(\mu\), calculating \(M\) and \(s_M\), solving for \(t\)
  • For CI, rearrange to solve for \(\mu\)
    • Calculate \(M\) and \(s_M\), specify \(t\) (based on desired width of CI —99%, 95%, 90%, 80% etc), solve for \(\mu\)

\(t = \dfrac{M - \mu}{s_M}\)

\(\mu = M \pm t * s_M\)

Confidence interval interpretation

  • What does a confidence interval tell us?
    • Indicates precision of parameter estimate
    • Quantifies variability around a single point estimate
    • NOT “we are 95% sure the true population mean is within this range”
    • NOT “sample means from this population will fall within this range 95% of the time”

“The parameter is an unknown constant and no probability statement concerning its value may be made.”1

Factors that affect CI width

30
95

CI & NHST

  • \(p\) value and CI always agree about statistical significance if CI is \(1 – alpha\)
    • E.g. \(\alpha = .05\) and 95% confidence interval
  • If the \(p < \alpha\), the confidence interval will not contain the null hypothesis value
  • If the confidence interval does not contain the null hypothesis value, the results are statistically significant
  • Both significance level and confidence level define a distance from a mean to a limit
    • The distances in both cases are exactly the same

CI & NHST demonstration

15
0.7

Show:

p

Learning checks

  1. What value of \(t\) would you expect to see if the null hypothesis is true?
  2. Which combination of factors is most likely to produce a significant value for the \(t\) statistic?
    • Small mean difference and large sample variability
    • Small mean difference and small sample variability
    • Large mean difference and large sample variability
    • Large mean difference and small sample variability