TS - Test of Significance Module Overview

Test of Significance Overview

Introduction

question mark door introducton.jpgMany "claims" are made by businesses and individuals that may or may not be justified.   Just because a company says that their product performs better than a competitor's product does not necessarily mean that it is true.   Just because your friend says they have a particular baseball batting average does not make it true.   Testing the truth of statements like these is called performing a Hypothesis Test or a Test of Significance. There are specific procedures using data and logic to help decide if the "claims" are true or not. Even after careful selection of a sample, careful survey or experimental design, and careful analysis by a trained statistician the result must still be questioned.   We must determine if we have "significant" results, or if the results could have occurred simply by chance.   Probability is the tool that will assist in answering that question.   We will test the significance of our own data or data collected by others.

Essential Questions

  • Are we ever completely sure about anything?
  • How can we determine if the results produced by the data are significant and not simply the result of chance occurrence?
  • What is a null hypothesis?  
  • What is an alternative hypothesis?
  •  In statistics, what is meant by the P-value?
  • What does a test statistic estimate?
  • What is meant by a significance level?
  • When can we conclude a significant difference between two data sets?
  • What can go wrong with our conclusion despite the fancy calculations?

Key Terms

The following key terms will help you understand the content in this module.

Tests of Significance - formal inference method in which a null hypothesis is asserted and tested by asking how unlikely the observed outcome would be if the null hypothesis were true

Null hypothesis (H0)- the claim being assessed in a hypothesis test - typically a statement of "no change from the stated value" or "no difference" or "no relationship"

Alternate hypothesis (Ha)- suggests what should be concluded if the null hypothesis is found to be unlikely

P-value- probability of observing a value for a test statistic at least as far from the hypothesized value as the sample statistic value actually observed IF then null hypothesis is true. …small p-value indicates an improbable observation

Statistically significant- a P-value as small or smaller than the alpha level determines statistical significance

Test statistic- value that will be used to determine the P-value and used to determine rejection and acceptance regions

Significance level (alpha)- decisive value of P that determines the required level of evidence against the null hypothesis

Hawthorne effect- can occur when a participant's knowledge that they are part of a study influences the outcome

Type I error- the null hypothesis is TRUE but we mistakenly reject it - also known as a false-negative result

Type II error- the null hypothesis if false, but we fail to reject it - also known as a false-positive result

Power of a test- the ability to detect a false hypothesis when it is really false standard error (SE)| name given to the estimated standard deviation of a sampling distribution

Standard Error (SE)- name given to the estimated standard distribution of a sampling distribution

T distribution- used when the population standard deviation is not known or when the sample is too small for a Normal model. Margin of error will be slightly wider producing a wider confidence interval and a slightly larger P value than the Normal distribution

Degrees of Freedom- df = (n - 1) similar to finding the standard deviation when dividing squared deviations by (n - 1). Since the sample variance would tend to underestimate the actual population variance, a statistical correction (i.e., n-1) must be used in an attempt to correct for this bias

Robust- a confidence interval or significance test is robust if the confidence level or P-value does not change much when the assumptions are violated. The t-distribution is said to be robust

Pooled- technique used for two-sample t procedures where samples are independent and have equal variance.

Math_OverviewBottomBanner.png IMAGES CREATED BY GAVS