Lecture 2: Research Methods

Assessment of Observation (Measurement)

Observed Score = True Score + error
Error component may be either:
1. Random error = variation due to uncontrolled or unknown factors.
2. Systemaic error = variation due to systematic but irrelevant elements of the design

   Concern of Scientific Research is management or error component.

Number of criteria by which evaluate success:

1. Reliability

   Does the measure consistently reflect changes in what it purports to measure?

   Consistency or stability of data across time and circumstances
   Balance between consistency and sensitivity of measure

2. Validity

   Does the measure acutally represent what it purports to measure?
   Accuracy of the data (for what?)

   Number of types of validity:

   1. internal validity = effects of an experiment are due solely to the experimental conditions.
      Extent to which causal conclusions can be drawn
      Dependent upon experimental control
      Trade-off between high internal validity and generalizability

   2. External validity = can the results of an experiment be applied to other individuals or situations.
      Extent to which results can be heneralized to broader populations or settings.
      Dependant upon sampling subjects and occasions
      Trade-off between high generalizability and internal validity

   3. Construct validity = whether or not an abstract, hypothetical concept or idea exists as postualted

      Based on:

      Convergence = different measures that purport to measure the same construct should be highly correlated with one another.
      Divergence = tests measuring one construct should not be highly correlated with tests purporting to measure other constructs

   4. Statistical conclusion validity = the extent to which a study has used appropriate design and statistical methods to enable it to detect the effects that are present. The accuracy of conclusions about covariation made on the basis of statistical evidence. Appropriate statistical power, methodological design, and statistical analyses. Can have reliable , but invalid measure. IF measure is valid, then necessarily reliable.

3. Utility
   Usefulness of the method/measure
   Gauged in terms of efficiency and generality:

   Efficient methods provide precise, reliable data with relatively low costs in time, materials, equipment, and personnel
   Generality refers to the extent to which a method can be applied successfully to a wide-range of phenomena (generalizability)

   Threats to Validity Related to Experimental Design
   1. Threats to Internal Validity (Cook & Campbell, 1979)
      1. History = the specific events occurring to an individual
      2. Testing = repeated exposure to testing instrument
      3. Instrumentation = changes in scoring procedures or criteria over time
      4. Regression = reversion toward the mean or less extreme scores
      5. Mortality = differential attrition across groups
      6. Maturation = developmental processes
      7. Selection = differential composition of subjects among samples
      8. Selection by maturation interaction
      9. Ambiguity about causal direction
      10. Diffusion of treatments = information spread between groups
      11. Compensatory Equalization of Treatments = lack of treatment integrity
      12. Compensatory Rivalry = "John Henry" effect on nonparticipants

   2. Threats to External Validity (LeCompte & Goetz, 1982)
      1. Selection = results sample-specific
      2. Setting = results context-specific
      3. History = unique experiences limit generalizability
      4. Construct effects = constructs are sample-specific

   Related to Experimenter
   1. Noninteractional Artifacts
      1. Observer Bias = over/under estimate of phenomenon (schema)
      2. Interpreter Bias = error in interpretation of data
      3. Intentional Bias = fabrication or fraudulent interpretation of data

   2. Interactional Artifacts
      1. Biosocial Effects = errors attributable to biosocial attributes of researcher
      2. Psychosocial Effects = errors attributable to psychosocial attributes of researcher
      3. Situational Effects = errors attributable to the research setting and participants
      4. Modeling Effects = errors attributable to example set by researcher
      5. Experimenter Expectancy Bias = researchers treatment of participants elicits confirmatory evidence of hypothesis

      Comparison of Applied and Basic Research

      Differences listed in Robson (1993)

      Purpose
      Basic = expand knowledge

      Applied = understanding specific problem

      No real difference, just a matter of degree

      Context

      Basic = Academic, single researcher, less time/cost pressure

      Applied = Real-world setting, multiple researchers, more time/cost pressure

      Again, no real difference, just depends on situation (stereotype)

      Methods
      Basic = Internal validity, cause, single level of analysis, single method, experimental designs, direct observations

      Applied = External validity, effect, multiple levels of analysis, multiple methods, quasi-experimental, indirect observations

      Again, over generalization no real differences.

      Only substantial difference between applied and basic research:

      Basic = experimental control

      Applied = statistical control