Reliability (AQA A Level Psychology)

Reliability

• Reliability is achieved if a study has been set up so that the IV can be seen to affect the DV:

  • If the procedure is replicated, the study should show similar results

  • Replicating a study and finding similar results shows that the measure is consistent and not liable to fluctuate to any significant degree

• If a study is completed using a standardised procedure and obtains the same results, it can be said that it is reliable 

• Lab experiments are the most reliable method used in psychology as they:

  • take place in a neutral space under controlled conditions

  • follow a standardised procedure

  • use random allocation of participants to conditions

  • tend to use a control group as a comparison to the experimental group

  • generate quantitative data which is easy to compare and analyse

• Field experiments generate quantitative data and implement an IV but they are subject to an array of extraneous variables over which the research has no control, making them less reliable than lab experiments

• Natural experiments generate quantitative data but they use a naturally occurring IV over which the research has no control, making them less reliable than lab experiments

• There is both internal reliability and external reliability 

  • Internal reliability: The extent something is consistent with itself

  • External reliability: The extent a test measure is consistent over time

Assessing reliability

• The reliability of a questionnaire can be assessed using two methods:

  • The test-retest method measures external reliability: 

    • The same participants are given the same questionnaire at separate time intervals (e.g. with a 6-month gap between testing sessions) 

    • If the same result is found per participant then external reliability is established

  • The split-half method measures internal reliability:

    • The researcher splits the test in half and analyses the responses given to the first half of the questionnaire compared to the second half of the questionnaire

    • If similar responses are given in both halves then internal reliability is established

Inter-observer reliability

• For an observational study, the researchers will have previously agreed on which specific behaviours should be recorded so that all observers are on in agreement before the observation 

• Researchers can test the reliability of their observations by comparing them with another researcher's recording of their behaviours

  • The level of consistency between the two records is then compared

• Inter-observer reliability is the level of consistency between two or more trained observers when they conduct the same observation, as follows:

  • All observers must agree on the behaviour categories and how they are going to record them before the observation begins

  • The observation is conducted separately by each observer to avoid conformity (i.e. one observer may be influenced by one or more other observers)

  • After the observational period:

    • The observers compare the two independent data sets (often designed as a tally chart)

    • They then test the correlation between the two sets

    • If there is a strong positive correlation between the sets then this shows that there is good inter-observer reliability and that the behaviour categories are reliable

• Establishing good inter-observer reliability means that there is less chance that researcher bias has interfered with the observation

An example of a tally chart used to record categories of behaviour in an observation:

Improving reliability

• The reliability of lab experiments can be improved by

  • ensuring that all aspects of the procedure are controlled and standardised to test the reliability of the method

  • ensuring that the IV and DV are operationalised

  • testing participants using conditions which differ slightly from the original procedure to test the reliability of a specific finding

• The reliability of observations can be improved by ensuring that the behavioural categories

  • are operationalised

  • measure only observable behaviour

  • are distinct, with no overlapping

• The reliability of questionnaires can be improved by

  • running the test-retest method and excluding any questions which do not show consistency

  • running the split-half method and excluding any whole questionnaires that do not show consistency

• The reliability of interviews can be improved by

  • using the same interviewer for each separate interview

  • using trained interviewers (which is particularly important for the cognitive interview used with eyewitnesses)

  • ensuring that leading questions or ambiguous, double-barreled questions are omitted from the questioning