Overview of Instruments

Questionnaire Selection

Initially several questionnaires were identified that were valid options for use in the MIXER evaluation.  Many of the questionnaires measured similar items, therefore the set was reduced to the three questionnaires that:

a) Matched the learning outcomes the best

b) Were most suitable for use with children

Where possible the questionnaires were further reduced to individual factors i.e. the CQS consists of 4 factors; Behavioural, Cognitive, Metacognitive and Motivational. Only the Behavioural factor was relevant to the MIXER evaluation so the remaining 3 factors were discarded.

The following sections detail the selection rationale and developmental studies that were conducted to refine the three final questionnaires:

MESSY Scale (Matson et al., 2010)

Social skills are a fundamental factor for the formation of relationships, for the quality of social interactions (Hay, 1994; Parker & Asher, 1987). The MESSY Scale will be used to assess the following learning goal:

“Does MIXER support children in being fully present in attending to others verbal and non-verbal messages?”

Messy stands for the Matson Evaluation of Social Skills with Youngsters.  The MESSY Questionnaire is a social skills measure for children, aged 4 to 18. The original Matson Evaluation of Social Skills with Youngsters (MESSY) consists of 62 items divided into a four-factor model:

  • Factor 1: Aggressiveness/Antisocial Behavior
  • Factor 2: Social Skills/Assertiveness
  • Factor 3: Conceit/Haughtiness
  • Factor 4: Loneliness/ Social Anxiety

In the MIXER evaluation only Factor 2: Social Skills/Assertiveness is used. Factor 2 will be used to assess the child’s self-perception of their own social skills and assertiveness.  Factor 2 of the Messy Scale consists of 25 items, children respond to each item using a 5-point Likert scale ranging from Strongly Agree to Strongly Disagree, items are based on both appropriate and inappropriate social behaviours e.g. I helping a friend who is hurt, I feel good if I help someone.

CQS – Cultural Intelligence Scale (Ang et al., 2007) 

The Cultural Quotient Scale (CIS) will be used to assess the following learning goal:

“Does the use of MIXER support children to understand on a basic level the differences and similarities between another group and their own?”

The Cultural Intelligence Scale was developed to test Early and Ang’s (2003) conceptualization of cultural intelligence.  The CQS provides a valid and reliable measure of a person’s ability to function effectively in culturally diverse situations.  The CQS measures four factors:

  • Metacognitive CQ,
  • Cognitive CQ,
  • Motivational CQ and
  • Behavioral CQ.

For the purpose of evaluating MIXER only the Behavioral CQ factor of the scale will be used.  This decision was made following a review of the content of the CQS, the remaining 3 factors – Cognitive, Meta-cognitive and motivational were unrelated to assessing the learning outcomes of MIXER as they assess understanding of e.g. currency and laws of other countries etc.

The Behavioural CQ measures capability to adapt verbal and nonverbal behavior. It includes having a flexible repertoire of behavioural responses that are appropriate in a variety of situations and having the capability to modify both verbal and nonverbal behaviour based on those involved in a specific interaction or in a particular setting.

Bryant’s Empathy Index (Bryant, 1982)

Byrant’s Empathy Index for children will be used to assess the learning outcome:

“Does MIXER enable children to recognise their emotions (for example fear and anxiety) when dealing with

the novel / unknown behaviours of another group?”

Bryant’s Empathy Index will be used to measure children’s empathic ability in regards to their emotional ability to understand happiness and sadness.

The empathy index for children and adolescents, developed by Bryant, (1982) and initially discussed in the Milestone 3 report produced by Jacobs University (MS3), looks at empathic traits in people. It examines the relationship of aggression and attitudes towards the general population.

This will allow us to determine whether children who have higher levels of empathy engage more with the MIXER application and/or are more accepting of different backgrounds/cultures. Bryant’s Empathy Index is a three-factor questionnaire. Factor one deals with understanding feelings, Factor two covers feelings of sadness and Factor 3 asks about tearful reactions.  Responses to sections F2 and F3 bear no relevance in the evaluation of MIXER and will therefore be excluded from the evaluation.

For the MIXER evaluation we will use factor one only which focuses on the understanding of feelings and emotions.

Instrument Rationale

Cultural Intelligence Scale

The behavioural subscale of the CQS will be used as a pre and post measure of a child’s capability to adapt verbal and nonverbal behaviour in different situations/cultures. Answering the following question: “Do children who have a more flexible repertoire of behavioural responses in culturally diverse settings recognise more emotions/behaviours in the MIXER application”. This will address aspects of the behavioural and emotional learning outcomes.

Bryants Empathy Index

Factor One from the Bryant Empathy Index will be used as a measurement of children’s empathic behaviour and styles. Answering the question: “Are children with higher empathy levels more able to recognise and accept emotions in novel situations? This will address the Emotional goal of the learning outcomes: “Be able to recognise your emotions when dealing with strange behaviours of another group”.
MESSY – Matson Social Skills Factor two and four of the MESSY questionnaires have been selected as a baseline assessment of a child’s capability to adapt to verbal and nonverbal behaviour in different situations/cultures to assess the behavioural goal from the learning outcomes; “Be fully present in attending to others verbal and nonverbal messages”.

 Table 1: Summary of Instruments

Questionnaire Refinement 

The development of the activity/work books took place over five studies, in which specific factors of the questionnaires were selected and adjusted to match the ability of the children that may take part in the final evaluation. An expert discussion group was held after study three, in which the specific factors were chosen in relation to the learning goals.

Pilot studies of the three questionnaires were conducted in the form of comprehension studies in Sunderland, UK.   This study was performed to test children’s understanding of the questions that were to be used in the evaluation as some of the words and phrases used were deemed to be a little too advanced for the target age group of 9-11 year olds. 60 children were given a copy of the CQS and asked to underline any words or phrases they did not understand.  The children then used iPads to look up the definitions of these words and were asked to provide suggestions for replacements. These words/phrases were replaced in the CQS.

The original version of Bryants Empathy Index contains questions which refer to boys and girls, as the evaluation of MIXER is non-gender specific we have replaced the words ‘boy’ and ‘girl’ with kid/s (kid is also used throughout the original questionnaire).  By making the questionnaire non-gender specific we were able to further reduce the number of questions as some questions were duplicated i.e. ‘Girls who cry because they’re happy are silly.’ and ‘Boys who cry because they’re happy silly.’ will be replaced in the MIXER evaluation with ‘Kids who cry because they’re happy are silly.’