Cultural Knowledge Test Adaptation

ASSESSMENT OF THE CONSTRUCT

Cultural intelligence (CQ) is “an individual’s capability to adapt effectively to new cultural contexts” (Early & Ang, 2003, p. 59). To measure CQ the Cultural Research Group at Florida Tech University have used a multidimensional scale with four specific components which they use as the initial indicators of CQ: Cognitive CQ, Meta-Cognitive CQ, Motivational CQ, and Behavioral CQ. The Cultural Intelligence Scale (CQS) was developed by Ang, (Ang et al, In Press) to assess CQ. Van Driel et al (van Driel et al., 2008) attempt to challenge the validity of Ang’s construct of the CQS. In the Cultural Knowledge test van Driel et al focus on Cognitive CQ so it is part of their latent construct just for this test. Van Driel et al take Ang’s assumption that cognitive CQ is any information that can be learned from other cultures inside or outside a classroom. Unlike Ang however, Van Driel et al develop their own indicators for Cultural Knowledge or Cognitive CQ which include: Legal and Economic Systems, Languages, Religion and Values, Marriage Systems, Arts and Crafts, Non-Verbal Display Rules, and Geography. This is a multidimensional scale as persons may know much about one of the indicators through previous study but little to none about other indicators (van Driel et al., 2008). From these indicators they created specific questions which are deemed to convey Cultural Knowledge. As an example the question: “What is the hajj?” would be under the Religion and Values indicator. A full list of questions assigned to indicators in available in Figure 1 of the appendix.

The content validity of this model has serious problems as there are many important indicators of Cultural Knowledge missing. The missing elements impair the ability of the test to accurately measure cultural knowledge. Some indicators which we believe are missing include time perception and eating behavior and traditional foods. Also it is much more useful to combine the indicator “Language” together with the indicator for “non verbal displays” because separating the variables loses important areas like greetings and verbal communication displays that may not exactly fall under language or non-verbal displays. Thus we suggest an indicator for verbal and non verbal communication as this includes both language and non verbal displays but also provides for questions about greetings and colloquial speech. Lastly we also suggest that the indicator “Arts and Crafts” should be replaced with the indicator Arts which would cover cultural areas such as music, theater, and artwork that were previously left out by the original construct. This clearly is still not a completely comprehensive construct, however we find that assessing all areas of possible cultural knowledge would be too taxing for a survey taker to engage in and thus our construct focuses only on the most important and relevant parts of cultural knowledge. A new construct has been proposed correcting for content validity in Figure 2.

After fixing the problems of content validity in our questionnaire it is apparent this questionnaire is extremely adaptable to other cultures. All items on the questionnaire should be etic as they are assessing knowledge that anyone could know. Thus for applying the construct to German culture it was unnecessary to change the construct at all. All indicators are relevant and would be understood by Germans. Therefore for German culture the construct (when corrected for content validity) is again the latent construct of Cognitive CQ with the indicators Legal and Economic Systems, Verbal and Non-verbal Communication Rules, Religion and Values, Marriage Systems, Arts, Geography, Time Perception, and Eating Behavior and Traditional Foods. Thus it is unnecessary to redraw the measurement model. Also as all aspects of this survey are etic there is no change in the items from one culture to another.

As the questions are all knowledge based and factual we assume there to be no reference group effects on the questionnaire as a whole; also we find no order effects as the questions could be written in any order to achieve the same results. The questionnaire does not use a scaled response and as they are factual questions the answer choices are appropriate. There is nothing on the questionnaire which respondents could not know, however all questions are subject to Foddy’s first law. This is expected and acceptable because the aim of the test is to assess what factual information respondents knew about culture; thus respondents need not answer correctly to every question. The questions are generally easy to understand however so the answer given by each respondent correctly evaluates how much the person knows about this indicator of Cognitive CQ. It is difficult to assess whether this questionnaire fully measures Cognitive CQ because obviously not every aspect of every culture can be asked in the Cultural Knowledge test; however with our new construct assessing the flaws in the content validity we hope that it adequately measures cultural knowledge for the purposes of assessing overall cultural intelligence. The questionnaire in general could be used in either individual research to assess the cultural knowledge (a subset of cultural intelligence) of an individual and thus assess his potential to succeed is various cultural situations or could be used at the cultural level to assess the cultural awareness of various cultural groups. As all items in the questionnaire are factual based they are not at risk of having any social desirability factors effecting the answers. The only concern for response bias of this survey is whether individuals would use outside sources to help them answer the questions on the survey in hopes to achieve a higher score and show they are more culturally aware which would affect the overall outcome. Therefore the last question of the survey has been added to help detect invalid data.

REVISION OF ORIGINAL ITEMS AND ADAPTATION FOR USE IN GERMAN

Not all the items in the original questionnaire meet the criteria specified in Brislin’s list (Brislin, 1986). Therefore, it was necessary to rephrase some of them or express the ideas more clearly. Figure 6 gives an overview of the revised original questionnaire, and the translated German version. In question 1, the use of compound words such as “multiethnic” and “multireligion”, the latter not even being a word, made the question unnecessarily complicated so we decided to change the wording to “many ethnic groups and religions”. A problem with question 2 was that “arranged marriages” does not have an exact equivalent in German and moreover, the term leaves the level of coercion open. For example, it is not clear if it refers to marriages in which the bride and the groom have a veto power or not, or if it refers to a child marriage.

Question 22 was problematic in several respects. First of all it was grammatically inconsistent since the question and the different answering options did not create coherent sentences. Moreover, the first option uses the proverb “you reap what you sow” (Fig.6) and according to Brislin’s list proverbs should be avoided since it is unlikely that there are equivalents in other languages. Therefore we changed this option into a normal sentence, however, it is very difficult to describe a concept as complex as Karma accurately in just few words. The next option, “cultivate themselves morally” is not very clear in its meaning so we suggest “to increase one’s level of morality” instead. A difficulty item was also question 24 with regards to the fact that there are too many adjectives describing the same circumstances in the question (“The largest major far eastern Russian city, on the Pacific Ocean”). Therefore we simplified the question and changed it to “The major eastern Russian city, on the Pacific Ocean”. Another question which was phrased in a misleading way is question 29 (The unique Japanese religion is) because it can be understood in two ways: The religion which only exists in Japan which is what the question asks for, or the only religion in Japan. Therefore, we rephrased the question to “The religion which only exists in Japan”. A problem identified with the last question is that there is a double negative since the question says “I did not...” and the answering options are “I did not...” so it takes the participant some time to figure out what he is asked for and what he needs to reply. This double negation might cause confusion; hence it was changed for a clearer formulation. New items were then generated in order to measure the parts of the new construct which were missing in the original scale. We created one question for each new aspect in order to show how we would test them; they are summarized in Figure 5. All the questions are common to the two cultures.

In the next step the old items and those which were modified were translated into German. As all German speakers had seen the questions already a blind back-translation was not possible and in many cases not necessary since items such as “Brazilians speak…” (Fig.6) leave no room for different or misleading translations. In order to still test the validity of the translation to some extent, we randomly asked German students, who were not familiar to the questionnaire, to translate selected German questions (back) to English. Thereby, we were able to have a back-translation by an independent third person with satisfying outcomes. All in all there were no major difficulties with translating the items. However, there were naturally some problems which occurred during the back-translation process as well as problems with other questions. These are described in the following. In question 1 ”multiethnic state” can be translated as either “multiethnischer Staat” which is understandable in German and very close to the original in English or as “Vielvölkerstaat”, a much more common term which is familiar to more people and which was thus used in the final translation. Question 2 which was already considered critical in the original posed an even bigger problem in the translation because the German language has no fixed expression for “arranged marriage”. The only terms available are “Zwangsheirat” and “Kinderheirat”, however, those are equal to “forced marriage” and “child marriage”. Therefore, we had to find a preferably precise circumscription. The final version “Im Voraus arrangierte Ehe” (Fig. 6) is still a relatively awkward phrase though. Finding an appropriate translation for “more likely to” (Fig. 6) in question six is not easy since there is no such structure in German. The translation of question 9 required some careful consideration in so far as there is no equivalent to “urban” in German so a circumscription in which a noun was added had to be used. Question 21caused some confusion since we did not know that “established church” is equivalent to the German word “Staatskirche”. However, when we became aware of this the problem was solved easily. In question 33, a problem of a different sort occurred since “hajj” is an accepted spelling in German, however, the spelling “Haddsch” is much more common so both words appear in the translation. Moreover, there is no common translation for “the three-barred cross of the Russian-Orthodox church” (Fig.6) other than what would simply mean “The cross of the Russian-Orthodox church” in English. Question 34 posed a problem in so far as that the answering option “glassware, porcelain and chinaware” has no equivalents other than “Glasswaren und Porzellan” because there is no finer distinction. For the other questions and answering options there was no problem with finding appropriate translations.

VALIDATING THE QUESTIONNAIRE

It is true that the SE scale should diverge from the CKT, as it did in Van Driel et al. SE may in fact reflect actual knowledge, however, but not necessarily in this domain: people who know a lot tend to realize they do, if imperfectly. However, a more compelling case for divergence would be made if you could find a scale that would seem to be uncomfortably close to CKT, but could be shown to in fact diverge. Because CKT is not a psychological construct in the sense of attitudes or personality, such a scale might be hard to find. However, another way is to show that CKT is related to a broader construct, but not precisely (e.g., IQ). In other words, show that high IQ does predict higher CKT, but not very much because CKT is domain-specific.

Your interest in using the CEQ suggests some possibilities for criterion validity studies, which are missing in this draft of the project. You can use known-groups methods to see if the right sort of people have the right sort of CKT scores. “Right sort of people” includes a lot of possibilities, including demographics, education, type of eduction, etc.

For 100K euro, you could get a better sample…

To verify if the modified original instrument and the adapted instrument actually measure cultural knowledge, we propose a validation study with a sample of 500 subjects, 250 Germans and 250 native English speakers from various educational backgrounds. For instance, the German sample could consist of students who graduated from Hauptschule, Realschule, Gymnasium and university. The respondents will be asked to complete the original, modified version of the questionnaire or the adapted version, respectively, before completing the Cultural Experience Questionnaire (CEQ) and an IQ test.

Reword- you wouldn’t say “difference” shows “sameness”

I’m not following your logic here: how does comparing the original CKT with the revised CKT, in the same language, examine cross-cultural equivalence? Once you change the original English version, you are leaving it behind. The question is, does the new one, in English and in German, measure the right thing?

Using German Jacobs students does provide an interesting possibility since that group has great English and in most cases considerable cultural experience: you can have half the students take the English version and half take the German version, allowing you to look for purely language/translation problems. However, this is not a validation study.

Research by Van Driel et. al (2008) showed that intercultural experience measured by the CEQ correlates significantly and positively with cognitive and motivational components of cultural intelligence (Van Driel et al). Cultural experience, therefore, has a high association with cultural intelligence. Hence, we expect that the CKTs will show a significant positive correlation with the CEQ which would prove that the instrument possesses convergent validity. Similarly, an IQ test will be chosen to study divergent validity of the CKT. As IQ encompasses broader domains of knowledge, a low correlation between CKT and IQ scores would indicate the presence of divergent validity. In order to assess if the questionnaires have criterion validity the results of groups with different educational backgrounds are compared. If less educated people have lower scores this satisfies the criterion validity since the score can –to some extent- be predicted from the educational background.

By the inclusion of the steps presented in this paper, our multinational group was able to develop a German version of the selected questionnaire. After having identified strengths and weaknesses of the original, all items were translated, including back-translation of selected question by third persons. Our proposed validation study finally guaranteed a possibly effective way of testing the goodness of our Cultural Knowledge Test Adaptation.

[...]