The work at hand is part of a wider study by (
Este trabajo es parte de un estudio más amplio (
El treball que ens ocupa és part d’un estudi més ampli de (
Despite the importance of pragmatic competence in the field of communicative teaching of second languages, the evaluation of this competence is still a neglected area in many of the commercial and computer adaptive tests (CAT), such as the Oxford Online Placement test (OOPT), and in some cases it is even non-existent.
Furthermore, we have noticed that only a small number of studies have examined the relation between the level of proficiency in English and the pragmatic competence of EFL learners. The studies on this issue can be grouped into two blocks, on the one hand, those that result in a positive correlation, and on the other hand, those that show that the level of linguistic competence is not a sufficient condition to determine the level of pragmatic competence.
According to the results in this study, we cannot confirm that there is a relationship between the level of linguistic competence and the level of pragmatic competence of the participants in this research. This contradicts the results obtained in some previous research. Moreover, the results in our study also indicate that the weight of pragmatic competence in the configuration of the final score in the OOPT has been proved to be low.
High stake tests have the capacity to influence the programs and methodologies applied in the classroom, consequently, an effective systematic teaching of pragmatics throughout the different stages of education depends to a great extent on the way pragmatic items are dealt with in these influential types of tests. Hence, further research and work needs to be done which focus on the way pragmatics is being taught in the classroom and also on the improvement of construct validity of pragmatic items in high stake EFL tests, and more specifically, in CATs.
In this study we focus on the analysis of the pragmatic subtest of the OOPT, which is administered by the Faculty of Teacher Training (University of Valencia, Spain) to undergraduate students who need to certify their level of proficiency in English as a foreign language. After scoring a group of test takers, some uncertainties arose regarding the quality of the items in the test, since many examinees complained about their difficulty, and they disagreed with the scores obtained. We got the impression that those items which the students found more difficult were the ones corresponding to the pragmatics part of the test. In view of this, our objectives were the following: to analyze the pragmatics items validity; to see how much importance is given to pragmatic competence in the OOPT, and finally, we aimed to explore the relationship between grammatical and pragmatic competence of the participants in this study. To present the answer to these questions, this paper is organized as follows. Firstly, the theoretical background considered in this study is offered. Secondly, the students involved in the project, the materials and the method used are explained. Thirdly, the results extracted from our analysis are shown and finally, some conclusions are drawn.
Pragmatics, the systematic study of communicative effectiveness, as defined by
Taking this into consideration, it would not be wrong to think that the evaluation of pragmatic competence should, consequently, be included in all EFL domain tests.
But despite the importance of pragmatic competence in the field of communicative teaching of second languages, the evaluation of this competence is still a neglected area in many of the commercial and computer adaptive tests, and in some cases, it is even non-existent.
Furthermore, some of these tests do not contemplate the interactive nature of the language or its purpose. In the same way, contextualization is often poorly defined. It is also noticeable that the only commercial CATs which contain specific pragmatic items are the Test of English as a Foreign Language (TOEFL) in its reading comprehension section, and the Oxford Online Placement Test (OOPT), with a pragmatic subtest (
Knowledge of speech acts and their strategies.
Interpretation of implicatures.
Recognition of formulaic expressions and routines.
So, some other aspects such as production and recognition of speech styles, contextualization cues, discourse structure, sequence organization or the effect on interlocutor are hardly ever the object of studies in this field (
So far, we have seen some issues related to the treatment of pragmatic competence in the most widely administered CATs today, as well as some aspects that affect the validity of this construct. However, we must add to this some other facets that affect the design of the EFL domain tests. One of these aspects is their psychological effect on the examinees since this type of tests are often seen as a threat rather than an opportunity to learn and improve (
Despite everything, in the educational environment, teaching and language tests are aspects of the same basic problem according to
The Council of Europe project for adult language learning is an example of this systematic approach since it considers the purposes for which the members of the European community will most likely communicate since courses and evaluation are developed from them. However, the Common European Framework of Reference for Languages (CERF) (
The development of the learner’s
Assumed to be transferable from education and general experience in the mother tongue (L1)? Or facilitated:
By progressively increasing the complexity of discourse structure and the functional range of the texts presented to the learner?
By requiring the learner to produce texts of increasing complexity by translating texts of increasing complexity from L1 to L2?
By setting tasks that require a wider functional range and adherence to verbal exchange patterns?
By awareness-raising (analysis, explanation, terminology, etc.) in addition to practical activities?
By explicit teaching and exercising of functions, verbal exchange patterns and discourse structure?
Users of the Framework may wish to consider and where appropriate state
To what extent sociolinguistic and pragmatic competences can be assumed or left to develop naturally.
What methods and techniques should be employed to facilitate their development where it is felt to be necessary or advisable to do so. (p.154)
The CEFR points to pragmatics as part of communicative competences, dedicating several sections to it in the document but leaves it out of the evaluation in the ninth chapter. This lack of precision is due to the fact that the Council of Europe has no power to impose anything on any member of state and that it is only a point of reference and "not a means for coercing teachers, nor even a basis for measures of accountability", as
Another aspect worth considering is the one that draws our attention in this study, which is the degree of relation between the level of grammatical and pragmatic competence of EFL learners. We noticed that only a small number of studies have examined this correlation (
In the following sections, the method, the participants, and the procedure which have been carried out are explained.
Our interest in studying the OOPT emerged when a group of students complained after taking the test and obtaining their results. They found some items very difficult and they disagreed with the scores obtained. Regarding this, our hypothesis was that those items were the ones corresponding to the pragmatic part. Consequently, the following research questions arose:
Do the pragmatic questions in the OOPT hold an outstanding frequency of wrong answers which could indicate that further analysis of tests items is needed?
What is the contribution of the pragmatic part to the configuration of the final score?
Moreover, another question led the main objective of this study:
What is the magnitude of the correlation between the grammatical and pragmatic competence in English of the learners involved in this study?
The participants in this study were 34 Spanish undergraduate students at the Faculty of Teacher Training (University of Valencia, Spain). The sample was homogeneous in terms of age, between 18 and 25 years old, and in terms of proficiency in English since all the examinees ranged from A2 and B1 levels acquired through formal education. Four of the participants were men and 40 were women, so we can say that in this study the anthropometric variables have little variability. Only samples of the students whose results ranged between A2 and B1 levels of proficiency in English were selected for the analysis of the test because B1 was the expected level to be reached by those who did not have obtained it by the end of the first academic year.
None of the students had stayed in the target language community, so, they were provided with culture-rich, interactive materials to foster spoken interaction and spoken production. The teaching methodology implemented throughout the course was based on a pragmatic approach which supported learning awareness rising, in such a way that the students were aware of the importance of the context in the verbal communicative act and the significance of the inferential process in the decoding of implicit meanings.
The OOPT is an adaptive test that assesses the level of proficiency in English and it is distributed by Oxford University Press. The main objective of the OOPT is not only to measure the grammatical or lexical competence, but also the ability of the examinee to understand many grammatical forms and the meanings that they transmit in different contexts. It also measures to what extent learners can use these language resources to communicate in English language situations (
As stated by (
To address the data analysis, the first drawback we had to tackle with was the impossibility to carry out an item analysis to get the discrimination and difficulty indexes of the pragmatic items due to the adaptive format of the OOPT. So, the first research question had to be formulated as follows: “Does the pragmatic part in the OOPT hold an outstanding frequency of wrong answers?” To resolve this question a descriptive analysis of the erroneous answers by the examinees in each of the test items was carried out. Then, the results obtained were described, both, globally and by parts. Four blocks or parts totaling 44 items have been distinguished in the analysis, the first corresponds to grammar items, the second corresponds to pragmatics, the third one is devoted to writing skills and the fourth is the reading comprehension part. The second and third research questions were: “What is the contribution of the pragmatic part to the configuration of the final score?” and “What is the magnitude of the correlation between the grammatical and pragmatic competence of the participants in this research?” Therefore, to give an answer to these questions the relative weight of each block in the configuration of the final score was quantified to find out the degree of contribution of the pragmatic items in the OOPT final score.
The analysis, carried out to identify the blocks that most contribute to the configuration of the final score, consisted in the estimation of a multiple linear regression model with dependent variable, the score of the test, and independent variables, the number of failures (or equivalent, the percentage of failures) in each block of items. The selection of variables was carried out by means of a method of successive steps. The value of the coefficient of determination was deemed as an indicator of the degree of adjustment achieved with the model. The possible collinearity between the independent factors was evaluated, considering the condition and decomposition of the variance indexes.
A preliminary exploratory analysis of the correlation between parameters was carried out using the Pearson or Spearman correlation coefficient, according to the observed sampling distribution and the result of the Kolmogorov-Smirnov test (
The estimations of the regression coefficients were accompanied by the 95% confidence interval. The theoretical hypotheses of the model were validated, that is, the results met a series of hypotheses: they followed a normal distribution, they had a constant variability (homoscedasticity), and they were uncorrelated between them (this is seen through the Durbin-Watson test). After checking these points, we saw that the model was robust.
The 44 items of the test are grouped into four large blocks: the first block comprises the items that evaluate the grammatical form or use of English with multiple-choice questions; the second block is made up of multiple-choice pragmatic items; the third block is composed of fill-in-the-blank items that assess writing, and the fourth block includes reading comprehension items.
The first research question asked which part of the OOPT held the highest frequency of wrong answers.
The results obtained in the descriptive analysis of failures in the four blocks of questions that make up the OOPT are presented below. The percentage of errors on the total number of items in each block has been calculated (
n |
mean |
Standard deviation |
Minim |
Maxim |
Median |
|
---|---|---|---|---|---|---|
GRAMMAR (10 items) |
34 |
36,2 |
11,6 |
10,0 |
60,0 |
40,0 |
PRAGMATICS (12 items) |
34 |
45,8 |
16,6 |
8,3 |
75,0 |
50,0 |
WRITING (5 items) |
34 |
47,1 |
29,1 |
,0 |
100,0 |
60,0 |
READING COMPREHENSION (17 items) |
34 |
44,8 |
11,5 |
23,5 |
70,6 |
47,1 |
TOTAL (44 items) |
34 |
43,4 |
8,0 |
27,3 |
61,4 |
43,2 |
In the grammar or ‛use of English’ block (
For a better understanding of the sampling distribution of the parameter ‛percentage of errors’, in
Finally, the grammar items are revealed as the easiest to answer since only 50% of subjects fail more than 40% (the median) of the items in the block. It is noteworthy that the range of 'normal' values is the narrowest of all blocks. The participants present a more comparable level in terms of the results obtained. Now, there is a peculiar characteristic, it is the only block where atypical cases are identified, one above and one below.
The second and third questions addressed the contribution of the pragmatic part to the configuration of the final score
A multiple linear regression analysis was performed to determine the relationships between the (final) score, and the number of failures recorded in each block, and the relation between the grammatical, reading and writing levels of competence and the pragmatic level. Basically, the issue here is to discern how the final score is configured from the partial ones. To control the effect of the different number of questions in each block, we chose to use the wrong answers percentages rather than the absolute number as predictors.
Correlations |
Score |
Gramm. |
Pragmatics |
Writing |
Reading |
||
---|---|---|---|---|---|---|---|
Rho Spearman |
SCORE |
Correlation coefficient |
1,000 |
-,471(**) |
-,299 |
,012 |
-,516(**) |
|
|
Sig. (bilateral) |
. |
,005 |
,086 |
,945 |
,002 |
|
|
N |
34 |
34 |
34 |
34 |
34 |
|
GRAMMAR |
Correlation coefficient |
-,471(**) |
1,000 |
-,340 (*) |
-,029 |
,040 |
|
|
Sig. (bilateral) |
,005 |
. |
,049 |
,870 |
,822 |
|
|
N |
34 |
34 |
34 |
34 |
34 |
|
PRAGMATICS |
Correlation coefficient |
-,299 |
-,340(*) |
1,000 |
-,011 |
,269 |
|
|
Sig. (bilateral) |
,086 |
,049 |
. |
,952 |
,125 |
|
|
N |
34 |
34 |
34 |
34 |
34 |
|
WRITING |
Correlation coefficient |
,012 |
-,029 |
-,011 |
1,000 |
-,025 |
|
|
Sig. (bilateral) |
,945 |
,870 |
,952 |
. |
,888 |
|
|
N |
34 |
34 |
34 |
34 |
34 |
|
READING COMPREH. |
Correlation coefficient |
-,516(**) |
,040 |
,269 |
-,025 |
1,000 |
|
|
Sig. (bilateral) |
,002 |
,822 |
,125 |
,888 |
. |
|
|
N |
34 |
34 |
34 |
34 |
34 |
The result of the regression model is presented in
Model |
Unstandardized coefficients |
Standardized coefficients |
t |
Sig. |
95% confidence interval for B |
|||
---|---|---|---|---|---|---|---|---|
B |
Standard error |
Beta |
Lower limit |
Upper limit |
Lower limit |
Upper limit |
||
1 |
(Constant) |
61,705 |
7,315 |
|
8,435 |
,000 |
46,804 |
76,606 |
|
READING |
-,717 |
,158 |
-,625 |
-4,528 |
,000 |
-1,039 |
-,394 |
2 |
(Constant) |
79,443 |
6,581 |
|
12,071 |
,000 |
66,021 |
92,865 |
|
READING |
-,627 |
,121 |
-,547 |
-5,179 |
,000 |
-,874 |
-,380 |
|
GRAMMAR |
-,601 |
,121 |
-,527 |
-4,986 |
,000 |
-,847 |
-,355 |
The model (
Beta IN |
t |
Sig. |
Partial correlation |
||
---|---|---|---|---|---|
1 |
GRAMMAR |
-,527(1) |
-4,986 |
,000 |
-,667 |
|
PRAGMATICS |
,014(1) |
,094 |
,926 |
,017 |
|
WRITING |
-,018(1) |
-,127 |
,900 |
-,023 |
2 |
PRAGMATICS |
-,207(2) |
-1,841 |
,075 |
-,319 |
|
WRITING |
-,039(2) |
-,365 |
,717 |
-,067 |
The R2 or coefficient of determination achieved with the model obtained takes the value 0.662. That is, up to 66% of all the variability of the score can be explained from the percentage of failures made in the blocks of reading and grammar. Some students have high scores, and others have low ones. That is, the score is presented in the data with a dispersion or variability that can be explained by different factors. Among those that we study here, the results of the different blocks that serve to predict the score or assume part of the responsibility of that final score. The interpretation is that the pragmatic items have a certain relationship with the final score of the student, but they do not improve the prediction that the grammar part and the reading part alone provide. On the other hand, the pragmatic and grammar part exhibit an inverse correlation, that is, if a student is good at grammar, we roughly know that he/she is not so good at pragmatics; therefore, it is not necessary to explicitly incorporate the pragmatic score in the model to better predict the final score. The results of a forced introduction model of all the factors are shown in
Non standardized coefficients |
Standardized coefficients |
t |
Sig. |
95% confidence interval for B |
||||
---|---|---|---|---|---|---|---|---|
B |
Standard error |
Beta |
Lower limit |
Upper limit |
Lower limit |
Upper limit |
||
(Constant) |
87,354 |
7,883 |
|
11,081 |
,000 |
71,231 |
103,477 |
|
GRAMMAR |
-,685 |
,126 |
-,600 |
-5,425 |
,000 |
-,943 |
-,427 |
|
PRAGMATICS |
-,165 |
,091 |
-,207 |
-1,818 |
,079 |
-,351 |
,021 |
|
WRITING |
-,018 |
,046 |
-,040 |
-,392 |
,698 |
-,113 |
,077 |
|
READING |
-,548 |
,126 |
-,478 |
-4,342 |
,000 |
-,807 |
-,290 |
The situation is totally stable. The R2 of this model is 0.698, that is, with two more factors we hardly increase the explicability of the final score, so the previous, more simplified solution is preferable.
The starting hypothesis at the beginning of this study was that the type of pragmatic items in the OOPT could be negatively influencing the results obtained by the 34 students who took this test, given that, after listening to the students’ opinions, the items seemed to be of rather high difficulty. Then, the first research question arose: “Does the pragmatic part in the OOPT hold an outstanding frequency of wrong answers?” To resolve this question a descriptive analysis of wrong answers by the examinees in the test was completed and it was found that the ‛writing’ part of the test was the one holding the highest mean of wrong answers, followed by the pragmatics part. Therefore, our hypothesis is null since, although the difficulty of the items is moderate, they are not the most difficult items of the test.
As for the second research question, “what is the contribution of the pragmatic part to the final score”, a regression analysis was applied to solve it, which reveals that the pragmatic items have a certain relationship with the final score of the students but do not improve the prediction of the final score that the grammar part and the reading comprehension part alone provide. Therefore, the results do not confirm our starting hypothesis and the null hypothesis must be accepted since we verify that these items hardly influence the final score.
As for the third research question,
On the other hand, the lack of correlation between the level of grammar and the participant´s pragmatic competence in this study could be because, pragmatics is not being systematically or sufficiently taught, neither explicitly nor implicitly, unlike grammar. Taking this into consideration, the results are informing us of the need for pragmatics to be taught in a more resourceful and efficient way in the classrooms since, as teachers, we cannot expect this competence to be developed and acquired without any type of instruction, at the expense of the learning and acquisition of grammatical competence. It may be necessary to undertake the teaching of pragmatics more thoroughly, similarly to the new approaches to vocabulary teaching and by adopting, in our methodology; the contributions of the studies that are being carried out in the field of lexical pragmatics “a research field that tries to give a systematic and explanatory account of pragmatic phenomena that are connected with the semantic under specification of lexical items” (
This work has also revealed two facts that should be noticed. On the one hand, it is verified that the incorporation of pragmatic items in the EFL tests is generally deficient, since it has been seen that only two of the numerous current commercial high-stake tests include pragmatic items. This indicates that pragmatics is a competence to which EFL tests still fail to give enough importance. On the other hand, the weight of pragmatic competence in the configuration of the final score in the OOPT has been proved to be low compare to the weight allocated to the grammar part. This fact is paradoxical if one thinks that within the communicative approach, which is currently being applied in most EFL classrooms, students are intended to understand and express themselves adequately in the target language according to different communicative and socio-cultural contexts. We are aware that this research may have some limitations, for instance, the sample size and the nature of pragmatic features analyzed may have slightly influenced the outcome. Yet, the results indicate that it is worth asking whether there is a truly consistent relationship between language programmes and high stake, standardized measurement systems and to what extent they derive from an analysis of the learner's communicative needs. There does not seem to be a coherent link between the external tests and the classroom evaluation with the fundamental model of learning. High stake tests have the capacity to influence the curriculum and methodologies applied in the classroom. This is due to the influential power of this type of tests and their active role in shaping education policies. Consequently, an effective systematic teaching of pragmatics throughout the different stages of education depends, not only, but to a great extent on the way pragmatic items are dealt with in these tests. Hence, further work needs to be done which focus on the way pragmatics is being taught in the classroom and on the improvement of construct validity of pragmatic tests in EFL.
Lexicalized Trope Inferences are groups of fixed words the meaning of which meaning is usually known by any competent speaker and usually does not come exclusively from the isolated meanings of the words that constitute them (
Speech acts are, according to Austin, “[...] functional units of communication that have prepositional or locutionary meaning (the literal meaning of the utterance), illocutionary meaning (the social function of the utterance), and perlocutionary force (the effect produced by the utterance in a given context.” (
Lexicalized Trope Inferences are groups of fixed words the meaning of which meaning is usually known by any competent speaker and usually does not come exclusively from the isolated meanings of the words that constitute them (