(Jamaica Observer) The integrity of the region’s secondary examination has been brought into question as some educators are uncomfortable with the recently implemented electronic marking system.
The high school teachers, who spoke on condition of anonymity, said that the value of grades issued by the Caribbean Examination Council (CXC) has degraded since the system’s implementation.
“The major concern is the results, what we are expecting as teachers, we are not getting,” one teacher, who heads the English Department at a Corporate Area school, said.
The CXC syllabus covers two years — grades 10 and 11 — and it is within this period of teaching and assessing that teachers are able to pinpoint whether a child is likely to pass, and with what grade.
According to the CXC website, candidates are awarded grades between one and six, with a grade one representing an “outstanding performance” and grade six, a representation of “a very limited standard of performance”. “I am a teacher of English and there are some children in my class who I know can’t write a sentence and they go up and they get a [grade] one,” she said while noting acceptance that teachers aren’t always correct.
The department head related that the school’s equivalent in the Modern Languages Department also expressed concern about the performance of two students — one expected to do extremely well and another who consistently performed below average and was not expected to pass. When the results were published the usually poor-performing student was awarded a grade one, while the high achiever got a lower grade.
“It jumps out because this is the complaint all around and it’s not just this one case,” the teacher stated.
“We feel that the papers are not being marked properly because we are the ones who teach the students. We know what they are capable of, and students who are getting grades one and two we know — based on what we have seen with them for the last two years — that there is no way that they can get that grade,” an English instructor at the Corporate Area school added.
“A student at our school who can barely read got a grade two in English Language,” another English teacher from a rural high school told the Jamaica Observer. “Even students who failed the City & Guilds exam were passing CXC English.
“How can that happen?” he questioned. “The City & Guilds exam is usually for the slow students.”
The UK-based examination is dedicated to vocational studies and is aimed at recognising different types of learners. It provides certification for students who need an option as opposed to the CSEC exam.
Assistant Registrar, Public Information and Customer Services at the CXC Cleveland Sam, in a written response to the Sunday Observer noted that while there is no general pattern, candidates sometimes “exceed their teachers’ expectations” while others do not do as well as projected.
What is e-marking?
E-marking, which was first employed in 2013, facilitates the marking of scripts in an online environment rather than a physical location.
In its first year, there was over 90 per cent correlation between scripts marked electronically and manually, when compared.
The assistant registrar, in explaining the system’s stages, said the scripts are first scanned and separated by questions, then individual questions are assigned to markers. The scripts are then marked using random seeding to uphold quality standards, and finally samples of scripts at grade boundaries are reviewed to ensure cut scores are maintained each year.
Prior to e-marking, the CXC would conduct what was termed table marking, wherein papers would be marked manually in marking centres across the territory.
The chief markers, as explained by the English HOD, would be flown to a location to meet and determine the acceptable answers and mark schemes. These individuals would then return to their territories and train table leaders — who would be responsible for up to six markers — using the established scheme from the previous island.
“We would come up with other things in that training process and then the markers would come in. When they came in we went through the process again and we would add and subtract, it would be an argument, for want of a better word, in an attempt to come to the best mark scheme and then the marking would begin,” the teacher explained.
Recounting an instance where papers had to be recalled and re-marked following the re-evaluation of an answer that was initially considered incorrect, the teacher noted that table marking facilitated ease of communication.
Comparatively, the Corporate Area English teacher asserted that e-marking limits the accepted answers to the CXC’s prescribed responses and lacks effective communication.
“I know there is room for giving feedback, voicing your concerns and having the chief respond to you, but in terms of changing the seed (prescribed answers), now that is not done,” the tutor said. “What happens now is that if you keep marking away from the seed you’re gonna be thrown off or you get bumped out of the system, and you can’t continue marking until you are re-added by your supervisor.”
The current system uses a method called seeding, whereby for every few scripts seeds (pre-marked papers) and presented randomly to the marker as unmarked scripts. Each seed has a prescribed total and if the marker’s total does not match, the marker is stopped.
The marker is bumped from the system after three “errors”.
The English Department head believes this causes markers “to adhere to the mark scheme slavishly” instead of fairly considering a candidate’s response.
But CXC, in debunking the teacher’s claim, maintained that e-marking improves the quality of marking through consistent seeding — a more effective method to sampling done in the past.
“More robust controls are applied earlier since the marks are immediately available for analysis. In the past, the marks were captured via data entry process that was several days behind marking as well as had the potential to introduce errors due to wrong keying,” Sam’s response stated.
“It allows for wider participation, since CXC was unable to fly persons from all territories and sought to use more markers where the centres were located. Now, markers from all territories have equal opportunity to mark. [Additionally] marking can be started earlier, since teachers are not removed from classroom,” he added.
According to the CXC, e-marking is also beneficial as is it insulates the marking process from the escalating cost of travel and accommodation; and reduces/eliminates the manual operations around marking, scoring and grading.
“CXC believes that the papers are being marked properly given the multiple layers of quality assurance,” the response read.
The outlined multiple layers listed were the fact that potential markers are pre-qualified; a standardisation process is used to ensure specifications are met before going live; random seeding at 10 to 20 per cent is done to maintain marking quality; distribution of marks by measurement officers is reviewed; and sampling of marked responses at grade boundaries to ensure year on year standards is carried out.
Grades issued before marking
The educators also expressed concern that grades were released before all the scripts were marked.
The department said that in one instance, the day before the results were released, “thousands of scripts” were not yet marked.
“I thought it lacked integrity because I don’t understand how it is that, overnight, these unmarked scripts got marked,” she told the Sunday Observer.
She claimed that up to a week before results were released this year, the organisation was still sending e-mails to recruit markers.
“They were asking for markers for English Language, English Literature, Mathematics etc. They were asking for markers in these subject areas but the wording suggested to me that you don’t have to be a specialist in the area,” she said.
But the CXC, while denying the amount, said results have been published before all scripts papers were marked.
“There are cases of papers not being marked up to the release of results. This is what is referred to as ‘mop-up’ marking, but it’s not usually ‘thousands’. We have had mop-up marking since the implementation of e-marking,” Sam responded.
Regarding recruitment, he explained that non-teachers are eligible to mark scripts once they are able to pass pre-qualification and are marking to the specified standard.
He said that upon recruitment, individuals are taken through a process of pre-qualification wherein their certification is verified before they are permitted to mark scripts.
“Once this has been completed, they are taken through training on the marking tool, then standardisation on the question he/she is assigned to mark,” Sam stated. “The standardisation process involves the understanding of the marking instructions, then marking up to 40 items to ensure they are marking to the chief examiner’s specification. If within tolerance, they move on to live marking, otherwise, they are suspended.”
Unattractive remuneration
Pointing out that most teachers marked because “it was good money” and the experience in marking aided teaching the syllabus, one teacher argued that marking is no longer worth it.
“At the end of the day, sitting down for seven hours for five days in a week to mark hundreds of scripts…of course you expect that you are going to be paid and the pay is good,” she reasoned. “[But] when they started the online marking they changed the pay system and you were paid per script, but the money was quoted in Bajan (Barbadian) dollars and questions are weighted differently.”
Instead of having a flat rate, she noted that an individual marking an essay question would, for example, receive roughly BBD$1.58 (J$100.65) per script, while an individual marking comprehension would receive BBD$0.56 ($35.67) per paper.
“So when you do the conversion and you work it out, if you mark say 1500 scripts you don’t even make close to the $80,000 that you used to make, especially if you are marking comprehension,” she disclosed. electricity…because all they do is provide the software that you load on to your personal computer and you sit, on your time, and do it.”
However, CXC argued that the new payment plan is to ensure that persons were rewarded based on their level of work.
“With the flat rate everyone got the same reward regardless of the volume marked or the quality of their work,” the response noted.
“The new system seeks to reward those who complete more scripts with more pay. Those who are not able to maintain the standard are suspended and hence receive lower pay. We have also examined the effort on questions and have categorised them accordingly. As such, an average hour of effort should result in the same level of payment, even though the number of scripts marked may differ,” it continued.
