was a november day in 2011 when 15 boxes arrived at
the Bahen Centre at the University of Toronto. They were
stuffed with 15,000 16-page exams from the Canadian
Open Mathematics Challenge, and James Colliander had
to get them marked.
He ordered pizza and assembled 100 markers – a mix
of faculty, postdocs, grads and undergrads mostly from the
school’s mathematics department, where he’s a professor.
Together, they put in a tedious eight-hour day on this volunteer task. The marking was repetitive, and since each
grader would focus on just one section, there was a lot of waiting going
on as the person working on section D had to wait for section C’s grader
to finish up, and the like. There was a lot of flipping of pages and moving
booklets around. “It was a logistical nightmare,” recalls Dr. Colliander.
But that day triggered an idea: what if this kind of co-marking, which
happens all the time for large exams in mathematics and the sciences,
could be transferred online? No waiting for other sections of the paper to
be done. The opportunity to cut and paste often-used comments. No red
pens and pizzas.
So Dr. Colliander started working with mathematician and software
developer Martin Muñoz to develop Crowdmark, an online software platform. With the help of MaRS Innovation, an agency that facilitates the
commercialization of university research, the two formed a company. In
November 2012, Dr. Colliander orchestrated the marking of that same
exam, recruiting 150 markers from across Canada to log onto a beta version of Crowdmark. His new team pushed through the same number of
tests in 340 person hours, compared with 700 the year before.
Now, Crowdmark is being used in educational institutions in 36 countries, including 25 universities. It’s allowing teams to take paper exams on
just about any subject, scan them into a computer and mark them collaboratively. Bonus functions include allowing a course director to chart the
accuracy of graders, fend off cheating and change marks quickly when a
student successfully disputes a grade.
This new piece of software is just one innovation changing how academics mark. Changing ideas about learning, combined with technology,
are making this significant – and often time-consuming, stressful and
even boring – part of the job faster, fairer and better geared to learning.
Creating assignments and tests and marking them endures as one of
the biggest challenges of the teaching side of academia. And it takes a lot
of time. “There have been so many times I’ve missed out on being with
my friends or family, particularly in the spring when it’s lovely outside,
because I have thousands of papers to grade,” says Dr. Colliander.
For Anne McNeilly, associate professor of journalism at Ryerson University, the weekly assignments for her first-year students translate into
endless marking. “I feel like I’m working all the time,” says Ms. McNeilly.
She takes papers wherever she goes, marking at the doctor’s office or at the
arena while her son skates. In the morning, she’ll polish off a few papers
before leaving for work.
It takes Erin Aspenlieder about 20 minutes to mark an eight-to- 10
page essay – just part of the work for which she’s paid a full-time salary.
But Dr. Aspenlieder, educational developer with the office for open learn-
ing and educational support at the University of Guelph, is concerned
about part-time faculty who juggle four or five courses and get little sup-
port. “That takes time away from doing things which might get you off the
sessional treadmill, like research.”
And the time spent marking is often fraught. When the paper or exam
gets handed back, there’s often a lineup during office hours. “It’s exhaust-
ing to defend your grades,” says Dr. Aspenlieder. “Especially when the
reasons they give are so amazing: ‘Please give me this grade, I need to get
into vet school.’” With class sizes on the rise, the challenges of marking
The latest developments in marking fall into two categories: technology based and pedagogically based innovations. On the technology side,
advances over the years mean hundreds of paper-based, multiple-choice
exams can be marked automatically now with little human intervention.
That may be ideal for instructors in early-year survey courses, but it isn’t
ideal for students.
“Multiple choice is a great idea if you want to test one’s knowledge
of trivia and useless information,” says David DiBattista, professor emeritus in psychology at Brock University. Since the late 1990s he’s been
researching this form of testing and has concluded that most professors
write poor questions that don’t enable learning. He now spends his time
consulting with textbook companies – he calls the test items in most textbooks “junky” – and doing presentations on how to avoid testing for trivia
and get to higher-level thinking. (His tips on question creation include:
avoid “all of the above” and “none of the above” as answers; write questions that are in a question format, not sentence completion; and avoid
negatives in questions.)