Friday, 29 November 2013

Large Classes, Large Rewards: Economies of Scale and the Democratisation of Higher Ed


Today I am giving a presentation on my teaching model as part of the University of Canberra's weekly "Saffire Snippet" series.


Some time ago, I blogged about using video content with a transcript as a format for teaching materials. It was thought that the format would assist students to review content in various formats, using links in the transcript to additional information about key topics and so on. It was intended that the traditional two-hour lecture format would become obsolete, and classrooms would be "flipped" where the lectures were shorter but the tutorials were longer. In effect, lecturers would be "curators" of educational content. I recently trialled the flipped classroom and despite my initial scepticism, it appears to have worked. In addition, I have become less hands-on in my teaching as the numbers of students in my subjects climbs into the thousands. Here I discuss some of the key approaches that are enabling me to achieve quality in learning outcomes while achieving economies of scale. Given the recent push to democratise higher ed, and the death of the traditional lecture, a new approach to teaching is necessary and I think I am getting closer to an ideal for teaching large classes.


Video Lectures


I have discussed previously the use of video lectures and most recently one of my lectures was recorded for the University of Canberra's youtube channel. The first step was to reduce a two-hour lecture to some 15 minutes of "youtube" style video recording. I have tried a few different approaches but there are two different models. The first is to use content from previously recorded lectures and edit this using Windows Movie Maker. I then publish this to the University's video system, Echo360, via an import. A major advantage of using Echo360 is that students' download limits are not affected by the use of the in-house system. As the quality of video files increases, so does the size of the data, so using an in-house system rather than youtube was an important limitation. As it turns out, WIndows Movie Maker is easy to use and the formats are readily published via Echo360. I have managed to overcome copyright issues by purchasing stock photos and stock music and these complement the learning materials and give them a professional, TV program polish.

Reducing a two-hour lecture to around 15 minutes was an interesting exercise. In some cases, the content was rather light and could easily be reduced, with the classroom examples added as links in Moodle or in the video transcripts. In other cases, it was difficult to reduce the lecture to less than 30 minutes, indicating that there was too much content in that particular session. This forced me to reconsider the use of discrete weekly lectures, and to package a series of lectures as a module. More on this later.

The second lecture model - the "high-end" lectures - were recorded with assistance from my colleagues in the media area using either a green room or multiple camera angles. This was a different approach to developing content in that I spoke for 15 minutes on a key topic. In most instances this approach was much faster than editing pre-recorded content. Of course, it is resource intensive and requires additional people, so the approach is not always practicable. But to my surprise, it was obvious that the two-hour lecture was unnecessary for getting across content. Traditionally, the large lecture was designed to provide an economy of scale, in that one academic could reach the number of students who could be seated in the one lecture theatre. Of course, with new communications technologies, the lecture theatre is no longer necessary and video content can readily replace the face-to-face lecture. As most lecturers would have noticed - students simply stopped coming to lectures, and many of them did not watch the lengthy video recordings. Whether this is a lack of discipline on the part of the student or not is irrelevant: as we continue to democratise higher ed, we will take on students with a variety of language and intellectual skills, from a variety of cultural and socio-economic backgrounds. Put simply, insisting on disciplined lecture attendance doesn't work.



Once the video content was operational, the next step was to design the tutorials. Traditionally, students would attend a two-hour lecture each week followed by one-hour tutorial in groups of about 20 per tutorial class. There are many limitations, including the room size and availability of sessional teaching staff, so some aspects of traditional classes have remained unchanged. However, getting students to attend a one-hour tutorial every week started off with near-full attendance and dwindled off to a handful of students, sometimes none, in the final weeks of semester. Attempts to counteract the lack of attendance rested on an arbitrary "participation" mark. There are two problems with this mark: how can you actually assess performance in class and make the grade consistent across classes? You can't. In many cases, students were awarded for attendance, despite policies which clearly state that you cannot be assessed on attendance. In many cases, participation grades have been used as a "fudge" factor to bring grades up or down as part of the moderation procedure. These have long been seen as overly subjective but without an assessment item to motivate students to attend class, classrooms would be empty each week. Using the flipped classroom model, however, I set up a series of Moodle quizzes, and increased the length of tutorials to two-hours, with an assessed quiz conducted after each tutorial class. Of course, two-hour tutorials are potentially more expensive than one hour tutorials, so I reduced the number of tutorials so that the total number of hours remained unchanged. So a student attends a two-hour class one week then completes an online Moodle quiz (typically a multiple choice question test) the following week. What shocked me most was that tutors were reporting almost 100% attendance throughout the semester, as the discussion in class were focused on the content assessed in the quiz. This enabled me to rid myself of the evil participation grade while motivating students to attend class.

Of course, not all students want to study in face-to-face mode, so I have been working on a variety of "fully flexible" models over the years. One problem is that students need to be able to choose either face-to-face tutorials or online tutorials, and this needs to remain consistent throughout the semester, otherwise it is impossible to manage sessional teaching staff workloads and students' timetables. This has largely been achieved by allowing students to enrol in two modes (either on campus or online) and using the existing timetable system for the on campus students, and then using the groups function in Moodle to add online students to a forum for conducting online tutorials. The distinction is important: being able to isolate the online students from the on campus students and put them into discrete groups of twenty online classes enables me to arrange the online teaching workload into an online equivalent, and manage online students' expectations about contact hours and activities. Of course, the literature on establishing online communities still applies, and although it is difficult to encourage participation, especially from "lurkers", many students appreciate the flexibility of online study. Further, it appears from initial feedback that online equivalents to two-hour face-to-face classes are too onerous, and I am considering breaking down the online study into weekly modules with a quiz at the end of each two weeks, for example. Although this is substantially different from the on campus approach, it seems to work better in the online environment.


Online Quizzes

There are two major challenges with managing online quizzes to ensure the integrity of the quiz questions: 1) ensuring there are sufficient questions to prevent the quiz questions becoming common knowledge; and 2) ensuring quiz questions are randomised sufficiently, especially when quizzes are left open over a period of time. Two solutions have been found which seem to work (after a long period of trial and error. The first is to use Respondus 4.0 to manage the quiz questions, and the second is to use Moodle's randomisation function. I have recently discovered a workaround for uploading quiz questions to Moodle from Respondus. It seems that Moodle's plug-in which enables uploads from Respondus is problematic. The solution is quite simple: change the format of the Respondus file to Blackboard, and then publish the quiz questions as a file for manual upload. Moodle enables the upload of Blackboard question files, so simply import the Blackboard file and you can then use Respondus for all Moodle quizzes. What I find most useful about Respondus, in addition to creating Moodle quizzes, is that I can produce hard-copy multiple choice question exams and ensure exam questions are not duplicated, including random sets of hard-copy exams for different campuses, final versus deferred exams and so on. Of course, Moodle has a randomisation function which works quite well, and provided students cannot see the correct answer for quiz questions until after all students have completed the quiz, the integrity of the quiz for future use remains unchallenged. It also means that online quizzes an be left open for extended periods, as no two quizzes will be exactly the same. This provides greater flexibility for students and has proven successful. Average quiz results suggest that there is little room for cheating due to the large number of questions which are randomly selected by Moodle, and that questions and answers are all randomised so that each student's view of the quiz is unique. Respondus has enabled me to readily update quiz questions so that the question bank remains relevant as content changes. Without doubt, Respondus has proven to be the most useful question database software and the workaround for uploading to move has provided increased efficiency in creating and modifying quizzes.


Marking Rubrics

Although Galileo may have influenced the marking of assignments by suggesting that we should “measure what can be measured, and make measurable what cannot be measured”, I remain a sceptic. I truly believe that students use rubrics as substitutes for reading and learning acceptable standards through experience, especially in essay writing. Rather than read journal articles, I find too many students focus on the rubric and then argue for higher marks on the basis of the rubric. Rubrics are meant to be less subjective than other approaches, but I am not convinced. If we could design the perfect rubric, then there is no need for me to teach essay writing skills.

Nonetheless, as we democratise higher ed and the number of students increases, the likelihood of inconsistency in marking increases, especially where teaching staff are drawn form different academic disciplines. I have conducted numerous informal marking experiments where two or more political scientists mark the same work. Repeatedly, scholars within the discipline mark students' work within one or two marks of each other. However, once teaching staff are drawn from other disciplines such as economics, public administration, business or management, for example, the different training leads to substantially different outcomes in terms of marking. Much of this relates to different expectations and interpretations of learning outcomes, and in particular essay standards and expectations. To make this clearer for both teaching staff and students, especially for delivery on multiple campuses and by teachers from various disciplines, I am currently designing rubrics for marking essays.

The approach so far is to identify a number of criteria from a sample of essays graded at each of the grade levels, i.e. fail, pass, credit distinction, high distinction. Each criterion will be given a mark out of 5, where 1 represents a fail and 5 represents a high distinction. Once I can create a rubric which reflects the results of essays which have been assessed using the more subjective methods of my discipline, I believe I will have a model to enable teachers from various disciplines and across various campuses to mark essays consistently. The rubric will also be developed as part of an exercise in mapping learning outcomes directly to assessment items - a procedure that has tended to be deduced rather than deliberate in the past.


Feedback from Tutors


I will not have formal feedback from students until early next year, but to date informal feedback has been positive. Here is some feedback from the tutors' perspective:
  • I was able to put more time into essay writing skills before the first essay which led to better essay results.
  • A 2 hour tutorial enables more in-depth coverage of course material and gives time for useful group work as well as being able to schedule a number of different activities in the 2 hour period.
  • One on one feedback seems to be the most useful [approach] for improving the quality of student written work.
  • [The approach] seemed to result in more students reading more of the textbook (and purchasing it) than in previous [units]. They read the section required for the quiz, then moved on without having the burden of remembering everything for a final exam. At the same time they knew where to find relevant information for their essays.
Students seem to have liked the shorter video lectures with a direct link to a quiz. Many preferred regular quizzes rather than one big exam at the end of the semester. Comments from students included:
  • I watched all the 20 minute lectures but wouldn't have gone to all the long ones.
  • The mini lectures were really good for covering what was needed in the quiz.
  • I liked to focus on one part of the course, do the reading and do the quiz then move on.
One of the major challenges was to ensure that students were aware of the date and time of their tutorial class. Some found it difficult to remember which week their class was on. Nonetheless, I am confident students will learn to manage their time effectively, especially as the flipped classroom becomes more commonplace.

Overall, the flipped classroom appears to be a success, and although the nature of my role is changing significantly,  I believe it is a necessary part of the democratisation of higher ed. I have also debunked the myth that economies of scale cannot be achieved in higher ed: after years of hard work I have been able to manage larger numbers of students without a detrimental effect on the quality of learning outcomes. But in terms of debunking myths, the flipped classroom is a real winner. Despite my initial scepticism, students' results speak for themselves. With careful curriculum design and some investment of time and resources, teaching large classes can be rewarding for both staff and students.

Creative Commons License Except where indicated otherwise, Le Flâneur Politique by Michael de Percy is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 Australia License. Based on a work at Background image © @redshinestudio