ding ding's List: classroom response system

• answer two step multiple choice problems after a discussion within their group
• students are comfortable with this technology
• interactive lectures are useful
• e is an improvement in students' exam performance, but there are too many competing factors to clearly say that this improvement is solely due to the use of the classroom communication system. Even though this paper is based in physics and a physics example is used to illustrate points, the technique can be applied to other discipline areas.

2 more annotations...

• Consequently, I primarily used the PRS questions as a means of anonymously testing students’ understanding of material recently covered

• 1998 American  Association of Physics Teachers.
• while clearly of growmg importance, these are also encountering  constraints.
• A. Question format
• (a) exploration, to gauge the opinion of the students of  their own understanding (for example, "Have you understood  my arguments regarding this equation?");
• to test the ability of students to apply  the work to specific situations
• (d) organization ("Are you ready for me to continue?").
• B. Lecture format
• a short talk on how the system works,
• The mean score  for non-APF students was 5.1; for APF students this rose to 6.7.
• both increasing the mean pass rate of individuals  exposed to it and reducing the variability between the achievements  of different students.

8 more annotations...

• • it's about how EVS is used in Glasgow university from Oct 2001-Mar 2002; it focuses on general feedbacks from students and teachers; benefits and problems are reported, but not focuses on specific model.
• (Oct 2001- March 2002)
• level 1 to level 4
• with audience sizes from 20 to 300,
• been used in lectures and formative assessment sessions
• The lecturers
• felt the benefits outweighed the difficulties
• the main focus of the evaluation so far has therefore been on observation and collection of data, and not on the analysis of data and the production of written reports.
• Observation
• Informal discussions with students
• Use of the handsets to evaluate the use of handset
• Written comments from students
• Questionnaires
• Discussions with lecturers before and after
• Written feedback from lecturers
• too much time sometimes is involved in setting up the equipment
• essential feature
• both students and lecturer get to know the distribution of responses and, in confidence, how their own response relates to that
• anonymity
• With the handsets I could see exactly which points I had not conveyed clearly and could rectify it straight away
• the idea worked well, but that the time it took with a large group was too long.
• they could test their understanding of the topics as they went along
• gave me some idea
• A slight problem is the time it takes to register the students answers and you have to time this into the lecture.
• The students on the other hand do not mind the
   delay.
• I would need a plan B which would involve a fuller explanation. So it would affect the way I plan lectures.
• it was useful in finding out how much they knew
• 61 students
• Extremely useful 18.0%
   Very useful 21.3%
   Useful 37.7%
   Not very useful 21.3%
   Not at all useful 0.0%
   No rating 1.6%
• how he/she is unsure of the subject without embarrassing themselves.
• where we are in relation to class mates
• encourages us to participate
• Benefits
• Problems

32 more annotations...

• • 2002, by Steve Draper. the paper discussed the design retionale of EVS questions. and it also pointed out at the end that "The studies in prospect with this equipment should eventually allow us to pronounce on the validity of the design rationales presented in this paper."
    I think although question design is important, more should be discussed about after voting--how to use the new input?
• The effectiveness of the equipment for learning depends mostly on what pedagogic method is employed
• The feedback cycle here takes about two minutes per item (somewhat longer if explanations are given).
• at the end could demonstrate to what degree the group now understood the topic.
• (e.g. on whether the pace is too fast or too slow, the jokes too numerous or infrequent, the examples too many or few)

  • these questions would work when being asked at the end of a session, without the expectation that the questions can adjust the following session.
• Perhaps this is not surprising: Landauer (1995) found it very hard to discover evidence of economic benefits for using computer technology in general.
• he main problem is usually analysed as to do with the lack of interaction and the consequent extreme passivity imposed on the audience
• by giving them much better, faster, and more detailed information on what the learners are thinking now, and where their problems are at each point.
• pedagogic modes
• The exploratory studies should yield practical knowledge such as question banks for the participating disciplines, and how much support is needed for first time use (a new lecturer and students who haven't used the equipment) and for regular use.
• The studies in prospect with this equipment should eventually allow us to pronounce on the validity of the design rationales presented in this paper.

9 more annotations...

• highlighting (for example) the development of models that seek to abstract and share practice.
• A typical pattern of use
• The first review took place in 2002.
• The follow on review took place in 2006
• a consistent terminology has still not emerged
• In review 1, electronic voting systems were used in a variety of contexts.
• The range of disciplines continues to grow and in review 2,
• In review 1, voting systems were aimed primarily at managing interaction with large groups, with class sizes ranging from 50 to 300
• There were some exceptions
• In review 2, while there are mentions of use with smaller groups (Draper & Brown, 2004), most of the focus continues to be on large groups with class sizes ranging from 100 to nearly 400
• the importance of sequencing of activities
• importance of sequencing of activities rather than merely whether to use the system or not
• importance of sequencing of activities rather than merely whether to use the system or not
• There is more discussion about how often a voting system is used within a course
• Nicol and Boyle (2003) explore the merits of different discussion sequences, drawing on earlier teaching approaches, and Cutts, Carbone and van Haaster (2004) explore the relationship between revealing the voting responses and discussion.
• between 40% and 60%
• In both reviews, many papers propose that anonymity encourages student contribution.

•   Whilst voting systems can support teaching and learning within lectures, any benefits will depend on how effectively they are used on each occasion
• The biggest learning gains [...] are likely to come from the much better and quicker feedback from learners to teachers, allowing better atunement of the delivery; and from the method of teaching by questions, i.e. of discussions in class (whether in small groups, plenaries, or a combination) initiated by well designed questions and by getting each individual to start by committing to an initial position. (Draper et al., 2002)
• It is possible that interactive teaching will not automatically result in active cognitive experiences.
• Use of voting systems to stimulate discussion starts to emerge in review one,
• , the relationship of voting systems and discussion is more prevalent in review 2,
• It is important not to lapse into thinking of it as a magic bullet but as more similar to central heating in contrast to individual fires. It is not essential to achieving learning or interactivity, which for millennia have been frequently achieved by other means; but nevertheless it makes the desired effect [...] easier to achieve more often, in more contexts, and with much less effort and attention. (Draper & Brown, 2004: p. 93)
• models of discussion are explored and the merits of each evaluated according to context, most notably by Nicol and Boyle (2003)
• but also in attempts to create representations of practice (such as models) that abstract principles in a form that can be shared and used to improve other peoples' practices.
• whereas disadvantages have changed (Draper & Brown, 2004).
• In review 2, the benefits identified appear to have remained stable.
• It also to be expected that as these systems are used in more contexts, different problems will be identified.
• There is now evidence that students are not universally supportive of such systems, in spite of the many positive class evaluations reported in the literature.
• are "over-used or inappropriately used they could quickly lose impact and lead to uncritical 'button pushing' and student detachment" (2003, p.45).
• Cutts and Kennedy's analysis also "showed that students who used the EVS more often did better in assessments, although level of correctness of responses did not correlate."
• As noted repeatedly in the literature, the potential of these systems depends on the skills and approach taken by teachers.
• However, more recent papers have interpreted this in a more sophisticated and holistic way, looking beyond the assumption that using handsets equates to meaningful interaction and also more broadly than just using the system for one teaching goal, such as 'checking understanding'.
• In both early and recent papers, the motivation to use a voting system effectively arises from the view that active learning and engagement of students is worthwhile.
• More specific and contextualised pedagogical approaches and theories are being used in developing practice; an indication of this increased expertise and the greater reference to pedagogy is the emergence of models that distil this practice and experience

  • this is what I am doing.
• A different view is offered by Draper and Brown (2004). They are also driven by educational goals, but appear to see technology as a definite solution rather than a possible option
• Practice may be becoming more sophisticated but there are still new case studies as people discover these systems for the first time in their own context (e.g. Stuart & Brown, 2004)
• It is worth noting that many of the uses identified can be (and sometimes already are) currently undertaken in lectures using non-technological methods.
• if voting systems have a role in teaching,
• thoughtfully exploring how they can be best used

38 more annotations...

• The class without the voting pads improved their average score by 3.4, from 7.7 to 11.1 out of 20. The class that had been using the voting pads increased their scores from 7.6 to 14.0, with a quite staggering average progress score increase of 6.4.

• A scenario (from Italian, that which is pinned to the scenery) is a synthetic description of an event or series of actions and events.

• These combinations and permutations of fact and related social changes are called "scenarios."

• By sensitive material, do you mean issues that students might not want to discuss in front of others? 
   
   Ross : Right. We can have those conversations without people identifying who they are...
• Most of us don't have the time to learn complicated types of instructional technology. Most of us also don't want to be dependent on ... IT personnel to bail us out every time

• Have student (groups) design EVS questions and present them as part of talks given to the rest of their class. The discussion that ensues during the design of the questions, alternative responses, and justifications to be used when explaining the answers, can be highly generative of learning.

• This is the home page for some web documents about interactive lecture methods in general, and using classroom electronic voting systems (EVS) in particular.

• Evaluating evidence on EVS effectiveness

   There are basically three classes of evidence to consider:
   
  • Improvements in exam marks and/or dropout rates 
  • Improved voluntary attendance by students 
  • Favourable data on attitudes e.g. from questionnaires 

• Exposition is the first; and in considering its wider place, we are concerned with the first 4 activities: not only exposition by the teacher, but re-expression by the learner, and sufficient iteration between the two to achieve convergence of the learner's understanding with the teacher's conception.
• At least in HE, learning is not automatic but on the contrary, highly intentional and is managed by a whole series of decisions and agreements about what will be done. Students are continually deciding how much and what work to do, and learning outcomes depend on this more than on anything else.
• This is probably the point to discuss the painful reality that in many lectures, students do not learn, but only collect material for later learning.
• Another claim about the functions of lectures are that they can or should be demonstrations of thinking in that field
• Finally, a very important claim about the role of lectures is that they sustain community.
• How many questions? How long do they take?
   A rule of thumb for a 50 minute lecture is to use only 3 EVS questions.

  • why?

4 more annotations...

• This paper describes the design rationale for introducing electronic equipment for student interaction in lecture theatres, and the studies now in prospect of the use of this equipment.
• Thus one or several such question items at the start of a class could be used to select a topic for detailed coverage, while the same or similar items at the end could demonstrate to what degree the group now understood the topic.
• e.g. on whether the pace is too fast or too slow, the jokes too numerous or infrequent, the examples too many or few
• For example if each student has to give a verbal presentation and this is peer assessed, then at the end of their talk the teacher can display (say) each of 10 criteria in turn, and get the other students to enter their mark for this anonymously but on the spot, with the totals displayed.
• For example, at a research meeting start by asking people's ages (which illustrates the advantage of anonymity), and the kind of department or institution they come from, and some alternative reasons for attending. At the start of a class, I might ask whether each student is straight from school or not, their gender, which faculty they belong to, whether they signed up for the course because it is their main interest, a side interest, or are just making up the number of courses they do.
• for topics that concern human responses, a very considerable range of experiments can be directly demonstrated using the audience as participants. For instance visual illusions may be displayed and the equipment used to show what degree of uniformity of response is found.
• to initiate a discussion. Here, a carefully chosen MCQ is displayed and the learners register an answer, thus privately committing to a definite opinion.
• Perhaps this is not surprising: Landauer (1995) found it very hard to discover evidence of economic benefits for using computer technology in general.
• Making adjustments to the teaching every 30 minutes
• In a review of a number of applications (Draper 1998), I argued that most applications showed no significant improvements over what they replaced, but that the few striking positive exceptions were characterised by "niche-based design": by a good fit between a particular learning situation and a specific technical solution.
• large classes in large lecture theatres,
• Actually, with highly skilled learners and a teacher reasonably in tune with the group, this can nevertheless take place: for instance, where the learners take notes that are not mere dictation, but substantial re-formulations of what is being talked about.
• However this degree of skilled, silent interaction is not often present in undergraduate teaching, and large numbers usually prevent learners asking sufficient questions to repair the attunement between speaker and audience, from both a pragmatic (there isn't time for many people to ask questions) and a social (it just feels too embarrassing) viewpoint.
• alternatives?
• give a show of hands. This equipment crucially offers more privacy (it's a secret ballot, and important for just the same reasons)
• The electronic version also provides faster and more accurate counting of the results:
• the instructional design mostly isn't in the equipment or software, but in how each teacher uses it.
• On the other hand, note too that this design does not fit with a simplistic interpretation of the slogan "learner-centered".
• how much support is needed for first time use (a new lecturer and students who haven't used the equipment) and for regular use.
• They will also yield evaluation results on what benefits can be demonstrated. We hope to use a version of the method of Integrative Evaluation
• Investigating more deeply than general student preferences will require more, and more sophisticated, measures.
• Some of the most important evaluation issues can be organised around the notion of interactivity.
• In other words, the measure of it would be the number of mental and verbal responses a learner makes (in discussion) rather than the number of button presses on the handset.
• We are writing software to smooth the integration of the equipment with other lecture facilities (e.g. the use of powerpoint presentations), and with keeping records of the interactions.

  • which has been solved by Promethean's Activinspire
• independently to generate an answer
• Currently infrared proves cheapest.
• motivatio
• When the latter becomes widely available, we can at last address a fundamental problem of discussion groups (such as research seminars) where many people want to ask a question: which is the best question to take for the group as a whole?
• With textual group responses, everyone's questions could appear in front of the speaker and/or facilitator, and could then be grouped, sequenced, and sorted by priority.

  • Expression gives the possibility to focus on padagogy rationale, great!
• anonymity

28 more annotations...