The Quality Lecture: How Do We Rate?

710 | P a g e M a y 8 , 2 0 1 3 The Quality Lecture: How Do We Rate? Tzong-Yang Pan, Cathy Owen ANU College of Medicine, Biology & Environment, Australian National Univerisity, Canberra, ACT Australia tzong.yang.pan@gmail.com Rural Clinical School ANU Medical School, ANU College of Medicine, Biology and Environment, Australian National Univerisity, Canberra, ACT Australia cathy.owen@anu.edu.au ABSTRACT


INTRODUCTION
Lectures remain a key, cost effective teaching method employed to some extent by most medical schools. Students who are satisfied with lecture quality achieve better on assessments of knowledge (Cohen 1981; Anderson et al. 1991; Koon and Murray 1995). Lectures are criticised for their passive nature as being ineffective for teaching (Golden 1989; Davis et al. 1995). Factors identified as important in shaping an effective lecture include introduction and summary of talk, demonstration of the topics importance, organized structure and clarity of presentation, enthusiasm, demonstration of knowledge of subject, encouragement of audience interaction, observation and response to understanding of audience, audio-visual aids of good quality, material presented at an appropriate pace and level and engagement of the audience with the material (Gelula 1997;Bligh 1998;Newman, Lown, and Jones 2009;Delaney et al. 2010). This review analyses the quantitative and qualitative evidence supporting each of these factors.
Research suggests an introduction outlines the content, engages the audience and provides a framework for the audience to place their current knowledge in context (Cooper 1982;Smith and Swinney 1992;White 2011). Similarly, the conclusion stimulates thought and recall (Cooper 1982).
Demonstrating the lecture"s content relevance and importance to the audience can help stimulate interest and subsequently gain the attention of the audience, encourage engagement and promote intellectual stimulation (Copeland et al. 1998;Wood 1998). Audiences are enthused to want to learn more about a subject by describing applications related to the audience or providing links back to the audience"s previous experience. This promotes enthusiasm and interest (Cantrell 1971). Studies have shown that material with interesting topics and examples increases intellectual activity and improves learning and recall of the content (Solomon, Rosenberg, and Bezdek 1964;Cantrell 1971;Murray 1983;Fink 1989;Shimoda 1993;Copeland et al. 1998;Wood 1998). Whether the perception of a topic"s relevance and importance affects learning directly by promoting thought, or through augmenting attention, is not well studied.
The structure and clarity of the lecture is central to its quality. A clear, logical structure with one idea leading to the next smoothly can assist in the audience to organize their thoughts and improve factual learning (Solomon, Rosenberg, and Bezdek 1964). Explaining jargon used and elaborating the concepts and ideas with clarity beginning from relatively simple concepts leading up to the more complex helps avoid confusion among the audience (Cantrell 1971). Clarity of lecture delivery increases both student satisfaction and final examination scores (Rosenshine and Furst 1973).
Lecturer enthusiasm can take various forms including the volume and variation in pitch of speech, body language such as gestures, eye contact and the energy of the delivery (Solomon, Rosenberg, and Bezdek 1964;Fink 1989; Newman, Lown, and Jones 2009; Long and Lock 2010). Enthusiasm promotes audience attentiveness, interest and enjoyability of the lecture and is thought to improve comprehensive learning (Solomon, Rosenberg, and Bezdek 1964;Gelula 1997). The perceived effectiveness of teaching has been shown to be closely related to the enthusiasm of the lecturer (Murray 1985;Erdle and Murray 1986). Students give higher ratings to lecturers which deliver lectures more expressively compared to those that do not, and students achieve higher examination scores (Ware and Williams 1975;Abrami, Leventhal, and Perry 1982). Enthusiasm improves audience recall of the content in the short term, measured in post-lecture multiple choice question scores (Coats and Smidchens 1966).
A lecturer"s demonstration of subject expertise increases the quality of a lecture, perhaps increasing intellectually stimulating discussions and promoting thinking in the audience (Newman, Lown, and Jones 2009; Kessler, Dharmapuri, and Marcolini 2011). Students perceive knowledgeable lecturers as important in effective lecturing and frequently indicated that they believe effective lecturers to have in-depth knowledge of the subject which also helps develop rapport (Delaney et al. 2010). Rapport has been identified by trained observers and experts as being important in a lecture possible due to its effects on promoting student engagement and approachability of the lecturer (Murray 1983;Fink 1989;Delaney et al. 2010). Student perceptions also extended to the lecturer"s knowledge of teaching methods and this included the ability to respond to the audience"s needs within the lecture (Delaney et al. 2010). There are opposing studies which suggest that a lecturer"s professional status, age, or qualification do not correlate with lecture quality, suggesting that more senior presenters are not necessarily better at teaching (Cantrell 1971). When both views are taken together, it appears that expertise in the subject does not directly influence teaching or quality of the lecture but rather, provides opportunities to engage the audience, build rapport, stimulate thought and develop interest in the subject.
Active learning techniques in lectures encourage thinking and participation in the audience (Gelula 1997;Long and Lock 2010). The best lectures encourage the audience to think about the questions and problems, and the lecturer guides the audience towards the conclusions (Cooper 1982). The final examination scores of students receiving active teaching methods were greater than that of students taught with passive methods (Yoder and Hochevar 2005). The Audience Response System (ARS) is an electronic device that is able to send individual responses anonymously to a central software, creating interactivity between a presenter and audience. Students rate ARS as helpful, by encouraging participation, discussion, motivation to think and engagement with the material (Gauci et al. 2009; FitzPatrick, Finn, and Campisi 2011). Students who answered the ARS questions scored better than those that did not in the final examination (Gauci 2009). In contrast, another study which looked at the use of active learning strategies with or without ARS showed that there was no difference between groups in student examination marks (Stoddard and Piquette 2010).
A good lecturer can respond to the audience by adjusting pace and offering alternative explanations if required (Cantrell 1971; Newman, Lown, and Jones 2009). However the direct influence of flexibility and responsiveness on lecture quality has not been studied in detail and there is limited support for this criterion. Experts view this as an important aspect of lecture delivery (Newman, Lown, and Jones 2009). The audience also value methods used to check their level of understanding during the lecture, such as use of the ARS system or polling the audience (FitzPatrick, Finn, and Campisi 2011). M a y 8 , 2 0 1 3 Delivery of content via visual aids such as diagrams and flow charts present this material in an additional form to the audience. Visual aids can help the audience organize their thoughts and provide a graphical representation of the content. It has been shown that material organised graphically assist students in learning hierarchical and coordinate relations; this was reflected in their ability to apply and integrate the knowledge in subsequent tests (Robinson and Kiewra 1995). There are two important characteristics of audio-visual aids: they must be able to be seen and heard and the content should be helpful in consolidating information or assist in the audience"s understanding (Smith and Swinney 1992; Newman, Lown, and Jones 2009).
The content in a lecture should be balanced as too much material or an excessively fast pace may not provide opportunities for the audience to think about the content and understand it, leading to confusion (Rosenshine and Furst 1971;Gelula 1997;Bligh 1998;Long and Lock 2010). Whilst too little material or an overly slow pace may not utilize allocated teaching time effectively. The most accurate determinant of whether the difficulty and pace is appropriate is the audience"s perspective and response. Therefore the most reliable ratings of this must originate from the audience"s opinion (Newman, Lown, and Jones 2009).
The audience may take a greater interest in a thought provoking lecture and see the relevance of the lecture to them at a personal level (Gelula 1997;Long and Lock 2010). Intellectual stimulation may increase active learning or extend prior knowledge (Fink 1989). Deep engagement and intellectual processing of material increased short-term and long-term conceptual understanding (Sokoloff and Thornton 1997). Copeland et al (Copeland 2000) found that the strongest predictors of overall lecture quality score were engaging the audience, lecture clarity and using case based material.
These characteristics are thought to be an important part of a quality lecture and there is support for each criterion by a range of perspectives including exemplary lecturers, experts, trained observers and students. (Solomon 1966;Cantrell 1971;Murray 1983;Fink 1989;Copeland et al. 1998Copeland et al. , 2000; Banwet and Biplab 2003; Newman, Lown, and Jones 2009; Kessler, Dharmapuri, and Marcolini 2011).The use of lectures as a main medium to deliver information to a large audience will continue to be used. By having identified various aspects of a lecture which assist in the learning process and recognising that lecturers can change following training (Fink 1989), it suggests that improvements in tertiary education can be achieved by changes in the delivery method of lectures.
This research draws together these recognised hallmarks of better lectures to establish a quality baseline for a medical school. The next step is to assess lectures according to these criteria and identify areas requiring improvement and criteria most strongly associated with overall lecture ratings. The identification of these criteria will then provide direction for improvements to the lecture component of the medical program.

METHOD
The study took place in a graduate entry four-year program using problem based learning, small group clinical skill development and large group lecture programs as key educational methods. Ethics approval was obtained.
Author TYP (a medical student) attended 148 out of 171 lectures presented during the study period. All lectures were allotted 55 minutes duration for delivery with the exception of nine lectures: two lasted 30 minutes and seven were twohours. During the assessment period, the course covered a range of disciplines including medical science, population health, ethics, and social philosophy disciplines.
The delivery of each lecture was scored by TYP on a scale of 5 (1 excellent, 3 adequate, 5 non-performance) using the Instrument for Peer Assessment of Medical Lecturing with permission (Newman, Lown, and Jones 2009). The instrument was modified to include 3 additional criteria identified in the literature review above as supporting effective learning. These 3 criteria included the difficulty of material presented, pace of presentation and encouragement of deep engagement with material. A total of 14 criteria for effective lecturing were assessed along with an overall score (see appendix 1).

RESULTS
The results from 148 lectures are shown in the following tables. 109 were delivered in a series by regular lecturers. 39 were delivered by guests as occasional, often single lectures. As the research intention was to improve the overall lecture program (rather than individual lecturer skills) ratings were deidentified and demographic data was not collected with individual lecture scores. M a y 8 , 2 0 1 3

1: Clearly states goals of the talk
Most lecturers scored a rating of adequate or better n=87 (58.8%). Scoring for this criterion was based on the structure, clarity and detail of the overview. The use of specific learning goals were generally required to achieve a high rating. Lectures delivered as part of a long series by an individual lecturer typically scored better in this criterion than lectures delivered as part of a short series or a single presentation.

2: Communicates or demonstrates importance of the lecture's topic(s)
This was largely achieved with 134 (90.5%) scoring adequate or better. Lectures that had clinically focused topics were more likely to score well for this criterion, likely due to the immediate relevance to the students" clinical knowledge and practice. Basic science lectures which scored well usually included, case scenarios, examples and other explanations integrated throughout the presentation which placed the content in a clinical context, more relevant to students. Lectures which scored poorly tended to be basic science lectures which referenced scientific experiments and other detailed research with little immediate relevance to students.

3: Presents material in a clear, organized fashion
Most lecturers presented with clarity as 141 (95.3%) scored a rating of adequate or better. Scoring for this criterion was based on the flow of the lecture which can be observed from the overall coherence of the framework. The use of outlines and quick reviews of previous content to help establish links to new material help produce smooth transitions between subtopics within and across lectures. Lectures which were delivered as part of a longer series by an individual lecturer tended to score better than lectures that were delivered as an individual presentation or part of a short series. M a y 8 , 2 0 1 3

4: Shows enthusiasm for topic
Half the lectures assessed rated as excellent on enthusiasm for the topic. Assessment of this criterion was based on the lecturer"s display of enthusiasm through body language, expression of voice, facial expression and the overall liveliness conveyed. Lower scores were given when the lecturer appeared tired during the presentation.

5: Demonstrates command of the subject matter
Most lecturers demonstrated command of their subject with 90 (60.8%) scoring the maximum rating and 145 (98.0%) scoring a rating of adequate or better. Methods such as providing succinct and informative answers to questions, citing literature, promoting discussion about current controversies in the field and building upon the content within slides by using examples all contributed positively to this criterion.

6: Explains and summarizes key concepts
Most scored well on this criterion with 145 (98.0%) as adequate or better. The use of short summaries throughout the lecture and making connections between subtopics within or across related lectures led to better scores. Other techniques included the use of analogies, providing examples, summarising in graphic formats and taking a step-by-step approach to explaining material when appropriate.

7: Encourages appropriate audience interaction
This was largely well done with 135 (91.2%) rating equal to or above adequate. Maintenance of eye contact, asking questions, initiating discussion and appropriately managing its flow were all approaches which contributed positively to this element. Lectures on basic and clinical sciences usually had minimal encouragement to interact. In contrast, lectures with social and professional aspects usually had excellent discussions, which is highly appropriate for the topic. Lectures which had poorer ratings in this criterion tended to be because of very limited eye contact with the audience as a result of reading off the slides or paper.

8: Monitors audience's understanding of material and responds accordingly
Performance was poor by comparison in this criterion with a mean rating of 2.69, σ = 1.428. Although many scored well, 22 (14.9%) did not demonstrate this strategy at all. This is of particular concern as it recognized that effective learning requires understanding of the material. Scoring was based on the methods used to monitor student understanding and more importantly, the degree of adaptation of the presentation to suit the learner"s needs. The audience"s understanding of material can be assessed by asking related questions following the delivery of content or inquiring if the content is clear. The lecture can be adapted to the student"s response by adjusting the pace of delivery, paraphrasing or repeating key sections. Poorly rating lectures were delivered as a continuous talk without opportunities for interaction and to ask questions to ensure that the content is clear. Lectures which scored well in this criterion usually had a structured format with a number of distinct sections, each of which ends with an opportunity for students to ask questions or seek further clarification.

9: Audio and/or visual aids reinforce the content effectively
The audio and visual aids utilized in each lecture were rated on its effectiveness in reinforcing lecture content. With the exception of one lecture which used video as the only aid, all presentations were given in conjunction with PowerPoint slides. The aids utilized in many lectures were usually effective in reinforcing the presentation. Ratings were determined by how the audio-visual aids used stimulated thought and recall. Presentations with slides which contained only text or only graphics tended to score poorly in this criterion. In contrast, slides which contained a mixture of text and graphics were usually rated highly in this criterion as it complemented the verbal delivery well. These aids are also useful for later revision.

10: Voice is clear and audio-visuals are audible/legible
This criterion rated the clarity and audibility of the speaker"s voice along with that of the audio-visual aids utilized. 113 (76.4%) scored the highest rating with no lectures being rated as "does not demonstrate". This is the best performing criterion among all fourteen criteria that were assessed. Lecturers infrequently asked the audience whether themselves and the audio-visual aids used were audible and clear. Despite this lack of confirmation, almost every lecture presented was audible and clear to the audience.

11: Provides a conclusion to the talk
63 (42.6%) lecturers failed to provide a conclusion. Better scores were when the conclusion captured the key concepts in the content and provided an opportunity for students to ask any final questions. Lecturers which presented well-structured M a y 8 , 2 0 1 3 and succinct conclusions usually did so for each of their presentations; typically with a slide listing the key points or outcomes.

12: Presents material at an appropriate level to the audience.
Most presenters pitched correctly with 142 (95.9%) scoring an adequate or above. This criterion would be difficult to rate by peer assessment, but a student assessor enabled rating from the learner"s perspective. Scoring was based on whether the student assessor thought the content presented was of an appropriate level for the audience, based on previous lectures given. Lectures delivered sequentially in a discipline usually by an individual scored higher with one lecture building on the previous. However, individual lectures at times were both too hard and too easy in those who scored poorly.

13: Presents material at an appropriate pace
Most lecturers paced well with 132 (89.2%) scoring equal to or above adequate. Pacing was assessed based on the actual duration of presentation in relation to the volume of content delivered. Variation in pace across subtopics within a lecture based on its breadth and difficulty also contributed towards a better score. The most common reason for scoring poorly in this criterion was running short of time towards the end of the lecture, which pressures the lecturer to hasten their presentation and may skip sections as a result.

14: Encourages deep engagement with material
Four lectures had scored the lowest rating in this criterion which is of concern because engagement with the material is a key factor to successful learning. Scores were based on the opportunities given to learners to relate to the learning material. The use of case studies, eyewitness accounts and other forms of anecdotal illustration that encourage the students to reflect on the content contributed favourably to the rating in this criterion. The most frequently used method to engage the students was the use of case studies or clinical scenarios. This method was felt to be highly effective as it puts the lecture content presented in an auditorium into a setting where the knowledge is clearly relevant and applicable.

Overall impressions
The ratings for the fourteen criteria allowed the identification of the three best and worst performing criteria. The three best were explaining and summarising key concepts, presentation of material at an appropriate level to the audience, and the use of clear audio-visual aids and voice. The three worst were stating goals of the talk, monitoring audience understanding and responding appropriately, and providing a conclusion to the talk.
An overall rating for each lecture was given, independent from each criterion. This rating was based on the general perception of the student assessor towards the lecture. The mean rating was 2.07, σ = 0.881, with 41 (27.7%) scoring the highest rating and 139 (93.9%) scoring equal to or above adequate. The remaining 9 (6.1%) lectures rated below adequate with one rating the lowest possible score. Although a small number, this reflects that improvements can be made. A correlation analysis was conducted on these data and it was established that a number of criterion had the strongest correlation with the overall rating. These included enthusiasm, command of topic, audience interaction, monitoring audience understanding and engagement, each with a Pearson"s correlation coefficient of above 0.5. These results share some similarities with earlier work (Copeland 2000) who identified engagement strategies, clarity and using a case based format as their key predictors of lecture quality.
There are limitations in this study that need to be considered. Primarily, only one student assessor was involved in the assessment process and there are advantages and disadvantages to this. The advantages are readily obtaining a snapshot of a lecture program prior to any intervention to improve lecturing. This is a low cost low fuss method that has built on criteria supported by educational literature. The key disadvantage is the potential for bias in rating performance over time and from demographic factors. The assessor has a strong background in medical sciences from previous education and this may have affected the perception of difficulty level in certain lectures. Bias may be present when rating certain lectures depending on the style of delivery and the assessor"s subjective preferences. The source of these biases have been minimized by rating the lectures based on objective targets in the assessment tool, and avoiding ratings based on the subjective impression of lecture.

CONCLUSION
Improvements to the quality of education delivered to students is a goal tertiary institutes constantly strive to achieve. With these data collected using a student assessor, an overview of the current quality of lectures has been obtained and the performance in each criterion has been identified. By using these data, areas requiring improvement can be investigated with possible implementation of interventions. Further research will examine the effect of sharing these data with lecturers, especially targeting areas most desirable for improvement. There are positive aspects of the lecture program recognized M a y 8 , 2 0 1 3 by using this rating tool that should be an encouragement to lectures. There are however, key areas for improvement ripe for intervention and follow-up research. Clearly states goals of the talk During introduction, communicates purpose of the presentation. For example, may provide an overview of content, state expected learning outcomes, pose rhetorical/challenging questions to be answered.

ACKNOWLEDGMENTS
States the goals, but description is limited in scope (e.g., only states topics to be covered or provides the format of talk).
Does not provide overview or communicate the goals of talk. 2 Communicates or demonstrates importance of the lecture"s topic(s) Clearly explains the topics" and subtopics" relevance, context, applicability, and/or the significance to the audience (e.g., presents compelling information, case, or data; uses a "hook").
States the importance of topic, but provides limited description of why learners need to know the material.
Does not communicate or describe why the topic is of importance.

Presents material in a clear, organized fashion
Uses an explicit, organized framework so that the presentation flows logically (e.g., articulates a structure and sequence to the talk, frames subtopics, links concepts).
Presentation has some organization, but limited in structure, linkage, and/or sequence.
Does not present material in a clear, organized fashion.

4
Shows enthusiasm for topic Demonstrates keen enthusiasm for topic through voice, eye contact, energy, movement and/or body language (e.g., varies pitch, inflection, tempo, and volume; gestures to emphasize importance).
Shows some enthusiasm for topic, but limited in display.
Does not show enthusiasm for the topic.

5
Demonstrates command of the subject matter Demonstrates strong understanding of subject matter (e.g., cites the literature, refers to overarching subject area, draws upon personal experiences, speaks to advances or current controversies in the field, provides informative answers).
Demonstrates some command of subject, but breadth of understanding is limited (e.g., unable to elaborate with greater detail or information).
Does not demonstrate a command of subject matter. Encourages some interaction or uses less effective strategies (e.g., asks close-ended questions, offers little wait time, often turns back to audience, and reads from slides).
Does not engage or encourage interaction (e.g., reads all slides without looking at audience; defers questions, yet does not answer them).

8
Monitors audience"s understanding of material and responds accordingly At appropriate intervals assesses and responds to audience"s understanding of material (e.g., asks probing questions or polls audience; asks if material is clear, then tailors response by rephrasing or providing alternative examples; adjusts the pace of lecture to accommodate learners).
Pays some attention to the audience"s understanding of topic, but tailoring of response is limited.
Does not pay attention to the audience"s understanding of material, or checks in but doesn"t respond accordingly.

9
Audio and/or visual aids reinforce the content effectively Appropriately chooses and designs instructional material to emphasize key points, demonstrate relevance of material, or stimulate thought.
Some of the audio and/or visual aids reinforce content, or material is less than effective (e.g., slides are cluttered).
Audio and/or visual aids do not reinforce content.

10
Voice is clear and audiovisuals are audible/legible Sensitive to the setting and tailors audio and visual aids so all can see and hear (e.g., checks if audience can hear/see material; talks to audience not to blackboard, laptop, or screen; visual material is well organized, text is legible, and graphics are clear).
At times voice is unclear or audiovisuals are inaudible/illegible. Voice is unclear and audiovisuals are inaudible/ illegible.

11
Provides a conclusion to the talk Concludes presentation by synthesizing information, summarizing main points, and inviting/responding to questions (e.g., repeats or rephrases questions as needed). Open to hearing learners" perspectives/opinions. Provides synthesis and/or summary of talk, but limited in scope. Invites few questions and/or provides limited or ambiguous responses.
Does not synthesize or summarize information.
12 Presents material at an appropriate level to the audience Presents material at an introductory level for unfamiliar subjects, or at a level above but clearly builds upon existing previous knowledge of the learners.
Presents material well above an introductory level for unfamiliar subjects or at a level well above current understanding of learners, but learners are still able to cope with some of the content.
Material presented is at a level well beyond the learners" grasp and/or generates confusion or misunderstanding in learners.

13
Presents material at an appropriate pace Paces lecture content appropriately and completes within the allocated time. Time spent on each subtopic is relative to the content"s breadth and difficulty.
Majority of the lecture is paced appropriately with few sections being covered briefer or faster than intended. Lecture is not completed prematurely or with greater than a five minutes extension.
No apparent pace to presentation of material and/or skips entire sections. Alternatively, the lecture finishes much too early or runs well past the allocated time and/or end before all content is covered.
14 Encourages deep engagement with material Provides opportunities for learners to intellectually and/or emotionally relate to the learning material. (e.g., uses personalized case scenarios or eyewitness accounts to encourage significant reflection on content).
Provides limited opportunities for learners to relate to teaching material.
Does not provide means of engaging learners to learning material.
Overall, how would you rate this lecture (please circle):

Excellent Very Good Good Fair Poor
Comments: