(Our Intranet is a complementary resource)
Last updated: September 9, 2014
Created and maintained by: Jennifer Murdock
Academic Handbook for Instructors (updated October 2013, pdf)
Academic Handbook for Instructors (updated October 2013, html)
Sessional Dates (St. George)
Sessional Dates (UTM)
Course Information (Calendar, sessional dates, etc) (St. George)
Teaching Resources at A & S (St. George)
Sessional Dates (St. George)
Sessional Dates (UTM)
Times and locations for final examinations (St. George)
· You can view the course outlines of your colleagues for more samples by visiting their course websites listed in the undergraduate course time tables St. George or Mississauga or by finding a link to it through the faculty member's own homepage
· ***Your course outline must be uploaded to our departmental website: http://www.economics.utoronto.ca/index.php/index/teaching/coursesForUser***
· The syllabus and the first day of class are your chance to communicate your expectations to your students. Making your expectations and what students should expect as transparent as possible right from the start will ensure a smoother course and greater student satisfaction with your course.
· You may require students who miss an assessment due to illness to provide a U of T Verification of Student Illness or Injury Form (pdf version) or other documentation.
Teaching Workshops at the Center for Teaching Support & Innovation (CTSI) (physically on St. George campus, but all are welcome)
Academic Skills Centre (Mississauga)
· CTSI gives seminars and courses to help you in your professional development as a university teacher. Visit their web site to sign up. Members of the Economics Department have attended numerous sessions and found them to be useful and informative.
· You may choose to include a mid-course review of your own design. Students typically given more substantive and constructive comments on these than the A&S end-of-term evaluations. CTSI has a booklet entitled "Gathering Formative Feedback with Mid-Course Evaluations: A Guide for Faculty" that you can request (e-mail them) or you can borrow a copy from me. SurveyMonkey is a useful, easy, quick, and free tool that lets you create a professional looking online survey and easy-to-review results.
· Subscribe to the CTSI listserv to ensure you receive timely information about upcoming events, workshops, lunch-time roundtable discussions, etc. To subscribe, you may send an e-mail to CTSI asking them to add you.
· You may consider joining (subscribing) to the Society for Teaching and Learning in Higher Education (STLHE), which is a Canadian association dedicated to the advancement of teaching and learning, or subscribing to a teaching newsletter such as The Teaching Professor, which will help you keep up-to-date with developments in teaching and learning and keep your courses on the cutting edge without taking up much time
· You may periodically check for interesting articles (full text, searchable, pdf) in The Journal of Economic Education. From a U of T computer you can get full electronic access directly at http://www.tandfonline.com/toc/vece20/current.
Teaching Evaluation for Promotion and Tenure (Provost's Guidelines; general for U of T)
A&S Guidelines for the Assessment of Effectiveness of Teaching in Tenure and Promotion Decisions (Research stream; St. George Campus)
A&S Guidelines for Evaluation of Teaching Activities and Pedagogical/Professional Development (Teaching stream; St. George Campus)
Copy of blank student survey form (used
until 2012/13) (St. George Campus)
ASSU Anti-Calendar (St. George Campus) (published summaries of faculty members' student survey results)
Tips & Templates from OSAI NEW (as of September 2014) and nicely organized
Academic Handbook for Instructors (See Section 12)
· Dr. Kristi Gourlay is the Manager of the Office of Student Academic Integrity (OSAI). She is extremely willing to help. She strongly encourages faculty to contact her with any concerns about academic integrity such as inappropriate use of medical notes, plagiarism, cheating or other such problems. OSAI also distributes a newsletter for faculty. It is not available online, but you can ask Kristi to e-mail a copy if you cannot locate yours.
University of Toronto Libraries Exams and Course Collections (St. George campus)
See Using multiple choice questions on this website
· Scheduling assessments during the term and obtaining appropriate testing rooms can be challenging. To avoid dealing with student conflicts with other courses you can try to schedule assessments during lecture time. You can also try to minimize student conflicts by avoiding dates with religious observances.
· Typically your regular lecture room is not appropriate for testing. To request a room you may either: (1) ask Deborah Navarro to book a room for you (make sure to specify the course, sections, number of students taking the test, date, and time) or (2) make a booking request directly to the Academic and Campus Events (ACE, formerly OSM). Do this early (before classes start) to ensure that it is possible for you to get an appropriate room at the requested time: sometimes the University will not accommodate you. You will receive a confirmation e-mail: bring a copy to the examination room as proof that you have reserved the room for that time if there is a conflict. If you have 50-minute-long lectures holding assessments during lecture time can be limiting. For year-long courses (ECO###Y) you can request time for a midterm test (2 or 3 hours) during the December examination period by contacting Undergraduate Administrator Robbie Innes early in the term and the University will schedule your exam and assign a room (you lose control of the time and date).
· University sets the times and locations for all final examinations with the schedule published a couple of months in advance.
· Final examinations from previous years are publicly available from the University of Toronto Libraries Exams and Course Collections by entering the course identifier (ex. ECO200) under "ERes QuickSearch:". If you wish not to have your examination made publicly available see Section 9.4 in the Academic Handbook for Instructors.
Writing at the University of Toronto (an excellent and comprehensive website for faculty and students)
· See the A &S Academic Handbook for important grading information including the U of T guidelines on mark distributions.
· Given that marks can be curved up but not curved down, there appears an incentive to write assessments that are difficult or time-pressured to ensure that marks do not exceed the U of T guidelines and then to retroactively curve the marks up as need be. This is highly unpopular with students. Even if after the curve the marks are typical, students seem to never forget (or forgive you for) the initially low un-curved marks. While there are several possible explanations, one is that students believe that the assessment reflects your expectations of them and if they do not do well on the assessment they feel they have fallen short of expectations possibly despite extensive studying and preparation. The ex-post curve does not change this feeling. Many students will blame you for this failure because you have not effectively communicated your expectations to them or are perceived as having expectations that are impossible for them to meet.
· Term marks (not inclusive of the final examination) should give students a good idea of how they are doing in your course. This means that you should strive to follow the guidelines during the term. It is not fair to students to give easy term work that is generously graded and then hit them with a much more difficult final examination that differs from their expectations based on the term work.
· Students' official transcripts indicate not only the student's grade but also the overall course average.
For multiple choice tests the Economics Department recommends using a bubble form and having the papers machine marked. The TA Scanning Coordinator (Adam Lavecchia firstname.lastname@example.org in 2014/15) in the Department of Economics is responsible for running the scan. This document describes all the steps you need to take with respect to a multiple-choice test.
The TA Scanning Coordinator will scan the papers for you and e-mail a file containing the raw scanned answers. Your TA can run the marking software below. I would recommend having your TA visit this web site (instructions for your TA (txt)). You will need to provide your TA with a solution key and a spreadsheet with the list of students registered in your class and the TA Scanning Coordinator will e-mail the raw data with scanned answers. Your TA can set up and execute the code. Your TA needs to have access to Stata version 10 or higher to do this, which is available at computing labs around campus. Your TA can give you the automatically produced report summarizing student performance, a spreadsheet with the marks, and a file that you can post to your course web site so students can see their answers and their marks.
Marking Software: version (April 2014)
I have written software to grade and analyze student performance on multiple choice questions. This description explains what it can do.
Software to mark multiple choice question data (STATA do-file) version (April 2014)
Raw Data from scanning machine (spreadsheet) (note: names and student numbers are fake)
Course List from ROSI (spreadsheet) (note: names and student numbers are fake)
Report on student performance on Term Test #1 (text
This illustrative example uses real data (but with fake student names and student numbers). For each question, the report on student performance (sample report) indicates the percent of students that chose the correct answer, the percent that picked each of the wrong answers, the percent that chose the correct answer broken down by quartiles (first quartile, second quartile, third quartile and fourth quartile based on overall performance on the multiple choice questions), the Discrimination Index (DI), R-squared (r2) and the slope. The DI, r2 and slope are all statistics that can be used to assess the effectiveness of each of your questions.
· The DI is the difference in the percent correct for the top and bottom quartiles of the class. One measure of the quality of a particular multiple choice question is the DI: the higher the DI the better the question is at separating proficient students from incompetent students. If a question is too easy the DI will be small: proficient students and incompetent students get it right. If the answer is obvious even if you don't understand the concept the DI will be small: proficient students and incompetent students get it right. If the question is way too hard the DI will be small: proficient students and incompetent students get it wrong. If students who understand an important concept get the question right but others get it wrong then the DI will tend to be large: proficient students get it right and incompetent students get it wrong. If you write a really poor or misleading question it is possible that you get a negative DI index: proficient students are misled while incompetent students guess and some get it right. In general, the larger the DI the better.
· The R-squared (r2) uses the information in the data more efficiently than the DI index, which makes it a better measure. The R-squared for a question tells you what percent of the variation in the students' overall percentage correct on all other questions is explained by their performance of this question. A relatively high R-squared indicates that the question is a good predictor of the students’ performance on the other questions. This is one sign of a good question. A bad question that confuses students who know the material would be expected to have a low R-squared. A hard question where most students guess would again have a low R-squared.
· The slope for a question measures the how much higher on average the percentage correct on all other questions is for students that got this question correct. A high slope is an indicator of a good question, whereas a low slope is an indicator of a bad question.
· If you see a relatively low R-squared and a relatively large slope that is a sign that the question is relatively easy. Hence most students got it, but those who missed it really didn't know the material and also did very poorly on other questions. It is a good idea to try to include some questions like these to separate the incompetent students (F's) from the borderline competent students (D's and low C's).
· If you see a question with a low R-squared and a low slope, this is a sign that the question might have problems. It means that this question is a poor predictor of students' success on other questions and students who got this question right didn't tend to fare better than average on other questions. This could be consistent with: an overly hard question where all students (even top students) are guessing, confusing wording that leads proficient students astray, or an overly obvious question where all students (even incompetent students) can easily spot the right answer. You should review the specific question you wrote and also look at which distracters students selected.
By carefully reading the report, you can study exactly what happened on each question and potentially identify questions that did not work well. For well-written questions you can identify common misconceptions amongst the students if you see numerous students selecting the same wrong answer.
Pros and cons of using multiple choice questions in your assessments
· You control marking
· Ex post you can decide to accept alternate answers
· Marking is objective
· Marking is completely consistent across students
· Marking is done by machine so that even for hundreds of students it takes only a few hours total
· Students can get quick feedback: each student can have their marked paper "returned electronically" within hours of the assessment (responses and marks posted on website)
· Allows for more frequent testing: for example, with quizzes
· Potentially frees some TA hours for more student contact
· Gives very detailed feedback on students’ performance on each question
· You "set the agenda" for the assessment so students cannot avoid the questions (i.e. give long and irrelevant answers)
· No opportunity for ex post cheating (i.e. not possible to modify the test paper and then submit it for a remark)
· It is time consuming to write multiple choice questions
· Likely not a useful tool if you have few enough students such that you can mark most assessments yourself
· Poorly written questions lead to extremely frustrated students: see basic tips for writing multiple choice questions
· Poorly written questions can "give away" answer: see basic tips for writing multiple choice questions
· Multiple choice questions are not suitable for all of the subject matter taught in our department
· Multiple choice questions do not give students a chance to improve their writing skills
· Guessing can impact students marks and create spurious variation across students
· Some students complain that there is no partial credit (even though that's not true in expectation)
· Because forms are machine read, students that fail to follow instructions could face catastrophic consequences (failure)
U of T has adopted Blackboard as the university-wide learning management system (LMS) that typically involves a web presence for courses, downloadable handouts, course announcements, online discussion boards, online tutorial sign-up, student e-mail lists, online marks posting, online student surveys, and sometimes online testing and other advanced features. Blackboard training is available through CTSI. You may log in to the Blackboard Portal using your UTORid and password. If you do not have a UTORid or forgot your password go to UTORid Management.
MAKING YOUR OWN COURSE WEBSITE:
Instructors that choose to may continue to use self-maintained web pages (i.e. through the Economics Department server or through CHASS) and possibly in conjunction with Blackboard. For example, Martin Osborne uses the Economics Department server (http://www.economics.utoronto.ca/osborne/2030/index.html) and Matthew Turner uses the CHASS server (http://homes.chass.utoronto.ca/~mturner/ec2020/eco2020_main.htm). (Or you could develop your own learning management system from scratch: http://mjo.osborne.economics.utoronto.ca/index.php/course/index/1.)
Graduate Student Directory (for e-mail addresses/telephone numbers)
· TA assignments usually happen around the time that classes begin and sometimes a bit after they have started.
· At the beginning of the course you are required to tell your TAs the breakdown by task of the hours they will spend on the course. Martin Osborne has created a web-based form you need to fill out for each TA. Please log-in to our Economics Department Intranet.
· The Economics Department expects you to evaluate each of your TA's at the conclusion of your course: you will receive an e-mail with links from the Economics Department Associate Chair, Graduate Affairs near the end of an academic session.
· Duties typically assigned to TAs include: marking, conducting review sessions, holding office hours, and invigilation. It is often advisable to have TAs attend your lectures and especially for those TAs that will have student contact through review sessions or office hours. Even for 100 and 200 level courses, the assumption that the TA is sufficiently familiar with the course material to teach it is often a poor one. Further, TAs that are not in lecture cannot be expected to know what you covered each week and what was emphasized or possibly how you taught it (i.e. with graphs or with or without calculus). Top research universities, such as Yale, require their TAs to attend all lectures including 100-level courses and courses the TA has done before. While we do not have an unlimited number of TA hours, for large enrolment courses we do have enough to require lecture attendance of at least some of the TAs assigned to the course. For a half-year course, class attendance uses 24 hours total and for a full-year course, 48 hours total.
· If your course has a tutorial hour (third hour per week), you can use this for weekly TA sessions. This is convenient as students should not have conflicts during this time and you will automatically be assigned a classroom. If you have a course that meets only two hours per week, you can request a third hour for next year by contacting the Economics Department Associate Chair, Undergraduate Affairs Dwayne Benjamin.
· For weekly TA review sessions (outside of the third hour of the course) and special review sessions before tests and exams you should request a room. To request a room you may either: (1) ask Deborah Navarro to book a room for you (make sure to specify the course, sections, number of students taking the test, date, and time) or (2) make a booking request directly to ACE: Academic and Campus Events (formerly OSM).
· For TA office hours or office hours for part-time instructors that do not have an assigned office, you can e-mail email@example.com to reserve time in the Economics Department. You can check the availability of our rooms. TA's need to pick up the key from Nada during regular business hours. Alternatively, TA's can meet with students in common areas (such as the Undergraduate Student Lounge on the main floor of the Economics Department, which is also open during regular business hours).
· For accurate marking, it is strongly recommended that you provide TAs with complete solutions (including alternate solutions that would also be correct). In addition, you may provide them with marking guidelines and a marking rubric.
ASSU Anti-Calendar (St. George Campus)
· Students are required to have and check a University of Toronto e-mail address.
· While it is not required that you communicate with undergraduate students via e-mail, you are required to clearly state an e-mail policy and to be otherwise accessible to your students.
· The Department expects you to have at least one hour of office hours per week per undergraduate course during the time when you are teaching.
· You can request a spreadsheet (Excel) that contains the contact information, including telephone number, e-mail address, and postal address, for all students registered in your courses. At the St. George Campus this request should be directed to the Undergraduate/Graduate Office Assistant Deborah Navarro. The names and e-mail addresses can be imported into a mailing list (see help for your e-mail system for specifics). Alternatively, you can download your class list from Blackboard.
university teaching INNitiatives:
When used properly, PowerPoint (or other slide producing software) can be an effective tool in some branches of economics. PowerPoint allows you to include more graphs, tables, game trees, flow charts, data, computer output, etc. Each instructor should choose a lecture style that works best and PowerPoint is not for every instructor or every course. Below is some useful information if you decide to use PowerPoint (or other software such as LaTex) to give lectures at U of T.
Logistics of creating PowerPoint slides and publishing slides:
· If you publish your slides (i.e. post your lecture notes online for your students), you also need Adobe to convert the PowerPoint slides to pdf documents. You can purchase Adobe Acrobat Professional 11.0 at the Licensed Software Office at Robarts Library.
· When converting PowerPoint slides to pdf documents using Adobe, it is important that you embed the fonts. Failure to embed the fonts will mean that some students will not be able to properly view and print out your lecture notes. This is also useful if you are converting other documents you want others to be able to reliably read and print. For Windows users, you can set embedding the fonts (recommended) to the default: (1) Install Adobe (reboot), (2) Click on start (if have option, choose Settings), (3) Select "Printers and Faxes", (4) Right click on "Adobe PDF" , (5) Select "Properties", (6) Click button "Printing Preferences...", (7) Select the "Layout" tab, (8) Click button "Advanced...", (9) Under "Graphic", "True Type Font:" click on what is shown (probably "Substitute with Device Font") select "Download as Softfont" from the drop down menu that appears, (10) Click button "OK", (11) Select the "Adobe PDF Settings" tab, (12) Under "Adobe PDF Conversion Settings", "Default Settings:" choose "High Quality" from the drop down menu, (13) UN-check the box "Rely on system fonts only; do not use document fonts" ("Do not send fonts to "Adobe PDF""), (14) Click button "Apply", (15) Click button "OK". If any of this is confusing, you can see pages 4 - 6, How to PDF document that illustrates most of these steps (in some cases the instructions are slightly different, follow those given on this web site). You only need to do this once and not every time you convert a document.
· To convert a PowerPoint file to pdf: (1) Open the PowerPoint file you wish to convert, (2) Click on "File", (3) Select "Print" from the drop down menu, (4) Under "Name:" select "Adobe PDF" from the drop down menu, (5) If you want to double-check your settings you can click button "Properties" and you should see the default settings you set above (i.e. Layout, Advanced, Graphic, Download as Softfont; Adobe PDF Settings, Default Settings, High Quality; Adobe PDF Settings, Do not send fonts to "Adobe PDF" is unchecked), (6) Do NOT check "Print to file" box (even though you will be creating a file and not a hardcopy), (7) Under "Print what:" choose desired format (i.e. one slide per page, six slides per page, slides with lines for notes, etc.), (8) Click button "OK", (9) In Adobe, review the pdf file you have created and if you need to rotate the image: select Document, Pages, Rotate, OK, and then if it looks right, File, Save, (10) In Adobe, you can check whether you have successfully embedded the fonts by selecting File, Document Properties..., Fonts, and making sure in parentheses you see (Embedded) or (Embedded Subset) next to each font: for illustration see page 9, How to PDF document.
Logistics of delivering lectures using a data projector (at St. George campus):
· For most classrooms you need to bring your own laptop to class with you. However, if your class meets in a large lecture hall it might have a U of T teaching station. If there is a teaching station, you can just plug a memory stick into one of the USB ports. To access the teaching station you need to be U of T staff (faculty, sessional or TA) with a valid UTORid and password.
· To give PowerPoint presentations you will need to request the use of a data projector in your classroom. The data projector will either be permanently installed in the classroom (an "electronic classroom") or it will be wheeled in on a cart prior to each of your lectures ("portable data projector"). If you will be using PowerPoint for only some of your lectures, a regular classroom with a portable data projector will suffice. However, it you plan to use PowerPoint for most or all of your lectures I strongly recommend that you request an electronic classroom. You should make this request as early as possible to Undergraduate Administrator Robbie Innes. Planning for classroom assignments begins in December for the upcoming academic year. Further, if you are teaching more than one section back-to-back you should request that you be assigned the same classroom for those consecutive sections as you need time to set up your laptop and the data projector. The Department can then try to get your requests filled with the Office of Space Management (OSM). There is sometimes a shortage of electronic classrooms, so as a courtesy to your colleagues you should not request one unless you plan to fully utilize it.
· As soon as you have your classroom assignment, which is typically posted a month or so before classes start on the undergraduate course time tables, you can check out the classroom specifications (most classrooms on campus). You can also look at OSM's document containing answers to frequently asked questions about using AV equipment.
· In rooms without a U of T teaching station, you should contact Brian Usher (416-978-6544) as soon as possible and preferably via e-mail to request AV equipment. Indicate the course code (ex. ECO220Y1Y), the lecture days and times (ex. TR 11-1), the class room (ex. SS 2135), and what you will need (ex. I want to give PowerPoint presentations in all of my lectures and I also need a microphone). I also highly recommend that you also request (in this same e-mail) to have a demonstration visit with the technician for your classroom. Ideally you will choose a time before the first day of class. The technician will show you how to use the equipment and will give you any codes you will need to access the equipment. Ask for the best telephone number to call for your classroom should a problem arise during or before lecture. You should receive an OSM e-mail containing the confirmation: double-check that the times, dates, locations, and equipment are correct.
· Emergency telephone numbers to call if the AV equipment is not working properly: 416-978-0423 or 416-978-6968.
· Make sure you completely shut down all data projection equipment when you are done lecturing (unless AV has specifically instructed you to do otherwise). Leaving this equipment on not only uses up valuable bulb life, which increases the chance of bulb burn out during lecture, but it also makes this equipment vulnerable to misuse.
"Art" of effective PowerPoint use: Some tips, questions, answers:
Q: Do we have reliable technical support or will I be wasting
a lot of valuable lecture time trying to get projection equipment working?
A: In nine years and in about 1400 hours of lecture time (all using the data projector) I have lost a total of less than 2 hours due to technical problems. The single worst case scenario I have encountered is when the bulb for the data projector expired in the middle of lecture and it took 15 minutes before a portable data projector was delivered and operational. This is not too bad at all: I have found that the AV support at OSM is very responsive and quick. You should be proactive though: arrive 10 minutes prior to lecture, have a demonstration with a technician before first day of class so you know how to operate everything, and immediately report any problems with the equipment (especially if the image is not a bright as usual as this is a sign the bulb is about to go).
Q: How about hybrid lectures: part PowerPoint and part
A: Good idea in theory but hard to implement given that most classrooms have the projection screen directly on top of the black board (white board). A great alternative is to use a tablet PC with a pen so you can write in real-time directly on your slides.
Q: Isn't it time consuming to create
A: Yes, it is fairly time consuming. How time consuming depends a lot on you and the course material. You may want to try creating a couple of PowerPoint lectures to see how long it takes.
Q: Should I post all of my lecture
notes on my course web site?
A: Understandably, students really, really want the lecture notes posted. However, to make sure that students attend lectures and keep up with the course in-between assessments, lectures have to have "value added." Each instructor can use some creativity to figure out a way to ensure lectures have "value added" even if all the lecture notes are posted before lecture such that students can print them out and bring them to lecture. In my case, have found that peppering the lecture notes with questions that are only answered during lecture and using iClickers helps ensure high attendance. There are many possibilities. For an econometrics course you may present a table of estimation results and discuss the interpretation in class (rather than write out the interpretation in the slides). For a course more focused on public policy, you could present the facts and assumptions in the slides but draw out the policy implications in class. For a theory course, you could present a model and derive the results in the slides but work through the intuition and the importance of various assumptions in class. Of course there are more direct means of ensuring class attendance such as having a TA pass around a sign-in sheet to take attendance, including "class participation" in the marking scheme, pop quizzes or other short in-class exercises. Make it very clear to your students that you expect them to take notes and attend class.
Q: Does using PowerPoint mean that I just read from
slides in class?
A: Try hard not to read your slides aloud in lecture: this is boring and students get frustrated because they can read faster than you can talk. Focus on providing context, interpretation, explanations, elaborations, insight, and encouraging some class room discussion (if appropriate).
Q: Does using PowerPoint mean I should change the
content of my course?
A: The means of delivery should not dictate the content. If you feel that the content of your course is not well-suited to using PowerPoint then don't use PowerPoint.
Using Classroom Response Systems ("IClickers") in the Classroom (prepared by U of T's, CTSI)
For two reports on the use of iClickers in our department – 2009 and 2011 -- see our intranet and look under Teaching (iClicker report).
For an April 2014 report on Piazza see our intranet and look under Teaching (Piazza report).