(Our Intranet is a complementary resource)
Last updated: July 8, 2016
Created and maintained by: Jennifer Murdock
A&S Academic Handbook for Instructors (updated October 2013, pdf)
A&S Academic Handbook for Instructors (updated October 2013, html)
New University of Toronto Provostial Guidelines on the Use of Digital Learning Materials (i.e. what you can do in terms of asking students to pay to access online materials, iClickers, etc.)
· ***Your course outline must be uploaded to our departmental website: http://www.economics.utoronto.ca/index.php/index/teaching/coursesForUser***
· The syllabus and the first day of class are your chance to communicate your expectations to your students. Make your expectations clear and precise for a smoother course and improved student satisfaction.
· You may require students who miss an assessment due to illness to provide a U of T Verification of Student Illness or Injury Form (pdf version) or other documentation.
Teaching Workshops at the Center for Teaching Support & Innovation (CTSI) (St. George campus, all faculty are welcome)
Teaching and Learning Symposium (annually at U of T)
Academic Skills Centre (UTM)
· You may periodically check for interesting articles (full text, searchable, pdf) in The Journal of Economic Education. From a U of T computer you can get full electronic access directly at http://www.tandfonline.com/toc/vece20/current.
· CTSI gives seminars and courses to help you in your professional development as a university teacher. Visit their web site to sign up. Subscribe to the CTSI listserv to ensure you receive timely information about upcoming events, workshops, lunch-time roundtable discussions, etc. To subscribe, you may send an e-mail to CTSI asking them to add you.
· You may choose to include a mid-course review of your own design. Students typically given more substantive and constructive comments on these than the A&S end-of-term evaluations. CTSI has a booklet entitled "Gathering Formative Feedback with Mid-Course Evaluations: A Guide for Faculty" that you can request (e-mail them) or you can borrow a copy from me. SurveyMonkey is a useful, easy, quick, and free tool that lets you create a professional looking online survey and easy-to-review results.
· You may consider joining (subscribing) to the Society for Teaching and Learning in Higher Education (STLHE), which is a Canadian association dedicated to the advancement of teaching and learning, or subscribing to a teaching newsletter such as The Teaching Professor, which will help you keep up-to-date with developments in teaching and learning and keep your courses on the cutting edge without taking up much time.
Online searchable database of undergraduate course evaluation results since 2012/13 on portal: click “Course Evals” tab in top right, scroll down, and click on “Course Evaluation Student Feedback” (under A&S, UTM, etc.)
Teaching Evaluation for Promotion and Tenure (Provost's Guidelines; general for U of T)
A&S Guidelines for the Assessment of Effectiveness of Teaching in Tenure and Promotion Decisions (Research stream; St. George Campus)
A&S Guidelines for Evaluation of Teaching Activities and Pedagogical/Professional Development (Teaching stream; St. George Campus)
New Changes as of June 2015 affecting Teaching-Stream Faculty (all campuses; these will result in changes to the guidelines above, but that is still in progress)
ASSU Anti-Calendar (St. George Campus) (published summaries of faculty members' student survey results; the last publication is Summer 2012)
Copy of blank student survey form (used
until 2012/13) (St. George Campus)
Tips & Templates from OSAI Nicely organized
· Dr. Kristi Gourlay is the Manager of the A&S Office of Student Academic Integrity (OSAI). She is extremely willing to help. She strongly encourages faculty to contact her with any concerns about academic integrity such as inappropriate use of medical notes, plagiarism, cheating or other such problems. OSAI also distributes a newsletter for faculty. It is not available online, but you can ask Kristi to e-mail a copy.
University of Toronto Libraries Exams and Course Collections A complete repository of old final examinations across all three campuses
See Using multiple choice questions on this website
· At U of T there is a major distinction between term work (which is anything you administer during the term such as term tests, essays, etc.) and final examinations (which you must assist in but you are not in charge of administering it: i.e. you do not control the date, illness policies, or the reporting of results for final examinations to students)
· Scheduling your term tests and obtaining appropriate testing rooms can be challenging.
o To avoid student conflicts with other courses schedule tests during lecture or tutorial time. Note: Section 6.3 of A&S Academic Handbook: “If a student has a conflict between a course holding a test outside its normal class hours and a test or required obligation for a class regularly scheduled into that hour, the regularly-scheduled academic obligation has precedence. The course with the irregularly-scheduled test must accommodate the student in some appropriate way.”
o Avoid dates with religious observances.
o If you are teaching a 200-level course, coordinate with other instructors on term test dates as many students are simultaneously enrolled in intermediate micro (ECO200/204/206), quantitative methods (ECO220/227) and intermediate macro (ECO202/208/209). All streams of intermediate micro should strive to overlap on test dates (as these courses are exclusions for each other so no student could have a conflict). Similarly all streams of intermediate macro should strive to overlap on test dates. We should strive for minimal overlap in test dates between micro, quantitative methods and macro.
o Typically your regular lecture room is not appropriate for testing. To request a room you may either: (1) ask Deborah Navarro to book a room for you (make sure to specify the course, sections, number of students taking the test, date, and time) or (2) make a booking request directly to the Academic and Campus Events (ACE, formerly OSM). Do this early (before classes start) to ensure that it is possible for you to get an appropriate room at the requested time: sometimes the University will not accommodate you. You will receive a confirmation e-mail: bring a copy to the examination room as proof that you have reserved the room for that time if there is a conflict. If you have 50-minute-long lectures holding assessments during lecture time can be limiting. For year-long courses (ECO###Y) you can request time for a midterm test (2 or 3 hours) during the December examination period by contacting Undergraduate Administrator Robbie Innes early in the term and the University will schedule your exam and assign a room (you lose control of the time and date).
· A&S sets the times and locations for all final examinations with the schedule published a couple of months in advance.
· Final examinations from previous years are publicly available from the University of Toronto Libraries Exams and Course Collections by entering the course identifier (ex. ECO200) under "ERes QuickSearch:". If you wish not to have your examination made publicly available see “Restricted Exams” in Section 9.4 in the A&S Academic Handbook for Instructors.
· For term tests, Accessibility Services will contact you (via e-mail) if there are students in your course that require special accommodations. (In most cases you will have at least one student requiring accommodation.) Accessibility Services coordinates all of this: you simply reply to their e-mail and e-mail them a copy of your test.
Writing at the University of Toronto (an excellent and comprehensive website for faculty and students)
New Writing Economics: A Guide for Harvard Economics Concentrators (October 2014 version)
· See the A &S Academic Handbook for important grading information including the U of T guidelines on mark distributions.
· Given that marks can be curved up but not curved down, there appears an incentive to write assessments that are difficult or time-pressured to ensure that marks are not too high and then to retroactively curve the marks up as need be. This is highly unpopular with students. Even if after the curve the marks are typical, students seem to never forget (or forgive you for) the initially low un-curved marks. While there are several possible explanations, one is that students believe that the assessment reflects your expectations of them and if they do not do well on the assessment they feel they have fallen short of expectations possibly despite extensive studying and preparation. The ex-post curve does not change this feeling. Many students will blame you for this failure because you have not effectively communicated your expectations to them or are perceived as having expectations that are impossible for them to meet.
· Term marks (not inclusive of the final examination) should give students a good idea of how they are doing in your course. This means that you should strive to follow the guidelines during the term. It is not fair to students to give easy term work that is generously graded and then hit them with a much more difficult final examination that differs from their expectations based on the term work.
· Students' official transcripts indicate not only the student's grade but also the overall course average.
For multiple choice tests the Economics Department recommends using a bubble form and having the papers machine marked. The TA Scanning Coordinator (Congfei Li, email@example.com, 647-965-5801 in 2015/16) in the Department of Economics is responsible for running the scan of bubble forms for multiple-choice tests. This document describes all the steps you need to take with respect to a multiple-choice test.
The TA Scanning Coordinator will scan the papers for you and e-mail a file containing the raw scanned answers. Your TA can run the marking software below. I would recommend having your TA visit this web site (instructions for your TA (txt)). You will need to provide your TA with a solution key and a spreadsheet with the list of students registered in your class and the TA Scanning Coordinator will e-mail the raw data with scanned answers. Your TA can set up and execute the code. Your TA needs to have access to Stata version 10 or higher to do this, which is available at computing labs around campus. Your TA can give you the automatically produced report summarizing student performance, a spreadsheet with the marks, and a file that you can post to your course web site so students can see their answers and their marks.
Marking Software: version (April 2014)
I have written software to grade and analyze student performance on multiple choice questions. This description explains what it can do.
Software to mark multiple choice question data (STATA do-file) version (April 2014)
Raw Data from scanning machine (spreadsheet) (note: names and student numbers are fake)
Course List from ROSI (spreadsheet) (note: names and student numbers are fake)
Report on student performance on Term Test #1 (text
This illustrative example uses real data (but with fake student names and student numbers). For each question, the report on student performance (sample report) indicates the percent of students that chose the correct answer, the percent that picked each of the wrong answers, the percent that chose the correct answer broken down by quartiles (first quartile, second quartile, third quartile and fourth quartile based on overall performance on the multiple choice questions), the Discrimination Index (DI), R-squared (r2) and the slope. The DI, r2 and slope are all statistics that can be used to assess the effectiveness of each of your questions.
· The DI is the difference in the percent correct for the top and bottom quartiles of the class. One measure of the quality of a particular multiple choice question is the DI: the higher the DI the better the question is at separating proficient students from incompetent students. If a question is too easy the DI will be small: proficient students and incompetent students get it right. If the answer is obvious even if you don't understand the concept the DI will be small: proficient students and incompetent students get it right. If the question is way too hard the DI will be small: proficient students and incompetent students get it wrong. If students who understand an important concept get the question right but others get it wrong then the DI will tend to be large: proficient students get it right and incompetent students get it wrong. If you write a really poor or misleading question it is possible that you get a negative DI index: proficient students are misled while incompetent students guess and some get it right. In general, the larger the DI the better.
· The R-squared (r2) uses the information in the data more efficiently than the DI index, which makes it a better measure. The R-squared for a question tells you what percent of the variation in the students' overall percentage correct on all other questions is explained by their performance of this question. A relatively high R-squared indicates that the question is a good predictor of the students’ performance on the other questions. This is one sign of a good question. A bad question that confuses students who know the material would be expected to have a low R-squared. A hard question where most students guess would again have a low R-squared.
· The slope for a question measures the how much higher on average the percentage correct on all other questions is for students that got this question correct. A high slope is an indicator of a good question, whereas a low slope is an indicator of a bad question.
· If you see a relatively low R-squared and a relatively large slope that is a sign that the question is relatively easy. Hence most students got it, but those who missed it really didn't know the material and also did very poorly on other questions. It is a good idea to try to include some questions like these to separate the incompetent students (F's) from the borderline competent students (D's and low C's).
· If you see a question with a low R-squared and a low slope, this is a sign that the question might have problems. It means that this question is a poor predictor of students' success on other questions and students who got this question right didn't tend to fare better than average on other questions. This could be consistent with: an overly hard question where all students (even top students) are guessing, confusing wording that leads proficient students astray, or an overly obvious question where all students (even incompetent students) can easily spot the right answer. You should review the specific question you wrote and also look at which distracters students selected.
By carefully reading the report, you can study exactly what happened on each question and potentially identify questions that did not work well. For well-written questions you can identify common misconceptions amongst the students if you see numerous students selecting the same wrong answer.
Pros and cons of using multiple choice questions in your assessments
· You control marking
· Ex post you can decide to accept alternate answers
· Marking is objective
· Marking is completely consistent across students
· Marking is done by machine so that even for hundreds of students it takes only a few hours total
· Students can get quick feedback: each student can have their marked paper "returned electronically" within hours of the assessment (responses and marks posted on website)
· Allows for more frequent testing: for example, with quizzes
· Potentially frees some TA hours for more student contact
· Gives very detailed feedback on students’ performance on each question
· You "set the agenda" for the assessment so students cannot avoid the questions (i.e. give long and irrelevant answers)
· No opportunity for ex post cheating (i.e. not possible to modify the test paper and then submit it for a remark)
· It is time consuming to write multiple choice questions
· Likely not a useful tool if you have few enough students such that you can mark most assessments yourself
· Poorly written questions lead to extremely frustrated students: see basic tips for writing multiple choice questions
· Poorly written questions can "give away" answer: see basic tips for writing multiple choice questions
· Multiple choice questions are not suitable for all of the subject matter taught in our department
· Multiple choice questions do not give students a chance to improve their writing skills
· Guessing can impact students marks and create spurious variation across students
· Some students complain that there is no partial credit (even though that's not true in expectation)
· Because forms are machine read, students that fail to follow instructions could face catastrophic consequences (failure)
U of T has adopted Blackboard as the university-wide learning management system (LMS) that typically involves a web presence for courses, downloadable handouts, course announcements, online discussion boards, online tutorial sign-up, student e-mail lists, online marks posting, online student surveys, and sometimes online testing and other advanced features. Blackboard training is available through CTSI. You may log in to the Blackboard Portal using your UTORid and password. If you do not have a UTORid or forgot your password go to UTORid Management.
MAKING YOUR OWN COURSE WEBSITE:
Instructors that choose to may continue to use self-maintained web pages (i.e. through the Economics Department server or through CHASS) and possibly in conjunction with Blackboard. For example, Martin Osborne uses the Economics Department server (http://www.economics.utoronto.ca/osborne/2030/index.html) and Rob McMillan uses the CHASS server (http://homes.chass.utoronto.ca/~mcmillan/ug_teaching.html). (Or you could develop your own learning management system from scratch: http://mjo.osborne.economics.utoronto.ca/index.php/course/index/5.)
Graduate Student Directory (for e-mail addresses/telephone numbers)
· TA assignments usually happen around the time that classes begin and sometimes a bit after they have started.
· At the beginning of the course you are required to tell your TAs the breakdown by task of the hours they will spend on the course. Martin Osborne has created a web-based form you need to fill out for each TA. Please log-in to our Economics Department Intranet.
· The Economics Department expects you to evaluate each of your TA's at the conclusion of your course: you will receive an e-mail with links from the Economics Department Associate Chair, Graduate Affairs near the end of an academic session.
· Duties typically assigned to TAs include: marking, conducting review sessions, holding office hours, and invigilation. It is often advisable to have TAs attend your lectures and especially for those TAs that will have student contact through review sessions or office hours. Even for 100 and 200 level courses, the assumption that the TA is sufficiently familiar with the course material to teach it is often a poor one. Further, TAs that are not in lecture cannot be expected to know what you covered each week and what was emphasized or possibly how you taught it (i.e. with graphs or with or without calculus). Top research universities, such as Yale, require their TAs to attend all lectures including 100-level courses and courses the TA has done before. While we do not have an unlimited number of TA hours, for large enrolment courses we do have enough to require lecture attendance of at least some of the TAs assigned to the course. For a half-year course, class attendance uses 24 hours total and for a full-year course, 48 hours total.
· If your course has a tutorial hour (third hour per week), you can use this for weekly TA sessions. This is convenient as students should not have conflicts during this time and you will automatically be assigned a classroom. If you have a course that meets only two hours per week, you can request a third hour for next year by contacting the Economics Department Associate Chair, Undergraduate Affairs Gillian Hamilton.
· For weekly TA review sessions (outside of the third hour of the course) and special review sessions before tests and exams you should request a room. To request a room you may either: (1) ask Deborah Navarro to book a room for you (make sure to specify the course, sections, number of students taking the test, date, and time) or (2) make a booking request directly to ACE: Academic and Campus Events (formerly OSM).
· For TA office hours or office hours for part-time instructors that do not have an assigned office, you can e-mail firstname.lastname@example.org to reserve time in the Economics Department. You can check the availability of our rooms. TA's need to pick up the key from Nada during regular business hours. Alternatively, TA's can meet with students in common areas (such as the Undergraduate Student Lounge on the main floor of the Economics Department, which is also open during regular business hours).
· For accurate marking, it is strongly recommended that you provide TAs with complete solutions (including alternate solutions that would also be correct). In addition, you may provide them with marking guidelines and a marking rubric.
ASSU (St. George Campus)
· Students are required to have and check a University of Toronto e-mail address.
· While it is not required that you communicate with undergraduate students via e-mail, you are required to clearly state an e-mail policy and to be otherwise accessible to your students.
· The Department expects you to have at least one hour of office hours per week per undergraduate course during the time when you are teaching.
· You can request a spreadsheet (Excel) that contains the contact information, including telephone number, e-mail address, and postal address, for all students registered in your courses. At the St. George Campus this request should be directed to the Undergraduate/Graduate Office Assistant Deborah Navarro. The names and e-mail addresses can be imported into a mailing list (see help for your e-mail system for specifics). Alternatively, you can download your class list from Blackboard.
university teaching INNitiatives:
New Integrate (see which teaching/learning technologies are already deployed at U of T and which are in the pipleline; can make suggestions)
Teaching and Learning Symposium (annually at U of T)
When used properly, PowerPoint (or other slide producing software) can be an effective tool in some branches of economics. PowerPoint allows you to include more graphs, tables, game trees, flow charts, data, computer output, etc. Each instructor should choose a lecture style that works best and PowerPoint is not for every instructor or every course. Below is some useful information if you decide to use PowerPoint (or other software such as LaTex) to give lectures at U of T.
Logistics of creating PowerPoint slides and publishing slides:
· If you publish your slides (i.e. post your lecture notes online for your students), you also need Adobe to convert the PowerPoint slides to pdf documents. You can purchase Adobe Acrobat Professional 11.0 at the Licensed Software Office at Robarts Library.
· When converting PowerPoint slides to pdf documents using Adobe, it is important that you embed the fonts. Failure to embed the fonts will mean that some students will not be able to properly view and print out your lecture notes. This is also useful if you are converting other documents you want others to be able to reliably read and print. For Windows users, you can set embedding the fonts (recommended) to the default: (1) Install Adobe (reboot), (2) Click on start (if have option, choose Settings), (3) Select "Printers and Faxes", (4) Right click on "Adobe PDF" , (5) Select "Properties", (6) Click button "Printing Preferences...", (7) Select the "Layout" tab, (8) Click button "Advanced...", (9) Under "Graphic", "True Type Font:" click on what is shown (probably "Substitute with Device Font") select "Download as Softfont" from the drop down menu that appears, (10) Click button "OK", (11) Select the "Adobe PDF Settings" tab, (12) Under "Adobe PDF Conversion Settings", "Default Settings:" choose "High Quality" from the drop down menu, (13) UN-check the box "Rely on system fonts only; do not use document fonts" ("Do not send fonts to "Adobe PDF""), (14) Click button "Apply", (15) Click button "OK". If any of this is confusing, you can see pages 4 - 6, How to PDF document that illustrates most of these steps (in some cases the instructions are slightly different, follow those given on this web site). You only need to do this once and not every time you convert a document.
· To convert a PowerPoint file to pdf: (1) Open the PowerPoint file you wish to convert, (2) Click on "File", (3) Select "Print" from the drop down menu, (4) Under "Name:" select "Adobe PDF" from the drop down menu, (5) If you want to double-check your settings you can click button "Properties" and you should see the default settings you set above (i.e. Layout, Advanced, Graphic, Download as Softfont; Adobe PDF Settings, Default Settings, High Quality; Adobe PDF Settings, Do not send fonts to "Adobe PDF" is unchecked), (6) Do NOT check "Print to file" box (even though you will be creating a file and not a hardcopy), (7) Under "Print what:" choose desired format (i.e. one slide per page, six slides per page, slides with lines for notes, etc.), (8) Click button "OK", (9) In Adobe, review the pdf file you have created and if you need to rotate the image: select Document, Pages, Rotate, OK, and then if it looks right, File, Save, (10) In Adobe, you can check whether you have successfully embedded the fonts by selecting File, Document Properties..., Fonts, and making sure in parentheses you see (Embedded) or (Embedded Subset) next to each font: for illustration see page 9, How to PDF document.
Logistics of delivering lectures using a data projector (at St. George campus):
· For most classrooms you need to bring your own laptop to class with you. However, if your class meets in a large lecture hall it might have a U of T teaching station. If there is a teaching station, you can just plug a memory stick into one of the USB ports. To access the teaching station you need to be U of T staff (faculty, sessional or TA) with a valid UTORid and password.
· To give PowerPoint presentations you will need to request the use of a data projector in your classroom. The data projector will either be permanently installed in the classroom (an "electronic classroom") or it will be wheeled in on a cart prior to each of your lectures ("portable data projector"). If you will be using PowerPoint for only some of your lectures, a regular classroom with a portable data projector will suffice. However, it you plan to use PowerPoint for most or all of your lectures I strongly recommend that you request an electronic classroom. You should make this request as early as possible to Undergraduate Administrator Robbie Innes. Planning for classroom assignments begins in December for the upcoming academic year. Further, if you are teaching more than one section back-to-back you should request that you be assigned the same classroom for those consecutive sections as you need time to set up your laptop and the data projector. The Department can then try to get your requests filled with the Office of Space Management (OSM). There is sometimes a shortage of electronic classrooms, so as a courtesy to your colleagues you should not request one unless you plan to fully utilize it.
· As soon as you have your classroom assignment, which is available earliest here (type your course code in search box, click search button, then click the hyperlink on your course code) and is typically posted a month or so before classes start on the undergraduate course time tables, you can check out the classroom specifications (most classrooms on campus). You can also look at ACE's document containing answers to frequently asked questions about using AV equipment.
· In rooms without a U of T teaching station, you should contact Brian Usher (416-978-6544) as soon as possible and preferably via e-mail to request AV equipment. Indicate the course code (ex. ECO220Y1Y), the lecture days and times (ex. TR 11-1), the class room (ex. SS 2135), and what you will need (ex. I want to give PowerPoint presentations in all of my lectures and I also need a microphone). I also highly recommend that you also request (in this same e-mail) to have a demonstration visit with the technician for your classroom. Ideally you will choose a time before the first day of class. The technician will show you how to use the equipment and will give you any codes you will need to access the equipment. Ask for the best telephone number to call for your classroom should a problem arise during or before lecture. You should receive an OSM e-mail containing the confirmation: double-check that the times, dates, locations, and equipment are correct.
· Emergency telephone numbers to call if the AV equipment is not working properly: 416-978-0423 or 416-978-6968.
· Make sure you completely shut down all data projection equipment when you are done lecturing (unless AV has specifically instructed you to do otherwise). Leaving this equipment on not only uses up valuable bulb life, which increases the chance of bulb burn out during lecture, but it also makes this equipment vulnerable to misuse.
"Art" of effective PowerPoint use: Some tips, questions, answers:
Q: Do we have reliable technical support or will I be
wasting a lot of valuable lecture time trying to get projection equipment
A: In nine years and in about 1400 hours of lecture time (all using the data projector) I have lost a total of less than 2 hours due to technical problems. The single worst case scenario I have encountered is when the bulb for the data projector expired in the middle of lecture and it took 15 minutes before a portable data projector was delivered and operational. This is not too bad at all: I have found that the AV support at OSM is very responsive and quick. You should be proactive though: arrive 10 minutes prior to lecture, have a demonstration with a technician before first day of class so you know how to operate everything, and immediately report any problems with the equipment (especially if the image is not a bright as usual as this is a sign the bulb is about to go).
Q: How about hybrid lectures: part PowerPoint and part
A: Good idea in theory but hard to implement given that most classrooms have the projection screen directly on top of the black board (white board). A great alternative is to use a tablet PC with a pen so you can write in real-time directly on your slides.
Q: Isn't it time consuming to create
A: Yes, it is fairly time consuming. How time consuming depends a lot on you and the course material. You may want to try creating a couple of PowerPoint lectures to see how long it takes.
Q: Should I post all of my lecture
notes on my course web site?
A: Understandably, students really, really want the lecture notes posted. However, to make sure that students attend lectures and keep up with the course in-between assessments, lectures have to have "value added." Each instructor can use some creativity to figure out a way to ensure lectures have "value added" even if all the lecture notes are posted before lecture such that students can print them out and bring them to lecture. In my case, have found that peppering the lecture notes with questions that are only answered during lecture and using iClickers helps ensure high attendance. There are many possibilities. For an econometrics course you may present a table of estimation results and discuss the interpretation in class (rather than write out the interpretation in the slides). For a course more focused on public policy, you could present the facts and assumptions in the slides but draw out the policy implications in class. For a theory course, you could present a model and derive the results in the slides but work through the intuition and the importance of various assumptions in class. Of course there are more direct means of ensuring class attendance such as having a TA pass around a sign-in sheet to take attendance, including "class participation" in the marking scheme, pop quizzes or other short in-class exercises. Make it very clear to your students that you expect them to take notes and attend class.
Q: Does using PowerPoint mean that I just read from
slides in class?
A: Try hard not to read your slides aloud in lecture: this is boring and students get frustrated because they can read faster than you can talk. Focus on providing context, interpretation, explanations, elaborations, insight, and encouraging some class room discussion (if appropriate).
Q: Does using PowerPoint mean I should change the
content of my course?
A: The means of delivery should not dictate the content. If you feel that the content of your course is not well-suited to using PowerPoint then don't use PowerPoint.
Using Classroom Response Systems ("IClickers") in the Classroom (prepared by U of T's, CTSI)
For two reports on the use of iClickers in our department – 2009 and 2011 -- see our intranet and look under Teaching (iClicker report).
Piazza is easy to set up and maintain. It is a far superior alternative to the discussion board on portal. Also, students that have no access to an online discussion board (or only a low quality one) sometimes make their own sites (e.g. on Facebook). Piazza is much more conducive to academic discussion and easy to monitor. I’ve very rarely had inappropriate posts but these can be flagged and deleted. Quick buttons let you mark good questions and good answers. You can also weigh in on the Q&A to correct misconceptions or answer questions only an instructor could address.
In 2014/15, Karen Bernhardt-Walther (ECO204Y, with >500 students) used portal tests and I tried portal tests in Summer 2015 (ECO220Y, with >100 students) and in 2015/16 (ECO220Y, with > 500 students). Portal tests are online tests run through the U of T portal. If you are familiar with Aplia or MyEconLab, which are for-profit online testing options that are sometimes packaged with textbooks, portal tests are the same idea. However, portal tests are free for our students but costly (your time/effort and/or TA hours) for you. In other words, you or your TAs prepare and input the questions into portal tests. During Summer 2015 and 2015/16, I prepared questions and answers in a plain text document and my TA input them into portal and set up the portal quizzes. However, Karen did everything herself: how much work the input part is depends on the number of questions, style of questions, and whether you want randomization. Next is Q&A to address likely questions. After that are some instructions for creating a test in portal (for you or your TA).
· Q: What is the potential value of portal tests?
· A: It can be an efficient way to administer frequent quizzes in your class without managing paper (i.e. no photocopies, no sorting, no returning papers). I would NOT recommend it as a substitute for regular tests because online tests are not invigilated. However, if you are looking for a way to help students regularly engage in your course – e.g. every week – in addition to regular tests, this is a potentially good option. Hence, while they are called “Tests” in portal, I view them more as weekly quizzes or online participation (not tests).
· Q: What kind of questions can I ask on portal quizzes?
· A: Pretty much any kind. It can be “Calculated Numeric,” which means that students type a numeric answer. It can be “Multiple Choice,” which means students select a letter answer (e.g. (A) – (E), but any number of alternatives are possible). It can be “Essay,” which means students type a written answer. It can be “Calculated Formula,” which means that students type a numeric answer to a question where the numbers have been chosen at random (within your specified ranges) and you have input the correct answer as a formula. There are lots more question types available. However, I think portal tests work well when the answers are numbers and/or words but not graphs, equations, etc.
· Q: Can I put tables, equations, graphs, etc. in the question?
· A: Unfortunately, it is NOT easy to put tables, graphics, or images directly in the question in portal tests (or if it is easy, I’m missing something). You can put hyperlinks. In my case, I have online lecture slides and homework (not graded) that are already publically posted. For questions with graphs, tables, equations, etc. I reference existing course materials (via hyperlink). Of course, some questions do not require giving a table, graph, or equation and those are easy to do in portal tests.
· Q: How are portal quizzes marked?
· A: It depends on the type of questions you set. For example, “Calculated Numeric,” “Multiple Choice” and “Calculated Formula” are marked automatically and immediately by portal. (These are marked flexibly: for example, you can specific a reasonable margin of error for number answers and you can accept alternate answers with partial credit for multiple choice questions.) Open-ended questions would be marked by you or a TA. The results from portal tests automatically appear in your Gradebook. If you like to keep your marks offline, you can download students’ marks on your portal quizzes as a spreadsheet. Note: If you realize the original solution entered was not correct, you can correct it and portal will remark everyone.
· Q: How easy is it to use portal quizzes?
· A: There is a lot of windows and clicking involved and some counter-intuitive navigation (just like with portal generally). Also, because portal is attempting to be super flexible, there are some confusing steps and unclear choices. Also, there are some limitations (see above about the difficulty of including anything that is not test or numbers). That said, I figured it out and gave my TA instructions and he did everything (I just wrote the text of the question, the answers, and the margins of error in a plain text file for him to input). However, coming back to it a couple months later, it still took me time to re-figure-out the steps. This is a general issue with portal: it is not that easy to do nearly anything the first time.
· Q: Why should I do all this extra work? Why not just use Aplia or the online question bank that comes with my textbook?
· A: Maybe you should use a paid service instead. It depends. If there is something already set up and ready-to-go and the questions work well for your course (and it costs students less than $60 for an H course or less than $120 for a Y course so that you comply with the Student Ancillary Fees rules), then that may be the best option. Also, some of these paid services have nice options (like embedded graphs that students can edit), which would be hard (or in many cases impossible) for you to do yourself within portal. However, if you dislike the pre-packaged questions or they don’t fit well with your course, than portal tests are a good option because you write the questions.
· Q: What about simply doing paper-based tests (not online)?
· A: Again, I do not view portal test as a substitute for regular tests (invigilated and paper-based). However, if you are teaching a medium to large (to very large) class, having students turn in weekly homework/problem sets or doing frequent paper-based quizzes in class/tutorials creates a lot of paper to manage (photocopy, sort, and return to students). Hence, for low-stakes, frequent, and graded work, online may be an attractive option. Also, grading may be facilitated: obviously for automatically graded question types grading is immensely facilitated but even open-ended human-graded questions may be facilitated because students type up their answers. However, if student answers are going to be in the form of graphs, diagrams, tables, equations, derivations, etc., a paper-based approach may be absolutely necessary. Also, we do not know if we’ll have Crowdmark for 2015/16, but this can help get some of the benefits of online tests while still having students write up their answers on paper.
· Q: How much should portal tests be worth?
· A: Obviously it depends. However, it seems like 5% - 10% of the course grade is often reasonable if students will have to do many portal tests (e.g. 10). That’s enough to motive students but not so much as to give an extra big incentive for cheating.
· Q: What about cheating?
· A: This is a big concern. Online tests (portal tests or paid services) are not invigilated. There are strategies to address potential cheating. There is test design: e.g. give students random subsets of questions or have randomly generated numbers in questions so that tests and answers differ across students. There are collaboration policies: e.g. allowing students to work in teams. I prefer to use both. Have randomization of tests across students and explicitly allow collaboration on portal tests. Here is my draft policy excerpt from an ECO220Y syllabus.
· Q: How long do students have to complete tests?
· A: You get to set this (and a very long list of other options). I like to give students a timer: e.g. 60 minutes once you start. This is another method to discourage cheating (but that is not the only rationale for having a timer).
· Q: What if students are late? Miss a portal test?
· A: Usually dilemmas. You can pretty much do whatever you want (portal has lots of flexibility here). I like to keep it simple and not allow late submissions and to have a make-up portal test that replaces the two lowest marks during a term’s 12 submissions (one per week), which also can replace zeros (missed tests). Here is my draft policy excerpt from an ECO220Y syllabus.
· Q: How do students review their work after it is marked?
· A: You need to give them instructions for how to find and see their answers (this is another case of portal being convoluted: students cannot find this on their own, and I do not blame them). Here are instructions you can give students (see paragraph that starts “How can I see my results for each portal quiz?”). Also, in portal you can choose when students can see their marked results (remember for some question types portal grades them instantly upon submission). I delay the availability of marks and answers until after the submission deadline for the entire class.
· Q: What about partial credit for trying (participation points)?
A: Unlike iClicker, which allows you to award points for attempting to answer even if the answer is wrong (i.e. participation points), this is not allowed in portal tests. There are some question types (e.g. “Multiple Choice”) where portal will let you automatically award partial credit for wrong answers (if you so choose). Of course if you use a question type that is graded by a human in portal (e.g. “Essay”), obviously partial credit is easily awarded for effort.
· Q: I’m thinking about trying portal tests in my class. How do I set them up?
· A: See below. (Note: These can be set up well ahead of time and will automatically deploy themselves at the right time and make announcements to students. TAs are usually less busy and the start of the term.)
Steps to create and deploy a portal test (yes, there are two separate steps). These can be done by either an instructor or a TA (or a combination) in portal:
1. Enter the portal site for your course.
2. Under “COURSE MANAGEMENT” in the bottom part of the left toolbar, click “Course Tools,” and then select “Tests, Surveys, and Pools.”
3. Click on “Tests” in the new window that opens.
4. Click on “Build Test.” (Note: You will be building the components for a test that will need to be deployed separately. Instructions to deploy a test for students are later in this list.)
5. At this point you are asked to fill in a “Name” (required) and have space for “Description” and “Instructions.” Just put a name (you will do that other stuff later). At this point you have a decision to make regarding test design. If all students are going to see the same test, it’s a bit easier. However, I like the idea of writing multiple versions of the same question (with different answers) and having each student get a randomly selected one of those questions. If you simply want one test then name this something like “Portal Quiz: Week 4” but if you are going to have random selections of questions the name this something like “Week 4, Q1.” In other words, you will enter more than one possible “Question 1” for students to get. (Note: It is important to name this something that will be easy to find later on such as “Week 4, Q3” or “Chapter 12, Q2”: i.e. make it clear which test it will belong to and what question number it will be.)
6. No matter what you decided in Step 5, the next step is to input your questions. Click “Create Questions” and select the question type from the drop down menu. Follow the instructions that appear. Note that only the stuff with an orange asterisk is required. Repeat until you either have: (A) entered all the questions on your test (i.e. if everyone is getting the same questions) or (B) entered all versions of that particular question (i.e. if you’re going to have multiple versions of each question where each student randomly gets one).
7. If you are doing a simple test (no randomization), SKIP Step 7. Otherwise, repeat Steps 5 and 6 for each question in your test. (You’re creating a bunch of mini-tests that will be put together later. In other words, one mini-test with all versions of question 1, another mini-test with all versions of question 2, etc.)
8. After finishing entering your test questions, you are now ready to deploy the test for students, which includes setting the test-taking parameters such as start date, due date, time limit, etc. To do this, make sure your “Edit Mode is ON” in the top right corner of portal. At the top of the left tool bar, click the “+” and select “Content Area.” Name it. This is going to be the “folder” where all of your portal tests are. Hence, name it appropriately: e.g. “Weekly Portal Quizzes.” You should check the box “Available to Users” because this does NOT mean students will be allowed to see your portal test before you intend them too (just that they can see a folder that will contain the tests). Hit submit. Click on your newly created “folder” in the left side bar. Click the “Assessments” drop down menu and select “Test.”
9. Now, if you are doing a simple test (no randomization), you can select the test you already created in Steps 2 – 6. In “Add an Existing Test” you will see all tests you have not yet deployed. Click on the one you want and hit submit.
10. If you are doing a simple test (no randomization), SKIP Step 10. Instead of “Add an Existing Test” (as in Step 9) click the button to “Create.” Name your test: this is the name students will see. E.g. “Portal Quiz: Week 5.” (You can edit the other parts later after you see the instructions portal will automatically give students advising them of deadlines, time limits, etc.).
11. If you are doing a simple test (no randomization), SKIP Step 11. Now click “Reuse Questions” and select “Create a Question Set.” In the pop up window, select “Tests” on left tool bar. Select the one that corresponds to the version of question 1 you wrote for this test. But, you’re not done. You’ll see the questions (i.e. all versions of that question) appear but you have to select them all. You can do this by marking the little grey box at the top (selects all) or by clicking each version separately. Click “Submit.” Now repeat this process for each question in your test. Click “OK” and click “Submit.” (I know, this part is weird.)
12. On this “Test Options” page you have A LOT of choices. I would recommend the following:
a. DEFINITELY click “Yes” to “Open test in new window” or there can be problems with certain browsers.
b. Under “TEST AVAILABILITY,” click “Yes” to “Make the link available.” (Note: You get to choose further down when it will be available. I know, this is not intuitive.)
c. Under “TEST AVAILABILITY,” you have a choice. Portal will automatically make an announcement to your students about the test if you click “Yes”: this option works well. You could also write your own announcement.
d. Under “TEST AVAILABILITY,” just skip over the next few things: i.e. leave them at their default settings (i.e. “Multiple Attempts,” “Score attempts using,” “Force Completion”). (Note: Do NOT turn on “Force Completion” or I promise you’ll regret it.)
e. Under “TEST AVAILABILITY,” make a choice about “Set Timer.” Both Karen and I used it and recommend it. If you want students to have a time limit (note they can close their browser and open it again but the timer keeps going regardless), then click it and set the time limit. 60 minutes works well. I click “ON,” which forces completion once the time runs out (it seems to me that if you’re going to have a timer in the first place, it needs to be binding).
f. Under “TEST AVAILABILITY,” DO select the options of “Display After” and “Display Until.” These determine the window during which students can access this portal test.
g. Under “TEST AVAILABILITY,” skip “Password.”
h. Under “TEST AVAILABILITY EXCEPTIONS,” just skip over it (i.e. leave it at default settings).
i. Under “DUE DATE,” select this and enter the date. I click “Do not allow students to start the Test if the due date has passed,” which doesn’t allow late submissions (except that there is a loop-hole: if a student starts a test at 5:59pm that is due at 6:00pm with a 60 minute timer they will get the full 60 minutes and portal marks their submission late and you/your TA have to go into the gradebook and accept them (or not)).
j. Under “SELF-ASSESSMENT OPTIONS” I unselect the first one as I do not use the score calculator tools in Grade Center (which cannot understand common things like dropping the lowest mark or reweighting).
k. Under “SHOW TEST RESULTS AND FEEDBACK TO STUDENTS”: Select “On Specific Date” from the drop down menu and choose a date and time that will certainly be after the last student submits the quiz. (Or, if you are not worried about answers getting out, you could allow students to see their results right away.) Also, click all of the boxes to the left of it (“All Answers,” “Correct,” “Submitted,” “Feedback,” “Show Incorrect Questions”) if you wish for students to see what they wrote and what was the right answer (which seems like a good idea in any typical scenario). (Note: If you want to keep marks secret until after the due date, you also need to hide the column of marks for that test in Grade Centre from students or they can see their mark there.)
l. Under “TEST PRESENTATION,” just skip over it (i.e. leave it at default settings).
13. If you wish to change your test’s questions or answers: in the main left toolbar of portal, click the folder with your portal tests in it (e.g. the name you picked in Step 8). Move your mouse on top of the portal test you wish to change, click the grey down arrow, select “Edit the Test” from the drop-down menu.
14. If you wish to change your test’s options (e.g. due date, time limit): in the main left toolbar of portal, click the folder with your portal tests in it (e.g. the name you picked in Step 8). Move your mouse on top of the portal test you wish to change, click the grey down arrow, select “Edit the Test Options” from the drop-down menu. (You can change the stuff you picked in Step 12.)
15. Now to make sure this test looks the way you think it should, preview it. In the main left toolbar of portal, click the folder with your portal tests in it. Click on the hyperlink on the test’s name. You now will see what students will see. Click “Begin.” You can answer your question and click “Save and Submit” and then “OK.” Note: You will not appear in your own Grade Centre but you can at least make sure things look right and you’re getting points for correct answers and no points for wrong ones.
16. If your question types can be automatically graded, you are basically done. To deal with any late submissions, you can go to Grade Centre and click “Needs Grading.” If you are doing open-ended questions with human grading, you go to Grade Centre and click “Needs Grading.”
If you choose to do this, there will invariably be more you and your TA will need to figure out as you go. Hence, while I believe there are strong pedagogical benefits, I am certain there are considerable costs. Hence, portal tests may be the best idea in larger enrolment courses where the fixed costs are spread over many students and where you have considerable TA hours and can delegate much of this.
What is Crowdmark?
Starting in 2014/15, select courses in Economics have utilized Crowdmark to manage term work electronically (returning term work to students electronically). Here is an Introduction to Crowdmark and if you visit Crowdmark’s website and scroll down there is a video and a simple graphic showing the six steps taking you from test preparation through the students receiving their graded work. Here is an overview of using Crowdmark at U of T (through portal). It may help to see a sample Crowmark test from an Economics course with 500 students. Each student gets a unique test paper (i.e. #1 - #500): here is paper #1 and the separate supplement (not through Crowdmark).
Why use Crowdmark?
There are many distinct advantages of Crowdmark that add up to a big advantage over the traditional entirely paper-based approach. Here is a summary of the advantages of Crowdmark most valued by those in the Economics Department who have used it:
· Substantial savings of TA hours
o Marking is more efficient and especially for tests marked by a team of TAs
§ When each TA specializes in a question or part of a question, papers can be physically separated so each TA gets her/his own pile and/or marking can be done entirely online
o Papers do not need to be sorted
o Marks entry is faster (and more accurate)
· More accurate marks entry (and no need to write subtotals on front of test papers, which are very prone to clerical errors)
· Greater ability of students to check for and efficiently report clerical errors
· Saving valuable class/tutorial time because do not have to return papers to students
· Faster return of students’ work: can send results to students as soon as marking is complete and do not need to wait until the next class/tutorial
· Improved student access to their own term work (no lost papers; convenient electronic copies)
· Allows instructors on-going and convenient access to all submitted term work (including for reviewing TA marking, preparing letters of recommendation, etc.)
· Elimination of the possibility of ex post cheating (falsified re-mark requests)
· Allows students to write term work entirely in pencil and use an eraser as needed, which is better for test-takers and markers
· More efficient handling of re-mark requests
· Potential for more comments on student work as longer comments could be added electronically
Also, you may request a copy of “Crowdmark Report: Select ECO Courses” by Jennifer Murdock, Economics Department, University of Toronto, August 12, 2015 by e-mail (email@example.com).
How can I use Crowdmark in my course?
Associate Chair of Undergraduate Studies (Gillian Hamilton) if you are
interested. Starting in April 2016 the cost is $2.75 per student, which is
shared by A&S and the Department. It makes most sense to use in larger
courses that will have multiple pieces of term work utilizing this technology.
(The cost is per student, not per piece of term work.) The TA
Scanning Coordinator will contact you (after approval by Gillian) to
explain how this works and to train your TAs. (Note:
Your TAs will do the work, the TA Scanning Coordinator just helps them get
going.) Even once approved, you may ONLY use Crowdmark for your term tests and
other term work: it is not currently permitted for final examinations.
Once approved, how do I actually start using Crowdmark?
While using a new technology can be daunting and especially when producing your tests on a deadline, using Crowdmark is quick and easy. The steps you will need to take to enable Crowdmark within your portal site are here. You can visit Crowdmark’s help page and they are anxious to help you personally: just e-mail them at firstname.lastname@example.org. They respond very quickly and will walk you through step-by-step over the phone if need be. Also, here is a partial summary of the collected wisdom of Economics after a couple years of using Crowdmark, which will help you with some common practical questions about using Crowdmark.