I asked Dr. Hull to reply to some specific questions I had about the views of the psychology department at Dartmouth. I am not interested in whether Dartmouth should stop giving credit for AP scores or any other issue - I think that's entirely that school's decision to make. But since all of the articles I've read have specifically pointed to students who made a 5 on the AP Psychology exam not performing to the standards of the psychology department, I wanted to hear directly from someone there. Dr. Hull was gracious enough to send the response below in reply to my questions.
-----------------------------------
From Dr. Hull:
First, note that this was not a scientific study. Nor did we intend it for public consumption. Indeed, we did not even intend these data to inform decision making at the college regarding AP credit in general. We simply collected these data to inform our own departmental decision making regarding whether to give Advanced Placement credit for Introductory Psychology.
We took three years of introductory psychology tests taken by Dartmouth students.
Each year has three terms of exams (we offer Introductory Psychology in the fall, winter, and spring terms). Each term has 4 exams (each exam over about one quarter of the material) plus a comprehensive final exam. Each exam has 50 questions. So, we ask 250 questions per course and 750 questions per year (although many of the latter are redundant – we don’t give the exams back to students and we do re-use questions).
Exams are over both lecture and book material. We combined the exams in a single year, discarded lecture based questions, discarded redundant textbook questions, discarded (a very few) questions flagged as bad questions by the faculty, and then randomly selected questions from the resultant pool. A score of 68% was required to pass.
This procedure was then repeated for three years.
Year 1:
97 questions
89 students with AP Psych scores of 5
7 passed (8%)
Mean score = 57%
Year 2:
97 questions
93 students with AP Psych scores of 5
8 passed (9%)
Mean score = 56%
Year 3:
80 questions
26 students
5 passed (19%)
Mean score = 65%
Overall:
208 students took it
20 passed (9.6%)
Keep in mind:
1. These tests were based on tests given to Dartmouth students in a Dartmouth course.
a. We use a high level textbook (Gleitman; Gazzaniga & Heatherton) whereas high school AP courses typically use a general level textbook (Myers).
b. We are a Department of Psychological and Brain Sciences and our textbook selection reflects this. This is appropriate for our majors, but probably not true of AP courses.
c. The difficulty of the questions used in our test reflects the difficulty of the questions chosen by our faculty for testing knowledge of students in our courses – i.e., we did not randomly select from a test bank pool provided by the publisher, but rather from a pool of items used by our professors in our courses.
d. It is out impression that students often take AP Psych courses earlier than other AP courses (Calculus) and may have forgotten the material after 2-3 years.
Finally, note that students who got an AP Psych score of 5, who fail our exam, and who subsequently take our introductory course do not do better than those without an AP course. Although a null effect, this is at least consistent with the notion that our exam is not unduly strict. In other words, it does not seem to screen out students who have mastered the material in a way that makes our introductory psychology course easier for them.
-----------------------------------
This is Steve again. If you have thoughts about this reply, I'd encourage you to post them in the comments below.
--posted by Steve
Posts
14 comments:
So, instructors wrote the questions given on their tests. I'd like to see the data on the reliability and validity of their questions.
If the focus at Dartmouth in an introductory psychology course is to introduce students to a more rigorous text, then make that be the case for requiring Intro of all students.
Dr. Hull says "students who got an AP Psych score of 5, who fail our exam, and who subsequently take our introductory course do not do better than those without an AP course." Could we see more numbers on this? I am having a hard time accepting the idea there is ZERO correlation between an AP Psych score of 5 and Intro grade. Could there be a statistically significant correlation here?
Thanks again Steve (and of course Dr. Jay Hull) for being open about Dartmouth's decision.
Sounds to me like more of a self-indictment of their introductory Psych class rather than an indictment on AP Psych students who scored a five and the high school curriculum. To me, they should be analyzing their own curriculum in the intro class and determining why the failure rate. IF this becomes (or is and I just don't know it) a larger scale problem - then it is worth looking into on the high school end. Otherwise, it remains a Dartmouth issue.
Todd,
I think you have misunderstood the statement. My interpretation is that Dr. Hull is indicating that the students with an AP background did no better than the students without an AP background, not that the AP students failed Psyc 101. AP was just irrelevant to their performance in the course. That would make sense if the content was simply different.
I think that the past three years of data would indicate when in the college proffession educators were downsized and found a way to save teaching positions. If 90 students can no longer opt out of General Psych that saves jobs.
I would also argue that the AP curriculum does not align with the Dartmouth curriculum. Could we possibly see a sylabus?
I did not get a class syllabus, but I found one online from 2005 and it looks to be the same structure as the class that Dr. Hull is describing. Each class lasts a term and there are three terms per semester - the course is about 11 weeks. The course is co-taught by four instructors who each take roughly 1/4 of the term. I searched earlier this week and discovered that the most recent enrollment numbers for the classes over the last three terms were 71, 119 and 127 students.
Here is the syllabus I found: https://dl.dropbox.com/u/26161916/Dartmouthpsyc1syllabus.doc
I also question the validity and reliability of the questions that are used on the final exam. After reading the syllabus you found, it seems as though the course content / test questions is at the will of one of the 4 lecturing professors. Each professor may choose to emphasize (or not) content according to their own interests or expertise. This content may be vastly different than our goal as AP teachers to give an overview of all of the major content areas of psychology.
For another (supportive) take on it: http://www.peerreviewedbymyneurons.com/2013/01/23/breaking-students-forget-things-they-memorize-for-a-test/
Thanks to Steve for all your work on this! Obviously a "hot topic" for AP Psych folks, and I'm grateful to you and Dr. Hull and Trevor Packer for all the different perspectives on the decision.
I'm glad and grateful that we're all talking about this in a respectful, non-defensive way. In the past, I think I remember a couple episodes when college folks did similar correlational studies between performance in AP psych and college classes, and I fear our community didn't react quite as respectfully as we should have.
A couple more thoughts to throw on the pile,: it does seem odd that students who were very successful at recalling psych concepts on the AP exam were not more successful on the Dartmouth tests. Makes me wonder (as others have) about what concepts/skills the Dartmouth tests measured. The AP exam has a "table of specifications" (what % of the test measures each chapter) and maybe a more fine grained analysis of what chapters different students did well or poorly on would reveal important patterns in their performance? The overall % correct might mask those patterns.
Last thought, sort of related to the article that Sue F. posted: It would be GREAT if our AP Psych classes helped students prepare for success in college classes in general (via skills learned related to studying and encoding, motivation, etc.) Apparently that didn't happen for these students at Dartmouth. This might be due to the ways the Dartmouth classes are graded, but maybe it's worth a discussion within our community about how deliberately we work with students regarding long-lasting skills related to being successful and what Psychology has to teach us about those skills?
The comments on here are very interesting to me. Several years ago (more than I care to tell!), I performed a simple study on HIGH SCHOOL psychology (not AP) and found what Dr. Hull found: there were no differences in grade based on whether or not students had previously taken psychology. I posted these results and heard from people that I didn't know about AP yet.
I took the time to get to know about AP psychology and became involved in the process. Since 1997 I have been involved at various levels in the AP process and have come away from that experience extremely impressed with what the AP program is trying to do and the efficacy of the scoring system as it reflects knowledge of basic psychological concepts.
To understand why I think this, I'd like to take a moment and describe how the AP exam is developed and how it is scored. I'd also like to talk about how the AP program evaluates the validity of the AP exam. Finally, I'd like to comment on why I think the people at Dartmouth College found the results that they did.
The AP psychology exam is developed through consultation with faculty from colleges/universities and high schools from around the country. Agreed upon standards are used to develop questions. These standards are modified by faculty as they evaluate the current state of the discipline. By doing so, faculty work hard to ensure that the questions being asked on the exam are relevant to psychology as it is presented today (at least according to dozens of texts and syllabi that are at the disposal of the committee).
After questions are selected, they are scrutinized for reliability and validity by the committee to ensure that the questions are fair, they have only one keyable answer and they are questions that reflect issues that are germane to the discipline.
Finally, the questions are pre-tested in college classes (the multiple choice questions anyway). The pretesting is designed to ensure that the questions are fair and good discriminators.
The free response items are evaluated over the course of several committee meetings and undergo rigorous evaluation prior to being administered.
The scoring of the free response items are done with a high degree of reliability and appear to be good evaluators of how much students know about general psychological concepts.
Now, I am not responding to this post because I am "just speaking the party line". I am posting because I believe AP does a fine job of measuring performance as it is described by the faculty who contribute as well as the faculty who write text books. The exam is representative of the current state and is fair, equitable and a good measure of how well a student has mastered the material at the time of the exam.
Further the College Board (which owns AP) has done studies that has demonstrated that students perform better in their SECOND course in the discipline than those who don't take AP.
ALl this is to say that I believe the study done at Dartmouth might be one that is specific to their university, idiosyncratic to the exam items selected by the faculty (which might be more specific to the area of expertise rather than general psychology facts), and not an indictment of the AP program. I hope that the faculty at Dartmouth reconsider their approach and perhaps, run an additional study where they ask questions from the test bank to evaluate students to see if perhaps the questions they are asking are too specific.
Dr. Hull, I hope you know this is in no means disrespectful or to question that your faculty at all. Dartmouth has a respect for teaching that I believe is unparalleled in many ways. Rather, I'd just like you folks to reconsider your decision and be open to dialog as to why the results came out the way they did.
Respectfully,
Chris Hakala
Professor of Psychology
Western New England University
Thanks to Steve, Dr. Hull, Trevor, and everyone contributing to the discussion here. Several things jump out at me. First as Dr. Hull writes, the Dartmouth texts (Gleitman et al and Gazzaniga et al) are definitely high level. I don't think that that serves a "typical" non-major student sampling intro psych as well as Myers, but be that as it may, Dartmouth's psych dept. can certainly arrive at the conclusion, for any number of reasons, that the AP course doesn't fit its requirements.
Seconding Chris' comments, I would hope that the psych dept. at Dartmouth would be willing to revisit their decision. A good place to start might be by having one of the faculty take part in the summer AP reading. Alternatively, I'm fairly certain that both college faculty and high school faculty attending the summer AP psych reading would appreciate it if Dartmouth would send a representative (at the College Board's expense) to explain where the AP psych program needs to beef up in order to meet Dartmouth's standards.
I think its more problematic though, as Dr. Hull writes, to use Dartmouth's psych conclusions as the basis for denying credit system-wide, if that is in fact what Dartmouth has done.
I agree with the questions about the validity of the exam. There are programs that can be used to determine the difficulty and discrimination of the test questions. I'd like to see that information.
I also wonder why Dartmouth is getting rid of IB credit as well. http://www.dartmouth.edu/admissions/apply/thinking/credit.html
Perhaps what the Dartmouth faculty are saying is that high school should be high school and college should be college.
I say, if high school teachers can effectively teach at the college level, let 'em do it. If the College Board wants to dictate a standard early-college curriculum, let 'em do it.
Then universities (and professors, like me) can skip teaching all those intro courses to disinterested students and focus on higher-level learning instead. Plus, we can streamline admissions by only taking students who have already "passed" the introductory curriculum. And a 3-year degree will cost parents less. Everyone wins.
This is now OLD news, but I found this update on the Dartmouth story VERY interesting!
http://thedartmouth.com/2013/02/13/news/ap-credit-policy-change-attracts-media-attention
Mastanduno responded more directly to what he considered misrepresented accounts of Dartmouth's policy change in a letter in The Washington Post, in which he wrote that "the decision's rationale is rooted in our faculty's belief that high AP exam scores are not a substitute for a Dartmouth undergraduate class."
Post a Comment