My Dissertation Study

My dissertation study

A lot of people have wondered what kind of dissertation someone does in English composition, or specifically what my dissertation is about. So here goes. I gave mine a nice lofty title, mostly as a way of positioning myself for a job someday—I know, but it's just a title. “Applying Modern Validity Theory to College Writing Assessment” or something like that.

The simple version:
I am trying to find out the experience students go through after they have been placed in Basic Writing (the lower level first year composition class). So I have gathered about 10 participants and I'll be interviewing them, reading some of their work—including the essay that got them placed into BW—and just trying to get inside the overall experience, the more global effects of being placed into BW.

The larger implications:
The larger issues here are educational tracking and assessment (or test) validity. People all over the world are tracked according to their “abilities” or “aptitudes,” ideally all done for their best interest, but the bigger question is whether tracking people this way is actually good or bad for them—and of course, if it's bad for them, who is benefiting..? In other words, if a student “belongs” in BW, we say this because we think they will be better off in the long run for having taken the course; we believe that are not yet ready for College Writing (the mainstream composition class), either because they don't have certain “skills” or perhaps they just need more time writing as they develop and mature. Also, we fear that putting them in CW will be bad because they will get low grades, or maybe fail; they will get discouraged because they will feel behind, all that kind of stuff. In this way, it could be the best decision.

But what if not? There's also the chance that this person needed that little boost saying, You're a better writer than you may realize or than your schools have let you know; you belong in the more advanced class; maybe they are the type of learner who rises to a challenge and responds more to a little added belief in themselves than in a kick-in-the-pants type of placement. We don't know. And the placement method itself won't tell us. What's more, the student themselves might not really know. To make it worse, this course may separate them from their friends, is usually non-credit-bearing (but still costs money) which means they will be a semester behind for graduation credits; often this course will not transfer—and since there may be a high correlation between BW placement and low socioeconomic status (SES), this can put undue burdens on someone who has to transfer because of financial constraints and now had to pay for a class that they can't even use toward graduation... The list goes on. One of the larger points is that this is a high stakes decision. Yes, there are educational matters at hand—being ready, “ready,” for CW, etc., but there are much more global issues of social identity, perception of self as a student, economic burden—things that may greatly outweigh the educational benefits of the placement.

So how to we find out? And why did I call this a matter of validity theory?

Validity, what it is.

Validity is “scientific inquiry into score meaning” (Samuel Messick, 1989). The old way of looking at it (pre mid-1950s) was as follows: a test score is reliable to the extent that the same results would happen over and over again (whether the student took the same test over, or whether several graders graded it); but its only valid if the test actually measures the thing it was supposed to test. We no longer quite see it this way. But just for an example, if you measure water temperature with a thermometer and it reads 80 degrees, it might be right, but lots of things could be wrong. So you could try it a few more times with the same thermometer, and you could try several different thermometers—if they all say 80 degrees, you have a very reliable measure. But, if the water is forming ice crystals, or is about to boil over, then clearly, the thermometer did not give a valid measure of the temperature. So reliability is the extent to which measures are consistent, and validity is (or used to be) the extent to which they are “correct.”

But the educational sciences outgrew this definition of validity for a few reasons. First, when you're talking about a psychological/mental construct—can this kid write well enough to take CW?--the issue of a “correct” answer just may not exist. In these matters, such as BW writing placement, there really is no correct or incorrect answer, but the answers are more or less plausible. But there's a bigger issue at stake. The measurement tool—the assessment procedure—is here to help us obtain evidence for the students' placement; it is not here to place them. This is huge. WE place them. The test is a tool that, if designed will (if it measures what it's supposed to measure) give us good evidence, but we the humans are the ones who make the decision. Validity, therefore, resides not in the measurement tool, but in the decision that we the humans make. So if the thermometer reads 32.5 degrees but we still go swimming and freeze to death? It's wasn't the thermometer's fault.

It has been about fifty years since we took validity out of the hands of the test itself and but the burden upon ourselves. What this means is that, in order to make the best possible decision based on an assessment, we need to accrue all possible evidence for and against the decision; thus, validity is seen now as a type of argument, not just as a matter of given fact. In terms of my study, I am investigating possible reasons for or against the placement of these particular students in order to examine possible issues in the larger matter of educational tracking.

One other major shift in validity theory is that the validity of the decision made and the social consequences of the decision are now seen as one, not two imperatives. (This was a major contribution of Samuel Messick, 1989; the validity as an argument idea is credited to Cronbach, 1988, and Kane, 1992, for those interested.) Again, my study seeks to fulfill this imperative. I have read tons of literature about the need to incorporate social consequences in validation inquiry, but I have never seen a study that actually does it. (Again, for the nerdier, Shepard, 1993, talks about it, but I don't think she goes far enough.)

So what happens now is that we've really opened a debate about science and scientific method in educational and social research. If we are to examine validity as a interpretive argument, with the assessment scores being the evidence and the decisions made being the conclusion—and if we are to take in social consequences as part of the evidence for or against the validity of this argument, we have graduated from the scientific paradigm that looks for the “correct” score as the valid score. There just isn't such thing in these matters. Our scientific inquiry into the score meaning of these placement essays must—if it is to follow the major ideas of validity theorists—examine phenomena that are not representable through quantitative means, and certainly not through a paradigm that even believes that there is objectivity is the measurement of this type of phenomena. Objectivity and correctness just don't come into play here because there just is no objectively correct answer to the big questions in life, such as, Did we place that student in the right class? The traditional view of validity would allow for a glance at transcripts and retention rates perhaps and be satisfied that the assessments were valid. But no longer. We know too much more now, and modern theorists do not but into the notion of excluding any possible evidence from the validity argument. (Pamela Moss is another major theorist to check out.)

Another reason for this study is that I see a lot of wondering in composition literature about what happens to basic writers as a result of their placement, but again, I don't recall ever seeing a study that sought to find out. That's what I'm trying to do. And I'm approaching this from a paradigm of phenomenology, which means that rather than giving surveys to thousands of BW students, and analyzing the data I receive on that level, I am attempting to get inside (and even help co-construct) the “lived experience” of these few students. This type of paradigm seeks insight, rather than objectivity or generalizability. And this this something I would love to fight about! The question is, Is the aggregate of common experiences of thousands of people more telling that the vivid life stories (in relation to our question) of a handful? Clearly the two inquiries are after different things, and each is good for something that the other isn't. But anyone who has read this and who sees no value in a phenomenological approach, as I've outlined here—let's fight it out! Maybe you're right. But maybe I'm right...

So far I have interviewed ten participants, and I've found some really interesting themes emerging—themes I could not have predicted ahead of time. I'll let you know as I gather more data. The interviews have been very open-ended, and I've decided to find times to let them know what I think I'm hearing thematically, to let them reflect and even construct experience right there with me. I'll be glad to talk more about this method and methodology in another post! It's solid...

JL