Wednesday, December 07, 2022

Class Assignments in an LLM World

 There has been a lot of discussion recently of OpenAI's recent ChatGPT-3, a significant improvement over previous iterations of GPT-3, and its possible implications for college pedagogy (for instance, here). GPT-3 is often called 'AI', but this is somewhat misleading in this particular context; suffice it to say it's more accurately called a large language model (LLM), a program that uses machine-learning algorithms and a very, very large training set of texts to process and generate blocks of natural language on the model of the texts in its training set. ChatGPT-3 does very well at this; you can get some plausible passages in a variety of genres out of it, without much difficulty. On the basis of this, some people are worried that the college essay is dead and it will become impossible to assess student writing properly. A few thoughts on this.

(1) I hate to break it to you, but if this is your worry, your approach to writing assignment design is twenty years out of date. For my entire professional career it has been easy for students to get unique papers from a paper mill, and the expense of this relative to the usual expense of failing a class has grown so small that the expense itself is not a significant deterrent even for relatively poor students. These papers are generally higher-quality than anything GPT-3 and the like can currently produce and (given the limitations of the model) than it is likely to be able to produce even for the near future.

This problem can be handled in a number of ways that, while not perfect safeguards (there are no perfect safeguards against cheating), do provide some obstacles. For instance, chained assignments -- an assignment that leads into another assignment that leads into another assignment, as when one requires a series of drafts, or a preparatory and a main assignment, or a main assignment and an outline or presentation assignment -- provide some measure of protection, as do assignments tied to things specifically discussed in class that are based on one's own research, as do in-class writing assignments (when feasible) and highly atypical writing assignments or assignments that are structured to assess obliquely rather than directly. There are lots of other things. 

This sort of thing has been known for ages; many of them pre-existed the cheap paper mill. But academics are very weird about these things. All my writing assignments have always been structured with this kind of thing in mind, but it has frankly sometimes been a struggle. Faculty evaluation processes, which often require samples of grading, are often not set up to accommodate these kinds of features; it's a lot easier to do a grading sample portfolio if your assignments are all of the straightforward, easily cheatable kind. My taste for indirect assessment -- for example, having students explain how they would select and use a field trip to teach a unit of the course, or having them turn in an illustrated paper -- has on more than one occasion gotten me dinged on faculty evaluations ('busy work' was one of the complaints, once). Admittedly, the things that make it hard to cheat an assignment do often make it harder to evaluate for faculty evaluation purposes, and indirect assessment is tricky business, and I certainly would not say that my attempts have always been successful. But the point is that there has really been no excuse for simply doing "Write on this" assignments for a very long time; academics do them because they are bureaucratically legible and because academics tend to be obstinate people who like to do things the way they have done them. (I am not an exception to the latter, of course; it's notable that I still do weird indirect assessments despite the fact that it's one of the few things I know might get complaints when I submit my evaluation portfolio. Obstinacy is a survival skill in academia, and selective obstinacy is part of how the academic game is played.)

(2) Ah, you might say, but the big game changer is that this is free! Not really. In this sense, it's like translation programs; instead of just saying, "Translate this", you have to find a way to design an assignment that is more like, "Translate this and then do such-and-such", where the latter component is something the translation can't or won't do, and which requires the student to do something they are inclined to shove off on the translation program. It's not that it's always easy, but you should already be designing writing assessments in view of the fact that students sometimes try to cheat, anyway. And we should never underestimate the superhuman ability of most students to fail completely to use resources that are freely available, an extraordinary ability that extends to means of cheating as much as it does to means of research. If it's not showing up on the first page of Google, they often have no idea how to access or use it, no matter how free it is. I'm old enough to remember when you had to go to the physical library to do research, and while databases were already very common, depending on the library it was sometimes still easier to use the literal card catalog for some kinds of research. Students today can call up libraries on their phone -- still not quite good enough for graduate or professional research, usually, but easily good enough for undergraduate, and getting better every year -- and it would never occur to most of them on their own to use these resources, even when they know they exist. Cheating we always have with us; most of it is much easier to catch than you would expect given the resources that are available for doing it; and one can often put a wrench into even a more competent use of cheating resources just by a well placed twist in the assignment. It won't handle everything; but it's always been remarkable how much you can impede cheating just by a few minor tweaks.

(3) Nonetheless, we are at a stage where it's probably best to de-emphasize ordinary writing assignments, even setting aside cheating concerns: students often don't know how to write, and assessment by writing requires that students be able to write at least well enough that they can be assessed on what they are writing. I don't know what's being taught in high school English classes these days, but whatever it is, it's usually not working. Combinations of writing + presentation seem to do better, as do multiple-revision assignments. Writing assignments are not eliminable -- relevant use of writing is one of the things students need to learn, and is sometimes important to assess in its own right -- but it makes sense to diversify the nature and structure of assignments in ways that don't entirely lean on writing itself. Alas, I think. It is a sign of deterioration, one of a great many. But in teaching you have to work with what you have, even if it sometimes seems like students have been actively made less prepared for succeeding in college.