The trouble with AI

[eller]

AAARGH. I just wanted chatgpt's help to structure a text. You know - what should be in the introduction, how long should each part be for easy reading, and so on. Unsurprisingly, I'm shit at this stuff, but usually, the AI is of great help - at least when it comes to nonfiction with clear structural requirements. (Letting the AI write texts is, of course, hopeless, so I won't even try. Letting the AI organize text structures before I just write stream-of-consciousness stuff, however? I mean, that could save me some headaches.) Trying to let it organize fiction, however? Wow. WOW. Today, I learned that chatgpt is really Very Fucking American.

Things I learned:
- The AI will not just try to reorganize the plot around an acceptable novella structure (which, after all, is what I asked it to do) but flag any character behavior for editing that does not conform to American cultural standards.
- The AI told me that my characters are too obsessed with honor and duty and I should consider editing that. I'm like... WAIT... I'm actually writing a Fantasy!Medieval!North!Germany setting. With Fantasy!Medieval!North!German characters with according cultural background and mindset. (Come on. It's fucking Germany. At least some of the characters take their oaths seriously...) Apparently, Germany written by a German is not acceptable by genre standards...
- The AI completely unasked (!) changed a scene description from a male character making tea for the group to a female character making the tea. Thanks for the casual sexism, I guess.
- The AI described a female character as "flirtatious". She's... not. She is, however, speaking to male characters. In, you know, plot-related ways. Apparently, that's yet another thing the AI can't handle. (Not a problem with the technology itself, I know, but definitely with the training dataset. WTF.)
- The AI completely unasked (!) tried to give a genderfluid character an issuefic subplot centered around Gender!Angst!American!Style. I mean, I onbviously don't expect an American piece of software to understand historical German ways of gender expression... which is why I didn't ask it to. This character has a perfectly acceptable subplot centered around military technology and espionage, and.no gender issues whatsoever, thanks.
- The AI really wants to change the magic system (which is, of course, North German as fuck, considering the setting) to something ripped off Tolkien.
- The AI is shit at interpreting character motivations in ways that are actually pretty hilarious.

Thanks for the non-help. -_-

Comments

[yhlee]

I've yet to see a single use case in the creative fields.

Patrick J. Jones says that for whatever reason, the visual AI generators are genius at color schemes.

ETA: Perhaps you wouldn't define this as a "creative field," but in professional Go/baduk/wei qi, the defeat of Lee Sedol by AlphaGo was hailed in that community BECAUSE AlphaGo made wild-ass moves that looked completely bonkers to the professional human players. There was a lot of excitement because literally the AI was making bonkers-to-humans moves that worked and this was hugely energizing for the potential to improve human play and understanding of Go strategy: Many top Go players characterized AlphaGo's unorthodox plays as seemingly-questionable moves that initially befuddled onlookers but made sense in hindsight:[72] "All but the very best Go players craft their style by imitating top players. AlphaGo seems to have totally original moves it creates itself."[68]

I have to agree with Eller that there are a lot of people whose interest is not being artists themselves in XYZ given field but "just" playing with output in a given discipline. Even someone who is, say, a professional weaver may not actually want to go all-in on learning to draw manga-style art for tinkering with TTRPG character portraits. One might look in music at precedents like 19th century Musikalisches Würfelspiel music generators; in literature, this really looks like it's not qualitatively different, only different in scale/extent, from some of the Oulipo experiments/works.

my evidence is that my students can't do any sort of advanced math or mathematical reasoning because people decided that rote learning is bad

Counterpoint: I remember the entire "students shouldn't have calculators because it'll make them bad at math." But computational methods are hugely important in e.g. numerical methods in calculus or computer programming. I'm not materially convinced that having to statistically crack Vigenere ciphers by hand (not even a four-function calculator) improved my understanding of the number theory involved. Hell, while we're at it, studies strongly suggest abacus use doesn't materially improve conceptual math understanding and often "manipulatives" can lead to rote muscle-memory manipulation without improving understanding of the underlying mathematical systems. I think this is a much stickier problem generally than "AI is bad" can explain.

Regarding "rote learning is bad" - again, this is a much stickier problem in math pedagogy. You can include rote learning as a prerequisite for many kinds of foundational math; it can be necessary but one can still have people who can handle the rote learning and flunk out of conceptual understanding, see Liping Ma's Knowing and Teaching Elementary Mathematics for an absolutely devastating critique of US elementary mathematics teaching in this regard (with comparisons to similar practices in China where the teachers have less formal education but use teaching methods that tend to produce better math understanding conceptually, starting with basics like "well, what IS place value and why do we care?"

[sabotabby]

I'm not good enough at math myself to weigh in—I just know that the discovery method or vibes-based math education they're using here has resulted in students who are increasingly unable to think mathematically or understand statistics. Not that I was any good at it, or the rote memorization stuff, but they are worse than I am. To the point where they can't see a problem, figure out what they'd have to do to find an answer, and use the calculator in their phones to calculate it. The second there's a number, they turn their brains off.

I don't get TTRPG or other hobby use of GenAI either, to be honest. For me, part of the joy in that is community and connection, it's one of the few things in life that remains free, and no one cares if you're good at it or not. My GM is always using AI stuff and it drags me immediately out of immersion because the style is so generic and hateful and I can't help but think of the environmental cost.

[yhlee]

As someone with an M.A. in math pedagogy (if that even matters anymore), I think it's likelier that they would have been shit at it in a traditional rote memorization context but the failure modes would have been different and would have presented differently. Every generation in the US (speaking from the context I know) has had some random-ass math reform that attempts to be the magic fix ("New Math" - I still think Venn diagrams should be near-mandatory although most civilians are not going to need an axiomatic system introduction to associativity; CPM; Cuisinart rods and manipulables; Soroban and abacus; graphing calculators). The truth is that there isn't a generalized magic fix to "make people learn to think mathematically." The failure modes keep changing and the magnitude of the failure modes change as well. But fundamentally, the problem (in the US) is that engaging people in genuine mathematical problem solving is pedagogically non-trivial and does not lend itself well (if at all) to the kinds of multiple-choice assessments that have taken over in the educational system here.

Sure, e.g. South Korea has consistently had much better results in math education. But South Korea also has elementary schoolers signed up for hagwon / night school classes and sometimes doing up to 16 hours of study a day stressed out of their everloving minds because of the examination system. I don't know that I consider that an educational win either.

I don't get TTRPG or other hobby use of GenAI either, to be honest. For me, part of the joy in that is community and connection, it's one of the few things in life that remains free, and no one cares if you're good at it or not.

GMing is a lot of work; AI-generated dungeon text is different in scale but not in kind from the GM rolling off ten different dungeon random encounter tables. If I had worked as 60-hour week and was tired and we only had three hours for a session, I'd be looking for shortcuts too.

Or people who don't have a gaming group for whatever reason. Solo journaling RPGs do exist and some are terrific but again, we're now looking at a difference in scale rather than kind.

[sabotabby]

It's complicated, and I have dyscalculia and probably never would have been good at math. But at least I learned enough, back in the day when rote learning was mixed with problem solving, that I can kind of figure out what the problem is. I think there's some middle ground between whatever the US is doing, what Canada's doing, and what South Korea is doing.

Weirdly, the GM isn't using AI-generated text. It's just images, which are completely unnecessary and just for flavour. He's got two artists in the group. But also, I just don't regard any of these expressions—art, writing, gaming—as needs, so when I weigh it against the ethical horrors of AI, I just kinda go, "well, no one made you do it."

[yhlee]

I would argue from a design standpoint that 70% of any game is "game feel" or "flavo(u)r," but that's a separate argument. (Edit: "70%" is a shitty made-up statistic.)

It's not that I don't think there aren't ethical considerations around AI (licensing, resource use, monetization and interaction with venture capitalism and commodity tech). It's that I find a number of the arguments in this space are (as detailed in other comments here) frustratingly ill-formed, alongside the ones that worry me (generalized fucked human attention spans caused by, as you say, affordances; "where do people get entry-level experience to get to higher levels of experience if no one does entry-level anymore?" raised by castiron) weighed e.g. against AI as accessibility aid. I know of someone who uses AI aids for writing as a government-permitted accommodation for weapons-grade dyslexia (not in the US or Canada).