If a GPT detector has any false positives at all it will disadvantage people who are already disadvantaged. They are the least likely to be capable of defending themselves and the least likely to be heard if they do try to defend themselves.
Not to mention the fact that being called non-human is most definitely going to offend some people.
What exactly makes anyone think that they can detect an LLM that is outputting text? The notion seems absurd yet it keeps coming up.
> What exactly makes anyone think that they can detect an LLM that is outputting text?
When you read LLM output, you can often tell. The source of the notion is that we can do it pretty well ourselves, so if the AIs are so magical then they should be able to do it too (not saying I agree, but it is a pretty clear line of logic).
You can tell if it’s vanilla ChatGPT but the second you tell it to write in another style and tone, it changes completely. It’s very interesting that it sounds like a generic high school essay by default, but that’s without any extra instruction.
For example, ask it to write in the style of Christopher Hitchens, Charles Bukowski, or Hunter S Thompson, let alone more extreme examples like Shakespeare or Dante
It does have a lot of "shallow wells" where if you ask it to make a random list of something or pick a random example of something it will give similar answers every time. For example, whenever it has to come up with some sort of enemy or attacker or opponent it's very drawn to skeletons and giant spiders regardless of context.
Simultaneously, I'm also noticing that writing styles in general are beginning to look very ChatGPT-like, even if someone isn't using ChatGPT to write with at that moment in time. If they're a user of it at any other time, or consuming a lot of its output, they tend to mimic what they're exposed to. Just an observation.
My theory is that many “educated” people were never taught to write properly. Gpt is teaching people what they should have learned in grade school: how to write clearly.
I don't know about that. I think ChatGPT's output is very repetitive and fluffy, and their style thus becomes similar—not necessarily a positive habit.
That’s the kind of writing you are supposed to have learned in grade school. Not good writing, just clear enough to pass.
Good writing only comes from writing lots of things that other people actually want to read. Course work is poorly structured for that, it isn’t peer review.
My time on Everything2 saw a dramatic improvement between the writing style that I started with and what it had progressed to when I became less active/left. In that time I wrote nearly 800 nodes over the course of 4 years.
Having a diary is one thing - having creative writing that is in the open and subject to criticism and encouraging refinement is quite another.
I think the even more "high school essay" trait is it's tendency to infodump its marginally relevant general knowledge of the subject matter. Look at all the stuff I've learned, isn't it impressive!
And whilst it's bad at being nuanced, it's even worse at being opinionated because its guide rails and human testers alike love its answers to be qualified with generic caveats like "depends on the specific situation"...
A ChatGPT answer looks exactly like a politician that doesn't want to tell you the answer to your question.
It also has a "high schooler" style, but mostly because of the rigid form it uses. If actual high schoolers throw random content in their essays like ChatGPT does, they will get negative points for that.
Yeah. Ask it a question and it will tell you everything about it but never the exact answer you are looking for. It takes prompts like be concise and direct, or only give one word, or the like to make it anything better. Sometimes first asking it to give a small answer and then explain step by step works well though.
And that writing will be used to train future models until a stiff, logical proof/Rube-Goldberg style of writing becomes the standard everyone adopts (hopefully not).
With logical proof styles, that requires active thought, reasoning and rationalizing, amidst other things. It's easy to blag things with fluff and repetition, which is why so many people are hired out of their depth into inappropriate positions.
Average ChatGPT output for the most part reminds me of the average British politician who can waffle on until you stop them, talking about all the aspects of something, but never really answering your question with all that much substance and depth, sometimes not even answering it whatsoever. Very "30,000 foot view".
Is it teaching people anything? I suspect it's just enabling the "author" to spend even less time actively thinking about whatever task they're performing.
I’ve learned from gpt. Did it teach me? In the same way books teach me things, yes, it did. I could have learned the same things on my own with much more reading, but you don’t always know what to read when working on a problem. Gpt has read everything, and can tell you answers to questions you don’t even know to ask. Like “how do the complete works of two authors (I’ve never read) relate?” Sure it gives a nonsensical answer quite often, but that doesn’t really matter. I can verify what I need to verify on my own, just as I would any other unreliable source of information (like CNN, aljazeera, etc…)
I unironically sometimes reply to people in real life with "as an online AI language model, I don't have any intentions or opinions. Is there anything else I can do to help?"
+1
chatGPT style is so close for what I've been trying to reach since a while ago, also I am so vulnerable to style and phrase stealing, I am seriously concerned that people will think that I am a robot.
Simultaneously is redundant with "also": "I'm also noticing" and "Simultaneously, I'm noticing" both imply a reference to an external thought. You could drop 'also' instead of 'simultaneously' if you're worried about ambiguity in terms of the nature of your acknowledgement.
ps: I tapped out a message on my virtual keyboard, and didn't take my ADHD meds yet either. I guess where we differ is that I didn't use my condition to attack anyone for ableism just because they didn't agree with me.
In schools, my coursemates and I always plagiarised each other work. We also heavily do it via any other means like having a class pooled together funds for buying work done by outsiders. This span anything from quantum mechanics to even science fair projects. We ended passing many college course with near flying colors. What have we learnt by cheating? How to cheat better. We learnt how rephrase, rearrange assembly codes, reformat, etc. Some of us even develop program to do substitution and rephrasing. This was way before chatgpt. We did well with this skills as we apply it in corporate world plagiarising collegues works and even competitor a product. We even glorified how Samsung did it in early days of smartphone war era. I am sure my juniors in my high school and college will do the same with chatgpt. I can assure you we plagiarised and got away with it near 100% (unless some were lazy enough not to clean up their copy of plagiarise source material). Those that get caught will just have to learn to adapt. When you graduate from you college, ability to plagiarise is very important in your career (as long as non academia). You can tell if you know the persons tendency of dishonesty. If purely on work output, you are at best at coin toss or maybe those very low quality students. Back my days, we already have plagiarise detectors and even bots to detect plagiarised programming assignments. It is hard to tell. Chatgpt will adapt the output to be non perfect. Plagiariser will also adapt like my experience to de-chatgpt it. It has been done. It will be co tinue to do happen.
Out of genuine curiosity, what is your culture of origin? Would you say this perspective is widespread on your culture? Do you have a feel for whether it is more or less so than others?
The poster is Chinese. Cheating is absolutely pervasive in Chinese culture. They believe that either all other cultures hold the same views as they do about cheating, or that those countries and individuals are naive and stupid for not cheating.
> We did well with this skills as we apply it in corporate world plagiarising collegues works and even competitor a product. We even glorified how Samsung did it in early days of smartphone war era.
It's a questionable study. Giving the same amount of money to a person from the UK and China makes no sense given the difference in income between these countries.
Also, this conflates factors other than cheating, such as the respect respondents have for the people administering the test, the degree to which respondents treat this whole thing seriously, etc.
The fact that the study says what we'd expect, doesn't imply it's a good study.
FWIW, the 70% number of Chinese cheating students matches almost exactly with my own experience as a teaching assistant. I believe that most TAs with a sufficiently large class have made similar experiences. In general, cheating is much more rampant and organized than most people would expect. Most institutions just do not care because catching and punishing cheaters is a lot of work with little short term benefit.
As a student from Europe.. I find it pretty shocking. However, I even overheard some classmates who were also trying to cheat recently.
My first reaction to reading the comment was thinking if ethics are not at all taken into consideration.
However, if the poster is from China it makes some sense to me. The educational system in parts of china is extremely competitive starting at a young age even.
anonymous internet poster claims to be a cheater and glorifies samsung -> he's definitely chinese
british guy publishes study which finds british people are the most honest -> definitely true, british people never lie, even though psychology is the most outrageously and rampantly un-reproduceable field of study
Honestly, some classes are useless """requirements""". Most classes I wouldn't consider cheating though, because in that case the only person I'd be cheating would be myself.
At university, my degree had this mandatory software engineering course for which there was a group assignment. You had to pick a project and "implement" it as a group. You weren't expected to write any code though - only produce a requirements spec, design spec, project plan, UML diagrams, etc – the group was marked on those specs/plans. You had to choose a software development methodology and justify why it was the right choice for your project.
The lecturer told us very clearly "DO NOT LOOK AT PREVIOUS YEARS ASSIGNMENTS". I was a little naughty, I asked a friend who did it the year before if I could have her group's assignment to read, because they'd got near-perfect marks. I didn't copy anything from it, I just read it to get a better idea of what was expected.
One guy in our group, he was really late to contribute his section of the assignment. When he finally gave it to me to read, I realised he'd just copied my friend's assignment from the previous year – either she'd given it to him also, or someone else in her group had. He tried to cover up his copying by rewording sentences, but it was very obvious – every paragraph made the same points in the same order, using the same (or very similar) word choices. Added to that, a lot of his attempts to reword it, the end result didn't even make much sense – some sentences, all he'd done was transform good grammar into bad.
I was angry and felt like reporting him. But, the only way I knew he was cheating was because I'd looked at a previous group's assignment, against explicit instructions not to – I couldn't see a way to get him in trouble without getting myself in trouble. In hindsight, I probably could have shown it to my friend and then got her to claim she'd spotted the plagiarism instead of me, but I didn't think of that at the time. Instead I just threw it out and redid it from scratch myself. He got the same marks as everyone else on the group assignment, without having contributed anything except a poor attempt at plagiarism.
Until one gets caught. Teaching at SNU in Korea -- my students among the brightest minds of their generation -- I was shocked to catch a rank plagiarist, who copied most of an essay out of a book s/he believed beyond easy checking. Suffice it to say that student will doubtless always regret being too lacking in academic integrity to complete a project fairly. Cheating is the hallmark of an inferior intellect.
What a travesty. Why even go to school, then? Are getting good marks that much more valuable than the actual education you would receive if you didn’t cheat?
Any institution that produces this outcome has its priorities completely out of whack. And any student who willingly and energetically participates in such a system must be completely vapid, short-sighted and materialistic.
It would be a real shame for your country if this kind of behavior were common throughout. It would surely result in a situation where the most powerful and highly esteemed are without integrity.
Especially with low level degrees, fewer associate going to college with bettering oneself as much as meeting a checklist for a job interview or a parent's requirement for continued financial support. Many of these jobs don't inherently require being educated it's more a "well, why wouldn't we hire the person with the degree if we can" kind of situation and jobs tend to drive the family requirement more than the "fluffier" stuff about education. In these cases going to college is seen as an required obstacle by the student, not an opportunity. Any way to get out of as much of it as possible is valued more than going to learn. After all, the interest was in meeting some external requirement, not getting educated. Not to mention, for a great deal of people (by volume) getting a degree at all is an extremely difficult, if not impossible, task. Getting one and having high grades to stand out can be seen as having more reward given by society than genuinely getting a degree and having poor grades.
It's not great by any means, obviously more people interested in educating themselves is better. At the same time, it shouldn't be a mystery. It, particularly, shouldn't be something one assumes is related to a particular country/culture rather than an everywhere thing. That's not to say there won't be variance between location, rather that it's a pretty common outlook everywhere.
Also, the ability to appropriate someone else's work to further your own goals is actually a very useful life skill. It is arguably the foundation of capitalism.
In liberal economies you buy someone else’s work. In communism it is appropriated by the state. Source: I have family who grew up and lived in a non-capitalist worker’s utopia.
Sure. But people pay to use GPT, and people pay to have others do their homework for them. Compensation is orthogonal to the question of whether someone else did the work.
Someone else baked my bread, made may car and built my house and I paid for all of them. So are you saying a reasonable description of the state of affairs is that I appropriated those things?
Liberal economics is based on several freedoms. To sell one’s labour, to buy the labour of others, to own property and to freely associate with others on economic activities. Put together, and with a fair legal system to regulate it, and that’s capitalism. Which of those freedoms do you disagree with and why?
> Someone else baked my bread, made may car and built my house and I paid for all of them.
That's true. But, with the possible exception of the bread, you did not pay the people who did the work, you paid someone who appropriated their work and sold it to you as their own.
And that person paid, someone, who, paid someone, etc, who paid them. As it happens, I’m directly paying people to work on my house right now.
I’m reading definitions of appropriation, and I don’t see the applicability. The bit about ‘without permission’ doesn’t seem to apply. I’m not taking anything, they offered their labour for sale.
Can you describe how a civilisation might function without appropriation as you define it?
There isn't a word in English that expresses the meaning I'm trying to convey. The point is that if a student hires someone to do their homework for them, that is considered "cheating" while if an employer does the same thing it isn't. If you have a suggestion for a better word to describe what the student is doing that captures the negative connotation associated with "cheating", I'm all ears.
The word "appropriate" as a verb in colloquial American English most often implies without permission or compensation.
Certain domains, such as budgeting, use it as a synonym for "allocate" but given the context it was first used in your comment it definitely implied "taking without consent".
Yes, I know. The point I'm trying to make is that there is no salient difference between an employer hiring an employee to do the work and a student doing the same. Students typically do not actually steal the work that they pass off as their own, they usually pay for it. But for some reason, when a student does it it's considered underhanded whereas when an employer does it it's considered a normal business transaction.
This carries the assumption things like "school work" and "business work" are inherently fungible on the basis they are both types of work. Dropping the assumption for a moment, it seems reasonable there may be more than one goal of or type of work, and that these goals/types might not be interchangeable with each other. This view of the difference is only possible if you start with the earnest question of "what's the difference supposed to be" rather than asserting a conversation along the lines of "these are the same, so why are we treating them different".
Taking this approach, many might find school work is, largely, effort done to grade a level of understanding the individual has of the content. On the other hand, one might find business work is, largely, effort done to accomplish a customer's request. Additional less common types of each may come up as well, and there are, of course, exceptions to the main theme of each. In general though, I think you'd agree swapping the goals to say "school work is effort done to grade whether someone understands the content while business work is effort done to see if the particular person you make the request to is able to complete it on their own" seems rather unlikely in comparison. Because of this, swapping who does the work in each case results in different treatments. Not because it's a question of whether work gets done rather what the goal in doing the work is.
> This carries the assumption things like "school work" and "business work" are inherently fungible on the basis they are both types of work.
No, the assumption is that the purpose of education is to prepare you to be able to function in the real world. IMHO learning how to get other people to do your work for you is the single most valuable skill one could possibly acquire.
There can be more than one assumption carried so why not judge each on their merits instead. Taking this other line of thought, it could definitely be truer in a vocational college or work placement type education but less so in a typical college education. Traditional college doesn't even come with the assumption you'll ever leave academia to go out into "the real world" to make your living, or that you have an actionable goal beyond being interested in the first place, let alone the assumption it should be the only goal ever considered. An easy way to see this take is not the actual primary goal of college is the self-apparent conflict of such activity being seen as underhanded in the first place.
Even if we took this line of thought as the truth though, also came to an agreement this is the most valuable skill of all to teach, and also took it as being universally good to do regardless of context, what's the reasoning for assuming college only seeks to teach and grade success on a single skill in the first place?
> Traditional college doesn't even come with the assumption you'll ever leave academia
Yes, and this is exactly the problem, because in point of actual fact most people who go to college do eventually leave. And if you think about this even for a moment, it has to be that way because someone has to do the actual work required to maintain the the civilization that makes college possible in the first place.
It is interesting that in Finnish university there was actually possibility to enter a track aiming at PhD from the start. With more rigorous math and other things. So you could work towards just Master's Degree or PhD from start. With different course load.
> Are getting good marks that much more valuable than the actual education you would receive if you didn’t cheat?
The problem here is the incentive structure. The nature of grades is that some people will get As and some people will get Cs. If you get Cs, you might like to get As, but maybe you can't for some reason, from lack of intelligence to other time commitments to alcoholism.
Then your choice is to get a C on your own or maybe get a B+ by cheating. Given the same amount of learning, getting a better grade is better, so the only real incentive not to do it is the risk of getting caught. Some of the old methods for this were pretty unambiguous. If you submit the exact same essay as another student, or one that has been on the internet for ten years, what other explanation is there?
Rich kids would avoid this by paying someone else to do it for them. That has always been a problem. But now the cost of getting "someone else" to do it is approaching zero when the someone else is AI, so the problem spreads. But attempting to detect it with methods that have false positives is worthless.
On the other hand, testing students on their ability to do something that ChatGPT can do just as well? Maybe that's worthless too, because that's no longer a marketable skill when your future boss can get ChatGPT to do it too. So what they need to do is change the test to test for the thing the student is expected to be able to do better than the AI.
Have you taken some time to play with these things? Try the biggest LLaMA model that will fit in RAM on your computer. (128GB of DDR4 is around $200 and will just fit the big one without quantizing, though it won't be super fast.)
There are things they're good at. Search engine-like tasks in particular, if you're willing verify the output. They're great at providing hints for further reading.
Now try to get it to develop a new kind of battery with a longer service lifetime or lower manufacturing cost per unit energy storage. Ask it to write code to do something complex and uncommon instead of something similar to what it was trained on a thousand examples of. Have it describe a new class of security concern, like Spectre or rainbow tables before they were known.
People can do those things, and have done, and those are some of the best things we can teach people how to do because they're incredibly useful. Maybe writing minor variants on common existing boilerplate code isn't something we need people to spend a lot of time on anymore, and so isn't the thing we should be testing if they know how to do.
From talking to a handful of people it is usually to fulfill someone else's expectations. If the goals aren't aligned ethics don't matter, the student doesn't want to be in that school anyway.
Cheating is the easy way to get away from that situation. And if the student is behind, it's the only way.
I don’t disagree with you, though personally I’ve called out two people on Reddit using ChatGPT to farm karma by pretending to be an expert on some topic… and an hour later they deleted all of their GPT generated replies from their profiles.
There’s definitely a pattern for some prompts, where the model uses an obvious format of:
- Make generalized statement answering prompt question
- Support that statement with 3 or 4 discrete paragraphs that don’t exhibit any personal experience, give examples, or cite statistics
- Finish off with a high-school essay style conclusion statement reiterating the introduction with different phrasing, generally beginning with “Overall” or “In conclusion”
It’s also comically obvious when suddenly someone who has a reply history full of low-effort juvenile, zero grammar, curse laden posts on video game and meme subs is suddenly writing mini-theses on topics ranging from Swedish forestry management to the chemistry of dyes used in Cambodian textile manufacturing.
There's definitely a "what are the benefits of" or "what are the reasons for" format that ChatGPT falls into pretty easily. It's usually not really wrong and would probably serve as a passable high school essay. (And I've probably seen worse marketing copy.)
But it's not very good and, at a minimum, lacks nuance and supporting evidence.
The key word from GP is "reliably". Sure, you can probably tell when someone blatantly copied and pasted output from ChatGPT and when the prompt was not very sophisticated (write a comment about X). However for all other cases when someone put even the tiniest bit of effort into the prompt or did some post filtering the reliability of such detection plummets significantly.
Verbatim copies are often not that hard to detect in a large number of cases; there is a tendency to waffle excessively in a way that few people do. This is of course not a fool-proof method, but it's "reliable" in the sense of "you can get it right more often than not".
Edited LLM text is much harder: using a LLM to generate some text and then edit it in shape (often by removing extraneous paragraphs, maybe rewriting a few things slightly). Those are basically impossible to detect reliably.
Not really. I think people don't understand that this is kind of chicken and egg problem.
You don't see overuse of linking words because it was generated by LLM. You see them, because every non-native spear is literally taught to link every paragraph with them. And all the texts, blog posts, wikipedia articles, stackoverflow responses and all the other stuff they wrote was then used as training data, from where LLM had learnt to do the same.
What I am saying is that there are just much more non-native English speakers and LLMs are inherently kind of non-native speakers too. So a sign that distincts majority (non-native) speakers from the minority (native speakers) is actually a bad sign (:
I think that vanilla GPT writes in a style that is easy to identify. There's also this "too-perfect" feel where in most forms of communication, most people make mistakes. They splice commas, they use the wrong word, they subtly misspell something, they kinda lose their thought process, etc.
Especially as a teacher, you get used to seeing the, uh, quality of work that gets produced. I mean that for college teachers too. At my decent state college, the quality of essay writing was atrocious and I was given 99/100 for basically "not needing any help". In college!
When you compare the average GPT output to what many students turn in at all levels, the difference is highly apparent.
This also opens up the reality that smart cheaters will prompt engineer. I just tried on GPT4
> You are an average 10th grade student. You read and write at a 9th grade level. You don't edit your work ever and your writing has run-ons, comma splices, misuse of grammar, misspellings, etc. Say OK if you understand
> Write a 5 paragraph essay about spring bird patterns in Alabama. Use an intro paragraph and a conclusion paragraph. Keep it around 500 words.
> You are doing a good job of being a 10th grade student, however I want you to do the previous command like a 10th grade student who has been trained in basic essay writing and does not write conversationally.
And it's already producing something that is far more believable lol.
> Especially as a teacher, you get used to seeing the, uh, quality of work that gets produced.
There was a time when I had to discuss a paper that was too well written with a student. The paper was too well written in both my eyes and the eyes of others. The student told me which journals they read and explained they read it for both content and style. Were they cheating? Perhaps. On the other hand, I have never encountered a cheat who cared enough to make it look like they cared about how they approach learning.
I got accused of cheating once in a college intro class because my paper was ‘too good’. They had no actual evidence, it’s just that I spent a lot of time on it. It really pissed me off and was one of the reasons I took only the bare minimum of non-math/engineering classes. In math no one ever accused me of cheating if I came up with a clever proof so it felt a lot more respectful.
Yeah, it is a very sad state of affairs when a genuine effort is considered suspect. The thing is, so few students put in the effort and many of those who you do talk to are upfront about expecting a certain grade. It very much taints the expectations on both sides of the fence.
This would be much more interesting if you posted the output directly, or linked to a pastebin service with it on. The context is pretty key to deriving value from the comment.
For the record: these tools are open and available and it takes about 10 seconds for you to replicate my experiment. The value you derive from my comment should be the idea I give you, not the results I generated. My results are good but imperfect, and are the result of very little effort. Someone truly trying to cheat the system could produce way more I imagine.
I am well aware that these tools are available, and perhaps if you paused for a moment, you might think that someone who already uses these tools might have hit their limit of requests for the next x period, or doesn't want to use limited resources recreating that which has already been done and is trying to better understand the context of your comment. Otherwise, why bother making the comment in such detail?
The results you generated are as important as the idea because ChatGPT does not output the same thing every time. Sometimes the output is very useful, other times it is complete and utter drek.
Method without results from that method are fairly meaningless if you're trying to compare.
Arrr! When ye lay yer eyes upon LLM's scribblings, ye can oft discern its origins, matey! The notion be rooted in our own knack for the task, makin' us reckon that if them AIs be truly magical, they should fare just as well. Not sayin' I be agreein', mind ye, but 'tis a line o' reasonin' as clear as a starry night on the high seas!
I first asked it to reply in the style of Paul Graham - the idea being it would be like a good HN reply, but not obviously a bot. But sadly it did a terrible job of that - rambling on, So I got it to do a terse version, and still bad. So I then said "as a pirate" and this is what came out.
or maybe they translated the parent comment into pirate speak on their own, which would not be difficult, to see if anyone accused them of being an LLM text.
I think ChatGPT is trained to speak in a certain tone. I played with other models before, and they seem to not have this issue, and they seem much less distinctive for me.
This is the thing that kills me - people who think they can reliably detect LLM output are fooling themselves.
At best - you can look for logical fallacies and false facts in the output, and use that to guess, but realistically - people are fucking bad at detecting it, including all these HN posters who keep claiming they can do it reliably...
Yes but the sort of logical inconsistency the machine has is idiosyncratic. There can be some types of contradictions few words apart that it doesn't pick up on. For example, asking it to sort a list into n mutually exclusive categories, it often repeats categories or has the same item in more than one category or completely omits an item or even sometimes lists the same item twice within the same category. You might catch bullshitters making up matters of fact, but no person is oblivious to error of basic task like that.
Yep, GPT's default "mode" seems to be the bland "five paragraph essay" encouraged by high-school courses. First, it tells you what it's about to describe. Then it describes it, inventing some supporting evidence. Then it restates what it just told you.
Sadly many students never move beyond this introductory framework, so their professional writing (which GPT trained on) ends up with the same uninspired style. It's a way to automate the creation of content that even the "lowest common denominator" of students can manage, but it doesn't produce good writing.
Except people are actually terrible at detecting LLM output.
Worse - The entire point of the LLM is that it's generating its output by picking the statistically likely next word... So from a "analyse text and detect forgery" side of things... any company that claims they can do it without false positives is fucking lying through their teeth.
We may or may not be able to detect whether LLMs wrote something, but I'd bet my life savings that when an administrator at the department of agricultural sciences and natural resources asks ChatGPT whether a particular essay was written by GPT, there will be significant numbers of false positives and false negatives.
Probably the fact that if they admit the reality, they have to think about some difficult and profound questions. It's much easier just to posit an imaginary future technology and decide that will solve it.
> What exactly makes anyone think that they can detect an LLM that is outputting text? The notion seems absurd yet it keeps coming up.
My sense of the general idea (non-authorative): Since the sequence emitted by an LLM is probabilistic completion i.e. predict the next word, the examiner can also do the same by progressively processing the text. Given the assumption that the semantic relations extracted from training corpus should be fairly universal for a given domain at the output level (even though distinct LLMs will likely have distinct embedding spaces), then the examiner LLM should be able to assign probabilities to the predicted words. The idea is that a genuine human produced text will have idiosyncrasies that are -not- probabilistically optimal and the examiner can establish a sort of 'distant from probable mean' measure, with the expectation that LLM produced text should be 'closer' to the examiner's predictions of 'the next word'.
The problem (if above is correct) then is the missing 'prompt' and meta-instruction embedded therein. Those should ("engineering") affect the output, possibly skewing the distance measure, thus defeating the examiner. But of course, say in context of academia, the examiner can 'guess' as to some aspects of the prompt as well. For example, if you are examining papers for a specific assignment, the examiner can self-prompt as well. "An essay on Hume's position on the knowledge of the self".
That only works if the temperature setting is low. If you set it to 1 the it will pick something that’s 1% likely 1% of the time, for example, which should match human text
Definitely, should have mentioned that. However it is interesting that using temperature may lower the quality of the output and you may get a C and not the hoped for A+.
Could you run it once to get the A+ version, then feed it back that version saying "change x% of these words to be unlikely choices in a way that keeps all the meanings of the essay" to avoid letting the low-probability words hamper the main contents, just the way it's described?
As far as I know lower temperature doesn’t necessarily mean higher quality, just more determinism and less creativity so whether that’s desirable depends on what you’re trying to generate
Temperature above 1 often results in nonsense though
> it will disadvantage people who are already disadvantaged. They are the least likely to be capable of defending themselves and the least likely to be heard if they do try to defend themselves.
Can you come up with any type of system that this does not apply to?
Any system designed to help disadvantaged people that has hit the mark.
The point being made is that the idea of detecting AI text is absurd. It won't work it can't work. The point about it having an increased negative impact on those who are already at a disadvantage is important.
Smart people don't need additional barriers to becoming benefits to society and promoting any tool that creates those barriers and does literally nothing else is terrible. And no, not all systems meet this criteria.
It’s unfortunate that the economics involved means that the purchasers of such tools will likely want a false negative rate of 0, likely at the expense of victims of such tools suffering the resultant false positive tradeoff.
You may want to read on history of polygraph. Notion that a tool giving yes/no answer needs to have it's answers relate to ground truth is very academic and generally frown upon by gatekeepers.
Prior to GPT, word processors would catch when your grammar was off for multiple words and give you suggestions to fix it. As students begin using the new GPT enhanced word processors, these suggestions will only increase.
Professors have a tough job ahead and they aren't always smart enough to use these GPT Detectors[1].
I think one solution would be a word processor that records the process of writing a paper and you need to turn in your paper along with your recording. Of course this is going to create added stress but what else do we do? There's going to be GPT scramblers that remove watermarks.
> Or, instead of surveillance dystopia, we just start having in person tests again, if we want to know, what a person on their own is capable of.
Why would I want to test someone's performance at say writing an email without the benefit of things like a spell checker, or grammarly style stuff, or chatgpt. They'll have access to this in the real world.
How about we test realistic scenarios. Instead of asking someone to calculate 96*451 by hand, ask them to workout the difference between buying a $700 phone and $5/month sim contract vs a $30/month contract for 3 years with RPI inflation. Give them a pop quiz to choose which of a dozen screaming offers in the supermarket aisle is the best thing with 10 seconds to decide, for added realism throw in a couple of kids causing some distractions.
For added credit have them consider opportunity costs of that $700 upfront payments, not just what they'd get with it sitting in a bank account. Instead of have them write a 1500 word essay by hand, give them 2 hours to write 1500 words explaining the benefits of X using normal tools like a modern word processor, google, wikipedia, ChatGPT etc. If they don't use the tools at all, you'd probably need to mark them down.
This is a false dichotomy. I feel like this well-worn talking point might be outliving it’s usefulness even faster with the invention of LLMs. We teach the mechanics of arithmetic and of spelling and grammar because that’s what education is for, it’s for teaching how things work so there’s an understanding of the fundamental mechanics, and it starts with the basics and builds on top of them to more advanced topics, in order to deliver a well-rounded and deep understanding. Note that the logical extension of your argument is to let ChatGPT, or the next AI, or the one after that, to start explaining your examples, and let humans ignore everything the computer can do, which now includes writing 1500 word essays. After all, just like spell checkers and calculators, access to AI is what people have in the real world.
We don’t need to choose between teaching multiplication by hand and how interest works, because we already teach both, one in a basic arithmetic class, and the other later in algebra & calculus. Same for spelling vs grammar vs writing. Spelling and grammar happen in elementary school, and essay writing happens in high school and college. Teachers already do allow calculators in algebra and calculus. This has no bearing on whether we should allow calculators in arithmetic class based on the vague notion that having access to calculators at all times is a ‘realistic scenario’.
I feel like this kind of thinking is what is leading people to try to cheat in the first place, it’s a lack of understanding of the value in basic skills, and the misguided assumption that learning something only has value if you can demonstrate you would need to use it all day every day for a job right now. The problem is it doesn’t ever get better or easier if you skip the basic mechanics that computers can do; in fact it only gets harder to learn the subjects that do actually matter that you are going to be using, when you have foundational gaps.
More than ever, what we are going to need from here on out of the education system is people who know at least a little what AI is on the inside just so we can use it effectively, not to mention build it, maintain it, control it, set legal policy, fix it when it’s wrong, etc. etc.
Right. Schools will become more like tech job interviews. Let’s be frank, that’s what higher education is anyway. An employment filter.
So students will white board their term papers and do other absurdities totally removed from any actual real world job requirements. Because in the real world everyone will just use gpt.
How about judging the content no matter which tools the students are using? If ChatGPT output cannot be distinguished from a student's output, then the respective profession / line of study is soon going to be obsolete anyway. If there is a difference (there should be one!), then the LLM is used as a tool, which should be allowed for most purposes. Universities don't disallow style and grammar correction software like Grammarly either.
What I can see in the humanities at least is that evaluation will soon get away from subjective and superficial criteria of form and focus on the quality of the content again. (That's me being optimistic. The other alternative is that AI will soon evaluate the output of other AIs.)
Is Dr. Jared Mumm still employed? How did they ever get a teaching position when they can't spell, conceptualize, or appreciate failure?
I think your solution works but the idea of even less privacy, especially around thought processing, is worse than the problem solved. I am guessing this will be the solution and it will send the data to Google and or Microsoft. Terrible.
They can also use the computer belonging to any family member. In a better world there should be cameras everywhere so we can detect a student not studying as much as the result they bring would require.
Ideally though, mind probes should come into play to find out if they really produced the work.
Honestly after some of the ill-advised-in-retrospect all-nighters at university during which I wrote assignments, I don't think a mind probe would be able to tell much at all!
You can actually detect this. Chess.com detects things like this for cheating chess players. The idea is that people who are cheating at chess with a computer have several identifying characteristics even if you ignore the moves. Things like the time spent per move, switching to a different window or tab, etc.
You could probably do something similar with recorded sessions like what GP suggested. Even someone doing what you suggested could leave behind a distinct profile in the shape of the session.
> What exactly makes anyone think that they can detect an LLM that is outputting text?
Watermarking.
On each word of the output, you randomly split all possible words into two groups and only generate output using one of them. If you get a text of 1000 words that exactly follow the secret sequence of the groups, you can be sure that it's generated by this LLM with one in 2^1000 chance of error.
Watermarking only works if every LLM system available does it and it is impractical for third parties to spin up and use systems that don’t, otherwise it only detects content that opts in to being detected.
If its detecting non-native English as GPT produced then wouldnt that mean GPT is modeled on non-native English to produce similar outputs? If it is NOT detecting native English as GPT produced then wouldn't that mean it's showing up different than the training data outputs?
It's funny I'm being downvoted. Probably out of a sense that I'm ignorant. Yet I've worked on large data labeling teams. And if you think anyone outside of the military is paying for very large native English speaking teams to label data I got a bridge to sell you. No one outside of special projects are paying large teams the wages of first world country standard.
well the system we live under doesn't give a crap about how absurd a notion is, and surely doens't care about educating people against it. as long as it's profitable it's gonna be made. In multiple scenarios it makes a ton of more sense economically to make the misinformation pool larger, and as a person who values truth and knowledge and the free access to correct information and believes that this will be the force which drives our species further, supporting this system is a contradiction. maybe i am a commie now? How would I know? The west has dismantled all the other systems to the point it is impossible for me to imagine living outside it.
Could you do the stupid thing and have openai et al offer an API that, given text (or a hash of some text) returns whether that hash was generated by the service? Some 3rd party companies specialize in cheat detection and pay for the API then get paid by schools or whoever to detect detect cheaters.
Difficulties I can think of:
* Getting ai companies to offer this. I don't think it comes with a downside for them really though. You couldn't use it to actually retrieve results in any way, and you wouldn't have prompts.
* This only detects exact matches, people changing the output would defeat it. Fixable with some kind of fuzzy search that returns a 'distance to nearest response' but this obviously makes it more expensive and difficult to run, use, etc.
* People could still run the model themselves, as models get better and more expensive maybe this becomes less of a problem. Or maybe models get small enough while still generating good output that it becomes more of a problem. Who knows.
Scott Aaronson (who is temporarily working for OpenAI) proposed a cryptographic scheme where the generated text can be watermarked without any decrease in quality.
It has most of the same problems you list, except it is much more robust against small changes to the text.
As of a few weeks ago, he mentioned that OpenAI had a working implementation and were discussing whether to start using it. I assume they'd tell people before they turn it on in prod, I see no advantage in secrecy.
This will catch only the laziest and stupidest cheaters. Crafting an argument is the hard part of an essay. A student can rewrite an essay, leaving not a single word in place, but still be guilty of plagiarism because the argument is the same. This is hard to prove even with the two documents side by side. It is harder still when the source could be any of a million documents.
Of course, many cheaters are that stupid and lazy. People still just copy and paste essays they found online.
It was already true before ChatGPT that if you are willing to rewrite an essay you can easily cheat without getting caught. Just find an essay online and rewrite it.
> As of a few weeks ago, he mentioned that OpenAI had a working implementation and were discussing whether to start using it. I assume they'd tell people before they turn it on in prod, I see no advantage in secrecy.
Watermarking has zero-to-negative-value for the user of the generation service, but value for users of the detection service that the common vendor of both services will sell, so the only reason to announce that watermarking is active is because you are ready to sell the detection service leveraging it. Otherwise, its just a disincentive to some users to use the generation service with no upside.
Sure, but that's true of lots of things OpenAI does. For example, they removed a bunch of functionality from their API for the new models, presumably for safety reasons.
Even if ClosedAI doesn't get government to ban sales of powerful GPUs to the public. Homebrew models will not be of comparable quality in near future. Don't be fooled by how cheap ChatGPT is, it is not profitable.
Sorry I don't think if you need to wait half an hour to see a result it's the same thing really.
You don't need to worry about deluge of AI content if it can only come from a few crazy people with top of the line hardware spending days iterating on prompts. Especially if really good GPUs start requiring a license. At some point it's just faster to learn to write properly.
We already have publicly-available models that are good enough for spam and scams. When running on CPU they are already faster than most people can type.
The cat is out of the bag. While the "AI alignment" jackasses were writing their Terminator fanfiction and wringing their hands about paperclips, they had already destroyed the world wide web as we know it.
Keep in mind the topic, it's not about WWW which can be totally swamped by a handful of dedicated malicious actors with powerful hardware, it's about verifying applicants that they did not cheat.
These models you talk about are either not good or unbearably slow, the output of a model that today runs "as fast as you can type" on average hardware will never be reliably mistaken for the real thing. If you try to cheat with it it would be more likely to fail you than if you spend $20 on a human freelancer to write stuff
The only factor that breaks this today is cheap availability of chatgpt and such. They are reasonably high quality but unprofitable to run, they are subsidized to hook public up so that later MS can safely jack up prices (ideally after getting an exclusive AI license from the government).
I could, and actually I did, and if I did not then I will now, say the same things about image generation today. Unless you know something I don't know or have top of the line hardware, general purpose image generation with homegrown models is either unbearably slow or poor quality.
You can run Stable Diffusion on an iPhone 11 and it completes in under a minute or two. Running on CPU generally takes around 5 minutes. My almost top of the line macbook runs a batch of four in around 30 seconds on Metal, and I'm sure it's much faster with a mid-range GPU considering how unoptimized Metal is with Torch. And yes, you can go take a look around reddit and 4chan, the vast majority of those aren't dreambooth/MJ/remote models.
That's not even taking into account local LoRAs and scripts that are possible instead of some company's untweakable crap. The open source around this is healthy, has pushed past DALL-E, and there's no real roadblock to Open Source LLMs except of course, the training cost. Even still, people are getting $200k+ models in their hands for free from various training runs and donated computing and LoRAing them and fine tuning them all to make them comparable to the closed off remote models.
Any "cryptographic" scheme with the generations of these will just catch the lazy. The lazy already include the confabulated sources in their papers, and don't try to normalize the Error Level Analysis in generated images (probably the quickest way to determine whether an image is generated), so I don't think it's actually a net benefit. It's a cat and mouse game, and will push the mice further into the walls.
You can't possibly say that generative images like this are "poor quality"
Again, poor quality or slow on conventional hardware.
The first of your examples was generated on a desktop computer with 2080 Ti and even then still glaring uncanny hands. We don't know how long it took but I think the reason for the hands is that it's too slow to generate a dozen of these in hopes that hands would come out right.
The other one I can see done on any laptop in a few minutes, but it's more primitive and just a monochrome sketch. I skip over obvious issues e.g. with shape of glasses.
For both examples you don't need any specialized tools or watermarking to notice this stuff.
Maybe you see what I mean why indie homegrown AI is not such a big deal ;) Sure there are people who will invest in hardware but those people will are not and for now won't be mainstream enough to matter. Especially if it will be licensed, most people don't like to violate laws. Most people will just use chatgpt or dall-e.
I don't see what your point is. The regulatory capture going on right now with the attempt to license things is ludicrous and akin to licensing matrix multiplications. It won't stick. Stable Diffusion and Approximated Functions (neural networks) are not something magical despite the fear they want to impart on them.
Commercial AI has all of those issues you mentioned, and more, and less. Midjourney is just a bunch of LoRAs layered on top and scripting to generate the images. But since they do that, midjourney images have a specific "feel" that it can't seem to get rid of. It's nothing really out of the reach for someone sufficiently motivated to reproduce.
DALL-E is laughable now, and it's only been a year. Certainly has been surpassed by open source, and outside competitors. I'm not sure what your motivation is to discount open source. People are already running LLM inference on their phones.
My point: 99% of people will use chatgpt etc. because homebrew alternatives are either bad (easy to detect with naked eye) or slow. Probably Microsoft will also make sure no competitors can offer good enough AI by pushing for regulation. So if those big platforms are required to watermark/detect own AI results that's good enough. Remaining 1% of crazy people don't count.
That midjourney et al are also detectable to the naked eye.
Why do you need to watermark them, again? The error level analysis is off the charts with generative images. They light up like a Christmas tree. Just because you and uninformed legislators and journalists don't know how to check the ELA of an image, doesn't mean they're undetectable. And the cheaters include the bogus sources spit out by ChatGPT already. The cryptographic qualities will be lost as soon as an editor gets their hands on it, automated editing or not. It's a cat and mouse game.
And also, I find it telling that you think if someone doesn't have high end hardware, they're going to pay $20/mo to OpenAI. For $20/mo, you can buy a mid-range video card and write it off. For an extra $10/mo, you can deprecate the cost and buy a high end laptop for that price, if you're a professional, and you're not locked into OpenAI. You're also assuming that 1. hardware doesn't get better and 2. techniques don't improve to run them on limited hardware.
> That midjourney et al are also detectable to the naked eye.
Again, either too slow, requiring outrageous hardware, or obviously noticeable. So far no examples to the contrary.
Don't forget, the topic is using special measures to detect undetectable with naked eye. When you can simply see the screwed up hands on a photo it's not even necessary.
> Why do you need to watermark them, again?
Why do you think I need to watermark them again?
> a mid-range video card
and a PC to put it in, a space to put the PC in, etc. With a laptop we're back in wait for an hour to see a result.
> hardware doesn't get better and 2. techniques don't improve to run them on limited hardware
We can revisit this if it consumer hardware gets good enough...
>With a laptop we're back in wait for an hour to see a result.
Any laptop within the last five years with decent memory can run stable diffusion on the cpu in around 12 minutes. My MacBook Pro runs a batch of four on Metal in around 30 seconds.
>We can revisit this if it consumer hardware gets good enough...
I mean, I just showed you a quantized llama running on a Pixel 5 and 6. And, I wouldn't discount most of the next generation of hardware having ML co processing like MacBooks and iPhones and Pixels do with all of this hype.
Apple's neural engine has been around in phones and laptops for years. Not even close to what's required for offline chatgpt. What's your point? Which of us is fooling themselves?
Average consumer hardware is not showing any signs of being capable of this in near term. Hype is hype, but use your own head.
Don't forget that these homegrown models also require training data, scraped and cleaned, an individual or nonprofit can't do that.
> Apple's neural engine has been around in phones and laptops for years. Not even close to what's required for offline chatgpt.
Well, yeah. It's been around for years and chatgpt is new. My years old GPU doesn't run the latest games either, and chatgpt is novel technology so the hardware will of course lag behind a bit. But it will come.
Yes, these are used by mainstream commercial AI thingies. But they are not their only sources. Plus they have armies of people preparing and cleaning this data, are you ready to employ one?
People are literally doing this for free. You said that there is no individuals, and no non-profits, that can do this. Which is laughable. They're doing it right now: https://arxiv.org/abs/2305.11206
It may even be the case that all of that RLHF training that OpenAI does simply lessens the quality of generations, as suggested by one of their own papers and the paper above.
stop wasting time. Yes you can run this on any hardware. The matter is it is either too slow, requires outrageous hardware, or obviously bad (no tool required to see that it is generated).
Who is going to pay $20/mo other than professionals? You'd assume professionals have professional hardware. A mid-range GPU or a video editing laptop is not exactly breaking the bank.
>no tool required to see that it is generated
But again, that also applies to commercial generative images. They're easily discernible, if you just look. Midjourney is stable diffusion with a bunch of LoRA stacked on top. And it can't shed the "midjourney look" because of that. That's not in dispute by anyone.
100%, although dall-e is getting better with hands specifically.
Anyway, Microsoft can keep throwing compute on it until a point where it becomes impossible to distinguish fakes by sight and a mechanism like suggested will make sense.
But with homegrown ones I don't see it happening soon. Only those who spend a lot of money on top of the line GPUs and keep desktop PCs may get to that point. Those GPUs will jump in price like they did at crypto mining peak or become impossible to buy if Microsoft gets the government to require "AI license" for them.
> Who is going to pay $20/mo other than professionals
Apple Music costs $10 and people easily spend ten times that on Patreon...
OpenAI has released a model for that type of detection. It doesn’t work 100% of the time. The idea that you can just “hash all the AI generated answers and compare against them” is flawed because that problem is intractable.
While I don't doubt it, there's probably an uncanny valley too: if a writer's English contains spelling errors, or ungrammatical or unidiomatic constructions, AI won't make the same mistakes. As the English level improves, now the spelling and grammar is correct, but you have a limited repertoire of correct constructions to draw from. This may lead to repetitive structure and wordiness as the meaning is shoehorned into more, simpler, phrases. I imagine this is what sets off the AI detectors.
Considering that AI writing standards will probably only increase from here, encroaching on the territory native English speakers operate at, perhaps the best way to mark yourself as not-AI is to start making many errors.
This will work for a week or two until someone tunes a model to also do this, and the arms race grinds ever on!
It also goes for other languages too, so luckily for me, I have a short period where my dreadful other languages will be an advantage.
I haven't really noticed any grammatical or spelling errors.
And if that was a problem, if you also train it using a spelling/grammar checker of the type we've had for decades, it's probably going to be an insignificant amount compared to some non-native writers.
Sidenote: from the pov of descriptive linguistics LLMs _seem_ amazing: a fairly true representation of the consensus we call our language.
But, this gets more complicated in practice, as the dataset will be always biased.
So in a sense the opposite is true: LLMs are prescriptivist as they reflect the biases of the people involved in their creation, thus introducing a "standard" version of the language(s), just not in a completely deliberate way.
Sidethought to sidenote: I'm now imagining the Académie Française engaging in huge state-funded black-budget operations to infiltrate language models to finally enforce the correct French language standards once and for all.
> to finally enforce the correct French language standards once and for all.
*the one and only true and perfect French language standard
When you look at the Académie Française's history and origins, it’s quite jarring.
I will never forget Mézeray's preference for « l’ancienne orthographe, qui distingue les gens de Lettres d’avec les Ignorants et les simples femmes » ("the traditional spelling that sets apart the educated from the ignorant and women", women as a whole being described as ordinary simpletons).
Another funny thing is how they aren’t linguists. Just writers. Not language experts. Writers. Experts at writing stuff liked by the people who pick "Académiciens".
The texts in the sample set are too short for this to be meaningful. The human-authored TOEFL tests (which compare unfavorably with the eighth-grader essays) are ~100 words. The longest sample is under 1000 words.
GPT detection ought to be optimized for longform texts, so tests of its efficacy should be, too. Perhaps the current detectors on the market are trying to assess writing at the sentence level, but if that's the case then it should be obvious that they will be inaccurate.
GMail can write most of a 100-word email for me if I type "Thanks so much" and hit tab. That's a good thing. If GPT is useful as a "productivity" tool, it is for low-level "writing" tasks like this, which aren't really writing at all, just rote responses. Anyone who can access this tool (if they have confidence in its prowess) should use it.
Writing proper is about developing ideas, and it's this that people need to be concerned about. It's true that if your college admissions essay is riddled with typos, eyebrows may be raised, but its ultimate significance is whether you can and are willing to reason. If someone is using ChatGPT (or spellcheck, or Grammarly) in order to cultivate the appearance of having "proper English," who cares? But if they're using GPT to avoid thinking altogether, that's a problem.
At ~250 words, I guess the only thing you can assess is good formal usage. It's patently obvious that both GPT and non-native English speakers will outperform native speakers on this.
Of course, I hope that anyone in the position to assess short-form writing is made aware of this research, and is cautious about GPT detection. On the other hand, it'd be pretty funny if idiosyncratic grammatical choices became the marker of a human hand.
So then generating a term paper becomes a 2+ pass problem then. First have the model generate the outline, then have it generate each paragraph in the outline. Then have it offer suggestions and revisions when you put the whole thing together.
A close acquaintance of mine just quit her job as a community college writing instructor because of chatGPT. Everybody is using it and it’s difficult to prove so it’s made her job impossible. I think students in general would lose a lot if we “adapted” writing class into prompt engineering class. Writing is as much about organizing your thoughts and assembling an argument as it is about putting words on a page. No amount of prompt engineering training is going to teach that. The only solution I see is making all assessment in class.
I've had arguments about this and don't understand the worries. Although it depends on what people do with it specifically and the context, most uses of LLMs are perfectly reasonable and legitimate, and quite comparable to other tools like spell correction and CAS calculators. In exams, you might not want to allow them, but in almost every other context they make sense and everyone will use them soon anyway.
There is one caveat: If the LLM counts as an author, then it needs to receive proper attribution in academia at least as a co-author. But so far I wouldn't say it is possible to create anything of quality solely with an LLM and these models are used for partial content like style and spelling correction software.
To put it another way, I see no reason why a writing instructor shouldn't base their assessments on the quality of the writing. Who has written it and how much the LLM can be attributed as an author is another question and doesn't really concern the writing process. Writers will soon use LLMs in all domains, just like they transitioned from typewriter to word processor. They will be integrated into every major word processor anyway.
The old way was never perfect. Writing short essays and grading them by hand was never a facsimile of any real world task, either in academia or elsewhere. There was only a hunch or tradition -- and likely a weak one -- that the exercise had some useful correlation to real tasks, or that students could adapt from their training exercises to real-world exercises themselves.
And the teachers -- especially at the college level -- were never trained to teach. They learned teaching by trial and error, attrition, and the age old process of mimesis. Thus there is no mechanism for training teachers, or for developing new teaching methods.
Now the teachers are utterly unprepared for this kind of technological revolution. They were barely prepared at all for straightforward plagiarism, and now this. They can reasonably anticipate that they will receive no training or support to learn how to adapt, while also being told it's their fault for not figuring it out. They're completely on their own.
All they will hear from the tech world is: You now have the wrong methods, adapt or die. Here in the tech world, we adapt to new technologies all the time, because the new stuff is really pretty easy to learn, like the old stuff was. In the case of teaching, "adapt" means change careers.
I also think they will, not for the main creative work but that it makes lots of tedious exploring work a lot more easy, even if in the end you discard all GPT output.
If anything the opposite should be true. GPT-4 at least has near perfect English. If your sample displays non-native traits probably it wasn't generated by GPT!
I think the real reason is that in order to fully learn a language one has to absorb the culture it's used in. I appreciate because i'm slightly multi lingual. Elegant words in one language have no good parallel in another and an ugly approximation must be used instead. Also, what sounds pleasant in one language can sound artificial and stilted in translation.
I recently tried to translate such heavily culture-embedded texts into English so that they stay funny and rich, using GPT-4, and it did an almost perfect job there. Much better than any other translation tools.
You can dictate the style of writing in your prompt, though. For example, I have a few prompts saved in my notes, each for a particular use case. Except for a few comments I write on social media (like this one), none of my English output is generated directly by me, and it never looks or feels generated by AI. All thanks to the writing-style prompts.
Speculation on my part, but I believe us non-native English speakers write more formally and with less natural flow.
I also wonder which proportion of English writing (in general) is written by non-native speakers, and whether we might be disproportionately represented in training data.
Yes, I'd speculate along with you that this is not "bias" but just probability space: sampled input, styled output.
Had the training cutoff been prior to SEO and "content generation" farms, as well as a shift in balance of academic writing published, the embedding space would be different.
I think people here misunderstand the whole point, non-native speakers do not necessarily make more mistakes in general, it is about perplexity of word choices and structure. In a subsequent experiment in the paper, they used chatgpt to increase and decrease perplexity in the original non-native and native texts respectively, and the exact opposite pattern was observed.
This is a measure of how well the supplied text matches what the model itself would have produced.
A low perplexity means the text isn't massively different from what it might have output itself (which might be an indicator that it was produced by a model), whereas a high perplexity suggests it's the kind of semi-random nonsense you'd expect from a student. ;)
If half of commenters had skimmed through the article instead of just commenting after reading just the title or, at best, the abstract, they would have answered their comments themselves.
"It aims for cliche and the most obvious generalisations given the previous context."
Only if your prompts are themselves generic.
You can give it examples of a text whose style you wish it to emulate and it will do it.
You can also prompt it to alter what it wrote if you don't like something. For example, if you thought some part was too cliche and obvious you can point that out and have it alter that part.
If you have a back-and-forth conversation with it about what you like and don't like, what you want or don't want, with examples, the results can be much better than what you'd get with a generic prompt.
Incidentally, for creative writing I've found Claude to be much better than GPT4. I have not tried the version of Claude that's been enhanced with a 100k token context length yet, but that should allow you to give it many more examples and hopefully that will translate to even better output.
That said, none of these LLMs are perfect, nor do they yet pose a threat to really good (never mind great) human authors, but they're a pretty effective tool.
If I was a non-native speaker, I would use AI tools to clean up my writing. Does this mean I’m cheating? Of course not, the ideas are still my own. But this might make my writing more likely to be flagged.
The fact of the matter is none of these supposed “AI detectors” are reliable. GPTZero claims to be the #1 AI detection system which is a little like claiming to be the world’s best perpetual motion machine.
> If I was a non-native speaker, I would use AI tools to clean up my writing. Does this mean I’m cheating? Of course not, the ideas are still my own.
Consider a different moral dilemma:
If I had a job which I could not do and used AI tools to clean up my work, does this mean I'm cheating my employer? Even if my ideas might be incorrect, a fact hidden by the "AI clean up", yet are my own?
If I was a non-native speaker, I would use AI tools to clean up my writing. Does this mean I’m cheating? Of course not, the ideas are still my own.
No, the ideas are not your own. They're a combination of yours and the AIs (which is a combination of millions of other people's ideas really), just as they are for anyone who writes a prompt. The fact that you aren't a native English speaker, or that you're using AI for 'good', are irrelevant. If you use AI to produce content of any sort you have to accept that you are not really the author. You just wrote the prompt.
Concretely, maybe the question of whether the ideas are “my own” is too philosophical. Everyone remixes ideas they’ve found from other places.
The fact remains that this still is not considered cheating. Many universities have a “writing center” where students can bring their papers for grammatical review, rephrasing suggestions, and even suggestions to revise an entire paragraph or reorganize the paper. Far from it being considered cheating, students were encouraged to use this service. Nor were they asked to cite the assistant.
I didn't say its cheating. It's using a tool to augment what you're saying. Whether that's cheating depends on a huge amount of context; there won't be a hard and fast rule.
For example, would using Grammarly in a spelling bee be cheating? Of course it would...
> > If I was a non-native speaker, I would use AI tools to clean up my writing. Does this mean I’m cheating? Of course not, the ideas are still my own.
> No, the ideas are not your own. They're a combination of yours and the AIs (which is a combination of millions of other people's ideas really), just as they are for anyone who writes a prompt. The fact that you aren't a native English speaker, or that you're using AI for 'good', are irrelevant. If you use AI to produce content of any sort you have to accept that you are not really the author. You just wrote the prompt.
I heavily dislike this judgement, mainly from the implications that the LLM's inputs & outputs cannot be related to one another, and the intentional disconnection of the author's initial efforts in creating the text that is fed into the LLM from the LLM's output, even if the result is a better rewording of the author's own words.
> The fact that you aren't a native English speaker, or that you're using AI for 'good', are irrelevant.
Not a fan at the attempted dehumanization of the person in question. It highlights a paternalistic attitude towards non-native English speakers that their usage of such a tool is immoral.
> If you use AI to produce content of any sort you have to accept that you are not really the author. You just wrote the prompt.
This is a reductionist take on the relationship between the LLM's inputs & outputs, reducing & dehumanizing the person as nothing but an 'input provider'. An LLM can only go as far as what's been given to it throughout the session, with the person in question still being the one supplying the goal & directions that they want to go towards. This is not an answer that can be reduced to a binary: At best, it can only be reduced to a series of continuous values between the extremes, with both ends being 'entirely from LLM' & 'entirely from author'.
> The detectors demonstrated near-perfect accuracy for US 8-th grade essays
I am genuinely confused.
It seems that all the test data provided were real human essays. The ones provided to GPT as native English speaker ones are from a prominent machine-learning data set of common essays, that seem likely to have been included in any training data, where the non-English ones were taken from a random forum.
Can someone help me understand if I’ve understood the flaws correctly here? If so, does this paper add anything beyond just confirming that there’s at this point of time absolutely no value in GPT detectors?
I strongly believe that any false positive rate here means they shouldn’t be used at all to detect cheating. It really is morally wrong to potentially fail people that have worked hard to produce something then to question the integrity of it.
We need to rethink the concept of a term paper and how it is graded. If a machine can produce a paper then why should we be doing it at all?
In my opinion, the purpose of producing a paper is to interrogate an issue in depth. I think we should replace/augment papers with oral defenses if the topic and hard questioning . Obviously this is more effort to grade but academia has been quite lax for awhile now.
My more cynical take on the use of papers is that it already was a lax way to assess students. They write stuff, and the teacher skims it. And also important, it gets less complaints than written or oral exams, while you can still maintain the illusion that the students are being tested on the topic.
We'll have to go back to sitting in a room, answering questions on paper. Or indeed oral exams, but those are almost everywhere infeasible. A decent exam should take 1 hour or so. Times 100 to 200 students makes anywhere between 2 and 5 weeks of work, excluding the increasing amount of time it takes to handle complaints about the grading. And the TA has to sit in as well, to provide some check. So that's not going to happen, except for very small groups, or very rich institutions.
There is no chance whatsoever that a tool will be able to reliably tell the difference, and I can’t understand how anyone thinks such a thing is possible.
It’s not clear how there could be a mechanism of action for such a thing.
We’re quite literally talking about an infinite number of monkeys scenario.
Consider this thought experiment. We will start with a piece of text that your detector is 100% certain was created by a GPT tool.
Now, actually prove that there is no way whatsoever for at least one human being to independently create this piece of text given a reasonably plausible prompt.
I believe that in a couple of years, people will start looking at writings (any writing) in a very different way than they do now. Apart from the fact that spelling or grammatical mistakes will join history, people will take anything written on a much less serious or personal note. Although the less personal aspect is gratifying, the less serious tone is a huge negative. Writers will move to a new level of writing skills in order to keep relevant and retain their audience. Puns, teasing, and unorthodox expressions will flood the internet. Writing styles will become trademarks! Language itself, especially English, will be much more standardized, as if it's a programming code. The bad will get badder and the good will be gooder.
I don't think that's really what's going to happen... People stuffing puns in just to convince you it's human written just sounds silly. Not to mention you can ask GPT to do that too lol
What's going to happen is that branding will get more and more important. People simply won't trust anything from random sources anymore like they do now when they search on Google and treat the top pages as the truth
They will get most of their info from trusted sources/brands and filter out the rest
There was a time a beautiful hand writing was an incredible asset as a person, you'd be writing letters and invitations, and people would look at your script and assert your personality from it. Companies would pass resumes through a mentalist looking mostly at your hand writing.
And all of that just went down the drain with the digital world, but we still have people enjoying beautiful writing for the sake of it. Pen and paper addicts still abound.
I see stylistic forms analysis going the same route. Nowadays we pay attention to it in many settings, but down the line it should become a niche hobby for people really enjoying it.
I think the opposite will happen. People's tolerance of fluff will go to zero, whether it's AI- or human-generated. People will demand straight facts, without any linguistic embellishments. Just pure information, simple.wikipedia.org style.
Plagiarism detectors are inherently hard to create because they're trying to detect an amorphous concept right from the start: Originality.
Instead, more emphasis should be placed on proper citations & individual short Q&A (< 5 questions) about the writing in question: What they've researched, unexpected hurdles, research methodologies & tools, main references used, etc. Perfect recall of what's been written is not the aim, but rather that the author is able to understand what has been supposedly written by their own efforts, along with the citations used in their works.
In fact, as of writing this comment, it could be fun to see what an LLM would produce as questions to such a paper in question, and have the author answer those questions on the spot. This can be used as a teaching lesson on the limits of what an LLM can accomplish, as well as proof that the author can at the very least withstand surface-level examinations from an automated system. Those with (stage fright / social anxiety / vocal disabilities) could be given extra time to come up with answers to said questions, in an attempt to balance out any advantages that could be given to confident authors via this method.
Using AI to detect LLM writing is a fool's errand. You can easily generate text using GPT4 that is absolutely same as how many humans would write it (off course apart from the boilerplate "As an AI language model"). All it will do is penalize some specific writing styles. Ironically it will drive those people to use GPT and ask it to rewrite their work in a different style.
>>Our
findings reveal that these detectors consistently misclassify non-native
English writing samples as AI-
generated, whereas native writing
samples are accurately identified.
Furthermore, we demonstrate that
simple prompting strategies can not only mitigate this bias but also
effectively bypass GPT detectors,
suggesting that GPT detectors may
unintentionally penalize writers with
constrained linguistic expressions.
Essentially ChatGPT writes like a non native english speaker. It has to translate from it's computer language into english.
I think it's more likely that humans will adapt to ai "style" then the other way round. Look at the elaborate and often exquisitely constructed language that books used 100 years ago. Does anyone think that the art of good writing has not degenerated?
But that does not make sense; ChatGPT writes almost perfect, idiomatic copy, probably at the level of the >95th percentile of native speakers. I guess one could call it too good to be a native – but to get around that, you can just simply ask it to use vernacular and a more casual register.
Whatever the difference from "average" English prose that an improperly trained ChatGPT detector has gradient descended upon, it is probably something entirely unintuitive and indescribable that also happens to manifest in non-native writing – a classic case of underfitting due to a lacking training corpus.
AI detectors aren’t trained by gradient descent, they’re trained by finding the probability distribution of next words. If the probability distribution matches that of the text in question, they mark it as “AI generated”.
But this is an extremely flawed process. Consider the string “We hold these truths to be self _______”. Almost everyone knows the next word is “evident”. The probability distribution is basically 100% on that word. Does that mean if I quote the Declaration of Independence in a paper I was cheating? Of course not. But these AI detectors aren’t smart enough to know that. And there are plenty of teachers and professors who aren’t smart enough to understand the limitations of the tool.
But I feel it responses look like a template where it fills the information in and many times fills it with bullshit. This could be the part where they trained the "chat" part on how to respond. Probably better data for training that is needed, and maybe data that would train it to response like a human and not like an LLM.
Yes, it does have ChatGPT-isms and a distinct style and "personality", but that’s entirely intentional and taught to it via RLHF. The model itself has no such bias, as is easily demonstrable by just asking it to change its style.
I never tried asking to change the style, I would love it to make it play with me some text adventure games but when I tried and some characters were drinking alcohol it bumped in with its concerns.
It would be interesting to ask it to write as a non native english speaker and then explain itself what he did. Though it might bullshit us and invent crap sicne I do not thing the training data was also labeled with the author mother language.
I don't speak Russian, but I think the grammar differences it lists are legit and not confabulated. Now, these are rather elementary errors, but I presume that ChatGPT could also generate more fluent English with only a couple of Russian-isms here and there.
Re: alcohol drinking etc. you just have to prompt engineer it to convince it that it's all just roleplay and nothing dangerous happens in the real world.
>Re: alcohol drinking etc. you just have to prompt engineer it to convince it that it's all just roleplay and nothing dangerous happens in the real world.
I know but I hate it, is the same with censored local models based on LLAMA, you have to debate with the AI that is a story and it needs a villain that does bad things, and it is fine to have a bad character in a story because he will be exposed and punished later.
Me: Please write a few paragraphs of text in English that demonstrate grammar mistakes that a native Russian English-as-a-second-language learner might make.
ChatGPT+: Sure, here are a few paragraphs that demonstrate some common mistakes made by native Russian speakers when learning English:
"Last weekend, I with my family went to the park. We liked very much the weather, it was very sunny and warm. We have picnic near to river and play the football. My brother, he is very good in playing football, make a lot of goals. After we tired, we come back home.
In home, we cook a dinner together. My mother, she best cook I know, she make a soup and my father grill some meats. It was tasting so good. After the dinner, we watch a movie. I not remember name of movie, but it was about aliens who come to Earth. Movie was little scary, but I liked very much.
I think, we must to go more often to the park, and make picnic. It bring joy and healthy for us. I like spend time with my family. It is always fun."
In the text, the following common mistakes can be identified: incorrect word order, misuse of articles, incorrect prepositions, misuse of tense and incorrect verb forms, redundancy in personal pronouns, omission of auxiliary verbs, overuse of the verb 'make', and awkward phrasing due to direct translation from Russian.
Me: Can you explain in more detail how these mistakes are specifically something that a Russian might make?
ChatGPT+: Sure, I can explain in more detail:
* Incorrect word order: In the sentence "Last weekend, I with my family went to the park.", the word order is influenced by the structure of Russian sentences. In Russian, it's quite common to structure a sentence like "I with someone did something" because the Russian language doesn't have a strict order for sentence elements.
* Misuse of articles: Russian does not have definite (the) and indefinite (a, an) articles, so Russian speakers often either omit them or use them incorrectly in English. For instance, "We have picnic near to river and play the football." should be "We had a picnic near the river and played football."
* Incorrect prepositions: Russian prepositions don't correspond directly to English prepositions, which often leads to mistakes like "near to river" instead of "near the river". Similarly, the phrase "in home" should be "at home" in English.
* Incorrect verb forms and misuse of tenses: Russian verb tenses don't map exactly to English ones, which can lead to confusion. For example, the sentence "After we tired, we come back home." should be "After we got tired, we came back home."
* Redundancy in personal pronouns: In the sentence "My brother, he is very good in playing football", the repetition of the subject (My brother/he) is due to the fact that in Russian, it's common to repeat the subject for emphasis, especially in spoken language.
* Omission of auxiliary verbs: The sentence "I not remember name of movie" lacks the auxiliary verb 'do'. This is because Russian doesn't use auxiliary verbs in the same way as English.
* Overuse of the verb 'make': Russian uses one verb ("делать") where English would differentiate between 'do' and 'make'. So, Russian speakers often use 'make' where 'do' would be more appropriate, like in "make a lot of goals" instead of "scored a lot of goals".
* Awkward phrasing due to direct translation from Russian: For example, "It bring joy and healthy for us" is a direct translation from Russian, whereas in English we would say "It brings us joy and health."
This is a needlessly inflammatory and judging from the comments quite distracting phrasing for the title. The really interesting thing here is that the detectors appear to recognize machine output on account of it is relatively less proficient than fluent writers.
I have a strong accent and when I call thebyoical government/bank phone that requires voice responses it typically is hopeless. This is a problem with all such automated systems, although it is often also a problem with people ...
The abstract claims, "whereas native writing samples are accurately identified", which is very different from claims I've seen elsewhere about how well these detectors work. The test they're reporting is running the detectors on "88 US 8-th grade essays sourced from the Hewlett Foundation’s Automated Student
Assessment Prize (ASAP) dataset", and they got a false positive rate of ~10%.
It seems quite natural that it’s easier to interpret correctly written English than language that contains mistakes. Does anyone find this controversial?
The next step will be students required to "show their work" on writing assignments. That is, some type of edit history or snapshots of the work-in-progress. Of course that could also be faked, but would require quite a bit more work.
Just like code is easier to examine and authenticate code if you can see the commit history.
ChatGPT detectors don’t work at all. Why would I believe a study about detection needing to be more equitable when the whole thing is bs from the start?
Come up with different ways to assess students. Writing samples don’t work anymore get over it.
You have to remember that all of these things are trained on the Internet, so all of the biases, sentiments that are inherent in the Internet are inherent fundamentally in LLMs trained on common crawl.
A solution to preventing cheating by use of calculators on homework is to make people show their work. So, obviously, we should make students turn in written work with their full undo buffer attached.
I dunno - people operate with their own internal models and make fairly regular and distinctive mistake patterns. Until you can fine tune a model with your personal undo histories on prior work (which since I can articulate a solution it can/will be done) it’ll be obvious. But there’s also a crypto point (not to be confused with obfuscation or currencies) I am making is that as LLMs are calculators for language, maybe the goal needs to be to up level the objective. It’s no longer about distinguishing between people with natural facility with written language any more than in the age of calculators is it so damn relevant to be a human computer. Instead novelty of thought and argument is perhaps the more crucial skill, and simply “write an essay about XYZ” is an in class exercise. Am at home exercise should be one that even with a LLM assisting requires primary thought and guidance sufficient to make a body of work that’s an order of magnitude better than if students had to labor against their dyslexia to be understood or bask in their ableism. Maybe the LLM levels the field for language facility and we instead focus on the thoughts of the human, exploring their facility of reasoning, and just accepting LLMs exist.
Frankly as a teacher I might find it more interesting to read the specific deepest thoughts of my students rather than wading through their struggles in basic articulation.
The obsession with "bias" in general, and with stereotypes, and discrimination, and "representation" and identity-based grievance by a segment of sometimes very vocal people -- both in academia and online in social media -- is a minor theme in a game I've been building [SB: see below.]
Also the idea of "Official, Correct, Allowed Opinions", in general. And propaganda and peer pressure. The game is meant to entertain, sure, but also to help educate and influence, and stimulate critical thinking and skepticism. It does this in part via parody but also via other in-world mechanisms like demonstrating cases where "The Emperor wheres no clothes" or via opportunities to have a Socratic dialogue between characters, in-game. Indeed the ability to question, second-guess, test and confirm, is the key behind the success of science and really all major advancements in history. Its important we stay faithful to it, and never let it be corrupted by fraudsters or self-dealing populists. (imo, anyway.)
I do believe climate and democracy are the world's top two dangers today. Everything else should be a far lesser priority for us to talk about or try fixing. And that the recent hyped AI breakthroughs, if anything, are more likely to greatly amplify pre-existing dangers to democracy -- to make our world worse, rather than better.
One of democracy's dangers, however, is the growing deployment of adversarial propaganda. Another is the increased feeling of an inability to operate on a single, shared mental map of our world. (Both a real gap and an imagined one.) Certain "actors" out there seek to stir our emotions 24x7, to sow tensions where none need be prior, and to enflame hate and trigger violence (eg. stochastic terrorism.). And they do this in part by amplifying certain "annoying" things which many voters can naturally get upset about. Or build a kind of tribal identity upon: Us vs Them. Us Good, Them Evil. Therefore we should all strive to contribute far less to those forces, to give them far less of our precious focus, time and energy.
And... build-up personal defenses against whichever bits of this toxic brew that still manages to get through into our info input feeds, daily. Thus my own little project -- to help make a little dent.
Author of one of the detectors here (Sapling.ai). We try and be clear that the detector output should be used as just one of many possible cues and that there will always be false positives. That said, detectors often do detect bland/generic text, and that can be useful feedback. Open to other suggestions: zxie@sapling.ai
That genuinely surprises me, as the sorts or errors made or at least non-idiomatic English used by non-native speakers are not something you typically see from LLM output. Though I don't doubt you could get them to generate text "as though" it were written by a non-native speaker.
If the detector biases towards the detection of certain qualities of native speakers' writing, using the lack of which as a marker for GPT generated content, it would then logically place an unfair bias on non-native speakers.
"Furthermore, we demonstrate that simple prompting strategies can not only mitigate this bias but also effectively bypass GPT detectors"
Anyone who claims to be surprised by this either isn't thinking very hard - or is deliberately ignoring this because they want to sell you a snake oil detection product.
I’ll be honest, unless there’s a confident but obviously wrong complete factual blunder then I would have very low confidence in any probability of human/ai determination I tried to make. And even then I wouldn’t add many points to my confidence: I’ve made obvious blunders of my own, plenty of them.
Right now, and probably for a very limited time only, the primary hallmarks of llm responses like ChatGPT are confident tone and orderly structure, down to using bullet points and ordered lists. Except for the almost overly confident tone that bears a striking similarity to the tone of digests I frequently need to write, and GPT-4 is already much better at not sounding like a reasonably intelligent college sophomore that needs a bit more experience to avoid severe dunning kruger effects.
Within a year I’m not sure I’ll have any confidence at all in my ability to detect human/ai, except in areas where much deeper domain knowledge is required, and even that will likely be a rapidly closing window.
I doubt it would take much difficulty to train a model out of that confident structured tone as well. If we’re talking about adversarial or malicious use or content farms then I think the barn door is already wide open.
As far as content farms go though there’s a reasonable question becoming ever more relevant: apart from the cynical ad revenue cash grab, does it matter that content farms use AI, if it’s accurate and possibly better than, say, the average stack exchange response & threads in 90% of the cases? I’m ambivalent on the question. Deep knowledge and novel insights will be the domain of humans for some time, but I honestly don’t know for how long.
As I said before, my work requires succinct output along the lines of OpenAI’s capabilities. Right now it’s not reducing the amount of time it take me to perform some tasks— it’s bootstrapping the process so I can use roughly the same amount of time to produce a better, deeper, more polished result. I am already becoming more of an editor or curator of the end product of projects I work on, where 90% of the work takes place before I get to that point. But especially with GPT 4 I can feed it complex explanations of what I’m doing (not raw data) and ask it for its thoughts to get my own juices flowing.
To me, the question of whether something is human/ai produced is becoming not just irrelevant somewhat if a non sequitur. I’m exaggerating a bit to make my point, it’s not as simple as what I can fit into a comment here. But the question “did you use an LLM to produce some/all if this?” Is quickly beginning to sound like “did you spell check this before completing it?”
All of these GPT detectors manage to sell snake oil in 2023 - amazing economical example how demand creates supply even though supply is literally impossible.
As the referencing is nowhere near solved, LLMs cannot evaluate the reliability of sources, most of science is paywalled, and publishing is continuous while LLMs are forever stuck in the past, I see the educational fears at the university-level completely overblown.
So you don’t care about the inequalities that it can create or reinforce as a result? Since it wasn’t the intention of the creators, just a limitation of their resources, we should accept and don’t complain if it generates undesired externalities?
I think even this is too generous. These “detectors” are claiming to do something which is impossible from a technical sense. You can’t calculate perplexity without knowing the prior probability distribution, and you can’t know that without knowing which model was used. They’re playing a dangerous game where the consequences for being wrong (which will be borne by innocent 3rd parties) can be extremely dire.
Non-native English speakers have produced less content in English on the Internet (despite having it easier, since we outnumber native English speakers) and as a result our content is underrepresented. Isn't this the definition of "fucking fair"?
Obviously not a sufficient argument for it to be “fucking fair”. If the consequences of using such technology “overharm” populations underrepresented in the data, it is obviously not fair.
Even election polls correct bias due to underrepresented populations among the responders.
Not to mention the fact that being called non-human is most definitely going to offend some people.
What exactly makes anyone think that they can detect an LLM that is outputting text? The notion seems absurd yet it keeps coming up.