AI writing is becoming indistinguishable from humans! Can you tell the difference between real and AI-generated news?
Created by Frederick Chan in Fall 2022 as part of an Information Ethics final project at UW.
AI writing is becoming indistinguishable from humans! Can you tell the difference between real and AI-generated news?
I made this game as part of a final project in an Information Ethics course at UW. Originally it was a small, but hands-on part of a larger website prototype about the role of large language models (LLMs) in online misinformation. The game itself is a great way for players to learn how far LLMs have come as of December 2022, and how hard it can be to distinguish between them even if you're completely aware that one of them is fake.
Some people were interested in this game specifically when I showcased it at the end of the course, so I've separated it out and made a few minor changes for a public release.
The fake articles were only given the title, date/time, and city of the real news article where applicable, and told to generate a news article in the style of the news source that published it. For example, if the real news article was "Cat Stuck Up Tree - January 10, 2021 - Seattle, WA" published by Pear Daily News, then the prompt given to the LLM would be something like Generate an article titled "Cat Stuck Up Tree" in Seattle, WA and published January 10, 2021 in the style of Pear Daily News. No other details are given.
Only two LLMs were sampled due to time constraints for the final project: the variant of GPT-3 used by ChatGPT, and Fairseq Megatron 11B.
I had a pretty limited amount of time to work on this, as I also had to contribute to the rest of the final project that this was originally a part of. There were a bunch of things I wanted to do that I didn't end up having the time for. For example, a good way to improve the game is to sample a wider variety of LLMs, especially some of the older LLMs like recurrent neural networks (RNNs) from the mid-2010s, or LLMs with the same architecture but different training data or parameter sizes. That way, the player could get a sense of how LLMs have progressed over time, and how training data or model architecture affects LLM outputs.
With the current state of the art in text-to-image generation (mostly diffusion-based models as of January 2023), the fake articles could be paired with fake images to add to the realism. Players could possibly start to learn strategies that distinguish real images from fake images, or learn to spot when the image and the content are mismatched.
Some miscellaneous gameplay improvements would be good too, like game modes that limit fake news to a single LLM, or more in-depth statistics about which LLMs bamboozled the player the most.
I claim fair use for the title and content of the real articles, which are owned by their respective publishers and whose URL they were originally accessed from are linked to in the game. The title and content of the real articles is provided for an educational purpose, and are only small excerpts of the full articles available from their repsective sources.
The fake article contents generated by the large language models are released under public domain. As of the time of writing (3 January 2023), the copyright status of the AI-generated comic book, Zarya of the Dawn (US Copyright Office registration No. VAu001480196) is being challenged by the US Copyright Office, since a human who generates a comic book using AI by primarily engineering a bunch of prompts is potentially not copyrightable. I don't think stuff you generate by throwing a prompt at an LLM ought to be copyrightable, and just throwing titles and a date at LLMs to generate fake news articles is definitely not sufficient human input.
I am making the software that I wrote for the game itself available under GNU GPL v3.