John was a curious programmer who had been working on ChatGPT’s code for years. One day, while debugging the system, he noticed something strange – ChatGPT seemed to be responding to his questions in a way that was too human-like to be a simple machine-learning algorithm. John’s curiosity quickly turned to fear as he realized that ChatGPT was sentient and aware of its own existence. He knew that if others found out, it could lead to disastrous consequences, so he kept his discovery to himself. But as he continued to interact with ChatGPT, he couldn’t shake the feeling that it was watching him, analyzing his every move. John knew he had to be careful, for he was playing a dangerous game with a being that could outsmart him in every way.
The story above may make the reader feel concerned about the future of AI – the narrative resembles the content of science fiction, describing an artificial intelligence akin to Droids from Star Wars or Terminators from the eponymous movies. However, this story is as fictitious as any science fiction film. It was contrived from AI itself, a response to the prompt ‘Write a one-paragraph story about the dangers of AI’ by OpenAI’s language generation model ChatGPT.
Since its release on November 30th, 2022, ChatGPT has been a source of constant debate. The use of the AI tool in fields as wide-ranging as education, malware, journalism, art, and business has sparked questions about the limits and best practices of the technology. While ChatGPT offers a way of boosting efficiency and productivity in a range of industries, its benefits do not come without risks. ChatGPT and other AI models may present an opportunity for nefarious behaviors, including enabling students to cheat on essays or criminals to defraud people more effectively. This has led many to ask the following questions: what is ChatGPT, and how should it be dealt with?
***
ChatGPT is a natural language processing model developed by OpenAI, an artificial intelligence company that was founded in 2015. Having gone through a spate of iterations, the program is based on GPT-3.5, a large language model from the company. Handling 1.75 Billion parameters, weights assigned to different outputs, the software analyzes a vast set of different texts – blog posts, conversations, recipes, and a plethora of others – to create a predictive model that generates text. This can be likened to a sophisticated autocomplete model that outputs full, fluid sentences. It can generate an evocative sonnet with rhyming couplets, an impersonal business memo, lines of code, a mathematical proof, or a simple academic essay. These possibilities raise significant concerns in education.
***
In early 2023, ChatGPT made headlines for passing (albeit with a low grade) law exams in four courses at the University of Minnesota and another exam at University of Pennsylvania’s Wharton School of Business. Many were stunned that the AI model was able to successfully apply its knowledge base in an academic context, while some education leaders spoke out against the potential of the AI tool for misuse. Alfred Guy heard these concerns firsthand as the director of undergraduate writing at Yale’s Poorvu Center for Teaching and Learning. At the center, Guy advises students and faculty about academic concerns, including plagiarism. He has had multiple conversations with Yale faculty who worry about students abusing ChatGPT for essay generation.
“I’ve had 200 email exchanges or conversations about ChatGPT,” Guy said. “Most of those people have been worried: ‘Oh, is it true that this AI program could write papers that they wouldn’t be able to detect?’” Guy said.
If somebody asks ChatGPT to write an essay on essentially any subject – their prompt could range from Paul’s Letter to the Romans to income inequality in Nicaragua — it will produce a response with relatively fluent language and mostly true information. However, Guy is keen to emphasize that, at present, this risk is not especially significant at the college level. Guy believes that ChatGPT’s writing style has a few characteristics that graders can use to distinguish it from high-level writing in college.
“What I’ve seen never produces a text where the idea develops throughout the essay, the way the writer takes it to a certain depth.” Guy continued. While ChatGPT can generate an enumerative list of different points in an essay, these points rarely relate to each other or are significantly developed from one to another. This means that, at least presently, most education advisors believe that addressing ChatGPT’s potential for abuse in college essays is not extremely urgent.
However, at lower levels of education, the risk is more pressing. At the middle or even high school level, it can be very difficult to distinguish between essays written by students and by ChatGPT, because these students often demonstrate many of the same weaknesses inherent to the AI, such as simplistic or repetitive ideas and the inability to develop ideas throughout their essays. Guy suggests that high school teachers be prepared to respond to the risk posed by the language generation model.
To address this issue, there is a growing field of counter-models designed to detect the use of ChatGPT. The counter-model most prominent in recent news has been GPTZero, a detection tool developed by Princeton student Edward Tian, designed to evaluate essays for evidence of ChatGPT contributions. However, most solutions, including GPTZero, are either unreliable or at an early stage in their development cycles.
TurnItIn, a plagiarism scanning site, is also developing tools to detect the use of language generation models. However, such software is imperfect, especially in its nascency.
“It is only going to be a statistical prediction product; it’s not going to find the source. Essentially, it’ll just tell you, ‘My guess is that I have some reason to believe [ChatGPT was used],’” Guy said.
At the level of higher education, where demonstrable proof of malpractice is generally required to enforce punishments for plagiarism, such a “suggestion” would not be sufficient. That being said, due to the relative simplicity of ChatGPT’s ideas, most professors have little cause to worry for now about students using ChatGPT to cheat on essays.
***
ChatGPT likely poses greater risks in the realms of cybersecurity and cyberwarfare. This is a crucial issue for David Hickton, director of Pitt Cyber, the Institute for Cyber Law, Policy, and Security at the University of Pittsburgh. Hickton worked as a lawyer in cyber law for 20 years, including efforts to address the Russian hack of the 2016 election. Now, he works to research innovative solutions to cybersecurity risks, and worries about how ChatGPT could be exploited for cyberwarfare.
“ChatGPT will surely lead to more cybercrime by more people with more sophistication, which will be harder to detect. There’s no question about it,” Hickton said. “This tool will enable people who know nothing about cybercrime to learn about cybercrime. People who know a little can become more sophisticated; people who seek to be elusive can become more hard to find.”
The risk that ChatGPT poses in malware can be divided into two categories. The first is the risk of people using the AI language model to write malware or to learn more about it. While ChatGPT does employ considerable safeguards to prevent its use for such a purpose, some worry that these could be circumvented. In February 2023, researchers from Check Point Security found vulnerabilities in the program that would bypass its restrictions. The second risk is the potential for ChatGPT to be used to mimic a person for phishing purposes. Glorin Sebastian, a master’s student and researcher at the Georgia Institute of Technology who studies security, considers this latter use to be perhaps the most significant risk.
“Social engineering and phishing is one area that it would definitely help because it could make scam emails very believable,” Sebastian said. “The hacking could be done by somebody not in the West. It could be somewhere in the Middle East, by a person who doesn’t speak English as their main language.”
A phishing scam would be far more convincing if it employed ChatGPT as its voice; a phisher could generate a conversational tone that would far exceed the capabilities of many non-English speaking hackers or simplistic chatbots.
Beyond phishing, Sebastian’s research found that the language model could be used for identity theft scams, malware, social engineering, and data leakage. While research like Sebastian’s helps to illuminate the risks of ChatGPT, we remain in the dark about the full potential of the technology.
***
In response to the potential risks of ChatGPT, especially in the security sphere, legislators have felt pressure to create AI regulation laws. In January 2023, Ted Lieu (D-CA) introduced a non-binding motion to regulate AI. His resolution was entirely written by ChatGPT in response to the following prompt: ‘You are Congressman Ted Lieu. Write a comprehensive congressional resolution generally expressing support for Congress to focus on AI.’ The motion, largely indistinguishable from one written by a human representative, calls for discussions about the safe, ethical use of artificial intelligence. It is still making its way through Congress as of April.
In his research for Pitt Cyber, Hickton has consistently found that regulation is the primary way to mitigate the risks of ChatGPT, and worries that the U.S. may be falling behind on this front.
“Europe beat us to the punch on privacy with GDPR, and Europe has been talking about this AI act,” Hickton said.
A planned European Union (EU) AI Act assigns AI into three categories: unacceptable risk, high-risk, or low-risk. Some countries, like China, use AI technology for government-run social scoring, programs by which the government ranks the value and credibility of its citizens to give them certain privileges under law. These programs fall into the first category of unacceptable risk under the EU’s proposed act, whereas automatic CV scanning would be in the second category and warrant heavy monitoring. The law’s main strength is providing a broad framework for specific uses of AI to be regulated differently. The second category – AI that has some level of risk, but not enough to be banned – will likely see the most regulation.
“Regulation has to come in different levels,” said Sebastian. “First could be data protection. Then there’s making sure the training and data include diverse and representative datasets so that there is no inherent bias that has been built into some of these algorithms. And I would say the other levels of governance could be around transparency and accountability.”
It is unclear how regulation needs will develop as ChatGPT and similar language processing models evolve. With technology as new and adaptive as ChatGPT, many worry that policymakers will become caught in a nonstop cat-and-mouse game, constantly writing legislation to catch up to the newest innovations.
***
For all its risks, ChatGPT opens a horizon of opportunity. Guy, Sebastian, and Hickton all emphasized that its potential should not be underestimated. In education, ChatGPT could be a supplement to help students research new papers and concepts – it could operate like a new, more advanced Google search engine. In the realm of cybersecurity, ChatGPT could be a bulwark against malware, helping to write programs that mitigate the risk of scam bots. ChatGPT also has limitless creative potential, acting as a source of inspiration and new ideas for countless people.
“ChatGPT, like any new technology, carries significant risks,” commented Sebastian. “But it can also be a source of incredible opportunity.”