Can you tell the difference between a human and ChatGPT?

ChatGPT, the popular OpenAI chatbot, is touted as a technology that can mimic human conversation and writing styles.

ChatGPT’s responses are based on the prompts it is given in the chat window. For example, if you tell it, “Describe Ottawa, Canada in one sentence” it will reply with this:

“Ottawa is the capital city of Canada, located in the eastern portion of southern Ontario on the banks of the Ottawa River.”

It’s free to use once you create an account and can be found online here: https://chat.openai.com/chat

But can you tell the difference between a simple news article written by a human being and one written by ChatGPT?

Here is the prompt CTVNewsOttawa.ca provided ChatGPT, which also served as the basis for the human writer.

Write a news article in Canadian Press style based on the following information:

This week, the NCC began the process of closing 24 Sussex Drive, the official residence of the Prime Minister of Canada.

The NCC is relocating residence employees and preparing the main building for abatement and other related work. Over the coming weeks, the site will be closed to provide easier access for proper planning of this work, which will include the abatement of designated substances such as asbestos, as well as the removal of obsolete mechanical, heating and electrical systems. Procurement for this project will occur over the winter months to ensure the prompt start of abatement work in Spring 2023. The work to be carried out as part of this project must be completed regardless of any future decision on the residence.

Over the years, the NCC has commissioned several reports and studies in order to plan for future work. The state of the asset is well known, and risks have been duly controlled. Today, with continuously aging and worsening materials and systems, more significant actions must be taken to mitigate matters of great concern such as potential fire hazards, water damage and air quality issues. Relocating employees and closing the site is required to prevent these concerns from materializing and to ensure the integrity of this important Classified Heritage Asset.

This is a press release issued by the NCC on Nov. 17, 2022.

Here are two news articles based on this information. Can you tell which one was written by a human? The answer is below.

EXAMPLE A

The National Capital Commission (NCC) has announced that it will begin the process of closing 24 Sussex Drive, the official residence of the Prime Minister of Canada, in order to conduct necessary abatement and maintenance work. The residence employees have been relocated, and the site will be closed over the coming weeks to facilitate proper planning and execution of the work.

The project, set to begin in Spring 2023, will include the abatement of designated substances such as asbestos, as well as the removal of obsolete mechanical, heating, and electrical systems. Procurement for the project will occur over the winter months in order to ensure a prompt start to the work.

The NCC has commissioned several reports and studies over the years to plan for future work on the residence. The state of the asset is well-known, and risks have been controlled, but with continuously aging and worsening materials and systems, more significant actions are needed to mitigate potential fire hazards, water damage, and air quality issues. By relocating employees and closing the site, the NCC hopes to prevent these concerns from materializing and to ensure the integrity of this important Classified Heritage Asset.

The work to be carried out as part of this project is a necessary step to protect this national heritage asset and will be completed regardless of any future decision on the residence. The National Capital Commission reminds the public that 24 Sussex Drive will remain closed to the public for the duration of the project.

EXAMPLE B

The National Capital Commission says 24 Sussex Dr. is officially closing.

The NCC says it began the process of closing the prime minister’s official residence this week, to prepare it for abatement and other related work.

Prime Minister Justin Trudeau has not lived at the dilapidated residence since his election in 2015, opting instead to reside in Rideau Cottage on the grounds of Rideau Hall; however, some employees still work at 24 Sussex, but they are now being relocated.

“Over the coming weeks, the site will be closed to provide easier access for proper planning of this work, which will include the abatement of designated substances such as asbestos, as well as the removal of obsolete mechanical, heating and electrical systems,” the NCC said in a news release.

A recent report found the building was in critical condition and in need of significant repairs. It could cost more than $36.6 million to restore the building to “good” condition, the report found.

The NCC said drastic measures are needed to deal with the building’s many issues, including fire hazards, water damage, and air quality issues.

“Relocating employees and closing the site is required to prevent these concerns from materializing and to ensure the integrity of this important Classified Heritage Asset,” the NCC said.

The building was first erected in 1868 and became the prime minister’s official residence in 1950.

THE ANSWER

The answer is Example B, which was written by CTV News Ottawa digital multi-skilled journalist Ted Raymond.

While the ChatGPT example faithfully reproduced the press release, and even knew that the NCC is the National Capital Commission, it could only publish something based on the information it was fed. It lacks the ability to provide context from previous reporting that a human writer is able to remember or research on his or her own.

ChatGPT is faster. It took our human writer a few minutes to research and write, but ChatGPT produced its text in a matter of seconds.

Among those who voted in the poll, 59 per cent correctly recognized that Example B was written by a human. Another 34 per cent guessed that Example A, the ChatBot text, was written by a human.

Seven per cent said they couldn't tell.

We hope you enjoyed this experiment.