Rise of AI-generated 'deepfake' porn shakes South Korea

Malicious use of artificial intelligence is plaguing South Korea. Beginning this week, the country's authorities have been investigating the mass sharing of nude images of real people generated through artificial intelligence, which is causing a major stir in the country.

According to reports, the images are being shared through Telegram groups. They consist of photographs, often pulled from social media or taken in public spaces. They are modified using A.I. to alter the image and make the person photographed appear naked.

South Korean authorities took action after the discovery of a massive network involving hundreds of victims, many of them minors. The revelation reflects the scale of the problem facing South Korea, which some researchers say is where half of the so-called "deepfake" porn videos around the world originate from.

The threat in South Korea seriously affects minors. In high schools and educational institutions, according to authorities cited by the The Wall Street Journal, these practices are becoming increasingly common. To deal with the problem in the classroom, the South Korean education minister is reportedly considering a series of extraordinary measures to increase punishments for those of school age, starting at age 10.

"Deepfake videos may be dismissed as mere pranks, but they are clearly criminal acts that exploit technology under the shield of anonymity," said South Korean President Yoon Suk Yeol, a former prosecutor, at a cabinet meeting on Tuesday. "Anyone can be a victim," he stressed in remarks reported by The Wall Street Journal.

About 500 educational institutions at all levels may have been affected by this wave of new illegal pornography. Universities are also at the center of the controversy. After messages from one of the Telegram groups that shared these images were made public, it became clear that many of them had been produced from photographs taken on university campuses.

These Telegram groups operate with their own rules, and to access them, potential members are often required to provide a real photograph of a woman, which adds to the bank of images that criminals use to generate deepfake pornography. Not only images are circulated. Names, women's telephone numbers and other private or sensitive information are also circulated, creating a web of harassment against them.

More than 6,000 South Korean women have so far in 2024 requested the removal of fake pornographic images made using their faces without their consent, according to South Korean government figures. The total for these same complaints in all of 2023 was 7,000.

Telegram refuses to exercise tight control over its users' content and messages. This stance and encouragement of freedom became a catalyst for the adoption of the social network by criminals and creators of deepfake porn.

Despite this, a Telegram representative assured the WSJ that the social network removes millions of pieces of harmful content every day on its platform, based on its moderation guidelines, A.I. tools and user reports. The social network remains, however, a favorite for many illicit activities and for communications meant to be encrypted and go unnoticed.

So far, the South Korean Police have claimed that the international location of Telegram's servers, which does not have infrastructure in South Korea, has made it very difficult to investigate cases of deepfake pornography.