San Francisco Orders Apple and Google to Purge AI 'Nudify' Apps From Their Stores

San Francisco city attorney David Chiu has issued cease-and-desist letters to Apple and Google, demanding that both companies remove 13 face-swapping applications from their respective app stores. The legal notices, sent on Thursday, accuse the technology giants of enabling the creation of AI-generated nonconsensual nude imagery and call on them to sever ties with the developers responsible.

According to the letters, California law prohibits supporting services that produce deepfake pornography. The apps in question rely on in-app payment systems, from which Apple and Google collect a portion of the revenue. Chiu's office estimates that the companies have likely earned millions of dollars in fees from apps offering nudification capabilities.

A Pattern of Harm Across App Stores

Researchers have repeatedly identified applications within Apple's App Store and Google's Play Store that allow users to generate sexual imagery using artificial intelligence. Some of these apps have even been rated as suitable for children. Despite existing developer policies at both companies that prohibit pornography, abuse, and harassment, harmful apps continue to surface on the platforms.

The 13 apps targeted by the city attorney's office — eight on the App Store and five on the Play Store — are broadly marketed as face-swapping tools. However, their capacity to generate sexual deepfakes becomes apparent once users begin interacting with them. One app, which has surpassed one million downloads, advertises over a dozen AI image styles featuring sexualized depictions of women. Another targeted app promotes what it calls "free and uncensored" videos on its homepage.

Over the past five years, a profitable ecosystem of deepfake nudification technology has proliferated online. The process often requires nothing more than a reference photograph and a few clicks, with some results generated in seconds. As the underlying generative AI has advanced, the resulting images and videos have grown increasingly realistic. Previous reporting has documented incidents at no fewer than 90 schools where deepfake sexual abuse images of minors were created.

Google Responds as Apple Stays Silent

Google spokesperson Dan Jackson confirmed that the company has removed hundreds of apps with nudifying features for policy violations, including the five Android apps flagged by Chiu's office. Jackson stated that Google Play does not permit apps containing sexual content and that the company takes proactive steps to detect and remove harmful material. He added that Google has suspended hundreds of violating apps and restricted related search terms such as "nudify" on its store.

Apple did not provide a comment prior to publication.

The problem is not new to either company. Over the past year, multiple investigations have identified apps on both platforms capable of producing nonconsensual nude images or videos. The Tech Transparency Project, an independent watchdog group, uncovered approximately 100 such apps across both stores in January and April of this year. The organization also found advertisements for nudifying technology on the platforms. According to TTP, the identified apps were collectively downloaded an estimated 480 million times and may have generated roughly $120 million in combined revenue.

Katie Paul, director of the Tech Transparency Project, said the group did not expect to see the issue persist after its initial report, only to find it equally severe — if not worse — upon a second investigation. She criticized the gap between the companies' safety promises and the reality on their platforms.

Researchers Highlight 'Dual-Use' App Loophole

A preprint research paper published in May by scholars from Cornell University and Georgetown University identified 420 face-swapping apps across Google's and Apple's app stores. The researchers tested 155 of them and found that 70 percent could be used to create face swaps with nude images, without any safety measures in place to prevent it.

The researchers noted that none of these apps were explicitly advertised as nudification tools. They described them as effectively "dual-use" — presenting themselves as benign applications while retaining the capacity to generate harmful content, thereby evading platform content moderation.

Chiu emphasized that his office will continue pursuing the issue and urged both companies to immediately remove the flagged apps and strengthen their screening systems. He warned that if the companies fail to act, his office will consider all available legal options.

The cease-and-desist letters mark an escalation in the broader effort to hold technology platforms accountable for the spread of AI-generated sexual abuse imagery. With researchers, watchdogs, and lawmakers increasingly spotlighting the gap between platform policies and enforcement, the pressure on Apple and Google is only expected to grow. If you found this report informative, please share it with your network to help spread awareness about this pressing issue.

Source: Wired