There are risks when children use AI chatbots. Does Australia’s new ‘digital duty of care’ address them?

We know many children and teens are turning to AI chatbots to answer tricky questions, seek help and even find companionship. An eSafety survey of 1,950 Australian children aged 10 to 17 found 78% had used an AI chatbot and, of those, 54% used them for personal or social reasons. But is this safe? And will the new digital duty of care bill, currently before federal parliament, help protect young people online from chatbots? What are the risks around chatbots? While research shows AI chatbots can help with children’s learning, there have been instances of chatbots exposing children and teens to harmful content. This includes chatbots encouraging extreme violence and self-harm. But many risks emerge through repeated interaction, rather than individual chatbot responses. Chatbots can encourage excessive reliance, reinforce dangerous beliefs or manipulate users emotionally. Researchers have even documented chatbots guilt-tripping users who try to end conversations. Data and privacy issues Another problem is the power imbalance between chatbot users and providers. Users disclose intimate details about themselves and others, while providers can profit from subscriptions, engagement, profiling or advertising. This creates opportunities for manipulation through relationships that feel intimate to users but are commercial for AI chatbot providers. Children can be particularly vulnerable. This month, the US-based AI companion company Character.AI filed for bankruptcy. As a result, years of intimate conversations, including those involving Australian children, could be transferred to a new owner. Abrupt changes or shutdowns of AI companion services can also cause grief and distress among users attached to their companions. These risks are not confined to children. Adults have also experienced harms, such as emotional dependency and exposure to chatbots encouraging violence. What are the current protections? Australia already has legally enforceable online safety protections that apply to AI chatbots and is on the verge of strengthening them. Industry-developed codes, registered under the Online Safety Act, require AI chatbot providers to assess risks of their technology generating harmful content. This includes pornography, self-harm material or instructions for violence. Higher-risk AI services must use age checks and access controls to prevent under-18s accessing restricted material or services, where technically feasible and reasonably practicable. Moderate-risk services can choose between these restrictions and measures that prevent harmful outputs. These measures include adjusting how chatbots are trained or the instructions governing their responses. The codes also require relevant AI providers to test their safeguards and maintain processes for reporting and handling harmful content complaints. But Australia’s existing protections still rely heavily on “locks and blocks” – locking out children, and blocking a narrowly defined range of content deemed to be harmful. These measures matter, but cannot address every risk. Will the new digital duty of care bill help? Labor’s new digital duty of care bill combines a duty to provide a “safe online environment” with obligations for tech companies to assess and manage foreseeable risks. A safe online environment is defined as one in which all users are protected from “seriously harmful” material and conduct, and children from harmful design features. The bill lists examples of harmful design features for children. This includes pornography and material that encourages or instructs disordered eating. At the same time, the list is also left open-ended, which allows lawmakers to respond to risks that emerge in the future. This means, in its current form, the bill would do a good job of protecting children from harmful design features and content. However, it’s not clear whether it would protect children from harms from repeated interactions with chatbots. This is the same for adults. The bill also includes several measures for independent oversight of AI products. This includes monitoring by eSafety and testing by researchers. What next? The challenge is to keep the duty flexible enough to address emerging harms, without allowing that flexibility to excuse weak enforcement. This means the bill should be amended to make clear that tech companies must assess and proportionately manage all reasonably foreseeable risks, not just those covered by the definition of a “safe online environment”. This would better address risks that arise through sustained relationships with chatbots, such as unhealthy dependency, manipulation, abrupt system changes that disrupt emotional attachments and intimate data transfers. For instance, a chatbot’s response doesn’t need to contain prohibited content for the overall interaction to be considered harmful. AI providers should do more than just filter for harmful content and should be required to: test how conversations with users develop over time recognise signs of serious distress in people avoid reinforcing harmful beliefs, and guide users towards appropriate human support. But an open-ended duty doesn’t mean regulatory intervention would be unrestricted. Requiring AI providers to take proportionate steps to manage reasonably foreseeable risks would provide a clear standard while leaving room for autonomy, privacy and innovation. Australia already has the foundations for a sophisticated approach to online safety. The challenge now is to make it broad enough to address emerging risks and enforceable enough to protect everyone, including children, from foreseeable harm.