What AI Has Done to the Phishing Threat and Why It Changes Detection

The Volume Framing Misses the Point

The previous article on the fraud killchain detection gap established where bank detection systems first have visibility of an APP fraud attack and how much of the social engineering kill chain completes before that point. This article examines something more specific: what artificial intelligence has done to the signal characteristics of phishing in the pre-transaction window and why the structural ceiling on existing detection is getting lower regardless of investment.

Fraud directors have received years of briefings on AI-enhanced phishing. Most of that framing has concentrated on scale: attacks are more frequent, more automated, better targeted. That is accurate. It is not the problem that changes the detection.

Volume is a capacity problem. Signal degradation is a structural one. A capacity problem is addressed by more of what you have. A structural problem is not.

What signal-dependent detection actually relies on

Existing phishing detection operates across several layers, each relying on identifiable anomalies in fraudulent content or infrastructure.

Content-level signals include spelling errors, grammar inconsistencies and tone mismatches that distinguish fraudulent from legitimate communications. At the structural level, domain names that approximate known organisations through typosquatting or suspicious top-level domains generate identifiable anomalies. Infrastructure signals include newly registered domains, low sender reputation scores and email headers inconsistent with the claimed origin.

These signals accumulated because traditional phishing was built for volume, not precision. An email sent to a million recipients could afford poor construction. The attacker’s economics favoured scale over quality and detection systems were calibrated accordingly.

AI inverts those economics.

What AI Has Changed in the Attack

Linguistic quality: the baseline signal that has gone

The most immediate change is linguistic. AI large language models produce grammatically accurate, contextually appropriate content at speed and at scale, without the spelling and structural irregularities that content filters use as primary signals. This is not incremental improvement in attacker craft. It is the removal of an entire signal category.

According to KnowBe4’s 2025 Phishing Threat Trends Report, 82.6% of phishing emails analysed between September 2024 and February 2025 contained AI-generated content.¹

The NCSC’s January 2024 assessment was direct on the consequence: GenAI and LLMs make it difficult for everyone, regardless of technical background, to assess whether a communication is genuine.² The NCSC’s May 2025 follow-up report confirms that this condition is now operational rather than forecast, with AI assessed as “almost certainly” continuing to make social engineering more effective and efficient through 2027. The same assessment explicitly notes that technical surprise in AI capability development remains likely.³

Personalisation: the economics of spear-phishing have changed

Traditional spear-phishing required skill, time and open-source research. Its economics made it rare. A sophisticated, personalised attack might take an experienced operator several hours to build, limiting its use to high-value targets.

AI tools change that arithmetic directly. The NCSC’s Annual Review 2025, covering September 2024 to August 2025, records fully automated spear-phishing campaigns as already deployed in the wild.⁴ This is a retrospective assessment of observed activity, not a prediction.

The Verizon 2026 Data Breach Investigations Report, the largest breach dataset the report has ever examined at over 22,000 confirmed breaches, quantifies AI integration into attacker toolkits directly. The median threat actor now uses AI assistance across 15 documented techniques, with some actors using it across as many as 40 to 50.⁸ The 2026 DBIR also finds that 62% of breaches involved the human element, a figure the report attributes in part to AI-assisted deception making social engineering more convincing and harder to identify.

The commercial dark web market reflects the same shift. Tools including WormGPT and FraudGPT offer subscription-based access to LLMs designed to generate personalised phishing content across multiple languages, without linguistic errors, at scale. Palo Alto Networks’ Unit 42 documented multiple multifunctional platforms in 2025 offering phishing generation, deepfake production and code obfuscation under SaaS-style pricing and support structures.⁵

The significance is not that these tools exist. It is what their existence means for detection. Quality of attacker output previously indicated attacker capability. When that capability is available on a monthly subscription, output quality no longer carries that signal.

The Platform Layer and Why It Matters

Where manipulation starts in 2025

Understanding the signal problem requires establishing where fraudulent contact actually initiates. UK Finance’s Half Year Fraud Report 2025 confirms that 66% of APP fraud cases in the first half of 2025 originated on online platforms, specifically social media, online marketplaces and messaging services, before any interaction with the victim’s financial institution.⁶

The figure covers all APP fraud categories by volume. The AI-enhanced extended manipulation pattern described in this article applies most acutely to investment and romance fraud, which together account for the majority of APP losses by value. Investment fraud alone recorded a 55% increase in losses in H1 2025, with an average loss per case more than twenty times that of purchase fraud.⁶ These are the categories where AI-assisted social engineering carries the most material financial consequence, operating over extended periods in channels invisible to bank detection.

This figure was 70% in full year 2024.⁷ The pattern is structural and consistent across multiple reporting periods. AI-enhanced phishing extends the sophistication and duration of manipulation in that unmonitored environment without producing any corresponding increase in signal detectable by downstream bank controls.

UK Finance’s own statement in the H1 2025 report is precise on this point: by the time a bank has any chance of identifying fraud, there has often already been extensive contact with the victim on social media or other platforms that is invisible to the financial sector.⁶

Infrastructure signals and how they are being bypassed

A historically reliable infrastructure signal in phishing detection was the domain name itself. Domains registered to approximate known organisations created identifiable anomalies in reputation scoring systems.

AI-assisted attacks increasingly route victims through legitimate intermediary services before reaching a fraudulent destination. The domain presented at the point of initial contact may be a legitimate messaging platform, URL shortener or QR code service, not a fraudulent domain. Reputation-based scoring has no mechanism to flag a legitimate service hosting a fraudulent redirect.

Separately, targeted attacks use domains registered specifically for a single campaign and decommissioned within hours, before reputation systems build a history. Neither technique is novel. AI has lowered the operational cost of executing both at scale. The technical resource required to manage ephemeral domain infrastructure has fallen accordingly, making these approaches accessible to a broader class of attacker.

What This Means for Detection Architectures

Why filter investment does not resolve a structural signal problem

The preceding sections establish a specific problem: the signal categories that content and infrastructure filters depend on are being systematically removed from attacker output. A filter cannot detect what produces no signal. Adding analyst resource, tuning detection rules, or increasing filter capacity addresses volume. It does not address the structural degradation of the signal inputs those filters consume.

This applies equally to AI-assisted content filtering. Machine learning applied to content classification faces the same structural ceiling as rule-based filtering when the content signals it is trained to detect have been removed by AI-generated output. The more significant defensive AI development is at the behavioural and session layer, where detection operates on the execution of an attack rather than its content and where AI-generated phrasing quality is not a relevant signal. That distinction matters for where defensive investment is directed.

The NCSC’s threat assessment sits alongside the FCA’s Consumer Duty obligations, which require firms to act to prevent foreseeable harm to customers. Where the threat environment is documented and the structural gap in detection architecture is identifiable, the case for acting upstream of that gap strengthens on both cyber security and regulatory grounds.

The NCSC’s May 2025 assessment, covering the period to 2027, confirms that AI will “almost certainly” continue to make social engineering more effective and efficient. Technical surprise in AI capability development, the assessment notes, remains likely.³ The Verizon 2026 DBIR independently confirms that AI is now a measurable component of active attacker toolkits, with the human element present in 62% of confirmed breaches analysed.⁸

Where signal remains available in the kill chain

The point in the APP fraud kill chain where AI has not yet eliminated signal is the pre-session window identified in the previous article: the period before a banking session opens, where the first five stages of an attack complete without any bank visibility. Detection at the device layer is the technical response to that timing gap. It operates on signals generated by the attack’s execution in that window, before the victim opens their banking application.

Social engineering attacks leave observable traces at the device level: URLs visited, content loaded, browsing sequences, time spent on specific pages. These are not linguistic signals that AI can replace. They are behavioural and environmental signals generated by the execution of the attack rather than its content. Detection at this layer does not require assessing the quality of a phishing message. It requires observing what is happening on the device while manipulation is in progress.

Pre-session device-level detection is not yet consistently deployed across the UK market, with adoption remaining uneven between institutions. It identifies where a detectable signal currently exists: one that content-level controls cannot access and that AI-generated content quality cannot retroactively remove from the interaction record. The NCSC’s assessment that AI capability development involves likely technical surprise means this position should be treated as the current strongest available signal rather than a permanent structural answer.

Detection at the device layer operates within an established legal and regulatory framework. UK GDPR Article 6 lawful basis requirements apply. Fraud prevention monitoring of this type typically operates under legitimate interests provisions, as recognised by ICO guidance on financial crime monitoring. The Data (Use and Access) Act 2025, discussed in the first article in this series, introduced a recognised legitimate interests lawful basis for crime prevention data sharing between institutions: that basis governs OCC participation. Device-level monitoring of an institution’s own customers operates under a distinct but equally established provision within the same legal framework. UK financial institutions are already navigating both in the context of existing session monitoring and behavioural analytics deployments. It is a framework that warrants careful implementation planning. It is not a barrier to the approach.

The Compliance and Cost Question That Follows

What the signal problem means for detection investment

AI has not created the upstream detection gap. The gap between where manipulation begins and where bank systems first have visibility existed before AI-generated phishing tools entered widespread circulation. What AI has done is widen that gap by accelerating and extending the social engineering phase and lower the barrier of entry to the class of attack that operates within it.

The implication for fraud directors is that detection infrastructure calibrated against historical signal characteristics faces an accuracy ceiling that additional investment in that infrastructure cannot resolve. Tuning existing filters does not restore signals that have been structurally removed from attacker output.

APP fraud losses totalled £450.7 million in 2024, per UK Finance.⁷ The cost cascade behind that figure, from investigation through to reimbursement liability, is examined in detail in a later article in this series. The structural point established here is that additional investment in signal-dependent filtering does not change the ceiling. It changes the cost of operating beneath it.

For compliance teams, the question that follows directly from this diagnosis is whether the frameworks built around existing detection approaches remain adequate when those approaches face a documented structural ceiling. Under the FCA’s Consumer Duty, firms are required to act to prevent foreseeable harm. Where the threat environment is documented and a structural gap in the detection architecture is identifiable, the case for acting upstream of that gap is both operational and regulatory. That argument is the subject of the next article in this series.