The National Cyber Security Centre (NCSC) in the UK recently published an in-depth and well researched article about the inefficacy of phishing simulations and the misleading metrics that are meant to support their use.
It also warned against holding employees accountable if they repeatedly succumb to the social engineering techniques deployed by many and advocated alternative training methods such as gamification.
For us at Phishd – where our service offering is built to help organizations identify weak points and ingrain higher employee security in their company DNA – the NCSC’s article aligns with views we’ve held for a long time, such as the fact that too many SaaS phishing vendors are conducting simulated phishing attacks and reporting click percentages without giving any context around them. Worse still, in our experience, such simulated campaigns are often fired off at near random, without any research informing the campaign content or the key training messages required by the individual receiving them.
So, how can phishing simulations be made more effective?
This article, based on our years of experience in assessing, crafting, and managing employees’ security behavior, will detail:
- How to give meaning to click rates
- Why context, research, training, and clear objectives are essential to a meaningful simulation campaign
- When employee accountability is effective and motivating – and when it’s not
- What types of training – in our experience – yield the right change in employee behavior
How to give meaning to click rates
The ‘success’ of click rates often isn’t embedded in how ‘good’ or ‘effective’ the scenario was in which they were measured. Template phishing emails, designed to be delivered at scale by a large number of 'pop-up' SaaS training providers, who possess little real-world offensive or defensive experience, are often where these infinitesimal click rates originate. Time and again, we see people boasting of a tiny click through percentage on their last phishing simulation, only for it to be revealed that the campaign in question was of a very low quality.
Campaigns of this nature will not be effective in achieving the behavioral and organizational changes needed to prevent and protect companies from the impact of a successful phishing campaign. Sending phishing emails is easy – planning, designing and orchestrating campaigns that change behavior effectively is not.
Face value metrics, such as click rates, only tell a small part of the larger story. So, how can organizations create context around these? When we create context by manually identifying the properties and attributes within a campaign to give it an effectiveness grade and calculate that effectiveness based on millions of rows of historic testing data, we can thoroughly identify the aspects with which employees are struggling, as well as show improvement or regression over time with a high degree of accuracy and normalization.
It is true that many behavior training programs for employees take too much time and that companies should be more creative with training. While we agree wholeheartedly in principle, from our experience, training videos and games are too generic and often dismissed as irrelevant.
Instead, we are in favor of short, focused light-touch training that focuses on the key phishing properties and attributes that people are struggling to spot, informed by data. Gamification, for example, when well thought out and implemented, can be an engaging and effective way of getting employees to engage with improving their security behaviour.
Should employees be punished if they repeatedly take the phishing bait? Again, it depends on context. There is a huge difference between 'blaming' employees that fall for APT-level attacks versus 'blaming' employees that fall for really obvious scams ten times in a row despite training, education, clear reporting processes, and a focused support program.
However, in our experience it is the integrity of the focused support program that leads to huge improvements in employee behavior. Let's take for an example a company that we partner with of just over 100,000 employees. 20% of those employees were 'repeat offenders' for clicking on low-level phishing scams. Those 20,000 employees underwent a brief, but comprehensive, support program, and so much improvement was made across the board that only three of that number ended up reaching disciplinary procedures. Zero have been fired.
We've seen good success where firms have taken this approach, with no impact on the relationship between security and the wider business. Positive reinforcement – i.e. rewarding people who report attacks, especially those who report attacks after clicking – is also key here.
The art of intelligent simulation
Simulated phishing campaigns play a key role in changing employee behavior still, but they need to be done intelligently, utilizing effectiveness gradings to give click rates meaningful context. The generic training approaches utilized by SaaS phishing providers must be dropped and replaced by micro, point-in-time training that's based on root causes and identified from contextualized metrics. This is the exact reason we used millions of sets of data to build our effectiveness grading and leverage humans during our campaign delivery process to apply those gradings and give context to our metrics.