How to Judge Security Products That Claim to Use AI

A simple way to call BS on security products that use 'AI'...

Security vendors are more desperate than ever to tell you they are using AI. It's been a trend for years but, since ChatGPT, almost every vendor now really wants you to know they are an 'AI company' under the hood.

Anyone at RSA last year saw this trend in full force. When creating marketing for the event, it looked like every vendor had taken their stand from 2023, added the letters A and I somewhere, and called it a day. Yet, anyone using these products day to day knows they've barely changed. It must be an absolute nightmare for buyers.

Modern AI is doing incredible things. It surely has the potential to make life significantly easier for security teams. But if every vendor claims to be an AI company, how do you know where to look?

AI and machine learning are complex topics. Many vendors use this complexity to exaggerate or outright lie about how their tech works. So what questions can you ask to cut through the noise and understand which products actually use AI?

First, a caveat. Just because you use LLMs or traditional machine learning in your product this does NOT guarantee that it will be better than one that doesn't. In fact, if the right level of skill and care isn't applied, the product will be worse. It's also true that in security we're close to the upper limits of what traditional software that doesn't leverage AI can do. Why else do you think we have so many lookalike products?

So, if we think AI has huge potential and that we're at the upper bound of what traditional software can do, where does that leave us? It tells us that if we want to find products that create step-change outcomes, we probably need to be looking for products that a) actually use AI, b) use it extensively and c) use it effectively.

So, when reviewing vendors, he are some things you can look into:

1) When was the product first built?

If it was built pre ~2015 it almost certainly doesn’t use modern machine learning at its core and is mostly rule-based. If it was built pre-2022 it almost certainly doesn’t use LLMs/AI Agents at its core, and is either rule-based or relies on more primitive machine learning.

Of course vendors can add features later, but they almost never rebuild and start again. As a result, the newer more powerful tech is used only for additive features, not to drive the core experience.

2) How extensively is AI used? Is it central to the product, or just used for minor features?

This is the most important question. The cause of most of the confusion around AI in security is vendors adding 1-n non-critical features that use AI or do something ‘intelligent’ and then claiming to be an AI company. You need to understand whether it’s absolutely central to the product, or just layered in on top.

3) Is the product able to do something meaningfully different?

LLMs or traditional machine learning used effectively should enable totally new use cases. For example, in email security, machine learning and behavioural analysis enabled highly targeted spear phishing attacks to be stopped, where threat intelligence could not.

If you’re looking at a product and they can’t articulate what they can do differently because AI and are just saying ‘well it makes us better’, walk away.

4) Is the product effective?

Arguably this is the hardest of all to measure from a distance. Speaking to peers is usually the only way to cut through the noise here. Or maybe if you’re satisfied with questions 1-3, you might just want to trial the product and see for yourself.

I don’t have time for all that, what’s the one question I can ask?

Okay so maybe you’re on your 18th AI based vendor pitch of the month and you don’t have time to be doing a bunch of diligence, fair enough.

Here is one simple question you can ask that gets to the heart of the issue:

“If your LLM provider stopped working, what would happen to your product?”

If the answer isn’t some variant of ‘the product will stop working’, then you know where you stand. The product might be using modern AI somewhere, or it might be using 2015-2020 ear machine learning, but not using modern AI at its core. This doesn’t necessarily make it a bad product, it’s just unlikely to be orders of magnitude better than anything you’ve used before.