As cited
Copy frozen at (site build).
threat intel
Choose your fighter: Balancing competing requirements to select models for your AI SOC
No summary had been written when this copy was frozen.
- First seen by Cybersecurity Tracker
Source attribution
Correction
Correction recorded as of .
threat intel
Choose your fighter: Balancing competing requirements to select models for your AI SOC
Cisco Talos evaluated 66 large language model (LLM) and reasoning combinations from Anthropic and OpenAI on a simulated log analysis task to guide model selection for security operations centers (SOCs) and incident response work. Rather than identifying a single best performer, the researchers developed a methodology using Pareto frontier analysis to balance four key factors: accuracy, cost, speed, and consistency. Key findings showed that higher reasoning effort did not reliably improve results, analyst role and prompts significantly influenced conclusions, and some models produced unacceptable failure rates at higher reasoning settings.
Why it matters: SOC teams choosing LLMs for incident response must benchmark models against their own production workflows and measure quality, cost, time, consistency, and failure rates rather than relying on generic leaderboards or assuming more expensive reasoning settings guarantee better outcomes.
- Source published
- First seen by Cybersecurity Tracker
Source attribution
Correction
Correction recorded as of .
threat intel
Choose your fighter: Balancing competing requirements to select models for your AI SOC
Cisco Talos evaluated 66 large language model (LLM) and reasoning combinations from Anthropic and OpenAI on a simulated log analysis task to guide model selection for security operations centers (SOCs) and incident response work. Rather than identifying a single best performer, the researchers developed a methodology using Pareto frontier analysis to balance four key factors: accuracy, cost, speed, and consistency. Key findings showed that higher reasoning effort did not reliably improve results, analyst role and prompts significantly influenced conclusions, and some models produced unacceptable failure rates at higher reasoning settings.
Why it matters: SOC teams choosing LLMs for incident response must benchmark models against their own production workflows and measure quality, cost, time, consistency, and failure rates rather than relying on generic leaderboards or assuming more expensive reasoning settings guarantee better outcomes.
- Source published
- First seen by Cybersecurity Tracker