Before creating a plan to build or improve threat intelligence for your organization, it is vital to establish where you are today and where you want to be within the next 12 to 18 months. Your stakeholders need to agree on the level of maturity in the intelligence capabilities you want to achieve.
We recommend creating a spider diagram (see Figure 1) that maps the eight capabilities against the five-stage scale to track your progress. This approach allows you to create a chart that visualizes your current situation versus your desired state, clearly showing your gaps.
Figure 1 - Tracking current vs. desired state
Creating spider diagrams is essential for tracking your maturity progress. However, this is just the first step toward achieving CTI maturity. Yes, showing management your progress as you move up the maturity scale is vital. But improvement only gets you so far; your outcomes must be measurable to drive results. To underscore this point, based on SANS 2022 research, three-quarters of CTI practitioners believe that CTI is helping, though less than 40% can calculate by how much. [1]
Measuring is important, but knowing what you are measuring is essential
To put this in perspective, as a CTI leader, you'll probably get a big thumbs up from your executives when you present your spider chart that shows your team moving from Stage 2 to Stage 4 in multiple areas. Though an admirable achievement, you'd undoubtedly prefer to show that you've increased the number of pre-emptive moves due to CTI by, say, 50%. Doing so requires adding a quantitative layer to your CTI maturity process.
However, before implementing a bunch of CTI metrics, it's worthwhile to pause and ask yourself what your metrics are measuring: if you are doing something correctly or if you are doing the correct thing. The former measures performance and the latter measures effectiveness. Both are important, but the risk of focusing on performance metrics alone is that you may be excelling at things that are not benefiting your organization. One way to look at this is that over-reliance on performance metrics drives a possibility rather than a probability mentality. For example, "Sending 50% more IOCs to the SOC increases your possibility of preventing a breach." In contrast, "Reducing your attack surface by 50% cuts your probability of a breach."
By adding effectiveness metrics, you can answer some of your most challenging questions:
- Given limited resources, how do I decide where to invest in improving our CTI maturity? If you must make budget cuts, understanding effectiveness allows you to protect the capabilities and associated practices that have the greatest positive effect on the organization.
- How do I know if my team is making a difference? Without an effectiveness measure, you only know how well your team is doing, but not its impact.
It's all about your stakeholders
Creating business value from threat intelligence relies on understanding key organizational stakeholders' information needs and requirements. These stakeholders are ultimately responsible for deterring, defeating, and preventing cyber threats (see Figure 2).
Figure 2 - CTI team stakeholder landscape
Therefore, start developing your metrics by aligning your metrics to your stakeholders' needs. Things to consider include what cadence they prefer to consume intelligence, the format of that intelligence, how their needs align with your priority intelligence requirements (PIRs), and what essential intelligence they require.
As you develop your metrics, you'll find they vary by stakeholder. For example, your security operations center (SOC) metrics are often performance based (e.g., percentage and number of incidents identified/ignored, rate and number of potential threat detections by firewalls and antivirus). In comparison, for risk management, effectiveness metrics are more appropriate (e.g., risk reduction over time, percentage of correct assessments by intelligence products).
To align the stages of the maturity model with your stakeholders' CTI requirements, the CTI-CMM provides metrics that help you quantify the following:
- Alignment between the business and your understanding of the threat landscape (Align) – Metrics measure how well your organization aligns the business or mission to your threat reality.
- Your understanding of the threat landscape (Understand) – Metrics measure how well your CTI team is functioning and engaging your stakeholders.
- Your ability to act on your understanding of the threat landscape (Act) – Metrics focus on the people, processes, and technology you have in place to act upon your understanding of the threat landscape and its potential impact on your attack surface.
At this point, you have the justification for a maturity model, an innovative CTI maturity model, and a general understanding of what metrics you must track to measure your progress. As you’ll read in our next blog, the metrics we define are pretty different than your typical SOC-focused metrics, such as mean time to detect and respond (MTTD/R). Though these metrics are vital, as you’ll read, there are a whole series of effectiveness and performance metrics that are unique to a CTI operation.
Reference
[1] SANS 2022 CTI Survey
