In our last blog, we explained how aligning the stages of the maturity model with your stakeholders' CTI requirements positions you to adopt a series of metrics to measure your effectiveness across each EclecticIQ CTI Capability Maturity Model pillar:

• Alignment between the business and your understanding of the threat landscape (Align) – Metrics measure how well your organization aligns the business or mission to your threat reality.
• Your understanding of the threat landscape (Understand) – Metrics measure how well your CTI team is functioning and engaging your stakeholders.
• Your ability to act on your understanding of the threat landscape (Act) – Metrics focus on the people, processes, and technology you have in place to act upon your understanding of the threat landscape and its potential impact on your attack surface.

This blog defines an approach that maps metrics to your stakeholders’ requirements. We call this approach the Pyramid of Gain (PoG) (see Figure 1). And yes, this intentionally resembles the well-known Pyramid of Pain.  While the Pyramid of Pain helps you shift your focus from IOCs to TTPs to better detect and respond to threats, the Pyramid of Gain enables you to progress from responsive to pre-emptive threat operations to better meet your stakeholders' needs. 

Figure 1 - The EclecticIQ Pyramid of Gain (PoG) (click the image to zoom in)

The components of the pyramid

The PoG depicts moving from lower to higher maturity through the five stages of the CTI-CMM. The five stages are grouped into tiers on the pyramid. The bottom tier is responsive CTI maturity, equating with Stages 1 to 3. The middle tier is predictive CTI maturity, equating with Stage 4. The top tier is pre-emptive CTI maturity, equating with Stage 5.

At each tier, your CTI team supports a different set of stakeholders:

• Bottom tier – This tier supports your operational stakeholders, feeding threat intelligence to your operational functions including the SOC, vulnerability management, and fraud operations. In other words, "Here is what we know; now go do something about this." Activities are primarily responsive.
• Middle tier – This tier supports your tactical stakeholders, including incident response, incident operations, and threat hunting. At this tier, you have appropriate controls and enough situational awareness to know you are compromised. In other words, "We know that actor A is exploiting a successful spear phishing attack, so we need to stop the attack, hunt for related breaches, and block future exploits."
• Top tier – This tier supports your strategic stakeholders, including compliance, risk, strategic planning, executives, and the board of directors. At this tier, you focus on the most strategic aspects of CTI. Here, you make pre-emptive decisions, as in "Kreplackistan's government is actively targeting industrial companies with ransomware and hacking into private company Internet traffic, so we are canceling our planned chemical manufacturing plant in the capital city, Kreplach."

Moving "up" the pyramid incrementally expands your CTI capabilities from responsive support of operational stakeholders to predictive support of tactical stakeholders to pre-emptive support of strategic stakeholders. At each tier, we recommend a series of effectiveness and performance metrics. Generally, the higher the level, the more your team focuses on effectiveness metrics. However, both types of metrics are essential for successful CTI operations.

EclecticIQ is taking a novel approach by precisely aligning metrics with your stakeholders' needs. We recommend picking a subset of metrics for each tier that best aligns with their goals. Many of these metrics draw on excellent work by several practicing threat analysts and researchers. [1]

Metrics at the responsive CTI operations tier

Align – Your threat reality alignment is quite limited at this tier. You may have defined PIRs, but many of your actions are entirely reactive, ad hoc, and operational (e.g., firewall block lists). Key metrics include:

• Number and percentage of RFIs answered and fulfilled
• Number of views and downloads of products
• Number of products created
• Number and percentage of survey replies
• Percentage of ad hoc PIRs
• Frequency of stakeholder surveys
• Number and percentage of threat actors catalogued
• Number of threats identified

Understand – At this tier, your CTI team supports your operational CTI stakeholders. Most of your efforts are focused on providing IOCs to your SecOps team and updated vulnerability data to your vulnerability and fraud operations teams. Key metrics include: 

• Percentage of IOCs enriched
• Quantity of IOCs observed across security solutions
• Intelligence to false positive ratio
• Frequency of source analysis and realignment (e.g., weekly, monthly, annually, as needed)
• Percentage of open source feeds
• Frequency of sharing

Act – Here, your emphasis is on reactive controls that attempt to block ongoing attacks, for example, feeding lightly filtered IOCs to your firewalls and endpoint protection controls. Your SOC tracks MTTD as a primary performance metric. Key CTI metrics that support your ability to act (and bring down MTTD) include:

• Number of intelligence packages delivered

• Percentage of packages with context/confidence

• Rate of disseminated intelligence targeting specific stakeholders

• Frequency of integrating multiple IOCs

• Number of sightings

• Number of detected vulnerabilities

• Percentage of new incidents discovered from sightings (e.g., false positives)

Metrics at the predictive CTI operations tier

Align - When you rise to this tier, you have procedures in place to briefly catch your breath by shifting from a reactive to a more proactive stance. At this point, you have well-formed PIRs, and your CTI team is receiving regular feedback to adjust the PIRs to reflect the shifting threat landscape. Further, you are including CTI in ongoing business and mission decisions. Key metrics at this level include:

• Percentage of stakeholders using intelligence products in decision making
• Percentage of products used in decision making for cyber-related issues
• Percentage of PIRs receiving intelligence feedback
• Percentage of qualitative feedback loops completed for standing PIRs
• Number of TTPs attributed to threat actors
• Quantity of intelligence products created that include forecasting and filtering per PIR

Understand – Your CTI team provides more actionable data based on more-robust feeds, IOCs, and case management practices. This CTI maturity tier makes it easier to support your tactical stakeholders with intelligence for conducting incident operations and looking for "unknown unknowns" via threat hunting. Critical metrics at this level include: 

• Percentage of internal incidents that have had CTI follow-up
• Number of new incidents resolved or supported through CTI
• Amount of new intelligence produced from IR cases
• Percentage of feeds providing unique analysis capabilities
• Feed turnover rate (new vs. dropped feeds) over the past year
• ISAC/NCSC engagement
• Data sensitivity tracking
• Percentage of supply chain engaged with threat intelligence
• Frequency of sharing intelligence with trusted partners

Act ¬– At this tier, your organization is shifting to a more proactive stance by enabling incident response, incident management, and threat hunting. You are leveraging context and insights from the CTI team to provide your response and hunting teams with actionable data. You are paying close attention to MTTR as a crucial operational metric. Other key metrics that support your ability to act (and reduce MTTR) include: 

• Percentage of targeted intelligence with input from stakeholders
• Frequency of invalid hypotheses based on stakeholder input
• Vulnerabilities proactively patched
• Percentage of IOCs with TTPs
• Percentage of IOCs with MITRE ATT&CK data

Pre-emptive CTI operations tier

Align – At this level of the pyramid, your business and mission are so closely aligned with your threat reality that you can become pre-emptive. For example, leveraging CTI that includes TTPs and projected threat actor targets, you may decide not to shift your operations from on premises to a recently breached cloud provider. Metrics at this level include: 

• Percentage of significant business decisions (e.g., M&A) incorporating CTI
• Percentage of PIRs adjusted per regular feedback
• Percentage of assessments made from incorrect intelligence products

Understand – Your team has raised the maturity bar to a level that provides pre-emptive threat management, giving high-level stakeholders insights that help achieve comprehensive situational awareness and risk management. Effectiveness metrics for understanding at this level include: 

• Prediction accuracy
• False positive ratio for ingested feeds (feed efficiency)
• Attacks stopped by partners based on your shared intelligence

Act – Here, you are disseminating actionable CTI to your partners and supply chain vendors, and integrating CTI into your controls and workflows with context, priority, specific courses of action, and intelligence that is meaningful to your highest-level stakeholders. For example, at this level, your threat modeling (i.e., red/blue and purple teams) includes MITRE ATT&CK TTPs from targeted threat actors to pre-empt future attacks. Metrics that support your ability to act pre-emptively include: 

• The number of CTI-driven pre-emptive moves. For example, intelligence dissemination leads a supply chain member organization to swap out a targeted software program.
• Alignment of your security controls to your threat landscape. For example, you are using active MITRE ATT&CK information based on current threat intelligence in your penetration testing.

Yes, we just provided nearly 50 metrics to implement to track your CTI maturity progress! We recognize that implementing and tracking metrics is a significant undertaking. On the other hand, so is evolving your CTI maturity. The important point is that this is a long journey that starts with the commitment to take the journey. Most organizations will only implement a subset of these metrics. However, any metric adoption is a move in the right direction. In our next blog, we map six steps to begin your CTI maturity adventure.

Reference

[1] Gert-Jan Bruggink, Freddy Murstad, and “How to Get Promoted: Developing Metrics to Show How Threat Intel Works” - SANS CTI Summit 2019, https://www.youtube.com/watch?v=-d38C3992aQ