Measuring execution capability increases the probability of technology initiatives succeeding.

Measuring execution capability provides decision makers scored data that identifies which projects are ready to move forward with a higher probability to succeed, and which projects need to correct critical vulnerabilities that will prevent them from meeting stated financial and performance outcomes.
Like a FICO® Score, SAT® score, or bond rating an execution capability score removes the biases in the decision process to filter strategic or project opportunities to maximize outcomes. When an initiative is scored and its vulnerabilities are identified, decision makers have a clear and precise understanding of what needs to be addressed before funding/approval, reducing failure rates and wasted resources.

Put execution capability measurement into practice.

Traditional government and private sector practices for determining which information technology projects should be approved and funded are greatly affected by the biases of decision makers. These biases have had a well-documented impact on the success rate of initiatives.

Studies have clearly shown that when decision makers are evaluating information to make a choice, they introduce biases into their rationale for approving or rejecting the idea or project. Many industries have instituted methods and tools to remove decision bias and focus on producing data that measures actual capabilities rather than perceived ones.

Leaders have been seeking an approach that identifies which strategies and projects are ready, and which ones have critical vulnerabilities which must be addressed before they are approved to move forward.

Common types of biases

Confirmation Bias: Occurs when decision makers seek out evidence that confirms their previously held beliefs, while discounting or diminishing the impact of evidence in support of differing conclusions.

Anchoring Bias: The over-reliance on a single piece of information or experience to make subsequent judgments. This limits one’s ability to accurately interpret new, potentially relevant information.

Overconfidence Bias: Occurs when a person overestimates the reliability of their judgments. This can include the certainty one feels in their own ability, performance, level of control, or chance of success.

Halo Affect Bias: This is an observer’s overall impression of a person, company, brand, or product, and it influences the observer’s feelings and thoughts about that entity’s overall character or properties.

There are many well-known execution capability methods and tools to counter decision bias in practice today:

The Credit Score
A credit score is a statistic forecast of individual credit risk, and substitutes human judgement in deciding who to lend money to, because it is more accurate and faster.

The SAT/ACT
The SAT test is an un-biased measurement of college readiness and forecasts future academic success.

Personality/Cognitive Profiles (Myers-Briggs, DISC, etc.)
Measure behavioral styles and provides potential employers data to forecast an individual’s suitability for a specific role.

Bond Ratings
Private independent rating services provide un-biased scoring of bonds that indicate the issuer’s strength to pay back the bond and indicate the risk of investing.

 

These execution capability measurements are filters that remove bias and measure an individual’s or institution’s ability to execute. Current processes for approving IT projects do not apply any bias-removing filters for execution capability in the decision-making process. As a result, many projects are approved too early to move forward, which are then impacted by vulnerabilities which could have been corrected before the project life-cycle was started.

What are the steps to succeed?

1. Determine the capability for implementing the project
2. Reduce the financial risk in implementing the project
Increase the probability of successful implementation of the project
3. A detailed report that identifies strengths and weaknesses should be addressed before spending funds for projects

The report is part of the overall decision making process to identify where vulnerabilities exist and how they are being corrected before project initiation. This provides the ability to address critical issues on their own, without expending project dollars.

The model is intended to work as indicated below:

Should You analysis addresses the value of the initiative to the organization, whether it adds value to the business or program, and what the financial impacts are forecasted to be. The assessment of this material continues to be the same in the decision process.

Can You focuses on execution capability. Here, the ability of the organization to execute on the idea or project is measured, and vulnerabilities are corrected. The purpose is to remove risks and obstacles before they affect project execution, and before the project life-cycle is engaged and costs are incurred.

The correction of vulnerabilities before the engagement of a project is much less than if they are corrected after a project has been initiated. Studies have indicated that many of the root-cause factors for bad project outcomes are not IT-related, and that the project management practices are only reactive once the project has been engaged.

Where do gaps in execution capability come from?

When a government agency or corporate business unit take on a new initiative/project, the outcome will in some way transform current people, processes, and technologies (PPT). Some elements of PPT that meet business needs today will no longer work in the next phase of the business model, which is being introduced through the project being implemented.

How far or aggressive the change/transformation is determines how many of the current practices will remain, while new PPT are necessary to work in the future model after the project is complete.

 

The more the project transforms the business, the larger the gap in capability exists in the future operating model. That gap represents the vulnerabilities that exist in the project because current PPT cannot meet the future state needs. The result is that people must be re-trained or new hires are necessary with new skill sets that don’t currently exist, processes must be re-engineered, and technologies are replaced.

This gap in capability factor is what decision makers most often overlook through their decision biases. The assumption is that the elements in current capabilities have the skills, processes, and technical know-how to meet the future need. This incorrect assessment leads to overlooking vulnerabilities that will undermine the ability of the transformation project to succeed.

Gaps in people, process, and technology capability factors are what decision makers most often overlook through their decision biases.

An independent global analysis of projects with low outcomes indicated the following ranked factors of impact:

  1. Change in scope by the business
  2. Bad estimates
  3. Insufficient resources
  4. Change in business environment
  5. Change in business strategy
  6. Lack of management support
  7. Imprecise goals
  8. Insufficient budget
  9. Insufficient motivation
  10. Poor communication
  11. Poor quality of deliverable
  12. Wrong project management
  13. Stakeholders not adequately involved

Of the 13 factors identified, 10 of them can be corrected before any project is approved. Factors such as wrong project management, poor deliverables, and poor communication are most commonly only identified while a project is in execution, and therefore cannot be addressed as a vulnerability before initiation.

The remaining factors are the basis for execution capability analysis prior to approving and engaging a project. Through the development of domains of measurement that are directly correlated to the most common factors (of which there are many), the gap in capability and associated vulnerabilities can be calculated.

How to measure execution capability.

Execution capability is measured and represented as an un-biased score (rather than an opinion based assessment), as with similar execution capability metrics identified earlier in this position paper. Initiatives are scored independently. A scoring of overall execution capability of an institution is insufficient, as conditions vary from initiative to initiative. An overall assessment of an agency’s execution capability will not identify accurately the specific vulnerabilities that exist for a specific initiative.

The following factors are recommended for measuring execution capability for technology resource projects:

Applying Domains of Measurement
An execution capability metric uses domains of measurement to identify classes and sub-classes of factors that affect projects. Each domain addresses a specific area of measurement (such as: alignment to strategy, operational capabilities, understanding of business rules & processes, clarity of vision, technical capabilities, management capabilities, etc.)
Indirect Data Collection
Data is collected through indirect survey methodologies to remove gaming and other forms of participant response manipulation. Each question must be weighted through the entire survey, and multiple domains are addressed indirectly.
Algorithm Based Scoring
As domain data is collected through indirect survey data, weighting and scoring of each domain is represented. Domain scores collectively contribute to a single execution capability score, which is weighted from method based criteria.
Vulnerability Identification
As scoring identifies the execution capability with each domain, vulnerabilities are listed from established criteria that relate to the specific domain and type of initiative being measured.

 
Compiled through structured methods of data collection and scoring (i.e. SAT, personality profiles, bond ratings and credit scores), execution capability for each specific initiative reflects a moment in time for the specific project and the specific conditions that will impact its execution.

The Origins of Execution Capability

Execution capability is a sub-category of decision bias, which is a principal factor measured in behavioral economic theory. Behavioral economics, along with the related sub-field behavioral finance, analyzes the effects of psychological, social, cognitive, and emotional factors on the economic decisions of individuals and institutions, and the consequences for market prices, returns, and resource allocation.

Behavioral economics is primarily concerned with the bounds of rationality of economic agents and functions as a method of economic analysis that applies psychological insights into human behavior to explain economic decision-making and the impacts of decision bias.

There are three prevalent themes in behavioral finance:

  1. Heuristics: Humans make 95% of their decisions using mental shortcuts or rules of thumb.
  2. Framing: The collection of anecdotes and stereotypes that make up the mental emotional filters individuals rely on to understand and respond to events.
  3. Market inefficiencies: These include mispricings and non-rational decision making.

Decision bias – also known as cognitive bias – are tendencies to think in certain ways that can lead to systematic deviations from a standard of rationality or good judgment, and are often studied in behavioral economics.

Although the reality of these biases is confirmed by replicable research, there are often controversies about how to classify these biases. Some are effects of information-processing rules (i.e., mental shortcuts), called heuristics, that the brain uses to produce decisions or judgments.

Biases have a variety of forms and appear as cognitive (“cold”) bias, such as mental noise, or motivational (“hot”) bias, such as when beliefs are distorted by wishful thinking. Both effects can be present at the same time.