Harvard Business Review

Susan,
yes we miss real measurement on the real economic benefits of our IT investiment.

And yes, MANY IT investments are just ONE of the multiple variable that effect performance (as per you example with CRM).

ROI is often useless (we don't substitute people with SW any longer). Is it usefull to invent metrics that we already know are more or less fake as ROI? 'increases to cross-channel sales' ... come on ....

We NEED numbers BUT we are FAR ... we probably need a completely different approach.

For ex. is the entepreneur/owner thinking at his strategic investment always in term of ROI?

PierG
http://pierg.wordpress.com

Another way to really measure ROI is experimentation. Steven Levitt gave a good talk at Chicago GSB on using experiments to quantify the ROI on Marketing- another area where the ROI is hard to find. Listen at link...
http://www.chicagogsb.edu/multimedia/podcast/archive.aspx?play=http://www.chicagogsb.edu/multimedia/audio/2006-06-28_XP_Levitt.mp3

If you do a staggered roll out of IT systems, you might be able to do an A/B style test to measure their effects.

Great article, however one barrier in tracking ROI for IT is People or better stated change management. How many times has a new technology been handed off to the business users only to have them immeadiately return to the old processes and not even come close to gleaning the business case they signed up for. How many projects are deemed IT failure when the root cause is the business refusal to adopt new processes designed to improve their results but require change. In my experiance it's almost alway a result of business failure, not software or hardware or IT related. It's time to hold the busniess leadership accountable both up front in agreed to business case and on the back end to ensure the ROI.

Surely, it would be unwise to invent metrics that are unrelated to the core mission and strategy if the business, but it is equally unwise to view metrics as "fake". The balanced scorecard work has demonstrated that mapping strategy to metrics provides keeps everyone focused and grounded as to whether the strategy and/or execution is on track.

This is true for start ups and ongoing ventures.

There seems to be a resistance to metrics because people think that it will constrain creativity or require too much effort. I just finished reading a book where the author defended his decision not to define metrics before starting an organizaitonal transformation, but then spent the rest of his book using metrics to demonstrate the success of the change. Imagine the power of educating an organization on the core operational metrics that drive the business so that leaders at every level can focus and manage their innovation efforts?

Intel IT has done a wonderful job working through new approaches for managing IT value - more information can be found at:http://www.intel.com/it/business-value/it-metrics.htm

In the organizations for which I have worked, the expected benefits are not defined or controlled by IT, but by the user and sponsor community. You are correct that they are not usually measured or enforced, but as a previous respondent has noted, that is not an IT issue per se.

DeMarco and Lister, in their book "Waltzing with Bears" addressed this issue. Businesses do not hold the sponsors and user communities to nearly the level of accuracy in benefits that they hold the IT community to for costs. Your list of exhibits shows the accounting for IT on late schedules (usually based on schedules that were set when 10% of the requirements were defined, in my experience). Why not an exhibit that shows that 70% of projects return less than their sponsors promised?

Gary and Randy emphasize an important point - also mentioned in the last paragraph of the blog. The key to returns from IT-enabled business initiatives is by holding business executives accountable for their benefit commitments. Accountability increases executive involvement that, in turn, helps ensure successful delivery that, in turn, results in higher returns.

Susan:

You make relevant and timely points, particularly as they relate to IT investments in managing a company's litigation and e-discovery costs.

The proliferation of email and electronic data has changed the landscape of litigation and costs associated with litigation. Arguably, IT investments have never been more important to an organization’s bottom line. For example, Phillip Morris was sanctioned $2.75 million when executives deleted emails – note that there was no finding of bad faith on the part of these executives. A negative inference for failing to preserve email resulted in a $29.3 million verdict to plaintiff in Zubulake v. UBS Warburg. In Perelman v. Morgan Stanley, a jury verdict awarded plaintiff $1.4 Billion (yes BILLION) after the judge placed the burden on Morgan Stanley to show that plaintiff was not defrauded after Morgan Stanley was found to have repeatedly failed to produce relevant emails. One may think that bad faith destruction of evidence is costly. However, I think the real lesson here is that ignorance of the laws is not a defense, so the ordinary destruction of data and emails by employees can be costly in the context of litigation.

Empirical data on past costs to preserve, collect, review and produce data in litigation may be one way to set benchmarks for determining ROI. Working with your in-house legal department to analyze past legal bills as it relates to e-discovery may prove useful. How much time did lawyers and forensics experts bill on the last litigation for collecting, processing, hosting, reviewing, and producing data? Can you break it down to a cost per document? Do you have the internal infrastructure in place to defensibly preserve, collect, analyze, and produce the data? If so, how do the costs of outsourcing those tasks compare to having done it internally?

With courts increasingly more willing to impose costly sanctions and negative inferences for violations of discovery rules, IT investments to ensure compliance with the Federal Rules of Civil Procedure (FRCP) as it relates to “electronically stored information” (ESI) should be on the radar of all IT leaders, particularly as states begin to adopt the FRCP – California is one such state.

The financial impact of an IT investment to manage e-discovery costs and reduce the risks of adverse jury instructions for failing to comply with the FRCP can be significant on reducing legal spend. Empirical data is likely available for benchmarking ROI of such an IT investment.

I am reading this article and could not agree more regarding specifying metrics and Baselines during the initial steps of the IT project.
Though it might not be easy or clear how to measure the baselines and returns, it is important to emphasize what area in the business the initiative might or well affect. This will help making critical decisions during the project.

On another note, one of the difficulties that might impede these measurements is the long cycle if IT implementation, things change from the time that the project first started till the time it was fully implemented. When I say things, I mean company internal and external changes; economic environment changes; new products, new competitors...So we have to think how to isolate these changes while conducting the analysis.

"What you must do to calculate it is exploit the fact that there's a direct relationship between processes and financial results."

Exactly! it is the business process effectiveness that is being measured and not the value IT has brought in.

"Therefore, the returns to measure for the CRM initiative would be increases to cross-channel sales and a reduction in the volume of calls to the customer care center"

If the business process is not defined correctly to enable an enterprise dervice value, the measurement on returns by automating the same would also not make sene.

Also note that in the above example, reduction in number of volume of calls does not necessarily mean satisfied customers (or does it?). It may also mean that if the process is not implemented properly, the customers have given up on call center and have probably reconciled to the bad service support or to a competetive offering.

Great article, and great feedback!

An interesting behaviour seems to sit under this problem. We put the supplier (IT) in charge of the agenda (planning, justifying and delivering IT projects), and then we are surprised when those on the demand side of the equation (the business, and especially the business leaders) do not make an all-out effort to reap the rewards. Frankly, we have it back to front!

The new International Standard for Corporate Governance of Information Technology (ISO/IEC 38500) provides in 15 pages a very important set of guidance for business leaders. They are responsible for the effective, efficient and acceptable use of IT. They should set the agenda (define the demand), and they should ensure that they get the payback. The standard is not talking about the minutiae - it's focused on the big picture.

In six principles - Responsibility, Strategy, Acquisition, Performance, Conformance and Human Behaviour - the standard makes it clear that success with IT is more about how the organization behaves, rather than about the capabilities of the IT supplier.

Those who take time to understand the standard will realise that what so many have thought of as "IT Governance" and which Ms Cramm so correctly lambastes, is really just the supply side management trying hard to fill the gaps created by an improperly engaged business.

Hi,

As a program manager with one of the world's leading IT organizations, i have seen this problem rear its head up in so many situations.
But to be honest, though we are acknowledging the existence of the problem (i.e. lack of metrics to define IT investment) we have kind of veered away from finding the solution.
Susan, others - can we throw more light on really how to figure out appropriate metrics for judging IT investments. I am sure it will help a lot of us here...definitely me.

Marwa,
you are absolutely right. That's why it is so important to have the right governance and controls in the project to make sure the achieved goals are in line with the target that was baselined at kick-off!