A FEW WORDS ABOUT

METRICS





A metric is a measurement that characterizes some aspect of an entity - be it a product, a process or a company. It's often been said that you can't manage what you can't measure, and without a metric it's difficult to set targets, monitor progress, track results and fix problems. Metrics help identify, analyze and report operating behavior.

Metrics are needed to measure all aspects of a company's performance. Some will be related to financial performance, some to operating performance. Metrics used at the corporate level - such as Earnings and Return on Assets (ROA) are often of little relevance at the level of a particular product or process. For example, the corporate assets may include the downtown headquarters skyscraper. Its value will probably be much more closely related to the state of the local market for office space than to the effectiveness of the product development process. Changes in Earnings may be more closely linked to fluctuations of the dollar on the foreign exchange market than to changes in the level of customer satisfaction.

Metrics such as Earnings and ROA are a useful guide to corporate performance for corporate management and stockholders, but they are less useful to the Engineering organization looking for metrics to help improve engineering performance. At the level of the Engineering organization, metrics that specifically address engineering performance are needed.

There are many aspects of engineering performance that can be measured. Metrics will be needed to address the various levels of engineering activity. For example metrics will be needed at the level of the engineering organization. They will also be needed for individual projects. Engineering budget is an example of a metric that can be applied at the level of the organization. The budget (in dollars or as a percentage) may be compared with that of engineering organizations in other companies. This metric doesn't really say much about performance since it only measures the input and not the output. However, there are many metrics that do address performance. Each organization has to decide which metrics are the most relevant for its activities and to define how they will be used. As the following list shows, for each aspect of engineering there are usually several metrics that can be used.
Different companies measure different aspects of their behavior. They also use different numbers of metrics. Some only use 1 or 2 metrics, others use 10 to 15. If few metrics are used they generally include headcount and budget, although, as mentioned above, these metrics measure only input and not output. Some companies are associated with particular metrics. For example, Six Sigma is associated with Motorola. Break-Even Time (BET) is associated with Hewlett Packard. BET is the elapsed time between the moment of initial spend on the development of a product, and the moment when net operating profit (sales less cost of sales) equals total cost of design and development.

Statistical Process Control (SPC) has been used widely and successfully in the manufacturing environment to reduce product and process variation. As statistical analysis methods are just as effective in an office environment as they are in the plant, it can be expected that some form of SPC will be used increasingly in the engineering environment. One of the key metrics that could be addressed with SPC is the difference between the planned and the actual product development cycle time. Focusing on this metric might help to reduce the number of projects that do not finish on time.

It's important that the metric-setting process be understood by the whole Engineering organization so as to obtain the necessary commitment to change. Many people in the Engineering organization should be involved in deciding which metrics should be used and how targets will be set. Their involvement will increase their awareness and understanding of both the current situation and the targets. It will make them more responsible and motivate them to achieve the targets they have been involved in setting.

People from other functions that are customers of the Engineering organization should be included in the metric-setting activity. This will provide a useful reality check in case the organization either underestimates or overestimates its expected performance.

Many individuals and teams can participate in the process of defining metrics and setting targets, but Engineering management has to take responsibility for the set of metrics and targets that is finally chosen. The set should be cohesive, realistic and acceptable to the whole organization. The metrics chosen should be easy to measure. Their introduction should not result in the need for an 'Engineering Metrics' overhead activity.

Having decided which aspects of its performance to measure, the Engineering organization then has to measure current performance levels and decide on future targets. The targets may be derived either internally or externally. Internal targets are usually extrapolations of past performance. They are of the form - we will do better by 20% next year. Such a target is of course useful, but it runs the danger of being unrelated to the performance of competitors. If the organization is 300% worse than its competitors, then a 20% change is not going to make much difference.

The other alternative - of deriving targets from external sources - avoids this problem. It involves benchmarking the performance of other companies and using their performance to set the target. As a result, the organization that finds it is 300% worse than its competitors might set a target of a 350% improvement.

One of the aims of benchmarking is to measure the organization's processes, systems and practices against those of other companies. Probably these companies will be renowned industry leaders and the toughest competitors. Benchmarking doesn't only involve identifying performance levels. It also involves understanding how other companies achieve their performance levels, and then adopting and adapting their strategies. Benchmarking establishes credible targets by basing them on external reality. Internally, benchmarking can drive consensus. If performance targets are set at the level of the best competitors, people will be less inclined to squabble about who should do what, and more likely to start working to achieve the targets. Seeing that better performance is possible is often a stimulus for a change in behavior.

The number of metrics used by the organization should be kept as small as possible - perhaps between 8 and 12 will be appropriate. This may mean eliminating some metrics that look interesting. The metrics should be prioritized so that it can be seen which are the most useful. The metric that has priority 15, even if it does look interesting, is going to be much less useful than those with priority 1 and 2. Maintaining it in the set of metrics will not only result in wasting time and money on measuring and targeting it, but more importantly will take the focus off the most important metrics.

Introducing a new set of metrics is a serious long-term issue. It is not something that should be done each year. The whole point of having the metrics is to be able to measure performance and to drive measurable performance improvement. This can't be done if the set of metrics, or the way they are measured, is changed each year.

It's a good idea to arrange for the introduction of metrics to coincide with the development and implementation of the Engineering strategy. The two are closely related - the targets for the metrics often being the targets set for the overall activities of the organization. Introducing metrics in parallel with plans developed from the Engineering strategy should help to ensure that they really do correspond to expected performance. It should also avoid the situation of applying them half-way through projects. Introducing metrics half-way through a project is not appreciated by project team members and can be expected to cause problems.

PRODUCT DEVELOPMENT WORLD | HOME | TOP OF PAGE

Copyright 1998 by John Stark