CVSS doesn't measure up.

I was doing some basic research into software metrics the other day and I came across something that I was probably taught once but had long since forgotten.  It was to do with the way we measure things and is covered in the Wikipedia article on Level of Measurement.

Basically there are 4 different scales which are available to measure things:

• Nominal scale - Assigning data to named categories or levels.
• Ordinal scale - A Nominal scale but the levels have a defined order.
• Interval scale - An Ordinal scale but the difference between, or units, of each level are well defined.
• Ratio scale - An Interval scale but with a non-arbitrary zero-point.

Why these scales are interesting is that only certain type of math, and therefore certain conclusions can be drawn from what you measure, depending on what scale the measurements belong.  For instance we can order the finishing place of a horse race into 1st, 2nd, 3rd etc. (an Ordinal scale), but we can't meaningfully say what the average finishing place of a horse is as there is no magnitude associated with the difference between the levels.  If on the other hand the races were over the same distance, we could could measure the time the horse took to complete the race (a Ratio scale) and calculate it's average time.

Sometimes we have an Ordinal scale that looks like an Interval or Ratio scale, for instance when we assign a numeric value to the levels e.g. ask people how much they like something on a scale of 1 to 5.  But this is still an Ordinal scale, and although we can assume that the difference between each level is a constant amount, nothing actually makes that true.  Thus calculating the average amount that people like something e.g. 2.2, is often a meaningless number.

When reading about this I was reminded of the way vulnerabilities are categorised and how we would so dearly like to be able to assign numbers to them so we can do some math and reach some greater insight into the nature of the vulnerabilities we have to deal with.  The Common Vulnerability Scoring System (CVSS) suffers essentially from this problem; vulnerabilities are assigned attributes from certain (ordered) categories, and then a complicated formula is used to derive a number in a range from 1 to 10.  It is basically optimistic to think that a complicated formula can bridge the theoretical problem of doing math on values from an Ordinal scale.  I wouldn't necessarily go to the other extreme and say it makes CVSS totally without merit - just that it's not the metric you likely wish it was.