Chasing the Holy Grail of Decision Making – Predicting Decision Quality

At Fingertip our biggest dream is this: is it possible to leverage artificial intelligence to predict the result of our decision before it is made? Quite obviously, the answer to such an overarching question has to be a resounding ”no”. Decisions are made under a fog of uncertainty and their results are frustratingly often affected by conditions beyond our control.

The mismatch between inputs and out is large as the results of decisions are often on wildly different scales: we don’t measure whether person A is a good hire the same way we measure whether marketing campaign B increased our sales of product C by x percentage points.

We can, however, attempt to hit a more moderate goal of predicting how our decision-making process is. And for this task, Fingertip has collected pretty unique data, as we have already glimpsed during a previous instalment of the scientific research blog series.

Back then, we looked at how the number of people involved in a decision correlates with the mood around the decision. But ”mood” is only one of a number metrics our users can evaluate in Fingertip.

Fingertip is a software built around our core ideology of social decision-making in which evaluating both the decision-making process and the result is an integral part. Indeed, the software is actively built to facilitate this and prod users to evaluate their decisions on a wide array of dimensions. This provides a treasure trove of data on how good the users find individual decisions to be.

Actually, the wealth of available metrics is both a blessing and a curse – rarely all of them are filled, leaving a lot of blanks on individual variables. There is also the issue of which is the metric we should predict on, mood, trend or process quality? For modelling purposes, the range of decision quality metrics are minnowed down to one all-encompassing number by averaging all aspects that have been evaluated. This value ranges from 0-5 with larger numbers indicating better decisions.

Obviously, how a decision pans out is affected by a whole range of variables and not merely by the count of people involved. We can for example see how much discussion has been held around the decision and the sentiment of those conversations, how often the decision is viewed, how many decisions the accountable has been involved in and whether the decision is part of larger Plan object or not. The data that Fingertip stores is never the whole picture of the decision-making process, but it is complete enough for clear patterns to emerge.

Using a range of predictor variables, we can train a model to predict how good a decision might be based on its attributes. To evaluate the model fit, we split a subset of observations that is not used for model training so we can see how well the model performs by comparing predicted values with the observed.

As we can see from the plot above, the model is far from infallible, but it is clearly far better than random guessing. The mean absolute error is slightly above 0.5 which means that average prediction is about 0.5 points off the mark.

As our dataset grows, our predictions will become more and more accurate. We also continuously attempt to improve our model, with the final aim of getting one to run in our software so all our users can benefit from the insight given by the AI assistant.

Our dream is not to let the machines make your decisions for you. That, alas, belongs to science fiction. Our dream is to augment, not replace, our human intelligence with machine intelligence. As the covid-19 pandemic has brutally shown, we need all the help we can get to make the correct decisions.