By  Andrew Smailes

This year is my 22nd year in data. I have been working in various industries, but the central theme in my work is using data to answer business questions and to improve businesses.

The use of data has expanded to support a wide variety of use cases in those years. The use cases can be as some form of decision support feedback loop ranging from interacting with an app responds in real-time, to a corporation or government creating scenarios looking ahead to the next five years.

The imaginative and the wonderful

I guess I am lucky to have experienced the mind-boggling changes of the last 22 years. It is amazing to see how disciplines separated at the start of this century have converged. For example, web technologies are now integrating the practices developed within data warehousing.

However, the progress in user-facing business intelligence and analytical tools has been very disappointing. We still look at graphs and tables and drill up or down, in spite of the fact that the way people interact with information has completely changed. We have gone from greenscreen monitors to smartphones. Apps bring us functionality and nuggets of information. They beep, flash and vibrate, so they will not be ignored. We are all participants, whether we want to be or not, in a huge, rapidly changing infrastructure that is collecting, transporting, aggregating and serving us information, continuously.

Personally, the introduction of the iPad was the milestone.

This was the first information interaction device that is intimate and where the technology has become totally transparent to the user. Though the smartphone is the token workhorse, the iPad was the first information device that people actually picked up without feeling intimidated by technology. I love the human-centric approach to information production and consumption it represents to me.

The availability of data-gathering technology and the amounts of detailed data it can generate comes with responsibilities for proper data management. And it is in the responsibilities where I experience that we are lagging.

The other side of the coin

Not so long ago, someone remarked to me, “You really dislike technology. Don’t you?”  No, I do not.

I can appreciate technology for what it can do with data or what kind of data it can produce. I have come to understand that you cannot sustain yourself in a technology-driven industry when you dislike the main force that propels it.

What has become problematic for me over the years is the misguided faith that technology will compensate for the lack of capabilities and knowledge of the humans that are working with data.

The humanities of working with data

Most people would rather look for silver-bullet solutions than face the music. But even taking this into account, the track record of success in data initiatives in businesses is not good. Over my years in the industry, I have seen huge sums of money put in projects, the project fails and then even more money is put into another project to try again.

For 22 years, I have wracked my brain to deconstruct the many failures in data initiatives. The simple answer is that we have been unable to sustain the necessary collaboration between people with different skillsets to work with data effectively. For a short period of time, we may manage to make some progress.

But afterwards, the energy flows away and the supporting solutions deteriorate through neglect or badly managed expansion.

Technology delivered us great new capabilities, but those capabilities are worthless if you don’t know how to use them to your advantage. The simple matter is that we focus on the data and we tend to overlook the context for which data is being used.

It is astounding to conclude, time and time again, that everyone involved has a different view of what we try to achieve with data and who and what is needed to get there. We fail to debate this in-depth before we start. Afterwards, we are disappointed that what we each intended to achieve did not materialise. More importantly, we fail to collectively learn from it.

This is not a comfortable place to stay. Luckily, there are perceptive people equally taken by the malaise and seek to apply their different skillsets on this fundamental problem.

Please reach out me to LinkedIn if you are one of them.