Friday, 31 October 2014

Give big data a chance to evolve

Data science is going to involve trial and error, and the costs of failure have to be contained

Listening to the presentations at the Information Age Data Leadership conference yesterday, it was noticeable that the term ‘big data’ didn’t come up with the frequency some of us expected. It was even treated a little disdain by a couple of speakers, portrayed as a term with more hype than substance behind it.

It reflected some of the lessons that emerged from the conference: that data has to be properly prepared to obtain clear insights; that it’s easier to prepare data held inside your enterprise; and that a lot of big data, despite all of its promises, resides outside the enterprise. The hard part in harnessing big data is not just to get at all that juicy information on the outside, but to get into a shape from which you can produce something worthwhile.

It was notable that in the stand out case study, on how Network Rail is learning a stream of valuable lessons from its data, which came predominantly from inside the organisation, making it much easier to manage. And a strong impression to emerge from the day was that you can get the best results in the short term by limiting your ambition, looking at what you have and can reasonably use rather than making grand plans to tap into streams from the outside world.

I’m not one to write off the potential of big data; it’s the continuation of a trend of bringing together and analysing information that is already producing plenty of real value for business. But it is probably being talked up by its evangelists to the point where it will disappoint a lot of expectations in the short term.

Harnessing all that unwieldy data from outside the enterprise is going to be a massive task, made more difficult by the unstructured nature of a lot of the information, and it will take a long time for best practice to develop. The emergence of data science will probably provide answers over time, but the discipline is in its early days and there aren’t yet many data scientists around. It will probably be well into the next decade before it matures, and some people are going to waste a lot of time and money in unproductive big data projects before then.

Which is why expectations should be kept in check, projects run on a small scale and not used for business-critical decisions until the techniques have been proven. Trial and error is inevitable in opening up any field of science, but data science is going to happen in the business world, not the controlled conditions of a laboratory, and it’s important that the errors are not too costly.

Business will benefit most from big data by allowing it evolve, not letting it loose in a big bang.

Mark Say is a UK based writer who covers the role of information management and technology in business. See