BI (Business Intelligence) is simply an umbrella term for a set of techniques that allow us to extract useful information from raw data. Ten years ago it was in its infancy, now it’s an integral part of many commercial data systems. However, despite the name, BI isn’t just about business data: it can be applied to any data set. For much of the time I work as a BI consultant/developer in the commercial world but I also work as an academic at Cambridge University. 

The principle of science is, surprisingly, pretty much as they teach it in schools – you start with an idea, collect a mass of data, extract the data and use that to prove or disprove your original idea. The area where science has lagged behind the commercial world is in adopting these excellent new commercial tools to help extract the information from the data. But the fact that these tools can equally well be applied to commercial and scientific data tells us that data is just data and that these tools can be applied universally. 

As it happens I work in the field of evolutionary biology in a team that has performed analysis which ultimately helped to re-write the history of how Charles Darwin developed the theory of Evolution. As far as we are concerned that is a fascinating outcome but we also think that the underlying point, that BI tools can beneficially be applied to any set of data, is far more important. Our work also taught us that it is worth collecting apparently inconsequential data because BI has the power to reveal hidden nuggets of information that a manual search would be very unlikely to unearth. 

So here is enough background to set the work itself in context and some discussion about the tools we chose and why we chose them.

Current history of Darwin’s discovery

Evolution is based on the premise that variation exists between individuals and that natural selection removes the less fit individuals, leaving the more fit to pass their genes on to the next generation. So in order to develop the theory of evolution it was vital that Darwin grasped the significance of variation, both within and between species. Few people were interested in the subject back in the early 1800s, let alone systematically studying it, so it has been commonly believed that Darwin’s interest in variation was first aroused by the examples he observed on the Galápagos Islands. For example, Wikipedia tells us that the islands “are famed for their vast number of endemic species and the studies by Charles Darwin during the voyage of the Beagle that contributed to the inception of Darwin's theory of evolution by natural selection.”

It’s a perfectly reasonable assumption, it just doesn’t happen to be true. Darwin studied at Cambridge as the protégé of Professor J.S. Henslow between 1827 and 1830. What we uncovered was the previously unknown information that Henslow was actively and systematically studying variation. Combine that with the fact that it was Henslow who got Darwin the job on the Beagle and you realise that far from stumbling upon variation in the Galápagos, Darwin had already been primed to observe and study variation by Henslow.