I had a difficult decision this month. There are two terms being thrown around conference rooms with the same mind-boggling frequency as “hit the ground running” and “make sure we are on the same page.” The funny thing about these new buzzwords is that they are really technology terms but they have been embraced by business folks who believe they are “thinking outside the box.” So I had a tough decision to make. Should I write about HTML 5—as in “our new website needs to be HTML 5,” or should I discuss “Big Data,” as in, “Big Data is going to fundamentally change the way we do business?” Big data won this month but with HTML 5 displacing Web 2.0 as a “value-added paradigm” it was a tough decision.
What is Big Data in real life? Generally accepted definitions are something like this: data sets that are so large and complex they cannot easily be processed or analyzed by traditional database tools. That doesn’t really tell us a lot. There are new tools that allow us to manage large amounts of data through distributed non-traditional methods. And that is very interesting to a technologist.
I have spent a lot of time working with web servers. There is a lot of data generated for a given server—things like internal event logs, error logs, web logs, performance logs, load balancer logs, and access logs. We are also able to track a lot of data about the consumers of the services delivered by these servers. Typically this is accomplished by using java script which allows us to collect client side data.
As mentioned earlier there is no true useful unstructured data, but there is a wealth of unclassified data that we may call unstructured. This is the kind of data that could be used to build truly intelligent systems. Artificial intelligence is a term thrown around much too loosely. IBM has gotten a lot of mileage around the Garry Kasparov – Deep Blue chess matches and the Jeopardy winning Watson machine. Both Deep Blue and Watson are specially designed systems with specially configured data to do one thing very well. As such they appear intelligent-like, but they are not examples of artificial intelligence.