The Lighthouse Blog

Explainer: Big Data

Explainer: Big Data
“Big data” is trendy in tech circles these days. And I’m not talking about the band, although they’re pretty popular too.

When it comes to Google searches, “big data” has been steadily rising in popularity since 2011. It’s another one of those industry buzzwords that people tend to throw around without fully understanding its meaning. I, too, am guilty of this.


Technically, big data is exactly what it sounds like – large data sets.

Let’s break that down into digestible pieces:

  1. A data set is a collection of data stored in a database.
  1. When you start relating, joining or connecting these data sets, then you set the stage for “big data”.
  1. The further you go, the bigger your data set becomes and, in theory, the more useful it becomes.

Sounds simple enough, right?

Now we’re going to get into the functional, slightly fuzzy aspects.

Big data sets in static form are nothing to get excited about. The real reason people go buck wild for big data is because of the opportunities it provides for analysis. In other words, it’s what you do with it that matters.

Computational analysis of data and statistics, also known as “analytics,” is a techie’s way of making otherwise intangible bits and bytes useful to decision makers. The key to big data isn’t the data itself, it’s how you query that data to extract meaningful trends and correlations that can help you anticipate future problems and identify opportunities for improvement.

So just how BIG does a database need to be to qualify as big data?

I couldn’t find the answer to this question, so I had to ask expert Christopher Barnatt:

“A big data set may be defined as a data set that cannot be handled via traditional computer hardware or software. In practice, that means a database that could not be stored on eight hard drives in a RAID array (which is the maximum that most motherboards can handle). So today, this means that a database beyond 64TB really has to be viewed as Big Data.”

Whoa. That’s a lot of data – more than the typical small or medium business has on hand. In fact, wielding big data is sounding more and more impossible for the Average Joe. Who has access to data sets that large, apart from big corporations or companies that focus on data harvesting?

But all is not lost.

Applying big data concepts to your IMS

Even though big data itself might be unattainable to most organizations, the idea behind it can still be applied in meaningful ways.

I’m going to use our web-based oil and gas solution, Oplii, to illustrate how this can actually work.

Oplii takes data sets related to oil and gas sites, assets (like pipeline segments and equipment,) inspections, maintenance and work orders and relates them to one another.

This is where the relevant value of big data emerges: Creating an opportunity to query an interrelated data set to detect patterns, trends and associations – giving you the foundation to delve into “predictive analytics.”

Here’s an example: If I were to query equipment information, I could tell you how many pressure vessels a company has, sorted by manufacturer. But unless this is part of an audit or inventory exercise, the resultant data is fairly useless.

Now, if I query equipment information in relation to its associated site and inspection data, I could tell you about inspection failure points, and their relationship to equipment location, age, manufacturer, or material, to show correlating trends.

When you have the ability to perform these types of queries and make more advanced correlations – resulting in really useful reports – by drawing on a multitude of conditions and factors, you can start making more informed decisions based on indications about what is going to happen in the future.

And that’s it: The key to big data is BIG DATA SETS and ADVANCED QUERIES. But even with a smaller data set, you can still perform ADVANCED QUERIES and, maybe even, delve into PREDICTIVE ANALYTICS.

Leave a Comment

Your email address will not be published. Required fields are marked *