May 16th, 2013. This is less than two years ago. At that time, a gentleman by the name of Bart Gellman, who was with the Washington Post, was contacted by someone who was using the codename Verax. So who was Verax? Well, Verax claimed that he was working with the National Security Agency, the NSA of the United States, and that he had acquired a huge amount of documents which he was prepared to leak showing what the United States government was doing in terms of its surveillance programs. Well, of course now we all know how this turned out, but at the time it was a mystery if this person was legitimate or not legitimate.
But it June of 2013, that person was identified as being Edward Snowden, and over the coming weeks and months we learned an enormous amount about the program referred to as PRISM. And in the PRISM program, the United States government and the NSA in particular was using surveillance techniques to look at literally millions upon millions of people, not only here in the United States but indeed all around the world. In particular, they were looking at emails and they were looking at telephone calls. But they weren’t actually looking at the specifics or the contents of those emails or the contents of those telephone calls. Instead, they were looking at what we refer to as metadata.
And this is an example of metadata. This is actually the metadata of an email, which we’ll be coming back to a little bit later in the program today, but at this stage it’s important to realize that what they were looking at was the identifying characteristics of those emails and those telephone calls, not the contents of the emails or the phone calls themselves. But of course, they were doing all of this surveillance in an attempt to supposedly mitigate the possibility of further terrorist attacks on America, which many people have debated and certainly that’s not the topic of today’s session.
But what is the topic is that we now know that the National Security Agency or the NSA had many programs, not just PRISM. PRISM was certainly one, one of the larger ones, but they also had a program called XKeyscore. They had one called MUSCULAR, one called STATEROOM, Tempora, Shotgiant, and Five Eyes among many others, but these were some of the major ones that we learned through the leaks coming from Edward Snowden.
For most people or for many people, this was their first introduction to the concept of big data. All of a sudden, this word big data was being used in the media mainstream and people were hearing about this for the very first time, and indeed, there is more and more data being generated all the time. So the analysis of this data is a natural evolution. The mobile industry, our mobile phones are generating an enormous amount of data and the applications on those mobile phones are generating an enormous amount of data. There are satellites that are orbiting this planet and others, taking photographs and measurements 24 hours a day, seven days a week, generating an enormous amount of data. There are RFID tags on many of the products we purchase and there are centers on many of the buildings we come to for work, hotels and a variety of other reasons.
Indeed, this is what people refer to when they refer to the Internet of things. There are so many things that are now connected. We have smart meters on our homes. We have the new Nest smart thermostats; it’s all the rage and thermostats like it are taking measurements of people’s usage of their thermostats in their homes. We are increasingly connected with our cars using apps and other applications like OnStar, and indeed our televisions are connected as well and a host of other appliances. This is what people refer to when they refer to the Internet of things. All of these devices are increasingly connected with each other and they’re transmitting that data over the Internet, but all the while they are taking measurements and accumulating an enormous amount of data.
And they’re not the only ones that are accumulating such large quantities of data. The government, not just here but governments all around the world are accumulating enormous amounts of data. Here in the United States, you can think of the United States Census Bureau, but there are many, many others. And in fact, there’s an e-government movement which has been going on now for a couple of decades already which is the effort of governments to deliver more of their services over the Internet, and as a result all of that digital activity those digital inquiries and transactions are all being monitored and recorded, an enormous amount of data.
And whether you like him or not, Barack Obama actually does deserve a little bit of credit because he after George W. Bush dramatically opened the data of government to what’s referred to as the open data movement to share as much data as possible in the public domain, and today in the United States, and of course you can access this from anywhere around the world, we have Data.gov which is a website that shares a total of, and it’s growing all the time but, currently, over 460,000 data sets from 172 different government agencies. And that’s just here in the United States. This is public data. This is data that anybody can access, and indeed, there are entrepreneurs all around the world and certainly here in America who are digging into these data sets. This is big data that’s available for public consumption. And how many people are actually analyzing that data and looking for the insights that are available inside that data? Those data sets are available and they’re available for anyone to look into.
To give you an idea, the stock market generates about one terabyte of data every day. That sounds like a lot of data, of course, but in comparison Facebook generates 30 terabytes of data each day. That’s people like you and me uploading photos and videos and posts and adding things to our profile, 30 times as much data as the stock market. Well, that actually, believe it or not, sounds small compared to the Airbus A380, the new double-decker aircraft which is now the largest commercial aircraft for passenger flight in the world. The A380—listen to this—generates 30 terabytes of data per engine every 30 minutes, every half hour. And they have four engines, so that means that it’s 120 terabytes per 30 minutes, 240 terabytes per hour. [00:07:46] That equals from one flight from San Francisco to New York City is 1440 terabytes of data, which is over one petabyte of data being generated on a single flight. Indeed, the amount of data is scaling by orders of magnitude.
And of course, one of our most familiar uses of big data whether we refer to it this way or not is Google itself. Every time we do a search on Google, the algorithm is not actually going out into the public Internet to deliver answers; instead, it’s going into a cached version of the Internet. Google sends out programs which index the entire Internet, everybody’s websites including yours and mine, and creates an equivalent of the entire Internet on its own servers. So how big is the Internet today? Well, it’s hard to get exact figures but the latest figure that we can get our hands on is it’s roughly one million terabytes, which equals one exabyte of data—that’s the size of the Google cache—or at least the most recent version that we can get our hands on. The amount of data is scaling by orders of magnitude.
One of the first applications of big data was our credit scores. The credit rating agencies have been using enormous quantities of data being collected on each and every one of us to generate credit scores for already a couple of decades and in fact longer than that, but in terms of the sophistication of their algorithms it’s really been in the last two decades that it’s become incredibly sophisticated and the reality is that your credit score today is a remarkably accurate assessment of your character. It’s a remarkably accurate assessment of your character, and in fact that same data usage that the credit rating agencies have been tapping into for decades is now being used by the credit card companies as well to limit credit card fraud. And these are things that perhaps many of us on the call today have experienced, is that if a faulty purchase goes through, they know. They know before we know and they know because they have looked at the data, they have seen the patterns and they can automatically without any human being’s involvement, they can identify transactions that are suspicious and likely to be fraud.
The amount of data is scaling. It’s going up along an exponential curve and it’s very important that we talk about that a little bit because back in the day, 10, 20 years ago, when we were analyzing data, the data sets were manageable enough that we could essentially bring the data to a CPU and use something called relational databases to find a result with whatever it is the analytics of the analysis that we were looking for. But today, the data sets are so big that we have to break them up into pieces and bring them to individual CPUs and then get the results with something called parallel processing. Parallel processing is essentially the defining feature of big data. What is big data? It’s using a data set that’s so large that it requires parallel processing. Now, of course, that definition isn’t even so important, but it is important to mention because it’s heralding in an entirely new category of software which is required to process data sets of this size.
Many of you may be familiar with Hadoop and MapReduce. They tend to go hand in hand. Hadoop is an open source platform; MapReduce was first pioneered by Google. It essentially looks like this, and I’m not going to go into this in any detail, but they’re taking an enormous data set and breaking it up into pieces, analyzing each piece individually, and then creating essentially a table of contents at the end to pull out the insights from all the different CPUs. Hadoop and MapReduce are forming today the backbone of big data, and we’ll be speaking about some providers and some of the resources that are available at the end of this session. And I should mention as well that there will be an announcement from Simplilearn at the end of this presentation as well, a very important announcement actually about some of the resources that they have available, but for now just know that Hadoop and MapReduce are playing a critical role in the development of big data.
So data is scaling. The amount of data being accumulated is going up along an exponential curve but it’s not the only thing that’s following that curve. Many of you have heard of Moore’s law, and Moore’s law refers generally to data processing and how data processing, the capacity of a processor, is going up along an exponential curve. But even that is not the only piece that’s expanding along that exponential curve. Data processing is following it, but so is data bandwidth—it’s also following an exponential curve—and the third one is data storage. Data storage is also following an exponential curve.
So you have these three really pillars of technology—data processing, data bandwidth and data storage—that are all expanding along an exponential curve. Well, the inverse of that reality is that the cost for any particular amount of processing bandwidth or storage is going down dramatically. It’s getting cheaper and cheaper and cheaper. And as long as these three pillars of technology follow this same exponential curve, the cost eventually is going to approach zero. What if it were free? What if it were free? What would you do if it were free?
Think back for a moment to April 1st 2004. At that time, Microsoft’s email provider called Hotmail offered two megabytes of free storage for every account. Well, April 1st 2004 was also the day that Gmail launched for the first time, offering one gigabyte of storage for every user, 500 times as much as Hotmail. How is that possible? How can that possibly be a good decision? Because in fact, when you look back at that time, offering one gigabyte of storage for every email account was not profitable at all. It wasn’t profitable to do that. But Google knew that data storage was following an exponential curve, which means the cost of delivering one gigabyte of storage would go down every single year. They knew that it was going to become profitable in the future.
I grew up in Canada myself and one of our national heroes in Canada is a gentleman named Wayne Gretzky, who played hockey. Some of you may be familiar with the name Wayne Gretzky. And he always used to say that he doesn’t skate to where the puck is, he skates to where the puck is going. And similar people in athletics have made similar statements, of course, but the bottom line is that we have to see where these trends are heading. What would you do if data processing, bandwidth and storage were free?
Now, listen, we all know that there will always be a cost to these things. The bottom line is that the cost of these core elements will continue to go down and what will inevitably become more expensive or at least remain expensive is the software to manage the process and the implementation of that software, and finally, the execution of the insights generated by that analysis. But as people and as business executives, we all have to think about, what would we do as a company or as an organization or as a government if data processing, bandwidth and storage were free? We have to plan for tomorrow and not today. We see where these trends are pointing; we just have to plan accordingly. That’s what big data is all about.
Now, we’re going to be looking at some case histories and there are some fascinating case [00:16:49] histories. I can do that as essentially the highlight of today’s session. But before we get to that, I’d like to make two very important distinctions. The first distinction is between structured and unstructured data. So this is an example of very structured data. Think rows, columns, think an Excel spreadsheet, clean numbers, and it’s just a question of running equations, algorithms, analyses on this structured data. But the truth is that only 5% of data is structured; 95% is what they call unstructured data.
What is unstructured data? Well, let’s go back to our metadata, which we introduced earlier in today’s session. This is unstructured data. These are machine logs. These are machine logs that are being generated when I write an email. This is the metadata of an email that I’ve written.
So let’s just take a look at what information actually exists inside this metadata. Well, right up at the top, we can see who it was delivered to. We can see the IP address of where it was received. We could also see the IP address of where it was set. A little bit lower down, we can see the from email address, as well we can see the exact time and date when the email was sent, and finally we can see the subject of that email. This is all in this unstructured data.
So it turns out that there’s very valuable information in this unstructured data. The problem is that it’s harder to get at. It’s more complicated. It requires a higher level of sophistication on the part of your software to identify the pieces of information that are important, primarily because machine logs are notoriously inconsistent from one platform to another, so if you’re using one type of browser or one type of computer or another, or your ISP is different or perhaps the underlying architecture of that ISP is different, maybe it’s in a Linux server or a Microsoft server. Whatever those differences may be, the machine logs are always slightly different, so it’s difficult in many cases to get this unstructured data out, but the value is enormous.
Let’s take a look at an example of what could happen and a problem essentially that could take place that could be mitigated if we found a way to dive into this unstructured data. So here’s a case history. At the beginning, an order is placed, and this is the machine log of that order opening; someone goes online and begins an order. But here in the shopping cart, it turns out that an error took place at approximately four minutes after two in the afternoon. So an error has been created. Roughly two hours later, a phone call comes in to the customer service line and the person references that same order which had an error in the shopping cart, that this call which is to the customer service department remains on hold for four-and-a-half minutes and eventually the caller disconnects the call. And just a few minutes after that, the owner of that order writes a tweet and on that tweet says, “Just bought this device from Acme. Doesn’t work. Called, gave up on waiting for them. Retweet if you hate Acme.”
So here we had a story that unfolds in front of our eyes over the course of roughly two-and-a-half hours where a person tries to place an order, the technology fails, they get frustrated and call customer service, their call isn’t answered, and finally they send out a tweet which could potentially be very destructive in the public domain on Twitter chastising the company for their poor performance. Meanwhile, we had roughly two-and-a-half hours to solve this problem. These are the sorts of things that can be pulled out of unstructured data. If we have the systems in place, we could see that there is a critical situation unfolding in front of our eyes. There is something important happening. We need to intervene. Something needs to happen before the customer gets angry about the situation they find themselves in.
The second distinction I need to make is that of data sampling. The first was structured versus unstructured data and now data sampling because, again, looking back in time, if we wanted to understand a large data set, what did we do historically? We took a sample. We took a sample where we took a certain number of data points from that larger data sample and we studied those particular data pieces and then at the very end tried to figure out what are the trends that we’re seeing in this data, and then we can assign a trend line perhaps or some sort of an algorithm to extrapolate insights about the larger data set from the small sample that we study. And this does indeed work fairly well for the vast majority of data points. The problem comes in with all those outlier data points, and almost every data set has those outliers that don’t fit the primary trend. In that case, by sampling those outliers are almost ignored. They’re essentially ignored because they don’t fit the primary trend and as a result of that we can’t account for those by using a sampling technique. There just isn’t any capacity to do that. The difference with big data, the difference today, is that we can literally study the entire data set.
Ladies and gentlemen, this sounds like a simple distinction, I do understand that, but it’s critically important. It’s critically important because when you can study the entire data set, we can include the insights that come from those outlier data points. We can include all that ugly data that is off on the side and doesn’t fit the primary trend. We can see it all. We can look at all of the data.
Now, I need to give some examples of why this is important and probably the best example is Google Translate. Google was not the first company that created automated translation algorithms. Many companies did it before and their primary methodology was to essentially have an online thesaurus which would just switch out words and switch out short phrases with their equivalent in the other language and then deliver the final result. But as we all know, the quality was very poor. Sometimes it was virtually unreadable because those simple bottom-up algorithms were unable to incorporate all of the intricacies in the conceptual context of those words and phrases.
What Google Translate did by comparison is they included literally the entire Internet of translated material. Some of it was translated well; some of it was translated not well, poorly. The good examples were examples like Canada, my home country, which has two official languages – English and French, and so all of the material that’s provided by government is translated officially by a well-paid translator who specializes in doing exactly that, and of course there are many, many, many countries around the world that have those types of translation requirements as part of their countries and the multiple languages that they support, but there’s also plenty of companies that translate into different languages. And again, some of it is good quality, some of it is bad quality, but when you include the entire universe of translated material that’s been done by millions of people all around the world, you can now for the first time truly calculate the probability, the probability of the accurate translation given the contextual context of the words and phrases in the original piece. Those probabilities are incredibly important when you have a huge data set and you’ve included all of it, the good data and even the ugly data. That ugly data adds to those probabilities and it actually made the quality of translation much, much better for Google Translate.
The same was true for the algorithms that were being created to play chess. There were chess algorithms that would, you know, you’re essentially a human being that’s playing chess against the computer, and for years and years and years the humans could still win. The humans could still win because essentially the algorithm, the chess algorithm, would calculate all of the possible moves 15 steps into the future and get a weighted average of what’s the best possible solution, what’s the best possible move to make right now. But that still didn’t include all the unique situations that a human being might bring to a chess game.
What they did afterwards was they included literally tens of thousands of different chess games that had been played by human beings in the past. Some of them were winners, some of them were losers, some of them were played well, some of them were not played well, but when they included that enormous, enormous data set of actual human experiences and how they translated on the chessboard, they were finally able to develop an algorithm which could win, and in 1997 famously, Garry Kasparov was beat by a computer named Deep Blue. And now, today, computer programs with all of that big data, with the entire data set being included, the computers can now win every time.
We’ve all heard about the ugly duckling—I forgot about one slide—the whole idea of including all the data, the ugly duckling being rejected by its flock. We have to embrace the ugly ducklings. We have to embrace the ugly data. The value, the largest value that’s available in the data that you either have already or have the opportunity to collect or purchase, the most valuable data is the ugly unstructured data that’s sometimes hard to get at. That’s where the value is. That’s also the place where most people don’t go because it’s more difficult. The opportunity is in the ugly data. The opportunity is in the unstructured data. So include that in your plans.
So let’s take a look at some success stories, and I alluded to this earlier in the program. This is what I think is the more important part because, hopefully, each of you who are on this call can extrapolate different possibilities that you can incorporate into your own company from one or another or maybe even multiple of the stories we’ll go through here.
One of the best known stories, perhaps some of you have heard this before, I don’t know, but there was a gentleman in Minneapolis, Minnesota who walked into a Target store and was furious because his 17-year-old daughter was receiving coupons in the mail for prenatal products and diapers and things to do with babies, assuming that she is going to have a baby sometime soon. She was 17 years old. The father was furious. How could Target send coupons to a 17-year-old girl? Is Target encouraging young teenage girls to get product at such a young age? He was furious.
The manager of that Target store apologized profusely and apologized for it very much. In any case, the father ended up leaving and three days later the manager of that Target store actually called the customer again just to apologize again and say that he had taken care of it and removed his daughter’s name from that mailing list. But on that second phone call, the father said, “You know, I actually owe you an apology because, as a result of this situation, we’ve had some conversations in my house and there were things taking place that I was unaware of and my daughter is due in August. My daughter is due is August.” So Target knew that the man’s daughter was pregnant before her own father knew. How is that possible?
You see, there are a few times when a person’s shopping habits change dramatically during their life, and one of those times is when they have a baby. When people have a baby, all of a sudden their shopping patterns are completely in flux and they’re open to new suggestions, new places to buy what they need. And so of course when you have a baby, that’s public record, and so as soon as you actually have the baby, then you get an enormous amount of coupons and things in the mail to help you make those new decisions. But Target wanted to find those people before they had had their baby, and so they did correlation analysis using the people who they knew and had babies and then going back six, eight, nine months and seeing how their shopping patterns had changed.
And what were the winners of their analysis, the ones that had the highest correlation? Prenatal vitamins and scent-free soap. So it turns out that if you’re purchasing prenatal vitamins and scent-free soap, there’s a very high probability that you’re in your second trimester of pregnancy. This is what big data is all about. This exemplifies the opportunity in big data. Netflix knows what movies you’ll like before your own friends know what movies you’ll like. Pandora knows what music you’ll like before you know what music you’ll like.
This is all a function of what they call predictive analytics, and some even refer to it as predictive marketing. We are using algorithms, we’re using usage patterns, we’re using all the data we can extrapolate from—online behavior, transactions, product usage, social media—anything we can get our hands on, to predict who is a customer for our product or service. It allows us to take a population of prospects, people who we have not yet met, and segmenting them according to other behaviors, other information that we have about them into different categories and perhaps offering different services or different products to different groups.
In the end, this is going to make our world more intuitive. The offers that we receive as consumers and as people in our world will be more closely related to things we actually want and need. In some cases, many of us have already had the experience where we received an advertisement for something and said, “How in the world did they know that this was something I needed?” Those are the sorts of things that we can get to using big data, by looking at those algorithms.
But it’s not just companies that are selling products and services that can benefit from this type of analysis. There’s a story of New York City, it’s a large a city and they had a real problem with manhole covers exploding because of fires that were taking place underneath these manhole covers, and they were happening in such a random and unpredictable way that they felt like they were really just in a reactive setting waiting for something to go wrong. But imagine the scale of the problem. These can cause serious problems, cause serious injury, but across the city, it has 52,000 manholes. They have 23,000 miles of cable. At the time they did this analysis, they had 61,000 troubled tickets that they had kept from the past and they had 126 years of maintenance records.
And so they hired some data visualization specialists, data engineers, data scientists to dig into all of this information and try and develop an algorithm which would identify the key performance indicators for a potential fire. What are the biggest reasons—which manholes are most likely to catch fire or have an electrical problem as a result of the information we have? And of course it’s not surprising that it was old wiring which was the single most important measure that they could look at, even though they included so many other measurements as well. But this analysis reduced their fire and explosions incidents by over 80% because they could now do preventative medicine, preventative action. They could go to the manholes that were most likely to have a problem and fix those problems before they occurred. Saved the city an enormous amount of money to do this.
There was a set of students at MIT that was conducting research by looking at cell phone activity by intersection, by intersection in their city, which was Boston, and they were looking at all of the different cell phone activities and this was for entirely different [00:35:11], but what they soon realized was that there was a number of spikes in cell phone activity at times that they would not have expected. In other words, we can all imagine that cell phone activity on street corners would go up during rush hour, during lunchtime, during maybe an afternoon break, but there were other times during the day when they would see real spikes in cell phone activity that they didn’t expect. And so they reached out to the city of Boston and sold them this data which Boston then used to calibrate their street maintenance schedules and garbage removal schedules. The data had value to someone entirely other than the original user of the data.
A similar example, a cell phone company in India realized that the payment record of their users, they had literally hundreds of millions of customers, well, I don’t know if it was hundreds, it was over 100 million customers, and they realized that the payment history of those customers could serve as a good surrogate credit rating that banks could use when making loans to those same people. India at the time of this story did not have the same credit rating agencies that we have here in America or in Europe. So their cell phone company realized that their data, their payment record history with their customers could be used by, again, someone entirely different – banks, when making loans. So they sold their data to the banks.
A lot of police stations around this country have installed cameras similar to these, and these cameras are on 24 hours a day and they look for license plates and they take photographs of license plates whether the car is moving or parked in front of them. Wherever they see a car license plate, that camera takes a photograph and tags the photograph with the time and date and the location. And in the end, with so many police cars on the roads, they have hundreds of millions of photographs of cars by their license plates in different locations at different times, and they sell that data to insurance companies and even private investigative units that are doing research on particular people. Again, their data has value to people other than themselves.
And of course, there are weather companies that are constantly monitoring high pressure zones and low pressure zones in our areas—probably the most famous is the Doppler radar system—and they accumulate all of this data and feed it into weather algorithms and they then sell that data to television stations who give us our weather forecast in the morning and in the evening. Again, they’re accumulating the data, extrapolating the value, and then selling that value to someone else who can use it for their purposes.
The question is, can you sell your data? Can you sell your data? Imagine the data you have. Who else could benefit from that data aside from just you? Let’s see what you can do it for yourself.
Walmart is another excellent example. They did correlation analysis of all of their sales data and different weather patterns. They wanted to see if weather patterns were resulting in different purchase activity, and in particular they looked at hurricanes. The beauty or the interesting thing of hurricanes is that you have usually six or seven or eight days’ notice, and so you know one’s coming. And so of course a lot of purchase activity takes place leading up to a hurricane; people are anticipating the storm.
And when you think to yourself, “What sorts of things go up in sales when a hurricane is coming?” there are a few usual suspects that come to mind – people buying water, flashlights or gasoline, canned goods. But it turns out that there was one product which was increasing in sales 600% in the few days leading up to a hurricane and it wasn’t any of the things you see on your screen today. Believe it or not, it was strawberry pop tarts. Strawberry pop tarts went up in sales 600% increase in the days leading up to a hurricane.
And so the immediate thought is to ask yourself why. Why would that happen? Who cares? Who cares why it happened? This is one of the things we all have to embrace in big data, that the analyses of our thought processes, our behavioral activity, our purchase activity are not always logical. We don’t need to understand why we do these things. But as a store, as a retailer, Walmart needs to know so that when a hurricane’s coming they can stock up on strawberry pop tarts and put them right at those end caps by all of the aisles and have the inventory to make those purchases, because if they didn’t have the inventory they would just sell out and lose out on that revenue opportunity.
We can learn even our illogical behaviors by analyzing the data. So can your data improve your cash flow? Can your data improve your revenue by seeing what people want to buy and in what circumstances do they want to make those purchases?
Google uses keyword search activity to see where the flu is breaking out and they can [00:40:59] it right down to specific neighborhoods within specific cities. Of course, the CDC, which is the government organization, the Center for Disease Control here in the United States, they do the same thing but they get their information from doctors’ offices, and there’s generally a two-week delay between what’s happening on the ground and when the CDC realizes that it’s taking place. By contrast, Google knows almost before it’s started because as soon as people start feeling sick they go to Google and start searching for flu-related keyword phrases. This is actually 2012. Two thousand twelve was one of the worst flu epidemics here in the United States that we’ve had in recent memory and Google kept on top of it neighborhood by neighborhood during the entire process by analyzing their enormous, enormous [00:41:50] of keyword search data.
Police departments is another fascinating example. Police and departments actually originally in Santa Cruz, California realized that similar to earthquakes, many crimes have aftershocks. Let me say that again – crimes tend to have aftershocks. So once one crime takes place, quite often there are other similar crimes in the same area following the initial crime, and if you’re curious, the reason for that is that the perpetrators of the crime, once successful, they immediately go tell all their friends what they just did and create the competitive environment where they all want to outdo each other. And so in the same neighborhood, you end up having multiple crimes taking place one after another. But they used this data in some very sophisticated analyses. Santa Cruz originally sold this procedure to the Los Angeles Police Department, but now their police department’s using it all across the country to literally develop real-time crime hot maps where they can calculate the probability of crime taking place one neighborhood by another and deploy their limited police resources in the areas where they’re most likely to be needed.
Can your data improve the world? These are the questions you have to ask yourself. Can you sell your data to somebody else? Can you use your data to improve your own revenue or your own cash flow? Or can your data improve the world as a whole? Start thinking out of the box of what your data truly has the capacity to do. There’s enormous value in this data; we just have to find it.
It all boils down to algorithms. Algorithms, the first step is to accumulate the data, and many of you are probably already doing that but you could probably do more. Think about the mobile apps on your telephone. They almost all request the permission to get location data even though most of them are not location-specific because the people who created those apps realize that having location data adds enormous value to the other data they’re collecting about the usage of their app, and so they’re asking permission, and most of us don’t look at it and so we’d say yes. That location data, they’re accumulating more data by design to add to the value that that data has in the end.
So the first step is to accumulate the data. The second step is to analyze that data, the analytics, to go through it and structure it and understand it and get attribution, not just correlation but causation. That’s a very important distinction as well. It’s easy to see which things are correlated, but to find out which is causing the other, and in some cases a third item is causing the two of them so there’s no causation between the two of them, but it’s a very, very tricky thing, is that the talent required is data engineers, data scientists and statisticians to go through and see where those correlations are, where the causation is, and then develop algorithms which can predictively anticipate future outcomes based on new data coming into the system today. Those algorithms hold the key.
Monetizing big data is about getting to algorithms that work. If you can generate an algorithm that works, it’s worth money. Someone needs that data. Someone needs that predictive capacity. So think about algorithms. Think about predictive analytics. That’s the value. That’s the monetization opportunity of big data.
Already today, we have algorithms that can generate updates, journalistic updates, on sporting events as they’re taking place with nothing more than the game’s statistics as an input. These algorithms have been generated with the full English language or whatever language that they’re reporting in, and all the phrases that are specific to the game and the colloquialisms, all of that information and the grammatical…all of that stuff is included in the algorithms. So you can feed in just the data of what happened and it can generate an update which can go out on websites all around the world. That data can be sold. That algorithm presents value that can be sold.
We have algorithms that can write music similar to famous composers like Bach and Mozart and Tchaikovsky, so good in fact that experts on those particular composers cannot tell the difference between the music written actually by Mozart and the music written by an algorithm that’s attempting to mimic the patterns that they see in Mozart’s writing. These are very, very sophisticated algorithms that have already been sold, and they can then sell that music as background music for whatever, elevators or restaurants that there are no copyright payments required because it’s unique music, which means that they can sell music that’s been generated by an algorithm. The algorithm is the core to the monetization opportunity.
Even Google, surprise, surprise, they’re literally at the cutting edge of big data in so many ways and they’re using algorithms to enhance their human resources effectiveness. In particular, they’re looking at valuing top performers. They’re looking at productivity optimization, retention maximization, leadership characteristics, discovery and learning, and effective hiring practices. These are just some among many. I’ve done research on this. They have algorithms that take in data and attempt to provide predictive insights based on information that’s coming in today.
Go to where the opportunity is going. Go to where the trends are going, not where they are today. What would you do if processing bandwidth and storage were free? This is where the opportunity lies and these are some of the larger players that are providing impressive platforms in the big data space.
This is one of my favorite quotes, “The future is already here; it just isn’t evenly distributed yet.” The future’s already here. We see what people are doing with big data already. We just have to learn from them and apply it to our own organizations.
This is all about thinking bigger, and I’ll close with this concept because thinking bigger—and I’m not talking about thinking a little bit bigger, I’m talking about thinking way bigger, like 10 times bigger, of what is the true opportunity of the data you have, because there’s a really fascinating domino effect that takes place when you think bigger. There are three things in particular that tend to happen when you think truly visionary large things. The first is that your life’s just a lot more exciting. It’s more fun to chase big goals. Secondly, you end up inspiring everyone else around you including your own customers, including your own employees, including even your suppliers and even your competitors. It’s inspiring when someone’s chasing a huge goal.
But the third one is maybe the most important: When you think really big, you end up having almost no competition along the way, because no one else has the guts to go after the big goal. They’re too busy chasing realistic goals. But the big goals, the unrealistic goals, those are ones where you end up having almost no competition along the way and people get inspired by what it is you’re doing. Richard Branson is probably the poster child of thinking bigger. He started Virgin Galactic, and you can’t get much bigger than that. Any competition? Not much.
What about Elon Musk, you heard about him? He’s the guy who started Tesla. Any competition there? Not much. None actually. No one thought to produce a high-performance electric car. And then he started SpaceX. Any competition there? Nope. Just one, in Europe, wherever.
What about Steve Jobs? Guys, everything this guy did, he got there first. He was a visionary. He chased the big goals, and because of that he had no competition when he got there. That’s where we are with big data. We’re talking about leadership. Who’s going to embrace the opportunity and go after it and get that data and invest the money that’s required to analyze it and get the attribution, get the correlations, build the algorithms, and start building an entire new revenue stream. This is all about planning for tomorrow, it’s all about embracing the ugly data and it’s all about transforming your industry. That’s what big data is.
And I hope that you guys can all take something away from this that you can use when you’re developing your big data strategy going forward for your own company. I’m very happy to be here and contribute. Once again, my favorite quote, “The future is already here; it just isn’t evenly distributed yet.” You have the opportunity to be ahead of the curve.
Thanks so much for attending this session. Again, we’re going to hear now from Simplilearn, who has some amazing resources. I think it’s very important. They put this on and they have some important announcements to share, so I’ll pass it off to them at this time.