Hadoop, utilities and that pesky bandwagon
A colleague of mine recently asked for help in a workshop with a major global utility customer. As part of his preamble, he described the utility as being in the “trough of disillusionment” phase with their Hadoop implementation. I wasn’t surprised.
Now let’s be clear: this isn’t a Hadoop-bashing blog. Teradata has great relationships with the big names in Hadoop; our ecosystem features Hadoop strongly; and I’m entirely convinced that there’s a place for the technology in the modern, data-driven utility. No, this is a blog about believing the hype . And where that can get you.
Many utilities today have Hadoop implementations at one level or another. Very few have achieved anything worthy of note with that implementation. French networks business Enedis are an exception , it has to be said. But most….well….there’s not really much to say. Why is that? Here are three reasons:
The bandwagon effect
A problem that amazes me every time I encounter it is that utility companies all over the world seem happy to “get some Hadoop” before they have any idea what they might use it for. For example, a utility company I know well will go to tender for “some Hadoop” in the next few months. Their favourite consultancy / systems integrator assures me that the client has no idea what for . They just know they need some. Because how will they been seen as innovative if they don’t have any? Part of this keenness might be due to my second point, below.
The engineering mind-setIn the main, utilities are asset management businesses, run by engineers. Engineers really like new technologies. They like stuff they can play around with. Especially if it’s free. (Which clearly, it isn’t.) And they tend to think that “hey, it can’t be so hard”. Especially since engineers probably already run the SCADA and associated OT systems, often entirely outside the traditional IT domain. Which leads on nicely to my third point.
Lack of skilled resourcesActually, being a Hadoop Mahout  isn’t so easy. It’s about managing and deriving value from huge amounts of data in an open-source distributed file system. And the state of the art is changing every day. Utilities don’t really have guys with those sorts of skills hanging on hooks waiting for stuff to do. And even if they hire a team, it can be difficult to keep hold of them, given the continued attractiveness of Hadoop skills on a CV. Like it or not, the utility industry is not the most exciting place for the young Big Data specialist.
In summary then: Hadoop is having a hard time in many utilities because to a great extent, they don’t really know what to do with it. And even if they did, they don’t have right the people to do it. Simple as that. And here’s the kicker: that’s not Hadoop’s fault. It’s your fault.
Hadoop really is a useful technology. But it’s not the answer to all your hopes and dreams. And it doesn’t make you innovative as soon as you spin up a cluster. Yes, you can store huge data sets very cheaply on it. You can even offload certain workloads and storage to it from other systems. Sometimes even from…gasp… Teradata. But it’s not a straight-up replacement for anything . No matter what the hype, or what the unscrupulous vendor says.
Instead, Hadoop is a key part of a wider data & analytics ecosystem. Something that by definition has other components too. Gartner like to call that ecosystem the Logical Data Warehouse. Teradata call it the Unified Data Architecture. It doesn’t matter what you call it. If it’s done right, it’s a seamlessly integrated combination of tools and technologies that meet your data & analytics needs. Needs such as overall cost, of course. But also security; accuracy; availability; usability; the opportunity to discover entirely new insights; and to serve the CEO at the same time as the Data Scientist (if you have them) or the Analyst if you don’t. And all points in between.
If you work for a utility struggling with Hadoop, or about to go to Tender for “some Hadoop”, perhaps you ought to help them think again. Perhaps you should consider how your utility might avoid that trough of disillusionment by taking a different approach. One that looks at the problems they’re trying to solve first and the technology second. If that sounds like it might have some merit….well, we can help. And we’d love to talk to you.
――――――――――――――――――――――――Footnotes  Yes, I know they have customers too. But let’s face it most still call them consumers and see them as a necessary evil.  I know. I’m one of them.  Elephant handler. It’s all elephant-related in the Hadoop world. No, seriously.