Cloudera: How real-time data streaming is driving innovation

Now more than ever, businesses are making use of their data. Today, data is arguably the most valuable asset that an organisation has, and companies are taking advantage of the insights it provides to make decisions, shape their overall direction, and much more. 

However, as valuable as data can be, the real reward is in real time. The old adage ‘time is of the essence’ is especially true when it comes to data. As many businesses have come to know, customer data even as young as a month old should be used with caution. Why? Because the customer base is continuously and quickly evolving, and anything that isn’t ‘right now’ might not be right at all.

Real time, on the other hand, allows businesses to respond to customers and events exactly as they happen. Like the name suggests, real-time data streaming describes the continuous flow of data that can be processed, analysed, and actioned when being generated from various sources – all in real time, of course. 

In a podcast with EM360, Simon Elliston Ball, Director Product Management, Streaming Analytics, and Cyber ML at Cloudera, walked us through the opportunities that real-time data streaming presents. First and foremost, he outlined that the definition of ‘real time’ is highly contextual. “Real time means a lot of things to different organisations. I’ve heard people say ‘we absolutely have to have real-time everything’...or, I’ve even heard people say ‘as long as it’s done within 15 minutes, that’s real time enough.’” However, Simon identifies that most often, real time refers to responses that are needed in under a second. 

Arguably one of real-time data streaming’s most important benefits is that it is empowering the edge. As Simon pointed out in the podcast, there are numerous situations in which ultra-low latency/super-fast networking to a large data or cloud data centre is lacking. However, businesses still need to make immediate decisions, despite their connections or environments not being optimised to do so.

This, in turn, gives edge computing an opportunity to take centre stage. There are two sides that Simon identifies: “The more interesting analytics tend to require more than just the information available to one small edge, so that’s where you move from the edge to the core data centres and use the more heavyweight analytics platforms to be able to do that kind of real-time analytics.

Of course, the other side is that once you’ve found insights from across, say, your entire fleet of trucks, you can also push those back down to the edge and start to use things like lightweight machine learning there, complemented by the more thoughtful, more informed machine learning that’s possible from large amounts of data collected in core inner real-time streaming.”

Industries across the board can tap into the opportunities that real-time data streaming enables. With Simon’s expertise, we explore some of the innovations that real-time data streaming is driving in various sectors.

Healthcare

“One of the coolest use cases we’ve seen of real-time stream processing is the monitoring and collection of device data from intensive care wards, and the immediate alerting of nurses which enable a much more efficient approach to healthcare,” Simon shared. Before, it was typical for nurses to have to walk around the ward, check the vitals of their patients, and then come back fifteen minutes later to do it again. However, real-time stream processing has changed the game. Now, healthcare professionals can make use of real-time streaming analytics platforms and good edge collection devices. With these in tow, healthcare staff are instead immediately alerted to anomalies, rather than subject to only discovering them on a quarter-hourly basis.

Increasingly, we are also seeing the use of wearable devices in healthcare. For disease, patient, and clinical management, gadgets such as watches and necklaces are among the most highly anticipated innovations in the healthcare space. These can deliver information to healthcare practitioners in real time, allowing for quicker diagnoses or improved symptom management.

Cybersecurity

As the cybersecurity mantra goes: hackers only need to get it right once, so businesses need to get it right all the time. With ‘time’ being imperative to a security professionals’ role, it doesn’t get much better than instantaneous insight. 

Cybersecurity has long been plagued by delays. In fact, the average time it took to identify a breach in 2019 was 206 days – and that only accounts for the breaches that are reported.

“The quicker you can find a breach, the quicker you can solve it.” Simon explained. “The time between those events is now measured in tens of seconds, so being able to get that data faster, react to it faster, and find the anomaly faster really does make your recovery a lot less, as well as, of course, reducing the risks of anything bad happening in the first place.”

With capabilities such as instantaneous alerts, businesses can take comfort in knowing that they have round-the-clock notifications exactly as they happen. In turn, cybersecurity professionals can respond quickly to causes for concern and minimise any damage that occurs.

Meeting customer expectations

The abundance and utilisation of data today has reshaped the business-customer relationship. 
Thanks to the actionable insights data provides, companies can enrich their relationship with their clientele by understanding their needs and catering towards them more quickly. Therefore, businesses can take pride in a ‘you asked, we listened’ persona, which customers have since come to expect.

However, in the wake of the COVID-19 pandemic, customer expectations have climbed even higher. Research by KPMG concluded that in 2020, customer expectations have experienced their greatest increase yet, averaging at +4% globally. So, customers want it all, and they want it now.

“Customers are getting much more demanding.” Simon explained. “They’re expecting immediate results, and that’s where the real-time streaming element comes in. You can’t wait for a batch process to tell you how your business needs to run, and to adjust to an individual customer’s behaviour anymore.”

Simon continued to describe how customer expectations are now being shaped by, for example, increased mobile use and new usage patterns in light of people becoming more mobile or home-focused. He also highlighted heightened telecommuting and internet centricity, which is especially the case amid the pandemic. Simon concluded that “this brought around a whole step change in the levels of expectations that people have for immediate and accurate results.”

The businesses that have successfully tapped into real-time data streaming will be able to become more receptive and more responsive to their customers. Furthermore, down the line, it enables companies to generate new products and revenue sources and, ultimately, help them to remain competitive.