73% of companies are investing in DataOps this year, according to research firm Pulse Q&A. The survey, commissioned by software company Nexla, documented how 266 data professionals interact with and process data.
DataOps, or data operations, is the latest innovative methodology used by analytic and data teams. It encompasses data management processes that streamline the speed and accuracy of analytics.
Tamr Founder Andy Palmer coined the term in a blog post in 2015. He defined DataOps as a “data management method that emphasises communication, collaboration, integration, automation and measurement of cooperation between data engineers, data scientists and other data professionals.”
DataOps recognises the interconnections in data engineering, data integration, data quality and data security. According to Palmer, it “aims to help an organisation rapidly deliver data that accelerates analytics and enables previously impossible analytics.”
Nevertheless, half of the survey respondents said that they lacked a sufficient number of backend data engineers on their data teams. The average company possessed just a single data engineer for every 5 business users.
85% of data professionals said that their company was also working on machine learning and AI. This percentage is forecast to grow, with 83% of businesses planning to invest even more in the emerging technologies next year.
Investment in data is set to soar, too. A new report from consulting firm SNS Telecom & IT found that insurers may invest a massive $2.4 billion in big data.
Despite this, Pulse Q&A found that 85% of companies are ingesting data from third parties. The survey found that a massive 54% of businesses are ingesting data from more than 10 partners.
Many data teams are unable to capitalise on this mass amount of data. Troubleshooting costs the average company 180 hours a week and takes up approximately 18% of a data engineer's time.
As a result, overworked data teams are begging for automation. 56% of respondents said that data clean-up would benefit from automation in the next two years, and 47% felt it could streamline analytics and integration.
Data format consistency was the biggest issue for data professionals. 39% of respondents cited data consistency as a key challenge, which could be attributed to the 60% of companies that ingest data in three or more formats.
“It's clear that backend data teams are strapped for resources, which is why 73% have plans to invest in DataOps,” Saket Saurabh, Nexla Co-founder and CEO, commented. He added “to really drive value from machine learning, AI, and advanced analytics, data teams need to stop troubleshooting and start automating.”
