Immuta: The key challenges of COVID-19-driven cloud migration

The post-pandemic world is going to be a cloud-driven one. After dealing businesses a lifeline amid the COVID-19 outbreak, cloud computing is the hero that empowered organisations to maintain business as usual, despite the unusual circumstances.

In turn, cloud initiatives have quickly climbed the business agenda. Plans for cloud migration went from “let’s definitely look into that this year” to “how fast can we do it?”

However, we know not to rush our work, and cloud migration is no exception to the rule. As many data teams know, moving to the cloud is challenging and requires careful planning and execution. 

The company Immuta is an expert in safely executing cloud migrations and helps global organisations harness the value of their data in a way that preserves data privacy and security.

So, to answer our questions about cloud migration, we spoke with Immuta’s Sumit Sarkar, Senior Director of Product Marketing, and Stephen Bailey, Applied Data Science Director, in a podcast investigating data governance and sensitive data protection considerations for data architects and engineers planning a cloud migration. 

More businesses than ever are looking to unlock potential cloud-based use cases, primarily those responding to rapidly evolving technology and economic conditions. For instance, Sumit shares an example of HR departments utilizing cloud resources to measure the workforce’s performance amid the sudden shift to remote working. And data science and analyst teams want the ability to acquire deeper insights to help answer the unprecedented business questions our current environment has raised.

COVID-19 has made cloud migration an even more compelling option for organizations, but it still presents real challenges for data teams. During our discussion, Sumit and Stephen outline the main obstacles data engineers and architects face on their cloud migration journeys, and how they can plan to mitigate them to execute a seamless transition.

Decentralisation of control

One of the most prominent yet unanticipated challenges for data teams when migrating to the cloud is relinquishing control. The inability to manage assets on-premise leaves a palpable void for data engineers and architects and can be a source of unease. 

One of the main concerns, Stephen explains, is that the cloud enables anybody within an organisation to spin up resources or a new analytics tool before connecting the data, and quickly putting the data into production.

For centralised IT organisations, this is a real paradigm shift. Rather than exercising full control over data resources, data teams end up taking on more of a leadership role, in which they outline the necessary data privacy standards and regulations, as well as disseminate the requirements and best practices for maintaining compliance. The end goal is that everyone within an organisation can take advantage of the cloud’s fluidity and ease of use without compromising compliance standards or having to be individually monitored. However, this also means data teams have the important responsibility of communicating these best practices adequately to stifle the possibility of employee compliance violations.

The decentralisation is also very technically demanding. Stephen points out that there are many, complex and diverse services available for data teams. And the plethora of services makes it difficult for data teams to implement policies that apply to all services and can be communicated consistently across the organisation.

But this paradigm shift doesn’t have to be a burden. An automated data governance solution, like Immuta, allows data architects and engineers to implement fine-grained access controls and policies across systems to easily reduce risk and streamline data protection responsibilities. This not only eliminates the need for manual, risk-prone human processes, but it is also a scalable and dynamic solution for sensitive data security. 

Grappling with regulations

Consumers are increasingly concerned about the collection and use of their personal data. Regulations like GDPR and CCPA, while reassuring, mean that the pressure is on for businesses — and most specifically, data teams — to get it right. If they don’t, they can face significant financial and reputational damage.

Amid this mounting pressure, the severity of the repercussions leaves little room for error and can cause concern among data engineers about their personal liability for data protection violations. In fact, in our podcast, Sumit shares that a data professional he’d spoken with had actually been losing sleep over whether or not he’d filtered out some records the day before, and expressed unrest over his personal liability under the imposed rules and laws. Moreover, building transparency is no easy feat for data teams, who are responsible for providing detailed audits and reports for compliance and legal teams that show who accessed the data, why and when.

For data teams, complying with these regulations, sufficiently protecting sensitive data and proving compliance can lead to an increased workload and added stress. However, with the right strategy and tools in place, data architects and engineers can feel confident that their data security measures are compliant and adequate in staving off data privacy attacks. 

TDWI’s checklist, Seven Steps for Migrating Sensitive Data to the Cloud: A Guide for Data Teams, sponsored by Immuta, outlines the challenges associated with cloud migration, and provides guidance on how data teams can proactively develop plans to mitigate them. Consider it a useful blueprint for avoiding common obstacles in the cloud migration process and executing a smooth and secure transition to the cloud. 

Cloud migration’s benefits far outweigh the potential challenges of executing the process securely. Again, many of these challenges can be avoided with the right foresight and plan in place. To give data architects and engineers some (much deserved) peace of mind, we highly recommend the TDWI/Immuta checklist.