Twitter turns to Google Cloud for infrastructure support

Twitter, one of the world's largest social media networks, appears to be growing so fast that it's had to turn to an outside company for infrastructure support. 

Companies turn to each other for additional infrastructure, and sometimes they even start out using only third-party infrastructure.

A few months ago it emerged that Apple was using Google Cloud for some its iCloud storage and functions. Now it appears Twitter has also turned to Google Cloud for similar support.

In a company blog post, Twitter CTO Parag Agrawal says the Hadoop compute system is the core of the social media website's data platform.

“Twitter runs multiple large Hadoop clusters that are among the biggest in the world,” says Agrawal. “In fact, our Hadoop file systems host more than 300 petabytes of data across tens of thousands of servers.”

Agrawal went on to explain that Twitter is now going to move what it calls its “cold data storage” and “flexible compute Hadoop clusters” to the Google Cloud Platform.

Agrawal says: “This will enable us to enhance the experience and productivity of our engineering teams working with our data platform.”

When complete, Twitter says the migration will enable faster capacity provisioning, increased flexibility, access to a broader ecosystem of tools and services, improvements to security and enhanced disaster recovery capabilities.

Architecturally, says Agrawal, Twitter will also be able to separate compute and storage for this class of Hadoop workloads, which has a number of long-term scaling and operational benefits.

In response from the other side, Brian Stevens, CTO of Google Cloud, says: “There is strong alignment with Twitter's engineering strategy to meet the demands of its platform and the services Google Cloud offers at a global scale.

“Google Cloud Platform's data solutions and trusted infrastructure will provide Twitter with the technical flexibility and consistency that its platform requires, and we look forward to an ongoing technical collaboration with their team.”