—
The Microsoft product leader believes AI infrastructure must become predictive, intelligent and capable of responding before customers experience a failure.
Sudeept Srivastava is a director and principal product leader in cloud and AI infrastructure at Microsoft R&D in Bengaluru. Across more than two decades, he has worked in engineering, enterprise software and product leadership at Microsoft, VMware, Accenture, Infosys, Infogain and Capgemini. His career has focused on cloud modernization, autonomous operations, predictive infrastructure health and large scale lifecycle management.
That experience has convinced Sudeept that the operating model created for conventional cloud applications is no longer sufficient for artificial intelligence. Traditional infrastructure teams could often rely on reactive monitoring, scheduled maintenance and manual intervention. Large AI workloads change the consequences of failure. A single hardware, network or scheduling problem can interrupt thousands of GPUs working on the same training job and erase valuable progress completed since the last checkpoint.
“Reliability should become an intelligent, AI assisted capability embedded directly into the infrastructure,” Sudeept says. The objective is not simply to restore service faster. It is to predict risk, coordinate workloads around potential problems and act before customers experience an outage.
Sudeept begins with an important calculation. If every GPU had an optimistic average operating life of five years, a fleet of 16,000 GPUs could still experience a component failure approximately every 2.7 hours. At that scale, failure is not an unusual event. It is a permanent operating condition.
His framework replaces the scheduler’s basic question, “Is this GPU available?” with a more useful one: How likely is this GPU to complete the next several hours of work without causing an interruption?
Each GPU, node and pod would receive a continuously updated readiness score based on hardware health, network stability, topology, failure history and the potential impact of a problem. The scoring would also change according to the workload. A GPU showing early signs of memory errors might remain suitable for an inference task that can restart easily. The same GPU could be too risky for a large training job where one interruption might stop the entire operation.
This approach gives the healthiest and best connected infrastructure to long training workloads. Flexible tasks, including batch inference and evaluation, can use capacity that might otherwise remain idle. The platform therefore matches each resource with work that reflects its condition and risk.
Sudeept also argues that available GPUs do not always equal usable capacity. A large training job needs accelerators in the correct topology, connected through compatible network domains. A fleet may appear to have enough free capacity while still being unable to assemble the right group of resources. His model treats topology as an important resource and uses smarter placement to reduce this fragmentation.
His thinking builds on earlier cloud platform work. According to his professional record, Sudeept has led product strategy for an Azure VMware Solution platform covering more than 14,000 hosts and 25,000 lifecycle operations. Predictive maintenance, self healing and automated workflows helped prevent approximately 300 virtual machine disruptions each month, manage more than 3,700 operations through over 50 workflows and reduce operational effort by about 40 hours a week. Across the workflows Sudeept implemented for Azure VMware Solution, customer adoption reached 97 percent.
Sudeept is careful not to present those results as guaranteed outcomes for GPU infrastructure. Some elements of the newer framework have been introduced, while others remain in development. He expects the complete financial return to emerge over three to four years.
Using an estimated cost of $2 to $4 per GPU hour, he calculates that a 16,000 GPU cluster could represent between $280 million and $560 million in annual capacity expense. A small improvement in useful computing time could therefore generate millions of dollars in yearly value.
For Sudeept Srivastava, the larger opportunity is to help companies complete more valuable work with the infrastructure they already own. “At scale, you do not win by eliminating problems,” Sudeept says. “You win by seeing them early, planning around them and making them boring.”
—
