Data and Analytics on the Cloud – Cost Management on the Cloud

February 1, 2021

Over the recent years, we have seen the adoption of Cloud, even in Data and Analytics, accelerate. There is no dearth of reasons why everyone loves the Cloud, including, but not limited to, the greatly increased speed-to-market compared to what could be achieved on-premise, readily available security and disaster recovery tools, and the relative ease of portability.

Most organizations are recognizing that there is not necessarily an immediate cost savings on the Cloud, but this can change over time. There is also a significant difference in the cost of Cloud in the application space versus in data and analytics. In the case of Cloud cost for applications, the workload, demand on the system, growth projection and forecasts are fairly deterministic and linear, and therefore, our ability to understand and model the costs over time should be quite straight forward. On the other hand, when it comes to data and analytics, the workloads are highly variable and much more difficult to predict, which makes it much harder to model, estimate, and control costs on Cloud in the data and analytics realm.

In this video, Darren Edery, CEO, Zelusit Corporation, focuses primarily on the two biggest potential advantages of the Cloud for Data and Analytics – Cost Savings & Structure and how it relates to Flexibility and Elasticity.

The Old Versus New Paradigm

In the world of on-premise, you would essentially buy fixed capacity at a fixed cost, and consequently, the higher your utilization, the more value or perceived value you could extract from your investment. In contrast, when it comes to data and analytics on the Cloud, the capacity and costs are variable, analogous to a pay-as-you-go model. Each use has an incremental cost, so there is an incentive to minimize use.

Where on-prem, workloads/queries up to the maximum capacity of your server were essentially free or paid up and so any workload/query within that capacity was run regardless of its business value, on the Cloud, every query/workload has an incremental cost so the decision to run any optional workloads should ideally be linked to business value and positive ROI.

Consequently, in the on-prem realm, there was little incentive to write efficient code or do performance optimization, however, on the cloud, the cost variability between inefficient and highly efficient queries and processes can be 1000x or more, so the need and ability to optimize and write efficient code becomes dramatically more important.

Another difference is that while development costs were essentially decoupled from compute resources on-prem, on the Cloud both these costs need to be considered to assess the total cost of development efforts, making the calculation more complicated. Finally, while in the on-prem world, old workloads were rarely decommissioned, unused/orphaned Virtual Machines or workloads that persist on the Cloud can keep running your bills up.

Cost Management on the Cloud

While it is in the interest of Cloud platforms to drive consumption and spend, the risk of attrition or losing customers to a competitor acts as a strong incentive to make sure they keep costs commensurate with value. All three major Cloud platforms – Microsoft Azure, Google Cloud Platform (GCP), and Amazon Web Services (AWS) provide out-of-the-box tools for cost tracking, management, and forecasting. However, these generic tools often do not account for business specific use cases, and so they may not always be effective as a standalone cost management solution.

Zelusit has partnerships with all three Cloud platforms and based on our knowledge and experience with our customers, we have developed some tools to help organizations effectively control their Cloud costs. In addition to the capabilities available on the Cloud platforms, Zelusit provides tools for Automated Query Analysis, which can estimate, in advance, the cost of running a query and based on the cost relative to value, we can evaluate whether query optimization is needed.

Organizations can also define standards for service configuration, such as governing how big a query can be (in bytes) or setting restrictive maximum run times to protect against cost incidents. We also recommend linking resources and services to cost centers to make cost allocation easier for shared resources.

All platforms offer some capability to trade-off elasticity in exchange for better pricing, but these are complex decisions that need to account for the overall benefit of a lower cost vis-à-vis a bigger commitment.

For decision-making on all of these aspects, to respond to cost incident alerts, and to establish processes, best practices, and tools to proactively ensure efficient use of consumption-based resources on the Cloud, Zelusit recommends the establishment of a “Cloud Cost Governance Office”.

This video focuses at length on the benefits of the Cloud and the differences between the on-premise and Cloud paradigms, especially in terms of cost structure and flexibility. It also delves into strategies for effective cost management on the Cloud and charts out the role and responsibilities of a “Cloud Cost Governance Office”.