Soaring cloud costs Five ideas for effective data platform cost management

Sean Xu, Vice President -- Information Management, MGM Resorts International

Sean Xu, Vice President -- Information Management, MGM Resorts International

Even prior to the pandemic, many companies have been adopting modern, cloud-based data platforms to expedite data initiatives, improve data scientist productivity, and to reduce time to market. In the last 18 months, the pandemic has forced organizations to suddenly embrace a remote workforce; consequently, this has spurred an unprecedented wave of cloud application deployments.

While organizations surely enjoy the benefits ofgreaterdata platform capacity, better scalability, and improved agility, these same organizations are likely also shocked by the skyrocketing cloud costs. Coupled with the challenges of integrating rapidly evolving cloud data technologies such as storage, Database & Data Warehouse system, and real-time streaming, the higher costonly exacerbates the challenge and underscores the need to effectively optimize expenses to realize business benefits. Undoubtedly, every organization will have different demands and allotted budgets for data and cloudplatforms, but here are 5 tips that can get you started:

1. Identify and remove unused resources:

Any enterprise cloud data platform is likely to consist of a variety of applications and services, ranging from virtual machine to data warehousing to streaming. Especially after projects are completed, it is common to see organizations forget to remove/reprovision computing and/or storage resources. In other cases, there are proliferated versions of services like virtual machines and database systems being deployed with specific options and use cases. This hinders organizations from effectively managing and applying reserved instance discounts. In addition to identifying idle resources, another best practice for optimizing cost is to ensure standardization on types instances and consolidate workloads into fewer instances

2. Rationalize technology choices for end-to-end solutions

While it can be tempting to integrate cutting edge technologies, misuse or misunderstanding of cloud data technology can result in higher costs than desired. Many cloud data technologies already include multiple features or functions and generally come with different pricing models. For example, both Azure Data Factory (ADF) and Azure Glue can handle data migration, job scheduling, and ETL, but that does not mean it should be placed for the functions by default.

While it can be tempting to integrate cutting edge technologies, misuse ormisunderstanding of cloud data technology can result in higher costs than desired

Especially when there are many technology options and methods to implement, architects and developers must take into consideration the end-to-end solution and the user experience relative to the cost to ultimately rationalize the best choice for the organization.

3. Right Size computing services with auto-scaling:

Right sizing of resources/instances for a cloud data platform involves appropriately designing of a database’s framework and storage capacity and then configuring resources to be able to scale appropriately based on workload needs. Oversizing instances for data pipelines, data warehouse, and database systems without effective auto-scaling are commonly seen in many organizations’ data platforms. This directly results in higher cloud expenses.

Organizations should analyze computing services like servers, databases, memory, and storage to modify them to the most efficient size. Furthermore, to take advantage of readily available and cloud computing and storage resources, organizations can utilize cloud auto-scaling to elastically provision resources to meet the dynamic workloads.

Right Sizing not only directly reduces cloud expenses, but also serves as the basis for addressing peak performance of tech resources/instances. Combined with auto-scaling, these are fundamental in helping organizations optimize their cloud expenditures.

4. Leverage reserved instances discounts:

Reserved Instances (RI) can provide organizations with a larger discount compared to on-demand Instance pricing. These larger discounts are based on upfront payment and time commitment. For some resources/instances widely used in cloud data platforms, such as Azure’s Synapse/SQLDB or AWS’s Redshift/RDS, RI savings can be up to 75% compare to on-demand pricing.

To reap the benefits of these RI discounts, organizations would have to commit to the cloud for the long term and develop resource implementation strategies aligned with their goals and workloads. Azure’s SQLDB and AWS RDS are two platforms that come with many customizable options for deployment. If organizations do the due diligence of carefully the plan workloads and standardizing into a few DB tiers for applications of RI discount,they can maximize the benefits of RI discounts and optimize the resources of any non-standard resources/instances.

5. Adopt Scheduling:

It is understandable for organizations that deploy cloud data platforms to run workloads on a 24/7 basis, but this also means that the usage and billing also continues. Often, many replicated staging and non-production environments do not require the deployed resources and instances around the clock which means that the deployed resources can be turned offwhen they are not in use. This means the billing stops.

A best practice is to introduce automatic scheduling, which will allow organizations can configure schedules to customize when to start and stop a variety of resources/instances depending on workloads and work hours. In addition to the cost benefits, this will also significantly improve elasticity and reduce wasted resources.

Make no mistake, cloud data platforms can certainly improve business agility and enhance business data-driven capabilities. During the rapidly changing demands of the remote workforce and the emerging technologies available to organizations, it is a challenge to implement these systems while effectively controlling the cost. The good news is these ideas can help better manage/control the cost, save money, and improve efficiencyas organizations continues improving and optimizing data platforms.