Introducing AI-Powered Rightsizing for AWS EC2 VMs

Summary

• AWS EC2 VMs are poorly utilized. It is possible to see CPU utilization under 10%. Many VMs are oversized, leading to unnecessary costs.  Doubling utilization, which is often possible, halves costs.
• Current rightsizing solutions like AWS Compute Optimizer often (1) require manual effort  (2) do not use the full set of golden metrics to optimize applications (e.g., using utilization metrics only and not considering latency and errors, and (3) do not perform safety checks to validate the change can be made.  
• Sedai’s AWS EC2 VM optimization finds the lowest cost VM type subject to performance and reliability requirements. 
• Early users of Sedai's optimization technology have found reductions in cloud costs without impacting application performance

Introduction

In environments where applications are not suitable for microservices architectures, rightsizing and in particular vertical scaling becomes a critical strategy to achieve cost-effective operations while meeting performance requirements. This approach involves choosing the right virtual machine type based on the CPU and memory resources required. Rightsizing is a known best practice for AWS EC2 VM cost optimization but is hard to implement in practice.

‍

The Rightsizing Problem 

Public Cloud VM Utilization is Low

While AWS utilization data has not been disclosed publicly, analysis of the most recently released Azure VM usage dataset (available on Microsoft’s Github here) shows that VM users had an an average utilization of just 8.2%, with 72% of users having an average utilization of below 20% (see distribution below). A common pattern uncovered in the dataset was the selection of a small number of powerful, but oversized VMs.

Source: “Using Virtual Machine Size Recommendation Algorithms to Reduce Cloud Cost”, March 2023

‍

Below is an example from a Sedai AWS EC2 customer environment of an underutilized instance with averaging just a few percent of CPU being used across a one month period:

‍

Causes of Overprovisioning for AWS EC2 VMs

Developer Bias to Overprovision

Effective vertical scaling is also important as developers often default to overprovisioning AWS EC2 VM resources, opting for a simpler and quicker setup rather than conducting extensive testing across multiple instance types. This approach, while expedient, typically results in selecting VM configurations that exceed the application's actual requirements, leading to increased costs. The reluctance to engage in detailed testing stems from the time and complexity involved in evaluating each instance type's performance under different workloads. Consequently, developers lean towards a 'better safe than sorry' strategy. Although reducing the risk of underperformance this inefficiently raises cloud costs.

Bursty Workloads

Many Virtual Machine workloads have bursty traffic patterns, especially for small databases, development environments, and low-traffic websites.  For example, in the case below of a dev/test workload CPU utilization stays around 5% but surges a few times to the 30-40% range. 

Given warm up periods may range from a few minutes to a few hours, horizontal scaling may not be viable.  Finding suitable burstable instance types may be the preferred approach.  In the absence of that, low average utilization will be achieved.

App Architecture Limits Horizontal Scaling

A high proportion of applications running on AWS EC2 run directly on virtual machines.  These applications have not been replatformed to a microservice architecture such as Kubernetes (including AWS EC2 Kubernetes Service (AKS)), or serverless frameworks (AWS EC2 Functions). One key reason is that many of these applications do not benefit significantly from the horizontal scaling capabilities offered by microservices architectures. They may have architectural or design constraints that make such a transition complex or suboptimal,  limiting their ability to efficiently use newer computing paradigms.

Complex Set of VM Choices

Vertical scaling is also complicated by the many types of VMs available.  There are currently709 types of AWS EC2 Virtual Machine options offering varying Compute, Memory and other characteristics.  

Asking an engineer to make the optimal choice across potentially hundreds or thousands of services can be challenging, especially if new code updates change the service's characteristics and then require a new determination of the right instance type.

Importance of Vertical Scaling for AWS EC2 VMs

Vertical scaling is particularly advantageous for applications that require high-performance levels from single instances or have dependencies that complicate distribution across multiple servers. By optimizing the configuration of AWS EC2 VMs to align closely with actual workload requirements, organizations can ensure that their applications perform optimally without incurring unnecessary costs from overprovisioning. This method allows for more precise control over resource allocation, leading to enhanced performance and reduced expenditures.

Rightsizing AWS EC2 Virtual Machines with Sedai

Key Capabilities

Sedai’s Automated Optimization utilizes advanced AI technology to deeply comprehend AWS EC2 VM configurations and their impact on application cost and performance. This results in AWS EC2 VMs that are optimally sized and configured to meet the specific needs of applications without incurring unnecessary costs or performance issues. Key benefits include:

• Cloud Cost Efficiency: AWS EC2 VM costs can be reduced by up to 30% or more through optimized resource allocation.
• Performance Improvement: Enhance customer-facing services with up to 25% better latency, ensuring a smoother user experience.
• Reduced operations effort.  The time needed to rightsize VMs is reduced by up to 90%.

Sedai’s Automated Optimization uses advanced AI that not only deeply understands AWS EC2 VM configurations and how they are impacting application cost and performance. This results in VMs that are optimally sized and configured to meet the specific needs of their applications without any excess cost or underperformance.

How It Works

Our AI-driven platform continuously analyzes your AWS EC2 VMs to detect inefficiencies. It then autonomously implements optimizations, adjusting resources in real-time without requiring manual intervention.

The Sedai platform operates on a simple yet effective process: Discover, Recommend, Validate, Execute, and Track:

• Discover: Sedai first discovers your AWS EC2 VM infrastructure and application pattern, going through three steps:some textIdentifying the app boundary by looking at traffic patterns (e.g., because they use a common load balancer, or by virtual machine tagging). A set of virtual machines doing the same task and expected to behave similarly can be termed an application. This definition means that a collective action will be taken on all the instances of the app.Standardizing metrics for optimization.   In a heterogeneous fleet, a service may use Node Exporter for Linux, or WMI Exporter or Windows exporter for Windows. It is important that the metrics are labeled correctly such that the system can precisely identify the metrics of a specific application.Identifying golden signals to drive optimization.  Finding the right signal to listen to can seem like finding a needle in a haystack.  Sedai will look for the best golden metrics (latency, error, saturation, and throughput of an application) so that this information can be used in algorithms and machine learning systems such that a recommendation can be generated.
• Identifying the app boundary by looking at traffic patterns (e.g., because they use a common load balancer, or by virtual machine tagging). A set of virtual machines doing the same task and expected to behave similarly can be termed an application. This definition means that a collective action will be taken on all the instances of the app.
• Standardizing metrics for optimization.   In a heterogeneous fleet, a service may use Node Exporter for Linux, or WMI Exporter or Windows exporter for Windows. It is important that the metrics are labeled correctly such that the system can precisely identify the metrics of a specific application.
• Identifying golden signals to drive optimization.  Finding the right signal to listen to can seem like finding a needle in a haystack.  Sedai will look for the best golden metrics (latency, error, saturation, and throughput of an application) so that this information can be used in algorithms and machine learning systems such that a recommendation can be generated.
• Recommend: then recommends optimal settings based on deep insights into service behavior and dependencies. Recommendations may be provided on a manual basis or occur automatically based on user settings.
• Validate: After validating potential changes through multiple safety checks, a sequence of steps so that it could be performed safely on the customer environment:some textSafety check: If there is an action, we need to ask whether this action can be safely performed on that application without risk. If you have a green signal there, you go to the next one.Timing check: We see if it is the right time to apply the action, or is there a later preferred time to execute this particular action on this application?
• Safety check: If there is an action, we need to ask whether this action can be safely performed on that application without risk. If you have a green signal there, you go to the next one.
• Timing check: We see if it is the right time to apply the action, or is there a later preferred time to execute this particular action on this application?
• Execute: Once we have a go-ahead for these validation steps, Sedai goes ahead and performs the action.
• Learn: After performing the action, we need to figure out if the app is healthy. Updates are also tracked with a full audit trail of changes made to the infrastructure.  This step is also important because this allows us to close our learning loop and use this information for further actions.

These capabilities form part of Sedai’s overall AWS EC2 VM optimization approach which can be seen below:

‍

‍

Some of the key elements above are:

• Access to Cloud APIs which allow Sedai to identify and discover the components of your infrastructure. Sedai’s inference engine actually utilizes this information to build a topology.  With this topology information, we deduce the application. 
• Metrics exporter & Sedai Core.  With the information about the application, Sedai’s metric exporter takes the data from all the monitoring providers. With the information about the application and the metrics, Sedai machine learning algorithms can generate optimization and remediation opportunities.
• Execution engine.  The recommendations are given to the execution engine. The execution engine utilizes AWS APIs to perform the actions on the EC2 VMs.

Example 34%  Cost Saving from AWS EC2 Rightsizing

Early adopters have seen significant improvements in both performance and cost efficiency. For instance, a technology company has identified over $75k of annual savings, a 34% saving, in its dev / test environments through rightsizing using Sedai’s optimization:

To gain insights on the state of your VM fleet you can scan it at a glance to see where applications are over or under provisioned as well as optimized based on Sedai’s findings.  The example below only 12% of the apps have been optimized to date (shown as green).

Pricing and Availability

The service is available now, with flexible pricing based on the scale of your AWS EC2 VM deployment. Request a demo to see how Sedai can help you rightsize your AWS EC2 VMs.