Documentation and Best Practices

Learn how to use Cloudability and get the most out of our cloud cost management tool.

Follow

Important Utilization Metrics And How To Leverage Memory Data

Cloudability’s Rightsizing Engine evaluates the underlying resource utilization for each EC2 instance and recommends instance types that are well matched to each utilization profile. The end goal is to keep your costs down while being mindful of operational risks.

To get the most accurate recommendations there are four utilization metrics that need to be assessed: CPU, Disk IOPS (in the case of instances that have local disks), Network Bandwidth and Memory Utilization. For a number of very good reasons, we’ve taken the approach of pulling this data directly from CloudWatch. The key reason being that hypervisor level metrics are saved to CloudWatch by default for each instance without you doing anything. This leaves Memory Utilization as the only metric that requires a little extra effort on your end to publish to CloudWatch. This is done using what AWS calls Custom Metrics which you can read about here.

The good news is that AWS has come up with standard formats for memory data publication and we’ve taken a streamlined approach to ingest that data. We require only one custom metric be published using either the older perl based agent or the current unified agent (recommended). 

AWS Unified Agent

This is the preferred method for memory data collection going forward. Cloudability supports the standard location and naming conventions that the unified agent uses when writing custom metrics to Cloudwatch. These details are:

metric-name: "mem_used_percent" (for Linux)

metric-name: "Memory % Committed Bytes In Use" (for Windows)

namespace: CWAgent

dimensions: InstanceId (it’s important to only add this one dimension)

unit: percent

(note that the only difference between Windows and Linux based machines is the metric name. All other details are consistent between them) 

Instructions

AWS provides a number of options for installing the agent which you can find here.

The unified agent provides capabilities to publish many types of custom metrics. A minimal config to publish just the memory info looks like this:

{
"metrics": {
"metrics_collected": {
"mem": {
"measurement": [
"used_percent"
],
"resources": [
"*"
]
}
},
"append_dimensions": {
"InstanceId": "${aws:InstanceId}"
}
}
}

When the custom metric is published it's important that it only has the InstanceId dimension. This is because as described here by AWS each dimension combination constitutes a completely separate metric, one that must be queried with all dimensional data. 

If you do wish to publish multiple dimensions you can rely on the aggregation_dimensions keyword as show below. The extra dimension in this case is AutoScalingGroupName. In this example you will end up with three published memory metrics each with a different dimension combination 1) just InstanceID  2) just AutoScalingGroupName 3) both InstanceID and AutoScalingGroupName.

{
"metrics": {
"metrics_collected": {
"mem": {
"measurement": [
"used_percent"
],
"resources": [
"*"
]
}
},
"append_dimensions": {
"InstanceId": "${aws:InstanceId}",
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"aggregation_dimensions": [
[
"AutoScalingGroupName"
],
[
"InstanceId"
]
]
}
}

Perl Based Agent

Please note that AWS has deprecated the perl based agent but it remains available for downloading. Cloudability will continue to support this method however we recommend you switch to the newer unified agent if possible.

As stated above only one custom metric is required. The perl based metric looks like this:

metric-name: MemoryUtilization

namespace: System/Linux or Windows/Default

dimensions: InstanceId (it’s important to only add this one dimension)

unit: percent

Instructions

Install the perl CloudWatch monitoring scripts as described on this AWS webpage. You will find instructions on this page for installing the pre-requisite packages on machines running the Amazon Linux AMI and other popular operating systems such as Red Hat.

Here is the crontab configuration we use at Cloudability for Linux machines:

* * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --from-cron

This will publish exactly what Cloudability requires and nothing more. You should be able to confirm that the memory metric is reported at 5 min intervals.

Others options: Other options include creating your own agent which integrates with the AWS SDK. Here is an example using Golang that some of our customers have had success with.

Verifying Results

Normally within 24 hours of configuring your EC2 instance this memory information will become available within Cloudability rightsizing.

Here is an example of the resultant memory data when viewing in the AWS Cloudwatch console. This can be helpful for debugging purposes. 

image2.png

Note: this is example data from the perl based agent. Unified agent data will appear similar but with the relevant namespace and metric name.

Final Word

Having this memory data within CloudWatch is going to provide benefits well beyond Cloudability and we’d highly recommend going down this path. For example, you could use the memory data to trigger autoscaling events or trigger alarms. Once you find what method works for you it’d be a good idea to roll that up into your configuration.

Was this article helpful?
2 out of 2 found this helpful
Have more questions? Submit a request

0 Comments

Article is closed for comments.