Documentation and Best Practices

Learn how to use Cloudability and get the most out of our cloud cost management tool.

Follow

Important Utilization Metrics And How To Leverage Memory Data

Cloudability’s Rightsizing Engine evaluates the underlying resource utilization for each EC2 instance and recommends instance types that are well matched to each utilization profile. The end goal is to keep your costs down while being mindful of operational risks.

To get the most accurate recommendation there are four utilization metrics that need to be assessed: CPU, Disk IOPS, Network Bandwidth and Memory Utilization. For a number of very good reasons we’ve taken the approach of pulling this data directly from CloudWatch. One of these reasons is that three of the metrics are saved to CloudWatch by default for each instance without you doing anything. The fourth one however, Memory Utilization, requires a little initial effort on your end to publish to CloudWatch. This is done using what AWS calls Custom Metrics which you can read about here.

The good news is that AWS has come up with a standard format for memory data publication and we’ve taken a streamlined approach to ingest that data. We require only one custom metric be published:

metric-name: MemoryUtilization

namespace: System/Linux or Windows/Default

dimensions: InstanceId (it’s important to only add this one dimension)

unit: percent

Here is an example of a Windows/AWS Cloudwatch config.json that we have verified works with our platform:
{
   "logs":{
      "logs_collected":{
         "files":{
            "collect_list":[
               {
                  "file_path":"",
                  "log_group_name":"."
               }
            ]
         }
      }
   },
   "metrics":{
      "namespace":"Windows/Default",
      "append_dimensions":{
         "InstanceId":"${aws:InstanceId}"
      },
      "metrics_collected":{
         "LogicalDisk":{
            "measurement":[
               "% Free Space"
            ],
            "metrics_collection_interval":60,
            "resources":[
               "*"
            ]
         },
         "Memory":{
            "measurement":[
               {
                  "name":"% Committed Bytes In Use",
                  "rename":"MemoryUtilization",
                  "unit":"Percent"
               }
            ],
            "metrics_collection_interval":60
         },
         "Paging File":{
            "measurement":[
               "% Usage"
            ],
            "metrics_collection_interval":60,
            "resources":[
               "*"
            ]
         },
         "PhysicalDisk":{
            "measurement":[
               "% Disk Time",
               "Disk Write Bytes/sec",
               "Disk Read Bytes/sec",
               "Disk Writes/sec",
               "Disk Reads/sec"
            ],
            "metrics_collection_interval":60,
            "resources":[
               "*"
            ]
         },
         "Processor":{
            "measurement":[
               "% User Time",
               "% Idle Time",
               "% Interrupt Time"
            ],
            "metrics_collection_interval":60,
            "resources":[
               "*"
            ]
         },
         "TCPv4":{
            "measurement":[
               "Connections Established"
            ],
            "metrics_collection_interval":60
         },
         "TCPv6":{
            "measurement":[
               "Connections Established"
            ],
            "metrics_collection_interval":60
         }
      }
   }
}

It’s generally recommended to publish this information every minute. Once you have this data flowing into CloudWatch we’ll pick it up without requiring you do anything more.

If you’d like to validate within CloudWatch itself you can do so in the AWS console and you should find graphs similar to this:

image2.png

 

Linux Options

Install the standard CloudWatch monitoring scripts for Linux available on the Amazon site. This is the most popular and straightforward option. Here is the crontab configuration we use at Cloudability which is described below :

* * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --from-cron

This will publish exactly what Cloudability requires and nothing more. Other options include creating your own agent which integrates with the AWS SDK. Here is a great example using Golang that some of our customers have had success with.

The method for enabling the perl script is as follows (written for Amazon Linux, but similar for other flavours)  - the full details can be found here.

Firstly, bind your IAM role to the EC2 instance with the required permissions to publish events to Cloudwatch.

SSH to the EC2 instance ; and 

Update your machine (optional, but recommended/good practice) 

sudo yum update

Install Perl (if not there already)

sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64

Download the AWS Monitoring script from AWS

curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O

Unzip this locally to ~ and change to the directory where this exists 

unzip CloudWatchMonitoringScripts-1.2.2.zip && \
rm CloudWatchMonitoringScripts-1.2.2.zip && \
cd aws-scripts-mon

Create /Update Crontab 

crontab -e

Add the following line to your crontab (Use I to Insert, the editor is native VI)

* * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --from-cron

Save and Exit with

:wq!

Confirm that the metrics in 5 min intervals are arriving in Cloudwatch metrics, normally within 24 hours they will start showing up as metrics in Cloudability / Rightsizing.

...Another Linux Option

Amazon have recently released the Cloudwatch Agent which also provides the ability to return additional metrics from your EC2 instances to Cloudwatch (and in turn (Cloudability). Details of this are covered here but in summary are as follows :

Firstly, bind your IAM role to the EC2 instance with the required permissions to publish events to Cloudwatch.

SSH to the EC2 instance ; and 

Update your machine (optional, but recommended/good practice) 

sudo yum update

Download the Cloudwatch agent to ~

wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

Install using Redhat Package Manager 

sudo rpm -U ./amazon-cloudwatch-agent.rpm

Using VI ; create a file called xnixcwagent.json 

vi xnixcwagent.json

Use I to insert and paste the following text in

{
    "metrics": {
        "namespace""System/Linux",
        "append_dimensions": {
            "InstanceId""${aws:InstanceId}"
        },
        "metrics_collected": {
            "mem": {
                "measurement": [
                    {
                        "name""mem_used_percent",
                        "rename""MemoryUtilization"
                    }
                ],
                "metrics_collection_interval"60
            }
        }
    }
}

Save and Exit with

:wq!

 Enable the metrics collection by entering the following command ;

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:./xnixcwagent.json -s

Confirm that the metrics in 5 min intervals are arriving in Cloudwatch metrics, normally within 24 hours they will start showing up as metrics in Cloudability / Rightsizing.

 Windows Options

Traditionally, the standard approach from Amazon was to use a PowerShell script with the AWS SDK, however the DLL used has been deprecated for some time. You could potentially use your own agent connected with the newer AWS SDK however it would need to adhere to the same output format that Cloudability expects with the respective dimensions in place. Cloudability fully supports the newer AWS Universal agent with the above config.json / respective IAM role in place bound against each EC2 instance in supporting memory metrics. This can be enabled by the following (on an individual EC2 basis where the IAM role is created and bound to the respective EC2 as well) ;

Fetch installer

wget https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/amazon-cloudwatch-agent.msi -outfile amazon-cloudwatch-agent.msi

Once downloaded, install it on the machine 

msiexec /i amazon-cloudwatch-agent.msi

CD to the location where the Agent is installed (C:\Program Files\Amazon\AmazonCloudWatchAgent) to and copy the config.json from above to this location. 

Start the agent pointing to the respective config as required :

./amazon-cloudwatch-agent-ctl.ps1 -a fetch-config -m ec2 -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json" -s

More details here with respect to the AWS Universal Agent (for both Windows and Linux) 

Final Word

Having this memory data within CloudWatch is going to provide benefits well beyond Cloudability and we’d highly recommend going down this path. For example you could use the memory data to trigger autoscaling events or trigger alarms. Once you find what method works for you it’d be a good idea to roll that up into your configuration.

Was this article helpful?
2 out of 2 found this helpful
Have more questions? Submit a request

0 Comments

Article is closed for comments.