Documentation and Best Practices

Learn how to use Cloudability and get the most out of our cloud cost management tool.

Follow

Rightsizing for Amazon S3 (Simple Storage Service)

About

Amazon S3 (Simple Storage Service) is a popular cloud object storage service. Organizations use S3 for storage scenarios such as images and videos, static website content, big data analytics, backups, and file sharing, among many others.  In terms of popularity, across current Apptio Cloudability customers, S3 is usually the third or fourth-largest AWS spend category by service, following virtual machines (EC2), persistent disks (EBS), and relational databases (RDS),

Buckets, objects, and storage classes are key concepts in S3 you should know about. S3 is organized around buckets, which are the top-level containers. Inside these buckets reside the individual objects, which can additionally be organized hierarchically within folders. Storage classes are tailored based on access patterns and durability needs, ranging from frequent to infrequent (archival) storage. It’s important to note that an object, not a bucket, is associated with a storage class.

Storage Classes

Here are the currently available S3 storage classes, each differentiated by price, performance, availability, and minimum size and duration requirements.

s3-storage-classes.png

 

Challenges

Organizations can have up to one thousand buckets per account, each of these containers might contain a few to several hundred thousand (even millions) of individual objects, up to five terabytes storage per bucket. Costs are incurred at the rate for the object’s storage class and factor by the allocated storage space (GB/month), duration, the number of operations (read, write, lists, delete), and volume of data transferred (egress).  Depending on the storage class, additional fees might be incurred such as monitoring, storage class transitioning, early delete, and restoration fees.

By default, S3 objects are created in the highest-priced storage class - S3 Standard. In fact, our analysis shows that over 85% of all objects belong to this storage class.

To manually optimize objects, you need to identify the rates for each storage class by region, collect the past access patterns and operations, then evaluate each of the alternatives considering all the trade-offs. Now scale this by hundreds of buckets and millions of objects, and it’s easy to understand how optimization is complex and challenging.

Insights and Optimization

Rightsizing recommendations for S3 will provide insights and visibility into critical bucket level attributes and metrics, such as storage class, storage size, count of objects, and operations. Furthermore, in the case where a bucket consists of objects of heterogeneous storage classes, we will provide the break out of these classes. Lastly, based on the rightsizing model, we will provide a recommendation for optimal storage class and estimated cost savings. Our analysis has identified up to 35% savings opportunities are available by following these recommendations.

Rightsizing for S3

Daily, Apptio Cloudability will analyze your S3 cost, usage, and utilization metrics looking back the past 30 days and produce bucket level insights and optimization recommendations.

To access Rightsizing for S3, within Apptio Cloudability, navigate to Optimize, Rightsizing, then click S3

Recommendations List

The recommendations, refreshed daily, will be displayed in a list similar to below.

Note: Only buckets with incurred costs greater than zero will be displayed.

s3-rightsizing-recommendations-grid.png

Columns

  • Bucket Name - S3 bucket name.

  • Account Name - AWS account name.

  • Size - Size of all objects within the bucket.

  • Objects - count of objects within the bucket.

  • Requests - number of requests (read, write, list) done via the console or API.

  • Cost - the cost incurred for the past 30-days for the objects within this bucket.

  • Current - the derived storage class for the bucket. When 100% of all objects are in the same storage class, that storage class will be displayed. If a bucket consists of objects within multiple storage classes, then “MIXED” will be displayed. Clicking the details will display the relative distribution across these classes.

  • New - the recommended optimal storage class for all objects within the bucket.

  • Action - either Rightsize or No Action. See Taking Action below.

  • Cost Savings - estimated savings had you been using the recommended storage class for the past 30-days.

 

Details Pane

Clicking the Details button, or if you have enabled Jira integration, the three vertical ellipses will launch the Details Pane. The example below highlights a MIXED storage class bucket. Note this bucket is comprised of three storage classes, and relative storage amounts are also displayed.

s3-rightsizing-details-pane.jpg

 

Recommendation(s)

One or more recommendations will be displayed, ranked by cost savings and risk (0-5).

  • Savings - estimated savings percent had you been using the recommended storage class for the past 30-days.

  • Cost Savings - estimated savings amount.

  • Storage Class - recommended storage class.

  • Action - recommended action.

  • Risk - Measure (on a scale of 0-5) representing the number of storage class transitions from frequent to infrequent access.

Charts

  • STORAGE SIZE (BYTES) - displays storage size amount by storage class, possible values are:

    • Standard

    • Intelligent FA (Intelligent Tiering - Frequent Access)

    • Intelligent IA (Intelligent Tiering = Infrequent Access)

    • Infrequent Access

    • One Zone

    • Glacier

    • Deep Archive

    • Reduced Redundancy

  • REQUESTS (READ/WRITE COUNT)

    • Individually displays read and write requests performed against the bucket, this chart provides valuable insights into the usage category of the bucket such as mostly reads, balanced read-writes, or write oriented.

  • NUMBER OF OBJECTS (COUNT)

    • graphs the number of objects in the bucket over the time period.

  • DATA TRANSFERRED (BYTES)

    • amount of egress to the internet (or data transferred across regions).

  • DATA TRANSITIONS (BYTES)

    • amount of data transitioned across storage classes (excludes Standard).

  • EARLY DELETION (BYTES)

    • amount of storage incurring an early deletion fee.

Frequently Asked Questions

How can I take action on these recommendations?

To transition the storage class of objects in bulk, there are currently two options:

  • Object Life Cycle Management

A life cycle management rule can be enabled at the bucket level to transition all contained objects to the recommended storage class based on the age after the object creation date. For more details see, https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html

  • S3 Batch Operations

If you prefer a programmatic route, you can leverage S3 Batch Operations to change the storage class of all or selected objects within one or more buckets. For more details see - https://aws.amazon.com/blogs/aws/new-amazon-s3-batch-operations/

Why do many of my recommendations suggest Intelligent Tiering as the target?

As background, Intelligent Tiering is a hybrid storage class and service that monitors each object’s access patterns and attributes. Based on this data, the service will automatically transition objects between the Standard and Infrequent Access storage class for a per-object monitoring fee in addition to the storage class-specific rates

By default, all new S3 objects are created in the Standard storage class. In fact, an analysis of current customers found that over 85% of all buckets had all objects in the Standard storage class.

While this is a good option for cost optimization, it may not always be the appropriate choice. Our bucket level rightsizing model helps you to identify which buckets will benefit from this service. Since most objects are created in the Standard tier, you will initially see this often, as the recommended storage class. However, as your S3 buckets become optimized over time, additional incremental optimization recommendations such as Infrequent Access or even Glacier may be recommended.

Cloudability’s Rightsizing for Amazon S3 provides you insights and recommendations to confidently make these optimizations.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.