Skip to content

Object Storage - S3 compatible

Object storage (often just referred to as S3) is a type of storage that has grown in popularity since its AWS released its S3 product in 2006. It is an alternative to file or block storage and comes with its own set of tradeoffs. There's lots of discussion online about these tradeoffs, so we'll focus on the primary benefits for VTTI.

While there are a number of different object storage systems around, S3 has become the industry standard. Unfortunately, S3 has become an overloaded term because it refers to both the AWS product and also to the standard that can be adopted by other systems and services, sometimes called S3-compatible or S3-style. VTTI leverages some of these like Minio and Ceph.

Info

With object storage, you're technically storing objects, but you'll frequently see files and objects used interchangeably and synonymously including throughout this documentation.

Benefits

  • Accessibility from anywhere - Object storage is HTTP (web) based, so access can be allowed from anywhere in contrast to file based or block storage that require local network access. This will allow our storage to be accessed from other systems and networks, like VT's own ARC platforms, cloud computing resources, or other locations.
  • Security - Access to objects can be controlled in a variety of ways, and credentials can be distributed, used, and rotated in ways that provide more flexibility than traditional file systems allow.
  • Scaling and cost - In general, object storage systems can scale beyond traditional file system limitations. The cost of the underlying storage systems also tend to cost significantly less than the scale-out NAS (Network Attached Storage) systems VTTI has been using since ~2010.
  • Growing standard - More and more systems and applications are natively supporting object storage, some even expecting or requiring it for certain functionality.

Disadvantages

There are some disadvantages to object storage.

  • Whole object access only - Objects are stored and have to be updated as a singular items. If you need to update just a small part of a 5GB object, you have to update and replace the whole 5GB object
  • File seeking and random access - Objects can't be randomly accessed for reading in the quite the same way as file systems, though range read requests do allow portions of the object to be read without needing to read through the whole object

Uses and plans

What are VTTI's plans for object storage?

Application use

Applications can request and use object storage for a variety of use cases and needs

Small scale use

Users and other ad hoc purposes will generally be supported

Project data

Access to existing project data as well as storing new project data via our traditional project "folders" will be supported

AWS S3

We also have the ability to use AWS S3 when need dictates, though depending on the volume of data, the cost may need to be charged to project funds.

What's available now

Our current offering is a Minio system with a few TBs of storage. We plan to transition most use cases to a Ceph system during 2023, so we strongly recommend that projects and users be forward-thinking and not use Minio-exclusive features where possible. While much of the S3 standard and functionality is supported across multiple systems like Minio and Ceph, they do not all implement every feature and some are supported in ways outside of the standard.

Warning

While we will likely continue to maintain a Minio system for those use cases that require some of the special features it offers, we've learned from our use over several years that Minio, as an open source offering from a single commercial developer, can and does change their featureset based on their own commercial interests.

We have also been beta-testing object-based data access of our existing project storage.

Getting access

You can currently talk to Kenny or Neal to request access. Please see the access documentation for developer details.