The Mystery of Folders on AWS S3

The difference between objects and files, and what the AWS Console really does when you press the "create folder" button.

· 10 min read · 1911 words

At the risk of sounding obvious: buckets are not filesystems. Photo by Sixteen Miles Out on Unsplash At the risk of sounding obvious: buckets are not filesystems. Photo by Sixteen Miles Out on Unsplash

When learning about cloud ☁️ or talking to pernickety data engineers 👨‍💻, you’ve no doubt picked up that “object storage is not a file system” or that “there’s no such thing as folders on S3”. But what is the difference? Why does the difference matter? And most importantly: why do the AWS S3 Console (as well as GCP) have a “Create folder” button, and what does it do? Let’s find out!

Object storage ≠ file storage

The most defining difference between object storage and file storage is that there is no hierarchical structuring of files — there are no folders in object storage. Instead, there is just a “flat” set of named binary blobs that we could call files but refer to as objects instead. The set itself is referred to as a bucket.

The lack of hierarchy allows for buckets to achieve far greater size, durability and availability than conventional filesystems. The reason for this is technical: lack of hierarchy makes it easy for objects to be dispersed and replicated across different physical machines without any need for centralised bookkeeping.

Some object storage systems only allow storage, retrieval and deletion of objects by their name, also known as their key. However, most object storage systems, including S3, have some pretty powerful capabilities for object listing:

  • S3 allows quick filtering of files by prefix
  • S3 allows splitting object listings by a separator, most tools set / as default by convention

Thanks to this capability, object storage systems can appear similar to filesystems, although differences remain (see below) — whether these are important depends on circumstances.

Folders in the AWS Console

Unfortunately, cloud vendors like to make things complicated. AWS introduced the concept of a “folder” in the AWS Console — their web UI for S3. Does this not contradict the above? Why yes, yes it does!

AWS Console Screenshot

What does the create folder button do?

Let’s investigate what the “Create Folder” button does. Start by creating a bucket:

aws s3 mb t35tbuck3t --region eu-west-1

Now navigate to this bucket in the Console, and you’ll see a shiny button, “Create folder”. What gives? Folders on object storage? Let’s try pressing it.

Creating a folder flow Folder creation dialog Folder creation confirmation We are creating a “folder” in our bucket, t35stbuck3t.

Something happened, but what? The Console tells us it created a folder, but that can’t be it — remember that folders do not exist! We can investigate using some more low-level API calls:

aws s3api list-objects --bucket t35tbuck3t

This returns:

{
  "Contents": [
    {
      "Key": "path/",
      "LastModified": "2022-03-21T15:39:25+00:00",
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      "Size": 0,
      "StorageClass": "STANDARD",
      "Owner": {
        "DisplayName": "...",
        "ID": "..."
      }
    }
  ]
}

The Console made a single empty object named path/. This shows up in the Console as an empty folder, even though it is just a 0-byte object.

Creating folders using the CLI

We can use the CLI to achieve the same thing; let’s make a “folder” to/ “inside” path/:

aws s3api put-object --bucket t35tbuck3t --key path/to/

Confirm its successful creation in the Console:

Folder created successfully

Adding files to folders

Upload a file, object.txt, inside of the /path/to/ “folder”:

echo "testobject" | aws s3 cp - s3://t35tbuck3t/path/to/object.txt

You can see the object appearing inside the to/ folder you created. However, uploading a similar object to the path /path/two/object.txt without first creating a folder two/ works too:

echo "testobject" | aws s3 cp - s3://t35tbuck3t/path/two/object.txt

In this case, you can see both the folder two/ and the object object.txt inside appear.

Realize the truth and claim the power that is rightfully yours! Realize the truth and claim the power that is rightfully yours!

Remember — folders do not exist. The cp command creates objects, in this case, an object with key /path/two/object.txt inside of the set of objects called t35stbuck3t. The folder concept is just a useful lie resulting from the convention to split listings by the / character in the Console (and many other tools). Even though we never created the directory two/ in path/, the Console tells us it exists.

The "folder" two/ shows up, even though we did not create it. The “folder” two/ shows up, even though we did not create it.

In fact, we can remove objects s3://t35tbuck3t/path/ and s3://t35tbuck3t/path/to/ without it having any observable impact!

Difference in behaviour

From the Console, folders path/to/ and path/two/ look identical. However, they’re different and will behave differently: two disappears when the object inside is removed, while to can exist on its own 🤯.

We cannot see why from the Console, but we can by thinking. Or by simply listing the bucket’s objects:

aws s3api list-objects --bucket t35tbuck3t | jq '.Contents[].Key'

Result:

"path/"
"path/to/"
"path/to/object.txt"
"path/two/object.txt"

Indeed, path/to/ remains visible as a folder because there is an (empty) object with this name. Knowing this, path/two, which exists only as a prefix of path/two/object.txt, disappearing together with path/two/object.txt makes sense. Unfortunately, there is no way to discern the two situations from the Console.

Some more great ideas

Knowing that everything on S3 is just objects with weird names and that / is a legal character for use in object names might give you some pretty awesome ehrm I mean horrible ideas.

  • It is possible to create non-zero-byte files ending on /. Creating such objects is a great way to baffle your colleagues almost always a bad idea and looks confusing in the Console. You can even make this file unreachable (from the Console), e.g. by creating another file ending on two / characters.
aws s3api put-object \
   --bucket t35tbuck3t \
   --key horribleidea/ \
   --body /dev/fd/0 <<<'this is probably a bad idea'

An object, horrible/, with a name ending on a slash — a bad idea! An object, horrible/, with a name ending on a slash — a bad idea!

  • It is possible to have an object and a “folder” share the same name. This is also absolutely hilarious confusing and to be avoided.
aws s3api put-object --bucket t35tbuck3t --key path/
echo "testobject" | aws s3 cp — s3://t35tbuck3t/path/

An object "path" and a "folder" "path/" present in the same bucket — also a bad idea! An object “path” and a “folder” “path/” present in the same bucket — also a bad idea!

  • The “Create folder” button in the Console (apparently) does some bonkers stuff behind the scenes, as for it to work, you need permission to list the relevant part of the bucket (ListObjects), not just permission to use the PutObject action. If ListObjects requests are restricted to use / as a separator (often mentioned as a good idea by AWS, for example here and here), then the “Create folder” button will also not work 🤷‍♂️. The UI will incorrectly single out the s3:PutObject action as the culprit. (I confirmed this weird behaviour on 21/3/2022)

The "insufficient permissions" are, in this case, actually due to a condition put on the ListObjects action. The “insufficient permissions” are, in this case, actually due to a condition put on the ListObjects action.

Conclusion

AWS introduced the folder concept to its S3 Console to make it more user-friendly, introducing a lot of unnecessary confusion in many cases.

A harmful truth is better than a useful lie. — Thomas Mann

That’s it! Now you know everything!

…

Still here?

More on buckets and filesystems

Curious how proper hierarchy differs from the ability to list prefixes? If other differences exist between objects and files, buckets and filesystems? Read on! There’s always more to know.

Objects versus files

  • File metadata consists of a well-defined set of properties such as an owner and ownership bytes. Object metadata is typically more flexible, allowing the attachment of tags and other forms of user-defined metadata.
  • Files can be modified and appended; objects in object storage systems are immutable. You cannot append to an object or rename an object, except through a sequence of copy and delete commands.

Both don’t have to be true. Some object storage systems, not S3 but e.g. Ceph, do allow for “appendable objects”; this usually comes at the cost of object versioning. Many filesystems allow for file extended attributes.

Buckets versus filesystems

  • You can write objects (files) with lengthy prefixes within a bucket without first creating folders…. Because there are no folders!
  • No folders implies no folder operations. Operations on a prefix are not a single operation but an operation on every individual matched object.
  • No folders implies no folder properties and permissions. Within buckets, you cannot assign properties to a prefix and have them apply to objects sharing the prefix in the same way as on a filesystem.

The lack of move operations and folder-level operations is often hidden from you by higher-level operations in the CLI or SDK. However, you will definitely notice this when your “folder” has many objects: operations will take longer and become more expensive. In lieu of folder permissions, you can usually condition IAM policies or bucket policies on prefixes, but certain limitations apply.

Examples of filesystems and object stores

Interfacing

  • Object storage systems typically expose an HTTP interface (REST).
  • Filesystems are typically managed by your OS kernel and interfaced with through system calls.
  • Linux enables implementing userspace filesystems using FUSE. FUSE allows you to implement a filesystem on top of HTTP 😱 🤤.

Hybrid systems

The difference between object storage and file systems is not always clear-cut. Many storage solutions exhibit properties of both; many filesystems are not POSIX compliant. Many tools allow you to expose buckets or parts of a bucket as a POSIX-ish filesystem or folder through FUSE (gcfuse, rclone, s3fs-fuse, Goofys, …). Many programs will work with such “ish” filesystems just fine, while other systems might fail catastrophically, for example, because they rely on folder deletion being constant-time or atomic.

Layering

Object storage is generally considered more “low level” than file storage. A frequently seen pattern is implementing a filesystem or something filesystem-like on top of object storage by introducing a more centralised metadata layer component. Such layering underpins SeaweedFS Filer, CephFS, JuiceFS (Redis+S3) and many more filesystem servers. Azure Data Lake Storage Gen2 is similarly built on top of Blob Storage.