The real lesson of this discussion seems to be: metadata has failed our expectations.
All this ancillary stuff that we'd like attached to files, like dates, client names and projects, versions and so on, are metadata. Some systems keep metadata in files: EXIF, Word, PDF. Some systems have conventions for this instead: header blocks in source code. But if neither of those applies? Only place you can put it reliably is the filename :(
If OS vendors could get over their "not invented here" syndrome with respect to default file systems we might see something a little bit more sane than FAT as the one interchangeable format. This is necessary but not sufficient to make decent file systems widely available. Then we can start hoping for such trivial features to become available.
Everybody shops for the best filesystem around that they can use. The problem is, licensing is a problem.
In other times, the industry would solve this by creating a standard for metadata interchange, but one of the biggest players gets a too big to ignore amount of revenue from FAT, and is able to block any attempt at standardizing.
>lesson of this discussion seems to be: metadata has failed our expectations.
I've written several "disk and file catalog" utilities over the years so I inevitably spent a lot of time thinking about the "metadata" problem.
I think the issue is that it's impossible to solve metadata in a universal way that satisfies all scenarios. This is why metadata often ends up being inscribed into the filename. It's the "least worse" solution.
Let's take one example of the scientific data of csv files. Typical Comma-Separated-Value files do not have metadata fields such as author, measuring device, timestamp of readings, GPS coordinates. (Yes, csv files sometimes have a first line for "column names" which is arguably metadata but that's not the higher-level metadata I'm talking about.)
Exactly where does one put that high-level metadata?
1) If one makes a new pseudo-standard that signifies any lines at the top the csv beginning with "//" as metadata, that means that modifying any metadata of a 100GB csv file (e.g. change author from "John Doe" to "Jacob Doe" is rewriting the whole 100GB file to add 1 byte.) As a related issue, let's say you have hash of "e1bb76e7391b93eb12" for the csv file. You really want a stable hash that represents the actual "raw data" of the csv file. You don't necessarily want the hash to change just because the metadata changed. In this case, embedding metadata into the file itself makes certain operations worse since typical hash utilities don't have "intelligence" about which parts of the file is "important" for hashing. (A similar problem is scanning mp3 files for duplicates. If 2 mp3 files have bit-identical audio output but the metadata tags are different, are they the same or different?!? It depends.)
2) if you put metadata at the end, typical utilities won't know about about it. (UNIX has "tail" command but standard MS Windows does not. The tail command is also unstructured and read-only which makes it a non-solution for managing end-of-file metadata fields. Also, the "quick" view of GUI file managers show the top of the file and not the bottom of it.)
3) If you put metadata in a separate file, it easily gets lost. File managers like MacOS Finder and MS Windows Explorer don't know when 2 files are supposed to be "treated as one unit" vs separately.
4) If you try to put metadata in a separate special area using os file system features suchs MS "NTFS alternate data streams" or Mac OSX "resource forks", they will get lost when transferring across incompatible filesystems or uploading to Amazon S3.
If one is feeling uncharitabe, one could say the MS WinFS[1] was a spectacular failed attempt at unifying metadata. (A relational database that makes metadata more of a 1st class concept.) Nobody has tried it on that level since. Even Apple's new file APFS system didn't have the same metadata ambitions as WinFS.
The combination of tradeoffs leads everybody to re-invent the idea of embedding metadata (including namespacing hierarchies) into filenames. The article's suggestions for scientific data filenames looks very similar to filenames that companies end up using for ETL pipelines.[2]
"File managers like MacOS Finder and MS Windows Explorer don't know when 2 files are supposed to be "treated as one unit" vs separately"
They sort-of do, each in their own way. Mac OS has packages: directories with files that the Finder treats as a single item. I think this is mostly implemented outside of the file system; if you give a directory a file name with an extension that some application claims to be an extension for a package, the Finder treats it as a package.
On Windows, when you export a web page from Internet Explorer as "web page, complete", you get a file and an associated folder containing the images of the file. MS Explorer shows them as separate icons, but knows that the two form a unit; when you delete one, it informs you about the existence of the other. I don't know how this is implemented, but suspect it is 100% outside of the file system, too.
Also, classic Mac OS had resource forks: a single alternate stream for every file. Text editors used it, for example, to store the cursor position, line wrap settings, etc. across saves.
Couldn't we just come up with some convention for "expanded filenames" where the meta-data is included in the file name itself? In the UI portion, you see what you see now, no difference, but say anything after the // delimiter in the file name is considered meta data and not shown in the windows/terminal UI.
Not sure if it's a good solution but if I were to put the meta data somewhere I would somehow try to put it in the identifier of the file (the name) as it is data that would help me identify the file AND it's content!
This is partially what filesystem forks/streams were supposed to be for. The big problem, as mentioned, is that this works for the OS, but isn't somehow transferred to third parties.
All this ancillary stuff that we'd like attached to files, like dates, client names and projects, versions and so on, are metadata. Some systems keep metadata in files: EXIF, Word, PDF. Some systems have conventions for this instead: header blocks in source code. But if neither of those applies? Only place you can put it reliably is the filename :(