What are resource forks in Mac OS X? How can they be used? What applications are using them? Are they being deprecated? Are developers encouraged to make use of them? Are they the future? Why is there a lack of up-to-date documentation? These are just a handful of questions that I had about the underlying Mac OS X filesystem. It took me several hours and hundreds of Google queries only to return with conflicting information. Through a process of gleaning information from hundreds of emails on mailing lists and usenet groups, sifting through outdated documentation, and hands-on experimentation I formed a coarse understanding of this technology. The following are my compiled observations which may or may not be correct.
I am writing this under the assumption that the target system in question is running the HFS+ filesystem.
Extended attributes vs. resource forks
First, I present an excerpt from the Wikipedia page on Resource forks:
The resource fork is a construct of the Mac OS operating system used to store structured data in a file, alongside unstructured data stored within the data fork.
From what I have gathered, there was a distinct difference between resource forks and extended attributes at one point. However, this appears to no longer be the case as I will attempt to explain in the next section. In short, the terminology usage has become muddied, and this seems to be the source of most confusion.
Resource forks are being deprecated
There is quite a bit of noise saying that resource forks are going away. Based on the information I found, this assertion is only partly true. Indeed, resource forks have already been phased out by way of Apple silently changing the underlying API implementation. Whereas this information was previously stored in the resource fork, it has now been moved to the data fork. From what I can tell, the majority of this data now manifests in the “Resources” directory that can be found within application bundles. However, there is still more resource fork data and this is now stored as an extended attribute under the key “com.apple.ResourceFork.”
At this point I must interject that it appears that Mac OS X continues to support named forks. There does not seem to be any limits on the number or size that these may be, but I’m not aware of any applications that make use of them. The majority of metadata appears to be contained within the extended attributes tree. This metadata is key-value based and generally very small in size.
Below is a bullet list of my observations.
- Apple’s resource fork was simply one of the many different forks that the Mac OS X filesystem can apparently handle.
- Apple’s resource fork is dead, but
- Apple’s APIs for interacting with their resource fork now store this information as an extended attribute (com.apple.ResourceFork) rather than using a separate fork.
- Mac OS X continues to support named forks.
- I have not seen any applications (Apple or otherwise) that use a named fork beyond the original resource fork.
- Extended attributes are used extensively by Apple products.
Accessing extended attributes (EA) and thereby com.apple.ResourceFork
Apple provides an API for interfacing with extended attributes, but they can be accessed without writing an application by using Terminal and the xattr command. For example, xattr -p com.apple.ResourceFork example.txt will display the data of the com.apple.ResourceFork attribute [if it exists]. Using the @ option with the ls command will give you a listing of all attribute keys for each file [e.g. ls -la@].
A neat trick that is still available in Mac OS X Leopard [10.5] from the days when ResourceFork was stored in its own fork is using ..namedfork/rsrc. If you don’t provide a name for the fork, then it defaults to Apple’s ResourceFork. For example, cat example.txt/rsrc will produce the same output as xattr -p com.apple.ResourceFork example.txt.
Gotcha!
File sizes calculated in terminal do not include the size of extended attributes, including the ResourceFork. This must be obtained separately through alternative means. For example, using ..namedfork/rsrc.
Clear as mud
I hope that this article sheds some light on the situation. Extended attributes are used quite a bit by Apple products. For example, Spotlight comments are stored as an extended attribute key-value pair. When a file is deleted using Finder, a key-value pair is written that indicates where the file originated. Time Machine uses extended attributes to keep track of backups, and the list can continue.
Extended attributes should not be feared and are a great way to store additional metadata. Unfortunately, support for extended attributes is not ubiquitous. Even some of Apple’s own bundled command line programs do not fully or properly support extended attributes. As such, from a programming point of view it might be wise to avoid use of extended attributes unless the data is transient. Any important data should be stored along with the file in the data fork.
Here’s hoping that EA support improves in the near future!
Please leave a comment if the information I have gathered is incorrect, or can be better clarified.
Everyone is pretty lose to the answer. I did a lot of research on this when I was adjusting a backup product to backup extended attributes and the resource fork.
Extended attributes and the resource fork are implemented with the same underlying mechanism in HFS+, they are all named forks.
Generically, HFS+ supports for named forks can be of any size (same limitations as the data fork). However, OS X does put some limits on top of the named forks. In OS X, the resource fork is a full named fork that can be any size. For extended attributes however, OS X limits the size of the named fork to a single block (4K). The single blocked can be inlined with the file entry. Anytime a named fork goes beyond a certain size, it needs to keep track of the blocks via an overflow list/tree. Because of this, thats probably why apple decided to limit the size of all extended attributes / named forks; except for the named fork that happens to be designated as a resource fork. It’s only called that because of the history. HFS supported only two forks, but HFS+ supported named forks generically but in the past up until 10.4, OS X only supported using one named fork (the resource fork). When OS X 10.4 came out, it supported extended attributes which (like I said above) are limited size named forks.
I have added a comment here, which may be helpful.
I spent about 2 hour to find out what really take the size in file. One of my jpeg image is showing 64kB in finder but console ls cmd showing 39kB. Now I saw xattr that shows me where those size come from in finder. It’s okay with me that apple has EA in filesystem but I wish that there is an switch that could control if finder should display the file actual size or the size combined with EA. Because my colleague is having really big issue to find out the real file size for a bunch jpeg size. She is not familiar with console operations…This is really frustration… I might end up remove all the EA of those jpeg files
Dear All,
I am confused in differentiating Extended attributes, resource forks and alternate data stream.
From reading different sources I got two different ideas:
1). these three are same things, in linux it is called extended attributes, in mac it is called resource fork, and in windows it is called alternate data stream (ads).
2). Or, only resource fork and ADS are same things and extended attribute is different from these two. And the difference between extended attribute and the other both (fork, ADS) is the size difference only. As extended attributes are smaller in size.
Please tell me which is correct. If both are incorrect then please send me the correct differences between these three.
Thanks to all
Syed
Syed,
It’s confusing and difficult to explain. Here’s my short answer that’s even more confusing:
Extended attributes and filesystem forks are different things, but they can also be the same thing in core concept and depending on the implementation.
A tad more detail:
There is a difference between a resource fork and an extended attribute. My understanding is that an extended attribute is typically constrained in size because this information is stored inline with the filesystem tree. A filesystem fork on the other hand can typically be as large as the maximum filesystem file size. Some systems such as Windows may store their extended attributes as a resource fork which can blur the line.
So, there is a technical distinction between a fork and an extended attribute. However, it seems that we generally use the term ‘extended attribute’ to refer to the same concept. That concept is attempting to answer the question, what do we do with that extra information that is important and specific to a certain file, but does not belong within the data stream itself?
Some information is pretty generic and can be applied operating system wide such as the time a file was created. [side note: each Operating system might have different interpretations of what this means though. Take a look at the difference in ctime between Unix and Windows systems.]. This can be rolled into the filesystem inline if the creators wanted to do it that way. But what do we do with other metadata that’s not so generic and can be quite arbitrary, and perhaps even large? That’s where the concept of extended attributes and resource forks enters the scene.
So, to answer you questions directly:
So, to recap. Filesystem forks and extended attributes are different things technically, but are part of the same concept of trying to answer the question what do we do with the metadata? Storing extended attributes in a fork is one possible implementation method and one that I believe Windows uses.
That’s my best explanation as I understand it. No guarantees on accuracy. Be sure to read up on Wikipedia to verify and further clarify the matter.
Edit: I should also note to readers that after writing this response, I think some of the details of my original observations in this blog post are slightly incorrect. As I demonstrate in this response, there is some grey area between the difference of an extended attribute and a resource fork. There is a technical difference between the two. Some operating systems build one on another which can further muddy the waters, but in general the overarching concept of how to handle the metadata is the same.
I am thankful to you for the reply, your reply would definitely help me a lot.
Regards
Syed Rahman Mashwani
PhD Student
University of Peshawar, Pakistan.
I don’t think that is right.
What it looks like to me is:
(1) Previously, (at least before OS X), resource forks were stored in the file system.
(2) Mac OS X (or HFS+, I don’t know) introduced Extended Attributes, which are stored in the directory structure.
(3) Once Extended Attributes are available, resource forks are stored as Extended Attributes. You can see that in the ls -@ command.
(4) I guess there is a special trick in ls, which allows the one special Extended Attribute ‘com.apple.ResourceFork’ to be accessed as though it were a separate file below the normal file and called ‘rsrc’.
So Extended Attributes and Resource Forks are different, it is just that Extended attributes are now used to implement the old Resource Forks.
Here’s a command that will list all files with resource forks, in case you want to Rsync to a Linux machine and are wondering what might be missing in the process:
find . -type f -exec ls \-l {}/rsrc \; 2> /dev/null | awk ‘ $5 != “0” { print $0 }’
First ‘cd’ into the directory you want to search, then run the command above
It would be nice to have a list of actions that can accidentally strip off a file’s extended attributes, such as copying to a Windows machine and back. (It used to be that even cp was not resource-fork-aware, but I believe that is fixed now.)
Indeed. The situation is inconsistent enough now on the same operating system, but becomes even more muddled when crossing OS boundaries.
I think rsync3 has extended attribute support now as an optional item.
As I am currently writing a paper on EA and used your website as inspiration, I’d like to clarify a bit on the fork part by showing an example I made.
*snip*
So, does this mean that all files with a /rsrc automatically gets the com.apple.ResourceFork Extended Attribute?
$ touch emptyfile
$ ls -la@ emptyfile
-rw-r–r– 1 klaag staff 0 13 Jan 11:23 emptyfile
echo “hello resource” > textfile
$ cat textfile
hello resource
$ cp textfile emptyfile/rsrc
$ ls -la@ emptyfile
-rw-r–r–@ 1 klaag staff 0 13 Jan 11:26 emptyfile
com.apple.ResourceFork 15
$
$ xattr -lp com.apple.ResourceFork emptyfile
com.apple.ResourceFork: hello resource
$
Yes, it seems that simply making something in /rsrc enables the com.apple.ResourceFork Extended Attribute.
*snip*
Are you aware of the mdls command? Where does all that data come from? mdls seems to have output even when there are no xattrs or resource forks for a given file.
From my searching, mdls can return information about files that do not necessarily need to be stored explicitly as metadata or with a resource fork. For example, the demonstration on MacRumors:Guides shows that mdls can be used to get the width of an image. From what I remember, this is not information that is going to be explicitly stored as metadata since this attribute does not differ from what the image width actually is.
Are there any advantages of this resource fork/ data fork aspect to apples operating system over however windows do it?
I haven’t really explored the NTFS alternate data streams, so I did a quick search and ran across the following article that you might find interesting: http://www.symantec.com/connect/articles/what-you-need-know-about-alternate-data-streams-windows
I have not spent any time using ADS on Windows, but I’ve found Apple’s implementation less than ideal. For example, not all utilities are aware of or work properly with the extended attributes. For example, the version of rsync that comes with OS X does not include full support (rsync 3 fixes this). This probably stems from the fact that not all GNU/BSD utilities support xattr , at least not yet.
They are both approaches at solving the metadata problem. Unfortunately, there seems to be some uncertainty surrounding them (are they going to be here tomorrow?). Ideally I envision a universal filesystem incorporating metadata management that greatly increases interoperability, but this is not realistic within the next 5 years. A more achievable goal would be to either drop the idea completely or fully incorporate the capabilities into the filesystem and APIs with a pluggable translation layer to make moves between filesystems seamless.