March 14, 2019 at 13:13How the browser helps Comrade Major
You know, when I almost accidentally discovered, reading Dmitry Ketov’s excellent book “The Linux Internal Device” (and this is not an advertisement), that every file we download from the Internet with the light of the browser leaves it in the extended file attributes (in the inode of the file, not in the file itself ) at least a full link to where it was downloaded from, I got a little excited and wanted to immediately rush to save the world. Later, having studied this question a little, reading the “commits” and “comments” that said that this feature, which has been built into the GNU / Linux kernel, has been at least 10 years old, that it has been used not only by browsers, but also, for example, by the popular wget utility(curl not), and that this thing is considered almost the norm in Linux (and, as it turned out later, in MacOS), I was a little reassured. One thing haunted me: a search for the getfattr keyword on google on the Habr site:
Disclaimer * . Due to lack of time I will add as many letters as possible. So far, only specifics.
Yes, each downloaded file using any Chromium-compatible browser (checked in the lxc-container on Chrome, Chromium, Yandex and the latest Opera) writes in the extended file attributes hidden in the bowels of the file system, here is the following conclusion:
However, they all honestly do not write anything if you download the file in the "incognito" / "private browsing" mode
Did you know about this? If yes, then good. If not, now you will know. As for Firefox and Palemoon , these guys honestly ignore this help to fellow major. I have not tested the whole bunch of programs for browsing the Internet, but I think everything that is based on Chromium does this, most likely not.
Now about MacOS and Windows. In makoshi, this function also works in Chrome:
... she works in Safari. I haven’t tested it in FF, but I believe that the pretty-hearted Firefox for MacOS is also deprived of this “privilege” as it is on Linux.
It is alleged that this feature is used to determine the file "downloaded from the Internet" and it is precisely by the presence of this attribute in the "makoshi" that a warning about the danger of launching pops up, etc. Hmm ... Strange ... But here we will smoothly move to Windows.
"Venda" thing is certainly mediocre and watching everyone with overt cynicism. (What are non-disconnectable updates in W10!). In it, in NTFS, loopholes are also hidden for writing anything to extended file attributes. They are called streams.and they can all do the same. They say that in these “streams” hidden by the file system they periodically like to write something and write viruses, since all other programs rarely use them. I can’t say anything, I haven’t been using Windows for the past 10 years. But still. At first approximation in Win10, I did not find such a “trifle”, namely, the fact that Google Chrome wrote links to NTFS advanced attributes in the same way as it does on Linux and MacOS. I’ll make a reservation: this is a first approximation, since it is argued that streams have several layers , as it were, and not all of them are directly accessible.
So, in short, the situation is with how not only browsers are arranged, but also file systems with an extremely brief look.
I will still edit this article. But in advance I ask you to refuse comments in the form: "So what, I have nothing to hide." In general, we all, as a rule, also. But the most zealous “we have nothing to hide”, if there are any, I can advise you to turn on the built-in camera and microphone, and give a public link here in the comments, so that we all can see this. A joke of course.
These are the jokes.
UPD *
So. What turns out, thanks to your numerous comments and additional study of the material:
In all three operating systems: Linux, MacOS and Windows, browsers, primarily Chrome and Chromium-based, write the path to the source of the downloaded files at the file system level.
On Linux, Google Chrome-based browsers write a full link to the file’s original location in the fields:
this is done using the Linux kernel feature, which has appeared in it since 2002 in the form of extended file attributes under the general name xattr [ Wikipedia ], which are available in almost all popular FS: ext2, ext3, ext4, Squashfs, ReiserFS, XFS, Btrfs, ZFS. With kernel options or mount flags, support for extended attributes can be disabled, but it is used for containerization (attributes for namespaces), SELinux, and is used by Xs in some cases with the freedesktop.org easy hand (for example, to determine the MIME type of a file) [ link ].
These fields are used by some programs. Like, for example, the Chrome, Chromium, Yandex, Opera browsers mentioned in the article, in some cases Firefox and the wget console utility , as well as, judging by Wikipedia: curl, Beagle, OpenStack Swift, Dropbox, KDE. According to statements in the comments, wget does not write the latest version 1.20.1 already (I have 1.19 - it writes, it also does not write curl for me, but judging by the wiki - it should). Apparently, among developers, a discussion periodically takes place to include or not to include such a feature in the next release). In Firefox for Linux, as I checked, nothing is saved, but for MacOS and Windows, as others checked - yes. It is possible that this is somehow related to the policies of the OS itself from Apple and Microsoft.
In general, there are no restrictions on creating and managing extended attribute fields in Linux. You yourself can create and write in the advanced attributes the information you want. At the kernel level, the restrictions are: 255 bytes for the name and up to 64KB for the field value. On some file systems they are (XFS and ReiserFS), on others they are smaller (ext2 / 3/4 and btrfs).
A similar situation on MacOS and Windows with their FS HFS + and NTFS . The only difference is that such attributes are called, hidden and shown in different ways. In Ubuntu 18.10, you will not see them in the default file manager
victoriously :
In the comments, there are many options for avoiding recording information or deleting links to files for a particular operating system. For example, for Linux, I liked the simple idea of mounting it in a folder where files will be downloaded by the browser on another file system - without the flag for writing extended attributes by default. For example, in ReiserFS -> the original comment
The idea of disabling attributes in general at the stage of mounting in / etc / fstab
firstly, they will not cancel existing fields, but the main thing, as it seems to me, is fraught with disruption of a number of services (SELinux, LXC). (-> full comment )
The easiest way is to clean the command / script in the console or in powershell:
Linux:
original
macOS comment :
original
Windows comment :
original comment
General summary:
The possibility of assigning any fields to the file and filling them with values that appeared quite a long time ago (in Linux since 2002) began to be implemented in the form of recording the full path to the download source quite recently. Somewhere after 2015-2016. So, for example, I have had files on my Mac since 2015 downloaded from the Web and (version 10.6.8) and I have not found any links. Someone from the commentators checked on previous versions of Debian (8-pk) and also did not find anything. link to comment .
When it appeared on Windows - there is no strength to find out, but judging by this comment, the function was "brought" not only in W10 and W7, but even in W_XP;)
I repeat a simple thought: It is strange not what it is, it is strange that it appeared quietly and almost imperceptibly everywhere. That is why, in the latest version of the utility for downloading files in Linux wget, the former function of saving links was removed there:
link to comment
For, it seems to me, such issues should be controlled and decided by us users.
Phew
site:https://habr.com getfattras well as on ENT: site:https://linux.org.ru getfattr did not really give anything. “So,” I thought, “this topic has not been particularly discussed yet.” - Well, then it's time to discuss it, but first I had to study a little. And here is what I found: Disclaimer * . Due to lack of time I will add as many letters as possible. So far, only specifics.
Yes, each downloaded file using any Chromium-compatible browser (checked in the lxc-container on Chrome, Chromium, Yandex and the latest Opera) writes in the extended file attributes hidden in the bowels of the file system, here is the following conclusion:
i@ars:~$ getfattr -d logo.png
# file: logo.png
user.xdg.origin.url="https://habr.com/images/logo.png"
user.xdg.referrer.url="https://habr.com/images/logo.png"However, they all honestly do not write anything if you download the file in the "incognito" / "private browsing" mode
Did you know about this? If yes, then good. If not, now you will know. As for Firefox and Palemoon , these guys honestly ignore this help to fellow major. I have not tested the whole bunch of programs for browsing the Internet, but I think everything that is based on Chromium does this, most likely not.
Now about MacOS and Windows. In makoshi, this function also works in Chrome:
iMac-Igor:~ [censored]$ xattr -l habr_logo.png
com.apple.metadata:kMDItemWhereFroms:
00000000 62 70 6C 69 73 74 30 30 A2 01 02 5F 10 20 68 74 |bplist00..._. ht|
00000010 74 70 73 3A 2F 2F 68 61 62 72 2E 63 6F 6D 2F 69 |tps://habr.com/i|
00000020 6D 61 67 65 73 2F 6C 6F 67 6F 2E 70 6E 67 5F 10 |mages/logo.png_.|
00000030 20 68 74 74 70 73 3A 2F 2F 68 61 62 72 2E 63 6F | https://habr.co|
00000040 6D 2F 69 6D 61 67 65 73 2F 6C 6F 67 6F 2E 70 6E |m/images/logo.pn|
00000050 67 08 0B 2E 00 00 00 00 00 00 01 01 00 00 00 00 |g...............|
00000060 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 51 |...Q|
00000074
com.apple.quarantine: 0001;5c8a21e7;Google Chrome;26D6C537-E6B0-4715-9E77-656FF1C5B7A9... she works in Safari. I haven’t tested it in FF, but I believe that the pretty-hearted Firefox for MacOS is also deprived of this “privilege” as it is on Linux.
It is alleged that this feature is used to determine the file "downloaded from the Internet" and it is precisely by the presence of this attribute in the "makoshi" that a warning about the danger of launching pops up, etc. Hmm ... Strange ... But here we will smoothly move to Windows.
"Venda" thing is certainly mediocre and watching everyone with overt cynicism. (What are non-disconnectable updates in W10!). In it, in NTFS, loopholes are also hidden for writing anything to extended file attributes. They are called streams.and they can all do the same. They say that in these “streams” hidden by the file system they periodically like to write something and write viruses, since all other programs rarely use them. I can’t say anything, I haven’t been using Windows for the past 10 years. But still. At first approximation in Win10, I did not find such a “trifle”, namely, the fact that Google Chrome wrote links to NTFS advanced attributes in the same way as it does on Linux and MacOS. I’ll make a reservation: this is a first approximation, since it is argued that streams have several layers , as it were, and not all of them are directly accessible.
So, in short, the situation is with how not only browsers are arranged, but also file systems with an extremely brief look.
I will still edit this article. But in advance I ask you to refuse comments in the form: "So what, I have nothing to hide." In general, we all, as a rule, also. But the most zealous “we have nothing to hide”, if there are any, I can advise you to turn on the built-in camera and microphone, and give a public link here in the comments, so that we all can see this. A joke of course.
These are the jokes.
UPD *
So. What turns out, thanks to your numerous comments and additional study of the material:
In all three operating systems: Linux, MacOS and Windows, browsers, primarily Chrome and Chromium-based, write the path to the source of the downloaded files at the file system level.
On Linux, Google Chrome-based browsers write a full link to the file’s original location in the fields:
user.xdg.origin.urland user.xdg.referrer.urlthis is done using the Linux kernel feature, which has appeared in it since 2002 in the form of extended file attributes under the general name xattr [ Wikipedia ], which are available in almost all popular FS: ext2, ext3, ext4, Squashfs, ReiserFS, XFS, Btrfs, ZFS. With kernel options or mount flags, support for extended attributes can be disabled, but it is used for containerization (attributes for namespaces), SELinux, and is used by Xs in some cases with the freedesktop.org easy hand (for example, to determine the MIME type of a file) [ link ].
These fields are used by some programs. Like, for example, the Chrome, Chromium, Yandex, Opera browsers mentioned in the article, in some cases Firefox and the wget console utility , as well as, judging by Wikipedia: curl, Beagle, OpenStack Swift, Dropbox, KDE. According to statements in the comments, wget does not write the latest version 1.20.1 already (I have 1.19 - it writes, it also does not write curl for me, but judging by the wiki - it should). Apparently, among developers, a discussion periodically takes place to include or not to include such a feature in the next release). In Firefox for Linux, as I checked, nothing is saved, but for MacOS and Windows, as others checked - yes. It is possible that this is somehow related to the policies of the OS itself from Apple and Microsoft.
In general, there are no restrictions on creating and managing extended attribute fields in Linux. You yourself can create and write in the advanced attributes the information you want. At the kernel level, the restrictions are: 255 bytes for the name and up to 64KB for the field value. On some file systems they are (XFS and ReiserFS), on others they are smaller (ext2 / 3/4 and btrfs).
A similar situation on MacOS and Windows with their FS HFS + and NTFS . The only difference is that such attributes are called, hidden and shown in different ways. In Ubuntu 18.10, you will not see them in the default file manager
Nautilusas well as in the console until you install a package attrthat includes utr viewing and setting utilities (getfattr and setfattr, respectively). You can see on the Mac either in Finderby cmd + I, or using the built-in xattr. On Windows, they say, in the properties of a file, only the flag of the presence of a hidden link in the file is visible (hands did not reach to check where), either through powershell or the program streams : victoriously :
Здесь некто советует помимо powershell использовать программу streams. Субъективно удобнее.
streams [-s] [-d] <файл или каталог>In the comments, there are many options for avoiding recording information or deleting links to files for a particular operating system. For example, for Linux, I liked the simple idea of mounting it in a folder where files will be downloaded by the browser on another file system - without the flag for writing extended attributes by default. For example, in ReiserFS -> the original comment
The idea of disabling attributes in general at the stage of mounting in / etc / fstab
mount -o nouser_xattrfirstly, they will not cancel existing fields, but the main thing, as it seems to me, is fraught with disruption of a number of services (SELinux, LXC). (-> full comment )
The easiest way is to clean the command / script in the console or in powershell:
Linux:
setfattr -hx user.xdg.origin.url имя_файла
original
macOS comment :
xattr -c -r ~/Downloadsoriginal
Windows comment :
get-childitem «D:\Downloads\» | unblock-fileoriginal comment
General summary:
The possibility of assigning any fields to the file and filling them with values that appeared quite a long time ago (in Linux since 2002) began to be implemented in the form of recording the full path to the download source quite recently. Somewhere after 2015-2016. So, for example, I have had files on my Mac since 2015 downloaded from the Web and (version 10.6.8) and I have not found any links. Someone from the commentators checked on previous versions of Debian (8-pk) and also did not find anything. link to comment .
When it appeared on Windows - there is no strength to find out, but judging by this comment, the function was "brought" not only in W10 and W7, but even in W_XP;)
I repeat a simple thought: It is strange not what it is, it is strange that it appeared quietly and almost imperceptibly everywhere. That is why, in the latest version of the utility for downloading files in Linux wget, the former function of saving links was removed there:
* Changes in Wget 1.20.1
** --xattr is no longer default since it introduces privacy issues.
link to comment
For, it seems to me, such issues should be controlled and decided by us users.
Phew
UPDATE 2
In Chromium under Linux, saving user.xdg.origin.url and user.xdg.referrer.url to xattrs will no longer be done.
chromium-review.googlesource.com/c/chromium/src/+/1407441