June 19, 2012 at 10:34Inferno Part 0: Namespaces
- Transfer
FAQ: What is OS Inferno and why is it needed?
I promised, but have not yet written a post on how to install and run Inferno, and do incredible things; I inflated it to a series of articles, of which this is the first, introductory part. I'm going to talk a little about namespaces. This is a critical part of the OS, and one of those things that make Plan 9 and Inferno very different from what is currently running on your computer.
Inferno is little known, and I do not intend to evangelize her. I want to help narrow the gap between background and practice, to explain how to use Inferno, and why it works the way it works.
I have to stray a little to the side. I'm going to talk about the evolution of namespaces, not quite in chronological order, rather in the order of developing capabilities, and then show how to use the capabilities of Inferno.
Remember DOS? You had A: and B :, your drives. Was C :, your hard drive, etc. You had a directory hierarchy on each of these drives. (We omit unnecessary details.) You can think of it as a flat, upper-level, single-letter, namespace tied to iron. One drive - one letter.
On Unix, however, there was a top-level directory, "/". A root disk was mounted there, and all other disks were mounted in subdirectories of the root directory, in subdirectories of subdirectories - where would you like to see this disk. You did not need to know what partition or disk, unless you were involved in system configuration or one of the partitions did not run out of free space. You can think of it as a hierarchical namespace common to the entire OS.
This is a much more flexible and powerful approach. Programs and users do not need to know what is on which drive.
Later, a new type of file system appeared in Unix: synthetic. For example, / proc. Linux also has / sys, and it’s not uncommon for / dev to also have a synthetic file system. There is no physical disk on which files of these file systems would be stored, but, nevertheless, programs do not need to know anything about it. Files are visible in the namespace just like any other files, and programs do not need to be modified to be able to read or write files on these file systems.
In a sense, there are file systems between them, which, although they store files on disk as usual, the user probably cannot know or prove this. For example, NFS is a remote service containing a hierarchy of files and directories.
Next is FUSEproviding file systems in user space. Using FUSE to interact with the local OS kernel, and implementing what you were going to write the driver for, you can create a service for any synthetic file system - without having to write a real driver for the OS kernel. (I hush up BSD; it seems like there were fewer features from Plan 9, although DragonFly developed some interesting ideas.)
On the other hand, since the namespace is common to the whole system, you need to be root to modify it. Imagine a random user who mounted something on top of / sbin, even ditching the umount command. Custom mounts are a minor exception, in a sense a crutch that doesn’t fundamentally change anything: you can mount the FUSE file system in any _ your_ directory. (By the way, Linux has a driver for 9P2000, the protocol used by Plan 9 and Inferno.)
Previously, the file system could only be connected in one place. Bind mounts on Linux offer a little more flexibility - if the file system (or part of it) is already available somewhere in the namespace, you can connect it to another location through
Plan 9 and Inferno took several large chunks of historical Unix semantics and threw them out the window, threw out some of the key OS assumptions, assumptions that were largely implied and implicit. I'm talking about Inferno now, but almost all of this also applies to (and first appeared in) Plan 9. When I say "in Inferno" you can safely add in brackets "and in Plan 9". These assumptions are not indicated in any particular order; I simply numbered them for convenience.
The first assumption, which was the easiest to break, and perhaps the most obvious in retrospect, is that files and directories in the namespace represent objects on disk.
The second assumption is that there is a one-to-one relationship between file systems and mount points. Those. that this file system is connected only in one place, and that only this one file system is connected to this place. To get rid of this assumption means that the file system and the place where it is connected are independent of each other, and that any number of file systems can be connected to the same place.
The third assumption is that there is a one-to-one relationship between a running system and a namespace. If namespace changes are visible only to the process that changed the namespace and its descendants, there will be no need to protect users from namespace changes made by other users, and the restriction on who can mount what and where will go. It is enough for the user to have read permissions to the file system to mount it.
Throwing away these assumptions about the namespace made it possible to get a much more powerful system, and this difference may not be visible so clearly at first glance, but in fact it is as big as the difference between the DOS and Unix namespace.
These three assumptions were violated to varying degrees in Unix-like operating systems, and to varying degrees of success. The first assumption was successfully broken on Unix. You have / proc, for example, on almost all Unix. FUSE allowed users to mount file systems, even hiding them (not ideally, of course) from other users, which gave a semblance of personal namespace, almost violating the third assumption. Bind mounts made it possible for one part of the namespace to appear in another part, approaching a violation of the second assumption, but requiring root. Union mounts also made their way to Unix, sort of. But, with the exception of the first assumption, all these solutions are not complete. What if we start from the very beginning without carrying Unix luggage?
Since the namespace of each process now has its own (more precisely, any process can, but does not have to, fork the namespace), anyone can mount anything anywhere, or connect (bind) anything anywhere, without creating any problems security. Since the mount point can contain any number of mounted systems, and all will be simultaneously accessible, it becomes possible to overlay one file system on another. Since any file system can be “real” or “synthetic” while it uses the appropriate protocol (9P2000 in the case of Inferno), anything can be a file system and any program that works with files can use it.
Many things, which are either hacks or crooked solutions, suddenly disappear, because they are no longer needed. Obviously, much of what has been delegated to root is no longer needed, just like sudo. chroot is no longer needed. Symbolic links are not needed. A huge number of drivers built into the kernel of the OS are not needed. NSF and / etc / exports are not needed (good riddance).
One of the things that annoys me in the documentation for Inferno (and Plan 9) is that there are scientific articles, man pages, source code, but almost no howto-style textbooks or documents. Maybe I'm spoiled by the Linux that comes with the HOWTO how to make coffee . Getting started is not an easy task, especially if you don’t know enough about the system to use the system to study yourself. You try, but there is an unfamiliar shell that runs unfamiliar commands, and man-pages written as a reference do not really help.
So I can evaluate the document trying to answer the question “Well? I have heard tempting promises before. How can I use this? ” and I'm going to combine all of this with a few specific examples, explaining what is happening.
Remember all these things that are no longer needed? Each of them was replaced by something better, and some of them by several things, each of which is better. I'm going to (for simplicity) ignore authentication in these examples, but I will say that you have a choice of authentication and encryption methods in Inferno.
It is pretty simple:
This mounts the prog file system (analogous to / proc) in / n / prog. By the way, you can manage, debug, and do everything that is usually with running processes through the / prog file system, just like in Unix. The difference is that if you locally mount / prog the remote machine, you can do all these things with the remote processes without any ssh and even without the debugger running on the remote machine. Yes, you can execute locally
I have a bunch of .iso files on my computer. Suppose they are in / n / cds, and I want to connect one of them:
Now the current process and its descendants can see the contents of this CD. Nothing special is required; 9660srv (4) knows how to read the ISO-9660 file system on the one hand and supports the 9P2000 protocol on the other.
Inferno also comes bundled with, for example, tarfs (4) , which treats the tar archive as a read-only file system.
Well, most. Those that do not just export the file system, and finally you can forget about ioctl (). The drivers provided by the kernel can be seen in the / dev / drivers file, and this is a rather short list, shorter than the output
Where you would need a kernel driver to support the new file system on most operating systems, Inferno does not need modprobe (and thus root) and does not touch the kernel to add a new file system. You can get ext2 (read-only) support by downloading ext2fs , just like the kernel did not need to know about ISO-9660 in the example above.
Since a process can create a namespace for itself, anything that might be dangerous to show to an untrusted process can be unmounted from the namespace, and the pctl (2) system call can be used to prevent the process from mounting drivers. An extreme example that everything will unmount (and therefore cannot do anything other than using the built-in shell functionality):
Of course, you need to act a little more carefully than just unmount everything if you want something useful, but the idea itself is simple.
You can also make a temporary sandbox if you want by mounting the file system in memory on top of the root directory:
And, of course, / asdf will not appear anywhere. (It should be noted that this only applies to creating new files.)
bind solves this problem by providing the ability to put some directories on top of others. For example, I like the tinytk icon set more than the default icons, so I override the default icons with them before starting the window manager:
You can even put one file on top of another:
Since symlinks are unnecessary and absent in Inferno, the problem of circular references in the file system almost disappears.
I hate messing around with $ PATH. The big chunks of my .bashrc are about figuring out who goes first, and not adding redundant paths or paths that point to binaries for other architectures. I have Plan 9 Port installed , but I don’t want the Plan 9 man version to conflict with the Linux version, for example.
In Inferno, everything you want to run is in / dis (/ bin in Plan 9). If you want to run programs from another place, you just plug it in before or after / dis.
So if you want to install something, but cannot write to / dis, install it wherever you want, and plug it on top of / dis. sudo is not required. The same goes for / lib and / module.
I'm going to ignore authentication (-A flag) and encryption, so we will export something not important: again memfs (4) . First car:
Second car:
Since most file server commands can work via stdin / stdout instead of connecting to a mount point, the file system does not even need to be mounted locally to make it accessible over the network:
And on the second machine:
The listen (1) and dial (1) commands are just handy wrappers for interacting with the / net file system, which is the network interface for Inferno. Unix sockets are handled by the BSD sockets library (with corresponding system calls) and are much more difficult to use. You can think of listen and dial as a netcat counterpart. styxlisten is the same, it just expects to transmit the 9P2000 protocol (also called Styx until it was accepted in Plan 9) instead of a byte stream.
All this is achieved with a few short commands, which in themselves are just wrappers around the Inferno system calls. You have bind (1), mount (1), and unmount (1) . If you need introspection, ns (1) will help you .
Representation of all OS objects as file systems means that programs like cat, sed and echo can be used to, for example, access the network using the ip (3) file system . ftpfs (4) allows you to use ls and cp without an interactive client, which simplifies the writing of scripts (there is no need for redirection via FTP client). In fact, your scripts generally do not need to know that they work with an FTP server and not with a regular file on disk. zipfsallows you to mount a .zip file. By combining them, you can mount .iso, which is inside .zip, which is on the FTP server, and then copy individual files from it.
We are already at an interesting point. Most of the population owns at least a laptop and a smartphone, and, of them, most of them also own a home computer, have a computer at work, a tablet, etc. In the heyday of Unix, there was one computer and many users. This seems obvious, but we are quickly approaching the point where a common case will be many computers for one user and not vice versa.
Managing all these resources (disk and CPU, in particular) is already a big problem. File Sync, Contact Sync, etc. If your phone does not have enough resources to process something, you need to go to a regular computer (physically or via ssh). If you want to watch a movie on your phone, you need to copy it there. What do you do if you read an article on your phone and decide to continue reading on your laptop? Send yourself an email or IM? (This is still better than DBUS, but this is a digression.)
I don’t think it will continue that way. As a temporary solution, you can give all your data to Google, Facebook, or any “cloud” service, but such a solution leaves much to be desired (and listing the reasons is another digression).
A software solution to the problem of managing computing resources and combining all of these machines (local and remote) is inevitable. At least at the moment, I see Inferno as the best candidate for the role of a platform that will make all this possible. If this does not fully explain why I suddenly fell in love with this system, then at least it does it partially.
This description is very superficial. “Everything is a file system” is just one of the great ideas Inferno builds on.
I promised, but have not yet written a post on how to install and run Inferno, and do incredible things; I inflated it to a series of articles, of which this is the first, introductory part. I'm going to talk a little about namespaces. This is a critical part of the OS, and one of those things that make Plan 9 and Inferno very different from what is currently running on your computer.
Inferno is little known, and I do not intend to evangelize her. I want to help narrow the gap between background and practice, to explain how to use Inferno, and why it works the way it works.
I have to stray a little to the side. I'm going to talk about the evolution of namespaces, not quite in chronological order, rather in the order of developing capabilities, and then show how to use the capabilities of Inferno.
Content
- Dos
- Unix, Linux
- Plan 9, Inferno
- … Well?
- Finally, something you can type
- Powerful primitives mean you'll never regret
- Somewhere in the future
- To be continued
Dos
Remember DOS? You had A: and B :, your drives. Was C :, your hard drive, etc. You had a directory hierarchy on each of these drives. (We omit unnecessary details.) You can think of it as a flat, upper-level, single-letter, namespace tied to iron. One drive - one letter.
Unix, Linux
On Unix, however, there was a top-level directory, "/". A root disk was mounted there, and all other disks were mounted in subdirectories of the root directory, in subdirectories of subdirectories - where would you like to see this disk. You did not need to know what partition or disk, unless you were involved in system configuration or one of the partitions did not run out of free space. You can think of it as a hierarchical namespace common to the entire OS.
This is a much more flexible and powerful approach. Programs and users do not need to know what is on which drive.
Later, a new type of file system appeared in Unix: synthetic. For example, / proc. Linux also has / sys, and it’s not uncommon for / dev to also have a synthetic file system. There is no physical disk on which files of these file systems would be stored, but, nevertheless, programs do not need to know anything about it. Files are visible in the namespace just like any other files, and programs do not need to be modified to be able to read or write files on these file systems.
In a sense, there are file systems between them, which, although they store files on disk as usual, the user probably cannot know or prove this. For example, NFS is a remote service containing a hierarchy of files and directories.
Next is FUSEproviding file systems in user space. Using FUSE to interact with the local OS kernel, and implementing what you were going to write the driver for, you can create a service for any synthetic file system - without having to write a real driver for the OS kernel. (I hush up BSD; it seems like there were fewer features from Plan 9, although DragonFly developed some interesting ideas.)
On the other hand, since the namespace is common to the whole system, you need to be root to modify it. Imagine a random user who mounted something on top of / sbin, even ditching the umount command. Custom mounts are a minor exception, in a sense a crutch that doesn’t fundamentally change anything: you can mount the FUSE file system in any _ your_ directory. (By the way, Linux has a driver for 9P2000, the protocol used by Plan 9 and Inferno.)
Previously, the file system could only be connected in one place. Bind mounts on Linux offer a little more flexibility - if the file system (or part of it) is already available somewhere in the namespace, you can connect it to another location through
mount --bind. This is a relatively recent addition, and before the 2.6.26 kernel it was impossible to mount the same file system in one place in read-write mode and at the same time in another in read-only mode.Plan 9, Inferno
Plan 9 and Inferno took several large chunks of historical Unix semantics and threw them out the window, threw out some of the key OS assumptions, assumptions that were largely implied and implicit. I'm talking about Inferno now, but almost all of this also applies to (and first appeared in) Plan 9. When I say "in Inferno" you can safely add in brackets "and in Plan 9". These assumptions are not indicated in any particular order; I simply numbered them for convenience.
The first assumption, which was the easiest to break, and perhaps the most obvious in retrospect, is that files and directories in the namespace represent objects on disk.
The second assumption is that there is a one-to-one relationship between file systems and mount points. Those. that this file system is connected only in one place, and that only this one file system is connected to this place. To get rid of this assumption means that the file system and the place where it is connected are independent of each other, and that any number of file systems can be connected to the same place.
The third assumption is that there is a one-to-one relationship between a running system and a namespace. If namespace changes are visible only to the process that changed the namespace and its descendants, there will be no need to protect users from namespace changes made by other users, and the restriction on who can mount what and where will go. It is enough for the user to have read permissions to the file system to mount it.
Throwing away these assumptions about the namespace made it possible to get a much more powerful system, and this difference may not be visible so clearly at first glance, but in fact it is as big as the difference between the DOS and Unix namespace.
These three assumptions were violated to varying degrees in Unix-like operating systems, and to varying degrees of success. The first assumption was successfully broken on Unix. You have / proc, for example, on almost all Unix. FUSE allowed users to mount file systems, even hiding them (not ideally, of course) from other users, which gave a semblance of personal namespace, almost violating the third assumption. Bind mounts made it possible for one part of the namespace to appear in another part, approaching a violation of the second assumption, but requiring root. Union mounts also made their way to Unix, sort of. But, with the exception of the first assumption, all these solutions are not complete. What if we start from the very beginning without carrying Unix luggage?
Since the namespace of each process now has its own (more precisely, any process can, but does not have to, fork the namespace), anyone can mount anything anywhere, or connect (bind) anything anywhere, without creating any problems security. Since the mount point can contain any number of mounted systems, and all will be simultaneously accessible, it becomes possible to overlay one file system on another. Since any file system can be “real” or “synthetic” while it uses the appropriate protocol (9P2000 in the case of Inferno), anything can be a file system and any program that works with files can use it.
Many things, which are either hacks or crooked solutions, suddenly disappear, because they are no longer needed. Obviously, much of what has been delegated to root is no longer needed, just like sudo. chroot is no longer needed. Symbolic links are not needed. A huge number of drivers built into the kernel of the OS are not needed. NSF and / etc / exports are not needed (good riddance).
mount -o loopnot needed. mount --bindnot needed. FUSE is not needed. Even $ PATH is no longer needed, and with it the need to be root to install software.… Well?
One of the things that annoys me in the documentation for Inferno (and Plan 9) is that there are scientific articles, man pages, source code, but almost no howto-style textbooks or documents. Maybe I'm spoiled by the Linux that comes with the HOWTO how to make coffee . Getting started is not an easy task, especially if you don’t know enough about the system to use the system to study yourself. You try, but there is an unfamiliar shell that runs unfamiliar commands, and man-pages written as a reference do not really help.
So I can evaluate the document trying to answer the question “Well? I have heard tempting promises before. How can I use this? ” and I'm going to combine all of this with a few specific examples, explaining what is happening.
Finally, something you can type
Remember all these things that are no longer needed? Each of them was replaced by something better, and some of them by several things, each of which is better. I'm going to (for simplicity) ignore authentication in these examples, but I will say that you have a choice of authentication and encryption methods in Inferno.
root is not needed to mount
It is pretty simple:
bind '#p' /n/prog
This mounts the prog file system (analogous to / proc) in / n / prog. By the way, you can manage, debug, and do everything that is usually with running processes through the / prog file system, just like in Unix. The difference is that if you locally mount / prog the remote machine, you can do all these things with the remote processes without any ssh and even without the debugger running on the remote machine. Yes, you can execute locally
kill 15and this will kill process 15 on the remote machine.mount -o loop and FUSE are not needed
I have a bunch of .iso files on my computer. Suppose they are in / n / cds, and I want to connect one of them:
; 9660srv /n/cds/dfly-i386-3.0.2_REL.iso /n/dfly
; ls /n/dfly
Now the current process and its descendants can see the contents of this CD. Nothing special is required; 9660srv (4) knows how to read the ISO-9660 file system on the one hand and supports the 9P2000 protocol on the other.
Inferno also comes bundled with, for example, tarfs (4) , which treats the tar archive as a read-only file system.
No kernel drivers needed
Well, most. Those that do not just export the file system, and finally you can forget about ioctl (). The drivers provided by the kernel can be seen in the / dev / drivers file, and this is a rather short list, shorter than the output
lsmodon a typical Ubuntu machine or even Arch. Where you would need a kernel driver to support the new file system on most operating systems, Inferno does not need modprobe (and thus root) and does not touch the kernel to add a new file system. You can get ext2 (read-only) support by downloading ext2fs , just like the kernel did not need to know about ISO-9660 in the example above.
chroot is not needed
Since a process can create a namespace for itself, anything that might be dangerous to show to an untrusted process can be unmounted from the namespace, and the pctl (2) system call can be used to prevent the process from mounting drivers. An extreme example that everything will unmount (and therefore cannot do anything other than using the built-in shell functionality):
; load std
; pctl newns
; pctl nodevs
; unmount /
; ls /
sh: ls: './ls' file does not exist
Of course, you need to act a little more carefully than just unmount everything if you want something useful, but the idea itself is simple.
You can also make a temporary sandbox if you want by mounting the file system in memory on top of the root directory:
; mount -bc {memfs -s} /
; echo Mad with power! > /asdf
And, of course, / asdf will not appear anywhere. (It should be noted that this only applies to creating new files.)
Simlink not needed
bind solves this problem by providing the ability to put some directories on top of others. For example, I like the tinytk icon set more than the default icons, so I override the default icons with them before starting the window manager:
; bind -bc /icons/tinytk /icons/tk
You can even put one file on top of another:
; echo Hello > some-file
; bind some-file /NOTICE
; cat /NOTICE
Hello
Since symlinks are unnecessary and absent in Inferno, the problem of circular references in the file system almost disappears.
$ PATH, $ LDPATH and root privileges are not needed to install software
I hate messing around with $ PATH. The big chunks of my .bashrc are about figuring out who goes first, and not adding redundant paths or paths that point to binaries for other architectures. I have Plan 9 Port installed , but I don’t want the Plan 9 man version to conflict with the Linux version, for example.
In Inferno, everything you want to run is in / dis (/ bin in Plan 9). If you want to run programs from another place, you just plug it in before or after / dis.
; bind -bc $home/dis /dis
So if you want to install something, but cannot write to / dis, install it wherever you want, and plug it on top of / dis. sudo is not required. The same goes for / lib and / module.
NFS is not needed
I'm going to ignore authentication (-A flag) and encryption, so we will export something not important: again memfs (4) . First car:
; mount -c {memfs -s} /n/mem
; echo something > /n/mem/asdf
; listen -A 'tcp!*!1234' { export /n/mem & }
Second car:
; mount -A 'tcp!thefirstmachine!1234' /n/remotememfs
; cat /n/remotememfs/asdf
something
Since most file server commands can work via stdin / stdout instead of connecting to a mount point, the file system does not even need to be mounted locally to make it accessible over the network:
; styxlisten -A 'tcp!*!1234' {memfs -s}
And on the second machine:
; mount -Ac 'tcp!thefirstmachine!1234' /n/remotememfs
The listen (1) and dial (1) commands are just handy wrappers for interacting with the / net file system, which is the network interface for Inferno. Unix sockets are handled by the BSD sockets library (with corresponding system calls) and are much more difficult to use. You can think of listen and dial as a netcat counterpart. styxlisten is the same, it just expects to transmit the 9P2000 protocol (also called Styx until it was accepted in Plan 9) instead of a byte stream.
Powerful primitives mean you'll never regret
All this is achieved with a few short commands, which in themselves are just wrappers around the Inferno system calls. You have bind (1), mount (1), and unmount (1) . If you need introspection, ns (1) will help you .
Representation of all OS objects as file systems means that programs like cat, sed and echo can be used to, for example, access the network using the ip (3) file system . ftpfs (4) allows you to use ls and cp without an interactive client, which simplifies the writing of scripts (there is no need for redirection via FTP client). In fact, your scripts generally do not need to know that they work with an FTP server and not with a regular file on disk. zipfsallows you to mount a .zip file. By combining them, you can mount .iso, which is inside .zip, which is on the FTP server, and then copy individual files from it.
Somewhere in the future
We are already at an interesting point. Most of the population owns at least a laptop and a smartphone, and, of them, most of them also own a home computer, have a computer at work, a tablet, etc. In the heyday of Unix, there was one computer and many users. This seems obvious, but we are quickly approaching the point where a common case will be many computers for one user and not vice versa.
Managing all these resources (disk and CPU, in particular) is already a big problem. File Sync, Contact Sync, etc. If your phone does not have enough resources to process something, you need to go to a regular computer (physically or via ssh). If you want to watch a movie on your phone, you need to copy it there. What do you do if you read an article on your phone and decide to continue reading on your laptop? Send yourself an email or IM? (This is still better than DBUS, but this is a digression.)
I don’t think it will continue that way. As a temporary solution, you can give all your data to Google, Facebook, or any “cloud” service, but such a solution leaves much to be desired (and listing the reasons is another digression).
A software solution to the problem of managing computing resources and combining all of these machines (local and remote) is inevitable. At least at the moment, I see Inferno as the best candidate for the role of a platform that will make all this possible. If this does not fully explain why I suddenly fell in love with this system, then at least it does it partially.
To be continued
This description is very superficial. “Everything is a file system” is just one of the great ideas Inferno builds on.