November 22, 2012 at 14:29Philipp Torchinsky of Semonix: installing SmartOS and using Node.js on it
On November 29, at the Yandex office in St. Petersburg, I will conduct a scientific seminar “DTrace - Verification Work for Your Code”. Those at YaC 2012 listened to my talk on the Illumos Kernel Cloud Computing Infrastructure — and many others — know that at Semonix, I work on cloud technologies that are closely related to SmartOS. At the seminar I will tell you how to use DTrace technology to conduct an in-depth performance analysis and study in detail the operation of the application. A search in Habré finds only one article about SmartOS , and so that more people who are already familiar with it come to the seminar, I decided to write in advance about installing SmartOS and using Node.js on it.
First, I’ll tell you what SmartOS is for. In short, its purpose is to be a host system for virtual machines. It is often used as the basis for public and private clouds, such as the Joyent and MITAC cloud services. LinkedIn uses the Joyent service: all of its mobile server part is made on Node.js, which runs in the Joyent cloud. We wrote about this in detail on the Semonix blog in articles about SmartOS and on the cloud based on illumos , and I talked about in a talk at YaC 2012 that I already mentioned.
Typical applications of SmartOS are data center systems, systems for developing and operating a loaded web application, and a platform for creating a private or public cloud. When working on this article, I actively usedan article by Stu Rudnage , who walked about the same SmartOS installation path as I did, but using VMware instead of VirtualBox.
Let's say you are one of those who are interested in trying SmartOS. How is it easier to do? I went along the simplest way, which would prevent Windows or Linux from running on a virtual machine, but would allow using the ready-made SmartOS + Node.js. I will launch SmartOS in VirtualBox. It is clear that KVM in a virtual machine for VirtualBox will not work, and the rest should earn a bang.
SmartOS can be a host system for guests running Windows, Linux, and SmartOS itself. In this article, I only consider the option with guest systems running SmartOS. They implement lightweight virtualization (the same as zones in Solaris and something similar to cells in FreeBSD), and therefore, they do not require KVM, which is only supported on physical computers. The entire configuration is shown in the figure below. SmartOS (host) is a system that is a guest in relation to Windows on my laptop, and a host - in relation to SmartOS systems running in zones inside this SmartOS.

We will install SmartOS in a virtual machine with two network interfaces. What for? Very simple: I want to connect a regular ssh to SmartOS from my host system (in this case, Windows) so that I can open several windows - at least one for each guest system inside my SmartOS.
Why not use the graphical mode in the console and run the right amount of xterm or gnome-terminal there? The fact is that the graphics ("X" - X) are not installed in SmartOS. There is a project to port X11 support to SmartOS, but it has not yet been completed, and Jonathan Perkin, the lead volunteer for this project, informed me that the deadline for working on it is unknown. Therefore, if I want to open five windows from ssh to my SmartOS, then I need TCP access from my host system to SmartOS in VirtualBox.
In addition, VirtualBox Guest Additions cannot be installed in the SmartOS guest machine, allowing cut-and-paste between the guest system and the host system in which I write the article (and I want to copy texts from the terminal into it). Theoretically, I think you can install Guest Additions (you basically need to install kernel modules there), but for Solaris-like systems Guest Additions comes in the form of a .pkg package, which is not supported in SmartOS. I decided that there would be more trouble with unpacking the package and manually screwing the module to the kernel than with ssh access from the host system.
Why do we need a second interface? So SmartOS can go through NAT for packets on the Internet. This restriction on the conditions of the task is imposed by VirtualBox - to make a virtual machine with NAT / DHCP accessible from the host system in it otherwise it is only possible through the bridged network, and this is not always convenient. A discussion of cases where it is not very convenient (or not supported on certain host systems or versions of VirtualBox) is beyond the scope of this article, but believe me - this happens. The simplest options are: the DHCP addresses on the network where the physical computer is connected to you have run out, or you do not want to look at each time what address this time was issued to the system before ssh hooked to it.
Configuring the virtual machine so that it has 2 network adapters is simple: on the Network tab, you just need to indicate that not only the first adapter is active in host adapter mode, but the second in NAT mode (see the figure below). Please note that for SmartOS it makes sense to choose the type of Solaris 10 virtual machine 10/09 or later (64 bit). The Solaris 11 option may be better. It worked without problems and delays.
You must attach the latest.iso file to the virtual machine’s CD-ROM - SmartOS will boot from it.

After booting for the first time, the system will ask a few questions about the settings (in particular, the root password), format the disk, create a ZFS pool on the disk and write the settings to disk. After rebooting, you should see something similar to the picture below.

Log in as root using the password that you set earlier. Already now you can connect to SmartOS using ssh, for this you just need to find out what server address is:
The system configuration in SmartOS is stored in the / usbkey / config file. The / usbkey file system is mounted from disk, unlike / etc, which is always loaded from a LiveCD or LiveUSB and mounted on a RAM disk. Therefore, the changes made to files from / etc will be lost upon reboot, and in / usbkey they will be saved.
The file / usbkey / config is created the first time SmartOS is started on the computer, and by default it describes only the first interface in the system. In order to see the second one, you need to slightly modify the file. To modify it, we need to find out the MAC addresses of the interfaces. To do this, you can use the command
Now let's change the / usbkey / config file to look like this:
First you need to download the image of the virtual machine. To do this, update the list of images:
Now let's see what ready-made images are in the repository:
Now we have a list of images available for installation. Here is only part of the list - it is much longer. At the time of writing, 96 images were available in the repository. We are interested in the base64 image - a minimal system that includes node.js. Currently, the latest version of this image is 1.8.1 (includes node 0.8.9), and its UUID is 60a3b1fa-0674-11e2-abf5-cb82934a8e24.
Download the image of the virtual machine:
Now let's make a description of the virtual machine, which we will create on the basis of the just loaded image and run it inside the SmartOS host.
The description is made in the form of a JSON file (the details of the process are described in the documentation on smartos.org, I’ll just put a working example here - let it be in /zones/template.json for us). The file should be in the file system, which is located on the disk and saved between reboots, most conveniently in / zones.
Pay attention to dataset_uuid - this is the same UUID that belongs to the image just received from the repository. There will also be two network interfaces in the zone - “internal” and “external”. This setting is used in data centers in real life: an internal network is connected to the internal interface, in which data center administrators work, and an external network is connected to the Internet, where clients come from.
Now we are creating a new SmartOS virtual machine (for brevity, we will continue to call it “zone”):
If everything turned out fine, you can see the launched new zone by giving the command
Now you can connect to the new zone either from the SmartOS console or via ssh:
Please note: The UUID of a zone is different from the UUID of a zone image, because you can create many zones from the same image. At the same time, up to 8191 virtual machines can be launched in one SmartOS host system.
To connect to the zone via ssh, you need to clarify which address is given to it; for this, you first need to join it through zlogin, and then find out the address:
Now you can install any packages we need in the zone. To do this, you need to update the information about available packages using pkgin update , get their list pkgin list or search for what is required using pkgin search . To install the package, use pkgin install .
Suppose you were not familiar with Node.js before. Then, for starters, you can make a very simple web application that produces the trivial Hello, World! You can start mastering Node.js by reading a well-known guide . Following it, we create our Hello, World.
To do this, connect to the zone
and create, for example, in the / home / node directory the server.js file with the following contents:
Now run node /home/node/server.js and access our zone from a regular browser. In my case, this required configuring the routing between the zone and the Windows host system:
In the zone: route add 192.168.56.1 192.168.56.101 on
Windows: route add 192.168.56.102 192.168.56.101
Address 192.168.56.1 is the VirtualBox interface on Windows, 192.168. 56.101 - SmartOS host system, 192.168.56.102 - zone in it.
The DTrace provider for node is called (surprise!) Node. Since this is a provider of the USDT type, its sensors must be specified either in the form of node * ::: <sensor name> , or with a specific PID of the node process, for example, node3297 ::: <sensor name> .
A live example of using DTrace to monitor the operation of your server under Node.js is to record calls to it. To find out what requests servers from different zones receive (a typical case for a hosting provider), run a short script:
To see which addresses the requests come from, you can slightly modify the script:
The following sensors are available for node in DTrace:
gc-start
gc-done
http-client-request
http-client-response
http-server-request
http-server-response
net-server-connection
net-socket-read
net-socket-write
net -stream-end
If you find that some sensors are unavailable, this is a sure sign that you do not have the /usr/lib/dtrace/node.d file. Then you need to create the / var / lib / dtrace / directory, download the file there (it is located at https://raw.github.com/joyent/node/master/src/node.d ) and run dtrace with the –L / switch var / lib / dtrace / (it means "include the directory specified after the L key in the DTrace library search path).
A detailed description of the sensors with code examples (in English) is on the page dtrace.org/blogs/rm/2011/03/01/dtrace-probes-for-node-v0-4-x/#httpd (there, however, it is outdated version of Node.js).
The special charm of SmartOS settings is that once installed, they do not need to be repeated after the update. You can download the latest latest.iso, replace it with the previous one and boot from the new one. All settings, virtual machines and files inside them will be picked up automatically, since nobody changes anything in them - only the downloaded image changes, and the settings exist separately from it.
Those who are interested in listening to my story about DTrace will be waiting for November 29 at 19:00 in the St. Petersburg office of Yandex. The seminar is completely free, but the number of seats is limited, so you need to register . For those who do not fall on it, the organizers will provide an online broadcast. And for those who are going to, I’ll say that you need to bring laptops with SmartOS installed under VirtualBox, because the important part of the seminar will be the solution of practical problems.
The meeting will be useful to anyone interested in analyzing application performance on UNIX-like systems, especially Solaris software developers, Node.js software developers, system administrators and others responsible for performance (if they work with FreeBSD, Solaris, QNX, OpenIndiana, SmartOS and others supporting DTrace).
First, I’ll tell you what SmartOS is for. In short, its purpose is to be a host system for virtual machines. It is often used as the basis for public and private clouds, such as the Joyent and MITAC cloud services. LinkedIn uses the Joyent service: all of its mobile server part is made on Node.js, which runs in the Joyent cloud. We wrote about this in detail on the Semonix blog in articles about SmartOS and on the cloud based on illumos , and I talked about in a talk at YaC 2012 that I already mentioned.
Typical applications of SmartOS are data center systems, systems for developing and operating a loaded web application, and a platform for creating a private or public cloud. When working on this article, I actively usedan article by Stu Rudnage , who walked about the same SmartOS installation path as I did, but using VMware instead of VirtualBox.
We plan a test installation
Let's say you are one of those who are interested in trying SmartOS. How is it easier to do? I went along the simplest way, which would prevent Windows or Linux from running on a virtual machine, but would allow using the ready-made SmartOS + Node.js. I will launch SmartOS in VirtualBox. It is clear that KVM in a virtual machine for VirtualBox will not work, and the rest should earn a bang.
SmartOS can be a host system for guests running Windows, Linux, and SmartOS itself. In this article, I only consider the option with guest systems running SmartOS. They implement lightweight virtualization (the same as zones in Solaris and something similar to cells in FreeBSD), and therefore, they do not require KVM, which is only supported on physical computers. The entire configuration is shown in the figure below. SmartOS (host) is a system that is a guest in relation to Windows on my laptop, and a host - in relation to SmartOS systems running in zones inside this SmartOS.

We will install SmartOS in a virtual machine with two network interfaces. What for? Very simple: I want to connect a regular ssh to SmartOS from my host system (in this case, Windows) so that I can open several windows - at least one for each guest system inside my SmartOS.
Why not use the graphical mode in the console and run the right amount of xterm or gnome-terminal there? The fact is that the graphics ("X" - X) are not installed in SmartOS. There is a project to port X11 support to SmartOS, but it has not yet been completed, and Jonathan Perkin, the lead volunteer for this project, informed me that the deadline for working on it is unknown. Therefore, if I want to open five windows from ssh to my SmartOS, then I need TCP access from my host system to SmartOS in VirtualBox.
In addition, VirtualBox Guest Additions cannot be installed in the SmartOS guest machine, allowing cut-and-paste between the guest system and the host system in which I write the article (and I want to copy texts from the terminal into it). Theoretically, I think you can install Guest Additions (you basically need to install kernel modules there), but for Solaris-like systems Guest Additions comes in the form of a .pkg package, which is not supported in SmartOS. I decided that there would be more trouble with unpacking the package and manually screwing the module to the kernel than with ssh access from the host system.
Why do we need a second interface? So SmartOS can go through NAT for packets on the Internet. This restriction on the conditions of the task is imposed by VirtualBox - to make a virtual machine with NAT / DHCP accessible from the host system in it otherwise it is only possible through the bridged network, and this is not always convenient. A discussion of cases where it is not very convenient (or not supported on certain host systems or versions of VirtualBox) is beyond the scope of this article, but believe me - this happens. The simplest options are: the DHCP addresses on the network where the physical computer is connected to you have run out, or you do not want to look at each time what address this time was issued to the system before ssh hooked to it.
How to make two network interfaces in VirtualBox
Configuring the virtual machine so that it has 2 network adapters is simple: on the Network tab, you just need to indicate that not only the first adapter is active in host adapter mode, but the second in NAT mode (see the figure below). Please note that for SmartOS it makes sense to choose the type of Solaris 10 virtual machine 10/09 or later (64 bit). The Solaris 11 option may be better. It worked without problems and delays.
What else needs to be configured?
You must attach the latest.iso file to the virtual machine’s CD-ROM - SmartOS will boot from it.

After booting for the first time, the system will ask a few questions about the settings (in particular, the root password), format the disk, create a ZFS pool on the disk and write the settings to disk. After rebooting, you should see something similar to the picture below.

Log in as root using the password that you set earlier. Already now you can connect to SmartOS using ssh, for this you just need to find out what server address is:
ipadm show-addrHow to make two network interfaces in SmartOS
The system configuration in SmartOS is stored in the / usbkey / config file. The / usbkey file system is mounted from disk, unlike / etc, which is always loaded from a LiveCD or LiveUSB and mounted on a RAM disk. Therefore, the changes made to files from / etc will be lost upon reboot, and in / usbkey they will be saved.
The file / usbkey / config is created the first time SmartOS is started on the computer, and by default it describes only the first interface in the system. In order to see the second one, you need to slightly modify the file. To modify it, we need to find out the MAC addresses of the interfaces. To do this, you can use the command
dladm show-phys -m
LINK SLOT ADDRESS INUSE CLIENT
e1000g0 primary 8:0:27:f7:e9:a3 yes e1000g0
e1000g1 primary 8:0:27:d0:35:3b no --
Now let's change the / usbkey / config file to look like this:
#
# This file was auto-generated and must be source-able by bash.
#
# admin_nic is the nic admin_ip will be connected to for headnode zones.
admin_nic=8:0:27:f7:e9:a3
admin_ip=dhcp
admin_netmask=
admin_network=...
admin_gateway=dhcp
external_nic=8:0:27:d0:35:3b
external_ip=dhcp
external_netmask=255.255.255.0
external_gateway=dhcp
dns_resolvers=8.8.8.8,8.8.4.4
dns_domain=
ntp_hosts=pool.ntp.org
compute_node_ntp_hosts=dhcp
Creating a virtual machine inside SmartOS
First you need to download the image of the virtual machine. To do this, update the list of images:
imgadm updateNow let's see what ready-made images are in the repository:
imgadm avail
UUID OS PUBLISHED URN
78ab4d60-2610-11e2-b3f7-b3bd2c369427 linux 2012-11-04 sdc:jpc:ubuntu-12.04:2.1.2
6a67c702-2083-11e2-b4fa-03f9d1d64ef0 linux 2012-10-28 sdc:jpc:ubuntu-12.04:2.1.1
b00acc20-14ab-11e2-85ae-4f03a066e93e smartos 2012-10-12 sdc:sdc:mongodb:1.4.0
1fc068b0-13b0-11e2-9f4e-2f3f6a96d9bc smartos 2012-10-11 sdc:sdc:nodejs:1.4.0
8700b668-0da4-11e2-bde4-17221283a2f4 linux 2012-10-03 sdc:jpc:centos-6:1.3.0
55330ab4-066f-11e2-bd0f-434f2462fada smartos 2012-09-25 sdc:sdc:base:1.8.1
60a3b1fa-0674-11e2-abf5-cb82934a8e24 smartos 2012-09-25 sdc:sdc:base64:1.8.1
...
Now we have a list of images available for installation. Here is only part of the list - it is much longer. At the time of writing, 96 images were available in the repository. We are interested in the base64 image - a minimal system that includes node.js. Currently, the latest version of this image is 1.8.1 (includes node 0.8.9), and its UUID is 60a3b1fa-0674-11e2-abf5-cb82934a8e24.
Download the image of the virtual machine:
imgadm import 60a3b1fa-0674-11e2-abf5-cb82934a8e24
Now let's make a description of the virtual machine, which we will create on the basis of the just loaded image and run it inside the SmartOS host.
The description is made in the form of a JSON file (the details of the process are described in the documentation on smartos.org, I’ll just put a working example here - let it be in /zones/template.json for us). The file should be in the file system, which is located on the disk and saved between reboots, most conveniently in / zones.
Pay attention to dataset_uuid - this is the same UUID that belongs to the image just received from the repository. There will also be two network interfaces in the zone - “internal” and “external”. This setting is used in data centers in real life: an internal network is connected to the internal interface, in which data center administrators work, and an external network is connected to the Internet, where clients come from.
{
"brand": "joyent",
"dataset_uuid": "60a3b1fa-0674-11e2-abf5-cb82934a8e24",
"max_physical_memory": 1024,
"nics": [
{
"nic_tag": "admin",
"ip": "dhcp",
"gateway": "dhcp"
},
{
"nic_tag": "external",
"ip": "dhcp",
"gateway": "10.0.3.2"
}
]
}
Now we are creating a new SmartOS virtual machine (for brevity, we will continue to call it “zone”):
vmadm create -f /zones/ template.jsonIf everything turned out fine, you can see the launched new zone by giving the command
vmadm list
UUID TYPE RAM STATE ALIAS
570cccb2-0511-400b-9143-7616b2ca8a3d OS 1024 running -
Now you can connect to the new zone either from the SmartOS console or via ssh:
zlogin 570cccb2-0511-400b-9143-7616b2ca8a3d
Please note: The UUID of a zone is different from the UUID of a zone image, because you can create many zones from the same image. At the same time, up to 8191 virtual machines can be launched in one SmartOS host system.
To connect to the zone via ssh, you need to clarify which address is given to it; for this, you first need to join it through zlogin, and then find out the address:
zlogin 570cccb2-0511-400b-9143-7616b2ca8a3d
ipadm show-addr
Package management
Now you can install any packages we need in the zone. To do this, you need to update the information about available packages using pkgin update , get their list pkgin list or search for what is required using pkgin search . To install the package, use pkgin install .
How to run Node.js
Suppose you were not familiar with Node.js before. Then, for starters, you can make a very simple web application that produces the trivial Hello, World! You can start mastering Node.js by reading a well-known guide . Following it, we create our Hello, World.
To do this, connect to the zone
zlogin 570cccb2-0511-400b-9143-7616b2ca8a3d
and create, for example, in the / home / node directory the server.js file with the following contents:
var http = require("http");
http.createServer(function(request, response) {
response.writeHead(200, {"Content-Type": "text/plain"});
response.write("Hello World");
response.end();
}).listen(80);
Now run node /home/node/server.js and access our zone from a regular browser. In my case, this required configuring the routing between the zone and the Windows host system:
In the zone: route add 192.168.56.1 192.168.56.101 on
Windows: route add 192.168.56.102 192.168.56.101
Address 192.168.56.1 is the VirtualBox interface on Windows, 192.168. 56.101 - SmartOS host system, 192.168.56.102 - zone in it.
How to use DTrace with Node.js?
The DTrace provider for node is called (surprise!) Node. Since this is a provider of the USDT type, its sensors must be specified either in the form of node * ::: <sensor name> , or with a specific PID of the node process, for example, node3297 ::: <sensor name> .
A live example of using DTrace to monitor the operation of your server under Node.js is to record calls to it. To find out what requests servers from different zones receive (a typical case for a hosting provider), run a short script:
# dtrace -L /var/lib/dtrace -n 'node*:::http-server-request{printf("%s: %s of %s\n",zonename, args[0]->method, args[0]->url)}' -q
570cccb2-0511-400b-9143-7616b2ca8a3d: GET of /
570cccb2-0511-400b-9143-7616b2ca8a3d: GET of /favicon.ico
570cccb2-0511-400b-9143-7616b2ca8a3d: GET of /
570cccb2-0511-400b-9143-7616b2ca8a3d: GET of /favicon.ico
570cccb2-0511-400b-9143-7616b2ca8a3d: GET of /
570cccb2-0511-400b-9143-7616b2ca8a3d: GET of /favicon.ico
To see which addresses the requests come from, you can slightly modify the script:
# dtrace -L /var/lib/dtrace -n 'node*:::http-server-request{printf("%s: %s of %s\n",args[1]->remoteAddress, args[0]->method, args[0]->url)}' -q
192.168.56.1: GET of /
192.168.56.1: GET of /
192.168.56.1: GET of /favicon.ico
192.168.56.1: GET of /
192.168.56.1: GET of /favicon.ico
192.168.56.1: GET of /
192.168.56.1: GET of /
The following sensors are available for node in DTrace:
gc-start
gc-done
http-client-request
http-client-response
http-server-request
http-server-response
net-server-connection
net-socket-read
net-socket-write
net -stream-end
If you find that some sensors are unavailable, this is a sure sign that you do not have the /usr/lib/dtrace/node.d file. Then you need to create the / var / lib / dtrace / directory, download the file there (it is located at https://raw.github.com/joyent/node/master/src/node.d ) and run dtrace with the –L / switch var / lib / dtrace / (it means "include the directory specified after the L key in the DTrace library search path).
A detailed description of the sensors with code examples (in English) is on the page dtrace.org/blogs/rm/2011/03/01/dtrace-probes-for-node-v0-4-x/#httpd (there, however, it is outdated version of Node.js).
Conclusion
The special charm of SmartOS settings is that once installed, they do not need to be repeated after the update. You can download the latest latest.iso, replace it with the previous one and boot from the new one. All settings, virtual machines and files inside them will be picked up automatically, since nobody changes anything in them - only the downloaded image changes, and the settings exist separately from it.
Those who are interested in listening to my story about DTrace will be waiting for November 29 at 19:00 in the St. Petersburg office of Yandex. The seminar is completely free, but the number of seats is limited, so you need to register . For those who do not fall on it, the organizers will provide an online broadcast. And for those who are going to, I’ll say that you need to bring laptops with SmartOS installed under VirtualBox, because the important part of the seminar will be the solution of practical problems.
The meeting will be useful to anyone interested in analyzing application performance on UNIX-like systems, especially Solaris software developers, Node.js software developers, system administrators and others responsible for performance (if they work with FreeBSD, Solaris, QNX, OpenIndiana, SmartOS and others supporting DTrace).