What is allowed by Jupyter?
Our story began with a seemingly simple task. It was necessary to set up analytical tools for data science specialists and just data analysts. This task was addressed to us by colleagues from the retail risk and CRM divisions, where the concentration of data science specialists is historically high. Customers had a simple desire - to write Python code, import advanced libraries (xgboost, pytorch, tensorflow, etc.) and run algorithms on data raised from the hdfs cluster.
Everything seems to be simple and clear. But there were so many pitfalls that we decided to write a post about it and post the ready-made solution on GitHub.
First, some details about the source infrastructure:
As the main environment for the work of analysts, they decided to choose JupyterHub, which de facto has already become one of the standards for working with data and developing machine learning models. Read more about it here . In the future, we already imagined JupyterLab.
It would seem that everything is simple: you need to take and configure a bunch of Python + Anaconda + Spark. Install Jupyter Hub on the application server, integrate with LDAP, connect Spark or connect to hdfs data in some other way and go ahead - build models!
If you delve into all the source data and requirements, then here is a more detailed list:
This post is designed for IT professionals who are faced with the need to solve such problems.
There is nothing unusual here. JupyterHub and Cloudera product clients are installed in the container (as - see below), and the configuration files are mounted from the host machine:
start-hub.sh
For integration with the Active Directory / Kerberos iron and not very hosts, the standard in our company is the PBIS Open product . Technically, this product is a set of services that communicate with Active Directory, with which, in turn, clients work through unix domain sockets. This product integrates with Linux PAM and NSS.
We used the standard Docker method - unix domain sockets of host services were mounted in a container (sockets were found empirically by simple manipulations with the lsof command):
start-hub.sh
In turn, PBIS packages are installed inside the container, but without executing the postinstall section. So we put only executable files and libraries, but do not start services inside the container - this is superfluous for us. PAM and NSS Linux integration commands are run manually.
Dockerfile:
It turns out that the clients of the PBIS container communicate with the PBIS host services. JupyterHub uses a PAM authenticator, and with properly configured PBIS on the host, everything works out of the box.
In order to prevent all users from AD from entering JupyterHub, you can use the setting that restricts users to specific AD groups.
config-example / jupyterhub / jupyterhub_config.py
When logging in to JupyterHub, PBIS caches the user's Kerberos ticket in a specific file in the / tmp directory. For transparent authentication in this way, it is enough to mount the / tmp directory of the host in the container and set the KRB5CCNAME variable to the desired value (this is done in our authenticator class).
start-hub.sh
assets / jupyterhub / dsai.py
Thanks to the above code, the JupyterHub user can execute hdfs commands from the Jupyter terminal and run Spark jobs without additional authentication steps. Mounting the entire / tmp directory of the host into the container is unsafe - we are aware of this problem, but its solution is still under development.
Here, it would seem, everything is simple: you need to install the necessary versions of Python and integrate them with Jupyter, creating the necessary Kernel. This issue has already been covered in many places. Conda is used to manage Python environments. Why all simplicity is only apparent will be clear from the next section. Kernel example for Python 3.6 (this file is not in git - all kernel files are generated by code):
/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/share/jupyter/kernels/python3 .6.6 / kernel.json
To integrate with SPARK clients, you also need to create Kernels. Kernel example for Python 3.6 and SPARK 2.
/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/share/jupyter/kernels/python3.6.6-pyspark2/kernel.json
Just note that the requirement to have Spark 1 support has developed historically. However, it is possible that someone will face similar restrictions - you cannot, for example, install Spark 2 in a cluster. Therefore, we describe here the pitfalls that we met on the way to implementation.
First, Spark 1.6.1 does not work with Python 3.6. Interestingly, in CDH 5.12.1 this was fixed, but in 5.15.1 - for some reason not). At first, we wanted to solve this problem by simply applying the appropriate patch. However, in the future, this idea had to be abandoned, since this approach requires the installation of a modified Spark in a cluster, which was unacceptable to us. The solution was found in creating a separate Conda environment with Python 3.5.
The second problem prevents Spark 1 from working inside Docker. The Spark driver opens a specific port through which Worker connects to the driver - for this the driver sends it its IP address. In the case of Docker Worker, it tries to connect to the driver via the IP of the container, and when using network = bridge it doesn’t work out quite naturally.
The obvious solution is to send not the IP of the container, but the IP of the host, which was implemented in Spark 2 by adding the appropriate configuration settings. This patch was creatively redesigned and applied to Spark 1. Spark modified in this way does not need to be placed on the cluster hosts, so there is no problem similar to incompatibility with Python 3.6.
Regardless of the version of Spark, for its functionality it is necessary to have the same Python versions in the cluster as in the container. To install Anaconda directly bypassing Cloudera Manager, we had to learn to do two things:
This turned out to be a fairly simple task. All you need is:
The process is described in more detail on GitHub , there is also a validator code there. We borrowed metadata in the official Anaconda parcel for Cloudera, creatively reworking it.
This practice has proven useful for two reasons:
As a bonus, as a result of solving the problems above, we received:
To install parcel in Docker, Cloudera Manager is first installed from the RPM packages. For the actual installation of parcel, Java code is used. The client in Java knows what the client in Python cannot do, so I had to use Java and lose some uniformity), which calls the API.
assets / install-parcels / src / InstallParcels.java
To manage host machine resources, a combination of DockerSpawner is used - a component that runs Jupyter end users in a separate Docker container - and cgroups - a resource management mechanism in Linux. DockerSpawner uses the Docker API, which allows you to set the parent cgroup for the container. There is no such possibility in the regular DockerSpawner, so we wrote simple code that allows us to set the correspondence between AD entities and parent cgroup in the configuration.
assets / jupyterhub / dsai.py
A small modification has also been introduced that launches Jupyter from the same image from which JupyterHub is launched. Therefore, there is no need to use more than one image.
assets / jupyterhub / dsai.py
What exactly to run in the container, Jupyter or JupyterHub, is determined in the startup script by environment variables:
assets / jupyterhub / dsai.py
The ability to start Jupyter Docker containers from the JupyterHub Docker container is achieved by mounting the Docker daemon socket in the JupyterHub container.
start-hub.sh
In the future, it is planned to abandon this decision in favor of, for example, ssh.
When using DockerSpawner in conjunction with Spark, another problem arises: the Spark driver opens random ports, through which Workers then establish an external connection. We can control the range of port numbers from which random ones are selected by setting these ranges in the Spark configuration. However, these ranges must be different for different users, since we cannot run Jupyter containers with the same published ports. To solve this problem, code was written that simply generates port ranges by user id from the JupyterHub database and launches the Docker container and Spark with the corresponding configuration:
assets / jupyterhub / dsai.py
The disadvantage of this solution is that when you restart the container with JupyterHub, everything stops working due to database loss. Therefore, when you restart the JupyterHub for, for example, a configuration change, we do not touch the container itself, but only restart the JupyterHub process inside it.
restart-hub.sh
Cgroups themselves are created by standard Linux tools, the correspondence between AD entities and cgroups in the configuration looks like this.
Our solution is publicly available on GitHub: https://github.com/DS-AI/dsai/ (DSAI - Data Science and Artificial Intelligence). All code is arranged in directories with serial numbers - the code from each subsequent directory can use artifacts from the previous one. The result of the code from the last directory will be a Docker image.
Each directory contains files:
In order to completely rebuild the Docker image, it is necessary to run clean.sh, assets.sh, build.sh from the directories according to their serial numbers.
For assembly, we use a machine with Linux RedHat 7.4, Docker 17.05.0-ce. The machine has 8 cores, 32GB RAM and 250GB of disk space. It is strongly recommended that you do not use a host with the worst RAM and HDD settings to build it.
Here is the help for the names used:
Alas, the assembly may crash for reasons that we were unable to fix (for example, tar is dropped during the assembly of parcel . In this case, as a rule, you just need to restart the assembly, but this does not always help (for example, Spark assembly depends on external resources Cloudera, which may no longer be available, etc.).
Another drawback is that the parcel assembly is not reproducible. Since the libraries are constantly updated, repetition of the assembly may give a result different from the previous one.
Now users are successfully using the tools, their number has exceeded several dozen and continues to grow. In the future, we plan to try JupyterLab and are thinking of connecting the GPU to the cluster, since now the computing resources of two fairly powerful application servers are no longer enough.
Everything seems to be simple and clear. But there were so many pitfalls that we decided to write a post about it and post the ready-made solution on GitHub.
First, some details about the source infrastructure:
- HDFS Data Warehouse (12 Oracle Big Data Appliance nodes, Cloudera distribution). In total, the storehouse has 130 TB of data from various internal systems of the bank; there is also heterogeneous information from external sources.
- Two application servers on which the deployment of analytical tools was supposed. It is worth mentioning that not only advanced analytics tasks are “spinning” on these servers, so one of the requirements was to use containerization tools (Docker) to manage server resources, use various environments and configure them.
As the main environment for the work of analysts, they decided to choose JupyterHub, which de facto has already become one of the standards for working with data and developing machine learning models. Read more about it here . In the future, we already imagined JupyterLab.
It would seem that everything is simple: you need to take and configure a bunch of Python + Anaconda + Spark. Install Jupyter Hub on the application server, integrate with LDAP, connect Spark or connect to hdfs data in some other way and go ahead - build models!
If you delve into all the source data and requirements, then here is a more detailed list:
- Running JupyterHub in Docker (base OS - Oracle Linux 7)
- Cloudera CDH 5.15.1 + Spark 2.3.0 cluster with Kerberos authentication in Active Directory configuration + dedicated Kerberos MIT in the cluster (see Cluster-Dedicated MIT KDC with Active Directory ), Oracle Linux 6
- Active Directory Integration
- Transparent authentication in Hadoop and Spark
- Python 2 and 3 support
- Spark 1 and 2 (with the ability to use cluster resources for training models and parallelizing data processing using pyspark)
- Ability to limit host resources
- Library set
This post is designed for IT professionals who are faced with the need to solve such problems.
Solution Description
Launch in Docker + Cloudera Cluster Integration
There is nothing unusual here. JupyterHub and Cloudera product clients are installed in the container (as - see below), and the configuration files are mounted from the host machine:
start-hub.sh
VOLUMES="-v/var/run/docker.sock:/var/run/docker.sock:Z
-v/var/lib/pbis/.lsassd:/var/lib/pbis/.lsassd:Z -v/var/lib/pbis/.netlogond:/var/lib/pbis/.netlogond:Z
-v/var/jupyterhub/home:/home/BANK/:Z -v/u00/:/u00/:Z -v/tmp:/host/tmp:Z
-v${CONFIG_DIR}/krb5.conf:/etc/krb5.conf:ro
-v${CONFIG_DIR}/hadoop/:/etc/hadoop/conf.cloudera.yarn/:ro
-v${CONFIG_DIR}/spark/:/etc/spark/conf.cloudera.spark_on_yarn/:ro
-v${CONFIG_DIR}/spark2/:/etc/spark2/conf.cloudera.spark2_on_yarn/:ro
-v${CONFIG_DIR}/jupyterhub/:/etc/jupyterhub/:ro"
docker run -p0.0.0.0:8000:8000/tcp ${VOLUMES} -e VOLUMES="${VOLUMES}"
-e HOST_HOSTNAME=`hostname -f` dsai1.2Active Directory Integration
For integration with the Active Directory / Kerberos iron and not very hosts, the standard in our company is the PBIS Open product . Technically, this product is a set of services that communicate with Active Directory, with which, in turn, clients work through unix domain sockets. This product integrates with Linux PAM and NSS.
We used the standard Docker method - unix domain sockets of host services were mounted in a container (sockets were found empirically by simple manipulations with the lsof command):
start-hub.sh
VOLUMES="-v/var/run/docker.sock:/var/run/docker.sock:Z
-v/var/lib/pbis/.lsassd:/var/lib/pbis/.lsassd:Z -v/var/lib/pbis/.netlogond:/var/lib/pbis/.netlogond:Z -v/var/jupyterhub/home:/home/BANK/:Z -v/u00/:/u00/:Z -v/tmp:/host/tmp:Z
-v${CONFIG_DIR}/krb5.conf:/etc/krb5.conf:ro
-v${CONFIG_DIR}/hadoop/:/etc/hadoop/conf.cloudera.yarn/:ro
-v${CONFIG_DIR}/spark/:/etc/spark/conf.cloudera.spark_on_yarn/:ro
-v${CONFIG_DIR}/spark2/:/etc/spark2/conf.cloudera.spark2_on_yarn/:ro
-v${CONFIG_DIR}/jupyterhub/:/etc/jupyterhub/:ro"
docker run -p0.0.0.0:8000:8000/tcp ${VOLUMES} -e VOLUMES="${VOLUMES}"
-e HOST_HOSTNAME=`hostname -f` dsai1.2
In turn, PBIS packages are installed inside the container, but without executing the postinstall section. So we put only executable files and libraries, but do not start services inside the container - this is superfluous for us. PAM and NSS Linux integration commands are run manually.
Dockerfile:
# Install PAM itself and standard PAM configuration packages.
RUN yum install -y pam util-linux \
# Here we just download PBIS RPM packages then install them omitting scripts.
# We don't need scripts since they start PBIS services, which are not used - we connect to the host services instead.
&& find /var/yum/localrepo/ -type f -name 'pbis-open*.rpm' | xargs rpm -ivh --noscripts \
# Enable PBIS PAM integration.
&& domainjoin-cli configure --enable pam \
# Make pam_loginuid.so module optional (Docker requirement) and add pam_mkhomedir.so to have home directories created automatically.
&& mv /etc/pam.d/login /tmp \
&& awk '{ if ($1 == "session" && $2 == "required" && $3 == "pam_loginuid.so") { print "session optional pam_loginuid.so"; print "session required pam_mkhomedir.so skel=/etc/skel/ umask=0022";} else { print $0; } }' /tmp/login > /etc/pam.d/login \
&& rm /tmp/login \
# Enable PBIS nss integration.
&& domainjoin-cli configure --enable nsswitch
It turns out that the clients of the PBIS container communicate with the PBIS host services. JupyterHub uses a PAM authenticator, and with properly configured PBIS on the host, everything works out of the box.
In order to prevent all users from AD from entering JupyterHub, you can use the setting that restricts users to specific AD groups.
config-example / jupyterhub / jupyterhub_config.py
c.DSAIAuthenticator.group_whitelist = ['COMPANY\\domain^users']
Transparent authentication in Hadoop and Spark
When logging in to JupyterHub, PBIS caches the user's Kerberos ticket in a specific file in the / tmp directory. For transparent authentication in this way, it is enough to mount the / tmp directory of the host in the container and set the KRB5CCNAME variable to the desired value (this is done in our authenticator class).
start-hub.sh
VOLUMES="-v/var/run/docker.sock:/var/run/docker.sock:Z
-v/var/lib/pbis/.lsassd:/var/lib/pbis/.lsassd:Z -v/var/lib/pbis/.netlogond:/var/lib/pbis/.netlogond:Z
-v/var/jupyterhub/home:/home/BANK/:Z -v/u00/:/u00/:Z -v/tmp:/host/tmp:Z
-v${CONFIG_DIR}/krb5.conf:/etc/krb5.conf:ro
-v${CONFIG_DIR}/hadoop/:/etc/hadoop/conf.cloudera.yarn/:ro
-v${CONFIG_DIR}/spark/:/etc/spark/conf.cloudera.spark_on_yarn/:ro
-v${CONFIG_DIR}/spark2/:/etc/spark2/conf.cloudera.spark2_on_yarn/:ro
-v${CONFIG_DIR}/jupyterhub/:/etc/jupyterhub/:ro"
docker run -p0.0.0.0:8000:8000/tcp ${VOLUMES} -e VOLUMES="${VOLUMES}"
-e HOST_HOSTNAME=`hostname -f` dsai1.2assets / jupyterhub / dsai.py
env['KRB5CCNAME'] = '/host/tmp/krb5cc_%d' %
pwd.getpwnam(self.user.name).pw_uidThanks to the above code, the JupyterHub user can execute hdfs commands from the Jupyter terminal and run Spark jobs without additional authentication steps. Mounting the entire / tmp directory of the host into the container is unsafe - we are aware of this problem, but its solution is still under development.
Python versions 2 and 3
Here, it would seem, everything is simple: you need to install the necessary versions of Python and integrate them with Jupyter, creating the necessary Kernel. This issue has already been covered in many places. Conda is used to manage Python environments. Why all simplicity is only apparent will be clear from the next section. Kernel example for Python 3.6 (this file is not in git - all kernel files are generated by code):
/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/share/jupyter/kernels/python3 .6.6 / kernel.json
{
"argv": [ "/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
Spark 1 and 2
To integrate with SPARK clients, you also need to create Kernels. Kernel example for Python 3.6 and SPARK 2.
/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/share/jupyter/kernels/python3.6.6-pyspark2/kernel.json
{
"argv": [
"/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3 + PySpark 2",
"language": "python",
"env": {
"JAVA_HOME": "/usr/java/default/",
"SPARK_HOME": "/opt/cloudera/parcels/SPARK2/lib/spark2/",
"PYTHONSTARTUP": "/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/shell.py",
"PYTHONPATH": "/opt/cloudera/parcels/SPARK2/lib/spark2/python/:/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.7-src.zip",
"PYSPARK_PYTHON": "/opt/cloudera/parcels/Anaconda-5.3.1-dsai1.0/envs/python3.6.6/bin/python"
}
}
Just note that the requirement to have Spark 1 support has developed historically. However, it is possible that someone will face similar restrictions - you cannot, for example, install Spark 2 in a cluster. Therefore, we describe here the pitfalls that we met on the way to implementation.
First, Spark 1.6.1 does not work with Python 3.6. Interestingly, in CDH 5.12.1 this was fixed, but in 5.15.1 - for some reason not). At first, we wanted to solve this problem by simply applying the appropriate patch. However, in the future, this idea had to be abandoned, since this approach requires the installation of a modified Spark in a cluster, which was unacceptable to us. The solution was found in creating a separate Conda environment with Python 3.5.
The second problem prevents Spark 1 from working inside Docker. The Spark driver opens a specific port through which Worker connects to the driver - for this the driver sends it its IP address. In the case of Docker Worker, it tries to connect to the driver via the IP of the container, and when using network = bridge it doesn’t work out quite naturally.
The obvious solution is to send not the IP of the container, but the IP of the host, which was implemented in Spark 2 by adding the appropriate configuration settings. This patch was creatively redesigned and applied to Spark 1. Spark modified in this way does not need to be placed on the cluster hosts, so there is no problem similar to incompatibility with Python 3.6.
Regardless of the version of Spark, for its functionality it is necessary to have the same Python versions in the cluster as in the container. To install Anaconda directly bypassing Cloudera Manager, we had to learn to do two things:
- build your parcel with Anaconda and all the right environments
- install it in Docker (for consistency)
Assembly parcel Anaconda
This turned out to be a fairly simple task. All you need is:
- Prepare parcel content by installing the required versions of Anaconda and Python environment
- Create metadata file (s) and put it in meta directory
- Create parcel with simple tar
- Validate parcel utility from Cloudera
The process is described in more detail on GitHub , there is also a validator code there. We borrowed metadata in the official Anaconda parcel for Cloudera, creatively reworking it.
Install parcel in Docker
This practice has proven useful for two reasons:
- ensuring Spark operability - it is impossible to put Anaconda into a cluster without parcel
- Spark 2 is distributed only in the form of parcel - you could, of course, install it in a container just in the form of jar files, but this approach was rejected
As a bonus, as a result of solving the problems above, we received:
- ease of setting up Hadoop and Spark clients - when installing the same parcels in Docker and in the cluster, the paths on the cluster and in the container are the same
- ease of maintaining a uniform environment in the container and in the cluster - when updating the cluster, the Docker image is simply rebuilt with the same parcels that were installed in the cluster.
To install parcel in Docker, Cloudera Manager is first installed from the RPM packages. For the actual installation of parcel, Java code is used. The client in Java knows what the client in Python cannot do, so I had to use Java and lose some uniformity), which calls the API.
assets / install-parcels / src / InstallParcels.java
ParcelsResourceV5 parcels = clusters.getParcelsResource(clusterName);
for (int i = 1; i < args.length; i += 2) {
result = installParcel(api, parcels, args[i], args[i + 1], pause);
if (!result) {
System.exit(1);
}
}
Host resource limitation
To manage host machine resources, a combination of DockerSpawner is used - a component that runs Jupyter end users in a separate Docker container - and cgroups - a resource management mechanism in Linux. DockerSpawner uses the Docker API, which allows you to set the parent cgroup for the container. There is no such possibility in the regular DockerSpawner, so we wrote simple code that allows us to set the correspondence between AD entities and parent cgroup in the configuration.
assets / jupyterhub / dsai.py
def set_extra_host_config(self):
extra_host_config = {}
if self.user.name in self.user_cgroup_parent:
cgroup_parent = self.user_cgroup_parent[self.user.name]
else:
pw_name = pwd.getpwnam(self.user.name).pw_name
group_found = False
for g in grp.getgrall():
if pw_name in g.gr_mem and g.gr_name in self.group_cgroup_parent:
cgroup_parent = self.group_cgroup_parent[g.gr_name]
group_found = True
break
if not group_found:
cgroup_parent = self.cgroup_parent
extra_host_config['cgroup_parent'] = cgroup_parent
A small modification has also been introduced that launches Jupyter from the same image from which JupyterHub is launched. Therefore, there is no need to use more than one image.
assets / jupyterhub / dsai.py
current_container = None
host_name = socket.gethostname()
for container in self.client.containers():
if container['Id'][0:12] == host_name:
current_container = container
break
self.image = current_container['Image']
What exactly to run in the container, Jupyter or JupyterHub, is determined in the startup script by environment variables:
assets / jupyterhub / dsai.py
#!/bin/bash
ANACONDA_PATH="/opt/cloudera/parcels/Anaconda/"
DEFAULT_ENV=`cat ${ANACONDA_PATH}/envs/default`
source activate ${DEFAULT_ENV}
if [ -z "${JUPYTERHUB_CLIENT_ID}" ]; then
while true; do
jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
done
else
HOME=`su ${JUPYTERHUB_USER} -c 'echo ~'`
cd ~
su ${JUPYTERHUB_USER} -p -c "jupyterhub-singleuser --KernelSpecManager.ensure_native_kernel=False --ip=0.0.0.0"
fi
The ability to start Jupyter Docker containers from the JupyterHub Docker container is achieved by mounting the Docker daemon socket in the JupyterHub container.
start-hub.sh
VOLUMES="-v/var/run/docker.sock:/var/run/docker.sock:Z
-v/var/lib/pbis/.lsassd:/var/lib/pbis/.lsassd:Z -v/var/lib/pbis/.netlogond:/var/lib/pbis/.netlogond:Z -v/var/jupyterhub/home:/home/BANK/:Z -v/u00/:/u00/:Z -v/tmp:/host/tmp:Z
-v${CONFIG_DIR}/krb5.conf:/etc/krb5.conf:ro
-v${CONFIG_DIR}/hadoop/:/etc/hadoop/conf.cloudera.yarn/:ro
-v${CONFIG_DIR}/spark/:/etc/spark/conf.cloudera.spark_on_yarn/:ro
-v${CONFIG_DIR}/spark2/:/etc/spark2/conf.cloudera.spark2_on_yarn/:ro
-v${CONFIG_DIR}/jupyterhub/:/etc/jupyterhub/:ro"
docker run -p0.0.0.0:8000:8000/tcp ${VOLUMES} -e VOLUMES="${VOLUMES}"
-e HOST_HOSTNAME=`hostname -f` dsai1.2
In the future, it is planned to abandon this decision in favor of, for example, ssh.
When using DockerSpawner in conjunction with Spark, another problem arises: the Spark driver opens random ports, through which Workers then establish an external connection. We can control the range of port numbers from which random ones are selected by setting these ranges in the Spark configuration. However, these ranges must be different for different users, since we cannot run Jupyter containers with the same published ports. To solve this problem, code was written that simply generates port ranges by user id from the JupyterHub database and launches the Docker container and Spark with the corresponding configuration:
assets / jupyterhub / dsai.py
def set_extra_create_kwargs(self):
user_spark_driver_port, user_spark_blockmanager_port, user_spark_ui_port, user_spark_max_retries = self.get_spark_ports()
if user_spark_driver_port == 0 or user_spark_blockmanager_port == 0 or user_spark_ui_port == 0 or user_spark_max_retries == 0:
return
ports = {}
for p in range(user_spark_driver_port, user_spark_driver_port + user_spark_max_retries):
ports['%d/tcp' % p] = None
for p in range(user_spark_blockmanager_port, user_spark_blockmanager_port + user_spark_max_retries):
ports['%d/tcp' % p] = None
for p in range(user_spark_ui_port, user_spark_ui_port + user_spark_max_retries):
ports['%d/tcp' % p] = None
self.extra_create_kwargs = { 'ports' : ports }
The disadvantage of this solution is that when you restart the container with JupyterHub, everything stops working due to database loss. Therefore, when you restart the JupyterHub for, for example, a configuration change, we do not touch the container itself, but only restart the JupyterHub process inside it.
restart-hub.sh
#!/bin/bash
docker ps | fgrep 'dsai1.2' | fgrep -v 'jupyter-' | awk '{ print $1; }' | while read ID; do docker exec $ID /bin/bash -c "kill \$( cat /root/jupyterhub.pid )"; done
Cgroups themselves are created by standard Linux tools, the correspondence between AD entities and cgroups in the configuration looks like this.
config-example/jupyterhub/jupyterhub_config.py
c.DSAISpawner.user_cgroup_parent = {
'bank\\user1' : '/jupyter-cgroup-1', # user 1
'bank\\user2' : '/jupyter-cgroup-1', # user 2
'bank\\user3' : '/jupyter-cgroup-2', # user 3
}
c.DSAISpawner.cgroup_parent = '/jupyter-cgroup-3'
Git code
Our solution is publicly available on GitHub: https://github.com/DS-AI/dsai/ (DSAI - Data Science and Artificial Intelligence). All code is arranged in directories with serial numbers - the code from each subsequent directory can use artifacts from the previous one. The result of the code from the last directory will be a Docker image.
Each directory contains files:
- assets.sh - creation of artifacts necessary for assembly (downloading from the Internet or copying from the directories of the previous steps)
- build.sh - build
- clean.sh - cleaning artifacts needed for assembly
In order to completely rebuild the Docker image, it is necessary to run clean.sh, assets.sh, build.sh from the directories according to their serial numbers.
For assembly, we use a machine with Linux RedHat 7.4, Docker 17.05.0-ce. The machine has 8 cores, 32GB RAM and 250GB of disk space. It is strongly recommended that you do not use a host with the worst RAM and HDD settings to build it.
Here is the help for the names used:
- 01-spark-patched - RPM Spark 1.6.1 with two patches applied SPARK-4563 and SPARK-19019.
- 02-validator - parcel validator
- 03-anaconda-dsai-parcel-1.0 - parcel Anaconda with the right Python (2, 3.5 and 3.6)
- 04-cloudera-manager-api - Cloudera Manager API libraries
- 05-dsai1.2-offline - final image
Alas, the assembly may crash for reasons that we were unable to fix (for example, tar is dropped during the assembly of parcel . In this case, as a rule, you just need to restart the assembly, but this does not always help (for example, Spark assembly depends on external resources Cloudera, which may no longer be available, etc.).
Another drawback is that the parcel assembly is not reproducible. Since the libraries are constantly updated, repetition of the assembly may give a result different from the previous one.
Grand finale
Now users are successfully using the tools, their number has exceeded several dozen and continues to grow. In the future, we plan to try JupyterLab and are thinking of connecting the GPU to the cluster, since now the computing resources of two fairly powerful application servers are no longer enough.
