How to assemble Ceylon in a container if your container ship has a channel blocked

It so happened that in our team of kind, sweet and responsive mathematicians ( DMOiM ), Ceylon is used as a general-purpose language . Being responsive, we not only use this language, but also participate in its development, mainly bug reports. Pulkvest is worse, and the first reason for this is the lack of direct access to the Internet in the office, only through a proxy server. (Zero, of course, lack of time.)



Under the cut, details about exactly what problems arose when building the Ceylon project from the source code and how they were solved. At the end there are also a few words about her ultimate goal.

What kind of elephant is this

First, a few words about Ceylon as a project. (As for the programming language, who are interested, read here .) Two basic subprojects (besides such tools as IDE plugins, own repository) in it are ceylon and ceylon-sdk . The first includes the compiler itself and a set of console utilities written in Java. The second is a set of core libraries written in Ceylon itself. Each of the projects is built using Apache Ant, and of course, the installed JDK is also required.

The compiler is assembled by the teamant clean dist, after which you can copy it from the dist / dist directory to / usr / local / share / ceylon or somewhere else to your taste and make a link to the executable in the directory that is visible in $ PATH. Libraries are collected and copied wherever needed by the team ant clean publish.

If you have direct access to the Internet (along with the necessary versions of Java, Ant and Ceylon sources), the assembly proceeds without any difficulties.

Course problem

Security requirements in our company require the absence of direct access to the Internet from workstations and most servers, assuming that local mirrors and a proxy server are enough for all occasions. But the opposite cases, nevertheless, do happen. Even when setting up CI for their own projects, colleagues had to suffer because of this.

When compiling the bare iron of the workstation on an open deck , the compiler assembled quietly (maybe all the dependencies are included in the project, maybe the settings made earlier helped), but the library set didn’t want to be built. It would seem that there is a gateway proxy, and tanks with libraries (Ceylon Herd, Maven Nexus), but something is missing. Probably, "salty spray, gusts of furious wind ..." ⓒ Barash.


The funny thing is that the build system requires a Java dependency from Maven in the Ceylon ( car ) format , which should not be there in principle.

Packaging assembly in a container

What is a container, the local audience should be aware. For those passing by - link .

So, having not achieved success, I decided: I will assemble the Ceylon and Ceylon SDK on the high seas on a computer with direct Internet access and in a container I will deliver this assembly to the office network. In theory, the container will contain all the dependencies, after which it will be possible to edit the code for your own needs and pull requests and start rebuilding.

As an intermediate level, an image with JDK was prepared.


Dockerfile was borrowed from corporate configurations, commented on the creation of the Russian locale commented. Its main task is to install Oracle JDK (through an auxiliary program, by virtue of the Oracle license) and Maven, also at the beginning they are updated, and at the end the data for APT is cleaned. Ant is also present, apparently as a dependency.

The next step: make the image with the assembly Ceylon. When creating the image, the ceylon-sources directory should be next to the Dockerfile, and it should contain the ceylon and ceylon-sdk projects. At first I wanted to stick git clone directly into the image creation, but we will edit the source locally, but it makes no sense to clone twice.

Dockerfile kopilov / ceylon_build: 1.3.4-SNAPSHOT

FROM kopilov/java8:latest
#Именно эту версию мы будем собирать
ENV CEYLON_VERSION 1.3.4-SNAPSHOT
#Повторно скачаем данные APT (их удаляли, чтобы образ был тоньше),
#установим git (он каким-то образом участвует в сборке)
#и netcat (он потребуется немного позже) 
RUN apt-get update -y && \
    apt-get install -y git && \
    apt-get install netcat-traditional
#Копируем исходники с рабочей станции. Теоретически,
#тут может быть git clone, но это много трафика, который уже выкачан.
WORKDIR /usr/src/ceylon
ADD ceylon-sources /usr/src/ceylon
#Собираем компилятор, ставим в систему
WORKDIR /usr/src/ceylon/ceylon
RUN ant clean dist && \
    cp -a dist/dist /usr/local/share/ceylon-${CEYLON_VERSION} && \
    ln -s /usr/local/share/ceylon-${CEYLON_VERSION}/bin/ceylon /usr/local/bin
#Собираем библиотеки
WORKDIR /usr/src/ceylon/ceylon-sdk
RUN ant clean publish
#Удаляем ненужное
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

The assembly of this image will be successful only if there is direct access to the Internet. Having assembled, I posted it on hub.docker.com .

Closed Channel Reassembly

It is expected that the container with the finished assembly includes all the dependencies, and the Internet will no longer be required. We start docker run -it kopilov/ceylon_build, then ant clean publish- no matter how.


The same error on the computer where the image was created if you disable Internet access. What is still missing? Without a shark, traffic is not clear.

With Internet access turned off, traffic from Docker looks like this in Wireshark:



After several unsuccessful attempts to determine the IP of the repo1.maven.org server, the above error is displayed. But what happens if the connection is restored:



Paradox: the system makes a GET request to get a 404 error response, and then calmly continues the assembly. And if she doesn’t fulfill this request, the user is given an error that seems to be absolutely orthogonal to that GET request. Below you can see requests for modules.ceylon-lang.org (aka Herd ) via HTTPS, but first try to figure out the first one.

The first thing we decided to do was add the line “127.0.0.1 repo1.maven.org” to the / etc / hosts file. Now we need to somehow imitate the answer “404 NOT FOUND”. Recent habrasurfing has shown that netcat ( proof ) can act as the simplest web server . Before starting the assembly (but after starting the container) I type in the parallel terminal

docker container ls
#скопировать название_контейнера
docker exec -it название_контейнера bash
nc -lp 80

After that, I start the build (ant), wait for the GET request to appear in the terminal with netcat, and print Voila in response ! The assembly went on! Then the system makes one more exactly the same request (when building under JavaScript), and the process is successfully completed.
HTTP/1.1 404 NOT FOUND
Server: nc



Automation of the above and refinement of the library

The prepared image included the assembly of the original Ceylon SDK library, and the final goal was to assemble the modified one. Therefore, another Dockerfile was made, replacing the source:

FROM kopilov/ceylon_build:1.3.4-SNAPSHOT
ENV CEYLON_VERSION 1.3.4-SNAPSHOT
WORKDIR /usr/src/ceylon/ceylon-sdk
RUN rm -rf * 
ADD ceylon-sources/ceylon-sdk .

It should have been as simple as possible (after all, the image is recreated for each test rebuild - almost for every source revision), which is why netcat was installed in advance. The netcat trick was wrapped in the following script (plug.sh):

#!/bin/bash
IMAGE_NAME="kopilov/ceylon_patch_src"
CONTAINER_ID=$(docker container ls | grep "${IMAGE_NAME}" | sed 's/ .*//')
docker exec -i $CONTAINER_ID bash << END
echo "127.0.0.1 repo1.maven.org" >> /etc/hosts
echo "HTTP/1.1 404 NOT FOUND" > /tmp/notfound
echo "Server: nc" >> /tmp/notfound
echo "" >> /tmp/notfound
nc -lp 80 < /tmp/notfound
nc -lp 80 < /tmp/notfound
END

Another script to get the assembly from the container (get_built_ceylon.sh):

#!/bin/bash
CONTAINER_ID=$(docker container ls -a | grep  kopilov/ceylon_patch_src | sed 's/ .*//')
rm -r ~/.sdkman/candidates/ceylon/1.3.4-SNAPSHOT/
docker cp $CONTAINER_ID:/usr/local/share/ceylon-1.3.4-SNAPSHOT .
mv ceylon-1.3.4-SNAPSHOT /home/akopilov/.sdkman/candidates/ceylon/1.3.4-SNAPSHOT
rm -r ~/.ceylon/repo/
docker cp $CONTAINER_ID:/root/.ceylon/repo ~/.ceylon

Further, the desire for rationalization dried up, it remained to act manually. After each editing of the sources, first run in one tab of the terminal docker build -t kopilov/ceylon_patch_src . && docker run -it kopilov/ceylon_patch_src, then in the next ./plug.sh, then again in the first ant clean publish. And, if the assembly went without errors (and if you already have something to test) - ./get_built_ceylon.sh.

Results and Origins

The main result of the work done “for the future” was the ability of our (and maybe not only) team to send pre-tested pullrequests to the upstream of the project. At the moment, I personally sent this one, written along the way: github.com/ceylon/ceylon-sdk/pull/688

Docker images kopilov / java8 and kopilov / ceylon_build are available on hub.docker.com , if anyone needs to.

It all started with the fact that last week I had extremely few urgent, and not even very urgent tasks, even the head of the department left somewhere. And leaving, he said: “Think that you can tie something else useful to your project.” (The project is at the prototype stage.) And I wanted to screw internationalization, and not with crutches, but with a ready-made solution.

PS
KDPV background source