Writing a Web server on Common Lisp Part One
Not so long ago, I began to study Common Lisp. It may seem that learning a new programming language is not an easy task, especially if it is completely different from all those languages that you have encountered before. So I decided to start with Land Of Lisp . The book is very good, with interesting pictures and very good for beginners. One of the chapters described how to create a web server on Common Lisp. I decided to slightly develop this topic, and in the end I did not get exactly what was described in this chapter, but a very interesting web server. Source codes can be found here .
To write it, we need Linux with emacs, sbcl, slime and quicklisp installed. I will not describe how to install it, configure it, and how to use it - there are many articles on the Internet about this. Our entire web server will be in one package called myweb. Create a folder with this name, and in it create two folders log and web. The log folder will contain the web server log file. The web folder will contain html pages and images that the web server will give to clients. The entire web server consists of seven files.
Let's start with the file declaring the package and the asd file for the package description.
Create the package.lisp file:
As you can see, our web server consists of three packages:
The in-package function is usually placed at the beginning of the file and indicates the name of the package in which we declare variables and functions. In this case, since we are declaring packages, we must declare them in the main package: cl-user.
Note the : use and : export directives in package declarations. : use allows us to use functions from other packages without specifying the package name at the beginning of the function name, thereby reducing the amount of typed text. : exportsets the names of those functions that can be used outside the package. As you can see, we have in the package: myweb the functions: start-http and: stop-http. Being in the package: cl-user, we will not be able to call them through myweb: start-http unless we first declare them using the: export directive.
We already have an advertisement for packages, now it remains to write the source code of these packages. Create the web.lisp, util.lisp, and handler.lisp files and add an in-package call in each of them. For web.lisp - (in-package: myweb), for util.lisp (in-package: myweb.util), etc. We will also need to create a log.lisp file with a call (in-package: cl-log). This file is needed to run and configure the cl-log logging system .
The final touch to creating the file structure for the web server will be the creation of the myweb.asd file, which describes which files the asdf system needs to load so that everything works for us.
Key: serial t indicates that asdf downloads files in the same order in which they are listed.
Now we need to write the file load.lisp, which will load our package and start the swank server for slime.
To continue development, we need to already run swank and load all the necessary libraries using quicklisp. To do this, run sbcl while in the myweb directory and call the function (quicklisp: quickload “swank”). After installing swank, start the swank server by calling (swank: create-server) from the sbcl command line.
Using slime-connect from emacs, connect to running sbcl and call all other functions with quicklisp from load.lisp using slime-mode in emacs and the ctrl-e key combination. If you did everything correctly, then quicklisp will download all the necessary libraries and load them using asdf for you. Everything is ready to start development.
Let's start with the web server itself. For him we need sockets. I decided to work with sockets using the widespread usocket library. We will also need threads, for which we will use bordeaux-threads . But first, I would like to talk about the model of processing http-requests that we are going to create. Each request will be processed by a separate thread. We will have worker threads that will be created depending on the number of requests. Among them, we will have separate idle streams, which, after processing the request, will go into the condition-wait state, waiting for new requests. Thus, you can reduce the burden of creating new worker threads. It turns out a kind of thread pool mechanism for processing http requests.
Let's start by declaring sockets and variables for mutex s in the web.lisp file:
To accept and distribute requests across threads, we will use a separate thread, the pointer to which will be stored in * listen-thread *. Let's start with the start-http method:
This is a simple function to start the dispenser stream, which in turn will call the http-acceptor function. We also have two keys - this is worker-limit - the maximum number of workers, and idle-workers - the number of idle workers.
We write the query distribution function itself:
The first thing we do is socket-listen to the specified address and port. Further in the loop, we do socket-accept, resulting in socket on the connected client, which we must process in the worker. Plus we assign request-id to the request. At this stage, we must decide what to do with the request and how to process it. First of all, we check the number of idle threads. If we have all the workers busy, we add the request to the queue for processing. If we have a free idle worker, then we again add the request to the queue, but this time we call (condition-notify (caar * idle-workers *))). And in the third case, we simply create a new worker and pass the request to it, which will be processed in the worker-thread function. Everything is quite simple. It remains only to write a function for processing the worker stream:
If we had a call with request-id, then we need to process the request first. We simply call the auxiliary function http-worker and pass the client socket to it. Next, we check if there are more processing requests: just remove the first request from the queue and pass it to worker-thread for processing, thereby calling the worker-thread function recursively. The question may arise, “will the recursion limit happen because the stack overflows at some point, for example, when there are a large number of requests in the queue?” Since after calling worker-thread nothing is called recursively in the function, then recursion limit will not happen. Almost all modern Common Lisp implementations support this optimization. Well, if the queue is empty, then we just have to check the number of idle workers. If everything is okay with us, then we just complete the request and remove the worker from the list of workers. If not, then we do a condition-wait, and thereby worker becomes idle worker.
If you notice, then we also call list-workers. This is a helper function that simply cleans the worker list of dead threads.
It remains to write the http-worker function:
Here we create socket-stream, parse the request and pass it to myweb.handler: process-request (we will talk about these functions in the second part). list-workers simply returns us a list of workers, previously clearing it of dead threads. We call this function in worker-thread before condition-wait.
The last thing we need to do is write a stop-http function that will stop our web server:
As you can see, everything is simple here - we stop the dispenser flow, kill all the workers and zero out the lists.
And so, everything is ready to process our requests. We will talk about this in the second part .
Thank you for your attention!
PS Thanks ertaquo for help with spelling and layout
To write it, we need Linux with emacs, sbcl, slime and quicklisp installed. I will not describe how to install it, configure it, and how to use it - there are many articles on the Internet about this. Our entire web server will be in one package called myweb. Create a folder with this name, and in it create two folders log and web. The log folder will contain the web server log file. The web folder will contain html pages and images that the web server will give to clients. The entire web server consists of seven files.
Let's start with the file declaring the package and the asd file for the package description.
Create the package.lisp file:
(in-package:cl-user)
(defpackage:myweb
(:use:cl:usocket:bordeaux-threads)
(:export:start-http:stop-http:list-workers:list-requests))
(defpackage:myweb.util
(:use:cl:local-time)
(:export:parse-request:read-utf-8-string:response-write:get-param:get-header:http-response:file-response:html-template:log-info:log-warning:log-error))
(defpackage:myweb.handler
(:use:cl)
(:export:process-request))
As you can see, our web server consists of three packages:
- myweb - will contain functions for starting and stopping the web server
- myweb.util - will contain functions to help handle requests
- myweb.handler - will contain the request processing code itself
The in-package function is usually placed at the beginning of the file and indicates the name of the package in which we declare variables and functions. In this case, since we are declaring packages, we must declare them in the main package: cl-user.
Note the : use and : export directives in package declarations. : use allows us to use functions from other packages without specifying the package name at the beginning of the function name, thereby reducing the amount of typed text. : exportsets the names of those functions that can be used outside the package. As you can see, we have in the package: myweb the functions: start-http and: stop-http. Being in the package: cl-user, we will not be able to call them through myweb: start-http unless we first declare them using the: export directive.
We already have an advertisement for packages, now it remains to write the source code of these packages. Create the web.lisp, util.lisp, and handler.lisp files and add an in-package call in each of them. For web.lisp - (in-package: myweb), for util.lisp (in-package: myweb.util), etc. We will also need to create a log.lisp file with a call (in-package: cl-log). This file is needed to run and configure the cl-log logging system .
The final touch to creating the file structure for the web server will be the creation of the myweb.asd file, which describes which files the asdf system needs to load so that everything works for us.
;; myweb.asd
(asdf:defsystem #:myweb:serialt:components ((:file"package")
(:file"log")
(:file"util")
(:file"web")
(:file"handler")))
Key: serial t indicates that asdf downloads files in the same order in which they are listed.
Now we need to write the file load.lisp, which will load our package and start the swank server for slime.
(in-package:cl-user)
(quicklisp:quickload"swank")
(quicklisp:quickload"usocket")
(quicklisp:quickload"bordeaux-threads")
(quicklisp:quickload"trivial-utf-8")
(quicklisp:quickload"cl-log")
(quicklisp:quickload"local-time")
(pushnew '*default-pathname-defaults* asdf:*central-registry*)
(asdf:load-system 'myweb)
(swank:create-server)
To continue development, we need to already run swank and load all the necessary libraries using quicklisp. To do this, run sbcl while in the myweb directory and call the function (quicklisp: quickload “swank”). After installing swank, start the swank server by calling (swank: create-server) from the sbcl command line.
Using slime-connect from emacs, connect to running sbcl and call all other functions with quicklisp from load.lisp using slime-mode in emacs and the ctrl-e key combination. If you did everything correctly, then quicklisp will download all the necessary libraries and load them using asdf for you. Everything is ready to start development.
Let's start with the web server itself. For him we need sockets. I decided to work with sockets using the widespread usocket library. We will also need threads, for which we will use bordeaux-threads . But first, I would like to talk about the model of processing http-requests that we are going to create. Each request will be processed by a separate thread. We will have worker threads that will be created depending on the number of requests. Among them, we will have separate idle streams, which, after processing the request, will go into the condition-wait state, waiting for new requests. Thus, you can reduce the burden of creating new worker threads. It turns out a kind of thread pool mechanism for processing http requests.
Let's start by declaring sockets and variables for mutex s in the web.lisp file:
(defvar *listen-socket* nil)
(defvar *listen-thread* nil)
(defvar *request-mutex* (make-lock"request-mutex"))
(defvar *request-threads* (list))
(defvar *worker-mutex* (make-lock"worker-mutex"))
(defvar *workers* (list))
(defvar *worker-num* 0)
(defvar *idle-workers* (list))
(defvar *idle-workers-num* 0)
(defvar *request-queue* (list))
To accept and distribute requests across threads, we will use a separate thread, the pointer to which will be stored in * listen-thread *. Let's start with the start-http method:
(defun start-http (host port &key (worker-limit10) (idle-workers1))
(if (not *listen-socket*)
(setq *listen-thread*
(make-thread (lambda () (http-acceptor host port worker-limit idle-workers)) :name"socket-acceptor"))
"http server already started"))
This is a simple function to start the dispenser stream, which in turn will call the http-acceptor function. We also have two keys - this is worker-limit - the maximum number of workers, and idle-workers - the number of idle workers.
We write the query distribution function itself:
(defun http-acceptor (host port worker-limit idle-workers)
(setq *listen-socket* (socket-listen host port :reuse-addresst:element-type '(unsigned-byte 8) :backlog (* worker-limit 2)))
(let ((request-id0)
(worker-id0))
(loop while *listen-thread* do
(let* ((socket (socket-accept *listen-socket* :element-type '(unsigned-byte 8))))
(progn (setq request-id (1+ request-id))
(acquire-lock *worker-mutex*)
(if (>= *worker-num* worker-limit)
(push (cons request-id socket) *request-queue*)
;; Get worker from idle workers
(if (> *idle-workers-num* 0)
(progn (push (cons request-id socket) *request-queue*)
(condition-notify (caar *idle-workers*)))
;; Add new Worker
(progn (setq worker-id (1+ worker-id))
(setq *worker-num* (1+ *worker-num*))
(setq *workers* (cons (make-thread (lambda () (worker-thread request-id socket idle-workers))
:name (concatenate 'string "socket-worker-" (prin1-to-string worker-id))) *workers*)))))
(release-lock *worker-mutex*)
t)))))
The first thing we do is socket-listen to the specified address and port. Further in the loop, we do socket-accept, resulting in socket on the connected client, which we must process in the worker. Plus we assign request-id to the request. At this stage, we must decide what to do with the request and how to process it. First of all, we check the number of idle threads. If we have all the workers busy, we add the request to the queue for processing. If we have a free idle worker, then we again add the request to the queue, but this time we call (condition-notify (caar * idle-workers *))). And in the third case, we simply create a new worker and pass the request to it, which will be processed in the worker-thread function. Everything is quite simple. It remains only to write a function for processing the worker stream:
(defun worker-thread (request-id socket idle-workers)
(if request-id
;; Process request if it is not nil
(progn
(with-lock-held (*request-mutex*)
(setq *request-threads* (cons (cons request-id (current-thread)) *request-threads*))
)
(http-worker socket)
(with-lock-held (*request-mutex*)
(setq *request-threads* (remove-if (lambda (x) (eq (car x) request-id)) *request-threads*))
)
))
(acquire-lock *worker-mutex*)
(if *request-queue*
(let ((requestnil))
(setq request (car *request-queue*))
(setq *request-queue* (cdr *request-queue*))
(release-lock *worker-mutex*)
(worker-thread (car request) (cdr request) idle-workers))
(if (< *idle-workers-num* idle-workers)
(let ((condition (make-condition-variable))
(idle-lock (make-lock))
(requestnil))
(push (cons condition (current-thread)) *idle-workers*)
(setq *idle-workers-num* (1+ *idle-workers-num*))
(release-lock *worker-mutex*)
(list-workers)
(with-lock-held (idle-lock)
(condition-wait condition idle-lock)
)
(with-lock-held (*worker-mutex*)
(setq *idle-workers* (cdr *idle-workers*))
(setq *idle-workers-num* (1- *idle-workers-num*))
(setq request (car *request-queue*))
(setq *request-queue* (cdr *request-queue*))
)
(worker-thread (car request) (cdr request) idle-workers))
(progn (setq *workers* (remove (current-thread) *workers*))
(setq *worker-num* (1- *worker-num*))
(release-lock *worker-mutex*)))))
If we had a call with request-id, then we need to process the request first. We simply call the auxiliary function http-worker and pass the client socket to it. Next, we check if there are more processing requests: just remove the first request from the queue and pass it to worker-thread for processing, thereby calling the worker-thread function recursively. The question may arise, “will the recursion limit happen because the stack overflows at some point, for example, when there are a large number of requests in the queue?” Since after calling worker-thread nothing is called recursively in the function, then recursion limit will not happen. Almost all modern Common Lisp implementations support this optimization. Well, if the queue is empty, then we just have to check the number of idle workers. If everything is okay with us, then we just complete the request and remove the worker from the list of workers. If not, then we do a condition-wait, and thereby worker becomes idle worker.
If you notice, then we also call list-workers. This is a helper function that simply cleans the worker list of dead threads.
It remains to write the http-worker function:
(defun http-worker (socket)
(let* ((stream (socket-stream socket))
(request (myweb.util:parse-request stream)))
(myweb.handler:process-request request stream)
(finish-output stream)
(socket-close socket)))
(defun list-workers ()
(with-lock-held (*worker-mutex*)
(setq *workers*
(remove-if (lambda (w) (not (thread-alive-p w))) *workers*))
(setq *worker-num* (length *workers*))
*workers*))
Here we create socket-stream, parse the request and pass it to myweb.handler: process-request (we will talk about these functions in the second part). list-workers simply returns us a list of workers, previously clearing it of dead threads. We call this function in worker-thread before condition-wait.
The last thing we need to do is write a stop-http function that will stop our web server:
(defun stop-http ()
(if *listen-socket*
(progn (stop-thread)
(socket-close *listen-socket*)
(setq *listen-socket* nil)
(setq *request-queue* nil)
(setq *worker-num* 0)
(setq *workers* nil)
(mapcar (lambda (i) (destroy-thread (cdr i))) *idle-workers*)
(setq *idle-workers-num* 0)
(setq *idle-workers* nil)
(release-lock *worker-mutex*)
(setq *request-threads* nil)
(release-lock *request-mutex*)
(setq *request-mutex* (make-lock"request-mutex"))
(setq *worker-mutex* (make-lock"worker-mutex")))))
(defun stop-thread ()
(if (and *listen-thread* (thread-alive-p *listen-thread*))
(destroy-thread *listen-thread*)))
As you can see, everything is simple here - we stop the dispenser flow, kill all the workers and zero out the lists.
And so, everything is ready to process our requests. We will talk about this in the second part .
Thank you for your attention!
PS Thanks ertaquo for help with spelling and layout
