Clack Web Applications

Clack — a web application environment for Common Lisp — loads verbatim via (ql:quickload "clack"), and clack:clackup runs a Clack application on the built-in clack-handler-rontolisp backend:

$ cat app.lisp
(ql:quickload "clack")
(clack:clackup
 (lambda (env)
   (list 200 '(:content-type "text/plain")
         (list (format nil "Hello, Clack! ~A ~A~%"
                       (getf env :request-method) (getf env :path-info)))))
 :server :rontolisp
 :port 5000
 :use-thread nil)
$ rontolisp app.lisp        # interpret; or -o App.class / -o app.wasm --component
$ curl http://127.0.0.1:5000/hello
Hello, Clack! GET /hello

There is no adaptation layer behind this: rontolisp's own server protocol is Clack's (see Serving HTTP), so the backend hands the application to the server as the handler and converts nothing per request — a Clack application is a valid rontolisp:http-handler handler, and vice versa.

The first run downloads clack, lack and their dependencies into ~/.rontolisp/quicklisp; the dependencies resolve to real libraries (alexandria, the ironclad slice) and to the built-in shim systems (bordeaux-threads, usocket, swank, uiop).

clackup semantics

The defaults work the way Clack users expect:

• :use-thread t (the default) returns a handler object while the server answers on a background thread (rontolisp:make-thread), and (clack:stop handler) shuts that server down.
• :use-thread nil serves in the foreground: clackup blocks until the process is stopped (Ctrl-C) — the script shape used above.
• :use-default-middlewares t (the default) wraps the application in lack's backtrace middleware through lack:builder.
• :address binds the listener (default 127.0.0.1); :silent t suppresses the banner and :debug nil the debug notice.

The application protocol

The application is a function from the standard Clack env plist to the standard (status headers body) list:

The response body may be a list of strings, a (vector (unsigned-byte 8)) (written byte for byte, so a binary response is byte-exact), a rontolisp stream, or nil; the two-element (status headers) form is valid too. A bare string signals a clear error, as Clack itself refuses strings; a pathname body (lack's file-serving form) is a distinct value here and is refused too, until the transport can serve it. A function body is supported in Clack's delayed-response form (the responder is called with the final response list); the streaming-writer form signals.

Getting from one handler to a set of routes

The application above is ONE function for the whole site. A routing library is what turns it into a set of routes, and tiny-routes loads unmodified (see the ASDF systems guide):

$ cat routes.lisp
(ql:quickload "clack")
(ql:quickload "tiny-routes")

(defpackage :demo (:use :cl :tiny-routes))
(in-package :demo)

(define-routes *app*
  (define-get "/hello" () (ok "hello world"))
  (define-get "/users/:id" (req) (ok (format nil "user ~A" (path-parameter req :id))))
  (define-post "/echo" (req) (ok (format nil "echo:~A" (request-body req))))
  (define-any "*" () (not-found "nope")))

(clack:clackup (pipe *app* (wrap-request-body) (wrap-query-parameters))
               :server :rontolisp :port 5000 :use-thread nil)
$ rontolisp routes.lisp
$ curl http://127.0.0.1:5000/hello
hello world
$ curl http://127.0.0.1:5000/users/42
user 42
$ curl -XPOST -d abc http://127.0.0.1:5000/echo
echo:abc
$ curl -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5000/zzz
404

Its request IS the env plist above and its response IS the response list, so nothing is converted at the boundary: wrap-request-body reads the :raw-body stream, wrap-query-parameters parses :query-string, the path template matches :path-info, and ok/not-found build (status headers body). The routes are read inside the application's own package, which is where the library is meant to be used from.

The same routes run WITHOUT a server on every backend — call the composed handler with a request plist you build yourself, which is what examples/asdf/tiny-routes-demo.lisp does. Serving them has the backend constraints below.

The other answer: ningle

ningle loads unmodified too, and it is a different model rather than a different spelling. The application is a CLOS object you hang routes on, each route is a setf, a controller receives the matched parameters (the request itself is in a special variable), and a controller that is not a function at all is answered as the body:

$ cat ningle-app.lisp
(ql:quickload "clack")
(ql:quickload "ningle")

(defpackage :demo (:use :cl))
(in-package :demo)

(defvar *app* (make-instance 'ningle:app))

(setf (ningle:route *app* "/") "Welcome to ningle!")
(setf (ningle:route *app* "/hello/:name")
      (lambda (params) (format nil "Hello, ~A" (cdr (assoc :name params)))))
(setf (ningle:route *app* "/submit" :method :POST)
      (lambda (params) (format nil "posted ~A" (cdr (assoc "q" params :test #'string=)))))

(clack:clackup *app* :server :rontolisp :port 5000 :use-thread nil)
$ rontolisp ningle-app.lisp
$ curl http://127.0.0.1:5000/
Welcome to ningle!
$ curl http://127.0.0.1:5000/hello/Eitaro
Hello, Eitaro
$ curl -XPOST -d q=abc http://127.0.0.1:5000/submit
posted abc
$ curl -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5000/zzz
404

Four differences are worth knowing before picking one:

• Routes are added, not listed. (setf (ningle:route ...)) mutates the application, so routes can come from anywhere — including from run-time data.
• Query and body parameters arrive in the same alist as the template's :name bindings (keyed by the string name), because ningle reads every request through lack-request. tiny-routes never touches that chain, and that is most of the size difference in a compiled module — an order of magnitude for the same two routes, with no ppcre-free opt-in to fall back on, since ningle's router compiles every rule to a scanner.
• The 404 is a method, ningle:not-found, rather than a catch-all route, and ningle:*response* is mutable — which is how a controller answers a status other than 200.
• A route can be chosen by something that is not the path. :accept negotiation is built in, and (setf (ningle:requirement app :key) fn) registers your own; the closure runs on every dispatch.

Backends

The :server :rontolisp line does not change between these — it means "serve on this target's native inbound transport", chosen at compile time:

• Interpreter — everything above.
• JVM class — the same program compiled with -o App.class; like every served program it needs the rontolisp jar on the runtime classpath (java -cp rontolisp-exec.jar:. App).
• WASM component (--component) — the host owns the socket: run with wasmtime serve -W gc=y -W exceptions=y -S cli=y -S tcp=y -S inherit-network=y app.wasm. The :port argument is ignored, :use-thread is effectively nil (the WASM backends are single-threaded, so it defaults to nil there) and clack:stop is meaningless — the host controls the server's lifecycle.
• WASM reactor (--no-wasi, or --no-gc) — the host calls the module instead of handing it a socket: the same program compiles to a module exporting handle-request (a JSON request string in, a JSON response string out), which a Cloudflare Worker, a browser page, node or a JVM host calls per request. :port is ignored and clackup returns at once — the next section has the details.
• WASM Preview 1 has no incoming TCP by design: the program compiles, and clackup signals HTTP-HANDLER requires --component ... at run time (catchable with handler-case).

A host that calls you: the reactor build

Some hosts never hand you a socket. A Cloudflare Worker, a browser page, node and a JVM embedding all parse the request themselves and then call an exported function. There is nothing for clackup to start there — but you still write clackup, and since :server :rontolisp picks the transport per target, nothing in the source has to change: compile the very same program with --no-wasi and the handler backend takes its reactor shape.

$ rontolisp app.lisp -o worker.wasm --no-wasi --optimize=size

run starts nothing here: it stores the application, and the compiler synthesizes the export the host calls (handle-request, a JSON request string in and a JSON response string out) from a marker the handler backend leaves behind. Nothing in your source names it, and the module imports nothing — no WASI shim on the JavaScript side.

One keyword is a property of the other backends, not boilerplate: :use-thread nil — on the interpreter and the JVM clackup defaults to running the backend on its own thread, and a script wants to serve in the foreground. clackup's default middlewares stay on everywhere: lack's backtrace middleware writes its report to *error-output*, which under --no-wasi is a discarding sink and on every other backend is real standard error.

A handler that fetches: --host-fetch

A reactor imports nothing, which also means it has no HTTP client — so an application that calls rontolisp:fetch (a proxy, an API gateway) needs one more flag. --host-fetch lowers fetch onto the host's own client as an env.fetch import for the request and the reply's head, plus an env.readResponseBody import the reply's body is pulled through; those two imports are the whole difference to the module above:

$ rontolisp worker.lisp -o worker.wasm --no-wasi --host-fetch --optimize=size

The route bodies stay synchronous. Only an async-defun / async-lambda body may await, so a route that needs a fetched value calls one and returns its future — the reactor transport resolves a future-valued response at the boundary, exactly as wasmtime serve does under --component. The fetch guide has what else is particular to this transport (a body pulled after the head, a future settled at the headers, and the JSPI obligation on the JavaScript side), and examples/cloudflare-workers/dog-fetcher is a routed Worker built this way — one source that also serves a socket on the interpreter and the JVM.

Driving the reactor by hand: clack-handler-reactor

A second built-in handler backend makes the reactor shape explicit and host-driven on every backend:

$ cat worker.lisp
(ql:quickload "clack-handler-reactor")
(load "app.lisp")                       ; defines app, an ordinary Clack application

(clack:clackup #'app :server :reactor :use-thread nil)

Where :rontolisp binds a socket on the interpreter and the JVM, this designator stores the application there too, and the host calls (clack.handler.reactor:dispatch request-json) — the same function the synthesized export calls — directly. That is how a Worker can be developed and tested without the Worker: the whole edit/run loop happens on the interpreter. A Worker itself no longer needs this designator; both ride the same machinery and the same application store, so the two cannot drift.

Underneath both is handle, and it needs no clackup and no Worker to try — it is an ordinary function of two arguments:

{"status":200,"headers":[["content-type","text/plain"]],"body":"GET /hi a=1"}

handle takes the application and one JSON request string and answers one JSON response string. It builds the Clack environment and normalizes the Clack response through the same code path a served request takes, so the application sees exactly what Clack promises — and it catches: on a host like this an uncaught error would take the whole instance down, so it answers 500 with the condition's report instead.

The envelope, in both directions:

{ "method": "GET", "target": "/path?a=1", "headers": {"host": "..."},
  "body": "", "scheme": "https", "remote-addr": "203.0.113.7" }

{ "status": 200, "headers": [["content-type", "text/plain"]], "body": "..." }

Two details the host side must get right:

• target is the raw request target — path and query still joined and still percent-encoded. The split and the decoding happen on the Lisp side, and :path-info / :query-string have to come from there for the application to see what Clack promises.
• Send content-length for a request with a body. lack/request parses nothing without it, and a request that arrived chunked carries none — set it from the bytes you actually read.

Response headers cross as an array of pairs, not an object, so an application that sets two cookies still answers two Set-Cookie headers.

Passing the body separately

The JSON above is the request head. handle and dispatch take one more optional argument, the body source, so the body does not have to ride inside it:

• nil — no body;
• a string — the body, already read;
• a function of no arguments — a pull source: each call answers the next chunk — a string, or an (unsigned-byte 8) vector for a host that hands over raw octets — with nil or an empty chunk for the end. It may answer a future, so a host that suspends while it reads can hand one over.

Octets stay octets. Both :raw-body shapes are byte streams -- the default asynchronous stream answers each chunk as an (unsigned-byte 8) vector, the buffered Gray stream stores the octets as they came -- so a chunk that arrived as octets reaches the application unchanged and a binary upload is byte-exact; a source handing over text is UTF-8 encoded once. Nothing decodes per chunk, so a chunk boundary inside a UTF-8 sequence (a host reading a socket knows nothing about code points) costs nothing: read-all decodes the whole body once.

A source that is empty at its first call is no body at all — :raw-body stays nil, exactly as for a request whose "body" is absent, because that is what upstream's (when raw-body ...) guards expect and a bodiless GET must not pay for a stream it would only find empty.

The envelope's "body" key is exactly the string case, and it is what is used when no source is passed — or when the source turns out to be empty, so a host may start handing a reader over without also having to stop filling the envelope. A host written against the shape above keeps working unchanged.

Taking the response body separately

Symmetrically, handle and dispatch take a fourth optional argument, the body sink: a function of one argument, called with each chunk of the response body. It may answer a future, so a host that suspends while it writes can hand one over.

Given a sink, the JSON answer is the response head and its "body" key is absent — so a host can tell "the body crossed out of band" from "the body is the empty string". A stream response body (a proxied fetch) is then forwarded chunk at a time instead of being collected into one string first.

{"status":200,"headers":[["content-type","text/plain"]]}
hello

The chunks cross before the head, because the head is the return value. So a head that does carry a "body" key wins over anything already written: that is how a handler that fails halfway through its body still answers one clean document — the 500 the transport catches into carries its report in band, and the host discards the chunks it already took.

An (unsigned-byte 8) response body reaches the sink as octets, not as text: a sink can write bytes, and a JSON head cannot carry them.

Passing no sink keeps the old shape exactly: the body rides the head, a stream body is drained into it, and octets -- an octet body, or a stream's octet chunks -- are rendered as the text their UTF-8 bytes spell, so a page a Clack application answered as octets crosses as the page it was. A JSON string is text, so that is the ONE place a binary response is not byte-exact, and it is the reason the sink exists: a host that answers binary passes one.

The WASM boundary: a head export and two body imports

On a --no-wasi WASM module neither a source nor a sink is a Lisp value the host can pass, so the boundary is three entries and the compiler writes the two imports for you:

module -> host   handle-request(headPtr, headLen) -> (ptr, len)   ; the JSON head
host -> module   env.readRequestBody(ptr, cap) -> n               ; up to cap octets
                                                                  ;   at ptr; 0 = end
host -> module   env.writeResponseBody(ptr, len)                  ; take these octets

The head is the JSON above without the "body" key, in either direction. The bodies cross as raw octets — in, into a buffer the module owns and reuses; out, straight out of the module's own memory — which is what a JSON string could not do: a binary body crosses exactly either way (the string boundary decodes UTF-8, and does not validate), and crossing costs the module no linear memory at all — the envelope used to hold the body several times over. Reading the request body is not yet free: whichever way a handler drains :raw-body, decoding the octets to text currently costs about fifteen times the body in linear memory, reclaimed for reuse at the end of the request.

Note the direction flip in the two imports. A chunk crossing in is a result written into a buffer the module passes; one crossing out is a parameter the host reads and must copy before the call returns. Both are the same rule — the caller owns the memory — and both mean the host may not hold on to a pointer.

Both imports are declared :async t, so the host chooses how it answers. Answering synchronously (read the body first, then call in; collect the response chunks as they arrive) is the simple host, and it is what the Worker examples do. Wrapping an import in WebAssembly.Suspending — pulling from the request's own reader, or writing to a stream that applies backpressure — is the streaming host: it must then enter handle-request through WebAssembly.promising and serialise its calls, because a suspended module can be re-entered — the module refuses that with a trap rather than corrupting both calls, and the build prints the obligation.

Under --component both bodies stay inside the envelope: a component's host functions cross the canonical ABI rather than a core import. So does a plain WASI command module that drives its own dispatch in-process (what the examples' check.lisp files do), whose host is wasmtime run and satisfies no env.* import. Everything above this section is unchanged either way, which is the point of the source and the sink being abstract values.

What the application then sees is the :raw-body mode. clackup and handle ask for the buffered one, the synchronous stream Clack promises (the source is drained into it whatever shape it had). A reactor built from a bare rontolisp:http-handler keeps that directive's default instead — a rontolisp stream, drained the same way as on every other backend:

A complete Worker built this way — the JavaScript side and the measurements included — is examples/cloudflare-workers/httpbin-clack/. Beside it, examples/cloudflare-workers/httpbin-clack-one-source/ deploys examples/net/httpbin-clack.lisp itself — the file that binds a socket when you interpret it — and so contains no Lisp file at all: one source, four hosts.

If the module size matters more than the clackup line, this adapter is small enough to write out by hand and skip loading clack entirely. examples/cloudflare-workers/httpbin/ is that: the same application, the same envelope, the same JavaScript side, and about half the module. The two directories are a measured pair — the per-request cost turns out to be identical, and what clack costs on a host like this is module size and isolate startup.

Current limits

• One Clack server per process: a second concurrent clackup replaces the first one's application.
• clack.socket (WebSocket) and :swank-port are unsupported (:swank-port reaches the swank stub, which signals).
• Streaming-writer responses and bare-string/pathname bodies signal, as noted above (delayed function responses work).

See also: Serving HTTP (http-handler) for the underlying server, and examples/asdf/clack-hello.lisp for the runnable demo with all per-backend commands.