Published: 30th January 2023; Last updated: 30th January 2023
Contents
Since a few years, I have been hosting a Mumble server on my Uberspace, for talking to my friends online while we are playing computer games (lately mostly Minecraft). As every good admin should to, I try to keep the software I'm hosting up-to-date. In January 2022, I received a notification that a new minor version of Mumble was released. This is where my learning journey began.
Up to version 1.3, I could download a static Mumble server binary for Linux from the release section of the Mumble-voip GitHub project.
This changed with Mumble release v1.4: the Mumble project maintainers decided to no longer provide a static server binary for Linux.
While the most common Linux distributions provide official packages for Mumble (client and server), this is not the case for CentOS 7, which is the Linux distribution Uberspace is based on. Remark: the server component of Mumble is usually referred to as "murmur", which I will also do from here on.
With CentOS 7 not providing a murmur binary in its package sources, I could not get an up to date version of murmur to install on my Uberspace host. This posed a major problem for me and others who operate a Mumble server instance on Uberspace.
Shortly after discovering this problem, I created a GitHub issue and asked the Uberspace community for input on the topic. Quickly, the idea came up to compile a static murmur binary for CentOS 7 by ourselves. Although there were some doubts about the implications, like where to publish the binary and who would create new versions in case of newer Mumble releases, we wanted to give it a try. We thought: how hard could it be to compile a murmur server? The Mumble maintainers have done it up to v1.3 - so it must be possible.
It turned out we were rather naive. In the end it took almost one year to self-compile a static murmur binary for version 1.4. that would actually work.
Although the build files were still present in the Mumble sources, it was not easy to figure out which build dependencies are needed for compiling murmur and even worse, the dependencies have different names depending not only on the Linux distribution, but even on the version of the distribution used. Example: a build dependency which is named "dev-dependency-x" on Ubuntu 20.04, was not found on Ubuntu 22.04 under the same name.
Another problem was the simple lack of knowledge about the true meaning of the word static in "static murmur binary". I mean, the idea is clear: a static binary comes with all things it requires to run, instead of looking them up as shared libraries on the host system. The question was: how do you figure out if the built binary is a truly static one? Disclosure: I'm mostly a Java Developer for Cloud software and have only had some brief touch points with compiling executables from sources during my studies. Neither any peer in the Uberspace community (call them "Ubernauts") who commented on the GitHub issue seemed to be very familiar with compiling software for Linux.
I started to write a bash script that would execute the build steps and loaded it into a ubuntu docker container.
In the container, I executed the bash script and let it output the binary to a shared volume again, in order to access it on my host system.
This approach was rather stupid, because the compilation on my laptop took several hours and with every compilation error, the progress was lost.
A fellow Ubernaut suggested to implement the build instructions in a Dockerfile. The advantage: with every successful build step, docker would "cache" the progress in a new image layer and a restart of the compilation process would not have to redo every step from start but continue from the last successfully finished layer. However, the build still took around 4 hours on my 6 year old laptop (featuring 4GB RAM and an i3 CPU).
The careful reader already noticed, that so far only the compilation of a murmur binary was successful, which doesn't mean that the binary would also run on our targeted distribution (Uberspace/CentOS 7).
When trying to run the murmur binary, it directly crashed with an error message that some shared library libdns_sd.so.1 was not found. However, this was not the only shared library that was not found but required to run murmur on CentOS 7. So the next problem to solve was: how to compile a truly static murmur binary?
With some help from a friend of mine, I was able to solve this problem, although with some hackish workarounds.
First we executed the file command on the non-functional binary to get some information about it:
$ file mumble-server mumble-server: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=110d5e0000c7eb347a06ab5a6bc1bd72d8b0e537, for GNU/Linux 3.2.0, not strippedThis gave us the information that it is an executable and linkable file format (ELF) targeting a 64-bit x86 architecture. The file is dynamically linked and is compiled for running on Linux kernels in version 3.2.x (and above). The key information is the "dynamically linked" part of the output string. It told us that the binary is not a static one. To find out which components are only linked dynamically, we used the
ldd command:
$ ldd mumble-server
libcap.so.2 => /usr/lib/libcap.so.2 (0x00007fce31e38000)
libdns_sd.so.1 => /usr/lib/libdns_sd.so.1 (0x00007fce31e2d000)
...
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fce31a8f000)
...
The command revealed a list of linked libraries (the example above omits some output for brevity).
Dynamic libraries typically end with the .so suffix, which stands for "shared object". We had to replace them with their static version, ending on .a, which stands for "archive".
For building murmur, ninja is used and the Mumble sources already provide a build.ninja file, containing the references to the dynamic libraries to be used.
We applied a hackish workaround which surprisingly worked well:
RUN sed -i -e 's/\.so/.a/g' build.ninjaWe added an additional instruction to the Dockerfile to replace every
.so suffix in the build.ninja file by .a, before invoking ninja.
Now we got a statically linked binary:
$ file mumble-server
mumble-server: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=2fda729a9d550bf1698d015f0a80684b2a3e1a36, for GNU/Linux 3.2.0, not stripped
$ ldd mumble-server
not a dynamic executable
During the compilation of murmur, the most time is spent in compiling the Qt5 framework which is a direct dependency for Mumble. Another fellow Ubernaut mentioned that he used GitHub Actions for compiling a murmur binary. This already reduced the long build times.
There was even more room for improvement by revisiting, what dependencies are actually needed. It turned out that in the initial approaches, many Qt modules were compiled that might be needed for a full mumble build (including GUI components for the client), but are not needed at all for building the murmur server. In the end, only the qtbase module was required for building murmur. Building the binary with GitHub Actions also brought a performance boost which allowed to clone the sources, compile the dependencies and the final murmur binary in under 20 minutes.
In retrospect, the first approaches might sound naive, but they were all part of my learning process. It was at least as much fun as tedious to drive this project, and in the end I tend to say that my tenacious efforts paid out. Not only did I have a drop-in replacement for my current murmur server, but also I had learned quite some stuff about building software from source.
The static murmur linux server binary (v1.4.287) is available for download on my GitHub repository, from the release section. For further reference on how to run it, you may have a look at the Uberlab Mumble guide.
What about updates you ask? As long as I self-host a Mumble server, I will try to continue maintaining static murmur builds for future Mumble releases. However, I can only do this best-effort, since I will not only have to monitor new Mumble releases, but also updates on the dependencies like OpenSSL. Furthermore, I do not guarantee functional correctness of the murmur binary (although it should be pretty stable, I think).
If you are interested in further details, you can find the Dockerfile which I use to compile the static murmur binary on GitHub.
To the top ▲