Preparing the Virtual Reality course at ICG

For a while now a lot of my time working was spent on preparing the technical part of a Virtual Reality course at ICG. Since the setup was fairly complex I thought a review might be interesting.

• This write-up contains notes on fabric, puppet, apt, dpkg, reprepro, unattended-upgrades, synergy and equalizer.
• I worked with Daniel Brajko, Bernhard Kerbl and Thomas Geymayer on this project.
• This post was updated 5 times.

The setup

The students will be controlling 8 desktop-style computers (“clients”) as well as one additional desktop computer (“master”) which will be used to control the clients. The master is the single computer the students will be working on - it will provide a “terminal” into our 24 (+1) display videowall-cluster.

Each of the 8 computers is equipped with a current, good NVIDIA GPU (NVIDIA GTX 970) which powers 3 large, 1080p, stereo-enabled screens positioned vertically along a metal construction. The construction serves as the mount for the displays, the computer at its back as well as all cables. Additionally, each mount has been constructed to be easily and individually movable by attaching wheels to the bottom plate. The design of said constructions, as well as the planning, organization and the acquisition of all components was done by Daniel Brajko. (You can find a non-compressed version of the image here.)

Preparation

I could go into detail here, how my colleague has planned and organized the new Deskotheque (that the name of the lab) as well as overseen the mobile mount construction. However, since I am very thankful for not having to deal with both shipping as well as assembly, I will spare that part. Instead I will tell how one of our researchers and I scrambled to get a demo working within little to no time.

All computers were set up with Ubuntu 14.04. We intended to use puppet, which was initially suggested by Dieter Schmalstieg, the head of our institute, from the start. At that time our puppet infrastructure was not yet ready, so I had to set up the computers individually. After installing openssh-server and copying my public key over to the computer I used Python fabric scripts I’ve written to execute the following command:

fabric allow_passwordless_sudo:desko-admin 
  set_password_login:False change_password:local -H deskoN

This command accessed the host whose alias I had previously set up in my ~/.ssh/config. The code for those commands can be found on Github. The desko-admin account has since been deleted.

A while later our puppet solution was ready and we connected those computers to puppet. There is a variety of tasks that is now handled by puppet:

• the ICG apt repository is used as additional source (this happens before the main stage)
• a PPA is used as additional apt source to enable the latest NVIDIA drivers (this happens before the main stage)
• NVIDIA drivers, a set of developer tools, a set of admin tools, the templates, binaries and libraries for the VRVU lecture are installed.
• unattended_upgrades, ntp, openssh-server are enabled and configured.
• apport is disabled. (Because honestly, I have no clue why Ubuntu is shipping this pain enabled.)
• deskotheque users are managed
• SSH public keys for administrative access are distributed

Demos

First impression

If you don’t care for ranting about Ubuntu, please skip ahead to moving parts, thank you. Setting up a different wallpaper for two or more different screens in Ubuntu happens to be a rather complicated task. For the first impression I needed to:

• log in as desko-admin
• create the demo user account
• have demo log in automatically
• log in via SSH as desko-admin
• add PPA for nitrogen
• install nitrogen and gnome-tweak-tool
• copy 3 distinct pictures to a given location on the system
• log in as demo
• disable desktop-icons via gnome-tweak-tool
• set monitor positions (do this the second time after doing it for desko-admin because monitor positions are account-specific. This, btw, is incredibly stupid.)
• set images via nitrogen (because who would ever want to see two different pictures on his two screens, right?)
• disable the screen saver (don’t want people having to log in over and over during work)
• enable autostart of nitrogen (that’s right, we are only faking a desktop background by starting an application that runs in the background)

Only after this had been done for every single computer, a big picture was visible: all the small images formed one big photograph and made an impressive multi-screen wallpaper - at least if you stood back far enough not to notice the pixels. Getting a picture that’s 3*1080 x 8*1920 is rather hard, so we upscaled an existing one.

The result of this pain is: One switches on all computers and they all start displaying parts of the same picture, logged in via the same account. You can immediately start a demo using all screens with this user. (This procedure was made even more simple by having puppet deploy SSH public and private keys for this user - so you instantly jump from one deskotheque computer to another if you’re demo.)

Moving parts

For the first big demo for a selected number of people during WARM 2015 I worked together with Thomas Geymayer which is the main developer of our in-house fork of synergy on setting up said program. It took us some attempts to get everything working in the first place since he had used Ubuntu 14.10 for development. The cluster however used the current 14.04 LTS I had rolled out earlier. Since by then the puppet solution wasn’t ready, we spent two frantic days copying, trying, compiling, trying again and copying via SFTP between the individual nodes in order to get everything to work properly. Thomas had to rework some of the implementation since our fork was originally invented for presenting, not remote-control of several devices which he did in admirably little time. Though we had some issues during the presentation the attendees seemed interested and impressed by our setup.

Soon after that deadline I prioritized finishing our puppet solution since I got very, very annoyed manually syncing directories.

Equalizer

Bernhard Kerbl wanted to work with the Equalizer framework in order to enable complex rendering tasks. Each of the computers in the cluster is supposed to compute a single part of the whole image (or rather 3 parts given that 3 monitors are connected to each node). The parts of the whole image must be synchronized by the master, so that the whole image makes sense (e.g. no parts of the image may be further ahead in a timeline than the others). Usually I expect bigger projects to either offer Ubuntu packages, prebuilt Linux binaries or even a PPA. Their PPA doesn’t offer packages for the current Ubuntu LTS though, so we ended up compiling everything ourselves.

That took a while, even after figuring out that one can make apt-get and use Ubuntu packages instead of compiling libraries like boost from source. After some trial and error we arrived at a portable (by which I mean “portable between systems in the cluster”) solution. I packaged that version using fpm. Since the students will be using the headers and libraries in the framework we could not simple ship that package and be done with it, we also had to ensure that everything could be compiled and run without issue. The result of that is a package with equalizer libraries and almost everything else that was built which has a sheer endless list of dependencies since we had to include both buildtime and runtime dependencies.

In order to package everything, we installed all the depencies, built out of source and packaged everything with fpm.

fpm \
-t deb \
-s dir \
--name "vrvu-equalizer" \
--version "1.0.1" \
--license "LGPL" \
--vendor "ICG TU Graz" \
--category "devel" \
--architecture "amd64" \
--maintainer "Alexander Skiba <skiba@icg.tugraz.at>" \
--url "https://gitlab.icg.tugraz.at/administrators/script-collection" \
--description "Compiled Equalizer and dependency libraries for LV VRVU
" \
--exclude "vrvu-equalizer.sh" \
--exclude "opt.zip" \
--verbose \
-d debhelper \
-d dh-apparmor \
-d gir1.2-gtk-2.0 \
-d icu-devtools \
-d libaacs0 \
-d libarmadillo4 \
-d libarpack2 \
-d libatk1.0-dev \
-d libavahi-client-dev \
-d libavahi-common-dev \
-d libavahi-compat-libdnssd1 \
-d libavcodec-dev \
-d libavcodec54 \
-d libavdevice53 \
-d libavformat-dev \
-d libavformat54 \
-d libavutil-dev \
-d libavutil52 \
-d libbison-dev \
-d libblas3 \
-d libbluray1 \
-d libboost-date-time1.54-dev \
-d libboost-program-options1.54-dev \
-d libboost-program-options1.54.0 \
-d libboost-regex1.54-dev \
-d libboost-regex1.54.0 \
-d libboost-serialization1.54-dev \
-d libboost-serialization1.54.0 \
-d libboost-system1.54-dev \
-d libboost1.54-dev \
-d libc6 \
-d libcairo-script-interpreter2 \
-d libcairo2-dev \
-d libcoin80 \
-d libcv-dev \
-d libcvaux-dev \
-d libdap11 \
-d libdapclient3 \
-d libdbus-1-dev \
-d libdc1394-22 \
-d libdc1394-22-dev \
-d libdrm-dev \
-d libepsilon1 \
-d libexpat1-dev \
-d libfaad2 \
-d libfl-dev \
-d libfontconfig1-dev \
-d libfreetype6-dev \
-d libfreexl1 \
-d libgdal1h \
-d libgdk-pixbuf2.0-dev \
-d libgeos-3.4.2 \
-d libgeos-c1 \
-d libgfortran3 \
-d libgif4 \
-d libglew-dev \
-d libglewmx-dev \
-d libglib2.0-dev \
-d libglu1-mesa-dev \
-d libgraphicsmagick3 \
-d libgsm1 \
-d libgtk2.0-dev \
-d libgtkglext1 \
-d libharfbuzz-dev \
-d libharfbuzz-gobject0 \
-d libhdf4-0-alt \
-d libhdf5-7 \
-d libhighgui-dev \
-d libhwloc-plugins \
-d libhwloc5 \
-d libibverbs1 \
-d libice-dev \
-d libicu-dev \
-d libilmbase-dev \
-d libilmbase6 \
-d libiso9660-8 \
-d libjasper-dev \
-d libjbig-dev \
-d libjpeg-dev \
-d libjpeg-turbo8-dev \
-d libjpeg8-dev \
-d libkml0 \
-d liblapack3 \
-d liblzma-dev \
-d libmad0 \
-d libmail-sendmail-perl \
-d libmng2 \
-d libmodplug1 \
-d libmp3lame0 \
-d libmpcdec6 \
-d libmysqlclient18 \
-d libnetcdfc7 \
-d libodbc1 \
-d libogdi3.2 \
-d libopencv-calib3d-dev \
-d libopencv-calib3d2.4 \
-d libopencv-contrib-dev \
-d libopencv-contrib2.4 \
-d libopencv-core-dev \
-d libopencv-core2.4 \
-d libopencv-features2d-dev \
-d libopencv-features2d2.4 \
-d libopencv-flann-dev \
-d libopencv-flann2.4 \
-d libopencv-gpu-dev \
-d libopencv-gpu2.4 \
-d libopencv-highgui-dev \
-d libopencv-highgui2.4 \
-d libopencv-imgproc-dev \
-d libopencv-imgproc2.4 \
-d libopencv-legacy-dev \
-d libopencv-legacy2.4 \
-d libopencv-ml-dev \
-d libopencv-ml2.4 \
-d libopencv-objdetect-dev \
-d libopencv-objdetect2.4 \
-d libopencv-ocl-dev \
-d libopencv-ocl2.4 \
-d libopencv-photo-dev \
-d libopencv-photo2.4 \
-d libopencv-stitching-dev \
-d libopencv-stitching2.4 \
-d libopencv-superres-dev \
-d libopencv-superres2.4 \
-d libopencv-ts-dev \
-d libopencv-ts2.4 \
-d libopencv-video-dev \
-d libopencv-video2.4 \
-d libopencv-videostab-dev \
-d libopencv-videostab2.4 \
-d libopencv2.4-java \
-d libopencv2.4-jni \
-d libopenexr-dev \
-d libopenexr6 \
-d libopenjpeg2 \
-d libopenscenegraph99 \
-d libopenthreads-dev \
-d libopenthreads14 \
-d libopus0 \
-d libpango1.0-dev \
-d libpci-dev \
-d libpcre3-dev \
-d libpcrecpp0 \
-d libpixman-1-dev \
-d libpng12-dev \
-d libpostproc52 \
-d libpq5 \
-d libproj0 \
-d libpthread-stubs0-dev \
-d libqt4-dev-bin \
-d libqt4-opengl-dev \
-d libqt4-qt3support \
-d libqtwebkit-dev \
-d libraw1394-dev \
-d libraw1394-tools \
-d librdmacm1 \
-d libschroedinger-1.0-0 \
-d libsm-dev \
-d libspatialite5 \
-d libspnav0 \
-d libswscale-dev \
-d libswscale2 \
-d libsys-hostname-long-perl \
-d libtbb2 \
-d libtiff5-dev \
-d libtiffxx5 \
-d libudt0 \
-d liburiparser1 \
-d libva1 \
-d libvcdinfo0 \
-d libx11-doc \
-d libx11-xcb-dev \
-d libx264-142 \
-d libxau-dev \
-d libxcb-dri2-0-dev \
-d libxcb-dri3-dev \
-d libxcb-glx0-dev \
-d libxcb-present-dev \
-d libxcb-randr0-dev \
-d libxcb-render0-dev \
-d libxcb-shape0-dev \
-d libxcb-shm0-dev \
-d libxcb-sync-dev \
-d libxcb-xfixes0-dev \
-d libxcb1-dev \
-d libxcomposite-dev \
-d libxcursor-dev \
-d libxdamage-dev \
-d libxdmcp-dev \
-d libxerces-c3.1 \
-d libxext-dev \
-d libxfixes-dev \
-d libxft-dev \
-d libxi-dev \
-d libxine2 \
-d libxine2-bin \
-d libxine2-doc \
-d libxine2-ffmpeg \
-d libxine2-misc-plugins \
-d libxine2-plugins \
-d libxinerama-dev \
-d libxml2-dev \
-d libxml2-utils \
-d libxrandr-dev \
-d libxrender-dev \
-d libxshmfence-dev \
-d libxvidcore4 \
-d libxxf86vm-dev \
-d mesa-common-dev \
-d mysql-common \
-d ocl-icd-libopencl1 \
-d odbcinst \
-d odbcinst1debian2 \
-d opencv-data \
-d po-debconf \
-d proj-bin \
-d proj-data \
-d qt4-linguist-tools \
-d qt4-qmake \
-d x11proto-composite-dev \
-d x11proto-core-dev \
-d x11proto-damage-dev \
-d x11proto-dri2-dev \
-d x11proto-fixes-dev \
-d x11proto-gl-dev \
-d x11proto-input-dev \
-d x11proto-kb-dev \
-d x11proto-randr-dev \
-d x11proto-render-dev \
-d x11proto-xext-dev \
-d x11proto-xf86vidmode-dev \
-d x11proto-xinerama-dev \
-d xorg-sgml-doctools \
-d xtrans-dev \
-d zlib1g-dev \
.

In the last weeks before this article, I’ve seen a 3D rendering on almost all screens of the cluster which was great. I enjoy seeing people use systems I helped building.

Puppet: apt or dpkg

Having a prepared .DEB file didn’t solve all my trouble though. I had two options for installing the file via puppet: apt or dpkg. Well, this was troubling. dpkg does not understand dependencies if used in this way - a bad thing given that the dependencies of our vrvu-equalizer package were a pretty long list. apt however didn’t offer to use a source parameter - therefore we had to offer a way to install the package from a repository.

After a bit of research I decided to set up an in-house repository for the institute, hosting those packages which we cannot comfortably use from other sources. At the time of this writing it holds patched versions of unattended-upgrades for Trusty, Precise, Wheezy and Jessie as well as our vrvu-equalizer version for Trusty. (I recommend against using our repository for your computers since I haven’t found the time to repair the slightly broken unattended-upgrades for systems other than Jessie.)

deb https://data.icg.tugraz.at/packages <codename> main

I created the repository using reprepro and we sign our packages with the following key: https://data.icg.tugraz.at/packages/ICG-packages.key.

Unattended-upgrades

I’ve automated installation of upgrades on most of our Linux based machines at the institute mostly due to the fact that I don’t want to babysit package upgrades when security critical updates are released. *cough* openssl *cough* However, I’ve run into one problematic issue. I’ve run out of space on the /boot partition due to frequent kernel updates which don’t remove the previous kernels.

I’ve since set the Remove-unused-dependencies parameter, but that didn’t do everything I wanted. This parameter only instructs the script to remove dependencies that happen to be no longer needed during this run. Dependencies which were “orphaned” before the current run will be ignored. This means that manual upgrades have the potential to lead to orphaned packages which remain on the system permanently.

Since the unattended-upgrades script is written in Python, I took a stab at implementing the functionality I wanted to have for use with our installations. After I had done that, I packaged everything for Ubuntu Precise Pangolin, Ubuntu Trusty Tahr and Debian Wheezy and put everything in our ICG apt repository to have it automatically installed.

Unattended-upgrades, again

A review of my previous modification to unattended-upgrades was necessary since root kept getting mail from the cronjob associated with unattended-upgrades even though I had specifically instructed the package via puppet to only do so in case of errors. Still, every few days, we would get emails containing the output of the script. Here’s an example.

/etc/cron.daily/apt:
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
(Reading database ... 117338 files and directories currently installed.)
Preparing to replace subversion 1.6.17dfsg-4+deb7u8 (using .../subversion_1.6.17dfsg-4+deb7u9_amd64.deb) ...
Unpacking replacement subversion ...
Preparing to replace libsvn1:amd64 1.6.17dfsg-4+deb7u8 (using .../libsvn1_1.6.17dfsg-4+deb7u9_amd64.deb) ...
Unpacking replacement libsvn1:amd64 ...
Processing triggers for man-db ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
Setting up libsvn1:amd64 (1.6.17dfsg-4+deb7u9) ...
Setting up subversion (1.6.17dfsg-4+deb7u9) ...

I am currently in the process of solving this by rewriting my modification in a cleaner, more structured way - a way which is a lot more influenced by the original script, keeping in mind that the necessary environment variable for debconf is set in the execution path.

My initial error with this was that cache.commit() in the script immediately applied all changes made to the cache. While I intended to only apply the deletion of marked packages at the point of my call to the method, this meant that all changes got applied - even those for installing/upgrading new packages. The script returned prematurely and stdout got written to. This in term meant that root would get mail, since root always receives mail of cronjobs produce output.

Update 1: While my current progress does no longer call commit prematurely, it still sends me e-mails. I probably forgot to return True somewhere.

Update 2: In the meantime I think I fixed that issue by returning the success status of the auto-removal process and assigning it to the pkg_install_success variable if it does not already contain an error.

Update 3: Fixed every issue I found and submitted a pull request on Github. However, I don’t know if it will be accepted since I implemented my preferred behaviour instead of the old one. I am not sure whether I should’ve added an additional parameter instead.

Update 4: Pull request was merged. Unfortunately I will be stuck patching my older systems, however.

Update 5: The change in behaviour implemented by me has been cherry-picked for both Ubuntu Trusty and Ubuntu Precise, both currently active LTS versions during the time of this writing, so I’m quite proud of my contribution having such a great reach and have removed the patched versions from the ICG repositories.