Work as a part-time system admin

I’ve wanted to write about my work at university for some time now but I’ve never really sat down and took the time to do that until now.

There are various things I do at the institute, some of which I don’t understand why they are my responsibility. Some of them I enjoy. Others I don’t, which is something very typical for any job, I suppose.

I am a part-time system administrator at the Institute for Computer Graphics and Vision.

Server Maintenance

Whether it’s searching lost files, looking through logs to find specific, maybe even security critical events or just reconfiguring an existing piece of software, maintaining the servers of the institute has become my job - that is if you subtract the physical maintenance and the purchasing process. I am extremely thankful that those areas are covered by my friendly full-time colleagues.

I try my best to investigate errors in logs, particularly recurring ones when there is time to spare.

Configurations

Obviously there’s a lot of command-line fu involved.

In the last few months I changed some processes which were occurring regularly:

1. Changed our servers to use unattended-upgrades where possible
2. Set up more reasonable e-mail notifications

Moreover, I blocked the 360Spider bot from constantly requesting files from our web server. I set up an OpenVPN server with guidance from Thomas. I configured a system in which an encrypted partition is automatically unlocked and mounted on login if you belong to the correct user group (I’m not sure whether I should be proud of that given the incredible hackyness of said system).

Currently I am looking into puppet for my next assignment. I’ve heard great things from friends about the software but my first experience was creating a VM for testing and realizing that the apt puppet module is currently broken (Dec 15th 2014).

Documentation

When I arrived I was pretty shocked to see that there was absolutely no internal documentation at all. I am working to remedy this situation whenever I have spare time between more immediate assignments. Since I consider the documentation my primary project I made all the important decisions myself. It’s written in Markdown and we are using our internal gitlab platform to host and distribute it. Gitlab provides a nice “edit” button too, for those unfamiliar with git.

As of the time of this writing, there is documentation about the most common workflows, some server configuration, deployment notes and command line hints.

Additionally I’ve written a Getting Started guide for new members of our institute in order to avoid explaining everything to every new person again (and again in case something was unclear).

Building stuff

Creating ways of interweaving technologies is easily my favorite task. I like to write scripts to automate laborious tasks that have to be done. On the other hand I am also looking for challenges in which proven ways have to be reassembled to fit a client’s needs (though they are not directly paying me, the members of the institute are “clients” in the sense that it’s my job to make their ideas or wishes in terms of infrastructure work).

In practice that means I’ve written several scripts and am in the process of rewriting most of the tools to use the awesome Fabric module for Python. This particular direction was inspired by my other friend Thomas, who suggested just the right kind of tool for my work - a tool which profits from my profound joy working with the Python language. Except when it comes to the byte string/unicode string problem in Python 2.x. My collection of administrative helpers is located at my github repository since I’ve liberated it from our internal gitlab.

It would’ve been easier - and probably more comfortable - to just stick the configuration in the scripts themselves instead of reading everything from JSON files. That would’ve meant at least three things I was uncomfortable with:

1. Sharing the code outside of our ICG admin team would have been impossible due to the risk of compromising confidential data. I preferred to share because I think that it’s hugely beneficial for any IT worker, be them programmers, sysadmins or similar to have a presentable portfolio of their work.
2. Asking peers for advice would have been impossible due to the same risk. I don’t consider myself a superior programmer and therefor like to get the opinion of my peers every so often to improve the quality of my code.
3. Hardcoding data where it is not strictly necessary feels unclean to me.
4. It would’ve been way harder for anyone who might like my work to use it themselves. I immensely dislike working against the Open Source idea where it’s so obviously unnecessary.

In combination with the work done on a server configuration project involving a cryptographic setup for groups I’ve also scripted a rather convoluted process of setting up new users for said system.

Administrative Chores

I’m not entirely sure why this belongs in my domain but I’m routinely tasked with entering content in our CMS of choice, Plone. That wouldn’t be as annoying of our instance of the system did not feel that broken and slow. Hm, I almost forgot “confusing”. Never had imagined that simply putting up a job offer needed so much administrative overhead just in a CMS.

I don’t have a lot of love for any database driven CMS, now that I think of it. Still, due to laziness I have not taken the time to move my personal site from WordPress to something like Jekyll yet.

And of course, there’s the usual “enter user X into the system, please” because others don’t have the same permissions that the system admin has. Cue “I am root” joke here. Actually, don’t do that. I did that once. Made a terrible mistake less than an hour later.

Tech (and Moral) Support

So you fix our computers, right?

Given you have acquired a certain knowledge of computers, operating systems and software over the years you will be tasked with fixing or configuring things that your co-workers simply cannot manage to do themselves. That’s okay. Sometimes you won’t be able to find the bugs, fix the errors or configure their thing to work. That’s okay too.

Your colleagues want you to try your best - if you manage to do the impossible on the way that’s great. If you’ve obviously done your best and invested multiple hours into research and experimentation concerning their problems, it’s very likely they will understand that it’s not possible for you to smooth out every little itching. And every once in a time, they’ll want your advice or input on a problem they’re trying to solve. You might know something. You might even guess something right - it’s not important. You’re there, supporting them with their issue. Maybe that will be enough; I’ve personally had more than a handful of these occasions during my half year at the institute.

People in my life know I’m good with computers. And they come to me and ask for advice. I can see the pain. They’ve been hurt. They want a savior.
And I’m tired. And I’m busy. And there’s so much to say. So much to teach. So much to do.
And I don’t want to be their savior right now.
And that hurts, too. ~SwiftOnSecurity

This passage from SwiftOnSecurity manages to catch my opinion on this issue pretty well. I’ll try to help everyone given the time but I sometimes I need my colleagues to understand that it’s outright impossible for me to be working on their issue right now. There may be more pressing problems, say I might have rm-ed a file we still needed or I’m in the middle of a project already.

I realized I need to work on my communication skills and the timing of e-mails in order to minimize stress - both for me and others.