Robin Powell

Senior UNIX/Linux Systems Administrator

robin@digitalkingdom.org

Objectives

I am a UNIX System Administrator, and have been for my entire adult life. While I am not normally seeking a new position, I am always open to particularily exceptional opportunities. To my mind, "exceptional opportunities" involve either extraordinary compensation now, or realistic promises of really extraordinary compensation in the future. I am especially interested in the challenge of building a company's infrastructure from the ground up (in other words, working for an early stage startup).

If the tone of the preceding paragraph surprises you, please understand that I am looking for an environment that I fit really, really well with. This includes a strong desire to avoid personality conflicts before they happen. Among other things, that means that I'm looking for a company where I can happily work with everyone else at making us all rich, because if you're not in business to make money, really, what's the point?

I am a very fast learner, and a very fast worker. I also excel at complicated trouble-shooting. Please don't take my word for it; give me a working interview. Make me configure Apache from scratch or debug a machine with a one character error in a config file or something. Just make sure the machine has man pages, because I'm offering you speed and skill, not an eidetic memory.

In terms of particular facets of sysadminning I'm interested in, I enjoy security related work (both in the sense of doing it myself and in the sense of being part of a small company that provides computer security products or services). I'd love to be involved in serious AI work. I've also spent time off and on being a de facto DBA, and have enjoyed it immensely.

Please note that I am not interested in moving away from the San Francisco bay area, nor in taking a position far outside of San Francisco proper, where I live. I am also not interested in contract positions, although remote consulting is certainly a possibility.

Skills

Each list is (more or less) priority ordered: the closer to the beginning of the list, the more recent and thorough my skill is with that thing.

• Networking: DNS, TCP/IP, NFS, SMTP, Samba, NNTP, DHCP, Ethernet networking, routing, Cisco routers and switches, NIS, NIS+, PPP, F5 3DNS, F5 BigIP
• Software Administration: Apache, E-Mail ( Sendmail, exim, a bit of postfix), Request Tracker, Subversion, RCS, CVS, Veritas Volume Manager, SMB, samba
• Hardware Administration: basically every recent x86 architecture with an emphasis on AMD chips, Sun Solaris machines, NetApp filers
• Markup Languages/Web Authoring: HTML, XHTML, CSS, CGI (Perl, PHP, others), LaTeX
• Protocols: HTTP, SMTP, PXE booting, DHCP
• Operating Systems: Debian Linux, Windows 2000/XP, Solaris, SunOS, RedHat Linux, NetBSD, FreeBSD
• Security: SSH, SSL, logchecker, chkrootkit, disaster recovery, post-incident forensics, PGP, GPG, one-way hashes, network security, Snort, cryptography theory
• Programming Languages: UNIX shell scripting, Scheme, SQL, PHP, Perl, C, Oz, Haskell

In addition to the above, I have substantial skill at porting programs (even very badly written ones) from one UNIX to another. I am able to perform well under pressure. I have excellent written and oral communication skills.

Work Experience

Job Description

I am a member of the Production Operations team, which is responsible for all customer-facing computer systems. I am personally responsible for all of the machines that run www.furl.net (about 25 machines of about six types, in terms of software) and a few medium sized web search engine clusters (between 10 and 100 machines, depending on the cluster). All of them are running Debian Linux on x86 hardware, mostly AMD, and mostly 64-bit capable. A fair complement are now running Debian amd64, something I hepled drive forward.

Duties And Responsibilities

• Maintenance of the www.furl.net infrastracture and several of our back-end search server pools. I have basically single-handedly turned Furl from something that kept oncall awake 2 or 3 times a week to something that never causes any trouble at all. This included MySQL server optimization, query optimization, capacity planning, server buildouts, and various kinds of scripting, including via cfengine.
• Imaging and deployment of new machines as necessary, including appropriate DNS modifications and adding servers to our centralized management pool.
• I have been almost entirely responsible for the suit of scripts that turn a machine from a freshly imaged bare Debian box into a complete server ready to fill a particular role. I have configured something on the order of 6 different classes of servers to immediately fix themselves in this fashion upon first bootup, based on the name of the server. Much of this is done via cfengine rules; the rest is shell scripting.
• LookSmart manages something on the order of 800 machines via cfengine. While I was not the person who original configured cfengine, I am responsible for a great deal of the current cfengine scripts, and am acknowledged as the secondary expert on the topic here.
• Deployment of new software from our development group to our various servers with a minimum of user-visible downtime (normally we manage to acheive none at all).
• Sharing in the on-call rotation; carrying a pager.
• Working with development to test and debug production outages and issues.

Please note that Symantec bought Recourse, so I never actually left my job at Recourse at all. I was moved from the former Recourse offices in Redwood City to Symantec headquarters in Cupertino.

Job Description

I was a member of the corporate IT team, which had a total of about 12 members. My particular group had 8 members when I left, one of whom is the primary admin for our mail and DNS infrastructure (he predates my joining the company), and it is this person that I worked most closely with.

I was the secondary admin for the corporate wide e-mail infrastructure (at least on the UNIX side; Lotus Notes is the only allowed mail client, so there are a lot of Notes servers on Windows boxes), as well as the DNS infrastructure and a lot of miscellaneous machine (more than 50).

Duties And Responsibilities

• Being the primary admin for any new RedHat servers (the installation of RedHat servers being a new initiative here).
• Dealing with most DNS change requests. This includes managing F5 3DNS and BigIP machines.
• Dealing with the ListServ servers, which handle some very large mailing lists (I've seen a flat file for the subscriptions to a list be larger than 35MB).
• Being part of the on-call rotation, which can get very hectic. One call I was on lasted 15 hours.
• Formulating purchasing requests for new hardware.
• Installation and upgrading of Solaris and Linux servers, both at the operating system and application level.
• Management of the corporate IPlanet/Sun ONE web servers.

Job Description

I was the sole company-wide UNIX administrator in a company of about 150 employees, about half of which are engineers on Linux boxes.

Duties And Responsibilities:

• Many day-to-day problem solving and user assistance activities, such as Solaris and RedHat troubleshooting, setting up out-of-office messages, e-mail troubleshooting and the like.
• Installation of Solaris x86 on Dell machines that Solaris really wasn't designed for and is not certified on, and then making images of those machines for quick shipment to customers of working Solaris x86 boxes. Once built 44 such machines in a day.
• Planned and deployed a new ticketing system for the IT department, using RT: Request Tracker.
• Maintained a RedHat Linux file server, which used Samba to serve files to mostly Windows 2000 clients. Retired said server and replaced it with a Maxtor MaxAttach 4300, running Windows 2000. Now administering that machine.
• New computer setup, currently RedHat 7.2 on Dell machines.
• Testing computer configurations. Everything from RedHat 7.2 to Solaris x86 2.6 on a Sun Ultra 2 to Solaris 8 on a Sun E-450.
• Computer trouble shooting, both preemptive and reactive.
• System administration in a mixed Solaris/Linux environment.
• Shell scripting.
• NIS, NFS, routing and DNS administration.
• Sendmail configuration, including mail virus protection.
• Internal Netscape server administration.
• Unix tools installation and configuration.
• User and group administration.

Job Description

I was a salaried, full-time employee of Taos. They then contracted me out to their clients as a consultant. Due to the nature of this arrangement, I got paid for overtime. Several of the job descriptions below are for clients of Taos.

Duties And Responsibilities:

• Computer trouble shooting, both preemptive and reactive.
• Informal UNIX training of PC and Oracle administrators.
• System administration in a Solaris environment.
• Firewall-1 configuration.
• Shell scripting.
• NIS, NFS, routing and DNS administration.
• Backup tape swapping.
• Sendmail configuration, including mail virus protection.
• Internal Netscape server administration.
• Unix tools installation and configuration.
• User and group administration.

Duties And Responsibilities:

• Computer trouble shooting, both preemptive and reactive.
• System administration in a Solaris environment.
• NIS, NFS and DNS administration.
• NetApp disk array management.
• Alexandria backup software management.
• Backup tape swapping.
• Unix tools installation and configuration.
• User and group administration.

Prior to March of 2000 I lived in Canada, and held various System Administrator, DBA, and programming positions going back to early 1995. My first position formally listed as systems administation started in 1998. More detailed information about this time period is available upon request.

Personal Programming Projects

• I've written a TUI client for the XMMS2 music server called Cluck.
• I've made a multilingual patch for the mooix MOO server, which so far as I'm aware turns it into the first fully multilingual MUD ever.

Systems Administration Writeups

• I have a method of personal data backup that seems to be fairly novel, in that it has all of the following properties: I back up multiple OSes on multiple machines, I synchronize it offsite, the offsite backup is encrypted, I don't have to synch everything every time I backup (which is daily, by the way) and it's free (except for the accounts).
• I've written a document on something that has been specifically declared impossible (look for "Can I `apt-get arch-upgrade` from an already installed i386 debian?"), which is always fun. I've written up how to upgrade a Debian system from i386 to amd64. This is very much not for the faint of heart.

Other Experience

Duties And Responsibilities:

I provide both the server and network connection for the lojban.org infrastructure, at my own expense. Also, I support a wide complex of tools for a variety of projects. The lojban.org infrastructure currently includes:

• An Apache web server environment, running TikiWiki, which is written in PHP, is locally hacked by me, and uses MySQL on the back end. Request Tracker (RT) is used for user request tracking, although most actual work gets done due to IRC and mailing list requests as I am in practice the only admin. TikiWiki has the ability to synchronize a mailing list and a web-based forum; this ability was broken when I found it, but I fixed it and now use it extensively.
• A home-brewed multi-lingual dictionary editing system written in Perl using Mason, with PostgreSQL as a back-end. I wrote about half of it (I took it over after the original author got too busy, and it was very much not finished at the time). The dictionary editing system fully supports UTF-8 input, as does many of the rest of the lojban.org systems. It even includes a DICT-based search tool (which was not written by me).
• A quite a few mailing lists, all of which are handled by Ecartis over an Exim mail server. Most of them are archived using MHonArc.
• A public CVS server, used for group text editing of translation files, and a public Subversion server used for a few programming projects.
• The world's only fully multilingual MUD. Thanks to my programming efforts on the mooix core, commands and descriptions of all kinds can be entered in any language. The actual Lojban MOO allows full use of all features in both Lojban and English.
• A wide variety of miscellaneous services (SSH, DNS, IceCast, MySQL, etc).

Duties And Responsibilities:

• User Training.
• Handling user requests that involved system administration.
• Keeping the machines stable and other routine system administration tasks.
• Keeping the members of the club happy with the computing facilities we provided.
• News server administration.

Education And Certifications

Solaris 7 System Administration I

As an example of my technical background and learning speed: I completed the online course for this certification in about 10 hours. 2 months later, I reviewed for about 4 hours, completed the 2 hour exam in 50 minutes, and got 84%.

Bachelor Of Mathematics

I was awarded a Bachelor Of Mathematics by the University of Waterloo in 2002, having begun the program in 1994. It took that long because in 1998 I went to work full time, and finished my remaining courses slowly by correspondance.

Other

References available upon request.