I am a web developer at UCSF Public Affairs. At the time I started this job in January 2001 we had a Solaris box managed by the IT department and a Mac Linux box run internally. I got a better Linux box and managed it for staging/development purposes, but became frustrated because 1) I couldn't get root on the production Solaris box; 2) Any non-urgent requests regarding that server were slow to be answered as we weren't actually paying the IT department for hosting; and 3) I couldn't get on-campus help managing the Linux box, and didn't feel confident that I could keep it running and secure, despite having taken an excellent Linux sysadmin class and having tried a few versions of Linux myself at home.
After several years of this situation I tried to get a real service level agreement with the IT department for better support, but the price they quoted was way higher than we could afford. So I approached another department, but they only supported Macs. I decided to take the plunge and recommend replacing our Solaris and Linux box with two identical Xserves.
The switch happened, and I was able to get all our software running on the new platforms. But the new department wasn't able to support me doing anything other than basic functions that were accessible through the Mac OS GUI. So I was on my own to do anything more advanced.
Getting help, or not
After a year of getting increasingly frustrated whenever I tried to do difficult installs by myself, like Apache with mod-ssl which was very strongly requested by one of our clients, I finally got a consultant hired. I was very much looking forward to working with them after all the red tape we had to go through to retain the company. Unfortunately the person assigned to me ended up being a total flake, not returning my e-mails for days on end and not getting any actual work done whenever we did speak, just saying he had to do more research and call me back. After a month of this runaround we finally scheduled a date to install the secure server, and he backed out at the last minute.
The company owner apologized and said he'd try to get us someone else, but that it was hard to find consultants willing to do this kind of work. I said that in the month I'd been waiting for this guy I probably could have figured out to install the secure server myself, and it was just one thing in a long list of things to do. So I said I would try again to do just that, and maybe use this consulting service of one of the less-urgent, long-term projects further down the list.
Meanwhile, at the movies...
All the time I was struggling with wearing a sysadmin hat in addition to my regular web programming/updating duties, I was trying to learn how to be a video editor, with no prior experience in multimedia at all. I actually enjoyed the effort, having made a double-DVD set of my own wedding in 2004, and progressed quickly from iMovie to Final Cut Express to Final Cut Pro. However it took quite a bit of time to edit each video clip, as I not only had to learn the software but sit down with the person writing the story and decide how to do the editing.
Then our division got a new Associate Vice Chancellor, who announced that podcasting was the latest and greatest thing. I had never to that date listened to a podcast, not understanding what the point of them was. But around the time we had an office retreat in November, I did start listening, most notably to This Week in Tech and its various spinoffs, and got hooked. So I had to learn how to do that too.
We've only produced two podcasts so far, and the second one was a real bear to edit. I was using software that I really wasn't skilled in (Soundtrack Pro), and the writer had me cutting out individual syllables and rearranging whole chunks of text. Plus, we were using inferior equipment, because A/V hardware and strategies were another area I was hoping to get consulting help in.
Nevertheless, we came through. But I still had to deal with figuring out how to use the new audio recording accessories that boyziggy very generously helped me find, and how to replace our aging and somewhat broken camcorder when most of the online stores I visited didn't take purchase orders.
Headless at the helm
And did I mention that during this time my supervisor, who I absolutely loved and got along superbly with and was one of the main reasons I didn't quit in the first year and run back to Berkeley, left for another job. This was over a year ago counting his internship there (before he officially resigned), and we have yet to replace him. So our communications director has to wear two hats, and I have to struggle because I've discovered one thing I really suck at is project management. I love the freedom to work on different things every day, but I really need someone guidance to help me prioritize.
We got as far as final candidate interviews, but then ended up having to re-list the job. I hope we get someone soon, but "soon" in University terms could be months, or years.
The last minute request
And to add to my burdens, our newest client, with a high-profile site, mentioned somewhat late in the game that they needed to access an Oracle database on another server, thus requiring me to download Oracle libraries (which required me to register as a developer with Oracle, not fast or fun) and rebuild PHP for the purpose. I was not obligated to do this as it was not part of our agreement (the person who approached me and signed the contract didn't realize these additional steps would be needed), but I figured I would give it a good try and get consulting help if it didn't work.
Hours of banging my head against the desk later, I e-mailed the client to tell him about our consulting woes and ask when he planned to launch, and how critical this component was.
In between e-mails about this I was working on yet another project. To organize all of our video clips, I had created a database with a web administration interface some six months ago, but later realized that I had programmed it so badly it really needed to be redone from scratch. As we have a home page redesign coming up next month that assumes the existence of this site, now was the time. I went back to the drawing board, studying a book on relational databases "clearly explained" (HA!) and sketching out on paper until I was ready to begin.
The server choked on a MySQL error. PHP couldn't access it. Since I figured I had to recompile PHP anyway, I tried including the Oracle support. Error, error, error. I did not have the energy to do my usual exhaustive search on Google and message boards to get this working, so I just relied on some tips from comments on the relevant PHP.net page. Edit, document, configure, re-edit, re-document, re-configure. Pain, pain, pain. Progress ground to a halt.
So by today I was at the breaking point, not wanting to face going to work at all. I dragged myself in, found that a power outage meant that I would have to reboot both servers so that they could be seen by our backup program again. So I had to stay a full day, the day before a University holiday (our one-day spring break). Great.
Then the following things happened.
After talking with my co-worker, whose counsel I don't normally seek on these things, I decided to phone up the client who wanted the Oracle DB. I had written our office's Service Level Agreement specifically to avoid these problems, after all. I explained to him that I really couldn't guarantee I could install it without consulting help, and that he would need to either migrate the data to MySQL or PostgreSQL with someone else's help, or keep that portion of the old web site on his old server running until he could find an alternative. I explained that if I had known he'd needed this support in the beginning, I might not have even agreed to host them, and that according to our SLA I really wasn't obligated to get this working, although I did try, and did some other nice things for him like configuring a search engine for them to use until Google has chance to index the new site. Luckily, he agreed that my statements and suggestions were fair, and thanked me for my help.
Then, I found an envelope on my desk containing a Macy's gift card from our AVC, the one who turned me onto podcasting, thanking me for all my work in that area.
Next, the new MacBook Pro I ordered, to replace our aging PowerBook with the broken CD drive, arrived a day early. I was happy to be able to set it up before the holiday, so we could use it for Monday morning's regular web team meeting. It took nearly three hours to set it up the way I wanted, but it was a joy to own and operate.
Finally, after I rebooted both servers on schedule at 4, I turned back to the secure server install. I had earlier removed any mention of Oracle from the PHP configure string, and PHP 5 now built correctly. But I wanted to use the latest Apache libraries for the PHP build, so I unzipped my previously downloaded Apache 2.2 distribution and started compiling away.
Apache choked on startup. Apparently several modules were now renamed or obsolete; it would have been smart to read the upgrade docs first. Edit, document, restart; edit, document, restart. Several rounds of this and I got Apache 2.2 running, without PHP or SSL.
PHP 5.1.2 came next. Luckily, it configured, compiled, and installed beautifully without the Oracle crap.
The final hurdle: mod-ssl. It was 5 and most of my work mates had gone home or to the bar across the street and I was going to miss out on lasertag with emmett_the_sane and Ziggy because I was so exhausted but I just had to get this done. Edit, document, restart, read docs, re-edit, document, restart, read more, pray, cry, scream.
FINALLY, I got it to work. With a self-signed dummy certificate that Firefox bitched about to be sure, but damnit, it was HTTPS and it had that little lock icon, and I was so happy I wrote Hallejuah five times into my notes, just like in the Messiah, Hal-le-ju-ah. I was not a useless geek wannabe after all; I was a true geek who could do useful things.
I'm far from done. I have to test the server, rigorously. I have to upgrade Apache and PHP on our staging and production machines. I have to request a real certificate and re-configure for that. I have to alert the client that has been waiting a year for this software that he now has to help me test it.
And that's not counting all the multimedia stuff and countless other things I still have to do. BUT
- I got the damn secure server installed and working,
- I got my MacBook Pro, and
- It's officially the weekend. Amen.