Linkstew

Biculturalism (UNIX culture vs Windows culture)

Add ESR's "The Art of Unix Programming" to my list of books I need to read. It sounds highly intriguing.

The X-Windows Disaster

This seems a little dated, but it was still an interesting read.

I was working with a bunch of Excel data this weekend for my CS project, and at some point I decided I wanted to transpose a table of data. You know, swap the rows and the columns? Simple, right?

Well, I tried searching Excel help, and had no luck. A quick google search didn't immediately yield the answer. And I asked my CS partners, but they also had no clue. A few of them even asked other people they knew in the lab, and still no luck.

For a bunch of CS majors, I was amazed by how little they all knew about Excel. Hell, that I was the one manning Excel in my group (because I knew the most about it) kinda says a lot by itself. I mean, I can get around in Excel, but I wouldn't dare list Excel on my resume. I know how complicated that sucker is, and I know that I can barely scratch the surface.

So I shrugged and decided I was going to write a perl script to transpose colon-delimited data that I could export from Excel. Perl is, afterall, my swiss-army chainsaw. What's the point of having a chainsaw if you're not going to use it? And besides, I figured it wouldn't take me more than 5 minutes to write, and I'd already spent at least that long trying to figure out how to do it with Excel.

But before I started, I had to go to the bathroom. And when I got back, one of my partners had found the "Paste Special" option, which, among a bunch of other handy features, includes a "Transpose" option. So much for showing off my perl-fu.

So at work, I've been doing a lot of work with Mac OS X's NetInfo database. In particular, I'm generating password and group files to import into the database via the niload command. Since I wanted "old" accounts to be deleted from my OS X clients, I looked at man niload, and found the following option:

-d: Delete entries which are in the directory, but not in the input.

Exactly what I wanted, right?

Heh, after I spent an hour tonight beating my head against `niload -d` not deleting entries that weren't in the input, I absent mindedly typed `niload --help,` and here's what I saw (emphasis mine):

-d: delete (override) existing entries from NetInfo when the input contains a duplicate name

-m: merge new values into NetInfo when the input contains a duplicate name

Note: only one of -d or -m may be used. If neither is given, existing entries in NetInfo will be unchanged if there are duplicate names in the input

Gah! Inconsistent documentation is the bestest! So this means that I've got to iterate through each entry currently in the NetInfo database, and if that entry isn't in the in the input (and isn't some form of "local" account), then use `niutil destroy ...,` which makes me sad.

YAFHW = Yet Another Hour Wasted.

Today while mucking around in NetInfo on the OS X machine at work, my jaw fell open when I noticed a more problematic local security problem than this nonsense. I immediately started composing my post about this problem in my head, and after mailing myself a few notes, I returned to configuring OS X.

But while searching for something else on google later today, I found that someone had beaten me to the punch, and had already written up this OS X security issue much better than I could have ever hoped to:

In brief, Mac OS X's NetInfo database stores crypted passwords that can be read by anyone with an account on the machine. Once the malicious user has the crypted passwords, all he has to do is sic a brute force cracking program on the hash until it finds a match.

To be fair, there is a way to secure your OS X box while still taking advantage of NetInfo as your primary configuration agent: You just need to make sure that your non-trusted users don't have permission to run tools like NetInfo Manager or nidump.

This UNIX story reminded me so much of The Tao of Computer Programming that I even went and checked multiple versions of the tao to see if it had been added to one of them.

But nope, the story in question doesn't appear to be part of the tao at all. Though it really should be.

So on top of the 8,000 other things I've been dealing with in the last week, I did some experimenting with rsync for work. I really liked it, and decided that once I got a chance, I wanted to switch to using rsync to upload my sources for my web sites. And so the weekend finally came, and after sleeping straight through Saturday afternoon/evening, I woke up and started playing with rsync, when Kevin messaged me:

Benjy (10:23 PM): Time to play with rsync.
Kevin (10:23 PM): I know it. I'm looking into it.
Benjy (10:24 PM): No, I mean, I'm actually playing with rsync right now.
Kevin (10:24 PM): Oh, did you read my just-finished post?
Benjy (10:24 PM): no...
Kevin (10:24 PM): That's pretty coincidental.

So before, I was using a Makefile + tar --after-date=($TODAY) ... + scp + ssh to copy updates to the web server. This worked okay, but it was pretty messy, and I had to manually specify the directories to include, and I had to manually exclude certain things from being copied every time, and so on. But here's what the new rsync based Makefile looks like:

all: clean put

put:
        rsync -avuz --exclude-from=exclude \
	--delete -e ssh gimli/ bsii@gimli:

get:
        rsync -avuz --exclude-from=exclude \ 
	-e ssh bsii@gimli: gimli/

clean:
        @echo 'Removing those pesky backup files'
        find gimli/ -name "*~" -exec rm {} \;
        find gimli/ -name "*bak" -exec rm {} \;
        find gimli/ -name "*#" -exec rm {} \;
        find gimli/ -name "#*#" -exec rm {} \;
        find gimli/ -name ".#*" -exec rm {} \;

The directory ~/gimli/ on my local machine contains everything that goes in my home directory on my web server. The file ~/exclude contains directories that should never be touched on either side -- notably, log files on the server side.

Because I use the `-e ssh` option to rsync, it uses ssh, so all of the communication is encrypted. And the best part is, you can do all of this on a default Mac OS X install, because it comes with rsync and ssh and make and everything you need! Yummy.

And then, as I make changes, I can just periodically type `make`, and it uploads just the bits that are different. And the advantage of this over my old system is that, if I make changes on the server side while I'm away from my own computer, I can type `make get` later and it'll update my local copy with what was changed on server. If you want more details about my setup, please ask.

disclaimer! It would be very easy to blow away your files by just running those commands that are in this post! Before you start playing with rsync, make sure you have a backup of all of your files on both ends! Also, be sure to use the -n option to rsync for a dry run -- it'll tell you exactly what it plans to do. Finally, be really careful with rsync's --delete. And if something does happen to your files, it's not my fault!

You know, I just realized yet another way that I'm a freak.

When I base a new password off of some dictionary word or phrase, instead of using 1337 sp34k s^b57i7u7i0n$, I use phonetic substitutions, based on the International Phonetic Alphabet where alphanumerically possible. I can't honestly account for when I started doing this, but I guess it had to have been some time after I took Ling 100.

So pardon my silence, but not only have I not had net at home, but servers have also been blowing up at work. And part of servers blowing up involves rebuilding them, which I was lucky enough to get to help with (because... don't get me started), and as part of this rebuilding we discovered problems with the backup scripts that were written long before even my time.

The backup command was doing `find $BACKUP_DIRS -xdev -not -type d -print | $TAR --create --verbose --files-from - --file -`. The thing is, that "-not -type d" in the find command meant it didn't include any directories at all, and the "-xdev" kept the find spanning file systems. The problem with this is, no directories were backed up, so when we untarred the backup, every directory was created when it was needed, which meant that the permissions on every directory on the system were wrong, and also that any empty directories weren't backed up at all (which has caused problems with emacs and postgresql, so far). So at one point in the last week, I took a list of every directory on the system (`find / -type d`) and with the help of a couple of other systems we had sitting around got to spend several fun filled hours manually reconstructing the ownership and permissions of 18,000 directories.

So uh, don't exclude directories from a find command you're piping to tar. It'll bite you in the ass later.

Hopefully I'll have net at home again soon, so I can resume my normal "post at 4 am from bed" habit.

So I've been using some some shell scripts I hacked up to act like an ssh agent, but they were really kind of stupid, and would frequently force me to manually reset things after a reboot. But I've been alerted to a proper ssh agent login process someone a friend of a friend wrote. So if you use ssh on OS X at all, SSHAgentServices is for you.

Basically, I'm completely useless without UNIX. Back in the day when I was interviewed for the Sysadmin position here where I work, one of the questions was "What's your favorite UNIX command and what's an example of something clever you've done with that command?" Back then, I answered grep, but my clever answer was pretty lackluster, and more programmery than sysadminy. Which probably partially accounts for why I was hired as a programmer and not a Sysadmin, but at least I didn't answer finger... Which one of my fellow rejected-for-sysadmin hired-for-programmers answered. ;-)

So without further ado, here are my five favorite unix commands, why, and maybe something clever I've done with each one.

1. perl - Claiming perl as my favorite command isn't really fair, since it slices and dices and is the ginsu of UNIX commands. Really, I don't use `perl -e` all that often... But I use scripts starting with #!/usr/bin/perl every day, so that's why this is number one. And being the shotgun it is, I'm not even going to try to give an example.
2. wget - The very first problem I had with Mac OS X 10.1 was that they replaced wget with curl. I assume that curl is equally capable, but I happen to know all of wget's switches back and forth, and I can recursively download a web tree or grab an mp3 in my sleep. I just prefer wget to download files to a web browser, because it's easier to pick where I want the file to go, it's more stable than a web browser, and the output is more useful. Any which way, a quick `wget url` is definitely my favorite method of downloading something.
3. grep - I use grep any time I'm looking through huge hunks of text. From `tail logfile | grep -v "stuff I don't want to see"` to `grep "email@address.com" ~/mail/*`, I basically use it as my first line search tool. My favorite use of grep was probably the time I was writing that paper on Dorian Grey, and I had a text file of the book. I was searching for references to opium, I needed context, so a simple `grep -A 5 -B 5 -i -E 'opium|opiate' doriangrey/* | enscript | lpr` gave me my pages of quotes just in time for my class.
4. dict - There really aren't any *clever* uses of dict, but I use it all the freaking time. For me, it doubles as both a spell checker and a dicitonary. Though now that I've been using Mac OS X a lot more, it's just my dictionary, thanks to OS X's integrated spell checking prowess.
5. (gnu) tar - (gnu)tar is really the lynchpin of the Makefile for my sites. Mind you, it's very important that that's gnutar, because other tar's just pale in comparisson. It's mainly the --exclude filename (which excludes every file in the given file name) and --after-date date (which only includes files who were modified after the given date that make it so insanely powerful for me, and allows me to only upload things which have been changed since I started working with a simple `make` call to make.

Other top contenders included:

• make - I use make a lot, but really it's just a way of making batch commands, and so for that reason it was dropped down.
• tail - In my line of work, I have to watch the log files from a lot of web servers, among other things, so I tail a lot of logs. But it's really grep that makes tail so useful, because those are really verbose logs, and I know I don't want to see everything in there.
• scp - This is what I use for 80% of my file transfers between servers (I use http for the rest), but it's kind of boring.
• ssh - I have tendrils all over the place with ssh, but it, too, is pretty mundane. I still couldn't get by without it.
• emacs/vi - I use both of these editors every day, and I really like both of them, but um, they're kind of mundane.

What're your favorite UNIX commands, and why?

I'm so amused by this perfectly correct, and yet so very undesirable crond behavior:

Date: Sun, 28 Oct 2001 01:59:00 -0700 (PDT)
Subject: another nightly cron report

Date: Sun, 28 Oct 2001 01:59:00 -0800 (PST)
Subject: another nightly cron report

Well, looks like it's time to start running that nightly report at some other time...