Peregrine

My dual Opteron workstation has been around for nearly 5 years now. It’s had some bumps and bruises along the way (some of which were due to my own actions), but has been a great machine. It still has very good performance, especially given it’s age.

When I first built it in May of 2004, Fedora Core 2 was barely out and was the first Fedora to sport an AMD64 (x86_64) 64-bit version. That was the first and last time that I installed Linux on this box, from scratch. Since then, I’ve upgraded it to FC3, FC4, FC5, FC6, F7, F8 and now F9 (I will upgrade to F10 in a week or so).

When I installed FC2, I used the ext3 filesystem for the root volume (I use LVM). I "converted" the root volume to the XFS filesystem on 2006/08/03. I also created a few volumes using XFS and reiserfs (v3.6) filesystems.

Over time, I’ve had a few minor problems with XFS. Recently, those problems grew in regards to the root volume to the point where I needed to convert it to something else, which I did the other day. The root volume is now on reiserfs. That leaves just 3 volumes that are still XFS.

After upgrading to F9 and installing updates, there were a couple of weird issues that I was dealing with. I also kept seeing some filesystem corruption messages (on the terminal, in the logs) for XFS volumes (but they don’t tell you which one). That’s it, I’m done with this XFS thing, so I’m going to convert those filesystems over to something else and get rid of XFS on this workstation.

The three volumes are for /usr/, /var/ and /var/log/. I could just drop to single user mode and convert /var/ and /var/log/ without any difficulty. For whatever reason, on Fedora and derivatives (including RHEL, CentOS, etc.), I have never been able to umount /usr/ successfully once it’s mounted. So, I’m using a rescue environment (to convert all 3) like I did for the root filesystem conversion, just so I don’t have to muck with it.

I haven’t had one problem at all with the two XFS volumes I have on my home file server (one for /music/ and one for /video/). That server is running openSUSE for a few years now. It’s also a much more complicated setup on that hardware, which I’ll talk about more in a later article.

For those who want to post comments that I’m an idiot for using reiserfs, please, don’t bother. I’ve heard every reason why this filesystem or that filesystem sucks and you should only use, “the other one,” instead. Look, it’s this simple: since the filesystem is the one piece of software where we just don’t tolerate buggy software, when something does go wrong the stories live on for years. I’ve heard horror stories of kinds that you might never be able to imagine describing data loss at the hands of ext2, ext3, reiserfs, XFS, JFS and many other filesystems. I’ve only experienced data loss with ext2 and ext3. XFS has given me problems, but thankfully not with files that I couldn’t easily replace. I haven’t hardly used JFS, but I do have a volume or two on my home file server that are JFS and there’s been zero trouble there.

Here’s my philosophy about filesystem type selection: use the right tool for the job.

It’s not always easy to say with perfect definitiveness that you should always use this filesystem here and that one there. Benchmarks show all 4 of ext3, reiserfs, JFS and XFS as having statistically equal performance for general use cases (like workstations). There are some rules by which I can say, from experience, that one of these will outperform the others for a particular use. I’ve been meaning to run another series of performance benchmarks on as many viable Linux filesystem types as I can. I’ll post results and talk about use cases then. For now, here are some basic tips from my experiences:

• Always check the “expiration date” on the horror stories that people tell you. It’s more likely to be old, as reiserfs, ext3, JFS and XFS have all been quite stable for many years now.
• ext3 will almost never outperform the others for a specialized task.
• reiserfs, JFS and XFS will almost always have roughly equal performance for most specialized tasks. This is primarily due to the fact that they share the very similar basic filesystem design concepts, though, obviously, the implementations vary. I’ve thought for many years that XFS was derived, in part, from reiserfs (due to some very hard to discount coincidences in XFS structures and code) but also shares some design elements in common with MacOS filesystems.
• If you’re going to have lots of files, big or small, then move away from ext3. Newer versions of ext3 (that are not backward compatible with older ext2/3 drivers) implemented some features (like hash-indexing) from reiserfs in order to improve performance in this area. Still, more than about one thousand files or so in a directory and ext3 starts to bog down quickly (when working in that directory). So, for example, ext3 is a really poor choice for spooling directories on busy servers or for proxy stores or any other application where tens if not hundreds of thousands of files will be created.
• ext3 has the worst file deletion performance of the group. Thus, for applications like print and mail servers, ext3 is a very poor choice. I have personally seen anywhere from 7 to 9 times better performance for print servers and from 10 to 23 times better performance for mail servers by simply converting the spool (and log, in the case of the mail servers) directories from ext3 to reiserfs.
• XFS and JFS have some specialized features that are very useful in high throughput applications. XFS has a bandwidth guarantee feature that is very useful with large media operations (like audio/video editing, compositing, etc.) and streaming. JFS has some sustained high throughput features that provide excellent performance for some types of databases (not database servers, but data operations by the servers).
• When it comes to databases, it’s very hard to predict which of these 4 will provide the best performance. It is very rare that ext3 is the winner, but it does happen. The only way to really know has been to create 4 volumes formatted with each filesystem type and run some benchmarks against the same DB on top of them. If you’re going to do this, make sure to use the database structures for the DB you want to test that you will be using in production. You don’t have to have “real” data, but make sure it is representative of the types and sizes of records that your database will be working with. Also, be sure the benchmarking test run “real” queries in the “right” ratios that you do (or expect to) see in your production environment. After all of that testing, you’ll probably see that one of the filesystem types outshines the rest.

Once, while I was consulting with a Fortune 500 company that will remain nameless, we saw that certain tables experienced huge performance benefits on one filesystem and other tables were significantly better on another. They actually reworked the application code to work with splitting the database into two databases, that were then stored on two different filesystems in order to take advantage of this.

Basically, each of these filesystem types have their advantages and disadvantages. There are other journalling and log filesystems available for Linux that are worth looking at for some applications. If you have a strong bias towards just one filesystem type and won’t even look at the others, then you are very likely missing out some benefits that you could have. If nothing else, it’s certainly an interesting topic … to some of us geeks.

A few weeks ago, I was looking through some Netfilter documentation, just poking around, looking at some modules I’ve never seen/played-with/hear-of and I came across the recent module. I decided to try it out on one of my servers that gets anywhere from zero (0) to tens of thousands of crack attempts via SSH per day and see if I could weed out some of these bot-nets. It also occurs to me that this could help fight email SPAM-bots, too.

Of course, it’s very important to have good, strong password security practices. If you have poor passwords, none of this will matter, as you’ve probably already been compromised whether you know it or not. This means that all users have to have strong passwords. Techniques for helping users to create and use strong passwords are beyond the scope of this article, but I will write articles about these things in the near future.

Here’s the configuration that I put into place. I’m showing this as the the iptables commands that you would run on the command line, adapt to however you persist your Netfilter configuration. Also note that these lines should replace anything that you now have in there for SSH. I’m also including the additional ESTABLISHED,RELATED rule here for completeness:

# iptables -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# iptables -A INPUT -i $OUTSIDE_INTERFACE -p tcp –sport $UNPRIV_PORTS –dport 22 -m state –state NEW -m recent –update –seconds 10 -j REJECT –reject-with icmp-host-prohibited
# iptables -A INPUT -i $OUTSIDE_INTERFACE -p tcp –sport $UNPRIV_PORTS –dport 22 -m state –state NEW -m recent –set -j ACCEPT

These three rules mean:

1. Accept traffic for “conversations” that are already in progress. This rule works for traffic in both directions and will handle everything in the ongoing connection.
2. If an IP packet matches these criterion:
   1. “-A INPUT -i $OUTSIDE_INTERFACE” — coming in on the Internet connetion (I create a variable with the value “eth0” or whatever it is and use that in my firewalling scripts);
   2. “-p tcp” — carry TCP (for Layer 4) in the IP packet payload;
   3. “–sport $UNPRIV_PORTS” — coming from an unprivileged TCP port (legitmate clients should only come from source ports 1024 through 65535, inclusive);
   5. “–dport 22 — destined for TCP port 22;
   6. “-m state –state” — the state module doesn’t have a record of this packet as being part of an existing connection;;
   7. “-m recent –update –seconds 10” — the update module has a record of another connection attempt matching this one within the past 10 seconds.

   Take these actions:

   1. “-j REJECT” — throw the packet away;
   2. “–reject-with icmp-host-prohibited” — return an ICMP host-prohibited response to the client who tried to initiate this connection.
3. The last rule is essentially the same as the second, except for:
   1. “-m recent –set” — make a note of the time that this connection attempt occurs at;
   2. “-j ACCEPT” — if all criterion match, accept the packet (which will stop further rule processing here, BTW).

Basically, what we’re trying to do here is to limit the number of failed connection attempts that are allowed.

Let’s say that human being tries to connect via ssh lamont.example.com and they don’t have an account or the mis-type their password several times to the point where sshd cuts off the connection, so they re-run their ssh command to try again. It will probably work and let them in, as it probably took them longer than 10 seconds from the first packet of the first TCP connection until the first packet of the next TCP connection.

However, the cracker-bot-nets don’t work like humans. They automate the process of trying to connect as fast as they can, so they will try only 1 or maybe as many as 3 or 4 passwords before closing the TCP connection and starting another. Since they don’t have to be as slow as people, they’ll usually be coming back again in under the 10 seconds. Most of them actually try to establish multiple connections (2-20ish at a time) in order to try more passwords.

Once the crack-bot starts seeing TCP connection failures, they usually skip your IP and go on to try and find softer targets. If they can’t connect to SSH, then why bother wasting time trying.

After setting up this configuration and letting it run for a week, I can report that it works marvelously. I’m getting under 10 break-in attempts per day, now. If you’re going to have SSH visible to the world (and why shouldn’t you?), then I would recommend adopting these Netfilter rules in your firewall configuration.

USA Today carried this story on the front page, Tuesday, (2007/12/04). It’s interesting that the headline for this story in the print edition and the online version are not identical.

The crux of the story is that the major U.S. airlines are shifting their schedules a little so that there will be fewer overall seats available throughout the U.S., despite stronger demand than last year. Such changes are not being made in their International schedules.

Part of the numbers come from switches to smaller aircraft, some are from schedule changes that will have a smaller total number of flights on some routes and other routes may be eliminated.

I also found it interesting that Delta Airlines, which I fly more often than all others combined, is the only major airline that is leaving their capacity and schedules virtually unchanged, with only a 0.6% overall U.S. routes capacity decrease planned for January 2008.

The biggest problem that could occur, whether you are flying with an airline that is making large changes or small, is that hiccups could take a day or two, rather than hours, to resolve. Such issues could bleed into other airlines, as cross-pollination can and does quickly fill the few available seats still open on all the other airlines. Given all of the problems that Northwest Airlines has been experiencing this year, I would hope that they don’t cut back too far. Granted, things are better at Northwest than they were in June of 2007, but the end of November continued to show larger numbers of cancellations than should normally be expected.

I just read an article from The New York Times website about Eve Online. I have been a player of Eve since March, 2007.

Obligatory quote:

At the strategic level, coalitions involving tens of thousands of players struggle for months over strategic objectives or simply to wipe out their enemies. For at least a year the most powerful group in Eve has been an alliance known as Band of Brothers, a self-appointed evil empire with the stated objective of taking over the galaxy. Against them is arrayed a motley batch of self-styled freedom fighters with names like the Red Alliance (mostly Russian), Tau Ceti Federation (mostly French), GoonSwarm (mostly obnoxious) and the Interstellar Alcohol Conglomerate (mostly drunk).

I finally got all of the blogs I manage updated to WordPress 2.3.1, today. Several friends who also use WordPress had experienced some difficulties upgrading to 2.3.0, so I had held off on the updates on some of the blogs until these past 3 days. There were two of them that I had some very small problems with, but they were solved very easily. One of those was this blog, the one you are reading right now.

First, when I was updating my wife’s blog, I got all in a hurry and forgot to make a backup of the database first. Then, when I connected to the wp-admin/upgrade.php script and clicked on the Upgrade WordPress button, all hell broke lose. I had DB errors left and sideways (thankfully, not quite right, though). At that moment, I realized that, “I could really use that DB backup right about now.” Well, I didn’t have it, so I tried the export function from the admin interface and that worked. At least I had her posts (there was a brand new one, too, which wasn’t in the most recent backup file that I did have).
Read the rest of this entry »

In dealing with nVidia and ATI drivers for Linux (both a kernel and X driver are needed), I’ve been using the Livna YUM repositories for Fedora to easily install them as RPMs using YUM.

I’ve run into trouble here and there as the Livna folks keep pulling RPMs from their repos for older versions of the kernels. At the very least, they should leave the kmod-* packages in there for the original kernels that shipped with each release. Then, people can install a release and get a good driver. I had to wait for about 3 weeks after I first put F7 on my home workstation (dual AMD Opteron) before I could get the nVidia driver from Livna because they didn’t have one for the older kernel packages and the newer kernels weren’t booting (turned out to be malformed initrd files, which I later fixed).

Yes, I understand that they take up some disk space, but it’s not really that much perhaps 100M per release to keep all kmod-* packages and their dependencies around.

Livna, if you’re listening, please, give us all the driver packages and don’t remove them. You don’t know which kernels are working for people and which aren’t, so you could really be making things pretty difficult for people.

A few minutes ago, I installed the kdebluetooth package. I was already logged in, so I had to launch the kbluetooth applet myself. I then clicked K Menu -> System -> KInputWizard, pressed the “reset” button on the bottom of my mouse and clicked “Add” in the Input Devices dialog. My mouse was discovered and I connected to it. Simple as that.

I have a Logitech bluetooth mouse that travels with me. I use it with my notebook computer, as I’m very, very not fond of trackpads. My favorite is the “TrackPoint” or “Eraser-head” mouse built into the keyboard, but this notebook didn’t come with one. Supposedly, I can buy a replacement keyboard from HP that includes the eraser-head pointer, but I have not yet done so.

When I wrote about installing Fedora 7 on this notebook (and now installing Fedora 8), one thing which I never documented was how I got the bluetooth mouse working with Linux (under F7). Now that I installed F8 from scratch, I need to set it up again.

When I installed F7, I spent hours dog-paddling through Google searches and horrible documentation and still hadn’t figured it out. Then, my friend and co-worker, Clint Savage (a.k.a. Herlo) popped into the office. It was him! He’s the one who has the exact same mouse as I do; I knew I’d seen it somewhere before I had bought mine. So, I asked him. He smiled and laughed, saying, “Not finding much useful documentation out there, eh?” He’d been through the same thing as me. He was impressed with how far I’d gotten through that process and estimated that I was probably 1-3 hours away from finding it myself, if I continued to follow the pattern he had. Well, he shared the information with me.

The good news was that it was pretty easy to get my bluetooth mouse talking with my bluetooth equipped notebook, just not really documented anywhere that one could point to just one thing (boy, I wish I’d documented those commands in a blog post; I’ll see if I can do just that next week, when I’m back at the office). The bad news was that one of them had to be run every time he started his computer. So, I put that command into a /root/bin/connect-to-my-bluetooth-mouse (or something like that) script. Then, a week later, I forgot to run that when I booted up and logged in, once, but was using the mouse anyway. I had discovered that it wasn’t necessary to run that all the time.

One of the reasons that it had been so difficult to setup bluetooth on Fedora 7 was that I was using GNOME on that installation. I stuck with entirely GNOME apps (except for Kdevelop) the entire time I had F7 on this notebook. Now that I have F8, I’ve gone back to KDE, which makes life so much better for me. GNOME still doesn’t have much bluetooth support and what is there is still very early half-baked and non-usable, for the most part. KDE’s bluetooth tools, on the other hand, seem much more comprehensive and “just work” for me.

The fact that Fedora (and by extension, RHEL, CentOS, etc.), supports bi-arch platforms is a great thing. However, it does get to be very irritating when YUM decides that it should just pull in 32bit versions on a system with no other 32 bit packages. I’ve experienced this problem during installations, as anaconda now uses YUM to process package selections (since FC5).

It doesn’t stop with just anaconda installation and yum update commands, either. Almost every yum install command that I run decides to install both 64-bit and 32-bit packages. That is, unless I explicitly specify that I only want the 64-bit for each and every package. For example:

# yum install foo.x86_64 bar.x86_64

Why is yum doing this? It didn’t used to. I started experiencing this a little bit on FC6, but F7 and F8 both have horrific troubles with it. I need to do some more digging through Red Hat’s Bugzilla bug/issue tracking system, however, my first pass didn’t find anything to help explain the changes. After a little more research, I’ll file this as a bug.

In the meantime, here’s a quick-n-dirty hack I put together to run updates. The first step is to capture the output of yum update to a file (be patient, this command can take for-freakin-ever to run). Step two is to run the update itself. Here it is as a shell script:

#/bin/bash
# Get a temporary file to use.
TO_UPDATE="$(mktemp)"

# Populate the temporary file with the list of available updates.
yes n | yum update > ${TO_UPDATE}

# Composite an update command that does not include any 32-bit stuff.
yum update $(for i in $(sed '/i[3456]86/d' ${TO_UPDATE} |
         sed '/^Updating/d' |
         sed '/^Installing/d' |
         grep -v "^$" |
         grep -v replacing |
         cut -d" " -f2); do
      rpm -q --qf "%{name}.%{arch}\n" $i; done |
   grep -v "is not installed$")

# Optional cleanup.
if [ "${1}" = "-k" ]; then
   mv ${TO_UPDATE} /root/$(date --iso-8601)-$(basename ${TO_UPDATE})
else
   rm ${TO_UPDATE}
fi

You will have to run a separate yum update command for any 32-bit stuff you really do have installed and want to update.

I know this could be streamlined (especially the part that constructs the yum update command), but as I’m not planing on making this a permanent fix, I’m just not going to bother with it right now. Still, feel free to comment or trackback with other solutions or optimizations of mine.

So, is anyone else experiencing troubles with the proprietary ATI driver on Fedora 8? How about on an updated F7 system?

My HP Compaq 6715b notebook comes with ATI Radeon X1270 video 128MB RAM dedicated plus 192MB RAM shared) and a 1680×1050 resolution 15.4 inch LCD (at 61Hz, it would seem). I’ve installed the proprietary ATI driver in order to get it working, as Fedora’s tools get really confused about widescreen setups, it would seem.

Here are the relevent package versions:

# rpm -qa | egrep ‘(fglrx|kernel)’
kmod-fglrx-8.42.3-8.lvn8
kmod-fglrx-2.6.23.1-49.fc8-8.42.3-8.lvn8
xorg-x11-drv-fglrx-8.42.3-7.lvn8.1
kernel-devel-2.6.23.1-49.fc8
kernel-devel-2.6.23.1-42.fc8
kernel-2.6.23.1-42.fc8
kernel-2.6.23.1-49.fc8
kernel-headers-2.6.23.1-49.fc8
xorg-x11-drv-fglrx-libs-32bit-8.42.3-7.lvn8.1

(As you can see, I haven’t removed the original kernel, yet. Maybe I’ll go do that now.)

However, I seem to be getting some fairly odd artifacts on-screen with this driver under F8, including some odd extra sprite garbage with the mouse cursor. I had experienced some oddities under F7, but they were confined to GNOME applications (no others exhibited any issues). It doesn’t matter if I enable or disable “Desktop Effects” either (they won’t successfully enable, anyway). A RAM test (memtest86+) shows that there’s nothing wrong with the system memory, but that doesn’t test the video card. There are ATI tools for testing the video card more fully, but I haven’t had time to try them out, yet.

Since FC6, Fedora systems rely on the X server detecting proper monitor and other configuration parameters every time it starts. This has been far less than reliable on a wide variety of machines that I’ve been running into over the past year. I’d like to get some more information about other people’s experiences with this, before I file a “bug” report about this. It’s really becoming an embarrassing problem as things worked much better when we would get a finished configuration file by default in FC5 and earlier.

Last night, while flying from Salt Lake (SLC) to Kansas City (MCI), I used a Fedora 8 x86_64 DVD I had burned earlier in the day to reinstall my notebook. Probably not the smartest way (that is, on an airplane without an Internet connection available) to do such an installation.

First, I tried to do an “Upgrade Install”, which didn’t surprise me by not working. Upgrading from 32-bit F7 to 64-bit F8 isn’t something that anaconda knows how to do, and I didn’t expect it would. Still, I tried it and know we know for sure. So, I did a fresh installation. I removed the root Logical Volume (I’ve been using LVM for my notebooks and workstations since long before Fedora started to default to it) and created new LVs for / and /usr/ LVs. Previously, under F7, /usr/ was on the root LV.

The install phase itself went just fine. 1478 packages were installed, including 389 32-bit (i.e. i386, i486, i586 and/or i686 RPMs). I ran a simple command to find and then remove all of them:

# rpm -qa –qf “%{name}-%{version}-%{release}.%{arch}\n” | grep “\.i[3456]86$” | xargs rpm -e

If I need any 32-bit stuff later, I’ll just reinstall as few such packages as are required.

My next issue was the same video problem as I had when I installed Fedora 7. The graphical installer couldn’t run and the resulting system had no working X server configuration. This was very easy to fix:

# wget http://rpm.livna.org/livna-release-8.rpm
# rpm -qp –qf “%{name}-%{version}-%{release}.%{arch}\n” livna-release-8.rpm
livna-release-8-1.noarch
# mv livna-release-8.rpm livna-release-8-1.noarch.rpm
# rpm -Uvh livna-release-8-1.noarch.rpm

(NOTE: I renamed the package file back to what it should have been in the first place. Though rare, if they update it, I’d like to notice the difference and be sure I’m using the latest one on some other machine in the future.)

After setting up the Livna repository for Fedora 8, I was able to install and activate the ATI driver:

# yum install kmod-fglrx x11-xorg-drv-fglrx
. . . output omitted . . .
# fglrx-config-display enable

Notice that the command name changed from Fedora 7 to Fedora 8; it used to be ati-fglrx-config-display.

The next thing I need to fix is to re-associate my Logitech Bluetooth mouse with the notebook. I had a script in /root/bin/ that would have taken care of that very easily. Unfortunately, I forgot about that until after I had installed Fedora 8, thus obliterating that file. Oh, well. This time, I’ll also document it elsewhere (perhaps here?) once I get it figured out again. I hope that will be tonight.

One last thing; with Fedora 7, I did my very best to stick with an all GNOME system. It was very irritating using applications that just couldn’t handle lots of basic things that I take for granted using other apps (mostly KDE). For Fedora 8, I’m going back to KDE, where things work much better.