Jul 28

Comment/Trackback Spam

Lately this site has been overrun with comment and trackback spam, not that it shows because I moderate everything. Still, my inbox has been full of junk because of it. I enabled a CAPTCHA image on comment posts. I hope this isn’t much of an inconvenience, but it was either this or disable comments completely. I also disabled trackbacks site-wide. I never used trackbacks anyhow, so it’s no big loss.

I don’t receive many comments, but when I do they are usually meaningful or informative in some way, so I did not want to drop them.

Now, if I can just find the time to post more often…

May 13

PHP Crashes Caused By Extensions II

As I wrote about previously, I have had problems with Apache and PHP crashing due to various PHP Extensions. I have come upon another combination that triggers a problem, but after investigating it a little I see that it’s been reported before, and nobody wants to fix it. PHP blames PHP accelerator systems, and Zend claims it’s a shared memory configuration problem (it isn’t — at least on my system)

The error happens whenever attempting a graceful restart of Apache via “apachectl graceful”:

  • [notice] seg fault or similar nasty error detected in the parent process

The environment:

  • Apache 2.2.4
  • PHP 5.2.1
  • Zend Optimizer 3.2.8

The culprit:

  • Some interaction between the Zend Optimizer being loaded along with the PHP pspell module.

If I disable one or the other, the crash goes away. Since this particular installation does not require the pspell module, I disabled it and things have been stable ever since.

I did follow Zend’s recommendations for increasing certain shared memory tunables on FreeBSD, as well as trying to recompile everything involved. For more information on shared memory tuning check the FreeBSD man page tuning(7) as well as this Zend Knowledge Base article. Note that certain sysctl settings may only be modified at boot time via /boot/loader.conf and/or /etc/sysctl.conf.

More information to come if I can find anything else…

Update 11/21/2007 – I found that in more recent version of PHP (Around 5.2.4-5.2.5) having pspell.so loaded before spl.so in extensions.ini will result in crashes when an httpd process is stopped/killed. Moving pspell anywhere after spl will clear this up (so far…).

Feb 17

Daylight Saving Time Strikes Again! Well, almost.

We lucky folks in Indiana have had a rough two years dealing with time. As I wrote about last year, Indiana just started observing Daylight Saving Time (DST) in 2006. Now, for 2007 we also have to change the dates on which DST starts and ends. DST now begins on the second Sunday in March, and ends on the first Sunday in November — This year it’s March 11th and Nov 4th. Why on earth we didn’t just wait to start along with the new rules is anyone’s guess. <rant>I don’t think we should be using DST at all, but that’s a story for another time</rant>

Here I was, all set for another round of server updates, reboots, etc. Turns out that I didn’t need to worry quite so much. When I updated all of the time zone files on our servers last year for Indiana’s initial DST switch, they had already made the changes with the new start and end dates for 2007 and beyond. You can confirm this on most UNIX systems as follows:

# zdump -v /etc/localtime | grep 2007
/etc/localtime  Sun Mar 11 06:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 EST isdst=0 gmtoff=-18000
/etc/localtime  Sun Mar 11 07:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime  Sun Nov  4 05:59:59 2007 UTC = Sun Nov  4 01:59:59 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime  Sun Nov  4 06:00:00 2007 UTC = Sun Nov  4 01:00:00 2007 EST isdst=0 gmtoff=-18000

If it says “Mar 11” and “Nov 4” you’re good. If it says “Apr 1” and “Oct 28” you need to update your time zone definitions. On FreeBSD, this can be as simple as downloading new zoneinfo files, recompiling them, and re-selecting the timezones:

  1. Download: ftp://elsie.nci.nih.gov/pub/tzdata2007b.tar.gz
  2. Exctact the contents to /usr/src/share/zoneinfo
  3. cd /usr/src/share/zoneinfo; make install
  4. tzsetup
  5. Choose your time zone again.

A reboot may be necessary to ensure that all running programs are on the same time zone. Currently running programs may not pick up the change. You could also update FreeBSD to a recent version, which includes these changes. If you choose to do the OS update, be sure to run “tzsetup” afterward to be absolutely certain that a new tz file gets installed to /etc/localtime. After you’re done, re-run the zdump command above to be check that you now have the proper DST change dates for 2007.

If you are running any Cisco gear (or other IOS-alike devices) this should work to make the change:

clock timezone EST -5
clock summer-time EDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00

Adjust the time zone to yours, of course.

There may be other programs that handle time zone data internally (such as Java and Outlook) so you’ll have to be sure there are no loose ends in that department. Those of us in Indiana have some practice with this, so at least for us it may not be that bad.

I am aware that many of these problems could be avoided by using UTC on all our server clocks. While that may be preferable, we like to have everything in local time. It’s a choice, and we deal with the consequences. One of which is we never schedule jobs to run overnight between 1-3am — they could be run twice or not at all.

UPDATE: 3/1/07: I have also been informed that you can copy the “/etc/localtime” file from an updated system to any other system that needs it. This could be especially useful if you are unable to update all of the Time Zone definitions for any particular reason.

Oct 18

PHP Crashes Caused By Extensions

Once again when faced with updating PHP on a few servers, I encountered my favorite of all PHP quirks: After rebuilding extensions, PHP crashes and/or takes Apache down with it. Here are the errors that tend to show up:

  • exited on signal 11 (core dumped)
  • exited on signal 6 (core dumped)
  • seg fault or similar nasty error detected in the parent process

And my personal favorite:

  • httpd in free(): error: junk pointer, too high to make sense

I have seen this on PHP4 and PHP5, and with Apache 1.3 and 2.x. I’m not sure if it’s a problem inherent to how the FreeBSD ports system builds and installs the modules or if it’s just a problem in general. I had read once upon a time that rebuilding extensions in a certain order would fix it, and it did. I never got around to figuring out why this worked. Turns out, rebuilding them doesn’t really matter, but the order of the extensions being loaded does. Rebuilding fixed it because when a php extension port is rebuilt, it gets placed at the end of extensions.ini. I solved the problem by editing /usr/local/etc/php/extensions.ini and placing the lines for mysql, imap, and sockets at the end and in that order:


I’m not sure if the conflict is only with those three, or with others as well, but that fixes it on my servers. I tried it on three different setups, and before the change they all crashed and after the change they’re all running OK.

Hopefully if anyone else runs across this, it will help. If I get more time, I’ll dig into it more later.

Update (11/25/06):
There has been some more discussion of this on the FreeBSD-Ports mailing list and the FreeBSD-STABLE mailing list. Apparently at least part of this is due to the PHP recode, MySQL, and IMAP extension ordering. These extensions rely on c-client libraries with different overloaded hash functions. So the “magic” order at the end of extensions.ini should be:


There is also talk of building some logic into the PHP extension ports to ensure the ordering of the extensions so as to avoid this bug. Best of luck to those working on it!

Edit 8/25/07: I wrote a very hackish shell script that gets the job done keeping the extensions in this order. It’s not pretty, but it works. It can be found here: http://www.pingle.org/files/fixphpextorder.sh. Read the full post here.

Edit 11/21/07: Lately pspell has also become picky about ordering. I recommend placing it at the end (or at least anywhere after spl.so)


Sep 05

FreeBSD On The Desktop (Part IV: A New Hope)

Due to my recent bad luck with electricity, I was using my home server as a desktop all last week. As a result, I have some more notes to add about using FreeBSD as a Desktop machine, which I hope others may find useful.

Read on for more about Printing, Firefox and Thunderbird interoperability, mounting a USB mass storage device, CD burning with K3B, and Video playback.

Continue reading

Sep 04

Magical Exploding Laptop

So a week ago I had a rather nasty shock. I was watching a TV show that I was playing on my laptop, which was hooked up to my DVD Recorder via S-Video and composite audio cables. Nothing I hadn’t done a few dozen times before. The difference was: I realized that I had not plugged in the laptop’s power cord. When I proceeded to plug in the laptop — *poof* — sparks flew and smoke rolled out of the laptop from under the headphone jacks. It’s an Acer not a Dell so this was truly a surprise :)

After some minor panicing, I found that the laptop would still boot (thankfully) but the audio was dead. I presume the S-Video port was also dead, but I was not about to test it. Sadly, my DirecTivo was also fried (also connected to the DVD Recorder via S-Video) but the DVD recorder is just fine. The jolt also fried a segment of coax cable between the Tivo and the Satellite dish: Specifically it was the segment that goes from the inside of the house to the grounding block outside.

Read on for all of the gruesome details…
Continue reading

Apr 02

The Horrors of Daylight Saving Time

So as a resident of Indiana, today is my first day of Daylight Saving Time and I am not happy at all. In addition to having to reset all of the clocks, I also had to change the time zones on all of the computers — Easy enough on Windows, you can just change the time zone to Eastern. However, all my FreeBSD servers required a download and recompile of new time zone definitions. Not just that, but most programs that were started before the time zone switch will not pick up the change until they are restarted. In some cases, it’s safer to just reboot the whole server to make sure everything is running on the same clock. And I hate having to reboot servers.

The reasoning behind the FreeBSD time zone updates is explained in this link. The gist of it is that past file dates can be miscomputed if the time zone is simply switched to Eastern.

Many users of calendaring programs will be bitten by similar problems, causing appointments set before the change to exhibit all kinds of odd behavior.

Personally, I have not seen enough compelling evidence to show that the clock changing business actually saves any energy. It does, however, mess with people’s internal clocks and cause many traffic accidents. Most of the figures done showing that it saves energy seem to be from the 1970’s. I doubt the governer thought about the thousands of man-hours lost in actually changing the clocks, on computers and servers especially – but he sure was sympathetic about the possible loss of an hours worth of drinking at bars!

The only positive effect is that the Indiana legislature will now debate over things that actually matter, instead of over DST.

I’m sure that come Monday I will have to deal with quite a few calls from people who, rather than changing the time zone on their PC, have simply run the clock ahead an hour – only to find out that the clock resets when the Internet Time function resyncs.

Dec 23

Merits of RBLs?

A couple weeks ago, the SORBS spamtrap list picked up a few Hotmail and Gmail servers, and a Yahoo mailing list server. This lead to me getting complaints that legitimate mail was bouncing. I’m all for letting the mail get blocked, because it’s the only way that large companies like Google and Microsoft will be forced to fix problems. Unfortunately, the end users don’t see it this way. They think because Hotmail user A can’t get mail to our user B, it’s a problem with our system and we need to fix it. Ignoring the fact that thousands of other ISPs who use the same RBL are also blocking mail from those people. Long story short, I was forced to remove the spamtrap RBL (by using all of the separate SORBS RBLs instead of the composite list) — the mail started flowing again and the complaints stopped.

This is leading to conversations on the general merit of RBLs in general, and whether or not we should use them because it’s allowing someone else to control whether or not mail gets to our users. Of course the people raising these questions do not have to listen to the end user complaints. People want all their mail and no spam, which of course is impossible.

Currently, between several different RBLs, we reject about 130,000 messages per day (~80% of the total daily mail volume) at the MTA level. Should we turn them off, everybody would notice. There are no other spam filtering techniques that have done as much to reduce our spam overall as RBLs. Sure, we could throw a million content filters at it, but that takes a lot of horsepower to run, and probably would not be as effective. I put more stock in RBLs than I do in content filtering. The only legitimate alternative to using RBLs at the MTA level is using them in SpamAssassin where they are ranked with scores based on the RBL’s reliability and such. However, performing the RBL checks in SpamAssassin also introduces a lot more delays in message delivery (and of course, if someone sends an e-mail and the other person doesn’t have it in less than a minute people call and complain too!)

Life would be so much easier if there was a secure and spam-resistant alternative to SMTP, but that won’t be happening anytime soon.

Dec 11

Ruby on Rails

So it’s been quite some time since I posted anything, mainly because I’ve been rather busy on a project at work. It’s a fairly complex system that we used Ruby on Rails for. I had never used Rails before (or Ruby for that matter!) but it was easy to pick up (I suppose having coded in a dozen languages or so at one time or another makes picking up new ones easier…) Rails really excels at taking care of the grunt work (db access/mapping especially) and lets you focus on what you’re trying to accomplish.

I did all of my coding using Eclipse, with Ruby Develpment Tools, Subclipse (for Subversion access), and most importantly the RadRails plugin. For the most part is was a good experience. There were a few times when renaming a file or performing miscellaneous actions that Eclipse locked up on me (at home and at work) but restarting Eclipse worked ok. The only feature missing that would have been nice is Rails debugging. It’s in the works in RadRails, but it’s a ways off.

More about Ruby on Rails can be found elsewhere.

I found the book Agile Web Development with Rails invaluable during the entire process. (Programming Ruby is another fine choice, too!)

Jul 10

Spammers Faking Received Headers (continued…)

A little update on the Spam I had been seeing recently.

After looking a little deeper, it must have been a single spammer using that tactic, and relaying the spam through a bunch of other hosts (likely compromized PCs, since many were cable modems). After a while, those particular faked Received: header styles were not showing up as often.

I’ve added a couple more RBLs and RHSBLs, updated SpamAssassin, and tweaked some default scores, and things seem to have gotten happier.

The number of rejections at the MTA level has held steady, but the overall volume has ebbed while the number of detections by SpamAssassin increased. Result: Less spam. For now.