Aug 27

Upgrading to FreeBSD 9.x with a whole-disk gmirror

My FreeBSD desktop workstation has been on 8.x since I built it just over two years ago. When I made it, I bought two 1TB HDDs and setup a RAID-1 volume with gmirror, following what was at the time the recommended instructions from the FreeBSD Handbook. After delaying the upgrade to 9.x repeatedly over the last several months I finally decided to give it a try after 9.1-RC1 was announced. Read on for how it went wrong, since FreeBSD 9.x more correctly handles partition integrity checks.

Continue reading

Aug 24

Setting Up a FreeBSD SVN Mirror

According to discussions on the FreeBSD-stable list, CVS support for FreeBSD sources is not just dying, it’s practically dead. (I’ll skip over the jokes about Netcraft confirming it…) So I setup an SVN mirror of the FreeBSD base and ports repositories and documented the process a little. I did this mainly because there are a lot of servers in two locations that need frequent access to the base and ports repositories, and there are currently cvsup mirrors running (one at each place) to keep the load off the upstream servers and for faster local access.

Continue reading

Apr 15

Growl Server for KDE to Receive Network Alerts

It’s no secret that I work with pfSense a lot, and a while back support was added to pfSense for Growl alerts over the network. Tired of being jealous of everyone else with a Mac (and even Windows!) who could get growl alerts, and not having any growl servers available for FreeBSD, I decided to dive in a little deeper.

After searches for a stand-alone growl server came up empty, I started looking for anyone who had implemented a server in another language like python, ruby or perl. Lo and behold, I found one in python! Just one problem: I use KDE, and that one was for gnome/dbus. Luckily the code is quite easy to follow and it was easy to modify. The kdialog program in KDE already supports growl style alerts by using kdialog –passivepopup, so it was just a matter of ripping out the gnome/dbus bits and replacing them with a call to kdialog.

Without further ado, I present: kdegrowl.py:

Continue reading

Feb 26

More PHP Woes: PHP 5.2.8 and libxml 2.7.x

EDIT: PHP 5.2.9 is out for FreeBSD, so this may be fixed.

There seems to be a bug in the way PHP’s xml library handles data when compiled against libxml 2.7.x. Supposedly, this will be fixed in the next release of PHP, but for now you have to back down to libxml 2.6.x, or compile against expat instead of libxml.

I first noticed this with a Joomla installation, a component was erroring out, saying “Fatal error: Call to a member function getTagName() on a non-object” (Z Weather, for those interested in knowing). Investigating this led me to this bug entry for PHP. I then found another server of mine with problems, this one was using XMLRPC and was getting back responses stripped of the < and > characters, rendering returned HTML code quite broken.

For those of you on FreeBSD who have no idea how to downgrade to the earlier version, it’s actually pretty simple, it can be done like so:

# cd /usr/ports/ports-mgmt/portdowngrade/
# make DEFAULT_CVS_SERVER="anoncvs@anoncvs1.FreeBSD.org:/home/ncvs" install clean
# portdowngrade libxml2

When presented with the choice, choose textproc/libxml2 (Probably option #2).
It will then start listing all prior version of the libxml2 port. When you see version 2.6.32, press enter. Use the most recent copy of 2.6.32, for me it was timestamped 2008/11/19 19:23:07.
Press the number (probably 3) at the start of the line for the version you want, and let portdowngrade do its thing. As suggested by the output of portdowngrade, finish up like so:

# portsdb -Uu
# portupgrade -f libxml2
# /usr/local/etc/rc.d/apache22 restart

Be aware that the portsdb -Uu run can take a while on older systems. Also, you may substitute that last line with whatever command you typically use to restart apache (shutdown and start again, not a graceful restart)

Danger Will Robinson!: Note that if you update your ports tree it will bring libxml2 back to the most recent version, be careful not to upgrade it again until after the next release of PHP!

Sep 22

PHP Crashes Caused By Extension Ordering: A Workaround

As I posted about nearly a year ago, I was (and still am) seeing Apache crashes caused by PHP extension ordering issues. So far, there has been no official or even unofficial workaround for the problem. I wrote a small shell script (/bin/sh for better portability) that will reorder the extensions in php.ini into the order that seems to cause the least problems for me.

Suggestions and improvements are more than welcome. I submitted this script to the PHP port maintainer for FreeBSD but have not yet heard back, which could be due to the hackishness of my script…

Anyhow, I’m pleased to announce that It Works For Me(TM) and you’re welcome to try it: fixphpextorder.sh

You may have to edit the file to correct pathnames and such, but if you build PHP from FreeBSD’s ports system, it should work. It’s especially nice when used with portupgrade like so:

portupgrade -A /root/bin/fixphpextorder.sh php5-\*

That will cause portupgrade to execute that script after each module is rebuilt. This will help if you have any cron or CLI PHP scripts that would reload modules while the upgrade is happening. I tried this method on several servers and it worked well. The only problem was a server running Cacti that polls every 5 minutes. I had two crashes while the upgrade was going on, but that is far better than the dozens it was getting when doing this by hand.

Update 11/21/07: I updated the script to also put pspell.so at the end of the file. It needs to be loaded after spl.so or PHP may crash Apache when a process terminates — either with a full shutdown or when an extra forked process is killed.

Update 6/25/08: Script updated to ensure spl.so comes before mysqli.so, which caused problems with PHP when used at the command line (CLI). Reported by Octaviao Ionescu.

Update 2/22/09: I updated the script again. I found that now sqlite.so must come after spl.so, or it complains about missing symbols. I also moved xml to the end hoping to fix another issue, but it did not help. It didn’t hurt, either, so I left the change in. Let me know if there are problems.

Update 1/21/10: Another script update. Had more crashes until I moved pdo/pdo_sqlite/pdo_mysql around a bit.

May 13

PHP Crashes Caused By Extensions II

As I wrote about previously, I have had problems with Apache and PHP crashing due to various PHP Extensions. I have come upon another combination that triggers a problem, but after investigating it a little I see that it’s been reported before, and nobody wants to fix it. PHP blames PHP accelerator systems, and Zend claims it’s a shared memory configuration problem (it isn’t — at least on my system)

The error happens whenever attempting a graceful restart of Apache via “apachectl graceful”:

  • [notice] seg fault or similar nasty error detected in the parent process

The environment:

  • Apache 2.2.4
  • PHP 5.2.1
  • Zend Optimizer 3.2.8

The culprit:

  • Some interaction between the Zend Optimizer being loaded along with the PHP pspell module.

If I disable one or the other, the crash goes away. Since this particular installation does not require the pspell module, I disabled it and things have been stable ever since.

I did follow Zend’s recommendations for increasing certain shared memory tunables on FreeBSD, as well as trying to recompile everything involved. For more information on shared memory tuning check the FreeBSD man page tuning(7) as well as this Zend Knowledge Base article. Note that certain sysctl settings may only be modified at boot time via /boot/loader.conf and/or /etc/sysctl.conf.

More information to come if I can find anything else…

Update 11/21/2007 – I found that in more recent version of PHP (Around 5.2.4-5.2.5) having pspell.so loaded before spl.so in extensions.ini will result in crashes when an httpd process is stopped/killed. Moving pspell anywhere after spl will clear this up (so far…).

Feb 17

Daylight Saving Time Strikes Again! Well, almost.

We lucky folks in Indiana have had a rough two years dealing with time. As I wrote about last year, Indiana just started observing Daylight Saving Time (DST) in 2006. Now, for 2007 we also have to change the dates on which DST starts and ends. DST now begins on the second Sunday in March, and ends on the first Sunday in November — This year it’s March 11th and Nov 4th. Why on earth we didn’t just wait to start along with the new rules is anyone’s guess. <rant>I don’t think we should be using DST at all, but that’s a story for another time</rant>

Here I was, all set for another round of server updates, reboots, etc. Turns out that I didn’t need to worry quite so much. When I updated all of the time zone files on our servers last year for Indiana’s initial DST switch, they had already made the changes with the new start and end dates for 2007 and beyond. You can confirm this on most UNIX systems as follows:

# zdump -v /etc/localtime | grep 2007
/etc/localtime  Sun Mar 11 06:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 EST isdst=0 gmtoff=-18000
/etc/localtime  Sun Mar 11 07:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime  Sun Nov  4 05:59:59 2007 UTC = Sun Nov  4 01:59:59 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime  Sun Nov  4 06:00:00 2007 UTC = Sun Nov  4 01:00:00 2007 EST isdst=0 gmtoff=-18000

If it says “Mar 11” and “Nov 4” you’re good. If it says “Apr 1” and “Oct 28” you need to update your time zone definitions. On FreeBSD, this can be as simple as downloading new zoneinfo files, recompiling them, and re-selecting the timezones:

  1. Download: ftp://elsie.nci.nih.gov/pub/tzdata2007b.tar.gz
  2. Exctact the contents to /usr/src/share/zoneinfo
  3. cd /usr/src/share/zoneinfo; make install
  4. tzsetup
  5. Choose your time zone again.

A reboot may be necessary to ensure that all running programs are on the same time zone. Currently running programs may not pick up the change. You could also update FreeBSD to a recent version, which includes these changes. If you choose to do the OS update, be sure to run “tzsetup” afterward to be absolutely certain that a new tz file gets installed to /etc/localtime. After you’re done, re-run the zdump command above to be check that you now have the proper DST change dates for 2007.

If you are running any Cisco gear (or other IOS-alike devices) this should work to make the change:

clock timezone EST -5
clock summer-time EDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00

Adjust the time zone to yours, of course.

There may be other programs that handle time zone data internally (such as Java and Outlook) so you’ll have to be sure there are no loose ends in that department. Those of us in Indiana have some practice with this, so at least for us it may not be that bad.

I am aware that many of these problems could be avoided by using UTC on all our server clocks. While that may be preferable, we like to have everything in local time. It’s a choice, and we deal with the consequences. One of which is we never schedule jobs to run overnight between 1-3am — they could be run twice or not at all.

UPDATE: 3/1/07: I have also been informed that you can copy the “/etc/localtime” file from an updated system to any other system that needs it. This could be especially useful if you are unable to update all of the Time Zone definitions for any particular reason.

Oct 18

PHP Crashes Caused By Extensions

Once again when faced with updating PHP on a few servers, I encountered my favorite of all PHP quirks: After rebuilding extensions, PHP crashes and/or takes Apache down with it. Here are the errors that tend to show up:

  • exited on signal 11 (core dumped)
  • exited on signal 6 (core dumped)
  • seg fault or similar nasty error detected in the parent process

And my personal favorite:

  • httpd in free(): error: junk pointer, too high to make sense

I have seen this on PHP4 and PHP5, and with Apache 1.3 and 2.x. I’m not sure if it’s a problem inherent to how the FreeBSD ports system builds and installs the modules or if it’s just a problem in general. I had read once upon a time that rebuilding extensions in a certain order would fix it, and it did. I never got around to figuring out why this worked. Turns out, rebuilding them doesn’t really matter, but the order of the extensions being loaded does. Rebuilding fixed it because when a php extension port is rebuilt, it gets placed at the end of extensions.ini. I solved the problem by editing /usr/local/etc/php/extensions.ini and placing the lines for mysql, imap, and sockets at the end and in that order:

...
extension=mysql.so
extension=imap.so
extension=sockets.so

I’m not sure if the conflict is only with those three, or with others as well, but that fixes it on my servers. I tried it on three different setups, and before the change they all crashed and after the change they’re all running OK.

Hopefully if anyone else runs across this, it will help. If I get more time, I’ll dig into it more later.

Update (11/25/06):
There has been some more discussion of this on the FreeBSD-Ports mailing list and the FreeBSD-STABLE mailing list. Apparently at least part of this is due to the PHP recode, MySQL, and IMAP extension ordering. These extensions rely on c-client libraries with different overloaded hash functions. So the “magic” order at the end of extensions.ini should be:

...
extension=recode.so
extension=mysql.so
extension=imap.so
extension=sockets.so

There is also talk of building some logic into the PHP extension ports to ensure the ordering of the extensions so as to avoid this bug. Best of luck to those working on it!

Edit 8/25/07: I wrote a very hackish shell script that gets the job done keeping the extensions in this order. It’s not pretty, but it works. It can be found here: http://www.pingle.org/files/fixphpextorder.sh. Read the full post here.

Edit 11/21/07: Lately pspell has also become picky about ordering. I recommend placing it at the end (or at least anywhere after spl.so)

...
extension=recode.so
extension=mysql.so
extension=imap.so
extension=sockets.so
extension=pspell.so

Sep 05

FreeBSD On The Desktop (Part IV: A New Hope)

Due to my recent bad luck with electricity, I was using my home server as a desktop all last week. As a result, I have some more notes to add about using FreeBSD as a Desktop machine, which I hope others may find useful.

Read on for more about Printing, Firefox and Thunderbird interoperability, mounting a USB mass storage device, CD burning with K3B, and Video playback.

Continue reading

May 11

Key Bindings in Bash and Vi

So I was typing away in an ssh window today when for the billionth time I had hit the END key expecting the cursor to jump to the end of the line, and it just printed a ~. As always, I just erased the character and then held down the arrow until I was at the end of the line. However, seeing as it was the billionth time I decided to figure out how to make it actually do what I want.

Not that I didn’t really know how to make it do what I want, I’ve just always been too lazy to actually look up the escape codes and such for the home and end keys, and then actually create the entries to fix it. So for the benefit of any other fellow lazy people, here’s what you need to do:

Create ~/.inputrc and put this in it:

“\e[1~”: beginning-of-line
“\e[4~”: end-of-line
“\eOH”: beginning-of-line
“\eOF”: end-of-line
“\e[H”: beginning-of-line
“\e[F”: end-of-line
That should cover most of the bases terminal-wise.

And to make sure that home/end work in Vi, this is what I added to .exrc:
map [CTRL-V] [HOME] ^
map [CTRL-V] [END] $

When I say [CTRL-V] I mean actually press the keys ctrl and v, not type that out, of course.

There are probably better ways to accomplish this, but this worked for me. Feel free to suggest a more elegant solution.