This post is mainly so there is a record of these error messages and what they might indicate. It’s also to help out others so they don’t shoot themselves in the foot as I did. I found no matching pages on Google when I tried to search for the errors I was seeing; now there should be at least one.
I was following along with this nice Cisco document that does a good job of explaining how to get a PIX and a SonicWall to talk IPSec back and forth. I’ll leave out the details of how I configured both sides as that document gets into more detail than most people need, and I’d rather not repeat things unnecessarily. Even though the SonicWall I was using was considerably older, the settings were similar. I did stray from the document in one way: I used 3DES/SHA. Due to the age of the SonicWall it did not support the newer/stronger methods.
When it was all said and done, the VPN appeared to work for a bit but then stopped, and I was getting errors on both sides. On the PIX side I saw (from “debug crypto isakmp”):
ISAKMP (0): retransmitting phase 2 (0/0)… mess_id 0x1eefa9e
ISAKMP: error, msg not encrypted
In the SonicWall log, I saw:
IKE Responder: IPSec proposal not acceptable
It turns out that I had made a typo in the subnet mask for the LAN side of the SonicWall when I entered it into an ACL on the PIX side. Gun, meet foot. Foot, meet gun. I noticed that on the PIX there were two SAs being created (“show crypto ipsec sa”) one for the proper subnet mask, as given by the SonicWall side, and one for the improper subnet mask as desired by the PIX. After I fixed the ACL and re-entered each of the lines that contained a reference to it, everything was copasetic.