Jay Taylor's notes

Configuring Mac OS X IPSec

Clipped on: 2017-04-24

The main computer on this network is an Apple Titanium PowerBook G4. This is running the excellent Mac OS X operating system which includes a large subset of FreeBSD including the kame IPv6/IPSec stack.

ISAKMP Configuration

Setting up IPSec under Mac OS X is the same as any other system that uses kame, notably FreeBSD. Kame is also available for OpenBSD, but OpenBSD's implementation may make better use of the Soekris hardware crypto accelerator. The kame ISAKMP daemon is named racoon and the configuration files are in /etc/racoon. OS X comes with a reasonable configuration just waiting to be enabled, but there are a few changes that need to be made. In the following snippets the original configuration is blue and modifications are green.

/etc/racoon/psk.txt contains "pre shared secrets", or passwords. This file tells racoon which password to use when negotiating with an IPSec endpoint. The hostname and password need to be added:

      # host          password
      192.168.1.1     secret

/etc/racoon/racoon.conf specifies ISAKMP negotiation parameters. This should be modified to prefer main mode in phase 1 negotiations. As of OSX 10.2 the layout of racoon configuration files has changed. The following configuration blocks were moved from racoon.conf to /etc/racoon/remote/anonymous.conf.

      remote anonymous
      {
          # Use main mode to force authentication
          exchange_mode main,aggressive;
          #exchange_mode aggressive,main;
          ...
      }

The following change keeps racoon from renegotiating a new session key every 30 seconds. The more frequent the rotation the more secure the session, but it is unlikely that anyone will be cracking 3DES in 30 minutes...

      sainfo anonymous
      {
          pfs_group 1;
        
          # Don't rekey so frequently
          lifetime time 30 minutes;
          #lifetime time 30 sec;
        ...
      }

Security Policy Configuration

Once racoon is configured to negotiate IPSec connections, security policy database (SPD) entries are needed. The SPD entries will ensure that packets are routed over a secure channel. IPsec may be tunneled as a packet within a packet, or with the IP header in plaintext and only the payload encrypted. The second choice allows normal routing with encryption only being handled at the source and destination. The AH protocol may be used to ensure that the header is not tampered with.

This example creates an IPSec tunnel which prevents a potential sniffer from analysing network traffic based on the final destination of the packet. The statements are 'sh' shell script, and a complete script is available here.

First define the local and remote destinations to apply the policy to, and determine the local IP address.

      IF=en1                        # the interface being secured

      any=0.0.0.0/0                 # the default route
      gateway=192.168.1.1           # the default gateway

      # local ip address on interface $IF

      my_addr=`ifconfig $IF | grep "inet " | cut -d " " -f 2`

Use setkey to configure the SPD database. The following is one big block that will be executed by setkey upon reaching the EOF line:

      setkey -c << EOF
      
      # flush any existing entries

      flush;
      spdflush;

      # require traffic from $my_addr to $any be tunneled via $gateway

      spdadd $my_addr $any any -P out ipsec esp/tunnel/$my_addr-$gateway/require;

      # require traffic from $any to $my_addr be tunneled via $gateway
      
      spdadd $any $my_addr any -P in ipsec esp/tunnel/$gateway-$my_addr/require;

      EOF

Finally start the ISAKMP daemon!

      # racoon

After racoon starts, which may take a little while, there will be messages in /var/log/system.log showing the Phase 1 and Phase 2 ISAKMP negotiation. Success is indicated by messages saying that the IPsec-SA has been established:

      Jan 1 00:00:00 host racoon: INFO: pfkey.c:1107:pk_recvupdate(): IPsec-SA established: ESP/Tunnel 192.168.1.1->192.168.1.2 spi=1234567(0x1A3BC4) 
      Jan 1 00:00:00 host racoon: INFO: pfkey.c:1319:pk_recvadd(): IPsec-SA established: ESP/Tunnel 192.168.1.2->192.168.1.1 spi=7654321(0x4CB3A1)

MTU

Since the 802.11b interface 'en1' defaults to a MTU of 1500, packets larger than this will need to be fragmented. The interface is not aware IPSec encapsulation is being used and will send out packets of MTU size. The additional data needed for IPSec will cause cause fragmentation. Also, Internet services will send max MTU packets which will be rejected as requiring fragmentation, and that will lead to packet loss. A more comprehensive description, including more details about IPSec secured 802.11b, is available here.

It is simple to change the MTU using ifconfig

      # ifconfig <if> mtu 1350

This is done in the IPSec configuration script

Next: Errata