-- Main.ctucker - 23 Nov 2010
How to Mount Kerberized NFS Shares as a Member of the CSCF Active Directory
Document UNDER DEVELOPMENT and intended for use in CSCF - University of Waterloo
For the purposes of this document, the NetApp server providing NFS shares is called fs02.student.cs.uwaterloo.ca
and the Ubuntu host which will be the client of this NFS server is called ubuntutest01.cs.uwaterloo.ca
. Both systems will be made members of the CS-TEACHING
Kerberos realm) within the CSCF
Active Directory forest.
Configuring The NetApp Server
Configuration of the NetApp server for kerberized NFS service follows consistently from sections 5.2 and 5.3 of Network Appliances Inc.'s white paper, 'Unified Windows and UNIX Authentication Using Microsoft Active Directory Kerberos'
by Ellie Berriman, 2006.
To start, the following NetApp server files must have the correct information. These files are normally accessed through a mount point on an administrative UNIX system such as cscf.cs:/autofs/NetApp/fs02.student.cs
. From there the path follows the mount point, through the NetApp volume root to the etc
directory. In the case of files under /etc
, they would be found on cscf.cs
hosts: files dns nis
passwd: files ldap
netgroup: files nis
group: files ldap
shadow: files ldap
defines the shares to be provided by the NFS server. For the purposes of this exercise we will be attempting to mount personal user disk space from the sandbox
share. This share's security settings are strictly kerberos (no sys
option as on the home
share). It is important to ensure that sys
security is not available on a share that must use kerberos because NFS client applications will default to sys
if a kerberized connection cannot be done - we definitely do not want that to occur. Although the shares are defined they do not become available (or update) until the exportfs -a
command is run on the NetApp server - see below.
As a sidebar, once configuration of the NFS server and client systems is complete and successfully using kerberos, an administrator can expand the permissions on a share like sandbox such that legacy systems with no TCP based kerberos client (our AD only uses TCP for kerberos operations) may still use the share using the old sys security mechanism. Chris Kranz's blog A NetApp Technical Diary outlines, with several examples, how various permission strategies can be accomplished in a NetApp exports file. Below is example of a modified sandbox share where the hosts cpu18.student.cs, student.cs and cscf.cs may obtain read/write access to the share through sys whereas all other hosts must use kerberos.
Within the sandbox
share, a user's home directories are located in a fashion consistent with their personal unixHomeDirectory
attribute value. That is /vol/sandbox/u#/long_username
. Unfortunately, the pam_mount.so
module for mounting user disk space to be used on the Ubuntu host or workstation, is too simple minded to expand the information in a user's unixHomeDirectory
attribute to locate the disk space under /vol/sandbox
. A simple /vol/sandbox/u/username
format has to be used to find personal disk space. As a result, all user disk space will need to be symbolically linked to a corresponding /vol/sandbox/u/username
The next step in the configuration of the NetApp server requires logging in directly to fs02.student.cs.uwaterloo.ca
and using the NetApp command interface. Using the options
command, we configure the DNS client for the server.
- options dns.cache.enable on
- options dns.domainname student.cs.uwaterloo.ca
- options dns.enable on
When you set the value for dns.domainname
you may be asked to update the corresponding value in /etc/rc
, this should be done to ensure the domain name is retained after any server reboot.
Now configure the CIFS service on the NetApp. It does not matter whether you intend to use CIFS, this procedure binds the NetApp server to the Active Directory domain. This will result in a computer object for the NetApp server being created within the Active Directory domain. You will be prompted for an account name and password with sufficient privileges to add a computer to the domain.
- cifs setup
Windows Domain Name: student.cs.uwaterloo.ca
Filer AD Name: fs02
Next configure the NTP client for time synchronization services. Kerberos authentication is very sensitive t time differentials between KDC and client systems.
- options timed.enable on
- options timed.proto ntp
- options timed.servers ntp.student.cs.uwaterloo.ca
If you wish users to be able to mount personal disk space directly during login then the following option must be disabled.
- options nfs.mount_rootonly off
Also configure the NetApp's LDAP client as this does not
happen as part of joining the NetApp to the Active Directory through the CIFS setup process above. The settings listed below are consistent with those used to add an Ubuntu system into one of our Active Directory domains (see: ADAddUbuntu
). There is no need to set the ldap.passwd
option in CSCF's current LDAP server setup so it is not listed.
- options ldap.ADdomain student.cs.uwaterloo.ca
- options ldap.base dc=student,dc=cs,dc=uwaterloo,dc=ca
- options ldap.base.group dc=student,dc=cs,dc=uwaterloo,dc=ca
- options ldap.base.netgroup dc=student,dc=cs,dc=uwaterloo,dc=ca
- options ldap.base.passwd dc=student,dc=cs,dc=uwaterloo,dc=ca
- options ldap.enable on
- options ldap.minimum_bind_level anonymous
- options ldap.nssmap.attribute.gecos name
- options ldap.nssmap.attribute.gidNumber gidNumber
- options ldap.nssmap.attribute.groupname displayName
- options ldap.nssmap.attribute.homeDirectory homeDirectory
- options ldap.nssmap.attribute.loginShell loginShell
- options ldap.nssmap.attribute.memberNisNetgroup memberNisNetgroup
- options ldap.nssmap.attribute.memberUid memberUid
- options ldap.nssmap.attribute.netgroupname cn
- options ldap.nssmap.attribute.nisNetgroupTriple nisNetgroupTriple
- options ldap.nssmap.attribute.uid sAMAccountName
- options ldap.nssmap.attribute.uidNumber uidNumber
- options ldap.nssmap.attribute.userPassword userPassword
- options ldap.nssmap.objectClass.nisNetgroup nisNetgroup
- options ldap.nssmap.objectClass.posixAccount user
- options ldap.nssmap.objectClass.posixGroup group
- options ldap.port 389
- options ldap.servers ldap.student.cs.uwaterloo.ca
- options ldap.servers.preferred ldap.student.cs.uwaterloo.ca
- options ldap.ssl.enable off
- options ldap.timeout 20
- options ldap.usermap.attribute.unixaccount unixaccount
- options ldap.usermap.attribute.windowsaccount windowsaccount
- options ldap.usermap.base
- options ldap.usermap.enable off
Now enable Kerberized NFS.
- options nfs.kerberos.enable on
- nfs setup
Enable Kerberos for NFS? y
The filer supports these types of Kerberos Key Distribution Centers (KDCs):
1 - UNIX KDC
2 - Microsoft Active Directory KDC
Enter the type of your KDC (1-2): 2
Kerberos now enabled for NFS.
NFS setup complete.
With this process complete the FS02
computer object in the Active Directory should have the following attributes defined with Kerberos principal names. It is important that these principal names be associated only with this computer object. If any of these names have become assigned elsewhere with the domain then the NetApp server may not be able to bind with the Active Directory and kerberos authentication will fail.
- userPrincipalName: firstname.lastname@example.org
If, at a later time, an administrator chooses to add an alternate hostname and IP address to the NetApp file server (due to firewall issues say) then the alternate hostname (fs02-alt.student.cs.uwaterloo.ca
for example) must be added to the servicePrincipalName
attribute for the NetApp server in a corresponding fashion.
The last step in this process is to create or update the shares for export on the NFS server. This command must be run each time the share definitions in /etc/exports
- exportfs -a (running exportfs -r will also work)
Configuring The Ubuntu Host
The first thing to be cognizant of when preparing an Ubuntu system for kerberized NFS mounting from a NetApp server is that it must have two (2) service principal names (SPN) associated with it in the Active Directory domain (Kerberos realm) where the Ubuntu host or workstation is to authenticate. And according to section 220.127.116.11 of the above mentioned Network Appliances white paper, this definitely requires two (2) user objects to represent the single Ubuntu host or workstation within the Active Directory domain in question. In the case of this exercise these principal names would be as follows.
- host/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA used for logins and network services such as ssh
- This one is regularly created when configuring an Ubuntu client to authenticate from an AD domain as outlined in ADAddUbuntu and ADAddSolaris10
- In accordance to those documents, the corresponding user object name in CSCF's Active Directory domain (CS-TEACHING for this example) for this principal would be ubuntutest01_host
- root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA used for mounting NFS mounts
- In a similar vein, the corresponding user object name in CSCF's Active Directory domain for this principal would be ubuntutest01_root
As with the principal names associated with the NetApp server, the host
SPNs for the Ubuntu host must not be mapped to any other objects than those listed above. Otherwise binding to the Active Directory for authentication may not work.
Standard Ubuntu Active Directory Configuration
Join the Ubuntu Linux host or workstation (ubuntutest01.cs.uwaterloo.ca
in this case) to an Active Directory domain as outlined in the ADAddUbuntu
wiki page. This system will be a client of the above configured NetApp server from where user disk space shall be mounted. For the purpose of this example the Ubuntu system will join (authenticate from) our CS-TEACHING
NOTE: CSCF uses a scripting mechanism on cscf.cs
to speed and stream line the joining process but it is just as valid to perform the join to CS-TEACHING
- During the join process the ubuntutest01_host user object is created in the CS-TEACHING domain for the Ubuntu system. As well, the service principal name (SPN) host/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA is mapped to the userPrincipalName attribute of the new user object.
- NOTE: For a user object representing a computer it is the userPrincipalName attribute that carries the host SPN and not the servicePrincipalName attribute. The same will be true for the root user object and SPN created in the next step.
Create the root User Object and SPN Within the Active Directory
Within the CS-TEACHING
domain create the root
user object and SPN for the Ubuntu host or workstation in a manner similar to that outlined in creating a host SPN and user object
. The difference being that a user object called ubuntutest01_root
will be created and the SPN root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
will be written to its userPrincipalName
attribute. You do not require the same password as used to create the ubuntutest01_host
user object, but it does make this process easier.
For this new user object, note the value of the msDS-KeyVersionNumber
attribute. This is equivalent to the KVNO
number which will be required for the Ubuntu keytab
file later. Every time a user object's password is set or reset this number increments. Therefore each time a keytab
file is updated it is important to know the value of this attribute. See Notes On KVNO Numbers and msDS-KeyVersionNumber Attributes
for more details.
Check The Ubuntu System's Kerberos Client Version
At a command prompt of the Ubuntu system enter the following command and check the krb5-user
package version number.
aptitude show krb5-user
If the version for krb5-user
is 1.8 or greater, then chances are the DES
type encryption may be disabled for the Ubuntu kerberos client. To be noted below, DES-CBC-CRC
is required for NetApp NFS servers when securing shares with kerberos. And even if you have legitimately created a keytab line for the root
instance using DES-CBC-CRC
encryption, it still won't work. And the local rpc.gssd
daemon will generate an error message in the /var/log/syslog
file similar to the one listed below when the mount is attempted (during user login).
Apr 8 12:12:47 nettop1 rpc.gssd: rpcsec_gss: gss_init_sec_context: (major) Unspecified GSS failure. Minor code may provide more information - (minor) No supported encryption types (config file error?)
A recommended workaround for this problem (aside from trying to recompile the kerberos client with DES
enabled) is to add the following line to the [libdefaults]
section of the Ubuntu system's /etc/krb5.conf
allow_weak_crypto = 1
Update The /etc/krb5.keytab File On The Ubuntu System
Because we have already joined the Ubuntu host to an AD domain, there should exist a valid /etc/krb5.keytab
on this system. Instead of transferring the AD generated keytab
file to the Ubuntu system, it is easier to simply add a new line to the Ubuntu systems' keytab
file for the root
SPN using the its local ktutil
utility. This process is similar to the manner used to correct bad keytab
files in the Troubleshooting kinit Errors section of ADAddSolaris10
- Make a backup copy of the existing /etc/krb5.keytab file on the Ubuntu system as a precaution.
- Take note of the value of the msDS-KeyVersionNumber attribute for the ubuntutest01_root user object for the KVNO number.
- At an Ubuntu command prompt, enter the ktutil utility.
- Read in the local keytab file:
- Add a new line for the root SPN:
addent -password -p root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA -k KVNO_number -e encryption_mechanism
- The encryption_mechanism used is the same one used for the host SPN entry in your keytab file. Most often it is either rc4-hmac or DES-CBC-MD5. If you are unsure as to which should be used, it does no harm to create an additional keytab line for a separate encryption mechanism.
- You will be prompted for a password at this point, enter the same complex password used to create the ubuntutest01_root account object in CS-TEACHING
- Add a second line for the root SPN specifically for DES-CBC-CRC encryption:
addent -password -p root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA -k KVNO_number -e DES-CBC-CRC
- Use the same complex password here.
- Write the updated keytab file:
- Quit addent: q
- Initialize the Kerberos client for both principal names.
- kinit -k host/ubuntu.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
- kinit -k root/ubuntu.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
- Create two cron entries to re-run initialization on a daily basis.
- Create an /etc/init.d so that kinit will also occur at boot time.
NOTE: The above steps for updating the /etc/krb5.keytab
are valid, however it is highly recommended that one create a script that will manage this process due to the greater complexity of the keytab
Install and Configure pam_mount and Other Additional Software
- Install the libpam-mount package onto the local Ubuntu host.
- On the local Ubuntu host run, sudo apt-get install libpam-mount.
- This will install the pam_mount.so module required for the local PAM stack as well as the configuration file /etc/security/pam_mount.conf.xml which will be used to define the mounts to be created during user login.
- Within pam_mount.conf.xml add the following lines in the Volume-related section
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u0/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u1/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u2/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u3/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u4/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u5/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u6/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u7/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u8/%(GROUP)" mountpoint="~" />
<volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u9/%(GROUP)" mountpoint="~" />
<msg-authpw>Please Enter Your Password:</msg-authpw>
<logout wait="750000" kill="yes" />
- For the above NFS volume mounts, only one is expected to be successful during user login and the others will fail with an error. However, in the CSCF environment, user home disk space is scattered over several partitions of the NFS server and all partitions need to be checked for each user. This is a consequence of the '~' symbol (home directory path) not expanding in the path option as it does in the mountpoint option.
- For the above NFS paths, the GROUP variable is used to represent the username portion of the unixHomeDirectory path. This is a consequence of CSCF's unique environment: home paths depend on the long form username string (full surname, longer than eight characters) which is not used as the actual login username - a maximum of eight characters and represented by the USER variable in the pam_mount.conf.xml file. The long form username is however the name of a user's default group in the CSCF environment. Thus we use the GROUP variable as part of the path option.
- The logout wait option is required by some X Windows GUIs, such as gdm. During logout, some GUI services attempt to unmount user disk space too soon causing the "device is busy" error found in /var/log/auth.log - the user disk space fails to unmount. The above command string delays the umount process for three quarters of a second to allow all files and directories used by X Windows within the user disk space to be closed. If X Windows is not complete after that period then all session X Windows processes are killed.
- Install the nfs-common as well.
- On the local Ubuntu host run, sudo apt-get install nfs-common.
- This will create the configuration file /etc/default/nfs-common.
- Within the newly created /etc/default/nfs-common file, set NEED_GSSD=yes and NEED_IDMAPD=yes
The Ubuntu host will require a reboot at this point.
Update The PAM Stack to Mount Personal NFS Disk Space At Logon
- Update the PAM stack.
- /etc/pam.d/common-auth needs to be slightly modified from the configuration laid out in ADAddUbuntu in order to accommodate the automatic mounting of user disk space from the NetApp. In particular, the pam_mount.so module handles the initial user password entry and the pam_krb5.so is instructed to use the first password pass obtained from pam_mount.so. For certain login services, such as gdm, pam_mount.so will not use a password or ticket obtained by the pam_krb5.so module. Though this anomaly does not affect authentication, the session stack will hang during the gdm login process if pam_mount.so does not obtain the user password during the auth stack:
auth required pam_mount.so
auth sufficient pam_krb5.so use_first_pass forwardable ignore_root
auth required pam_unix.so use_first_pass nullok_secure
- /etc/pam.d/common-session needs to be slightly modified from the configuration laid out in ADAddUbuntu in order to accommodate the automatic mounting of user disk space from the NetApp:
session optional pam_mount.so
session sufficient pam_krb5.so
session required pam_unix.so
session optional pam_foreground.so
- In this case the pam_mount.so is added to the top of the PAM stack as an option the other lines staying the same.
- The files /etc/pam.d/common-account and /etc/pam.d/common-password do not change from how they are set in ADAddUbuntu#Configuration_Of_The_Ubuntu_PAM.
Some Known Glitches
- pam_mount.so will attempt to mount disk space for root user even though that is a local account on the Ubuntu host with not disk space on the NetApp server.
- This mechanism will fail if a kerberos ticket cache called krb5_machine_realm_name is destroyed. This cache is not directly associated with the Ubuntu kerberos client and is not affected if a kdestroy command is issued. However, like any file it can be deleted by the root user. A system reboot will do the same thing since all kerberos tickets are cached in /tmp by default.
To fix this situation the rpc.gssd daemon must be restarted.
service gssd restart
A system reboot will have the same result.
The krb5_machine_realm_name cache will reappear after the first login with a user mounting kerberized NFS disk space.
We should come up with a mechanism to auto renew this cache on a daily basis.
Contents Of Common Kerberos Cache Files
Below is the contents of /tmp/krb5cc_0
, created as a result of the kinit
command when the kerberos client is initialized. The default principal is always the last one to be initialized by the system.
root@cseng10:/tmp# klist -e -c /tmp/krb5cc_0
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: root/cseng10.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
Valid starting Expires Service principal
12/01/10 10:57:44 12/01/10 20:57:44 krbtgt/STUDENT.CS.UWATERLOO.CA@STUDENT.CS.UWATERLOO.CA
renew until 12/02/10 10:57:44, Etype (skey, tkt): ArcFour with HMAC/md5, ArcFour with HMAC/md5
Below is the contents of /tmp/krb5cc_machine_STUDENT.CS.UWATERLOO.CA
which is created by the rpc.gssd
daemon as part of authenticating mounts from the NetApp NFS server fs02.student.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
and potentially others. Note that the encryption type for addressing the CS-TEACHING
domain is rc4-hmac
(though for some systems it is DES-CBC-MD5
) whereas the encryption type for addressing the NFS server is DES-CBC-CRC
. Hence the requirement of maintaining at least two entries for the root
principal in the Ubuntu host's krb5.keytab
file - each representing a different but required mode of encryption.
root@cseng10:/tmp# klist -e -c /tmp/krb5cc_machine_STUDENT.CS.UWATERLOO.CA
Ticket cache: FILE:/tmp/krb5cc_machine_STUDENT.CS.UWATERLOO.CA
Default principal: root/cseng10.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
Valid starting Expires Service principal
12/01/10 10:58:51 12/01/10 21:01:50 krbtgt/STUDENT.CS.UWATERLOO.CA@STUDENT.CS.UWATERLOO.CA
renew until 12/02/10 10:58:51, Etype (skey, tkt): ArcFour with HMAC/md5, ArcFour with HMAC/md5
12/01/10 10:58:53 12/01/10 20:58:53 nfs/fs02.student.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
renew until 12/02/10 10:58:51, Etype (skey, tkt): DES cbc mode with CRC-32, ArcFour with HMAC/md5