Upgrading Lustre

C H A P T E R 13

Upgrading Lustre

The chapter describes how to upgrade and downgrade Lustre versions and includes the following sections:

Lustre Interoperability

Upgrading Lustre from 1.4.12 to 1.6.6

Upgrading Lustre from 1.6.5.1 to 1.6.6

Downgrading Lustre from 1.6.6 to 1.4.12

13.1 Lustre Interoperability

For Lustre 1.6.6, the following upgrades are supported:

Lustre 1.4.12 (latest 1.4.x version) to Lustre 1.6.6 (latest 1.6.x version).

One minor version to the next (for example, 1.6.5.1 > 1.6.6 and 1.4.11 > 1.4.12).

For Lustre 1.6.6, downgrades in the same ranges are supported.

If you upgrade from Lustre 1.4.12 > 1.6.4, you can downgrade to version 1.4.12.

If you upgrade from Lustre 1.6.5.1 > 1.6.6, you can downgrade to version 1.6.5.1

Caution - A fresh installation of Lustre 1.6.6 is not guaranteed to be downgradable to an earlier Lustre version.

13.2 Upgrading Lustre from 1.4.12 to 1.6.6

Use the procedures in this chapter to upgrade Lustre version 1.4.12 to version 1.6.6.

Note - In Lustre version 1.6 and later, the file system name (--fsname parameter) is limited to 8 characters.

13.2.1 Upgrade Requirements

Remember the following important points before upgrading Lustre.

Upgrade MDT before OSTs. The upgrade procedure is:

1. Shut down lconf failover.

2. Install the new modules.

3. Run tunefs.lustre.

4. Mount startup.

A Lustre upgrade can be done across a failover pair, in which case the upgrade procedure is:

1. On the backup server, install the new modules.

2. Shut down lconf failover.

3. On the new server, run tunefs.lustre.

4. On the new server, mount startup.

5. On the primary server, install the new modules.

Note that when upgrading to Lustre 1.6.x, the file system name must be less than or equal to 8 characters (so it fits on the disk label).

13.2.2 Supported Upgrade Paths

The following Lustre upgrade paths are supported.

Entire file system or individual servers / clients

Servers can undergo a "rolling upgrade", in which individual servers (or their failover partners) and clients are upgraded one at a time and restarted, so that the file system never goes down. This type of upgrade limits your ability to change certain parameters.

The entire file system can be shut down, and all servers and clients upgraded at once.

Any combination of the above two paths.

Interoperability between the nodes

This describes the interoperability between clients, OSTs, and MDTs.

Clients

Old live clients can continue to communicate with old/new/mixed servers.

Old clients can start up using old/new/mixed servers.

New clients can start up using old/new/mixed servers (use old mount format for old MDT).

OSTs

New clients/MDTs can continue to communicate with old OSTs.

New OSTs can only be started after the MGS has been started (typically this means "after the MDT has been upgraded.")

MDTs

New clients can communicate with old MDTs.

New co-located MGS/MDTs can be started at any point.

New non-MGS MDTs can be started after the MGS starts.

Note - The limitation with interoperability is that old clients cannot mount a file system which was created by a new MDT.

Note - If your system is upgraded from 1.4.x to 1.6.x, you can mount the Lustre client on both Lustre versions. If the file system was originallycreated using Lustre 1.6.x, then you will not be able to mount the fildsystem created using an earlier Lustre version.

13.2.3 Starting Clients

You can start a new client with an old MDT by using the old format of the client mount command:

client# mount -t lustre <mdtnid>:/<mdtname>/client <mountpoint>

You can start a new client with an upgraded MDT by using the new format and pointing it at the MGS, not the MDT (for co-located MDT/MGS, this is the same):

client# mount -t lustre <mgsnid>:/<fsname> <mountpoint>

Old clients always use the old format of the mount command, regardless of whether the MDT has been upgraded or not.

13.2.4 Upgrading a Single File system

tunefs.lustre will find the old client log on an 1.4.x MDT that is being upgraded to 1.6. (If the name of the client log is not "client", use the lustre_up14.sh script, described in Step 2 and Step 3.)

1. Shut down the MDT.

mdt1# lconf --failover --cleanup config.xml

2. Install the new Lustre version and run tunefs.lustre to upgrade the configuration.

There are two options:

Rolling upgrade keeps a copy of the original configuration log, allowing immediate reintegration into a live file system, but prevents OSC parameter and failover NID changes. (The writeconf procedure can be performed later to eliminate these restrictions. For details, see Running the Writeconf Command.)

mdt1# tunefs.lustre --mgs --mdt --fsname=testfs /dev/sda1

i.--writeconf begins a new configuration log, allowing permanent modification of all parameters (see Changing Parameters), but requiring all other servers and clients to be stopped at this point. No clients can be started until all OSTs are upgraded.

[root@mds1]# tunefs.lustre --mgs --writeconf --mgs --mdt --fsname=ldiskfs /dev/hda4
checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata
Read previous values:
Target:		testfs-MDT0000
Index:		0
UUID:		mds-1_UUID
Lustre FS:		testfs
Mount type:		ldiskfs
Flags:		0x205
(MDT MGS upgrade1.4 )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:
Permanent disk data:
Target:		ldiskfs-MDT0000
Index:		0
UUID:		mds-1_UUID
Lustre FS:		ldiskfs
Mount type:		ldiskfs
Flags:		0x305
		(MDT MGS writeconf upgrade1.4 )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:
Writing CONFIGS/mountdata
Copying old logs

3. Start the upgraded MDT.

mdt1# mkdir -p /mnt/test/mdt
mdt1# mount -t lustre /dev/hda4 /mnt/test/mdt
mdt1 # df
Filesystem		1K-blocks			Used			Available			Use%		Mounted on
/dev/hda2		10080520			4600820			4967632			49%		/
/dev/hda1		101086			14787			81080			16%		/boot
none		501000			0			501000			0%		/dev/shm
/dev/hda4		23339176			455236			21550144			3%		/mnt/test/mdt

4. Upgrade and start the OSTs for the file system in a similar manner, except they need the address of the MGS. Old installations may also need to specify the OST index (for instance, --index=5).

ost1# tunefs.lustre --ost --fsname=lustre --mgsnode=mds /dev/sda4
checking for existing Lustre data: found last_rcvd
tunefs.lustre: Unable to read /tmp/dirQi2cwV/mountdata (No such file or directory.)
Trying last_rcvd
Reading last_rcvd
Feature compat=2, incompat=0
Read previous values:
Target:
Index:		0
UUID:		ost1_UUID
Lustre FS:		lustre
Mount type:		ldiskfs
Flags:		0x202
		(OST upgrade1.4 )
Persistent mount opts:
Parameters:
Permanent disk data:
Target:		lustre-OST0000
Index:		0
UUID:		ost1_UUID
Lustre FS:		lustre
Mount type:	ldiskfs
Flags:		0x202
		(OST upgrade1.4 )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=192.168.10.34@tcp
Writing CONFIGS/mountdata 11.1.5 Upgrading Multiple File Systems with a Shared MGS 
Ost-1# mount -t lustre /dev/sda4 /mnt/test/ost/
Ost1# df
Filesystem		1K-blocks			Used			Available			Use%		Mounted on
/dev/sda2		10080520			3852036			5716416			41%		/
/dev/sda1		101086			14964			80903			16%		/boot
none		501000			0			501000			0%		/dev/shm
/dev/sda4		101492248			471672			95781780			1%		/mnt/test/ost

13.2.5 Upgrading Multiple File Systems with a Shared MGS

The upgrade order is: MGS first, then for any single file system the MDT must be upgraded and mounted, and then the OSTs for that file system. If the MGS is
co-located with the MDT, the old configuration logs stored on the MDT are automatically transferred to the MGS. If the MGS is not co-located with the MDT (for a site with multiple file systems), the old config logs must be manually transferred to the MGS.

1. Format the MGS node, but do not start it.

mgsnode# mkfs.lustre --mgs /dev/hda4

2. Mount the MGS disk as type ldiskfs.

mgsnode# mount -t ldiskfs /dev/hda4 /mnt/mgs

3. For each MDT, copy the MDT and client startup logs from the MDT to the MGS, renaming them as needed. There is a script that helps automate this process--lustre_up14.sh

mdt1# lustre_up14 /dev/hda4 lustre
debugfs 1.35 (28-Feb-2004)
/dev/hda4: catastrophic mode - not reading inode or group bitmaps
Copying log 'mds-1' to 'lustre-MDT0000'. Okay [y/n]?y
Copying log 'client' to 'lustre-client'. Okay [y/n]?y
ls -l /tmp/logs
total 24
-rw-r--r--  1 root root 9448 Oct 22 17:46 lustre-client
-rw-r--r--  1 root root 9080 Oct 22 17:46 lustre-MDT0000 
mdt1# cp /tmp/logs/lustre-* /mnt/tmp/CONFIGS/
cp: overwrite `/mnt/tmp/CONFIGS/lustre-client'? y
cp: overwrite `/mnt/tmp/CONFIGS/lustre-MDT0000'? y

4. Unmount the MGS ldiskfs mount.

mgsnode# umount /mnt/mgs

5. Start the MGS.

mgsnode# mount -t lustre /dev/hda4 /mnt/mgs

6. Shut down one of the old MDTs.

mdt1# lconf --failover --cleanup config.xml

7. Upgrade the old MDT.

install new Lustre 1.6
mdt1# tunefs.lustre --mdt --nomgs --fsname=testfs \ 
--mgsnode=mgsnode@tcp0 /dev/hda4

(--nomgs is required to upgrade a non-co-located MDT).

8. Start the upgraded MDT.

mdt1# mount -t lustre /dev/hda4 /mnt/test/mdt

9. Upgrade and start OSTs for this file system.

ost1# lconf --failover --cleanup config.xml
install new Lustre 1.6
ost1# tunefs.lustre --ost --fsname=lustre --mgsnode=mgsnode@tcp0 \/dev/sdc 
ost1# mount -t lustre /dev/sdc /mnt/test/ost1

10. Upgrade the other MDTs in a similar manner. Remember:

The MGS must not be running (mounted) when the backing disk is mounted as ldiskfs.

The MGS must be running when first starting a newly-upgraded server (MDT or OST).

13.3 Upgrading Lustre from 1.6.5.1 to 1.6.6

To upgrade Lustre from version 1.6.5.1 to 1.6.6, perform these steps:

1. Check the current Lustre version on the MDS.

root@mds# uname -a 
root@mds# Linux mds.sun.com 2.6.18-8.1.14.el5_lustre.1.6.4.2smp #1 SMP Wed Jan 16 20:49:25 EST 2008 i686 athlon i386 GNU/Linux

2. Check the name of the file system.

[root@mds ~]# cat /proc/fs/lustre/mgs/MGS/filesystems
sunfs

3. Umount the old file system in this order.

Client

[root@client ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              6940516   4051136   2531132  62% /
/dev/sda1               101086     14484     81383  16% /boot
tmpfs                   271844         0    271844   0% /dev/shm
mds@tcp0:/sunfs        4128416    291800   3626904   8% /mnt
[root@client ~]# umount /mnt

4. Cross-check the unmount. You must verify the unmount before upgrading the Lustre version.

[root@client ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              6940516   4051136   2531132  62% /
/dev/sda1               101086     14484     81383  16% /boot
tmpfs                   271844         0    271844   0% /dev/shm

5. Unmount the file system on the MDT and all OSTs in a similar manner.

6. Install new Lustre version and restart the nodes with new kernel on the MGS and MDT.

[root@mds ~]# tunefs.lustre --mgs --writeconf --mgs --mdt --fsname=sunfs /dev/sdb
tunefs.lustre --mgs --writeconf --mgs --mdt --fsname=sunfs /dev/sdb
checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata
Read previous values:
Target:     sunfs-MDT0000
Index:      0
Lustre FS:  sunfs
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups
Permanent disk data:
Target:     sunfs-MDT0000
Index:      0
Lustre FS:  sunfs
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups

7. Write configurations to mount data.

[root@mds ~]# mount -t lustre /dev/sdb /mnt/data/mdt/
[root@mds ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              6940516   4173316   2408952  64% /
/dev/sda1               101086     14548     81319  16% /boot
tmpfs                   271804         0    271804   0% /dev/shm
/dev/sdb               1834832     90196   1639780   6% /mnt/data/mdt

8. Mount the OST.

oss #  mount -t lustre /dev/sdb /mnt/ost

If an error occurs, use this command:

oss # tunefs.lustre --ost --fsname=sunfs --mgssnode=mds /dev/sdb

9. After installing the new Lustre modules, mount the file system on the client side.

client # mount -t lustre mds@tcp0:/sunfs /mnt/client

13.4 Downgrading Lustre from 1.6.6 to 1.4.12

This section describes how to downgrade Lustre version 1.6.6 to version 1.4.12.

13.4.1 Downgrade Requirements

The file system must have been upgraded from 1.4.x. In other words, a file system created or reformatted under Lustre 1.6.x cannot be downgraded.

Any new OSTs that were dynamically added to the file system will be unknown in version 1.4.x. It is possible to add them back using lconf --write-conf, but you must be careful to use the correct UUID of the new OSTs.

Downgrading an MDS that is also acting as an MGS prevents access to all other file systems that the MGS serves.

13.4.2 Downgrading a File System

To downgrade a file system:

1. Shut down all clients.

2. Shut down all servers.

3. Install Lustre 1.4.x on the client and server nodes.

4. Restart the servers (OSTs, then MDT) and clients.

Caution - When you downgrade Lustre, all OST additions and parameter changes made since the file system was upgraded are lost.