Lustre Distributed File System

1 Intro
2 Setting up a Lustre Cluster
3 Managing Lustre
4 Monitoring File System Useage
5 Ubuntu Client
6 Compiling Patchless Client from Source (Red Hat)

Intro

The Luster administration manual is 500 pages, and while filled with great info and very complete it is a bit daunting to do a quick set up. Following these steps you should be able to get a test cluster up in a very short period of time.

NOTE: I have since found this Quick Start Blueprint which is extremely helpful (too bad I didn't find this BEFORE I had to figure most of this out!) General information is also available at the main site where this good FAQ can also be found.

Mailing List: http://lists.lustre.org/mailman/options/lustre-discuss/peby%40sagonet.com lust3r4m3669337

Setting up a Lustre Cluster

Temp Cluster: Our temporary Luster cluster is currently in DC1 3:4 on these IPs.

207.150.181.10 - MDS
207.150.181.12 - OSS1
207.150.181.14 - OSS2

A simple scenario will consist of 3 server: (1) Management Data Server (MDS) and (2) Object Storage Servers (OSS). All three will have CentOS 5.3.

(In the future we might try this config and add the /mds and /oss partitions rather than using entire drives as devices. This way we can run software RAID 1 on the nodes. We also have to look yet at configuring High Availability (HA) services through heartbeat and configuring the OSS nodes using the failover options supported by luster.)

MDS: Software RAID 1 / = 20GB /swap 2GB /mds = remaining

OSS: Software RAID 1 / = 20GB /swap 2GB /oss = remaining

Installing Lustre

Setup Script: I made a script to automatically install all the RPMs needed on the MDS and OSS nodes. To connect from a client to the Lustre Cluster see the section below.

Create the file systems on the nodes:

 
 On the MGS Server:
 # mkfs.lustre --fsname=lustre --mgs --mdt /dev/sdb
 # mount -t lustre /dev/sdb /lustre/
 
 On each OSS Server:
 # mkfs.lustre --ost --fsname=lustre --mgsnode=207.150.181.10@tcp0 /dev/sdb
 #mount -t lustre /dev/sdb /lustre/

Now you can access this using from a client, for example from a CentOS Client

You will need these installed on the client:

[root@cobbler lustre]# rpm -qa | grep lustre
lustre-client-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
lustre-client-modules-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
lustre-modules-1.8.1-2.6.18_128.1.14.el5_lustre.1.8.1
kernel-lustre-2.6.18-128.1.14.el5_lustre.1.8.1

You can then mount the target with:

[root@cobbler]# mount -t lustre 207.150.181.10:/lustre /lustre

And it will show up such as:

[root@cobbler lustre]# df -h

207.150.181.10:/lustre
                      147G  460M  139G   1% /lustre

Or see the OSS mounts in the pool with:

[root@cobbler lustre]# lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
lustre-MDT0000_UUID     130.4G    459.6M    122.5G    0% /lustre[MDT:0]
lustre-OST0000_UUID     146.7G      1.4G    137.8G    0% /lustre[OST:0]
lustre-OST0001_UUID     146.7G    459.6M    138.8G    0% /lustre[OST:1]

filesystem summary:     293.3G      1.9G    276.6G    0% /lustre

Managing Lustre

Please see the Quickstart Blueprint for an introduction to basic managment commands.

Stopping / Starting Storage Nodes:

[root@10-0-0-1 /]# lctl dl
  0 UP mgs MGS MGS 11
  1 UP mgc MGC207.150.181.10@tcp 286db1cc-9673-0e98-2a46-62f221910298 5
  2 UP mdt MDS MDS_uuid 3
  3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
  4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
  5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
  6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5

To deactivate the second OSS server, you would use:

[root@10-0-0-1 /]# lctl dl --device 6 deactivate

Monitoring Striping

See the "Using Strips" section of the Quick Start guide to view and modify how data is striped across OSS nodes.

[root@cobbler lustre]# lfs getstripe /lustre/10megtest.file

Monitoring File System Useage

This project uses snmp to monitor filesystem usage and support Lustre

http://robinhood.sourceforge.net/

http://collectl.sourceforge.net/

Ubuntu Client

(Hummm . . . does not look good. There are some Lustre thins in repos, but not the unpatched client. Might have to build from lustre-source pkg.) (I could not get the module to build correctly. Not sure why. Configure fails saying it is unable to build modules at all.)

archimedes@Anduril:/$ sudo apt-get install lustre-utils lustre-source liblustre

You will also need to have linux-headers-x.y.z for your kernel installed

Then build the module using module-assistant

archimedes@Anduril:/$sudo module-assistant

(The assistant will install any needed build essential tools, then use SELECT: Select module or source package to work on, find the Lustre module and proceed with building it.)

To not use module-assistant, look in /usr/src/modules/lustre for the source file and configure and build manually.

http://wiki.lustre.org/index.php/LibLustre_How-To_Guide