RAC Filesystem Options

RAC Filesystem Options

DBAs wanting to create a 10g Real Applications Cluster face many configuration decisions. One of the more potentially confusing decisions involves the choice of filesystems. Gone are the days when DBAs simply had to choose between "raw" and "cooked". DBAs setting
up a 10g RAC can still choose raw devices, but they also have several filesystem options, and these options vary considerably from platform to platform. Further, some storage options cannot be used for all the files in the RAC setup. This article gives an
overview of the RAC storage options available.

RAC Review

Let's begin by reviewing the structure of a Real Applications Cluster. Physically, a RAC consists of several nodes (servers), connected to each other by a private interconnect. The database files are kept on a shared storage subsystem, where they're accessible
to all nodes. And each node has a public network connection.

In terms of software and configuration, the RAC has three basic components: cluster software and/or Cluster Ready Services, database software, and a method of managing the shared storage subsystem.
The cluster software can be vendor-supplied or Oracle-supplied, depending on platform. Cluster Ready Services, or CRS, is a new feature in 10g. Where vendor clusterware is used, CRS interacts with the vendor clusterware to coordinate cluster membership
information; without vendor clusterware, CRS, which is also known as Oracle OSD Clusterware, provides complete cluster management.

The database software is Oracle 10g with the RAC option, of course.

Finally, the shared storage subsystem can be managed by one of the following options: raw devices; Automatic Storage Management (ASM); Vendor-supplied cluster file system (CFS), Oracle Cluster File System (OCFS), or vendor-supplied logical volume manager
(LVM); or Networked File System (NFS) on a certified Network Attached Storage (NAS) device.

Storage Options

Let me clarify the foregoing alphabet soup with a table:
Table 1. Storage options for the shared storage subsystem.

Abbrev.	Storage Option
Raw	Raw devices, no filesystem
ASM	Automatic Storage Management
CFS	Cluster File System
OCFS	Oracle Cluster File System
LVM	Logical Volume Manager
NFS	Network File System (must be on certified NAS device)

Before I delve into each of these storage options, a word about file types. A regular single-instance database has three basic types of files: database software and dump files; datafiles, spfile, control files and log files, often referred to as "database files";
and it may have recovery files, if using RMAN. A RAC database has an additional type of file referred to as "CRS files". These consist of the Oracle Cluster Registry (OCR) and the voting disk.

Not all of these files have to be on the shared storage subsystem. The database files and CRS files must be accessible to all instances, so must be on the shared storage subsystem. The database software can be on the shared subsystem and shared between nodes;
or each node can have its own ORACLE_HOME. The flash recovery area must be shared by all instances, if used.

Some storage options can't handle all of these file types. To take an obvious example, the database software and dump files can't be stored on raw devices. This isn't important for the dump files, but it does mean that choosing raw devices precludes having
a shared ORACLE_HOME on the shared storage device.

And to further complicate the picture, no OS platform is certified for all of the shared storage options. For example, only Linux and SPARC Solaris are supported with NFS, and the NFS must be on a certified NAS device. The following table spells out which platforms
and file types can use each storage option.
Table 2. Platforms and file types able to use each storage option

Storage option	Platforms	File types supported	File types not supported
Raw	All platforms	Database, CRS	Software/Dump files, Recovery
ASM	All platforms	Database, Recovery	CRS, Software/Dump
Certified Vendor CFS	AIX, HP Tru64 UNIX, SPARC Solaris	All	None
LVM	HP-UX, HP Tru64 UNIX, SPARC Solaris	All	None
OCFS	Windows, Linux	Database, CRS, Recovery	Software/Dump files
NFS	Linux, SPARC Solaris	All	None

(Note: Mike Ault and Madhu Tumma have summarized the storage choices by platform in more detail in this excerpt from their recent book, Oracle
10g Grid Computing with RAC, which I used as one source for this table.)

Now that we have an idea of where we can use these storage options, let's examine each option in a little more detail. We'll tackle them in order of Oracle's recommendation, starting with Oracle's least preferred, raw devices, and finishing up with Oracle's
top recommendation, ASM.

Raw devices

Raw devices need little explanation. As with single-instance Oracle, each tablespace requires a partition. You will also need to store your software and dump files elsewhere.

Pros: You won't need to install any vendor or Oracle-supplied clusterware or additional drivers. 
Cons: You won't be able to have a shared oracle home, and if you want to configure a flash recovery area, you'll need to choose another option for it. Manageablility is an issue. Further, raw devices are a terrible choice
if you expect to resize or add tablespaces frequently, as this involves resizing or adding a partition.

NFS

NFS also requires little explanation. It must be used with a certified NAS device; Oracle has certified a number of NAS filers with its products, including products from EMC, HP, NetApp and others. NFS on NAS can be a cost-effective alternative to a SAN for
Linux and Solaris, especially if no SAN hardware is already installed.

Pros: Ease of use and relatively low cost. 
Cons: Not suitable for all deployments. Analysts recommend SANs over NAS for large-scale transaction-intensive applications, although there's disagreement on how big is too big for NAS.

Vendor CFS and LVMs

If you're considering a vendor CFS or LVM, you'll need to check the 10g Real Application Clusters Installation Guide for your platform and the Certify pages on MetaLink. A discussion of all the certified cluster file systems is beyond the scope of this article.
Pros and cons depend on the specific solution, but some general observations can be made:

Pros: You can store all types of files associated with the instance on the CFS / logical volumes. 
Cons: Depends on CFS / LVM. And you won't be enjoying the manageability advantage of ASM.

OCFS

OCFS is the Oracle-supplied CFS for Linux and Windows. This is the only CFS that can be used with these platforms. The current version of OCFS was designed specifically to store RAC files, and is not a full-featured CFS. You can store database, CRS and recovery
files on it, but it doesn't fully support generic filesystem operations. Thus, for example, you cannot install a shared ORACLE_HOME on an OCFS device.

The next version of OCFS, OCFS2, is currently out in beta version and will support generic filesystem operations, including a shared ORACLE_HOME.

Pros: Provides a CFS option for Linux and Windows. 
Cons: Cannot store regular filesystem files such as Oracle software. Easier to manage than raw devices, but not as manageable as NFS or ASM.

ASM

Oracle recommends ASM for 10g RAC deployments, although CRS files cannot be stored on ASM. In fact, RAC installations using Oracle Database Standard Edition must use ASM.

ASM is a little bit like a logical volume manager and provides many of the benefits of LVMs. But it also provides benefits LVMs don't: file-level striping/mirroring, and ease of manageability. Instead of running LVM software, you run an ASM instance, a new
type of "instance" that largely consists of processes and memory and stores its information in the ASM disks it's managing.

Pros: File-level striping and mirroring; ease of manageability through Oracle syntax and OEM. 
Cons: ASM files can only be managed through an Oracle application such as RMAN. This can be a weakness if you prefer third-party backup software or simple backup scripts. Cannot store CRS files or database software.

Conclusion

We've seen that there's an array of storage options for the shared storage device in your RAC. These options depend on your platform, and many options don't store all types of database files, meaning they have to be used in conjunction with another option.
For example, a DBA wanting to use ASM to store database files might take a 12-disk SAN, create 11 ASM disks for the database files and flash recovery area, leave the 12th disk raw and store CRS files on it, and maintain separate ORACLE_HOMEs on the non-shared
disks on each node.
Table 3: Sample Disk Configuration w/ ASM

In shared storage subsytem: SAN Disk#	In each node: Internal Disk#
1	2	3	4	5	6	7	8	9	10	11	12	1	2
ASM disks DB files; flash recovery	RAW CRS	OS files	ORA HOME

When weighing the shared storage device options for your platform, start with the Oracle Real Application Clusters Installation and Configuration Guide for your platform, available from OTN. Section III has a platform-specific discussion of storage options.
Be sure to check the certification matrices on MetaLink as well.

» 

Natalka Roshak's blog 

Login to post comments

Oracle RAC Technologies Certification Matrix for Microsoft Windows Platforms 

 

Oracle will support the Oracle software on clusters comprised of RAC compatible technologies and supported software combinations
NOTE: Please consult your hardware and software vendors as not all vendors may choose to support their hardware or software in every possible cluster combination
For RAC database and server virtualization (Oracle VM, LPAR, vPAR, LDOM, etc.) support seehere.
For RAC database and SAP certification information see SAP support note 527843.
Hitachi Compute Blade logical partitioning feature (also known as BladeSymphony Virtage) are supported with Oracle RAC 11.2.0.2 and 12.1.0.1 or higher with Windows x64.

Technology Category	Technology	Notes
Server/Processor Architecture	x86 32-bit Architecture x86-64 (AMD64/EM64T) 64-bit Architecture Itanium Architecture
Network Interconnect Technologies	Ethernet 1 Gigabit NIC's and Switches	Cross over cables are not supported Use correct type of cable (Cat 5, 5e, or 6)
InfiniBand (IB) IP over IB is supported
VIA based NICs and Switches	Must be VIPL 1.0 compliant
Storage Technologies All storage products must be supported by the server (host) and storage vendors	Fibre Channel (FC)	Switch required for greater than two nodes (if the storage system does not have enough FC port for the cluster nodes)
SCSI	Support for two nodes only
iSCSI	Support up to 16 nodes. Private GigE storage network required. The iSCSI accessed storage must be supported by the system and storage vendors.

Submitted by John (not verified) on Fri, 2005-08-12 13:41.

Missed to include one of more complete cluster file systems - PolyServe File System (supports RAC on Windows, SuSE and Red Hat).

Otherwise quite informative.