I must thank my fellow DBA Franky Weber Faust for his publication in Portuguese OTN.
Purpose: To present the benefits and reasons for using ACFS and to initially configure ACFS for use.
The Automatic Storage Management Cluster File System (ACFS) was introduced in the second release of Oracle 11g in 2009 and initially there was no licensing cost as well as the other components of the Grid Infrastructure product. In version 11.2.0.2 it was renamed CloudFS and next to it came a licensing cost. In September 2014 Oracle did it again for no cost and remains so until now. You do not have a license fee to use any of the Grid Infrastructure components, but you can only do so if you have another licensed product such as Oracle Database or Oracle Enterprise Linux Premier Support.
In the latest versions of the Oracle Database Appliance CloudFS (ACFS) has become the default file system.
ACFS extends the capabilities of using ASM and provides a multi-platform and scalable file system for the purpose of storing any type of file, including audio, video or image files. It complies with POSIX standards for Linux and Unix operating systems and also with Windows standards.
For the operation of ACFS we need to configure ADVM (AMS Dynamic Volume Manager, which is responsible for delivering and managing volumes.) These volumes can be part of a cluster and can also be formatted on other file systems, in addition to ACFS, such as EXT4, EXT3, etc.
The following figure exemplifies the layers involved when we have ASM with ADVM and ACFS configured:
To use ACFS we need to have Grid Infrastructure configured, so we can not use ACFS to store Voting Disk, OCR and Grid Infrastructure Home and Diagnostic Dest. As the figure shows, ACFS is above ASM. Starting with version 12.1.0.1, ACFS now supports storing datafiles for databases of versions 11.2.0.4 or above.
There are some advanced features that bring advantages in using ACFS:
- Snapshots;
- Cryptography;
- Safety;
- Audit;
- Replication;
- HANFS;
- Tagging.
Some of these features have special issues in the handling of user licenses. See the table below that is presented in the documentation:
Note that all features are free from licensing if they are not used for database files
There are several benefits to using ACFS, and these benefits also add to those of ASM and ADVM:
- No need to purchase any other volume managers;
- Being part of a cluster, we gain in availability;
- It is easy to manage and eases administration;
- Monitoring can be centralized;
- Streamlines the provisioning and copying of databases;
- It can be easily displayed on all cluster nodes, including hub and leaf nodes in Flex Clusters.
If one of the hindrances would be the consumption of resources for the use of the ASM instance, now from version 12c this is no longer an excuse, because with the Flex ASM architecture no more is needed one instance per server.
In the figure above there are 5 nodes and only 3 of them have an ASM instance, the others access using the internal ASM network, which facilitates communication between the instances.
The step by step approach to having an ACFS filesystem is simple:
- Install the Grid Infrastructure;
- Configure ASM;
- Create the disk groups;
- Create the ADVM volumes;
- Format the volumes as ACFS;
- Create the directories that will be used by filesystems;
- Mount the filesystems.
After this brief introduction on ACFS we will see how to create the volumes in ADVM to then format them and mount the filesystems.
Initially, I have a new disk added to my Linux servers srv1 and srv2 (this operation should be performed from one of the nodes, only), so I will create a partition:
[root@srv1 ~]# fdisk /dev/sdf
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x7f9518b1.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):
Using default value 130
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Let’s check if the partition was created:
[root@srv1 ~]# fdisk -l /dev/sdf Disk /dev/sdf: 1073 MB, 1073741824 bytes 255 heads, 63 sectors/track, 130 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x7f9518b1 Device Boot Start End Blocks Id System /dev/sdf1 1 130 1044193+ 83 Linux
Done this we will create the disk in ASM with the name of ADVMVOL1:
[root@srv1 ~] # oracleasm createdisk ADVMVOL1 /dev/sdf1 Writing disk header: done Instantiating disk: done
Let’s list the discs to see what we have:
[root@srv1 ~] # oracleasm listdisks ADVMVOL1 CONFIG1 CONFIG2 CONFIG3 DATA1
Now we switch to the oracle user to access ASM:
[root@srv1 ~]# su - oracle [oracle@srv1 ~]$ . oraenv ORACLE_SID = [oracle] ? +ASM1 The Oracle base has been set to /u01/app/oracle
We go to the ASM instance:
[oracle@srv1 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.1.0.2.0 Production on Sat Oct 15 16:10:04 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL>
Now let’s list the disks to create the diskgroup:
SQL> set pages 200 lin 200 SQL> col path for a50 SQL> select path, header_status from v$asm_disk order by 2; PATH HEADER_STATUS ------------------------------------------ ------------ /dev/oracleasm/disks/CONFIG3 MEMBER /dev/oracleasm/disks/DATA1 MEMBER /dev/oracleasm/disks/CONFIG2 MEMBER /dev/oracleasm/disks/CONFIG1 MEMBER /dev/oracleasm/disks/ADVMVOL1 PROVISIONED
Next we create the diskgroup, I’ll call it ADVM:
SQL> create diskgroup ADVM external redundancy disk '/dev/oracleasm/disks/ADVMVOL1' attribute 'au_size' = '1M'; Diskgroup created.
Now, let’s create a volume using ASMCMD, and to create the volume we have to change the attributes of the diskgroup:
SQL> exit Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options [oracle @ srv1 ~] $ asmcmd -p ASMCMD [+]> volcreate -G ADVM -s 500M ADVMVOL1 ORA-15032: not all alterations performed ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher (DBD ERROR: OCIStmtExecute)
Let’s go back to SQL * Plus and change some attributes of our diskgroup so we can use ADVM. Oracle by default returns the compatibility for an earlier version, in our case as we want to have all the new features of 12c we will set everything to 12.1.0.2:
ASMCMD [+]> exit [oracle@srv1 ~] $ sqlplus / as sysasm SQL * Plus: Release 12.1.0.2.0 Production on Sat Oct 15 16:47:29 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter diskgroup ADVM set attribute 'compatible.asm' = '12.1.0.2'; Diskgroup altered. SQL> alter diskgroup ADVM set attribute 'compatible.advm' = '12.1.0.2'; Diskgroup altered. SQL> alter diskgroup ADVM set attribute 'compatible.rdbms' = '12.1.0.2'; Diskgroup altered.
Now let’s create our volume:
SQL> exit Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options [oracle@srv1 ~] $ asmcmd -p ASMCMD [+]> volcreate -G ADVM -s 500M ADVMVOL1
Let’s see the volumes created in the ADVM diskgroup:
ASMCMD [+]> volinfo -G ADVM -a Diskgroup Name: ADVM Volume Name: ADVMVOL1 Volume Device: /dev/asm/advmvol1-301 State: ENABLED Size (MB): 512 Resize Unit (MB): 64 Redundancy: UNPROT Stripe Columns: 8 Stripe Width (K): 1024 Usage: Mountpath:
I’m going back to SQL * Plus to show that it’s also possible to create volumes:
ASMCMD [+]> exit [oracle@srv1 ~] $ sqlplus / as sysasm SQL * Plus: Release 12.1.0.2.0 Production on Sat Oct 15 17:09:34 2016 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter diskgroup ADVM add volume ADVMVOL2 size 100M; Diskgroup altered.
For SQL*Plus itself, we were also able to query information from our volumes in the v$ASM_VOLUME view:
SQL> select group_number, volume_number, size_mb, usage, state from v $ asm_volume; GROUP_NUMBER VOLUME_NUMBER SIZE_MB USAGE STATE Eur-lex.europa.eu eur-lex.europa.eu ----------- -------- 3 1 512 REMOTE 3 2 128 REMOTE
Now through the operating system we can see the volumes created:
SQL> exit Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options [oracle@srv1 ~] $ ls -lh /dev/asm/ Total 0 brwxrwx --- 1 root oinstall 251, 154113 Oct 15 16:59 advmvol1-301 brwxrwx --- 1 root oinstall 251, 154114 Oct 15 17:35 advmvol2-301
We have the 2 volumes listed, Oracle inserts a suffix in the name of the volumes to guarantee their unique names. Notice in the first character the letter “b” which means that this file is a block device in the operating system.
Now let’s format these volumes. The first one will format as ACFS and for this I need to do the operation as root:
[oracle@srv1 ~]$ su - Password: [root@srv1 ~]# mkfs -t acfs /dev/asm/advmvol1-301 mkfs.acfs: version = 12.1.0.2.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/advmvol1-301 mkfs.acfs: volume size = 536870912 ( 512.00 MB ) mkfs.acfs: Format complete.
The other volume I will format as EXT4 to demonstrate that this can also be done:
[root@srv1 ~]# mkfs -t ext4 /dev/asm/advmvol2-301 mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) Stride=0 blocks, Stripe width=0 blocks 32768 inodes, 131072 blocks 6553 blocks (5.00%) reserved for the super user First data block=1 Maximum filesystem blocks=67371008 16 block groups 8192 blocks per group, 8192 fragments per group 2048 inodes per group Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729 Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 23 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
Note the difference in the output of the two commands. To format in EXT4 is not used any utility of Oracle but of the own Linux that is who formats for this type of filesystem. Formatting on any other filesystem other than ACFS we have lost the ability to use the ACFS snapshot, replication, encryption, and so on the feature on this volume.
Let’s now mount these filesystems in some directory:
[root@srv1 ~]# mkdir -p /mnt/advm/fs1 [root@srv1 ~]# mount -t acfs /dev/asm/advmvol1-301 /mnt/advm/fs1/ [root@srv1 ~]# mkdir -p /mnt/advm/fs2 [root@srv1 ~]# mount -t acfs /dev/asm/advmvol2-301 /mnt/advm/fs2/ mount.acfs: ACFS-00591: error found in volume disk header mount.acfs: ACFS-02037: File system not created on a Linux system. Cannot mount.
Note that when I tried to mount advmvol2 it was not possible as this is formatted for EXT4 and I tried to mount it as ACFS. I’m going to mount it now as EXT4:
[Root@srv1 ~] # mount -t ext4 /dev/asm/advmvol2-301 /mnt/advm/fs2/
Now it worked. Let’s list our assembly points:
[root@srv1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_srv1-lv_root 50G 21G 27G 44% / tmpfs 2.0G 1.2G 759M 62% /dev/shm /dev/sda1 485M 78M 382M 17% /boot /dev/mapper/vg_srv1-lv_home 45G 199M 43G 1% /home /dev/asm/advmvol1-301 512M 40M 473M 8% /mnt/advm/fs1 /dev/asm/advmvol2-301 124M 5.6M 113M 5% /mnt/advm/fs2
And how did it get on the srv2 server? Let’s check:
[oracle@srv2 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_srv1-lv_root 50G 22G 26G 46% / tmpfs 2.0G 1.3G 743M 63% /dev/shm /dev/sda1 485M 78M 382M 17% /boot /dev/mapper/vg_srv1-lv_home 45G 181M 43G 1% /home
There are none of the volumes assembled. Let’s create the directories and mount them:
[oracle@srv2 ~]$ su - Password: [root@srv2 ~]# mkdir -p /mnt/advm/fs1 [root@srv2 ~]# mkdir -p /mnt/advm/fs2 [root@srv2 ~]# mount -t acfs /dev/asm/advmvol1-301 /mnt/advm/fs1/ [root@srv2 ~]# mount -t ext4 /dev/asm/advmvol2-301 /mnt/advm/fs2/ [root@srv2 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_srv1-lv_root 50G 22G 26G 46% / tmpfs 2.0G 1.3G 743M 63% /dev/shm /dev/sda1 485M 78M 382M 17% /boot /dev/mapper/vg_srv1-lv_home 45G 181M 43G 1% /home /dev/asm/advmvol1-301 512M 78M 435M 16% /mnt/advm/fs1 /dev/asm/advmvol2-301 124M 5.6M 113M 5% /mnt/advm/fs2
Success. Let’s do a test generating a file on the srv1 server:
[root@srv1 ~]# touch /mnt/advm/fs1/teste_advm [root@srv1 ~]# ls -lh /mnt/advm/fs1/ total 64K drwx------ 2 root root 64K Oct 15 18:09 lost+found -rw-r--r-- 1 root root 0 Oct 15 18:13 teste_advm
Because / mnt / advm / fs1 is an ACFS filesystem this directory is in the cluster and the created file must also be visible from srv2. Let’s check
[root@srv2 ~]# ls -lh /mnt/advm/fs1 total 64K drwx------ 2 root root 64K Oct 15 18:09 lost+found -rw-r--r-- 1 root root 0 Oct 15 18:13 teste_advm
Great! It worked. The file is appearing on the 2 nodes of the cluster.
Let’s do the same test for the filesystem formatted in EXT4:
[root@srv1 ~]# touch /mnt/advm/fs2/teste_advm2 [root@srv1 ~]# ls -lh /mnt/advm/fs2/ total 12K drwx------ 2 root root 12K Oct 15 17:40 lost+found -rw-r--r-- 1 root root 0 Oct 15 18:16 teste_advm2
In srv1 ok, we will see list in srv2 to see if it appears:
[root@srv2 ~]# ls -lh /mnt/advm/fs2/ ls: cannot access /mnt/advm/fs2/teste_advm2: Input/output error total 12K drwx------ 2 root root 12K Oct 15 17:40 lost+found -????????? ? ? ? ? ? teste_advm2
Looks like you screwed up our filesystem, did not it? As it is EXT4 it is not clustered and should only be presented to the server which will be used, so let’s take it out of srv2:
[root@srv2 ~]# umount /mnt/advm/fs2/ [root@srv2 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_srv1-lv_root 50G 22G 26G 46% / tmpfs 2.0G 1.3G 743M 63% /dev/shm /dev/sda1 485M 78M 382M 17% /boot /dev/mapper/vg_srv1-lv_home 45G 181M 43G 1% /home /dev/asm/advmvol1-301 512M 78M 435M 16% /mnt/advm/fs1
We have seen a very simple example of using ACFS. This is how we started learning this fantastic filesystem. ACFS can also be used to centralize the database and application logs and can also be used as a shared Oracle Home for the Oracle Database installation binaries.
I will bring in a few more articles using the advanced features of ACFS.
ACFS is already a well-developed filesystem and has many features that in other Clustered Filesystems such as IBM GPFS, Red Hat GFS or Veritas CFS would have licensing costs.
References:
Https://docs.oracle.com/cd/E11882_01/server.112/e18951/whatsnew.htm#OSTMG94051
Http://www.oracle.com/technetwork/database/database-technologies/cloud-storage/cloudfs-datasheet-279846.pdf
Http://www.oracle.com/technetwork/database/database-technologies/cloud-storage/cloudfs-overview-wp-279856.pdf
Http://www.oracle.com/technetwork/products/cloud-storage/cloudfs-12c-overview-wp-1965426.pdf
Https://docs.oracle.com/database/121/OSTMG/GUID-C91D3A8E-F329-44BE-97B7-8AB41897D8E1.htm#OSTMG31000
Https://docs.oracle.com/database/121/DBLIC/editions.htm#CIHDDJCJ
Http://www.oracle.com/technetwork/database/database-technologies/cloud-storage/benefits-of-oracle-acfs-2379064.pdf
