Rebuilding / Moving MGMTDB on Grid Infrastructure 12c

I must thank my fellow DBA Franky Weber Faust for his publication in his blog.
In this article, we are going to know about how to rebuilding/moving MGMTDB on grid infrastructure 12c.

Stop the ora.crf resource on each node in your cluster and disable it so that it does not start automatically:

[root@rac1 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded

[root@rac1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

[root@rac2 ~]# crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded

[root@rac2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init

If your MGMTDB is functional you will be able to remove it through the DBCA. If it has a problem that does not let it start you will not be able to remove it. In my case, the tablespace UNDOTBS1 was corrupted, so it was not possible to open the database and consequently could not delete it via DBCA:

[oracle@rac1 ~]$ /u01/app/12.1.0.2/grid/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB
ORA-01219: database or pluggable database not open: queries allowed on fixed tables or views only

Browsing the MGMTDB alert.log I identified that the problem was the Undo tablespace:

Thu May 04 22:11:27 2017
Errors in file /u01/app/oracle/diag/rdbms/_mgmtdb/-MGMTDB/trace/-MGMTDB_ora_30436.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00376: file 4 cannot be read at this time
ORA-01110: data file 4: '+CONFIG/_MGMTDB/DATAFILE/undotbs1.259.941395093'

So I opted to manually remove the MGMTDB. Use asmcmd to find the datafiles:

ASMCMD> cd +CONFIG/_MGMTDB/
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y 4D39E5F3979F4E3AE0536538A8C0AD9E/
Y 4D3A16FC19245C2AE0536538A8C04036/
Y CONTROLFILE/
Y DATAFILE/
Y FD9AC0F7C36E4438E043B6A9E80A24D5/
Y FD9B43BF6A646F8CE043B6A9E80A2815/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/

Then inside the MGMTDB directory remove all your files:

ASMCMD> rm -rf *
ASMCMD> exit
Remove from the / etc / oratab file the line for MGMTDB:
[oracle@rac1 ~]$ vi /etc/oratab

You now have to remove the MGMTDB resource from the cluster. As we can see the database is still part of the cluster configuration:

 [root@rac2 ~]# srvctl status mgmtdb
Database is enabled
Database is not running.
Remove the MGMTDB from the configuration, this way it will be deleted from the Oracle Cluster Registry (OCR):
[oracle@rac1 ~]$ srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y

Once all this is done, we can start the re-creation. When executing the MGMTDB creation command we can define another diskgroup to store it. Note the -diskGroupName parameter below:

 [oracle@rac1 ~]$ /u01/app/12.1.0.2/grid/bin/dbca -silent -createDatabase -sid -MGMTDB \
-createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb \
-storageType ASM -diskGroupName +GIMR -datafileJarLocation \
$ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck

Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
After the CDB is created we need to create a PDB with the name of our cluster. If you do not know the name of your cluster run:
[root@rac1 trace]# cemutlo -n
rac
In the name of the cluster we will create the PDB:
[oracle@rac1 ~]$ $ORACLE_HOME/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB \
-pdbName rac -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb \
-PDBMetadataFile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true

Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file “/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/rac/_mgmtdb0.log” for further details.
The database creation is complete and we can see that it is already present in the cluster again:
[oracle@rac1 ~]$ srvctl status MGMTDB
Database is enabled
Instance -MGMTDB is running on node rac1
One more step is needed to properly configure the MGMTDB (this command does not return any rows but is part of the configuration):
[oracle@rac1 ~]$ mgmtca
Once this is done, just enable the ora.crf resource again and start it on each cluster node:
[root@rac1 trace]# crsctl modify res ora.crf -attr ENABLED=1 -init
[root@rac1 trace]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'rac1'
CRS-2676: Start of 'ora.crf' on 'rac1' succeeded

[root@rac2 ~]# crsctl modify res ora.crf -attr ENABLED=1 -init
[root@rac2 ~]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'rac2'
CRS-2676: Start of 'ora.crf' on 'rac2' succeeded
Note: ora.crf is the Cluster Health Monitor feature, is managed by the ohasd daemon, and is part of the Oracle Clusterware startup features.
Tagged:

About The Author

"The fragrance always remains in the hand that gives the rose." I am a person who believes in sharing knowledge among all. My Oracle rewards are OCI, OCP, OCA.

Comments

  1. Reply
    Kaushika Kasaragod

    Thanks for this wonderful Article.
    I have a question?, can we do this online in a Production system?
    What happens when the node goes down for maintenance, does the MGMT database automatically fails over to the existing node in the cluster?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.