I must thank my fellow DBA Franky Weber Faust for his publication in his blog.
In this article, we are going to know about how to rebuilding/moving MGMTDB on grid infrastructure 12c.
Stop the ora.crf resource on each node in your cluster and disable it so that it does not start automatically:
[root@rac1 ~]# crsctl stop resource ora.crf -init CRS-2673: Attempting to stop 'ora.crf' on 'rac1' CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded [root@rac1 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init [root@rac2 ~]# crsctl stop resource ora.crf -init CRS-2673: Attempting to stop 'ora.crf' on 'rac2' CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded [root@rac2 ~]# crsctl modify resource ora.crf -attr ENABLED=0 -init
If your MGMTDB is functional you will be able to remove it through the DBCA. If it has a problem that does not let it start you will not be able to remove it. In my case, the tablespace UNDOTBS1 was corrupted, so it was not possible to open the database and consequently could not delete it via DBCA:
[oracle@rac1 ~]$ /u01/app/126.96.36.199/grid/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB ORA-01219: database or pluggable database not open: queries allowed on fixed tables or views only
Browsing the MGMTDB alert.log I identified that the problem was the Undo tablespace:
Thu May 04 22:11:27 2017 Errors in file /u01/app/oracle/diag/rdbms/_mgmtdb/-MGMTDB/trace/-MGMTDB_ora_30436.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-00376: file 4 cannot be read at this time ORA-01110: data file 4: '+CONFIG/_MGMTDB/DATAFILE/undotbs1.259.941395093'
So I opted to manually remove the MGMTDB. Use asmcmd to find the datafiles:
ASMCMD> cd +CONFIG/_MGMTDB/ ASMCMD> ls -l Type Redund Striped Time Sys Name Y 4D39E5F3979F4E3AE0536538A8C0AD9E/ Y 4D3A16FC19245C2AE0536538A8C04036/ Y CONTROLFILE/ Y DATAFILE/ Y FD9AC0F7C36E4438E043B6A9E80A24D5/ Y FD9B43BF6A646F8CE043B6A9E80A2815/ Y ONLINELOG/ Y PARAMETERFILE/ Y TEMPFILE/
Then inside the MGMTDB directory remove all your files:
ASMCMD> rm -rf * ASMCMD> exit
[oracle@rac1 ~]$ vi /etc/oratab
You now have to remove the MGMTDB resource from the cluster. As we can see the database is still part of the cluster configuration:
[root@rac2 ~]# srvctl status mgmtdb Database is enabled Database is not running.
[oracle@rac1 ~]$ srvctl remove mgmtdb Remove the database _mgmtdb? (y/[n]) y
Thanks for this wonderful Article.
I have a question?, can we do this online in a Production system?
What happens when the node goes down for maintenance, does the MGMT database automatically fails over to the existing node in the cluster?
It works great. We don’t know why the mgmtdb was lost, but it’s running again!
Great article, help me a lot!