How to troubleshoot the OMS - Error 404--Not Found

What to do when we encounter this error in our Oracle Enterprise Manager 12c! 

 Error 404--Not Found

From RFC 2068 Hypertext Transfer Protocol -- HTTP/1.1:

10.4.5 404 Not Found

The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent.

I have created some aliases on my Linux box so i can make my life easier.

 [oracle@DCG023 bin]$ alias
alias agent='cd /u00/agent12.0.3/core/12.1.0.3.0/bin'
alias l.='ls -d .* --color=tty'
alias ll='ls -l --color=tty'
alias ls='ls --color=tty'
alias oms='cd /u00/oms12.0.3/oms/bin'
alias omslog='tail -f /u00/oms12.0.3/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
[oracle@DCG023 bin]$
  • 1- Check for OMS status 
    •  [oracle@DCG023 ~]$ oms
[oracle@DCG023 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /u00/oms12.0.3/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
  • 2- Check if your database and listener are up and receiving connections
    •  [oracle@DCG023 ~]$ . oraenv
ORACLE_SID = [EMR] ? EMR
The Oracle base for ORACLE_HOME=/u00/product/11.2.0/dbhome_1 is /u00/oracle
[oracle@DCG023 ~]$ ps -ef | grep pmon | grep EMR
oracle    2168     1  0 Feb16 ?        00:06:58 ora_pmon_EMR

[oracle@DCG023 ~]$ ps -ef | grep lsn
oracle   12871     1  0 Jan26 ?        00:11:46 /u00/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit



[oracle@DCG023 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 6 13:16:28 2014
Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


Session altered.


NAME                           DB_UNIQUE_NAME                 DATABASE_ROLE                  OPEN_MODE
------------------------------ ------------------------------ ------------------------------ ------------------------------
EMR                            EMR                            PRIMARY                        READ WRITE


   INST_ID INSTANCE_NAME    STATUS       HOST_NAME                      STARTUP_TIME
---------- ---------------- ------------ ------------------------------ --------------------
         1 EMR              OPEN         DCG023                         16-FEB-2014 12:48:19

SYS@EMR>

    All are up and running.

  • 3- Restart you OMS
    •  [oracle@DCG023 bin]$ ./emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
[oracle@DCG023 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
Check EM Server log file for details: /u00/oms12.0.3/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

    Still the error persists.

  • 4 -Check the OMS log output now
    •  Internal Exception: weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException:
 0:weblogic.common.ResourceException: Could not create pool connection.
 The DBMS driver exception was: ORA-00257: archiver error. Connect internal only, until freed.

    Got it !!!

  • 5- Fix for ORA-00257: archiver error - in the Oracle Enterprise Manager Repository.

    • This error is related to archiving, so the first place we will look is into database parameters and check the db_recovery_file_dest parameters values.

     SYS@EMR> show parameter db_recovery_file_dest;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /u00/oracle/flash_recovery_area
db_recovery_file_dest_size           big integer 20G

    Next see space used by archives:

    SYS@EMR>  SELECT * FROM V$RECOVERY_FILE_DEST;

NAME                           SPACE_LIMIT SPACE_USED SPACE_RECLAIMABLE NUMBER_OF_FILES
------------------------------ ----------- ---------- ----------------- ---------------
/u00/oracle/flash_recovery_are  2.1475E+10 2.1473E+10                 0             680
a

    Ops.. none left.

    To free up space follow the steps bellow: 

     [oracle@DCG023 bin]$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Thu Mar 6 13:28:02 2014

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: EMR (DBID=3814598061)

RMAN> CROSSCHECK ARCHIVELOG ALL;

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=145 device type=DISK
validation failed for archived log
archived log file name=/u00/oracle/flash_recovery_area/EMR/archivelog/2014_02_22/o1_mf_1_13811_9jj4y44s_.arc RECID=13809 STAMP=840153876
validation failed for archived log
.....
.....
.....



MAN> DELETE EXPIRED ARCHIVELOG ALL;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=145 device type=DISK
List of Archived Log Copies for database with db_unique_name EMR
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
13809   1    13811   X 21-FEB-14
.....
.....
.....
Do you really want to delete the above objects (enter YES or NO)? YES
deleted archived log
archived log file name=/u00/oracle/flash_recovery_area/EMR/archivelog/2014_02_22/o1_mf_1_13811_9jj4y44s_.arc RECID=13809 STAMP=840153876
deleted archived log
  • 6- Check OMS status and start it.
    • [oracle@DCG023 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Down


[oracle@DCG023 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Already Started
Oracle Management Server is Up


[oracle@DCG023 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
[oracle@DCG023 bin]$
  • 7- Log in into your Oracle Enterprise Manager 12c

    • Copyright 2020. All Rights Reserved