After a server restart on Red Hat Enterprise Linux 5.0 with Oracle 11g installed, you're encountering issues where the Oracle Enterprise Manager Database Control (dbconsole) fails to start, while the agent continues to run normally. This is a common scenario that many DBAs face after system reboots.
First, let's verify the exact status and error messages:
$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.1.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Oracle Enterprise Manager 11g is not running.
------------------------------------------------------------------
The key indicators here are:
- dbconsole reports "not running"
- Agent shows "Running and Ready"
- Start attempt results in multiple dots (...) then "failed"
Here are the most likely causes and how to address them:
1. Repository Connectivity Issues
The dbconsole relies on repository database connectivity. Check if the database listener is running:
$ lsnrctl status
If needed, start the listener:
$ lsnrctl start
2. Incorrect Environment Variables
After reboot, environment variables might not be properly set. Verify your ORACLE_HOME, ORACLE_SID, and other critical variables:
$ echo $ORACLE_HOME
$ echo $ORACLE_SID
$ echo $PATH
If they're not set, source your Oracle environment:
$ source /home/oracle/.bash_profile
or
$ . /u01/app/oracle/product/11.1.0/db_1/bin/oracle_env.sh
3. Repository Schema Issues
The dbconsole repository might be locked or in an inconsistent state. Try dropping and recreating it:
$ emca -deconfig dbcontrol db -repos drop
$ emca -config dbcontrol db -repos create
4. Port Conflicts
Check if another process is using the dbconsole port (usually 1158):
$ netstat -tulpn | grep 1158
If the port is taken, either kill the conflicting process or reconfigure dbconsole to use a different port:
$ emca -reconfig ports -PORT 1158 -EM_PORT <new_port>
For deeper troubleshooting, examine the log files:
$ cd $ORACLE_HOME/hostname_sid/sysman/log
$ tail -f emdctl.trc
$ tail -f emagent.trc
Common errors found in logs include:
- ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
- ORA-01034: ORACLE not available
- ORA-27101: shared memory realm does not exist
To prevent this issue after future reboots, modify your dbora script to properly sequence the startup:
#!/bin/sh
# chkconfig: 345 99 10
# description: Oracle auto start-stop script.
case "$1" in
start)
# Start the Oracle database
su - oracle -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME"
# Start the listener
su - oracle -c "$ORACLE_HOME/bin/lsnrctl start"
# Start Enterprise Manager
su - oracle -c "$ORACLE_HOME/bin/emctl start dbconsole"
;;
stop)
# Stop Enterprise Manager
su - oracle -c "$ORACLE_HOME/bin/emctl stop dbconsole"
# Stop the listener
su - oracle -c "$ORACLE_HOME/bin/lsnrctl stop"
# Stop the Oracle database
su - oracle -c "$ORACLE_HOME/bin/dbshut $ORACLE_HOME"
;;
esac
After implementing the solution, verify everything is working:
$ emctl status dbconsole
$ emctl status agent
Access the console in your browser at:
https://<server_name>:1158/em
After a server restart on RHEL 5 with Oracle 11g, many DBAs encounter the frustrating situation where emctl start dbconsole
fails silently. The agent may show as running (emctl status agent
), but the DBConsole service refuses to start properly.
First, let's verify the exact error by checking the log files:
cd $ORACLE_HOME/hostname_sid/sysman/log
tail -100 emdb.nohup
Common findings include:
- Port conflicts (usually 1158)
- Repository connection failures
- Permission issues in SYSMAN schema
Here's the complete recovery procedure I've used successfully in production environments:
# First, clean up any existing processes
ps -ef | grep emagent | grep -v grep | awk '{print $2}' | xargs kill -9
# Verify ORACLE_HOME and ORACLE_SID are set correctly
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export ORACLE_SID=orcl
# Reconfigure the DBConsole
emca -config dbcontrol db -repos recreate
# During configuration, you'll need to provide:
# SYS password
# SYSMAN password (or choose to reset it)
# Notification email (optional)
# Port number (default 1158)
Repository Issues: If the SYSMAN repository gets corrupted:
sqlplus "/ as sysdba"
SQL> ALTER USER SYSMAN IDENTIFIED BY new_password ACCOUNT UNLOCK;
SQL> EXEC SYSMAN.EMD_NOTIFICATION.QUEUE_CLEANUP;
Port Conflicts: Check and free up port 1158 if needed:
netstat -tulnp | grep 1158
# If occupied, either kill the process or reconfigure DBConsole to use a different port
emca -config dbcontrol db -PORT 1159
To prevent this issue after future reboots, modify your /etc/init.d/dbora
script to include:
#!/bin/bash
# chkconfig: 345 99 10
# description: Oracle auto start-stop script.
case "$1" in
start)
echo -n "Starting Oracle DBConsole: "
su - oracle -c "$ORACLE_HOME/bin/emctl start dbconsole"
touch /var/lock/subsys/dbora
;;
stop)
echo -n "Shutting down Oracle DBConsole: "
su - oracle -c "$ORACLE_HOME/bin/emctl stop dbconsole"
rm -f /var/lock/subsys/dbora
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
Then register it with:
chmod 750 /etc/init.d/dbora
chkconfig --add dbora
chkconfig dbora on
After implementing these steps, verify everything works:
emctl status dbconsole
# Should show:
# Oracle Enterprise Manager 11g is running.
# Logs are being generated at: /u01/app/oracle/product/11.2.0/dbhome_1/hostname_orcl/sysman/log
Access the console at https://yourserver:1158/em
(replace port if you changed it).