Tip of the week: Resolving TCM-00787: A repository error occurred.
Environment: Oracle WebCenter 220.127.116.11, Oracle Database 11gR2, RedHat Linux5, Windows 2008 R2
Issue: When accessing images from WebCenter Imaging, you receive oracle.imaging.ImagingException: TCM-00787: A repository error occurred.
Comprehensive analysis and resolution:
All services are up and running from Windows services console. Yet Universal Content Management service would show status as down in WLS Admin Console.
In order to generate the trace output, we started UCM via line command as opposed to Windows service.
Eg: f:\> startManagedWebLogic.cmd UCM_server1
The Content Server logs showed the database becoming unavailable, and Java unable to execute classes and methods.
We see a 324 MB Java.exe process attempt to start when we run the /startManagedWeblogic.cmd UCM_server1
In order to analyze the issue further by gaining the details in Java dump, we did a Java dump and a >> log.txt to collect everything. This was achieved with windows hit Ctrl + Brake in command window.
A Java dump was collected and it showed Java threads in a waiting state, waiting from a response from WLS over port 16200.
The requests over port 16200 were not being acknowledged, or replied to.
The port becoming unavailable can happen for a number of reasons:
1. The most likely is a firewall actively blocked the port 16200
*External testing on the firewall is advised. Review the firewall logs. Review the port restrictions to see if port 16200 was recently blocked.
*Can use the command ‘telnet <server name> <port>’ to test if the port is open. For example: telnet servername.com 16200
2. The socket became stuck or corrupts on the server where the port was not available.
*We rebooted the server and the port was still unavailable. If the socket is corrupt it would require an inspection of the network card for any hardware errors.
3. Another process started using port 16200, this would make port 16200 unavailable to the Content Server.
*We ran a ‘netstat’ command. Port 16200 did not appear on the list as being used by any other application.
The proposed Solution is to change the port from 16200 to another port that would be open.
We used the port 16202.
1. Log in to the WebLogic Admin Console. http://<hostname>:7100/console
2. Go to Servers > UCM_server1 > Lock and Edit
3. Update the ‘Listen Port’ from 16200 to 16202 > Save > Close the WLS Admin Console
4. Start the Content Server as an application: <Middleware>\user_projects\domains\base_domian\bin\startManagedWeblogic.cmd UCM_server1
Test the port 16200 to see if it is being blocked by a firewall. You can switch back to using port 16200 once the port is clear.
When trying to start the Content Server as an application it would hang after the notification that logging was being created.
We created a Java dump by clicking on ‘Ctrl + Break’ in the Windows command line.
The Java Dump shows all of the threads waiting.
“[ACTIVE] ExecuteThread: ’1′ for queue: ‘weblogic.kernel.Default (self-tuning)’”
id=22 idx=0x6c tid=3436 prio=5 alive, waiting, native_blocked, daemon
– Waiting for notification on: weblogic/work/ExecuteThread@0x000000000E6204 C0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
^– Lock released while waiting: weblogic/work/ExecuteThread@0x000000000E6204C0[fat lock]
Switch weblogic to use a port for the Content Server that is open.
A. Log in to the WebLogic Admin Console. http://<hostname>:7100/console
B. Go to Servers > UCM_server1 > Lock and Edit.
C. Update the ‘Listen Port’ from 16200 to 16202 > Save > Close the WLS Admin Console.
D. Start the Content Server as an application: <Middleware>\user_projects\domains\base_domian\bin\startManagedWeblogic.cmd UCM_server1
E. If the application starts where it says ‘Ready and Waiting on Port 16202′, close it as an application in the command line.
F. Start the Content Server as a service.
Revert back and test port 16200
A. Review the Firewall logs to see if port 16200 was blocked.
B. Use the command ‘telnet <server name> <port>’ to test if the port is open. For example: telnet servername.com 16200
C. Once the port is open change the UCM_server1 managed server in WebLogic to use port 16200 again.
NOTE: We were also notified by Oracle, that they will create a knowledge base article about UCM process not starting when the port is blocked.
Thanks to contributors Matt Drake and Dave Sagendoff.