Monday, March 3, 2008

Oracle application server- standard out and error

Recently I was troubleshooting an application under Oracle Application Server (OAS 10g) and I was wondering where the standard out and error were getting routed. I was looking for the script that starts the oc4j to see where it is getting redirected. However, oracle is doing things their own way and using a executable file to start the jvm if we are using oc4j as part of the opmn.
So I had to digup the oracle documentation and found where they are supposed to go and how to control them. This website has good documentation on the various setting. However if you like to setup quickly here is what I did and hope this will help.

1) Update your opmn config file so it prints the system out and system.err to a file.
To do this, edit the rocess-type definition related to the oc4j instance where the app is deployed and add the following line just below the java-options line under start-parameters.

If you are not comfortable editing the file manually, you can use the enterprose manager tool and add new properties (yourInstance->Administration->Server Properties)to the startup parameters. You will have to add one option per line.
Here is the sample process definition:


<process-type id="myappinstance" module-id="OC4J" status="enabled">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -Djava.security.policy=$ORACLE_HOME/j2ee/myappinstance/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Dstdstream.filesize=2.5"/>
<data id="oc4j-options" value="-out oc4j.out.txt -err oc4j.err.txt"/>
</category>
<category id="stop-parameters">
<data id="java-options" value="-Djava.security.policy=$ORACLE_HOME/j2ee/myappinstance/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false"/>
</category>
</module-data>
<start timeout="600" retry="2"/>
<stop timeout="120"/>
<restart timeout="720" retry="2"/>
<port id="default-web-site" range="12501-12600" protocol="ajp"/>
<port id="rmi" range="12401-12500"/>
<port id="rmis" range="12701-12800"/>
<port id="jms" range="12601-12700"/>
<process-set id="default_group" numprocs="1"/>
</process-type>


2) Restart the instance and now you should see the outputs from any System.out outputs in the new file.

The location of the files are little strange. You will find them under %ORACLE_HOME%\j2ee\home\%INSTANCE_NAME%_default_group_1.

1 comment:

Vishal Deore said...

Hi this is vishal I am also having problem for my application in OC4J cluster mode. When i run on stand alone mode it is working fine, when i keep it on the cluster environment in OC4J it responds slowly at initial first then after three to four days it is stoping respond at the third request then after one or two days it stops responding for the second as well as third request.
And finally it stops responding and OC4J stops or Hang or get crash after some time.
And no error is found in Application servers log file. I had got the error for the tile like this except nothing else.

[13:32:47,362 ERROR,InsertTag,AJPRequestHandler-HTTPThreadGroup-9] ServletException in '/views/menu.jsp': null
javax.servlet.ServletException
at com.evermind.server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:809)
at com.evermind.server.http.EvermindPageContext.handlePageException(EvermindPageContext.java:756)
at _views._menu._jspService(_menu.java:147)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:453)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:591)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:515)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_include(ServletRequestDispatcher.java:160)
at com.evermind.server.http.ServletRequestDispatcher.access$000(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$1.oc4jRun(ServletRequestDispatcher.java:97)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:102)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:396)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:349)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
at _common._layout._classicLayout._jspService(_classicLayout.java:84)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:453)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:591)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:515)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:259)
at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:302)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:259)
at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198)
at rbi.system.filter.LogoutFilter.sendRedirect(LogoutFilter.java:21)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:101)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:286)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.HttpRequestHandler.handleNotFound(HttpRequestHandler.java:1032)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:906)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:448)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:302)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:190)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)
[13:32:47,377 ERROR,LogoutFilter,AJPRequestHandler-HTTPThreadGroup-9] Error forwarding request to url: /default.do
09/04/08 13:34:12 The value of perform is--select.return
[13:34:40,905 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-8] Authentication event AuthenticationFailureBadCredentialsEvent: LUC00004VER; details: org.acegisecurity.ui.WebAuthenticationDetails@1de6: RemoteIpAddress: 172.16.8.12; SessionId: ac10138730eefd6fb4f4c6a040cf9b64d8c6915524c6; exception: Bad credentials
09/04/08 13:36:26 The value of perform is--loadCompanyTypePageUpload
09/04/08 13:36:28 The value of perform is--Search Company By Company Code
09/04/08 13:36:36 The value of perform is--Search By Code
09/04/08 13:36:38 The value of perform is--Select Return
09/04/08 13:36:52 The value of perform is--Next
09/04/08 13:36:58 The value of perform is--Next
09/04/08 13:37:20 The value of perform is--submitReturn
09/04/08 13:37:23 The value of perform is--Search By Code
09/04/08 13:37:25 The value of perform is--loadCompanyTypePageUpload
09/04/08 13:37:43 The value of perform is--select.return
[13:39:29,450 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-9] Authentication event AuthenticationSuccessEvent: BAN00007VER; details: org.acegisecurity.ui.WebAuthenticationDetails@ffffc434: RemoteIpAddress: 172.16.8.12; SessionId: ac10138930ee72792cd469dc442ab07fee6a8f17d7c2
[13:39:29,465 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-9] Authentication event InteractiveAuthenticationSuccessEvent: BAN00007VER; details: org.acegisecurity.ui.WebAuthenticationDetails@ffffc434: RemoteIpAddress: 172.16.8.12; SessionId: ac10138930ee72792cd469dc442ab07fee6a8f17d7c2
[13:40:16,736 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-7] Authentication event AuthenticationFailureDisabledEvent: CHE00099VER; details: org.acegisecurity.ui.WebAuthenticationDetails@ffff10d0: RemoteIpAddress: 172.16.8.13; SessionId: ac10138730ee804534c4dde04e0083c3951dc75d5820; exception: User is disabled
[13:41:16,633 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-10] Authentication event AuthenticationSuccessEvent: THI00213VER; details: org.acegisecurity.ui.WebAuthenticationDetails@0: RemoteIpAddress: 172.16.8.11; SessionId: ac10138830ee96b0363fca6741b2994eb060630cca6d
[13:41:16,633 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-10] Authentication event InteractiveAuthenticationSuccessEvent: THI00213VER; details: org.acegisecurity.ui.WebAuthenticationDetails@0: RemoteIpAddress: 172.16.8.11; SessionId: ac10138830ee96b0363fca6741b2994eb060630cca6d
[13:41:19,992 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-4] Authentication event AuthenticationFailureDisabledEvent: rbiusermum; details: org.acegisecurity.ui.WebAuthenticationDetails@0: RemoteIpAddress: 172.16.8.11; SessionId: ac10138830eedb0a44a2dd33427a9decd53abe765134; exception: User is disabled
09/04/08 13:41:25 java.lang.NullPointerException
[13:41:25,868 ERROR,InsertTag,AJPRequestHandler-HTTPThreadGroup-5] ServletException in '/views/menu.jsp': null
javax.servlet.ServletException
at com.evermind.server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:809)
at com.evermind.server.http.EvermindPageContext.handlePageException(EvermindPageContext.java:756)
at _views._menu._jspService(_menu.java:147)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:453)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:591)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:515)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_include(ServletRequestDispatcher.java:160)
at com.evermind.server.http.ServletRequestDispatcher.access$000(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$1.oc4jRun(ServletRequestDispatcher.java:97)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:102)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:396)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:349)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
at _common._layout._classicLayout._jspService(_classicLayout.java:84)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:453)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:591)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:515)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:259)
at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:302)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:259)
at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198)
at rbi.system.filter.LogoutFilter.sendRedirect(LogoutFilter.java:21)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:101)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:286)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.HttpRequestHandler.handleNotFound(HttpRequestHandler.java:1032)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:906)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:448)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:302)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:190)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)
[13:41:25,868 ERROR,LogoutFilter,AJPRequestHandler-HTTPThreadGroup-5] Error forwarding request to url: /default.do
[13:42:54,705 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-4] Authentication event AuthenticationSuccessEvent: MUM04929VER; details: org.acegisecurity.ui.WebAuthenticationDetails@ffffa64e: RemoteIpAddress: 172.16.8.12; SessionId: ac10138730ee3646ab2c10d047cb8204d4e639b3ea63
[13:42:54,705 WARN,LoggerListener,AJPRequestHandler-HTTPThreadGroup-4] Authentication event InteractiveAuthenticationSuccessEvent: MUM04929VER; details: org.acegisecurity.ui.WebAuthenticationDetails@ffffa64e: RemoteIpAddress: 172.16.8.12; SessionId: ac10138730ee3646ab2c10d047cb8204d4e639b3ea63
09/04/08 13:43:38 The value of perform is--select.return
09/04/08 13:44:27 The value of perform is--Select Period