Showing posts with label logging. Show all posts
Showing posts with label logging. Show all posts

Friday, September 24, 2010

Deleting old log files based on size and time

There are times when the log files take up too much space in the system and bring the system to run out of space.  Fortunately there are few steps you can take to keep the log files under control for development servers.  Here I have listed a simple strategy:
1) Create a dedicated partition for the log files so if the log files grow out of space, that still does not crash the system by making no space left for your data and configuration files.
2) Keep all log files for all applications under a common directory dedicated for logs.  Replace the location of the log files within your application with symbolic links to the common logs folder or change the log configuration files to directly point them to the log directory.
3) Create a simple script to delete files that are older than certain number of days.
4) For files that does not get rolled over by the application, have a script that trims the head of the file by certain percentage of their size when the file size grows over a certain size limit.

If you follow these simple steps, your log files and directories will stay tidy and you still have logs left when you need them to debug issues.   Here is a sample script that you can use to cleanup old files and trim larger files.
logdir="/tmp/test"
trimAmt=50
tempFile=/tmp/trimedfile.$$
sizeLimit=+1G
fileAge= +30

cd $logdir

find . -mtime $fileAge -exec rm -f {} \;

for FILENAME in $(find . -size $sizeLimit -print )
do
    filesize=$(wc -l $FILENAME |  awk '{print $1}')
    trimsize=$(( $filesize -($filesize / $trimAmt)))
    echo " $FILENAME $filesize $trimsize "
    tail -$trimsize $FILENAME > /tmp/trimedfile.$$
    cat $tempFile > $FILENAME
    rm  $tempFile
done


Thursday, October 8, 2009

Debugging Webservice clients - Log HTTP transport content

Many times it is useful to see the actual content that is send and received through the wire in order to troubleshoot webservice clients. Unfortunately, there does not seem to be a standard way to enable logging across all the webservice clients. Depending on the client you use for accessing the webservice, the debugging method varies. Here are a few tips that will help.

If you are using Axis client to access a webservice and using log4j for logging, you can enable printing of the http request and response contents by setting the "httpclient.wire.content" property to "debug". For example if using the log4j.properties file set the following property:

log4j.category.httpclient.wire.content=debug,stdout


If the webservice is connecting to a secure url using https protocol, setting the java system property "javax.net.debug" to "plaintext" will provide a hex dump of the content send and received.

If you are using the web service proxy provided by oc4j within jdeveloper or the oc4j container, you can enable the java logging property "oracle.webservices.client" in logging.properties file to FINE

oracle.webservices.client.level=FINE

Setting this option will allow a standalone client to log the request and response content to the log file specified by the logging.properties file.

If the client is running within the oc4j container, you can enable the oracle diagnostics logging for "oracle.webservices.client". This can be done by navigating to the <oc4j container> -> Administration --> Logger Configuration page and setting the log level as "FINE" for "oracle.webservices.client" property.
You can view the request and response contents in the log.xml file.

Saturday, August 16, 2008

Viewing JSP Stack Traces with OC4J 10.1.3.1.

When an error occurs in the translation or runtime execution of a JSP page, the container would return the exception, message and a stacktrace to the client. However, the exception is not displayed in the page by default in OC4j.

This can be a little annoying for development machines. There are several ways to change this behavior. The easiest way is to set the development flag at the container level. This can be done by updating the element of the global-web-application.xml file.

Here is an example:

<orion-web-app
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://xmlns.oracle.com/oracleas/schema/orion-web-10_0.xsd"
jsp-cache-directory="./persistence"
servlet-webdir="/servlet"
development="true"
jsp-timeout="0"
jsp-cache-tlds="standard"
schema-major-version="10"
schema-minor-version="0"
>


I like this option because, this can be set once for the container and dont have to worry about it.

If this cannot be done because you are working on a shared server, you can override individually by setting this flag on the application level by including orion-web.xml file in WEB-INF directory of your application. However, you have to make sure you remove this or change it when you deploy to production.

As a temporery measure, you can change it during deployment on the jdeveloper deployment wizard. The "Buttso Blathers" has a screen shot of this screen.

Tuesday, July 1, 2008

Struts/Tiles/oc4j logging

Recently I was troubleshooting an issue with a web application that uses struts/tiles and runs in an oc4j container. I application was getting a exception and oc4j will not print the exception stack trace in the output. It was hard to understand what is going on and it took me a while to figure out. So I am documenting here the process, so it can be a handy reference next time around or for some one who comes across the same issue.

To understand what was going on, the first step was to see what was the exception. To do this I set the development mode to true in the global_application.xml file. The global-web-application.xml file can be found under %ORACLE_HOME%/j2ee/%instance%/config/global-web-application.xml. Update the development attrubute on the orion-web-app tag to true and restart the instance.

<orion-web-app development="true" xsi="http://www.w3.org/2001/XMLSchema-instance" nonamespaceschemalocation="http://xmlns.oracle.com/oracleas/schema/orion-web-10_0.xsd" directory="./persistence" webdir="/servlet" timeout="0" tlds="false" version="0">


Now, if you visit the page on the browser, you will see the stack trace on the browser. This is a handy method, if your log files are cluttered. Alternatively you may view the stacktrace in the log files if you have configured oc4j to print standard out and error to a file. (for more information on how to setup refer to my earlier entry)

In my current situation, the stacktrace was not very useful as it did not point to any application code directly. The following is the stacktrace I received.


[ ERROR] [2008-07-01 09:23:13,500] [AJPRequestHandler-HTTPThreadGroup-5:] {{ServletException in '/StrutsAction2.do?QueryParm=parmvalue': null}} [org.apache.struts.taglib.tiles.InsertTag]
javax.servlet.ServletException: Error in servlet
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:757)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_include(ServletRequestDispatcher.java:160)
at com.evermind.server.http.ServletRequestDispatcher.access$000(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$1.oc4jRun(ServletRequestDispatcher.java:97)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:102)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:396)
at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:349)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:101)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:137)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:758)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:890)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:460)
at _layouts._template._jspService(_template.java:324)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:453)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:591)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:515)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:259)
at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:50)
at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193)
at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:283)
at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:261)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:237)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:300)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:711)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:368)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:866)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:448)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:302)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:190)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)


As you can see, all I can see is it getting a null exception when the jvm is trying to process the request for StrutsAction2. Unfortunately it was not printing the application class that was causing the exception. The reason for the issue is the exception is happening in a different thread than the one the stacktrace is printed.
Here is what was happening. I was actually calling url accociated with StrutsAction1. The tile definition for the StrutsAction1 includes StrutsAction2 in its response. The stacktrace oc4j was printing was from the thread that was processing StrutsAction2 which is a seperate thread. The only clue I could get from here is that the error is occoring during StrutsAction2. If I try to invoke StrutsAction2 directly it seems to be working fine.

This made things a bit more complex and I started adding more and more debugging statements to the strutsaction2. Then I realized there is a Custom tiles plugin we have defined for the app and it had a debug statement. On the outlook, the debug statement looked harmless to me. However, I was a bit surprised by the fact that I was not even seeing the debugline on the logfile even after turning debug on.

Here is the debug statement.


Logger.debug("Processing Request: " + request.getContextPath() + request.getServletPath() +" Req Detail: " + getLogString(request));

private String getLogString (HttpServletRequest req) {
StringBuffer buf = new StringBuffer("Request Parms:");
Enumeration parms = req.getParameterNames() ;
if (parms != null) {
while ( parms.hasMoreElements() ) {
String parm = parms.nextElement() ;
String pVal = req.getParameter(parm);
buf.append(parm + "=" + pVal + ";");
}
}
buf.append(": Req Headers: ");
Enumeration headers = req.getHeaderNames() ;
if (headers != null ) {
while ( headers.hasMoreElements() ) {
String parm = headers.nextElement() ;
String pVal = req.getHeader(parm);
buf.append(parm + "=" + pVal + ";");
}
}

Enumeration attribs = req.getAttributeNames() ;
if(attribs != null) {
while ( attribs.hasMoreElements() ) {
String parm = attribs.nextElement() ;
Object pVal = req.getAttribute(parm);
buf.append(parm + "=" + pVal.toString() + ";");
}
}
return buf.toString();
}


Not seeing the debug line in the log provided me the next clue. The issue might be somewhere in the debug statement. I came to the conclusion because this debug statement was the first thinging happening in the preprocess method for the plugin.

So my next step is to add a try catch around the debug line and printed the stacktrace in case of ecxeption. This pointed to the bug in the getLogString method. It was the line getting the attribute value "pVal.toString()" as the specific call flow, it was running in to an attribute that had a null value.

So I added a null check to the line as follows to get going. This was interesting beacause the common code that was suposed to be helping with debug was itself the one causing the bug. The leason here is be extra carefull with debug statements especially when adding to comon code.

buf.append(parm + "=" + (pVal==null?"null":pVal.toString()) + ";");"