Using Oracle ESB as a web service client to pull data periodically

In this article we will cover the detailed steps to create an ESB service that invokes a web service periodically and write the response to a file.  The web service in this case takes an input parameter and returns some response based on the input.  The client is expected to parse the response and take an element from the response and supply that as the input for the next call.

1.    Create a service group. Lets call it EsbWebServiceClient
2.    Create a xsd file to represent the configuration information that we will use to pass to the webservice.
3.    Create a xml file that complies with the schema and the initial parameters for invoking the web service. The ESB adaptor will update this file after each invocation.
4.    Create a file adaptor to read a file.  Don’t check the mark that says delete file after reading.  Specify the schema that you created in step 2 for the schema of the file. When you are done with step you should end up with boxes 1 & 2 shown in the diagram with the line connecting them.
5.    Create a ESB Service  for Soap service (Right click on design pane-> Create ESB Service->Soap Service)  On the pop up window, select the WSDL file representing the service.  For this demo, the web service is a simple service that takes a string as input and returns a string. This creates Box 3.with no lines connected.
6.    Now create a schema that represents the input elements to the webservice and the output of the webservice.  We will use this as the holding area to have both the request and response so we can write the output to a file and update the config file with the updated information.  In the example the schema is called serviceParameterAndResponse.
7.    Create a ESB routing Service and name it ConfigMapper.(Right click on design pane-> Create ESB Service->Routing Service)  On the pop up window, select the serviceParameterAndResponse schema for both request and response formats.. This creates Box 4.with no lines connected. This routing service provides the framework for us to merge the input parameters and the webservice output.
8.    Now create a file adaptor that can write an output file and name it WsResponseWriter. To keep it simple, specify the schema for the file as the same as the response from the webservice  This should create Box 5 with no lines connected.
9.    Now create a file adaptor that can write the config file and name that ConfigFileWriter.. Specify the schema and the location for the file as the same as the schema created in step 2. This should create Box 6 with no connector lines
10.    Double click box 2 and add a routing rule to invoke the method on the webservice.  This should create a reply line.  Now point the reply to the execute method on the routing service created on step 7.  This should create lines 8 & 9 on the esb diagram..
11.    Double click box 4 and add two routing rules.  One pointing to WsResponseWriter and the second one pointing to ConfigFileWriter.  This should create the lines 10 & 11 in the diagram.
12.    Now the diagram is almost complete.  You will see the mappings are still gray. We will add the mappings which will make them yellow.
13.    Double click on the top "X" and create a new mapping.  When the designed is displayed, drag a line from the config parameter to the input parameter of the web service.
14.    Double click of the bottom "X" of box 2.  Create a new mapping.  You will notice there is an option to "include the request in reply play load".  Make sure you select this option.  When the designer panel is displayed you should see a new node called “ESBREQUEST” on the left hand side.  To learn more about the $ESBREQUEST parameter refer to this article (http://www.soastation.org/2007/05/esbrequest-in-oracle-esb-routing.html)
15.    Now select the source view of the xslt and add a function to parse the required fields from the request to map to the output. In the example below, I have mapped the string after the pipe symbol to be stored as the parameter for the next call. And for the response, I have concatenated the input and the output to be written out to the file.  Please note that you will not be able to drat lines to or from the $ESBREQUEST variable on the designer view.

<xsl:param name="ESBREQUEST"/>
 <xsl:template match="/">
   <out1:serviceParameterAndResponse>
     <out1:param1>
       <xsl:value-of select='substring-after(/tns:getElapsedTimeSinceResponse/tns:return,"|")'/>
     </out1:param1>
     <out1:response>
       <xsl:value-of select='concat(substring-after(/tns:getElapsedTimeSinceResponse/tns:return,"|"),substring($ESBREQUEST/tns:lastTime,1.0,2.0))'/>
     </out1:response>
   </out1:serviceParameterAndResponse>
 </xsl:template>

 16.    Now double click on the "X" on box 4 and create new mappings.   These mappings should be straightforward as all the manipulation is done on the previous step.   Just draw a straight line from the corresponding field on the left to right.  Repeat this for both the ‘X” on the box.
17.    Now your diagram should look similar to the one shown above.  You can register the service to your ESB and test it.

If you like see a copy of the project that I used to create the demo, feel free to drop me an email.  My email address is jmeslie at gmail dot com.

Asynchronous Web Service Calls over HTTP using Oracle ESB and JMS

How to create an asynchronous webservice that supports failover using Oracle ESB?

This article describes a general design for invoking web services in an asynchronous fashion even if the web service provider only supports synchronous invocation. I have used JDeveloper to create this service however the same principles can be used for any development tool.

Use case:

Generally there are 3 different scenarios of using a web services.

1. Request/Response Real Time: .Here the response from the web service is critical for the application to proceed to the next step. If the web service is not available, user needs to know right away. Example: An application depends on a web service to display a list of accounts they can manage. In this case invoking the web service in an asynchronous fashion is not going to help much. A simple synchronous web service call will do the work without adding any complexity.
2. Request only – (delivery critical): Here, the client application needs to send a message to a web service, but it is not expecting any response from the service. The client wants to make sure that the message will be delivered to the service even if the service was down at the time of invocation.
3. Request/Response time (not critical, delivery critical): Here, the client application needs to send a message and receive response from the web service. The client wants to make sure that the message will be delivered to the service even if the service was down at the time of invocation. The client is not waiting for the response from the service. Instead, the client has provided a callback mechanism to receive the response.

The design described here suites for the third scenario and the second scenario. For the second scenario, only part of the design is needed.

The approach:

The idea here is to use JMS queues to receive the request and response messages to and from the web service. Clients will access the web service through a JMS-WebService adaptor. JMS listeners will invoke the web service and the callback service when messages are received in the request or response queues.

The following diagram describes the approach.

The web service is created and deployed in a remote server. In many cases this might be an external web service provided by a third party hosted outside your firewall. The client provides its callback mechanism as a web service and it is deployed locally. JMS queues are created in the local server for receiving the request and response messages. A JMS adaptor is created that exposes the request queue as a web service. Using JMS adaptor provides the following benefits:

• The client can communicate with the service using SOAP/HTTP(S) which is well supported by most tools than the SOAP/JMS.
• It hides the dirty details of the JMS details from the web service client developer.
• The web service client developer is using the same protocol to access the service if he has to call the service directly. (except for the return value)

A JMS listener is created to monitor the request queue and invoke the actual web service using a SOAP adaptor. The response from the web service will be written to the response queue. The listener and the adaptor can be configured to retry the service invocation if the service is down.

Another JMS listener is created to monitor the response queue and invoke the callback web service using a SOAP adaptor. The listener and the adaptor can be configured to retry the service invocation if the callback service is down when the actual web service responded.

Benefits:

This approach allows you to develop an asynchronous callback mechanism to your web service, even if the original service did not provide one. Most web services that are request/response provides only synchronous calling mechanism only.

It provides a way to decouple the dependency on the external web service without adding any custom client code. These days most application server vendors provide JMS and SOAP adaptor widgets that let you create these without having to write them yourself.

If you are using a web service that does not return and response back (one way request only), this approach can still work just be not creating the response queue and the related components (step 5 and beyond in the above diagram).

Limitation:

This approach assumes that you have enterprise grade tools such as JMS/SOAP Adaptors in your production environment. If such tools are not available, you may have to create these adaptors yourself. If you are developing them yourself, the time spent creating these generic services might be higher than adding failover logic to the application itself.

Creating the service

The instructions assume you are using Oracle application server 10.1.3.x and jDeveloper 10. The exact steps will be different for other application server and IDEs. Please note that these instructions assume that you are familiar with the above tools and does not provide step by step details of each widget.

1. Create and deploy the ‘actual WebService. If the service is provided to you by a third party you can skip this step.
2. Create and deploy the web service response receiver web service to your local server. If you are creating only a one way web service call you can skip this step.
3. Create JMS queues for request and response queues. This can be done through the Enterprise Manager application by navigating to à Administration à Enterprise Messaging Service à Create/delete/edit JMS destinations. Since we are going through this exercise to create a failover mechanism, I would suggest create these queues with File/Database persistence.
4. Create separate JMS Connection factories for each of these queues so we don’t get in to any resource contentions.
5. Create a new jDeveloper ESB project
6. Copy the WSDL for step 1 and 2 in to the project so we can reuse the schema elements.
7. Create JMS adaptor for writing to the request queue (produce message). Check the box that says “Invocable from external service”.

If you selected Oracle Enterprise Messaging Service in the adaptor configuration widget, select Produce Message as the operation type and specify a meaningful name (similar to the original service’s method name) for the operation. The WSDL exposed by this adaptor will be later used by the web service client. Select the request queue and the appropriate connection factory. Specify a large time out so if the external service is down, the message will not expire before the ESB exhausts all retry attempts. On the schema selection screen, select the message element that corresponds to the input message to the external web service from the WSDL.

8. Create an ESB Soap service (Create ESB Service à SOAP Service) and specify the WSDL for the actual web service.
9. Create JMS adaptor for reading from the request queue (consume message). If you selected Oracle Enterprise Messaging Service in the adaptor configuration widget, select Consume Message as the operation type. Select the request queue and the appropriate connection factory. On the schema selection screen, select the message element that corresponds to the input message to the external web service from the WSDL (schema and queue names same as previous step). JDeveloper should have created the adaptor service icon and the routing service.
10. Edit the routing service by double clicking and add a routing rule. For the consume message operation. Wire it to point to the appropriate operation of the SOAP adaptor from step 8. Create a transformation map to map the elements from the request to response. Also, you may want to make this execution asynchronous.
11. Create JMS adaptor for writing to the response queue (produce message).

If you selected Oracle Enterprise Messaging Service in the adaptor configuration widget, select Produce Message as the operation type. Select the response queue and the appropriate connection factory. Specify a large time out so if the callback service is down, the message will not expire before the ESB exhausts all retry attempts. On the schema selection screen, select the message element that corresponds to the input message to the callback web service from the WSDL (or the output message from the external web service).

12. Create an ESB Soap service (Create ESB Service à SOAP Service) and specify the WSDL for the callback web service.
13. Create JMS adaptor for reading from the response queue (consume message). If you selected Oracle Enterprise Messaging Service in the adaptor configuration widget, select Consume Message as the operation type. Select the response queue and the appropriate connection factory. On the schema selection screen, select the message element that corresponds to the input message to the callback web service from the WSDL. JDeveloper should have created the adaptor service icon and the routing service.
14. Edit the routing service by double clicking and add a routing rule. For the consume message operation. Wire it to point to the appropriate operation of the SOAP adaptor from step 12. Create a transformation map to map the elements from the request to response. Also, you may want to make this execution asynchronous.
15. Edit the routing service created in step 10, by double clicking and expand the routing rule for the consume message operation. Wire the response message from the service to point to the produce message operation of the JMS adaptor for the response queue. Create a transformation map to map the elements from the request to response. Also, you may want to make this execution asynchronous. The following diagram represents the ESB diagram for a project with these settings.
16. 
17. Deploy/Register the ESB project to the OC4J container.
18. Create a web service proxy for the WSDL exposed in step 7. Invoke the service using the generated proxy. If everything went well you should notice the response came back through your callback service.

Oracle ESB deploy script

If you need to deploy your ESB projects to multiple environments without jDeveloper, there is not a elegant solution. The ANT deployment tasks that comes part of the 10.3.1.3 packages helps a good deal. You can refer to the documentation on that at
http://www.oracle.com/technology/products/integration/esb/files/esbdeploymentautomation.zip or it is included in your ORACLE_HOME/integration/deployment/documentation.zip file.

Using the deployment script described there, you can extract the deployment plan from the esb project files. Once you have extracted the deployment plan file, all your settings will be in one single file which you can edit/make multiple copies with different settings for different environments. Even though this approach is far better than having to maintain several versions of several esb file, it still falls short of my expectation.

For most of the ESB projects dealing with various adapters and such, the deployment plan file has too much information it it so every time the ESB project changes (adding/removing adaptors or adaptor settings), you will have to regenerate the deployment plan file and make the copies for your different information. However, in most cases the only information that changes per environment are the endpoint properties that specify file directories, server specific properties etc. From a development and deployment perspective, it would be much nicer, if oracle had provided a way to specify the endpoint properties in a single location on the server, so every time a new version of ESB project is deployed, the endpoint properties are not impacted. In the absence of such feature, the closest that I could do minimize the impact of maintaining the endpoint properties is to generate the deployment plan file using the ant task and edit it using another custom task that takes the endpoint properties from a properties file that just has the properties you want to be different for each environment.

The source code and the sample ant task for this can be found here.

Using this approach, you create a property file that contains the endpoint property as the key and the value you want as a simple java style property file. You will need to create one such file per environment. The java program takes the template file, the properties file and the new deployment file name as arguments and creates the deployment file that can be used by the ant task that comes with oracle. Using this procedure, a typical build will have the following steps.
1) Extract deployment plan file using the oracle provided task.
2) Invole the ESBDeploymentPlanEditor class to create a server/environment specific deployment plan using the custom task
3) Deploy the ESB project using the deployment plan created on the above step using the oracle provided task.

ESB ftp adaptor missing class oracle.tip.adapter.file.outbound.FileIoInteractionSpec

If you happen to get an error in the ftp adaptor that it is unable to find the class oracle.tip.adapter.file.outbound.FileIoInteractionSpec, the most likely reason is that there is a class loading issue. To validate the issue, you can use the MBean for class loading and run the following command.

FindResource(oracle.tip.adapter.file.outbound.FileIoInteractionSpec)

If the above command indicate that the class is not found, you may run the following command to verify it it is atleast in your system.

Depends(oracle.tip.adapter.file.outbound.FileIoInteractionSpec)

If this command shows that the class in in the FTPAdaptor jar file, you may simply delete the /j2ee//connectors/FileAdapter/FileAdapter directory and restart the container. This will reimport the files in the fileadaptor directory with the right files.

ESB rejected Message Handlers doesnot work

If you ever wondered to change the default place where the ESB drops the files it rejected, you might have came across the endproperty "rejectedMessageHandlers". However JDeveloper 10.1.3.3 does not allow to add this property. You have to close the jDeveloper, edit the file manually and re-register the service. I also noticed that the property seems to be disappearing some times. (so keep a backup handy all the time)

There is also another catch to the way you specify the value of the property. The developer guide example specifies a sample value as below.

<property name="rejectedMessageHandlers" value="file://c:\rejectSample\reject"/>

However, it will not work. As pointed out in the OTN thread
you need to add a additional slash after the file: as shown below.

<property name="rejectedMessageHandlers" value="file:///c:\rejectSample\reject"/>

I agree this is pretty dump. However it if you are reading this you might be already used to such dump things from the SOA suite.


P.S. In case you are wondering the location of the default folder for the rejected messages, it is /j2ee/home/jca/service.name.routingservice.name/rejectedMessages.
I have not yet found a way to keep the name of the original input file.
rejectedMessageHandlers

How to: Get filename from ESB adaptor

Oracle ESB provides header functions that allows designers to access inbound and outbound header functions. The following are the functions available.

1. String getRequestHeader(String xpathExpression,String namespaceDecl)
2. void setOutboundHeader(String xpathExpression,String value, String namespaceDecl)
3. String getInboundResponseHeader(String xpathExpression,String namespaceDecl)
4. void setResponseHeader(String xpathExpression,String value, String namespaceDecl)
Where,
xpathExpression : XPath expression to get/set
value : value to be set for the xpathExpression
namespaceDecl : namespace declarations in the form ‘prefix=namespace;’


You can use the following code snippet in your transformation xslt to get the name of the file that is read by the file adaptor in ESB.

<xsl:variable name="GET_INFILENAME" select="ehdr:getRequestHeader('/fhdr:InboundFileHeaderType/fhdr:fileName',
'fhdr=http://xmlns.oracle.com/pcbpel/adapter/file/;')"/>

Where to find Oracle ESB documentation

Here is a list of ESB documentation files I found useful. You can check them out at http://www.technogemsinc.com/clientFile/esb.htm

For details about configuring the AQ, database, file, FTP, JMS, and MQ adapters in Oracle JDeveloper, see Oracle Application Server Adapter for Files, FTP, Databases, and Enterprise Messaging User's Guide. For details about configuring the Oracle application adapter for Oracle E-Business Suite, see Oracle Application Server Adapter for Oracle Applications User's Guide.

How to set the name of the file dynamically during transformation time in ESB fileAdaptor

Oracle ESB provides header functions that allows designers to access inbound and outbound header functions. The following are the functions available.

1. String getRequestHeader(String xpathExpression,String namespaceDecl)
2. void setOutboundHeader(String xpathExpression,String value, String namespaceDecl)
3. String getInboundResponseHeader(String xpathExpression,String namespaceDecl)
4. void setResponseHeader(String xpathExpression,String value, String namespaceDecl)
Where,
xpathExpression : XPath expression to get/set
value : value to be set for the xpathExpression
namespaceDecl : namespace declarations in the form ‘prefix=namespace;’


You can use the following code snippet in your transformation xslt to set the name of the file that is about to written by the ESB File adaptor. This overrides the name given in the design time.

<xsl:variable name="SET_OUTFILENAME"
select="ehdr:setOutboundHeader('/fhdr:OutboundFileHeaderType/fhdr :fileName', $GET_INFILENAME, 'fhdr=http://xmlns.oracle.com/pcbpel/adapter/file/;')"/>

The variable GET_INFILE is the name of the input file name as referred in the previous post.
If you like an example project, you can download it from oracle site here The sample project is under HeaderSupportDemo folder.

How to get the name of an ESB instance in transformation

A new XPath function was added in Oracle 10.1.3.3 to get instance IDs. If you are running an older version you have to upgrade to get this working. One note of caution is, after making the edits to your xslt, the designer view will not work. This is a limitation of jDeveloper.
Here is an example on how to get the instance id in the transformation.

1. Define a simple ESB project and define a simple File Adapter.
2. Perform a transformation in the routing service.
3 Perform the following steps to get the instance ID:

a. Go to Design View.
b. Select Advanced Functions in the Component Palette.
c. Select xpath-expression and drag it to the middle column.
d. Double-click the function icon.
e. Enter ehdr:getInstanceID() in the XPath Expression field.
f. Click OK

4. Click Yes when prompted with a warning.
5. Map the function to the InstanceID element and save the project.
6. Return to the routing service where the XSL transformation activity is defined.
7. Double-click the routing service.
8. Expand the Properties section and add the following property:
Map the function to the InstanceID element and save the project.
6. Return to the routing service where the XSL transformation activity is defined.
7. Double-click the routing service.
8. Expand the Properties section and add the following property:(Note this property is case sensitive.)

enableAccessBusinessEvent=true

9. Save and deploy the project.