Monday, December 14, 2009

Handling JSON arrays and objects

Recently I was using the Apache Wink client-side capabilities to interface with the RESTful APIs of several popular social media sites like Twitter and bit.ly, and I ran into an interesting problem regarding handling the JSON data being returned from those services. Since I think quite a few people may hit this same issue, I thought it may be helpful to share one way of dealing with the problem.

What is the problem? Well, it turns out that in dealing with these services they were sending back JSON in two different forms. One of those forms was the object form of JSON, which looks like the following:

{"name":"Dustin", "userid":"WebSphereClouds"}

The other form that was getting sent back was the array form of JSON, which starts with a [ instead of a {:

[{"name":"Dustin", "userid":"WebSphereClouds"}]

That may not seem like a big deal, but it does in fact affect the way the data is parsed and the Java object type to which the data is converted.

For instance, consider my application which uses the IBM JSON4J library. If I were trying to convert a response input stream containing data in JSON object form, I would call JSONObject.parse(responseInputStream) and that would return a JSONObject. On the other hand, if I were converting a response input stream containing data in JSON array form I would call JSONArray.parse(responseInputStream) and that would return a JSONArray. Clearly, my application needed a way to determine what form of JSON data it was handling.

To do this, I created a simple class that encapsulates this data type determination. It contains two methods to parse a response input stream and return a JSONArtifact type which is an interface implemented by both JSONObject and JSONArray.

private enum JSONType {
ARRAY_TYPE,
REGULAR_TYPE
}
public JSONArtifact parse(InputStream is) throws IOException {
JSONArtifact parsedJSON = null;
FilterInputStream fis = new BufferedInputStream(is);
JSONType jsonType = getJSONType(fis);
if(jsonType == JSONType.ARRAY_TYPE) {
parsedJSON = JSONArray.parse(fis);
}
else {
parsedJSON = JSONObject.parse(fis);
}
return parsedJSON;
}
JSONType getJSONType(FilterInputStream fis) throws IOException {
JSONType type = null;
fis.mark(1);
byte[] bytes = new byte[1];
fis.read(bytes);
fis.reset();
String firstChar = new String(bytes);
if("[".equals(firstChar)) {
type = JSONType.ARRAY_TYPE;
}
else {
type = JSONType.REGULAR_TYPE;
}
return type;
}

The class simply reads the first byte of data from the stream, resets the stream, and then determines, based on whether the first byte is { or [, whether to parse using the JSONArray or JSONObject class.

I also want to point out what I would consider to be a best practice with respect to JSON data sent back from a RESTful service. When developing RESTful services on the server-side, I suggest sending back the object form of JSON. If necessary, you can encapsulate arrays in this object form:

{"userInfo":[{"name":"Dustin", "userid":"WebSphereClouds"}]}

This will make for simpler parsing by clients of your services, and you can easily add data to what you return by adding more entries in the JSON without regressing existing clients.

Friday, December 11, 2009

Useful JAX-RS links for testing and handling error conditions

There's some interesting articles that Dave Artus has been writing as he becomes more proficient with Apache Wink.  He has a great (in-depth) detailed article about using JMock to test JAX-RS services that he's developing with Apache Wink as well as a good article about handling error conditions.  Check it out...

Tuesday, December 8, 2009

Paper to provide an overview of SAML and support in 7.0.0.7

I hope you found the SAML demo video from Greg useful. My team has put together a paper to provide an overview of the SAML technology introduced in WebSphere 7.0.0.7.

The paper is available for download here. This paper provides:

  1. Introduction to SAML standards (a very rich set of standards and specification)
  2. Comparison of SAML to LTPA and Kerberos technology and what problem SAML is solving
  3. Different trust models based on Bearer, Holder-Of-Key, and Sender-Vouches Confirmation Methods
  4. Self-Issuance of SAML token and external third party STS (Security Token Service) issuance
  5. SAML token propagation in Web Services including different considerations for handling token timeouts
  6. Client side SAML token caching
  7. List of supported features for SAML introduced in WebSphere 7.0.0.7
  8. Introduction to the default Policy Sets for enabling SAML Token Profile support

Here is the abstract of the paper:

Abstract
Security Assertion Markup Language (SAML) is an XML based framework developed by the Organization for the Advancement of Structured Information Standards (OASIS), used for the exchange of security information between different parties. This is an open standard that can be used to exchange security information between different products. SAML support in IBM® WebSphere® Application Server Version 7.0 Fix Pack 7 delivers a SAML solution that provides support for the most common scenarios SAML using Web services.

I hope you find the paper useful and please send us any feedback.

Wednesday, November 25, 2009

WebSphere Application Server 7.0.0.7 and JAXB performance improvements

As Andrew Spyker posted here, the 7.0.0.7 improvements to JAXB will help both JAX-RS and JAX-WS services when enabled.  These were enabled via tighter integration with our XML parser shipped in WebSphere Application Server.

The larger your payloads, the more benefit the user should see.  Let us know how it works out for you?

Friday, November 20, 2009

Which Identity is used for Authorization

In one of my recent discussions, I came across a question on which identity is used for authorization checks in WebSphere for SOAP over HTTP web services.   For the background of web services authorization in a JEE server, please read our previous blog entry (Web Services and Authorization).

Is it the transport level identity or instead the message level identity (assuming the identity is propagated using WS-Security)? The answer is it depends.

  1. If the web service is implemented as a Java Bean and hosted in a Web container, then the security model is based on the JEE servlet/JSP security model. The JEE role based authorization check is URL based.

    Underneath the covers, the Web container intercepts the HTTP request, authenticates the identity from the HTTP transport and performs the authorization check. The message level security is not processed yet, as the Web container does not understand or process the SOAP message.

    The transport level identity is used in this case for roles defined in the JEE deployment descriptors.

    But if you are using the programming API (e.g. WebServiceContext.isUserInRole) to performs authorization checks in JAX-RPC/ JAX-WS application handlers or web services provider implementation, then it could be either the transport level identity or message security identity. But the message level identity always overrides the transport level identity in security context since message level security is processed after transport level security.

  2. If the web service is implemented as stateless EJB, then the identity could be either the transport level identity or message security identity.   The message level identity always overrides the transport level identity in the security context since message level security is processed after transport level security.  This applies to both roles defined in the JEE deployment descriptors and programmatic APIs authorization check.

    The reason is that the web service EJB is leveraging the Web container to handle the HTTP protocol. In this case, the authorization is done by the Web container (Router Servlet) and the EJB container. The identity in message level security in the SOAP message is processed and set in the security context before calling the EJB container.  Therefore, either HTTP transport level identity or message level security identity could be used.

    In this scenario, you can use JEE role based method level authorization to protect the web services.


I hope this helps to clarify which identity is used for web service authorization check.

Wednesday, November 18, 2009

Web Services and Authorization

I've gotten involved in some discussions with customers asking questions about web services and authorization. In an attempt to clear up some of the ambiguities, I decided to start with going back to the basics and stating to customers what is the intended capabilities of the technology so that they can see for themselves a bit of what is the magic. The initial problem that some customers have is understanding how authorization works for Java EE-based web services.

Some of the facts are:

  • There is a security model for both Web Resources and EJB resources. Both support role based authorization to control access to resources
  • Web Resources are URIs and the methods on Web Resources are GET, PUT, POST, and DELETE
  • EJBs are objects/services and the methods/operations on EJB resources are more RPC-level methods
  • You can secure/authorize both types of resources appropriately...

Now, that said, let's add in "SOAP-based Web Services". To a great extent, this starts to blend the 2 models... (in the usage of HTTP as your transport)...
  • JAX-RPC and JAX-WS represent programming models for SOAP-based Web Services.
  • JAX-RPC defined a mapping model, as well as an unmanaged "client" programming model. There is no standard "deployment" model for exposing Web Services in JAX-RPC.
  • JSR 109 adds to that a "management" model, where you can now define "declaratively" Web Services to be exposed. In addition, you can now manage service references (clients) to those resources.
  • WebSphere uses JSR 109 metadata (i.e. service name or service reference name) in order to attach qualities of service to those resources
  • JAX-WS get's a *little* better in this space (since annotations can be used to denote service references - and clients) - which is why we are able to use a notion like "policy sets" to attach resources to the "attachment-points" defined by annotated classes...

Lastly, this brings us to the matter at hand - authorization....
  • There is one small fly in the ointment mapping SOAP-based HTTP-centric Web Services, and that is all services are issued over a POST.
  • For services hosted as EJBs, "users" can be identified and secured using standard HTTP-transport mechanisms (or Message-level mechanisms), and the authorization of methods can be handled via the normal authorization checks for method/operation level security in EJBs.
  • For services hosted as POJOs in a Web Container, since it's all a POST, *all* you can do is validate (if anything at all) that the user is a valid user (with the standard J2EE level mechanisms for security).
  • You *could* invent your own authorization logic to check (at the application-level) what role the caller is in, etc... (under the various operations) - but, that's essentially doing the same as what is built-for-you with using EJBs.
  • Lastly, there is nothing preventing you from doing this by delegating the request to an EJB container. The additional cost of the container breach is roughly what you are paying for (performance-wise).

Hopefully, this will help clear up whatever confusion users have with how authorization and web services works.

Monday, November 16, 2009

Demonstration of the SAML Token support in v7.0.0.7

I spent some time over the weekend recording a demo of a scenario which shows how to use our SAML support that shipped as part of v7.0.0.7 and was introduced on this blog last week.  The demonstration walks through a simple application, attaches some SAML-specific policy sets, and walks through a little of the configuration of the Policy Sets themselves.   A key takeaway is showing that nothing changes from an application programming model perspective.  All of the work is done by an administrator (and can be easily reused by other applications.  The demo is <10 minutes long, and is located here.

Hopefully, this will visually help others conceptualize how they will use this in their own environment.

Friday, November 13, 2009

Apache Wink 1.0 in WebSphere Application Server

I'm glad to see that Apache Wink has reached it's 1.0 status and can be officially downloaded.   As I've seen various people deploy Apache Wink to WebSphere, I've had to point out a few things that are (and are not needed) when deploying the Wink libraries to WebSphere.  I've made a short video which showcases some of these items.

However, for brevity sake, some of the basic items are listed as follows:
  • WebSphere Application Server v7.0 (and the Feature Pack for Web Services on v6.1) both include JAXB, StAX, and the Java Activation Framework libraries (so the support can be obtained from WebSphere)
  • slf4j-jdk14 should be used instead of slf4j-simple.  This will enable better integration of the emitted informational/trace/debug messages.
  • File Serving needs to be disabled in the ibm-web-ext.xml file (if a user wants to use a URL mapping of "/*"
  • Shared libraries currently work either attached to the Web Application or at the Application level with a classloader setting of Application.  A known JIRA is open to resolve a classloading issue for classloaders set to Module and attached at the Application level.
I hope this helps some folks get started using Apache Wink on WebSphere...

SAML for Web Services is now available with WebSphere 7.0.0.7

You probably have read the announcement letter on the XML Feature Pack, SCA Refresh, and SAML (Security Assertion Markup Language) for Web Services support coming to WebSphere 7.0.

I would like to give some additional details of what is in the SAML support.
  1. Supports scenarios targeting OASIS Web Services Security SAML Token Profile 1.1
    1. Supports SAML Token Assertion specifications v1.1 and v2.0
    2. Supports Bearer confirmation and Holder-of-key confirmation
  2. Configurable via policy sets
    1. Targets JAX-WS services
    2. Leverages Custom Token Support
  3. API to create and consume SAML assertions
    1. Allows customers to create SSO solutions independent from web services
  4. Issuing Token
    1. Supports an external STS (Security Token Service)
      1. Tested with Tivoli Federated Identity Manager
      2. API supports request and validation of SAML Assertions via standard WS-Trust v1.2 and v1.3 Protocols
    2. Supports sender (client) side SAML token caching for better performance
    3. Supports self-issuance
It is now available, so try it out and feel free to give us some feedback on your experiences.

P.S. Here is the SAML support Info Center link.

Wednesday, October 14, 2009

Tips on optimizing your JAX-RPC web services client

The new JAX-WS engine in WebSphere offers a lot of new capabilities, such as SOAP 1.2 support, MTOM-based attachments and Reliable Messaging. It also performs better in general. For new development projects, I usually use JAX-WS.

However, if you have some existing JAX-RPC client code and do not have the opportunity to migrate them to JAX-WS yet, there are some common techniques you can employ to make it perform better.

These are tips that myself and my web services colleagues in IBM GBS and ISSW have built up over the last 5 years.

Tip #1: Always cache and reuse WebSphere generated JAX-RPC stubs: Initialize and cache JAX-RPC stub and reuse it for subsequent invocations. Stub initialization is both CPU and memory intensive. These stubs are thread-safe and you could use the same initialized instance for all threads.

Tip #2: Remember to specify maximum amount of time the client would wait for a synchronous SOAP response: This setting is per service, defined in ibm-webservicesclient-bnd.xmi (syncTimout attribute). In WebSphere 6.0.2, this defaults to 300 seconds which is usually longer than typical requirements. This reduces the amount of time that the container needs to hold on to a thread in case of outages.

Tip #3: Use WebSphere web services caching for services that mostly provide static data (e.g. lookup tables):
This reduces the number of outbound calls required and the need for caching manually inside the application. See this link for details.

Tip #4: Enable the use of Local Transport Channel for intra-JVM service invocations: This eliminates the trip to the network stack for intra-JVM web services calls. (enableInProcessConnections)

Tip #5: Reduce data transmission time by compressing SOAP/http payload using gzip:
WebSphere supports the compression of request and responses.
(using Http Transport Property: com.ibm.websphere.webservices.http.requestContentEncoding=gzip)

Tip #6: Send large attachments out of band: If possible, send an URL where the client can pick up the attachment; instead of sending it as part of the SOAP with attachment message.

Tip #7: Use SSL instead of WS-Security (XML-Signature, XML-Encryption) if appropriate: This will improve throughput and lower memory requirement for your JAX-RPC client. SSL at the transport level is much more efficient than doing it inside the XML payload.

Tip #8: Tune WebSphere outbound HttpOutboundChannel: Monitor the usage of the outbound HttpConnection pool by this WebSphere trace settings (com.ibm.ws.webservices.engine.transport.channel.OutboundConnectionCache=fine) and adjust maxConenction accordingly.

Key settings include:
com.ibm.websphere.webservices.http.maxConnection (50)
com.ibm.websphere.webservices.http.connectionIdleTimeout (5 seconds)
com.ibm.websphere.webservices.http.connectionTimeout (300 seconds)

For a more detailed description of these settings, see HttpTransport properties in InfoCentre.

That's all for now... Hope this helps!

Billy.

(thanks to Russell Butek, Jackie Paradis and Greg Truty for reviewing this)


Tuesday, September 22, 2009

EJB 3.0 and JAX-WS Web Services

I was in a discussion with a number of my colleagues yesterday and the topic came up of using EJB 3.0 with JAX-WS web services. Essentially, they weren't sure if the @EJB annotation (which injected a link to an EJB 3.0 bean) could be used with JAX-WS endpoints. The question comes up since they only though injection worked on servlets (but this wasn't true). This is something that we think will be a typical usage scenario in that users can invoke existing services downstream on an EJB 3.0 bean. These scenarios may not necessarily be exposing EJB 3.0 beans as web services but merely using an EJB as part of your business logic to obtain/determine information needing to use or share back to a customer.


The general scenario looks a little something like the following image below:

The code to develop it is very straight forward. It's basically starting out with an EJB 3.0 bean like the code below:

import javax.ejb.Stateless;

@Stateless
public class SampleBean implements SampleBeanLocal {
public SampleBean() {}

public String echo(String s) {
return s;
}
}


with a simple local interface...

import javax.ejb.Local;

@Local
public interface SampleBeanLocal {
public String echo (String s);
}


it becomes easy to then inject a reference into a JAX-WS bean in order to get this to work... The code is also pretty trivial...

import javax.ejb.EJB;
import sample.ejb.SampleBeanLocal;

@javax.jws.WebService
public class SampleService{
@EJB(mappedName = "sample.ejb.SampleBeanLocal")
private SampleBeanLocal sample;

public String echo(String arg0) {
return sample.echo(arg0);
}
}


and that's about it. This is supported in IBM WebSphere Application Server v7.0. Let us know if you have any questions.

Monday, September 21, 2009

Asserting an Identity from JAX-RPC Web Services to WebSphere

I've received similar questions from multiple customers all about asserting identity on Web Services requests within JAX-RPC. Therefore, let me talk a bit about how this can be done with the support in WebSphere today.

The JAX-RPC web services security implementation has two phases of processing a security token embedded in the WS-Security header.
  1. Verify the validity of the security token.
    This is handled by the Token Consumer and/or JAAS Login Module configured in the deployment descriptor and binding.
  2. Verify the identity of the security token exist in the configured user registry and then create the WebSphere credentials.
    This is enabled via the Caller and it is optional.
In some business use cases, you may want to assert the identity based on trust and do not necessary want to add the identity to the configured user registry. Here is how you can do this (since 6.1.0.19).

Phase 1

The user must provide a custom implementation of a Token Consumer. Here is a developerWorks article with step by step instructions.

Phase 2

The user must override the default JAAS Login Configuration.

The default JAAS Login Configuration used by the Caller to create WebSphere Principal and Credential is “system.DEFAULT”. This JAAS Login Configuration is used by many other parts of the system. It is best to create a new JAAS Login Configuration with the same content of “system.DEFAULT” and ensure your custom JAAS Login Module is the first in the list, and then specify the Caller to use the new JAAS Login Configuration. The property to specify a different JAAS Login Configuration for the Caller (see figure below):


Name: com.ibm.wsspi.wssecurity.Caller.assertionLoginConfig

Value: name of the JAAS Login Configuration



Please follow the WebSphere Attribute Assertion framework in your custom JAAS Login Module implementations to assert the identity. The information is available here and here.

I hope you find this information useful.

Friday, September 18, 2009

Web Services Performance Tidbits

My team was recently working with a customer on their web services application and we wanted to document a few things we learned during the engagement.

Network bottlenecks

In our IBM labs, when we performance test web services, we set up a set of multi-threaded clients pounding a separate server system until the CPU utilization on the server system becomes saturated (that way - we can see where the time is spent processing). Working with the customer, they could only stress their system to a small portion of the CPU processing limit.

In the end, we discovered that the network was only a 100MB network and totally inadequate to handle the type of loads used in the test - whereas, in our IBM labs, we run on 1GB Ethernet. Using the size of the messages, we computed what the maximum achievable throughput would be on a 100MB network and these matched almost exactly to what was being reported for the customer. This showed us that the network was the bottleneck in the environment and the Application Servers are not being stressed at all - only the network connection.

One way to get around the network bottleneck in an interoperable way would be to use GZIP compression to compress the data (since the network is the bottleneck). See here and here for more information about enabling GZIP compression for Web Services.

Transaction Isolation Level

Another interesting point that we learned was that different Application Server vendors use different transaction isolation levels on the JDBC connection to the database. WebSphere, by default, uses Repeatable Read as a default isolation level which initially made our performance slower. This default was made to protect customers data. Another vendor used a default of Read Committed isolation level which offers higher throughput but more risk. The choice of which isolation level to use is up to the customer but in a comparison, this setting should be consistent to get apples-to-apples measurements.

Tuning

WebSphere does provide many parameters that can be tuned to increase performance. Some of the important ones for web services are JVM heap size and Web container thread pools. A white paper was recently published by our performance team that shows the impact of adjusting some of these. See WAS Perf Tuning for more info


We hope this information helps others as they try to build performant web service applications.

Thursday, August 27, 2009

Apache Wink 0.1 Goes Live!

After almost 2 months of being in the Apache Incubator, the Apache Wink team is excited to announce that we have published the first release of our JAX-RS-based REST runtime.

Wink was started as a joint project between HP and IBM in an effort to build a lightweight runtime focused only on REST. The Wink community is comprised of mostly developers from both organizations, with both teams having contributed their existing runtimes in the original proposal. That said, the project is starting to gain traction and we expect it to grow quickly. Although Wink is technically an incubator project, the code for Wink has been around for almost two years now, with some users already in production on a prior version of the runtime.

You can download the new runtime here, http://incubator.apache.org/wink/downloads.html.

Some of the features in the inital Wink release include:

A JAX-RS 1.0 compliant runtime

Wink 1.0 is fully compliant with the JAX-RS 1.0 specification.

Java Client API

Wink includes a Java client API that can be used to invoke REST-based services. The intent of the API is to model the remote resource with a local facade. This client API supports all of the data mappings that are supported on the server side by leveraging the same JAX-RS Provider classes to serialize and deserialize the data.

Simple servlet deployment

Like other JAX-RS runtimes that are available, you can package Wink as a library in your web application and add an entry for a new serlvet in your application configuration (web.xml).

Databinding support for JSON, XML (JAXB), Atom and more

JSON, XML and Atom are the standard databinding types used in REST-based applications. Wink includes support for each of these data types, and supports integrating other 3rd party libraries that handle these data types as well. Along with the standard types, Wink also includes data models for CSV data

Extensible runtime

The Wink architecture allows you to add in extensions by introducing custom handlers into the runtime. This is a perfect solution for adding in things like audit logging.

Dynamic resource registration

Need to register a new resource on the fly? Wink provides an API for doing so.


We encourage you to take a look at the runtime and tell us what you think. Feel free to join us in the community by registering and sending us a note via the mailing lists. Or, if you find an issue you'd like address or would like to make a suggestion for improvement, open an issue in our bug tracker and we will take a look.

Welcome to the WebSphere Web Services blog!

Welcome to the WebSphere Web Services blog! We've created this space to discuss topics related to web services in general with you. Specifically, this blog will cover both the traditional SOAP-based web services along with REST web services.