Azure Storage Services Asynchronously in Java

| Comments

When performing I/O bound operation, the program should use asynchronous approach.  This is particularly important when you access the Azure storage services.  As of now, Azure managed libraries for .NET and Java do not support asynchronous APIs.  Instead, by using underlying run time’s asynchronous programming approaches along with Azure storage services REST API makes you do I/O bound operations on Azure storage services.

In this post, I explain how to access Azure blob storage services asynchronously in Java.  I have used following libraries:

  • org.apache.commons.codec-1.6.jar (for base64 string encoding)
  • async-http-client-1.7.3.jar (Ning’s Async Http Client library – https://github.com/sonatype/async-http-client)
  • log4j-1.2.16.jar and slf4j-*-1.6.4.jar (Logging)

AzureStorage Class

The class “AzureStorage” contains the implementation to create HTTP request object for accessing the Azure RESTful resources.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

public class AzureStorage...

// fields
String storageMedium;
String accountName;
byte[] secretKey;
String host;
java.util.regex.Pattern urlAbsolutePathPattern;

//ctor
AzureStorage(String accountName, String storageMedium, String base64SecretKey)
// public method
Request get(String resourcePath)
// utility method
String createAuthorizationHeader(Request request)

The constructor requires storage account name, storage medium (this is neither Java nor Azure terminology, just identify whether you want to access blob, table or queue) and the primary shared key of the account.  In this post, I just provide simple get() method for GET related Azure storage APIs.  The input to the method is the resource path.  Most the REST API requires authorization which in-turn sign the particular request by shared key.   createAuthorizationHeader() method does this job.

The Ctor

1
2
3
4
5
6
7

public AzureStorage(String accountName, String storageMedium, String base64SecretKey) {
this.accountName = accountName;
this.storageMedium = storageMedium;
this.host = "core.windows.net";
secretKey = Base64.decodeBase64(base64SecretKey);
urlAbsolutePathPattern = Pattern.compile("http(?:s?)://[-a-z0-9.]+/([-a-z0-9]*)/\\?.*");
}

The host field contains the part of base Azure storage URL.  The primary shared key for the account has been converted to base 64 decoded byte array.  Since, there is no AbsolutePath facility from an URL in Java world, I have used regular expression here.  For example, the absolute path of the URL “https://myaccount.blob.core.windows.net/acontainer/?restype=container&comp=list” is “acontainer”.

The get() method

A HTTP request should be made to access a Azure storage with following details:

1
2
3
4

GET https://myaccount.blob.core.windows.net/acontainer?restype=container&comp=acl&timeout=90 HTTP/1.1
x-ms-version: 2009-09-19
x-ms-date: Fri, 20 Apr 2012 11:12:05 GMT
Authorization: SharedKey myaccount:9S/gs8jkAQKAN1Gp/y82B8jHR2r7HShZSiPdl2JSWQw=

The above request specifies the URL for the resource, the REST API version, the request time stamp, authorization header.  The get() method here frames these request headers.  To want to know the complete details of request and response of Azure REST API, visit http://msdn.microsoft.com/en-us/library/windowsazure/dd179355.aspx.

1
2
3
4
5
6
7

public Request get(String resourcePath) {
String RFC1123_PATTERN = "EEE, dd MMM yyyy HH:mm:ss z";
DateFormat rfc1123Format = new SimpleDateFormat(RFC1123_PATTERN);
rfc1123Format.setTimeZone(TimeZone.getTimeZone("GMT"));

// remaining code
}

The rfc1123Format is used to send request time stamp in RFC 1123 format as shown in the HTTP request.  The below code snippet creates the com.ning.http.client.Request object.

1
2
3
4
5
6
7
8
9
10
11

String url = "https://" + this.accountName + "." +
this.storageMedium + "." +  this.host + "/" + resourcePath;

RequestBuilder builder = new RequestBuilder("GET");

Request request = builder.setUrl(url)
.addHeader("content-type", "text/plain")
.addHeader("content-length", "0")
.addHeader(HeaderDate, rfc1123Format.format(new Date()))
.addHeader(HeaderPrefixMS + "version", "2009-09-19")
.build();

The below code creates signed Authorization header.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

String authHeader = "";

try {
authHeader = createAuthorizationHeader(request);
} catch (InvalidKeyException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

request.getHeaders().add("Authorization", "SharedKey " + this.accountName
+ ":" + authHeader);
return request;

This part in-turn calls method createAuthorizationHeader().

The createAuthorizationHeader() method

1
2
3
4
5
6
7
8
9
10
11
12
13
14

private String createAuthorizationHeader(Request request)
throws UnsupportedEncodingException,
NoSuchAlgorithmException, InvalidKeyException {

FluentCaseInsensitiveStringsMap headers = request.getHeaders();

StringBuffer stringToSign = new StringBuffer();
stringToSign.append(request.getMethod() + "\n");
stringToSign.append("\n\n0\n\n");
stringToSign.append(headers.get("content-type").get(0) + "\n");
stringToSign.append("\n\n\n\n\n\n");

// remaining code part
}

The authorization header should be like

1

Authorization="[SharedKey|SharedKeyLite] <AccountName>:<Signature>"

The createAuthorizationHeader() method mainly creates the “” string. The “Signature” is a HMAC-SHA256 based computed hash of the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

GET\n /*HTTP Verb*/
\n    /*Content-Encoding*/
\n    /*Content-Language*/
\n    /*Content-Length*/
\n    /*Content-MD5*/
\n    /*Content-Type*/
\n    /*Date*/
\n    /*If-Modified-Since */
\n    /*If-Match*/
\n    /*If-None-Match*/
\n    /*If-Unmodified-Since*/
\n    /*Range*/
x-ms-date:Sun, 11 Oct 2009 21:49:13 GMT\nx-ms-version:2009-09-19\n    /*CanonicalizedHeaders*/
/myaccount/myaccount/acontainer\ncomp:metadata\nrestype:container\ntimeout:20    /*CanonicalizedResource*/

For more details about this, visit: http://msdn.microsoft.com/en-us/library/windowsazure/dd179428.aspx
The above Java code adds the string starting from GET to Range. For this demonstration, I skipped most of the headers with newline and added only content-length and content-type headers. The below code constructs the CanonicalizedHeaders.

1
2
3
4
5
6
7
8
9
10
11
12

List httpStorageHeaderNameArray = new ArrayList();

for(String key : headers.keySet()) {
if(key.toLowerCase().startsWith(HeaderPrefixMS)) {
httpStorageHeaderNameArray.add(key.toLowerCase());
}
}

Collections.sort(httpStorageHeaderNameArray);
for(String key : httpStorageHeaderNameArray) {
stringToSign.append(key + ":" + headers.get(key).get(0) + "\n");
}

The below code constructs the CanonicalizedResource.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

java.util.regex.Matcher matcher = urlAbsolutePathPattern.matcher(request.getUrl());
String absolutePath = "";

if(matcher.find()) {
absolutePath = matcher.group(1);
}else {
throw new IllegalArgumentException("resourcePath");
}
stringToSign.append("/" + this.accountName + "/" + absolutePath);
if(absolutePath.length() > 0) stringToSign.append("/");

stringToSign.append("\n");

List paramsArray = new ArrayList();
for(String key : request.getQueryParams().keySet()) {
paramsArray.add(key.toLowerCase());
}
Collections.sort(paramsArray);
for(String key : paramsArray) {
stringToSign.append(key + ":" + request.getQueryParams().get(key).get(0) + "\n");
}

Finally, the whole string should be signed with by the shared key.

1
2
3
4
5
6
7

byte[] dataToMac = stringToSign.substring(0, stringToSign.length() -1).getBytes("UTF-8");

SecretKeySpec signingKey = new SecretKeySpec(secretKey, "HmacSHA256");
Mac hmacSha256 = Mac.getInstance("HmacSHA256");
hmacSha256.init(signingKey);
byte[] rawMac = hmacSha256.doFinal(dataToMac);
return Base64.encodeBase64String(rawMac);

The Calling Side

At the calling end, when you invoke the get() method, it returns the com.ning.http.client.Request instance.  You can make request asynchronously using Ning’s async library as shown below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

AzureStorage blobStorage = new AzureStorage("account-name", "blob|table|queue", "sharedkey");
Request request = blobStorage.get("ablobcontainer/?restype=container&comp=list");
AsyncHttpClient client = new AsyncHttpClient();
ListenableFuture response = client.executeRequest(request, new AsyncHandler() {
  private final Response.ResponseBuilder builder = new Response.ResponseBuilder();

    public STATE onBodyPartReceived(final HttpResponseBodyPart content) throws Exception {
        builder.accumulate(content);
        return STATE.CONTINUE;
    }

    public STATE onStatusReceived(final HttpResponseStatus status) throws Exception {
        builder.accumulate(status);
        return STATE.CONTINUE;
    }

    public STATE onHeadersReceived(final HttpResponseHeaders headers) throws Exception {
        builder.accumulate(headers);
        return STATE.CONTINUE;
    }

    public Response onCompleted() throws Exception {
        return builder.build();
    }

  public void onThrowable(Throwable arg0) {
      // TODO Auto-generated method stub

  }
});

Visit http://sonatype.github.com/async-http-client/request.html for more details about the above code. After that, you can do other computations. When you reach the place where you want the response for the asynchronous request, you can do the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// till here, there are other interesting computation done

while(! response.isDone()) {
  if(response.isCancelled()) break;
}

Response actualResponse;
try {
  actualResponse = response.get();

  System.out.println(actualResponse.getStatusCode());
  System.out.println(actualResponse.getResponseBody());

} catch (InterruptedException e) {
  // TODO Auto-generated catch block
  e.printStackTrace();
} catch (ExecutionException e) {
  // TODO Auto-generated catch block
  e.printStackTrace();
}

The “while” part just wait for the asynchronous operation to be completed. After that, it processes the com.ning.http.client.Response instance.

You can download the complete example from http://udooz.net/file-drive/doc_download/24-asyncazureaccessfromjava.html