Archive

Archive for the ‘Java’ Category

Using RDBMS on Google App Engine : Google Cloud SQL

December 16, 2011 5 comments

We have been working on Google App Engine since its early days and have developed and ported several applications on the platform for our clients. One of the major hinderance for our Enterprise clients for not being able to move to GAE has been Datastore being the only storage option available till date. Major reasons for these enterprises not using Datastore are:

  • (In case of migration) Substantial efforts in adapting the application code base to Google Datastore specifications
  • Potentially huge data-migration efforts
  • Vendor Lock-In as both the application code and data residing in Datastore can’t be taken to an in-premise Infrastructure or another cloud platform seamlessly

Keeping all these requests and potential business opportunities in mind, Google has recently announced Cloud SQL.

What is Google Cloud SQL

  • It’s a Relational Database on Cloud offering from Google.
  • Offers MySQL Database environment with JDBC support for Java applications and DB-API support for Python applications.
  • Being a service on Cloud, it offers traditional advantages of any Cloud service like no maintenance, administration, replication and backup efforts
  • To ensure high-availability, data gets replicated across multiple geographic regions
  • Easy to use Web-Interface for managing multiple Instances

Google Cloud SQL & Google App Engine

A very good integration has been provided for your applications running on Google App Engine to use Google Cloud SQL as back-end storage mechanism. Furthermore, one SQL instance can be shared across multiple Google App Engine applications. This feature should be quite helpful as often we have a family of Web Applications for different user-requirements but still sharing the same Database.
Read more…

Categories: Cloud, Java, Web Tags: ,

Efficiently Getting a Count of All Entities in Google App Engine

September 7, 2011 Leave a comment

While developing applications on Google App Engine, you might need to find out total number of an Entity type being stored in the underlying Datastore. For people like me, coming from RDBMS background, it means one line query statement :

SELECT COUNT(*) from TABLE_NAME;

People starting with GAE might be surprised to know that we don’t have anything similar in GQL. The reason behind this limitation is the way Google Big Table (underlying storage for Datastore) is structured. It doesn’t support row counts as a fundamental concept. The only way for Datastore to return the result for us would be to run through the entire application data-set and execute query for finding matching entities. Following Java code will give us the desired result:

import com.google.appengine.api.datastore.Query;
		Query query = new Query("Entity Name");
		DatastoreService datastoreService = DatastoreServiceFactory.getDatastoreService();
		List<Entity> allEntities = datastoreService.prepare(query).asList(FetchOptions.Builder.withDefaults());
                return allEntities.size();

Some of the potential issues with the above code are:

  • As the Datastore will execute queries to find matching entities, it will result in degrading performance as the number of entities increase over a period of time.
  • All the Datastore operations are charged. If we are going to require this information quite frequently, it might adversely impact our final bills to be paid to Google and even more with new Pricing model to be effective soon.
  • There is a hard limit of 30 seconds on very request to be executed. In case, we are dealing with large number of entities, it may just not work at all.

Following are a couple of different approaches for the problem:
Read more…

Categories: Cloud, Java Tags: ,

Achieve More with Fusion Tables using Fusion Tables API

August 22, 2011 1 comment

In my previous post, I gave a broad overview of Google Fusion Tables along with a brief Getting Started Guide. Continuing on the same subject, I would like to delve a bit deeper and discuss about Fusion Tables API.

Need of API As discussed in my earlier post, Fusion Tables is a Data Storage on Google Cloud. It allows us to store, share and execute SQL like queries on our data residing on Fusion Tables. We can of-course represent this data in multiple formats on Google Maps. As we saw in my previous post, publishing the contents of Fusion Tables data on a Google Map is quite easy. But more often than not, we would be dealing with dynamically changing data. For the Map to render the latest information, we will not be able to go through the route of importing tables at regular intervals. We would rather do these updates behind the scenes and let everyone have an up-to-date information all the time. That’s where Fusion Table API comes into picture.
Read more…

Categories: Cloud, Java Tags:

Extending Vosao CMS through Custom Plugin Development

In my last post, I shared our experiences about building a large organization’s Intranet on Google App Engine using Vosao as underlying CMS. During that time, I touched upon Vosao’s extensible architecture which lets developers easily extend it’s functionality through plugin-based framework. In this post, I would like to talk about one of the plugins, we developed for our Intranet application.

Our requirement was that only authenticated users should be able to access the Intranet. Once that’s done, we should be displaying a Welcome message with connected user’s name and also provide a log-out URL at the top of all the pages. Fairly standard requirements, we would say for any company’s intranet but Vosao is designed to develop any interactive web-sites. It doesn’t impose any authentication mechanism for accessing the web-application and this seems logical as well. The decision about whether and which type of authentication is required is completely dependent on the application we are trying to develop.
Read more…

Categories: Cloud, Java Tags: , ,

CMS for your Web Application on Google App Engine

August 12, 2011 2 comments

We have been working on Google App Engine right from its early stages and have developed several Web applications of various complexities on the platform. Recently we got a request from one of very large enterprise clients to port their existing (in-premise hosted) Intranet over to Google App Engine. As they had already decided to move their existing mail infrastructure to Google Apps, it was logical for them to leverage the benefits of Google App Engine for their Intranet as well.

In most of the scenarios, Company Intranet servers as a placeholder for sharing information among the employees. The information can be about new projects, upcoming events, quarterly/annual reports, birthdays, training calendar etc. and it generally keeps on changing at regular intervals. It would be ideal to have a CMS(Content Management System) embedded inside our web application. On one hand, Google App Engine would provide us the standard benefits of PaaS cloud while CMS would let us quickly and easily edit the contents without going through build and deployment cycles. For CMS, our preference was for Open Source solution for reasons of cost and also the possibility to modify the source to suit our requirements. During our search we came across following options:

Java based

Python based

Our client’s technical team is well-versed with Java and for long-term maintenance reasons, we decided to stick to Java based solutions. After evaluating all Java based options, we decided to use Vosao for building our Intranet. Below are some of the reasons for the decision:
Read more…

Categories: Cloud, Java Tags: , , ,

Google Apps Script And Getting Site Context

Google Script UI services enables to create rich UI in Google Sites. There is a bug where there is no way to find out directly where Google Apps Script is embedded in Sites. If you are faced with similar dilemma and need a workaround read on. This a critical issue when a script intended to be executed for multiple sites.

In our case the requirement was to fetch the data for site instances from the spreadsheet. Site instances were created from a template Site. The template Site containing the apps script required to know the active site, or in other words the UI apps script required to know the context of the site it is embedded in.

There is an existing method SitesApp.getActiveSite() which runs perfectly for the Script but fails when it is coded in Google UI Script. You can read more of it here.

The solution was suggested by Rahim where he found a way to wrap the Google Apps UI Script in a Google Widget and pass Site URL reference parameter to the UI script.

The Google Widget xml file has a reference to the published UI script url and it uses document.referrer javascript to get the URL where it is embedded in. This is how the Google Widget looks like.

<?xml version="1.0" encoding="UTF-8"?>
<Module>
<ModulePrefs title="Company Description" />
  <Content type="html"><![CDATA[<div id="txt"></div>
  <iframe id="gdgt" frameborder="0" height="100%" width="100%"></iframe>
  <script type="text/javascript">
  document.getElementById('gdgt').src='publishedUrl&siteurl=' + document.referrer;</script>
]]></Content>
</Module>

Read more…

Categories: Java, Web Tags: ,

Experiences at Cloud Computing Conference Pune 2011

I was one of the speaker of the second IndicThreads conference held at Pune on 3-4th June 2011.

Sessions at the conference dealt with key topics like Cloud Security, Amazon Elastic Beanstalk, Legal Issues in Cloud Computing, OpenStack, Xen Cloud Platform, Rails and CouchDB on the cloud, CloudFoundry, Gigapaces PAAS, Monitoring Cloud Applications, ORM with Objectify-Appengine, Scalable Architecture on Amazon AWS Cloud, Cloud Lock-in, Cloud Interoperability, Apache Hadoop, Map Reduce and Predictive Analysis.

My talk focussed on managing persistence on GAE. It dealt with choices available to a developer and then focussed on doing it with Objectify-Appengine.



Demo application is present here for download. For the instructions on how to run it please read the wiki.

Why CloudFoundry is Disruptive?

Recently, Inphina started exploring Cloud Foundry to add onto its cloud offerings. The initial reactions have varied from good to very good to but ‘could they have this…’ In general it looks like that Cloud Foundry (CF) would be disrupting the PaaS market eventually. Why? Primarily because of the following reasons

  • It is not tied to a particular environment or framework like .Net for Azure, Java/Python for the App Engine, Java for Beanstalk. CloudFoundry does support Java, Ruby, Grails, Node.js for now and the support for other languages / frameworks is on the roadmap.
  • No vendor lock-in and no cloud lock-in. That is right you can deploy on any cloud. CF PaaS can be deployed on Amazon.
  • Choice of Public and Private cloud infrastructures. This is one of the uphill battle that we are fighting with GAE at many enterprises. CloudFoundry PaaS can be deployed in your private datacenter and VMWare has enterprise support for that as well.
  • Open PaaS, so take the code and build on top of that and contribute back.
  • Backed by VMware and Spring (bought by VMware a couple of years back) which are known names in the enterprises. (I remember the day in earlier life when I wanted to bundle the Spring jar with my project and had to seek scores of approvals from the NY based financial house, but now no-one would blink an eyelid for Spring)

So does it mean that GAE and Azure would have to start losing sleep. Well, not at the moment because CF is still in beta. But, given the fact that GAE has partial support for Java and that Azure would be supporting .Net primarily (now that their Java offering has kind of fallen flat) the Open PaaS strategy of CF is intimidating. It enables developers to do what they do best i.e. develop in the language and framework of their choice and have little concerns about deployment environments.

One area where I feel still a lot of catch up is required is services. GAE as a PaaS offers a bouquet of services which are unmatched by any other PaaS. At least, at the moment. For a rapid development and prototyping approach I would still probably fall back to GAE where I get mails, namespaces, cache, channel api, oauth etc etc. Though CF could easily provide administration services with Hyperic and system level services with GemFire and RabbitMQ but other application level services which GAE can provide are still missing.

All in all, an Open PaaS which needs to followed closely. Enjoy the presentation …

Inphina, as an expert on Google App Engine and Google Apps has helped more than 10 medium to large organizations help take advantage of the cloud by building, migrating or re-engineering their complex line of business applications to the cloud thereby making significant reductions in their capex expenditure. If you are interested in an assessment send us a mail at cloud@inphina.com

Custom Versioning for Google Docs: Working with Google Docs on Google App Engine

May 16, 2011 1 comment

Inphina, as an expert on Google App Engine and Google Apps has enabled many medium to large organizations leverage the cloud by building, migrating or re-engineering complex line of business applications to the cloud making significant reductions in their capex expenditure. Contact us at cloud@inphina.com

As a last post in series of posts for building custom versioning for Google Docs, in this post, we would look at the interaction with Google Docs from Google App Engine. This piece is particularly interesting because we are talking cloud to cloud. Google Docs is hosted in the cloud and so is the app engine.

For working with Google Docs, there is a list of libraries that must be packaged with your application. The core set of libraries come from gdata-java-client and in this we are particularly interested in Google Documents List Data API.

First things first, you need to download the client library. Now, there are a select list of jars that we are interested in for Google docs, these are

<!-- Adding <span class="hiddenSpellError" pre="Adding ">Gdata</span> dependencies -->
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-collect</artifactId>
			<version>1.0-rc1</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-jsr</artifactId>
			<version>1.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-media</artifactId>
			<version>1.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-client</artifactId>
			<version>1.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-client-meta</artifactId>
			<version>1.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-core</artifactId>
			<version>1.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-docs</artifactId>
			<version>3.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.gdata</groupId>
			<artifactId>google-docs-meta</artifactId>
			<version>3.0</version>
		</dependency>

You would also need the mail, activation and servlet APIs.

<!-- Mail, Activation and <span class="hiddenSpellError" pre="and ">Servlet</span> -->

		<dependency>
			<groupId>javax.mail</groupId>
			<artifactId>mail</artifactId>
			<version>1.4.1</version>
		</dependency>
		<dependency>
			<groupId>javax.activation</groupId>
			<artifactId>activation</artifactId>
			<version>1.1.1</version>
		</dependency>

		<dependency>
			<groupId>javax.servlet</groupId>
			<artifactId>servlet-api</artifactId>
			<version>2.5</version>
		</dependency>

For the complete listing of my pom.xml click here pom

There are some libraries which are not present on the maven repository. You could use the following script to install them as a part of your local repo or in nexus.

if [ "$1" == "" ]
then
    echo "usage: $0 /path/to/gdata"
    exit 1
fi

GDATA_PATH=$1

mvn install:install-file -Dfile=$GDATA_PATH/java/deps/google-collect-1.0-rc1.jar -DgroupId=com.google.gdata -DartifactId=google-collect -Dversion=1.0-rc1 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/deps/jsr305.jar -DgroupId=com.google.gdata -DartifactId=google-jsr -Dversion=1.0 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/lib/gdata-media-1.0.jar -DgroupId=com.google.gdata -DartifactId=google-media -Dversion=1.0 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/lib/gdata-client-1.0.jar -DgroupId=com.google.gdata -DartifactId=google-client -Dversion=1.0 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/lib/gdata-client-meta-1.0.jar -DgroupId=com.google.gdata -DartifactId=google-client-meta -Dversion=1.0 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/lib/gdata-core-1.0.jar -DgroupId=com.google.gdata -DartifactId=google-core -Dversion=1.0 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/lib/gdata-docs-3.0.jar -DgroupId=com.google.gdata -DartifactId=google-docs -Dversion=3.0 -DgeneratePom=true -Dpackaging=jar
mvn install:install-file -Dfile=$GDATA_PATH/java/lib/gdata-docs-meta-3.0.jar -DgroupId=com.google.gdata -DartifactId=google-docs-meta -Dversion=3.0 -DgeneratePom=true -Dpackaging=jar

Ok, now we are all set to start coding.

The logic for the versioning component is simple. When there is a new file upload using the custom versioning component, it would

1. Check, if a file by the same name exists on the Google Docs for the user with whose credentials we are accessing the Google Docs. You could use oauth for authentication.
2. If there is no file with the name then upload the new document
3. else, version the earlier document with an incremented counter version number and assign it to an archival folder, then
4. upload the new file

The exact logic is written in the method below

Read more…

Custom Versioning for Google Docs: Handling File Operations on Google App Engine

May 16, 2011 1 comment

In the last post you saw how easy it was to use the functionality of GWTUpload and upload the file to the server. In this post we would look at the file handling for Google App Engine.

As you would know that File operations are not allowed on the app engine. You can however, read from a file if it is uploaded as a part of your application war or your application can use the IO classes which are useful for reading from the file system. The whitelisted list of classes is available here, but for us, the following are supported File, FileInputStream, FileReader, or RandomAccessFile.

By default the uploaded files are stored in the session for further processing by the GWTUpload servlet. We get all the files which are present in the session with the following code

@Override
	public String executeAction(HttpServletRequest request, List sessionFiles) throws UploadActionException {
		String response = "";
		List<FileItem> itemsTobeUploaded = new ArrayList();
		String description = getDescription(request);

		for (FileItem item : sessionFiles) {
			if (item.isFormField() == false) {
				itemsTobeUploaded.add(item);
			}
		}

		for (FileItem fileItem : itemsTobeUploaded) {
			try {
				documentService.uploadFileToGoogleDocs(fileItem, description);
			} catch (Exception e) {
				// catch all exceptions here for the purpose of this blogpost
			}
		}
		// / Remove files from session because we have a copy of them
		removeSessionFileItems(request);

		// / Send your customized message to the client.
		return response;
	}

As you would notice, we are overriding the executeAction method of the AppEngineUploadAction servlet. Here we are doing processing which is very specific to our application.

Here, we are passing the FileItem to the documentService which would upload the document on Google Docs. FileItem is an apache commons fileupload interface. It represents a file or a form item which was received within a multipart/form-data post request.

The uploadFileToGoogleDocs does some logic with Google Docs which we are not I nterested in right now. That would be the details of the post in which we talk about calling Google Docs from App Engine.

public boolean uploadFileToGoogleDocs(FileItem file, String description) throws MalformedURLException, IOException, ServiceException {
		boolean uploadStatus = false;

		//...  some logic

			uploadNewFile(docsService, file, description);

		//... some more logic

	}

For now let us look at the details of the uploadNewFile(docsService, file, description); method

private void uploadNewFile(DocsService docsService, FileItem file, String description) throws AuthenticationException, IOException,
			ServiceException, MalformedURLException {
		String mimeType = DocumentListEntry.MediaType.fromFileName(file.getName()).getMimeType();

		DocumentListEntry newDocument = new DocumentListEntry();
		newDocument.setMediaSource(new MediaByteArraySource(file.get(), mimeType));
		newDocument.setTitle(new PlainTextConstruct(file.getName()));
		newDocument.setDescription(description);
		System.out.println("Uploaded document with description " + description);
		docsService.insert(new URL(DOCUMENT_URL), newDocument);
	}

As you would notice that we get the contents of the file not as a file but as a byte[]. We have a method in DocumentListEntry called setMediaSource

setMediaSource
public void setMediaSource(MediaSource mediaSource)
Description copied from interface: IMediaEntry
Sets the MediaSource that contains the media data for the entry.

Through this method, we can set the MediaSource to be a ByteArray. The file.get() method results in a byte[] and the mimeType is easily obtained with file.getName().getMimeType()
Hence we are able to extract the contents of the file from the app engine and pass it onto the Google document service.
In the next post, we would get into the details of Google Document Service and see how we can use it to work with Google Docs.

Inphina, as an expert on Google App Engine and Google Apps has helped more than 10 medium to large organizations help take advantage of the cloud by building, migrating or re-engineering their complex line of business applications to the cloud thereby making significant reductions in their capex expenditure. If you are interested in an assessment send us a mail at cloud@inphina.com

Follow

Get every new post delivered to your Inbox.