Posts Tagged ‘GAE’

Using RDBMS on Google App Engine : Google Cloud SQL

December 16, 2011 7 comments

We have been working on Google App Engine since its early days and have developed and ported several applications on the platform for our clients. One of the major hinderance for our Enterprise clients for not being able to move to GAE has been Datastore being the only storage option available till date. Major reasons for these enterprises not using Datastore are:

  • (In case of migration) Substantial efforts in adapting the application code base to Google Datastore specifications
  • Potentially huge data-migration efforts
  • Vendor Lock-In as both the application code and data residing in Datastore can’t be taken to an in-premise Infrastructure or another cloud platform seamlessly

Keeping all these requests and potential business opportunities in mind, Google has recently announced Cloud SQL.

What is Google Cloud SQL

  • It’s a Relational Database on Cloud offering from Google.
  • Offers MySQL Database environment with JDBC support for Java applications and DB-API support for Python applications.
  • Being a service on Cloud, it offers traditional advantages of any Cloud service like no maintenance, administration, replication and backup efforts
  • To ensure high-availability, data gets replicated across multiple geographic regions
  • Easy to use Web-Interface for managing multiple Instances

Google Cloud SQL & Google App Engine

A very good integration has been provided for your applications running on Google App Engine to use Google Cloud SQL as back-end storage mechanism. Furthermore, one SQL instance can be shared across multiple Google App Engine applications. This feature should be quite helpful as often we have a family of Web Applications for different user-requirements but still sharing the same Database.
Read more…

Categories: Cloud, Java, Web Tags: ,

Efficiently Getting a Count of All Entities in Google App Engine

September 7, 2011 Comments off

While developing applications on Google App Engine, you might need to find out total number of an Entity type being stored in the underlying Datastore. For people like me, coming from RDBMS background, it means one line query statement :


People starting with GAE might be surprised to know that we don’t have anything similar in GQL. The reason behind this limitation is the way Google Big Table (underlying storage for Datastore) is structured. It doesn’t support row counts as a fundamental concept. The only way for Datastore to return the result for us would be to run through the entire application data-set and execute query for finding matching entities. Following Java code will give us the desired result:

		Query query = new Query("Entity Name");
		DatastoreService datastoreService = DatastoreServiceFactory.getDatastoreService();
		List<Entity> allEntities = datastoreService.prepare(query).asList(FetchOptions.Builder.withDefaults());
                return allEntities.size();

Some of the potential issues with the above code are:

  • As the Datastore will execute queries to find matching entities, it will result in degrading performance as the number of entities increase over a period of time.
  • All the Datastore operations are charged. If we are going to require this information quite frequently, it might adversely impact our final bills to be paid to Google and even more with new Pricing model to be effective soon.
  • There is a hard limit of 30 seconds on very request to be executed. In case, we are dealing with large number of entities, it may just not work at all.

Following are a couple of different approaches for the problem:
Read more…

Categories: Cloud, Java Tags: ,

Take care of Version Number while Deploying on Google App Engine

August 23, 2011 5 comments

Few days back, I spent hours and hours trying to figure out why my web application was not getting deployed on Google App Engine. To save others from similar trouble, I would like to mention the reason behind the problem in my case.

Problem Context Mine was a Java based web application but the language and the contents are really not important in this case. When I was running the application in local GAE environment, everything was working perfectly fine. There were couple of versions already deployed on Google App Engine and they were also accessible. I was owner of the application, so there were no issues from permissions side as well. But while deploying the latest version update on Google App Engine, I was consistently getting 500 Internal Server Error. The detailed error message was :

Unable to update app: Error posting to URL:
500 Internal Server Error

Server Error (500)
A server error has occurred.

Read more…

Categories: Cloud Tags: ,

Extending Vosao CMS through Custom Plugin Development

August 19, 2011 1 comment

In my last post, I shared our experiences about building a large organization’s Intranet on Google App Engine using Vosao as underlying CMS. During that time, I touched upon Vosao’s extensible architecture which lets developers easily extend it’s functionality through plugin-based framework. In this post, I would like to talk about one of the plugins, we developed for our Intranet application.

Our requirement was that only authenticated users should be able to access the Intranet. Once that’s done, we should be displaying a Welcome message with connected user’s name and also provide a log-out URL at the top of all the pages. Fairly standard requirements, we would say for any company’s intranet but Vosao is designed to develop any interactive web-sites. It doesn’t impose any authentication mechanism for accessing the web-application and this seems logical as well. The decision about whether and which type of authentication is required is completely dependent on the application we are trying to develop.
Read more…

Categories: Cloud, Java Tags: , ,

CMS for your Web Application on Google App Engine

August 12, 2011 2 comments

We have been working on Google App Engine right from its early stages and have developed several Web applications of various complexities on the platform. Recently we got a request from one of very large enterprise clients to port their existing (in-premise hosted) Intranet over to Google App Engine. As they had already decided to move their existing mail infrastructure to Google Apps, it was logical for them to leverage the benefits of Google App Engine for their Intranet as well.

In most of the scenarios, Company Intranet servers as a placeholder for sharing information among the employees. The information can be about new projects, upcoming events, quarterly/annual reports, birthdays, training calendar etc. and it generally keeps on changing at regular intervals. It would be ideal to have a CMS(Content Management System) embedded inside our web application. On one hand, Google App Engine would provide us the standard benefits of PaaS cloud while CMS would let us quickly and easily edit the contents without going through build and deployment cycles. For CMS, our preference was for Open Source solution for reasons of cost and also the possibility to modify the source to suit our requirements. During our search we came across following options:

Java based

Python based

Our client’s technical team is well-versed with Java and for long-term maintenance reasons, we decided to stick to Java based solutions. After evaluating all Java based options, we decided to use Vosao for building our Intranet. Below are some of the reasons for the decision:
Read more…

Categories: Cloud, Java Tags: , , ,

Experiences at Cloud Computing Conference Pune 2011

June 7, 2011 Comments off

I was one of the speaker of the second IndicThreads conference held at Pune on 3-4th June 2011.

Sessions at the conference dealt with key topics like Cloud Security, Amazon Elastic Beanstalk, Legal Issues in Cloud Computing, OpenStack, Xen Cloud Platform, Rails and CouchDB on the cloud, CloudFoundry, Gigapaces PAAS, Monitoring Cloud Applications, ORM with Objectify-Appengine, Scalable Architecture on Amazon AWS Cloud, Cloud Lock-in, Cloud Interoperability, Apache Hadoop, Map Reduce and Predictive Analysis.

My talk focussed on managing persistence on GAE. It dealt with choices available to a developer and then focussed on doing it with Objectify-Appengine.

Demo application is present here for download. For the instructions on how to run it please read the wiki.

Mailing Google Docs As Attachment On Google App Engine

November 17, 2010 1 comment

Our application works with Google Spreadsheet API to update spreadsheets. The application now needed to send mail with an attachment of a google spreadsheet. We posted our query on google docs community here and on Google app engine here and there were no clear directions of implementing it.

Let’s see how we went about implementing this feature.

Google provides its spreadsheet API and it allows you to create and retrieve data in Google Spreadsheets, but it does not allow us to either create or manage their permissions. Google also has Google Documents List API which can be used to create and retrieve a list of Google Spreadsheets. For our case Google Documents List API provides exporting functionality such that we can export Google documents in common formats like pdf, rtf, xls and others. Google also provides java mail api and with its Multi-Part Messages feature we can send attachment as well.

For implementing this, we used spreadsheet API to retrieve spreadsheet entry. Using the spreadsheet entry we used Google Documents List API to export it in the specified format and we used java mail api to send exported data as an attachment.

We used play framework for our implementation but it can also be applied on other java frameworks as well. The class EmailSender first creates a google spreadsheet service and document service.

public class EmailSender extends Controller {

private static final String META_FEED_URL = "";
private static String spreadSheetName = "spreadsheetname";
private static String yourApplicationName = "emailSender-version1";
private static String userEmail = "";
private static String recepientEmail = "";
private static String password = "password";

public static void emailSpreadsheetAsAttachment() throws AuthenticationException, MessagingException, IOException {
	SpreadsheetService spreadSheetService = createSpreadSheetService();
	DocsService documentService = createDocumentService();
	SpreadsheetEntry spreadsheetEntry = loadSpreadSheet(spreadSheetService, spreadSheetName);
	byte[] spreadSheetData = getSpreadSheetData(spreadSheetService, documentService, spreadsheetEntry);
. . .
. . .
private static SpreadsheetService createSpreadSheetService() throws AuthenticationException {
	SpreadsheetService spreadSheetService = new SpreadsheetService(yourApplicationName);
	spreadSheetService.setUserCredentials(userEmail, password);
	return spreadSheetService;

private static DocsService createDocumentService() throws AuthenticationException {
	DocsService docsService = new DocsService(yourApplicationName);
	docsService.setUserCredentials(userEmail, password);
	return docsService;
. . .
. . .

Read more…

Categories: Cloud, Java Tags:

Managing Wicket Serialization Problem On Google App Engine

October 20, 2010 1 comment

There are several problems of using Wicket on Google App Engine. We are porting a Wicket application on Google App Engine and faced several issues. This post will look at the problems we may encounter while working on a Wicket project on Google App engine and how may we overcome them.

Google App engine will it play suggests that if you have a wicket project you follow this workaround your Wicket project will start working on app engine. Still, one of the error you may encounter while working on app engine are Wicket Serialization errors.

Wicket is very powerful in maintaining application/session state. This enables for example a nice usage of back/forward buttons. In a typical Wicket scenario, this is backed by the DiskPageStore implementation. But on Google App Engine we can’t use the DiskPageStore as it relies on writing to the filesystem. Therefore HttpSessionStore is used in Wicket Application class. In this implementation Wicket components and associated model will be saved in Session and therefore in datastore.

This may result in the session getting bloated till we encounter this error$RequestTooLargeException: The request to API call datastore_v3.Put() was too large on datastore.

The way to manage this problem is that we use a different pageStore implementation which uses google memcache instead. If memcache implementation is used then the data: that is wicket components and its models will not be associated with the Session and therefore will not end up in datastore.

Memcache also has a limit on the data size limit and you may get an exception if data to be entered in it is high "Caused by:$RequestTooLargeException: The request to API call memcache.Set() was too large."

MemcachePageStore that we used has a MAX_PAGES_PER_MAP instance variable and using this we can avoid the "memcache.Set() was too large" errors. Let’s look at how we went about implementing it. We tell our Wicket Application class to use Memcache based page store instead of HttpSessionStore.

public class EhourWebApplication extends AuthenticatedWebApplication {
       . . .
	protected ISessionStore newSessionStore() {
		return new SecondLevelCacheSessionStore(this, new MemcachePageStore(3));
		// return new HttpSessionStore(this);
       . . .

Read more…

Categories: Cloud, Java Tags: ,

Managing Cold Start On Google App Engine

October 9, 2010 4 comments

One of the major problem working on Google App engine is managing cold start. Well what is it?

Google App engine spins down the application when it is idle. On Google App engine, application does not get a permanent instance of java virtual machine. If there are no activities for a while then the JVM goes cold and application will have to start again to render a request. This is cold start and your application will be sluggish if you render a request to it after a while.

GAE team has accepted the issue and introduced “Ability to reserve instances to reduce application loading overhead” into their roadmap.

This is an issue we traditionally are not aware about while developing an application on Google App engine. If we are working on a greenfield project and we have a liberty of choosing the frameworks then we can stay away from heavy frameworks.

The application we are porting is Spring/Wicket/JPA tehnology stack and we like others were suffering from cold start. When the application warms up all the Spring beans are initialized again. Since we use JPA the EntityManagerFactory adds a significant amount of time for startup.

The quick fix for the cold start till Google App engine team figures it out, is to ping the application every few minutes. This will keep the application from going into the cold storage.

For this implementation to work we require a Servlet which essentially does nothing apart from receiving the request. We also require a cron job which renders request every few minutes. Let’s look at the servlet code listing and its mapping in web.xml.

public class PingServlet extends HttpServlet {

   private Logger logger = Logger.getLogger(PingServlet.class);

   protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {"Keeping the application warm");

   protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
	doGet(req, resp);

Read more…

Categories: Cloud, Java Tags:

Phi Cast : An Introduction to Google App Engine Java Platform

October 7, 2010 Comments off

Very happy to share our first Video Cast where Meetu and Vikas are discussing about GAE/J platform, it’s key features and advantages both for Business as well as Developers. We would love to hear you comments and have your feedback.

Enjoy !
Inphina Team

Categories: Cloud, Java Tags: ,

Get every new post delivered to your Inbox.

%d bloggers like this: