While developing applications on Google App Engine, you might need to find out total number of an Entity type being stored in the underlying Datastore. For people like me, coming from RDBMS background, it means one line query statement :
SELECT COUNT(*) from TABLE_NAME;
People starting with GAE might be surprised to know that we don’t have anything similar in GQL. The reason behind this limitation is the way Google Big Table (underlying storage for Datastore) is structured. It doesn’t support row counts as a fundamental concept. The only way for Datastore to return the result for us would be to run through the entire application data-set and execute query for finding matching entities. Following Java code will give us the desired result:
import com.google.appengine.api.datastore.Query; Query query = new Query("Entity Name"); DatastoreService datastoreService = DatastoreServiceFactory.getDatastoreService(); List<Entity> allEntities = datastoreService.prepare(query).asList(FetchOptions.Builder.withDefaults()); return allEntities.size();
Some of the potential issues with the above code are:
- As the Datastore will execute queries to find matching entities, it will result in degrading performance as the number of entities increase over a period of time.
- All the Datastore operations are charged. If we are going to require this information quite frequently, it might adversely impact our final bills to be paid to Google and even more with new Pricing model to be effective soon.
- There is a hard limit of 30 seconds on very request to be executed. In case, we are dealing with large number of entities, it may just not work at all.
Following are a couple of different approaches for the problem:
Few days back, I spent hours and hours trying to figure out why my web application was not getting deployed on Google App Engine. To save others from similar trouble, I would like to mention the reason behind the problem in my case.
Problem Context Mine was a Java based web application but the language and the contents are really not important in this case. When I was running the application in local GAE environment, everything was working perfectly fine. There were couple of versions already deployed on Google App Engine and they were also accessible. I was owner of the application, so there were no issues from permissions side as well. But while deploying the latest version update on Google App Engine, I was consistently getting 500 Internal Server Error. The detailed error message was :
Unable to update app: Error posting to URL: https://appengine.google.com/api/appversion/create?app_id=itj-helloworld&version=4.0.0& 500 Internal Server Error Server Error (500) A server error has occurred.
In my last post, I shared our experiences about building a large organization’s Intranet on Google App Engine using Vosao as underlying CMS. During that time, I touched upon Vosao’s extensible architecture which lets developers easily extend it’s functionality through plugin-based framework. In this post, I would like to talk about one of the plugins, we developed for our Intranet application.
Our requirement was that only authenticated users should be able to access the Intranet. Once that’s done, we should be displaying a Welcome message with connected user’s name and also provide a log-out URL at the top of all the pages. Fairly standard requirements, we would say for any company’s intranet but Vosao is designed to develop any interactive web-sites. It doesn’t impose any authentication mechanism for accessing the web-application and this seems logical as well. The decision about whether and which type of authentication is required is completely dependent on the application we are trying to develop.
We have been working on Google App Engine right from its early stages and have developed several Web applications of various complexities on the platform. Recently we got a request from one of very large enterprise clients to port their existing (in-premise hosted) Intranet over to Google App Engine. As they had already decided to move their existing mail infrastructure to Google Apps, it was logical for them to leverage the benefits of Google App Engine for their Intranet as well.
In most of the scenarios, Company Intranet servers as a placeholder for sharing information among the employees. The information can be about new projects, upcoming events, quarterly/annual reports, birthdays, training calendar etc. and it generally keeps on changing at regular intervals. It would be ideal to have a CMS(Content Management System) embedded inside our web application. On one hand, Google App Engine would provide us the standard benefits of PaaS cloud while CMS would let us quickly and easily edit the contents without going through build and deployment cycles. For CMS, our preference was for Open Source solution for reasons of cost and also the possibility to modify the source to suit our requirements. During our search we came across following options:
Our client’s technical team is well-versed with Java and for long-term maintenance reasons, we decided to stick to Java based solutions. After evaluating all Java based options, we decided to use Vosao for building our Intranet. Below are some of the reasons for the decision:
Very happy to share our first Video Cast where Meetu and Vikas are discussing about GAE/J platform, it’s key features and advantages both for Business as well as Developers. We would love to hear you comments and have your feedback.
For the past few months we have been working on porting a time-sheet application on Google App Engine. The application we were trying to port had Spring for Dependency Injection, Hibernate for persistence and Wicket as the web framework. Looking at the will it play in app engine Hibernate was an easy call, it is not compatible on Google App Engine, wicket works with some workaround and Spring is fully supported. We decided to go with JPA as it is easier to convert a hibernate application to JPA than to other ORM framework. We decided to continue with Spring and Wicket.
We encountered several problems during porting the application. Most of the effort went in increasing the performance. We used Key-Only-Query, memcache to increase performance. We also had issues with handling consistency of datastore operations across entity groups.
When we select framework for application development, we think in terms of the application performance and the ease of application development. When we are not porting an application but doing a new development we can choose frameworks more freely. For new application development on Google App Engine Slim3 framework seems like a good option.
Google App Engine has a notion of entity groups. It is a very important concept and decides how the Entities will be mapped in JPA for the application.
There is a limitation that an Entity cannot have more than one parent. This can be a major issue while re-defining Entity associations for Google App Engine for the application. The Entity Groups also decide the transaction strategy for an application. Simply put you can only apply transaction on a single Entity Group. Is there a platform which allow us to provide transactions across Entity groups. Slim3 does allow us to work across entity groups.
Slim3 example for transferring amount across two accounts:
Google recently added support for multi-tenancy for applications via the Namespaces API. We are in process of porting an application to Google App engine. We were thinking of making our application multi-tenant. With the new Namespaces API in our arsenal we decided to give it a try. To our surprise implementing multi-tenancy is easy, we were done with it in couple of hours!
With multi-tenancy, multiple client organizations (or “tenants”) can all run on a same hosted application. In effect due to segregation of data using a unique namespace for each client. This allows us to serve the same application to different customers, with each customer seeing their own unique copy of the application.
Let’s see how we went about implementing it. First of all we needed a Filter it sets unique name-space for each request. Here is the code listing.
It has been around two months since we started to port an existing application on Google App engine. It used Hibernate for persistence, Wicket framework for web layer and Spring as an Ioc container. Looking at the will it play on appengine we had to change Hibernate to use either JPA or JDO. We decided on JPA purely on the basis of experience we had on JPA compared to JDO. Wicket is semi compatible and we made it work by the three standard workarounds described here.
Changing the persistence layer from Hibernate to JPA was the most challenging task. Not only we had to break the relationships between entities because of the Datastore’s notion of entity groups but also due to several performance optimizations we had to do later. We will have a look at how can we use keys-only query in JPA to increase performance.
We had trouble mapping the associations in JPA for Google App engine for which we blogged about the case for unowned keys and managing multiple parent problem. Understanding of Entity Groups and transactions are extremely important for implementing persistence on Google App Engine. This becomes especially important for porting an application to Google App engine.
In our application there were certain parts we could find were expensive either in terms of CPU usage or in Datastore calls. We were able to optimize application performance using keys-only queries.
We are porting an existing Wicket application on Google App Engine. This application uses Wicket, Spring and Hibernate. If you read will it play in app engine, you will notice that Wicket is semi compatible and it does work with workarounds. Hibernate on the other hand is incompatible and Spring is fully supported. We began porting this application after changing persistence to use JPA instead of hibernate. We used workarounds for wicket described here.
This application generates Excel sheets and used Apache POI for generating Excel reports. Apache POI is not supported in Google App engine and it seems that Apache POI will not change their implementation to make it work for Google App Engine in near future.
We started looking at ways to generate Excel sheets. We decided on using java Excel API for our application. Let’s see how can we generate excel files for a Wicket application on Google App engine.
First of all add the maven dependency for JExcel in the application:
<dependency> <groupId>net.sourceforge.jexcelapi</groupId> <artifactId>jxl</artifactId> <version>2.6.10</version> </dependency>
We are in process of porting an existing Wicket application on Google App engine. This application’s charting engine used Java color classes along with Swing components to generate dynamic images. These images are then used by Wicket to display on the front-end . Unfortunately Google app engine does not support these classes. We therefore had to find an alternative to generate Charts for our application.
We decided upon using google charts in our Wicket based project. We came across this wicket google charts sub project which has the capabilities to generate charts for the application. We found out later that it used awt classes at some parts to generate charts. Looking at the code from this project we realized that it is simple to do this on our own.
Google charts require an URL format which has data along with meta information on how to display it. This well formed URL is embedded in the image tag of the rendered html content is enough to display the chart on the browser. So, in order to generate charts we had to crunch the data in the application to generate a well formed URL. Then we simply had to embed this URL in the image tag to render it on the browser. For example an URL like this:
will generate an image like this: