Creating portable owned JDO relations on Google App Engine

Background

We have an application that runs on a traditional J2EE environment and Google App Engine, with as identical code base as possible. We use JDO for persistance. In Google App Engine you need to use Key objects as the primary key for owned relations in an entity group. Obviously we can’t use the Key class as our primary key if we want to be portable, since it’s not available in a non Google App Engine class path.

Google App Engine supports encoded strings as keys, however this breaks automatic identity value generation in most RDBMS. To work around this you would have to handle parent key creation on Google App Engine and have another  key factory for your RDBMS Using custom made factory for all keys is not good enough, leaving us with no other option than to use numeric keys on both platforms and loosing the entity group.

Another big problem with having Long keys is that in datanucleus-appengine-1.x you couldn’t specify unowned relations, so not only did we have problems with transactions, we couldn’t even make real relationships between the entities, we had to store the keys and manually resolve them in our DAOs.

This works fine as long as you don’t need to guarantee data consistency across entity groups in transactions. If you’re thinking that a put followed by a get will give you the same data, you are bound for getting some nasty “random bugs”, especially when you are using the High Replication Datastore.

“In a High Replication datastore the transaction is typically completely applied within a few hundred milliseconds after the commit() returns. However, even if it is not completely applied, subsequent reads, writes, and ancestor queries will always reflect the results of the commit() because these operations apply any outstanding modifications before executing. However, queries that span multiple entity groups cannot determine if there are any outstanding modifications before executing and may return stale or partially applied results.”, Transaction Isolation in App Engine, February 29, 2012.

Solution

After a lot of experimenting and DataNucleus code analyzation, I discovered that you can specify a jdbcType in the @Persistent annotation. The datanucleus-appengine plugin ignores jdbcType, so it doesn’t effect the Google App Engine environment.

@PersistenceCapable
public class Person {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Extension(vendorName = "datanucleus",
               key = "gae.encoded-pk",
               value = "true")

    // To make this work, you need to use our plugin
    @Column(jdbcType = "BIGINT")
    private String id;

    @Persistent(mappedBy = "person")
    private Collection<Address> addresses =
               new HashSet<Address>();
}
@PersistenceCapable
public class Address {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Extension(vendorName = "datanucleus",
               key = "gae.encoded-pk",
               value = "true")

    // To make this work, you need to use our plugin
    @Column(jdbcType = "BIGINT")
    private String id;

    @Persistent
    private Person person;
}

So I tried mapping our String to a BIGINT (Long), to allow valueStrategy=IdGeneratorStrategy.IDENTITY. This was in theory a good idea, but didn’t work out of the box. It turned out that DataNucleus don’t support mapping Strings to BIGINT by default, so I had to create my own mapper plugin for this.

I’m not going to explain all the findings I made about creating DataNucleus plugins, if you are interested in that, please see the DataNucleus documentation. Instead I will show you where you can find our plugin, how to enable and use it.

Using the plugin

Just add the datanucleus-stringtobigintmapper to your class path, and it will be activated. If you are using Maven, here is the dependency declaration for you.

<dependency>
    <groupId>se.weightpoint.datanucleus</groupId>
    <artifactId>datanucleus-stringtobigintmapper</artifactId>
    <version>1.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-rdbms</artifactId>
        </exclusion>
    </exclusions>
</dependency>

You can find the sources at our GitHub project or you can download the 1.0 release here.

We are using DataNucleus 3.x and the datanucleus-appengine-2.0 plugin, if you haven’t upgraded already, please see this page for instructions.

5 thoughts on “Creating portable owned JDO relations on Google App Engine”

  1. Nice post.

    Note that rather than just putting all as annotations you could have
    package-gae.orm
    package-mysql.orm
    and separate the GAE/RDBMS specific bits that way

  2. You know if is posible to do the same thing with :
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Column(jdbcType = “BIGINT”)
    private Key id;

    1. You could probably with some work create a plugin that could do that mapping, but i doubt it would really be useful, since you would depend on appengine classes. I see no reason not to just use Strings, in rdbms it becomes an bigint, and in appengine it becomes a Key, so we get a good native key for both platforms.

Leave a Reply

Your email address will not be published. Required fields are marked *


9 × three =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>