Using Google Datastore with akka-persistence

4 min readMar 6, 2020

abstract database visualization for akka-peristence-gcp-datastore

akka-persistence-gcp-datastore is a library we created in 2020 to use the google datastore (firestore with datastore mode) in combination with akka-persistence for event-sourcing and cqrs, because we mostly use the google cloud platform and not amazon with dynamodb.

The plugin supports the following functionality:

serialization of events and snapshots with play-json
peristence-query api
custom serialization

Scala 2.12 & 2.13, Java 8 & Java 11, akka 2.6.X are currently supported.

We also adapted the official Lightbend CQRS-Sample which implements a CQRS-ES design that will side-effect in the read model on selected events persisted to google cloud-datastore emulator by the write model.

We have created two new german blogposts about event-sourcing and cqrs, but the very exciting part of this library is not akka-persistence but the managed google cloud no-sql database.

Google Firestore with datastore mode

For our plugin we used the newest no-sql database from google cloud platform. It is a rebranding from google and we must to choose the datastore mode and not the native mode.

Based on the Google Megastore concept, firestore combines the best of two worlds. It has the scalability of no-sql databases and the easiness of a classiv rdbms. firestore in datastore mode is strongly consistent and high available (99,95%) which is very important for our akka-persistent plugin.

The datastore mode makes the database backwards-compatible with the “old” Google Cloud Datastore. Firestore in Datastore mode uses Datastore system behaviour but accesses Firestores storage layer, removing the following old limitation which the google datastore had:

Eventual consistency, all Datastore queries become strongly consistent.
Queries in transactions are no longer required to be ancestor queries.
Transactions are no longer limited to 25 entity groups.
Writes to an entity group are no longer limited to 1 per second.

Datastore mode disables Firestore features that are not compatible with Datastore:

The project will accept Datastore API requests and deny Firestore API requests.
The project will use Datastore indexes instead of Firestore indexes.
You can use Datastore client libraries with this project but not Firestore client libraries.
Firestore real-time capabilities will not be available.
In the Cloud Console, the database will use the Datastore viewer.

(source: https://cloud.google.com/firestore/docs/firestore-or-datastore)

Usage of akka-persistence-gcp-datastore

First you need to add the matching dependency to your scala project. You’ll find the matching version in the github repo

libraryDependencies += "de.innfactory" %% "akka-persistence-gcp-datastore" % "X.Y.Z"

After that you have to create a index.yml file with content bolow in the project that will use this plugin:

indexes:
  - kind: journal
    properties:
      - name: persistenceId
      - name: sequenceNr
        direction: desc  - kind: snapshot
    properties:
      - name: persistenceId
      - name: timestamp
        direction: desc  - kind: snapshot
    properties:
      - name: persistenceId
      - name: timestamp  - kind: journal
    properties:
    - name: tagsKey
    - name: timestamp  - kind: journal
    properties:
    - name: persistenceId
    - name: sequenceNr

and then deploy it to google cloud to tell gcp datastore to build indexes for the plugin based on the yaml file

gcloud app deploy index.yaml

Create a service account for read and write to datastore. Download the json and add it to the project

src/main/resources/datastore.json

Add the following to your application.conf for a basic configuration, more values are referenced in the reference.conf:

akka {
  # use google cloud datastore as journal and snapshot store
  persistence {

    journal {
      plugin = "gcp-datastore-journal"
      auto-start-journals = ["gcp-datastore-journal"]
    }

    snapshot-store {
      plugin = "gcp-datastore-snapshot"
      auto-start-snapshot-stores = ["gcp-datastore-snapshot"]
    }
  }
}

Persistence query API

The plugin supports the Persistence query APi, mostly used in CQRS applications to transform/migrate the events from the write side to the read side.

The ReadJournal is retrieved via the akka.persistence.datastore.journal.read.DatastoreScaladslReadJournal and akka.persistence.datastore.journal.read.DatastoreJavadslReadJournal. There is also a DatastoreReadJournalProvider.

import akka.persistence.datastore.journal.read.DatastoreScaladslReadJournal
import akka.persistence.query.{ EventEnvelope, PersistenceQuery }val system = ??? //ActorSystem akka-classic or akka-typed then system.toClassic is needed. see the example.
val readJournal =
    PersistenceQuery(system).readJournalFor[DatastoreScaladslReadJournal]("gcp-datastore-query")

Supported Queries

All queries are live streams and they are not completed when they reaches the end of the currently stored events, but continue to push new events when new events are persisted.

eventsByTag

eventsByTags is used for retrieving events that were marked with a given tag.

eventsByPersistenceId

eventsByPersistenceId is used for retrieving events for a specific PersistentActor identified by its persistenceId

Testing

To test this plugin

(Source: https://cloud.google.com/datastore/docs/tools/datastore-emulator)

gcloud components install cloud-datastore-emulator
gcloud beta emulators datastore start --no-store-on-disk --consistency=1.0
Set Env Variable DATASTORE_TESTHOST=http://<host>:<port> of datastore emulator
Execute sbt run
Before executing test reset datastore data: curl -X POST http://<host>:<port>/reset

There is also a shell script in the circleci folder which runs the test.

Credits

innFactory GmbH is a lightbend partner from germany. We are experts for Apps, BigData & Cloud Computing. If you need help with your next cloud project, feel free to ask for our support.