AppScale 3.5: The Datastore Viewer Explained

    Posted by Dmitrii Calzago on 3/22/18 10:52 AM

    tl;dr: Datastore Viewer introduced in AppScale 3.5 allows users to inspect the contents of the NoSQL database which is available to all AppScale applications out of the box.

    One of the key features of the new AppScale 3.5 release is the Datastore Viewer. Included as part of the Dashboard application, the Viewer is a browser-based interface to the Datastore. Until now, inspection of the Datastore contents could only be done programmatically, within the context of a specific application. This feature enables easier debugging for developers in QA deployments and easier troubleshooting or analytics for operators of production deployments.

    Admin, Developer, Users

    Google App Engine (GAE) and AppScale are multi-tenant systems, but AppScale, with its flexible deployments options, enables its users a choice of single vs multi-tenancy. From our experience, the greater majority of our users, deploy a single application per deployment: there are various reasons for it, from customization of the deployment, to easy billing, to separation of concerns and contention limitation. Below, we will reference Cloud Administrator, Developer, and User as the single user who controls the deployment, since this is the common dev/test deployment.

    Application IDs, Namespaces, Kinds

    In-depth coverage of the Datastore model is beyond the scope of this post (for that we would refer the reader to Google's documentation or the appropriate chapters in books about App Engine). Suffice it to say that the Datastore partitions the data

    • first by application ID (to keep persistent state of each app strictly separate),
    • then by namespace (an optional mechanism to facilitate multi-tenant applications),
    • and then by kind (for a collection of data items of the same nature).

    Although conceptually similar to a table in a relational database, a kind in Datastore can group items – called entities – that lack a consistent schema. Thus, even when querying entities of a particular kind, the "rows" returned are arbitrarily complex data structures, which may vary one row to the next. Each entity has an automatically generated key that is unique within the application.

    The Viewer allows navigation of the three partition options:

      1. Applications: When selecting Datastore Viewer under the Debugging/Monitoring section of the Dashboard, the user is presented with a list of deployed applications belonging to the user or, for the admin, the list of all deployed applications. If the only deployed application is the Dashboard, you will see the message "There are either no deployed projects, or you do not have ownership of any of them", even as an admin. (Although 'appscaledashboard' is an application with Datastore state, its entities are excluded from the Viewer.) 
        datastore viewer
      2. Namespaces: When an application is selected, entities for the selected kind in the default (Empty) namespace are listed, 20 at a time, sorted by key (which is decidedly not chronological). "Select a different namespace" link does the obvious, but assumes that you know the name of an existing namespace.
      3. Kinds: Persisted entity kinds are presented in a drop-down menu in alphabetical order, with entities from the first one in the list shown by default. If the application is deployed, but hasn't written to the Datastore yet, instead of showing rows of entities, the Viewer will state: "Either guestbook has no entities in the Empty namespace or entities have been added after the last time statistics were generated." It is possible to get that message if statistics for a recently created kind have not been generated yet. In this case, one can wait or trigger the generation manually by logging into the head node and invoking the 'appscale-groomer' command.


    Datastore entities consist of properties, each with a name and a value that is one of the primitive value types (integers, floats, strings, dates, binary data, etc). An entity of a given kind is not required to have the same set of properties as another entity of the same kind. When two entities have a property with the same name, the type of their values does not have to be the same. Furthermore, an entity may have multiple values for the same property.

    The Viewer will show each property of an entity in its own column, named after the property. This results in a table with at most 20 entity rows and as many columns as are needed to show all property names of the shown entities. Multiple values for a property will be shown using Python's list notation: [ val1, val2, ... ]. The first four columns are always the same:

        • Key - unique identifier for the entity across the deployment
        • Write Ops - an estimate of low-level database writes that were required during latest update to the entity (including the indexes)
        • ID - a numerical ID for the entity, generated by the system when not set by the user
        • Key Name - a string ID for the entity, when set by the user

    The remaining columns are user-defined and are sortable (by clicking on the column’s header). Clicking on the key link of an entity shows all properties of the entity vertically. This view reveals details that are truncated in the table view, such as long strings or list members.

    datastore viewer keys

    When one or more entities in the table are selected, the Delete button, located at the bottom of the list, becomes clickable. This allows removing entities from the Datastore, one screen at a time.

    Next to it, the Flush Memcache button triggers requests to all Memcache servers to dump any entity data that may not be consistent with the datastore. This can help when query results temporarily don't reflect the latest writes from the application.


    In addition to listing all Datastore entries sorted by one of the properties and paging through them (under 'Query by kind' tab), one can utilize query language called GQL to find specific entities or narrow down the list (under 'Query by GQL' tab).

    Although similar to SQL in appearance, GQL is constrained to a narrow set of use cases: Only SELECT queries over a single property are supported. (Composite queries, involving conditions for more than one property are not supported in the Datastore Viewer yet.) Queries may return either values for a single property or entire entities. No updates to entities can be done via GQL.

    Example A: Getting all entities based on a property value match:

    SELECT * from Person
    WHERE name = 'Roger Mohler'
    datastore viewer example a

    Example B: Getting all values of a particular property:

    SELECT places FROM Person
    ORDER by places ASC
    datastore viewer example b

    Example C: Getting all entities within a range of values of a particular property:

    SELECT * from Account
    WHERE timestamp > DATETIME('2018-03-20 06:30:00')
    AND timestamp < DATETIME('2018-03-20 06:30:10')
    ORDER BY timestamp DESC
    datastore viewer example c

     AppScale's Datastore Viewer is based on the code of the Entity Viewer that's bundled into Google's App Engine SDK. It therefore has some of the same limitations.

    Try AppScale 3.5 and tell us if you've found the Datastore Viewer useful for debugging and how we can make it better!


    Topics: AppScale News, AppScale Release

    Subscribe to Email Updates

    Most Popular

    Recent Posts