Your project’s data needs are complex. Entity Manager simplifies the problem by encapsulating a provider-agnostic, highly opinionated approach to the single-table design pattern.
At the end of the day, though, you have to implement a specific data model against a specific database platform. This guide presents a step-by-step Typescript implementation of a realistic data model against DynamoDB, with the help of Entity Manager.
The entity-manager-demo
repository contains the full implementation documented below.
This page is under construction! The Typescript refactor is nearly complete, and I’m busy building the demo & syncing up this documentation. Please check back soon for updates and drop me a note with any questions or ideas!.
An Overview
As the basis of this demonstration we will use the same data model, table design, and index structure we worked up in Evolving a NoSQL DB Schema. If you haven’t read this article yet, I recommend you do so before proceeding as it will help you understand why we settled on the design we chose.
If you just want to review the resulting design, see the Recap section at the end of that article.
While all Entity Manager entity records are technically sharded, by default each record’s shard key is an empty string, resulting in effectively unsharded data. To demonstrate Entity Manager’s ability to scale, we will configure a sharding schedule for each entity that will allow us to test both unsharded and sharded scenarios.
The details of an API implementation are beyond the scope of this demo, but we will develop the handler functions an API would call to perform the following operations against the above data model in DynamoDB using Entity Manager:
- Email entity:
- Create a new email record.
- Delete an email record.
- Retrieve a list of email records by
userId
.
- User entity:
- Create a new user record.
- Retrieve a user record by
userId
. - Update a user record.
- Delete a user record and associated email records.
- Retrieve a list of user records by various search criteria, including a multi-index match against both
firstName
andlastName
.
We’ll also demonstrate how to leverage your Entity Manager config to generate a definition for your DynamoDB table, and how to use the entity-client-dynamodb
to create this table and efficiently perform other table-level database operations in DynamoDB.
We’ll write mocha
/chai
unit tests to exercise all this functionality against DynamoDB using the DynamoDB Local emulator running in a Docker container. This will allow you to run the demo on your local machine without incurring any AWS costs (though these would be trivial) and without requiring an active connection to AWS.
You should be abe to pull the repository, install dependencies, and run all unit tests successfully within just a few minutes. So let’s get started!
For convenience, this repository uses my Typescript NPM Package Template, only I’ve stripped out the CLI & Rollup build and have disabled NPM publishing. So what remains is a pure, semantic-versioned Typescript “package” with a bunch of unit tests: perfect for a demo that can evolve over time, useless for anything else.
DynamoDB Local Integration
There are a couple of different ways to run DynamoDB locally. See the DynamoDB Local documentation for details.
To keep things simple, this demo executes its tests against DynamoDB running in a Docker image. The only requirement is that Docker Desktop be installed & running. When you execute a test, the test suite will download the Docker image if it isn’t already present, start the container, run the tests, and then stop & delete the container.
Setting up Docker Desktop is beyond the scope of this guide, but if you’re on a Windows machine and have Chocolatey, it’s easy: run choco install docker-desktop
from an admin prompt. You’ll want to restart your machine once installation completes.
If your first test execution seems to hang, check the output panel! The DynamoDB Local Docker image takes a couple of minutes to download & install, but you’ll only have to do that once. If your tests fail outright, make sure Docker Desktop is actually running!
Setting Up The Demo
After you have Docker Desktop installed & running, follow these steps:
-
Clone the
entity-manager-demo
repository to your local machine. -
Install dependencies by running
npm install
from the repository root. -
Optionally, install Mocha Test Explorer to make it a bit easier to follow along with the examples below.
That’s it! Check your work by running npm test
from the repository root. If all the tests pass, you’re ready to start exploring the code!
If you run into any trouble, please start a discussion and I’ll help!
Logger
All packages in the Entity Manager ecosystem perform extensive debug logging by default. This is often very useful when troubleshooting projects that leverage these packages, but it can also inject a lot of noise into the console when you’re trying to focus on your own code with its own debug logging.
Entity Manager packages also support an injected logger object. To address the noise issue, we will…
-
Alias
console
tologger
and use it everywhere we want logging in our demo. The demo will work just as well if you replaceconsole
withwinston
or some other logger of choice. -
Use the
controlledProxy
function to proxy thelogger
object and disable thedebug
endpoint. When we inject the resultingerrorLogger
object into ourEntityManager
andEntity Client
instances, all of their internal debug logging will be suppressed.
Visit logger.ts
to see this code in context.
import { controlledProxy } from "@karmaniverous/controlled-proxy";
// Use the console logger. This could easily be replaced with a
// custom logger like winston.
export const logger = console;
// Proxy the logger & disable debug logging.
export const errorLogger = controlledProxy({
defaultControls: { debug: false },
target: logger,
});
EntityManager
Configuration
Entity Manager configuration is a complex topic! Our purpose here is to focus specifically on our demonstration scenario. Click here for a deeper dive into all aspects of Entity Manager configuration.
Entity Manager is a Typescript-first tool! If you are writing Javascript, you can skip the type-related parts of this guide and your config will still be validated for you at run time. You just won’t get the compile-time type checking that Typescript provides.
The EntityManager
class configuration object is defined in the Config
type. Internally this type is very complex, in order to support all kinds of compile-time validations. Once the config is parsed, though, it takes on the ParsedConfig
type, which has the same form and is quite a bit easier to read!
Creating an EntityManager
instance is really about creating a valid Config
object.
The Config
type has four type parameters. Only the first one is required, and for this demo we will use defaults for the other three. Here they are:
-
M extends
EntityMap
- This is the most important parameter and will be different in every implementation. We’ll address it below. -
HashKey extends string
- This is the name of the generated hash key property that will be shared across all entities in the configuration. The default value is'hashKey'
, and we will use that here. -
RangeKey extends string
- This is the name of the generated range key property that will be shared across all entities in the configuration. The default value is'rangeKey'
, and we will use that here. -
T extends
TranscodeMap
- Relates the name of a transcode to the type of the value being transcoded. This parameter defaults to an extensible DefaultTranscodeMap type that will serve our purposes here.
The EntityManager
constructor also takes a logger
argument. To minimize noise in the demo console, the injected errorLogger
proxies console
to disable debug logging in the EntityManager
instance while keeping it enabled elsewhere. See my controlled-proxy
repo for more info!
Let’s compose our configuration’s EntityMap
.
MyEntityMap
Type
An EntityMap
is just a map of Typescript interfaces that define the structure of each entity in the configuration. The keys of the map are the entity names, and the values are the entity interfaces.
As a special convention, within each interface we identify generated properties (the ones marked with a ⚙️ in our table design) with a never
type. This is a signal to the Config
type that these properties require special support & configuration. See the EmailEntity
and UserEntity
interfaces below for examples.
While we are at it, we will also construct and export the Email
and User
types. These are the same as EmailEntity
and UserEntity
, respectively, but we’ve stripped out all properties with the never
type. These are useful for the rest of our application code, which doesn’t need to know about properties that are specific to data operations.
Here is the definition of MyEntityMap
from entityManager.ts
:
import type { EntityMap } from "@karmaniverous/entity-manager";
import type {
Entity,
PropertiesNotOfType,
} from "@karmaniverous/entity-tools";
// Email entity interface. never types indicate generated properties.
interface EmailEntity extends Entity {
created: number;
email: string;
userHashKey: never; // generated
userId: string;
}
// Email type for use outside data operations.
export type Email = Pick<
EmailEntity,
PropertiesNotOfType<EmailEntity, never>
>;
// User entity interface. never types indicate generated properties.
interface UserEntity extends Entity {
beneficiaryId: string;
created: number;
firstName: string;
firstNameCanonical: string;
firstNameRangeKey: never; // generated
lastName: string;
lastNameCanonical: string;
lastNameRangeKey: never; // generated
phone?: string;
updated: number;
userBeneficiaryHashKey: never; // generated
userHashKey: never; // generated
userId: string;
}
// Email type for use outside data operations.
export type User = Pick<
UserEntity,
PropertiesNotOfType<UserEntity, never>
>;
// Entity interfaces combined into EntityMap.
interface MyEntityMap extends EntityMap {
email: EmailEntity;
user: UserEntity;
}
config
Object
The config
object has a lot of moving parts, so it helps to come at the problem from a specific direction.
The properties of the demo config object defined below are arranged accordingly and explained in the comments. For the mopart, only the Email entity config is commented, since the User config follows the same pattern.
Here are some important references from the comments:
-
The demo table design.
-
The demo index design.
-
Transcodes perform and reverse the conversion of a value into a string, often at a fixed width, for inclusion in a generated property while preserving its sorting characteristics. The
defaultTranscodes
object is used here since atranscodes
object is not defined in the config. Available transcodes are:bigint
- Renders abigint
as a variable-width string.bigint20
- Pads abigint
to a fixed width of 20 characters.boolean
- Renders as't'
or'f'
.fix6
- Pads a number to a fixed width with 6 decimal digits.int
- Pads a signed integer to a fixed width.number
- Renders anumber
as a variable-width string.string
- Variable-length pass-through.timestamp
- Pads a UNIX ms timestamp to a fixed width.
Visit entityManager.ts
to see this code in context.
import {
type Config,
EntityManager,
} from "@karmaniverous/entity-manager";
import { errorLogger } from "./logger";
// Current timestamp will act as break point for sharding schedule.
const now = Date.now();
// Config object for EntityManager.
// Using default values for HashKey, RangeKey, and TranscodeMap
// type params.
const config: Config<MyEntityMap> = {
// Common hash & range key properties for all entities. Must
// exactly match HashKey & RangeKey type params.
hashKey: "hashKey",
rangeKey: "rangeKey",
// Entity-specific configs. Keys must exactly match those of
// MyEntityMap.
entities: {
// Email entity config.
email: {
// Source property for the Email entity's hash key.
uniqueProperty: "email",
// Source property for timestamp used to calculate Email
// shard key.
timestampProperty: "created",
// Email entity's shard bump schedule. Records created before
// now are unsharded (1 possible shard key). Records created
// after now have a 1-char, 2-bit shard key (4 possible shard
// keys).
shardBumps: [{ timestamp: now, charBits: 2, chars: 1 }],
// Email entity generated properties. These keys must match
// the ones with never types in the Email interface defined
// above, and are marked with a ⚙️ in the table design.
generated: {
userHashKey: {
// When true, if any element is undefined or null, the
// generated property will be undefined. When false,
// undefined or null elements will be rendered as an
// empty string.
atomic: true,
// Elements of the generated property. These MUST be
// ungenerated properties (i.e. not marked with never
// in the Email interface) and MUST be included in the
// entityTranscodes object below. Elements are applied
// in order.
elements: ["userId"],
// When this value is true, the generated property will
// be sharded.
sharded: true,
},
},
// Indexes for the Email entity as specified in the index
// design.
indexes: {
// An index hashKey must be either the global hash key or a
// sharded generated property. Its rangeKey must be either
// the global range key, an scalar ungenerated property, or
// an unsharded generated property. Any ungenerated
// properties used MUST be included in the entityTranscodes
// object below.
created: { hashKey: "hashKey", rangeKey: "created" },
userCreated: {
hashKey: "userHashKey",
rangeKey: "created",
},
},
// Transcodes for ungenerated properties used as generated
// property elements or index components. Transcode values
// must be valid config transcodes object keys. Since this
// config does not define a transcodes object it uses
// defaultTranscodes exported by @karmaniverous/entity-tools.
elementTranscodes: {
created: "timestamp",
userId: "string",
},
},
// User entity config.
user: {
uniqueProperty: "userId",
timestampProperty: "created",
shardBumps: [{ timestamp: now, charBits: 2, chars: 1 }],
generated: {
firstNameRangeKey: {
atomic: true,
elements: [
"firstNameCanonical",
"lastNameCanonical",
"created",
],
},
lastNameRangeKey: {
atomic: true,
elements: [
"lastNameCanonical",
"firstNameCanonical",
"created",
],
},
userBeneficiaryHashKey: {
atomic: true,
elements: ["beneficiaryId"],
sharded: true,
},
userHashKey: {
atomic: true,
elements: ["userId"],
sharded: true,
},
},
indexes: {
created: ["hashKey", "rangeKey", "created"],
firstName: ["hashKey", "rangeKey", "firstNameRangeKey"],
lastName: ["hashKey", "rangeKey", "lastNameRangeKey"],
phone: ["hashKey", "rangeKey", "phone"],
updated: ["hashKey", "rangeKey", "updated"],
userBeneficiaryCreated: [
"hashKey",
"rangeKey",
"userBeneficiaryHashKey",
"created",
],
userBeneficiaryFirstName: [
"hashKey",
"rangeKey",
"userBeneficiaryHashKey",
"firstNameRangeKey",
],
userBeneficiaryLastName: [
"hashKey",
"rangeKey",
"userBeneficiaryHashKey",
"lastNameRangeKey",
],
userBeneficiaryPhone: [
"hashKey",
"rangeKey",
"userBeneficiaryHashKey",
"phone",
],
userBeneficiaryUpdated: [
"hashKey",
"rangeKey",
"userBeneficiaryHashKey",
"updated",
],
},
elementTranscodes: {
beneficiaryId: "string",
created: "timestamp",
firstNameCanonical: "string",
lastNameCanonical: "string",
phone: "string",
updated: "timestamp",
userId: "string",
},
},
},
};
// Configure & export EntityManager instance.
export const entityManager = new EntityManager(config, errorLogger);
Item Types
The Email
and User
interfaces defined above are not useful for database operations because…
-
they don’t include the
hashKey
andrangeKey
properties that we specified in our Entity Managerconfig
object, and -
their generated properties all have a
never
type.
The ItemMap
type takes our EntityMap
type as a parameter and returns a map of the correct item types. Let’s do this and export the result for use in our endpoint handlers.
Visit entityManager.ts
to see this code in context.
import type { ItemMap } from "@karmaniverous/entity-manager";
// Construct ItemMap type from MyEntityMap.
type MyItemMap = ItemMap<MyEntityMap>;
// Export EmailItem & UserItem types for use in other modules.
export type EmailItem = MyItemMap["email"];
export type UserItem = MyItemMap["user"];
EntityClient
Configuration
The EntityClient
class combines the DynamoDBClient
and DynamoDBDocument
classes from the AWS SDK with some higher-level functions to provide a simplified interface over key interactions with a DynamoDB database, as well as improved handling of batch operations.
For example:
-
The
createTable
method leverages its internalDynamoDBClient
instance to create a table, then callswaitUntilTableExists
to block further execution until the new table is actually available for data operations. -
The
putItems
method breaks an array of entity items into multiple batches, then leverages theDynamoDBDocument
BatchWriteCommand
to perform throttled batched writes to the database in parallel.
The class also provides direct access to the underlying DynamoDBClient
and DynamoDBDocument
instances, so any operations not supported by enhanced EntityClient
methods are also available. This demo will provide examples of both modes of operation.
See the entity-client-dynamodb
repository for more info.
As with the EntityManager
instance configuration above, we have injected an errorLogger
object that proxies console
to disable debug logging in the EntityClient
instance while keeping it enabled elsewhere. See my controlled-proxy
repo for more info!
Otherwise, the EntityClientOptions
type is only a slight extension of the familiar DynamoDBClientConfig
type, so configuring the EntityClient
instance for this demo is straightforward.
Visit entityClient.ts
to see this code in context.
import { EntityClient } from "@karmaniverous/entity-client-dynamodb";
import { errorLogger } from "./logger";
export const entityClient = new EntityClient({
credentials: {
accessKeyId: "fakeAccessKeyId",
secretAccessKey: "fakeSecretAccessKey",
},
endpoint: "http://localhost:8000",
logger: errorLogger,
region: "local",
});
Endpoint Handlers
Every handler function defined below follows the pattern illustrated in the diagram above. Here’s a breakdown of the key elements in the diagram:
-
User
is the type used in your application to represent a User object, not including all the generated properties used by Entity Manager to support database operations. Depending on handler requirements,Email
is also available. -
params
are the parameters received by thehandler
function. In most cases this object’s type will be some variant on theUser
orEmail
type. -
UserItem
is the complete User type stored in the database, including all generated properties specified in theEntityManager
configuration. Depending on handler requirements,EmailItem
is also available. -
handler
is the function that handles the actual data request. As we will see below, with Entity Manager in place this code can be very compact and efficient! Internally, thehandler
function will use theUserItem
andEmailItem
types to interact with the database. -
entityManager
is an instance of theEntityManager
class, which is initialized with a configuration reflecting the design summarized here. This object gives our handler the ability to add and remove generated properties from aUserItem
orEmailItem
and to perform query and CRUD operations on these entities in the database. -
entityClient
is an instance of the DynamoDB-specificEntityClient
class, initialized to connect with the User Service table in DynamoDB. In principle we could accomplish everything in this demo usingDynamoDBClient
andDynamoDBDocument
from the native DynamoDB SDK, but theEntityClient
class simplifies database interactions and eliminates a lot of noise that would otherwise interfere with the clarity of this demo. We’ll also use a utility function from the same library to generate our DynamoDB table definition from our Entity Manager config! -
shardQueryMapBuilder
is aShardQueryMapBuilder
instance declared internally by handler functions (like search endpoints) that require the ability to perform cross-shard, multi-index database queries. Each instance is handler-specific, so this object is not shared between handlers. -
logger
is simply an alias ofconsole
. Feel free to replace it with your favorite logger!
Email Entity
Create Email
TODO
Read Email
TODO
Delete Email
TODO
Search Emails
TODO
User Entity
Create User
TODO
Read User
TODO
Update User
TODO
Delete User
TODO
Search Users
TODO