Entity Manager Configuration

Entity Manager is a highly generic tool that enables efficient put, get, and query operations across many entity relationships, indexes, and sharded partitions.

Entity Manager was designed to operate within the context of AWS DynamoDB, but should work equally well with any sufficiently similar NoSQL platform.

To accomplish this, Entity Manager needs to know:

• Which data types are indexable on your data platform, and how to represent those types within a compound index.

• Your entities, their properties and related types, and which properties will be generated by Entity Manager.

• The structures of your generated properties and indexes.

• Your partition sharding strategy for each entity.

This documentation takes a Typescript-first approach! All discussions & code examples will assume you are using Typescript, and we will call out Javascript-specific considerations where appropriate.

Generated Properties & Transcodes

As discussed in detail in Evolving a NoSQL Database Schema, Entity Manager indexes are supported by special generated properties.

Generated properties always have a string type. Within the Entity Manager config object, an entity generated property is specified by a simple array of its component property names. These components can be any non-generated property of the same entity, so long as that property is supported by a transcode.

Transcodes

A transcode is a pair of functions that convert a property value to and from a string type, such that the resulting strings are guaranteed to sort in the same order as the original values.

Transcodes and related mechanisms are actually defined in the @karmaniverous/entity-tools package, which is a dependency of both Entity Manager and the @karmaniverous/mock-db package used to test Entity Manager. For developer convenience these are re-exported from the @karmaniverous/entity-manager package.

In the current, inference‑first model, you author a “registry” of transcoders using a value‑first builder. Here is a minimal example that isolates the timestamp transcode:

import { isInt, isString } from 'radash';
import { defineTranscodes } from '@karmaniverous/entity-tools';

// Build a registry by providing encode/decode pairs.
// The builder enforces compile-time agreement between types.
export const timestampOnly = defineTranscodes({
  timestamp: {
    encode: (value: number) => {
      if (!isInt(value) || value < 0 || value > 9999999999999)
        throw new Error('invalid timestamp');
      return value.toString().padStart(13, '0');
    },
    decode: (value: string) => {
      if (!isString(value) || !/^[0-9]{13}$/.test(value))
        throw new Error('invalid encoded timestamp');
      return Number(value);
    },
  },
} as const);

radash is a key Entity Manager dependency, which provides a set of type-safe utility functions for working with data.

The purpose of this transcode is to convert a Unix timestamp (which is always a 13-digit integer) into a 13-character numerical string and back. The transcode’s encode and decode functions contain some type validation to catch invalid values in either direction.

timestamp is a simple case: because a Unix timestamp is an unsigned integer and always has the same length, its string representation will always sort properly.

fix6 is another default transcode that presents a more complex case: it handles a signed, fixed-point number with 6 decimal places.

import { isNumber, isString } from 'radash';
import { defineTranscodes } from '@karmaniverous/entity-tools';

export const fix6Only = defineTranscodes({
  fix6: {
    encode: (value: number) => {
      if (
        !isNumber(value) ||
        value > Number.MAX_SAFE_INTEGER / 1000000 ||
        value < Number.MIN_SAFE_INTEGER / 1000000
      )
        throw new Error('invalid fix6');

      const [prefix, abs] = value < 0 ? ['n', -value] : ['p', value];
      return `${prefix}${abs.toFixed(6).padStart(17, '0')}`;
    },
    decode: (value: string) => {
      if (!isString(value) || !/^[np][0-9]{10}\.[0-9]{6}$/.test(value))
        throw new Error('invalid encoded fix6');

      return (value.startsWith('n') ? -1 : 1) * Number(value.slice(1));
    },
  },
} as const);

fix6 uses the following techniques to meet transcode requirements:

• range checking (not required for timestamp because Unix timestamps are always positive and have a fixed length), and

• sign handling (prefixes positive numbers with p and negative numbers with n to ensure proper alpha sort), and

• zero-padding of small values (again to ensure proper alpha sort).

Entity Manager ships with a default registry:

Transcode	Description
bigint20	BigInt value with a maximum of 20 digits
boolean	boolean value
fix6	signed, fixed-point number with 6 decimal places
int	signed integer
string	string value
timestamp	Unix timestamp

Click here to review these default transcode definitions.

Tip: most applications simply import and use defaultTranscodes from @karmaniverous/entity-tools. If you add a custom transcoder, you can merge it with the defaults at runtime and pass the combined object into your config.

The TranscodeRegistry Type

A TranscodeRegistry is a type-level mapping from transcode names to the value types they encode/decode. The default registry used by Entity Tools and Entity Manager is exported as DefaultTranscodeRegistry.

When you author a registry with defineTranscodes(...), TypeScript can derive the corresponding registry type for you:

import type {
  TranscodeRegistryFrom, // derives registry type from a record of transcoders
  TranscodedType, // extracts the value type for a specific transcode name
  TranscodeName, // union of valid transcode names for a registry
} from '@karmaniverous/entity-tools';
import { defineTranscodes } from '@karmaniverous/entity-tools';

const mySpec = {
  boolean: {
    encode: (v: boolean) => (v ? 't' : 'f'),
    decode: (s: string) => s === 't',
  },
} as const;

const myTranscodes = defineTranscodes(mySpec);

type MyRegistry = TranscodeRegistryFrom<typeof mySpec>; // { boolean: boolean }
type BooleanValue = TranscodedType<MyRegistry, 'boolean'>; // boolean
type Names = TranscodeName<MyRegistry>; // 'boolean'

If you are only using Entity Manager’s default transcodes, you can skip defining a registry entirely and just set transcodes: defaultTranscodes in your config.

Custom Transcodes

Let’s say you are building a high-precision navigation application. Latitude & longitude values require three digits to the left of the decimal point, so a 64‑bit signed number leaves room for 13 digits of decimal precision. You will want to define a custom transcode for this data type—call it fix13.

import { isNumber, isString } from 'radash';
import {
  defineTranscodes,
  defaultTranscodes,
} from '@karmaniverous/entity-tools';

// Define just the custom encoder/decoder
const fix13Spec = defineTranscodes({
  fix13: {
    encode: (value: number) => {
      if (
        !isNumber(value) ||
        value > Number.MAX_SAFE_INTEGER / 10000000000000 ||
        value < Number.MIN_SAFE_INTEGER / 10000000000000
      )
        throw new Error('invalid fix13');

      const [prefix, abs] = value < 0 ? ['n', -value] : ['p', value];
      return `${prefix}${abs.toFixed(13).padStart(17, '0')}`;
    },
    decode: (value: string) => {
      if (!isString(value) || !/^[np][0-9]{3}\.[0-9]{13}$/.test(value))
        throw new Error('invalid encoded fix13');

      return (value.startsWith('n') ? -1 : 1) * Number(value.slice(1));
    },
  },
} as const);

// Merge at runtime when passing into the config (simple spread is fine)
const transcodes = { ...defaultTranscodes, ...fix13Spec };

The Entity Type (concept) and schema-first approach

Historically, the Entity type described your domain shapes as indexable records, with generated properties marked as never. You can still use it conceptually, but today we recommend a schema‑first approach using Zod:

• Define base/domain fields with Zod schemas.
• Infer your EmailItem / UserItem types from schemas.
• Use token‑aware helpers to get storage‑facing “records” when you need them (i.e., including generated/global keys).

For example, the domain shapes used in the demo:

// src/entity-manager/Email.ts
import type { EntityClientRecordByToken } from '@karmaniverous/entity-manager';
import { z } from 'zod';

import { entityClient } from '../entity-manager/entityClient';

/**
 * Email domain schema (base fields only).
 *
 * Generated/global keys are layered by Entity Manager at runtime. The inferred
 * EmailItem type represents the domain shape used by handlers. When you read
 * through the adapter, records will include generated/global keys; strip them
 * via entityManager.removeKeys('email', record) when you want pure domain
 * objects for API responses.
 */
export const emailSchema = z.object({
  created: z.number(),
  email: z.string(),
  userId: z.string(),
});

export type EmailItem = z.infer<typeof emailSchema>;

export type EmailRecord = EntityClientRecordByToken<
  typeof entityClient,
  'email'
>;

// src/entity-manager/User.ts
import type { EntityClientRecordByToken } from '@karmaniverous/entity-manager';
import { z } from 'zod';

import { entityClient } from '../entity-manager/entityClient';

/**
 * User domain schema (base fields only).
 *
 * This schema excludes all generated/global keys. Those are derived from this
 * base shape by Entity Manager according to the config. Handlers can rely on
 * domain types for input/output and only materialize keys when interacting
 * with the database.
 */
export const userSchema = z.object({
  beneficiaryId: z.string(),
  created: z.number(),
  firstName: z.string(),
  firstNameCanonical: z.string(),
  lastName: z.string(),
  lastNameCanonical: z.string(),
  phone: z.string().optional(),
  updated: z.number(),
  userId: z.string(),
});

export type UserItem = z.infer<typeof userSchema>;

export type UserRecord = EntityClientRecordByToken<typeof entityClient, 'user'>;

If you prefer the original “Entity + EntityMap” mental model, it still maps cleanly to the same configuration and generated tokens; the schema‑first approach just saves you from hand‑maintaining type definitions.

Values‑first configuration (recommended)

The EntityManager is created by passing a values‑first configuration literal (prefer as const) into createEntityManager(config, logger?). This is the simplest path with the strongest inference—no generics at the call site.

Here is a compact example:

import { createEntityManager } from '@karmaniverous/entity-manager';
import { defaultTranscodes } from '@karmaniverous/entity-tools';
import { emailSchema } from './Email';
import { userSchema } from './User';

const now = Date.now();

const config = {
  hashKey: 'hashKey' as const,
  rangeKey: 'rangeKey' as const,
  entitiesSchema: { email: emailSchema, user: userSchema } as const,
  entities: {
    email: {
      uniqueProperty: 'email',
      timestampProperty: 'created',
      shardBumps: [{ timestamp: now, charBits: 2, chars: 1 }],
    },
    user: {
      uniqueProperty: 'userId',
      timestampProperty: 'created',
      shardBumps: [{ timestamp: now, charBits: 2, chars: 1 }],
    },
  },
  generatedProperties: {
    sharded: {
      userHashKey: ['userId'],
      beneficiaryHashKey: ['beneficiaryId'],
    } as const,
    unsharded: {
      firstNameRangeKey: ['firstNameCanonical', 'lastNameCanonical', 'created'],
      lastNameRangeKey: ['lastNameCanonical', 'firstNameCanonical', 'created'],
    } as const,
  } as const,
  indexes: {
    created: { hashKey: 'hashKey', rangeKey: 'created' },
    // ...other indexes elided for brevity (see demo)
  } as const,
  propertyTranscodes: {
    created: 'timestamp',
    userId: 'string',
    email: 'string',
    // ...rest of mappings
  },
  transcodes: defaultTranscodes,
} as const;

export const entityManager = createEntityManager(config);

Type parameters & global keys (background)

For advanced use, Entity Manager exposes generic types that model hash/range keys, generated tokens, and transcodes. With the values‑first approach you don’t need to pass generics—the compiler derives them directly from your literal—but the rules remain:

• hashKey and rangeKey in the config are the canonical global key names; they must not collide with any domain property name.
• Generated and index tokens must be unique and consistent across entities.
• Transcodes must be provided for every domain property used in a generated token or ungenerated index component.

Entity configurations

The entities map defines per‑entity behavior:

• uniqueProperty — the field that uniquely identifies records (and anchors the rangeKey).
• timestampProperty — the field that anchors shard selection during queries (commonly created).
• shardBumps — the planned scale‑up schedule for shard suffix width (more below).

Query limits (defaults)

Entity Manager orchestrates cross‑shard, multi‑index queries under the hood. You can set sensible defaults per entity:

• defaultPageSize — target per‑shard page size (default 10).
• defaultLimit — target overall items per combined result (default 10).
• throttle — max shard queries in flight (default 10).

Callers can override these in their query options.

Indexes

Indexes are declared once in the config so:

• The query builder can be typed (index tokens and page keys).
• Table definition generation can include GSIs for your platform (DynamoDB in our case).
• Page keys can be dehydrated into a compact, cross‑index string (pageKeyMap) and restored later.

An index must satisfy these rules:

• hashKey: either the global hashKey or a sharded generated hash token (e.g., userHashKey).
• rangeKey: either the global rangeKey, an unsharded generated range token, or a transcodable scalar.
• projections (optional): list of projected fields for the index; do not include hash/range keys (global or index).

Generated properties

Each generated property is configured with:

• components (called elements in code) — the list of domain fields combined into the token, in order.
• atomic — if true, missing component values yield undefined rather than empty strings; useful when you need the whole composite present or not at all.
• sharded — whether this token carries a shard suffix (for hash keys).

A generated property’s component fields must appear in propertyTranscodes.

Element transcodes

Every ungenerated property that participates in a generated token or serves as a scalar index range key must have a transcode mapping. Common mappings include timestamp, string, and int.

Sharding strategy

Sharding is defined declaratively via shardBumps:

Property	Type	Description
timestamp	number	When this bump takes effect (ms since epoch).
charBits	number	Bits per shard character (1..5) — controls radix.
chars	number	Characters in the shard suffix (0..40) — controls width.

• If no bumps are specified, the effective default is a single unsharded partition (chars: 0).
• If you specify only future bumps, historical data remains unsharded; new data becomes sharded as of the bump timestamp.
• Queries that span bump windows will fan out to the relevant shard space for the window.

This staged approach lets you scale up gracefully without re‑keying historical data.

Delimiters

Generated value serialization uses three delimiters:

Property	Default	Description
generatedKeyDelimiter	' | '	Separates key-value pairs in a generated property.
generatedValueDelimiter	'#'	Separates key from value in a generated property key-value pair.
shardKeyDelimiter	'!'	Separates entity token from shard key in a sharded generated property.

An entity property used as a generated property element or an index component should never contain these delimiter characters! If unavoidable, override the delimiters to characters that never appear in your data.

Config transcodes

Set transcodes to a registry of encoders/decoders. Most applications use defaultTranscodes. If you extend, ensure the value types and keys agree; TypeScript will enforce this when you use defineTranscodes.

Runtime validation

TypeScript enforces structure at compile time, but some checks must occur at runtime (e.g., shard‑bump ordering and monotonic chars, delimiter collisions). Entity Manager validates your configuration with zod when you call createEntityManager(...). Invalid configurations throw with a helpful error.

Domain and record types (what you read/write)

With the schema‑first approach:

• EmailItem / UserItem (domain) are inferred from your Zod schemas.
• Storage “records” (with generated/global keys) can be typed via EntityClientRecordByToken<typeof entityClient, 'token'>.

Demo excerpts:

// Domain items from Zod
export type EmailItem = z.infer<typeof emailSchema>;
export type UserItem = z.infer<typeof userSchema>;

// Records (storage-facing shapes with keys)
export type EmailRecord = EntityClientRecordByToken<
  typeof entityClient,
  'email'
>;
export type UserRecord = EntityClientRecordByToken<typeof entityClient, 'user'>;

Handlers typically:

• Accept domain items as input.
• Use entityManager.addKeys('token', item) to materialize records for writes.
• Use token‑aware reads, then entityManager.removeKeys('token', records) to return domain items to callers.

Legacy note: older versions exposed an ItemMap utility that transformed a hand-authored EntityMap into storage “items”. You can still think in those terms, but the schema‑first approach saves boilerplate and tends to produce better inference.

Javascript

If you are working in Javascript, you can still use Entity Manager! Just be aware that you will not benefit from the compile-time validation that Typescript provides.

When defining custom transcodes, you will do so without reference to types:

import {
  defineTranscodes,
  defaultTranscodes,
} from '@karmaniverous/entity-tools';

const fix13Spec = defineTranscodes({
  fix13: {
    encode: (value) => {
      /* ... */
    },
    decode: (value) => {
      /* ... */
    },
  },
});

const transcodes = { ...defaultTranscodes, ...fix13Spec };

Your configuration object is authored as a values‑first literal. You then pass it to the factory:

import { createEntityManager } from '@karmaniverous/entity-manager';

const config = {
  hashKey: 'hashKey',
  rangeKey: 'rangeKey',
  // entitiesSchema may be omitted in JS; runtime validation still applies
  entities: {
    email: { uniqueProperty: 'email', timestampProperty: 'created' },
    user: { uniqueProperty: 'userId', timestampProperty: 'created' },
  },
  generatedProperties: {
    sharded: { userHashKey: ['userId'] },
    unsharded: {
      /* ... */
    },
  },
  indexes: { created: { hashKey: 'hashKey', rangeKey: 'created' } },
  propertyTranscodes: {
    created: 'timestamp',
    userId: 'string',
    email: 'string',
  },
  transcodes,
};

const entityManager = createEntityManager(config);

The factory will validate your configuration at runtime, so you can be confident that it is correct before proceeding. Having said that: if you are working in Javascript, you should really consider switching to Typescript! The benefits are enormous, and the learning curve is not as steep as you might think.