Java 21 • Project-scoped data platform

Ship data products
at warp speed.

Build and operate data services from one Java engine. Dobase combines stable append-only NDJSON storage, an opt-in typed binary collection preview, authenticated REST APIs, generation-verified secondary indexes, durable scheduled jobs, organization and project isolation, managed API keys, object storage, sandboxed functions, schema-driven field encryption, durable post-commit triggers, typed realtime events, signed webhooks, deterministic firewall policy, verified backups, controlled regional replication, and a built-in management console with an IDEL QL workbench.

dobase-api:8888
POST /generate-token
✓ authenticated as ADMIN
POST /db/perfdb/events/insert
✓ indexed + persisted
GET  /db/perfdb/events/find-by-id/evt-1001
✓ document found
POST /db/perfdb/events/compact
✓ background compaction queued

A focused engine for document workloads

The core data path is built around append-only persistence, bounded reads, durable ID lookups, and explicit operational limits.

Append-Only Writes

Inserts, updates, and tombstones append to NDJSON files with a per-collection write lock and configurable batched fsync.

Cursor Pagination

File-offset cursors avoid deep offset scans and return bounded pages with a next cursor and total document count.

Auto-Compaction

Tombstone ratio and file-growth thresholds queue background compaction, with task status exposed through the REST API.

Durable Indexes

ID and secondary-index snapshots are persisted atomically, tied to the source-data generation, and rebuilt after recovery whenever they are missing, corrupt, or stale.

Backpressure & Limits

Configurable read/write permits, page-size validation, queue depth limits, and bounded function execution protect the host.

Write-Time Schema Enforcement

Typed collection metadata is enforced for inserts, updates, atomic batches, and physical shards. Invalid JSON, mismatched IDs, reserved fields, and incompatible values fail before they enter the durable log.

Typed Binary Collections Preview

New schema-required collections can opt into checksum-framed, append-only binary records with stable field IDs, typed values, field encryption, crash-tail recovery, secondary indexes, and the same REST and IDEL QL read surface. Existing collections remain on the stable NDJSON engine.

Storage Telemetry

Inspect active documents, tombstones, file growth, compaction thresholds, JVM metrics, readiness, runtime configuration, and Prometheus metrics through an ADMIN-protected actuator endpoint.

Controlled Multi-Region

Run one writable primary with read replicas across servers and datacenters using an authenticated, fsync-backed ordered outbox, automatic retry, replay controls, lag status, and replica write protection. Automatic failover remains gated on recurring fencing and restore drills.

Indexed Queries & Aggregates

Case-consistent equality indexes accelerate document queries, while persisted materialized-aggregate definitions rebuild safely and stay current across recovery, single writes, atomic batches, collection recreation, and every physical shard.

Schema-Driven Field Encryption

Declare typed collection fields and mark only sensitive fields encrypted. AES-256-GCM envelope encryption protects WAL, data files, and backups while ordinary fields remain queryable.

Schedule work without treating memory as storage

Queue database mutations and sandboxed function invocations for now or later. Dobase persists the job before returning 202 Accepted, then executes it with leases and at-least-once delivery.

Scheduled Execution

Supply an ISO-8601 runAt value for delayed inserts, updates, deletes, or registered function invocations.

Durable by Default

The single-node backend appends and fsyncs every state change. Production can use Redis AOF with optional Redis 7.2+ WAITAOF confirmation.

Retries & Dead Letter

Worker leases recover abandoned jobs, exponential backoff handles transient failures, and exhausted work stays inspectable and can be retried manually.

Idempotent Enqueue

Project- and queue-scoped idempotency keys make client retries return the original job instead of scheduling duplicates.

Completion Webhooks

Attach an optional HTTPS callback to a job. Delivery is retried, can be HMAC-SHA256 signed, and rejects redirects and private network targets by default.

Realtime Console Updates

Every queue state transition publishes a project-scoped queues/* event, so all console versions show live completion, retry, webhook, and dead-letter notifications.

Durable Event Triggers

Match database, collection, document, bucket, and object events, then enqueue a project-scoped function or signed HTTPS webhook after commit with retries, idempotency, and dead letters.

Operate Dobase without a separate dashboard

The built-in console uses the same authenticated APIs available to your applications and keeps destructive actions explicit.

Management Console

Search, sort, and paginate databases, collections, documents, buckets, files, functions, users, organizations, and projects; preview files, define typed encrypted schemas, run IDEL QL, inspect events, manage triggers, rotate keys, and manage webhooks from one interface.

Health & JVM Metrics

Live health, readiness, counters, JVM memory, uptime, storage statistics, and realtime connection status.

Realtime Event Inspector

Connect with a one-time socket ticket, subscribe by resource and event type, and inspect structured payloads without polling.

Project API Keys

Issue named, role-scoped credentials for exactly one project, then search, update, disable, rotate, or permanently revoke them. Raw secrets are shown once and stored only as hashes.

Safe Defaults

Authentication is enabled by default, registration is disabled, and production requires stable JWT and encryption keys.

Verified Backup & Restore

Create scheduled or on-demand snapshots with SHA-256 manifests, configurable retention, isolated recovery drills, and staged restart-safe restore with automatic rollback protection.

Cluster & Regions

Register provisioned HTTPS regions, enable, disable, drain, or resume peers, and inspect node identity, journal capacity, health, lag, retries, and bounded replay operations.

Encryption Key Operations

Inspect master-key health and wrapped project-key versions, then rotate a project DEK without deleting the old versions required for reads, restore, and recovery drills.

IDEL QL Workbench

Execute bounded read-only document, file, function, and metadata queries from a dark code editor with live resource-aware autocomplete and copyable JSON results. Canonical OpenExecution syntax stays concise: show.collections.analytics sort="name" order="asc" limit=25.

Firewall Management

Search, simulate, publish, and audit endpoint policies covering IP networks, countries, identities, roles, API keys, organizations, projects, and Redis-backed rate limits.

Data, files, and code in one runtime

Dobase pairs its document API with project-scoped object storage and bounded server-side function execution.

Bucket & Object CRUD

Create and rename buckets, list objects, upload files, preview metadata, download content, and explicitly delete objects.

Encrypted Object Storage

Object bytes are encrypted at rest. Production requires a stable external encryption key and Redis-backed metadata.

Object Versioning

Store version history, list prior versions, roll an object back, or delete a specific version through GraphQL operations.

Multipart Uploads

Stream multipart files or resume encrypted chunk uploads. The object ceiling is operator-configurable with a 1 GiB default.

Sandboxed Functions

Publish immutable TypeScript or JavaScript function versions, activate a known-good revision, or run bounded read-only IDEL QL. TypeScript is compiled before activation and executable code runs inside a GraalJS sandbox with time and output limits.

REST, GraphQL & Realtime

REST drives database and operator workflows, GraphQL exposes a typed realtime subscription, and the WebSocket API supports multiplexed database, queue, storage, function, and system listeners.

Project Envelope Keys

Every project receives a random data key wrapped by an external versioned master key. Fresh nonces and location-bound authentication prevent ciphertext substitution across tenants, databases, documents, or fields.

Secure by default, scoped by project

Authentication protects the API by default, while roles and project identities separate operator actions from application access.

JWT Authentication

Signed bearer tokens carry roles, expire automatically, and use a stable secret supplied by production configuration. Logout, password changes, role changes, and account deletion rotate a persisted session generation so existing tokens stop working immediately.

Role-Based Access

Administrative routes require the ADMIN role while authenticated clients retain access to permitted data operations.

Project Isolation

Organizations group globally unique projects. Every API key is permanently bound to one project; JWT clients must send an explicit project context. Database ownership is enforced, and scoped data, queues, storage, functions, realtime tickets, IDEL QL, GraphQL, triggers, and webhooks cannot fall back to a username-shaped project identity.

Signed Webhook Delivery

Each delivery includes an event ID, timestamp, delivery ID, and HMAC-SHA256 signature. HTTPS, redirect blocking, DNS rechecks, and public-network validation are enabled by default.

Deterministic Application Firewall

First-match, versioned rules can allow, deny, or rate-limit endpoint patterns by authentication type, role, tenant, user, API key, numeric IPv4/IPv6 CIDR, and offline country lookup. Enforcement fails closed when its required authority is unavailable.

Country & Proxy Safety

Country decisions use a local MaxMind-compatible database—never a request-time third-party lookup. Forwarded client addresses are accepted only through explicitly trusted proxy networks.

Replicated, Auditable Policy

Atomic policy revisions enter the durable multi-region outbox, while bounded regional security events record decisions without request bodies, query strings, tokens, or raw API-key material.

IDEL Security Advisor

Evidence-backed recommendations highlight risky gaps and unusual activity. Advice is deliberately isolated from enforcement and never changes policy without an administrator.

Requirements, limits, and test scope

These numbers describe the current codebase. They are not all configuration settings, and the benchmark figure is not a capacity claim.

Requirement
Java 21
Build and runtime baseline

Maven compiles Dobase with --release 21.

Test profiles
10K–10M
NDJSON benchmark records

10K is the smoke run; 1M and 10M are opt-in capacity jobs.

Default
1 GiB
Streaming object limit

Runtime-configurable; JSON/Base64 remains 25 MiB by default.

Default
1,000
Records per page

Runtime-configurable; cursors handle million-record scans.

Production controls: Field master keys, wrapped project-key versions, object size, direct-upload size, resumable chunk size, free-disk reserve, session lifetime, preview size, and page size are environment-configurable without rebuilding Dobase.

What Dobase is building next

These initiatives have supporting primitives or design work, but remain roadmap items until their public APIs, lifecycle integration, failure handling, security review, benchmarks, and end-to-end tests are complete.

Timewave APIs

Time travel & change history

Expose safe AS OF reads, snapshot diffs, and preview-before-restore workflows for databases, collections, and documents, plus change-frequency and growth analytics.

AXIS client + server

Intent-bound, verifiable execution

Add a versioned, project-scoped AXIS registry and ingress with deterministic contract decisions, replay protection, proof and receipt verification, OpenLogs evidence, and Java and TypeScript clients. The execution chain will be IDEL → AXIS → registered handler with no unregistered fallback.

Write path

One-fsync durable writes

Collapse WAL and data persistence into one authoritative log, move fsync outside collection locks, and expand bounded group commit while preserving strict acknowledgements and crash recovery.

Query engine

Advanced query planning

Plan range, IN, OR, and posting-list intersections, then satisfy eligible sorting directly from index order instead of scanning and materializing matches.

Read path

Raw-byte streaming

Evaluate filters and search over streamed document bytes and return stored bytes directly where possible, reducing map allocation, garbage collection, and heap pressure.

Data lifecycle

Online encryption & schema migration

Run resumable, rate-limited field-key re-encryption and schema migrations in the background with checkpoints, pause and resume controls, audit evidence, and safe rollback.

Binary GA & IDEL-native model

Canonical, verifiable typed documents

Graduate the current typed-binary collection preview with online migration, compaction, audit, transactions, replication, restore drills, and compatibility tooling; then accept human-readable IDEL for canonical authoring, signatures, policy references, and evidence. IDEL QL already queries both collection engines through one API.

Resilience

Verified multi-region failover

Qualify automatic failover, fencing, promotion, replay, and encrypted restore through recurring drills with measurable RPO and RTO before multi-region operation is enabled.

Storage engine

Online compaction & retention

Add generation-safe compaction, tombstone reclamation, and retention enforcement so append-only logs remain fast and disk-bounded without long write pauses.

Operations

Tenant SLO & capacity guardrails

Add per-project quotas, latency and error SLOs, capacity forecasts, and performance regression gates so one tenant or release cannot silently consume shared resources.

Your first document and durable job, end to end

Provision a local administrator, create an organization and project, then issue a one-project API key. This copy-ready example uses curl and jq, waits for the accepted queue job to finish, and reads back the project-owned document.

1

Authenticate

POST /generate-token
2

Create a project key

POST /projects/demo/api-keys
3

Create database & collection

POST /db/create-database/perfdb
4

Insert project-owned data

POST /db/perfdb/events/insert
5

Queue a durable update

POST /queues/writes/jobs
6

Wait, then verify

GET job status → GET document
api-flow.sh
bash
# Terminal 1 — start with a local admin
export DOBASE_SECURITY_USERS_0_USERNAME="admin"
export DOBASE_SECURITY_USERS_0_PASSWORD="change-this-local-password"
export DOBASE_SECURITY_USERS_0_ROLES_0="ADMIN"
scripts/dev-dobase.sh

# Terminal 2 — authenticate the administrator once
BASE_URL="http://127.0.0.1:8888"
TOKEN=$(curl -fsS -X POST "$BASE_URL/generate-token" \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"change-this-local-password"}' | jq -er .token)
admin_api() {
  curl -fsS -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" "$@"
}

# Create a tenant boundary and issue a one-project credential
admin_api -X POST "$BASE_URL/organizations" \
  -d '{"id":"acme","name":"Acme"}'
admin_api -X POST "$BASE_URL/projects" \
  -d '{"id":"demo","organizationId":"acme","name":"Demo"}'
API_KEY=$(admin_api -X POST "$BASE_URL/projects/demo/api-keys" \
  -d '{"name":"Local app","roles":["ADMIN"]}' | jq -er .key)
api() {
  curl -fsS -H "X-API-Key: $API_KEY" \
    -H "Content-Type: application/json" "$@"
}

# Create project-owned data with typed collection metadata
api -X POST "$BASE_URL/db/create-database/perfdb" -d '{}'
api -X POST "$BASE_URL/db/perfdb/create-collection/events" \
  -d '[{"fieldName":"type","fieldType":"STRING","encrypted":false},{"fieldName":"userId","fieldType":"STRING","encrypted":false}]'
api -X POST "$BASE_URL/db/perfdb/events/insert" \
  -d '{"id":"evt-1001","type":"click","userId":"u-91"}'

# Queue the update, wait, then read the document
JOB_ID=$(api -X POST "$BASE_URL/queues/writes/jobs" \
  -d '{"operation":"DATABASE_UPDATE","database":"perfdb","collection":"events","documentId":"evt-1001","payload":{"id":"evt-1001","type":"processed","userId":"u-91"},"idempotencyKey":"process-evt-1001","maxAttempts":5}' | jq -er .job.id)

STATUS=""
for _ in {1..50}; do
  STATUS=$(api "$BASE_URL/queues/jobs/$JOB_ID" | jq -er .job.status)
  case "$STATUS" in
    SUCCEEDED) break ;;
    DEAD_LETTER|CANCELLED) echo "Job $STATUS" >&2; exit 1 ;;
  esac
  sleep 0.2
done
[ "$STATUS" = "SUCCEEDED" ] || { echo "Job timed out" >&2; exit 1; }
api "$BASE_URL/db/perfdb/events/find-by-id/evt-1001"

React to changes from application code: exchange an authenticated request at POST /realtime/ticket, connect to /ws/realtime, and subscribe to resource channels such as databases/app/collections/orders or storage/buckets/assets/*. Backend clients can also use the realtimeEvents GraphQL subscription at /graphql, or register a signed HTTPS receiver through POST /webhooks when they do not want to keep a socket open.