Append-Only Writes
Inserts, updates, and tombstones append to NDJSON files with a per-collection write lock and configurable batched fsync.
Build and operate data services from one Java engine. Dobase combines stable append-only NDJSON storage, an opt-in typed binary collection preview, authenticated REST APIs, generation-verified secondary indexes, durable scheduled jobs, organization and project isolation, managed API keys, object storage, sandboxed functions, schema-driven field encryption, durable post-commit triggers, typed realtime events, signed webhooks, deterministic firewall policy, verified backups, controlled regional replication, and a built-in management console with an IDEL QL workbench.
The core data path is built around append-only persistence, bounded reads, durable ID lookups, and explicit operational limits.
Inserts, updates, and tombstones append to NDJSON files with a per-collection write lock and configurable batched fsync.
File-offset cursors avoid deep offset scans and return bounded pages with a next cursor and total document count.
Tombstone ratio and file-growth thresholds queue background compaction, with task status exposed through the REST API.
ID and secondary-index snapshots are persisted atomically, tied to the source-data generation, and rebuilt after recovery whenever they are missing, corrupt, or stale.
Configurable read/write permits, page-size validation, queue depth limits, and bounded function execution protect the host.
Typed collection metadata is enforced for inserts, updates, atomic batches, and physical shards. Invalid JSON, mismatched IDs, reserved fields, and incompatible values fail before they enter the durable log.
New schema-required collections can opt into checksum-framed, append-only binary records with stable field IDs, typed values, field encryption, crash-tail recovery, secondary indexes, and the same REST and IDEL QL read surface. Existing collections remain on the stable NDJSON engine.
Inspect active documents, tombstones, file growth, compaction thresholds, JVM metrics, readiness, runtime configuration, and Prometheus metrics through an ADMIN-protected actuator endpoint.
Run one writable primary with read replicas across servers and datacenters using an authenticated, fsync-backed ordered outbox, automatic retry, replay controls, lag status, and replica write protection. Automatic failover remains gated on recurring fencing and restore drills.
Case-consistent equality indexes accelerate document queries, while persisted materialized-aggregate definitions rebuild safely and stay current across recovery, single writes, atomic batches, collection recreation, and every physical shard.
Declare typed collection fields and mark only sensitive fields encrypted. AES-256-GCM envelope encryption protects WAL, data files, and backups while ordinary fields remain queryable.
Queue database mutations and sandboxed function invocations for now
or later. Dobase persists the job before returning
202 Accepted, then executes it with leases and
at-least-once delivery.
Supply an ISO-8601 runAt value for delayed inserts,
updates, deletes, or registered function invocations.
The single-node backend appends and fsyncs every state change.
Production can use Redis AOF with optional Redis 7.2+
WAITAOF confirmation.
Worker leases recover abandoned jobs, exponential backoff handles transient failures, and exhausted work stays inspectable and can be retried manually.
Project- and queue-scoped idempotency keys make client retries return the original job instead of scheduling duplicates.
Attach an optional HTTPS callback to a job. Delivery is retried, can be HMAC-SHA256 signed, and rejects redirects and private network targets by default.
Every queue state transition publishes a project-scoped
queues/* event, so all console versions show live
completion, retry, webhook, and dead-letter notifications.
Match database, collection, document, bucket, and object events, then enqueue a project-scoped function or signed HTTPS webhook after commit with retries, idempotency, and dead letters.
The built-in console uses the same authenticated APIs available to your applications and keeps destructive actions explicit.
Search, sort, and paginate databases, collections, documents, buckets, files, functions, users, organizations, and projects; preview files, define typed encrypted schemas, run IDEL QL, inspect events, manage triggers, rotate keys, and manage webhooks from one interface.
Live health, readiness, counters, JVM memory, uptime, storage statistics, and realtime connection status.
Connect with a one-time socket ticket, subscribe by resource and event type, and inspect structured payloads without polling.
Issue named, role-scoped credentials for exactly one project, then search, update, disable, rotate, or permanently revoke them. Raw secrets are shown once and stored only as hashes.
Authentication is enabled by default, registration is disabled, and production requires stable JWT and encryption keys.
Create scheduled or on-demand snapshots with SHA-256 manifests, configurable retention, isolated recovery drills, and staged restart-safe restore with automatic rollback protection.
Register provisioned HTTPS regions, enable, disable, drain, or resume peers, and inspect node identity, journal capacity, health, lag, retries, and bounded replay operations.
Inspect master-key health and wrapped project-key versions, then rotate a project DEK without deleting the old versions required for reads, restore, and recovery drills.
Execute bounded read-only document, file, function, and metadata
queries from a dark code editor with live resource-aware
autocomplete and copyable JSON results. Canonical OpenExecution
syntax stays concise:
show.collections.analytics sort="name" order="asc" limit=25.
Search, simulate, publish, and audit endpoint policies covering IP networks, countries, identities, roles, API keys, organizations, projects, and Redis-backed rate limits.
Dobase pairs its document API with project-scoped object storage and bounded server-side function execution.
Create and rename buckets, list objects, upload files, preview metadata, download content, and explicitly delete objects.
Object bytes are encrypted at rest. Production requires a stable external encryption key and Redis-backed metadata.
Store version history, list prior versions, roll an object back, or delete a specific version through GraphQL operations.
Stream multipart files or resume encrypted chunk uploads. The object ceiling is operator-configurable with a 1 GiB default.
Publish immutable TypeScript or JavaScript function versions, activate a known-good revision, or run bounded read-only IDEL QL. TypeScript is compiled before activation and executable code runs inside a GraalJS sandbox with time and output limits.
REST drives database and operator workflows, GraphQL exposes a typed realtime subscription, and the WebSocket API supports multiplexed database, queue, storage, function, and system listeners.
Every project receives a random data key wrapped by an external versioned master key. Fresh nonces and location-bound authentication prevent ciphertext substitution across tenants, databases, documents, or fields.
Authentication protects the API by default, while roles and project identities separate operator actions from application access.
Signed bearer tokens carry roles, expire automatically, and use a stable secret supplied by production configuration. Logout, password changes, role changes, and account deletion rotate a persisted session generation so existing tokens stop working immediately.
Administrative routes require the ADMIN role while authenticated clients retain access to permitted data operations.
Organizations group globally unique projects. Every API key is permanently bound to one project; JWT clients must send an explicit project context. Database ownership is enforced, and scoped data, queues, storage, functions, realtime tickets, IDEL QL, GraphQL, triggers, and webhooks cannot fall back to a username-shaped project identity.
Each delivery includes an event ID, timestamp, delivery ID, and HMAC-SHA256 signature. HTTPS, redirect blocking, DNS rechecks, and public-network validation are enabled by default.
First-match, versioned rules can allow, deny, or rate-limit endpoint patterns by authentication type, role, tenant, user, API key, numeric IPv4/IPv6 CIDR, and offline country lookup. Enforcement fails closed when its required authority is unavailable.
Country decisions use a local MaxMind-compatible database—never a request-time third-party lookup. Forwarded client addresses are accepted only through explicitly trusted proxy networks.
Atomic policy revisions enter the durable multi-region outbox, while bounded regional security events record decisions without request bodies, query strings, tokens, or raw API-key material.
Evidence-backed recommendations highlight risky gaps and unusual activity. Advice is deliberately isolated from enforcement and never changes policy without an administrator.
These numbers describe the current codebase. They are not all configuration settings, and the benchmark figure is not a capacity claim.
Production controls: Field master keys, wrapped project-key versions, object size, direct-upload size, resumable chunk size, free-disk reserve, session lifetime, preview size, and page size are environment-configurable without rebuilding Dobase.
These initiatives have supporting primitives or design work, but remain roadmap items until their public APIs, lifecycle integration, failure handling, security review, benchmarks, and end-to-end tests are complete.
Expose safe AS OF reads, snapshot diffs, and
preview-before-restore workflows for databases,
collections, and documents, plus change-frequency and
growth analytics.
Add a versioned, project-scoped AXIS registry and ingress
with deterministic contract decisions, replay protection,
proof and receipt verification, OpenLogs evidence, and
Java and TypeScript clients. The execution chain will be
IDEL → AXIS → registered handler with no
unregistered fallback.
Collapse WAL and data persistence into one authoritative log, move fsync outside collection locks, and expand bounded group commit while preserving strict acknowledgements and crash recovery.
Plan range, IN, OR, and
posting-list intersections, then satisfy eligible sorting
directly from index order instead of scanning and
materializing matches.
Evaluate filters and search over streamed document bytes and return stored bytes directly where possible, reducing map allocation, garbage collection, and heap pressure.
Run resumable, rate-limited field-key re-encryption and schema migrations in the background with checkpoints, pause and resume controls, audit evidence, and safe rollback.
Graduate the current typed-binary collection preview with online migration, compaction, audit, transactions, replication, restore drills, and compatibility tooling; then accept human-readable IDEL for canonical authoring, signatures, policy references, and evidence. IDEL QL already queries both collection engines through one API.
Qualify automatic failover, fencing, promotion, replay, and encrypted restore through recurring drills with measurable RPO and RTO before multi-region operation is enabled.
Add generation-safe compaction, tombstone reclamation, and retention enforcement so append-only logs remain fast and disk-bounded without long write pauses.
Add per-project quotas, latency and error SLOs, capacity forecasts, and performance regression gates so one tenant or release cannot silently consume shared resources.
Provision a local administrator, create an organization and project,
then issue a one-project API key. This copy-ready example uses
curl and jq, waits for the accepted queue
job to finish, and reads back the project-owned document.
POST /generate-token
POST /projects/demo/api-keys
POST /db/create-database/perfdb
POST /db/perfdb/events/insert
POST /queues/writes/jobs
GET job status → GET document
# Terminal 1 — start with a local admin
export DOBASE_SECURITY_USERS_0_USERNAME="admin"
export DOBASE_SECURITY_USERS_0_PASSWORD="change-this-local-password"
export DOBASE_SECURITY_USERS_0_ROLES_0="ADMIN"
scripts/dev-dobase.sh
# Terminal 2 — authenticate the administrator once
BASE_URL="http://127.0.0.1:8888"
TOKEN=$(curl -fsS -X POST "$BASE_URL/generate-token" \
-H "Content-Type: application/json" \
-d '{"username":"admin","password":"change-this-local-password"}' | jq -er .token)
admin_api() {
curl -fsS -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" "$@"
}
# Create a tenant boundary and issue a one-project credential
admin_api -X POST "$BASE_URL/organizations" \
-d '{"id":"acme","name":"Acme"}'
admin_api -X POST "$BASE_URL/projects" \
-d '{"id":"demo","organizationId":"acme","name":"Demo"}'
API_KEY=$(admin_api -X POST "$BASE_URL/projects/demo/api-keys" \
-d '{"name":"Local app","roles":["ADMIN"]}' | jq -er .key)
api() {
curl -fsS -H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" "$@"
}
# Create project-owned data with typed collection metadata
api -X POST "$BASE_URL/db/create-database/perfdb" -d '{}'
api -X POST "$BASE_URL/db/perfdb/create-collection/events" \
-d '[{"fieldName":"type","fieldType":"STRING","encrypted":false},{"fieldName":"userId","fieldType":"STRING","encrypted":false}]'
api -X POST "$BASE_URL/db/perfdb/events/insert" \
-d '{"id":"evt-1001","type":"click","userId":"u-91"}'
# Queue the update, wait, then read the document
JOB_ID=$(api -X POST "$BASE_URL/queues/writes/jobs" \
-d '{"operation":"DATABASE_UPDATE","database":"perfdb","collection":"events","documentId":"evt-1001","payload":{"id":"evt-1001","type":"processed","userId":"u-91"},"idempotencyKey":"process-evt-1001","maxAttempts":5}' | jq -er .job.id)
STATUS=""
for _ in {1..50}; do
STATUS=$(api "$BASE_URL/queues/jobs/$JOB_ID" | jq -er .job.status)
case "$STATUS" in
SUCCEEDED) break ;;
DEAD_LETTER|CANCELLED) echo "Job $STATUS" >&2; exit 1 ;;
esac
sleep 0.2
done
[ "$STATUS" = "SUCCEEDED" ] || { echo "Job timed out" >&2; exit 1; }
api "$BASE_URL/db/perfdb/events/find-by-id/evt-1001"
React to changes from application code: exchange an
authenticated request at POST /realtime/ticket, connect
to /ws/realtime, and subscribe to resource channels such
as databases/app/collections/orders or
storage/buckets/assets/*. Backend clients can also use
the realtimeEvents GraphQL subscription at
/graphql, or register a signed HTTPS receiver through
POST /webhooks when they do not want to keep a socket
open.