ngn

What is ngn?

ngn is a cron scheduler for JavaScript and TypeScript task files. Write tasks as plain files, run one command, and every run — status, result, logs, timings, key-value state — is recorded into one SQLite file. Each task also gets its own SQLite database, and tasks can hand work to each other through job queues and keep out of each other's way with named locks. Everything lives in SQLite files on your disk; there is no server or broker to run beside it.

Installation

One package. Install it globally, or run it without installing:

npm install -g @apisurf/ngn
ngn <command>

# or, without installing
npx @apisurf/ngn <command>

Add it to the project too, for the TaskContext and defineConfig types:

npm i @apisurf/ngn

Commands

ngn init            # ngn.config.ts + tasks/
ngn add scrape.ts   # tasks/scrape.ts, from a template
ngn run             # schedule every task  (Ctrl-C to stop)

Every command takes --root to point at a project other than the working directory, and --help for its details. Quote your globs so the shell does not expand them.

run:once only picks from tasks the config match already covers. It retries as the task's retry allows, but the circuit breaker never holds it back.

run:single ignores ngn.config.ts entirely: no .env is loaded and the run database is :memory:, so nothing is recorded and its locks are not shared with other processes. To schedule one configured task, use ngn run --match.

ngn init

Flags write non-default values into the new config:

Configuration

ngn.config.ts (or ngn.config.js), written by ngn init and typed by defineConfig. Every option is optional:

import { defineConfig } from "@apisurf/ngn";

export default defineConfig({
  dbPath: "file:./ngn.sqlite",
  port: 4545,
  match: ["tasks/**/*.ts"],
  envFile: ".env",
});

A bad value stops the command with the option's name. Set dbPath to a file: URL to keep your run history, and to share locks between ngn run and a run:once in a second terminal.

Writing a task

A task file exports timing and task. Everything else is optional.

import { TaskContext } from "@apisurf/ngn";

// Every 5 seconds
export const timing = "*/5 * * * * *";

export const task = async (ctx: TaskContext) => {
  await ctx.log.info("fetching");

  const html = await fetch("https://example.com", { signal: ctx.run.signal }).then((res) =>
    res.text(),
  );
  await ctx.kv.set("last-length", String(html.length));

  // Recorded with the run, as JSON in task_runs.result
  return { length: html.length };
};

Your file and its relative imports are bundled. Package imports are left alone and resolved from your own node_modules at run time, so the version you installed is the version that runs. Anything else a task needs, it imports itself.

What a task can export

A duration is milliseconds or a string: "500ms", "30s", "4m", "2h", "1d". A bad overlap, budget, retry or exclusive fails when the task is scheduled, naming the task, not on its first tick.

The context

task and every hook get the same ctx:

Hooks

Five optional exports run around task:

// Before task. Returning true records the run as skipped; no other hook fires
export const shouldSkip = async (ctx: TaskContext) => {
  return (await ctx.kv.get("paused")) === "1";
};

export const onSuccess = async (ctx: TaskContext) => {
  await ctx.log.info("scrape ok");
};

// The run is already recorded as a failure; the error is rethrown after this
export const onError = async (error: Error, ctx: TaskContext) => {
  await ctx.log.error(error.message);
};

// After success or failure. Not called on a skipped run
export const onComplete = async (ctx: TaskContext) => {
  await ctx.kv.set("last-run", new Date().toISOString());
};

// Before each retry that retry allows. Returning false gives up until the next tick
export const shouldRetry = async (error: Error, ctx: TaskContext) => {
  return !error.message.includes("401");
};

Scheduling

Cron patterns

Six fields, seconds first, or the usual five, minutes first. With five fields, tasks run at second 0.

┌────────────── second (0-59, optional)
│ ┌──────────── minute (0-59)
│ │ ┌────────── hour (0-23)
│ │ │ ┌──────── day of month (1-31)
│ │ │ │ ┌────── month (1-12 or jan-dec)
│ │ │ │ │ ┌──── day of week (0-7 or sun-sat, 0 and 7 are Sunday)
│ │ │ │ │ │
* * * * * *

When a run is still going

A tick can arrive before the previous run has finished. By default ngn does not start a second run beside it: the tick is recorded as a skipped run, with skip_reason = 'previous run still in progress', and none of the task's code runs, shouldSkip included.

export const timing = "*/5 * * * *";
export const overlap = "allow"; // start another run anyway

How long a run has

ctx.run says when this run's slot ends, so a task working through a backlog can stop taking on more before the next tick and leave the rest for it:

export const timing = "*/5 * * * *";
export const budget = "2m"; // optional: stop sooner than the next tick

export const task = async (ctx: TaskContext) => {
  for (const item of await loadBacklog()) {
    if (ctx.run.signal.aborted) break; // the slot is nearly over
    await process(item);
  }
};

The deadline is a tenth of the period before the next tick, and never more than five seconds before it. budget brings it closer. run:once has no next tick, so it gets a deadline only from a budget. The signal is a request: ngn does not stop a run that ignores it. Times are epoch milliseconds.

When a run fails

retry retries a failed run inside its tick, but never into the next one: a retry that would not start before the next tick is left to that tick. Each attempt is its own run, with retry_of pointing at the tick's first. A task that fails to load is not retried, and shouldRetry gets the last word on the rest.

export const timing = "*/10 * * * *";
export const retry = {
  attempts: 3, // tries per tick, the first included
  delay: "30s", // before each retry
  breaker: { after: 5, maxSkip: 16 },
};

breaker stops a task that keeps failing from failing on every tick. After after failed ticks in a row, ngn skips the next tick, then lets one through; if that fails too it skips two, then four, up to maxSkip (32 by default). One tick that succeeds closes it. Skipped ticks are recorded with skip_reason = 'circuit open after repeated failures'. The streak is read from the run history, so restarting ngn run does not reset it.

Storing data with ctx.sqlite

Every task gets its own SQLite database, with nothing to configure. It lives next to the task file — tasks/scrape.ts writes to tasks/scrape.db — and the file is only created once the task actually uses it. This is your data, separate from ngn's own run history in dbPath.

export const task = async (ctx: TaskContext) => {
  await ctx.sqlite.execute("CREATE TABLE IF NOT EXISTS pages (url TEXT, seen TEXT)");
  await ctx.sqlite.execute("INSERT INTO pages (url, seen) VALUES (?, ?)", [
    "https://example.com",
    new Date().toISOString(),
  ]);

  const { rows } = await ctx.sqlite.execute("SELECT COUNT(*) AS n FROM pages");
  await ctx.log.info(`${rows[0].n} pages`);
};

initDB takes a path relative to the task's folder and runs each migration once, tracked in a _ngn_migrations table inside that database:

const cache = await ctx.sqlite.initDB({
  file: "cache.db",
  migrations: [
    { id: "2026-01-13-001", up: "CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)" },
    { id: "2026-01-14-001", up: "ALTER TABLE items ADD COLUMN qty INTEGER" },
  ],
});

await cache.execute("INSERT INTO items (name, qty) VALUES (?, ?)", ["widget", 3]);

Two tasks in one folder that open the same file share it, which is how they share a queue. A task can only reach databases inside its own folder or its subfolders — absolute paths and paths that climb out with .. are rejected. Connections are held open between runs and closed on shutdown.

Queues

A task that finds work and a task that does it can share a queue. It is a table, _ngn_jobs, in whichever database you open it on, and ngn handles the parts that are easy to get wrong:

• two runs, or two processes, never take the same job;
• a failed job backs off until the next tick instead of being retried at once, and is dead after maxAttempts;
• a job held by a run that died becomes available again once its lease runs out;
• the worker stops taking jobs before the next tick, and leaves the rest for it.

// Every 2 minutes, find pages
export const timing = "*/2 * * * *";

export const task = async (ctx: TaskContext) => {
  const db = await ctx.sqlite.initDB({ file: "pages.db" });
  const pages = await ctx.queue<{ url: string }>("pages", { db });

  const links = await findLinks();
  // A key already in the queue, in any status, is not added again
  const added = await pages.add(
    links.map((url) => ({ key: url, group: new URL(url).host, payload: { url } })),
  );
  return { found: links.length, added };
};

// Every 5 minutes, read as many as fit
export const timing = "*/5 * * * *";

export const task = async (ctx: TaskContext) => {
  const db = await ctx.sqlite.initDB({ file: "pages.db" });
  const pages = await ctx.queue<{ url: string }>("pages", { db, maxAttempts: 3 });

  // Return to complete a job, throw to fail the attempt
  return pages.work(async (job) => read(job.payload.url), { concurrency: 4, perGroup: 10 });
};

Both tasks open pages.db from the same folder, so they share the queue. Without db, a queue lives in the task's own database, where only that task sees it.

Working a queue

work() claims a few jobs at a time and stops taking new ones at ctx.run.deadline, finishing the ones in hand. It never works two jobs of one group (one site, one account) at once, and spreads each call across groups instead of draining one. It returns a summary; return that from task and every run records how the queue is doing:

{ done, failed, dead, released, remaining, oldestAgeMs, stoppedGroups, stoppedBy }
// stoppedBy: "empty" | "deadline" | "max"

The handler gets the job and a handle:

A browser-backed worker, with a bot wall that stops the whole site:

// One browser for every worker, launched by whichever worker needs it first
let browser: Promise<Browser> | null = null;

const summary = await pages.work(
  async (job, { resource: page, stopGroup }) => {
    const result = await extract(page, job.payload.url);
    if (result.blocked) return stopGroup(); // every page on this site would say the same
    await save(result);
  },
  {
    concurrency: 4,
    worker: {
      setup: async () => (await (browser ??= launch())).newPage(),
      teardown: (page) => page.close(),
    },
  },
);
await (await browser)?.close();

The queue itself

// A job goes in the same transaction as the row it is about
await db.batch([
  { sql: "INSERT INTO listings (url) VALUES (?)", args: [url] },
  pages.addStatement({ key: url, group: host, payload: { url } }),
]);

// Filters take { group, groupNotIn, key }
await pages.park({ groupNotIn: SUPPORTED_HOSTS }, "no extractor");
await pages.reopen({ group: SUPPORTED_HOSTS }); // a new extractor just landed
await pages.reopen({}, { from: ["dead"] }); // retry what gave up
await pages.prune({ olderThan: "7d" });

A job's status is one of ready, leased, done, dead or parked. A job's attempts counts the one in hand, and lastError holds what the previous one failed with.

Locks

Two tasks that must not work on the same thing at once can share a lock, by name. Locks live in ngn's database, so with a file: dbPath they also hold against ngn run:once in a second terminal.

// Hold a lock around part of a run. Waits for it, until the run's deadline
export const task = async (ctx: TaskContext) => {
  await ctx.lock("site:example.com", async () => {
    await scrape("https://example.com");
  });
};

// Or hold locks for the whole run. A run that cannot have them all is
// recorded as skipped, and the next tick tries again
export const exclusive = ["browser"];

A run skipped by exclusive is recorded with skip_reason = 'lock "browser" is held'. ctx.lock takes options as its third argument:

When the lock cannot be had in time, ctx.lock throws a LockBusyError. Check error.name === "LockBusyError"; it works even where instanceof would not.

Reading what your tasks did

Both readers open the file dbPath points at, so it must be a file: URL. Neither needs the scheduler running.

From the terminal

ngn sql passes any query straight to SQLite and prints a table. Rows go to stdout and counts to stderr, so piping into jq works. It never migrates the file, so a query cannot change its schema.

ngn sql "SELECT status, COUNT(*) FROM task_runs GROUP BY status"
ngn sql "SELECT * FROM logs WHERE status = 'error' ORDER BY id DESC" --json
ngn sql "SELECT id, skip_reason FROM task_runs WHERE status = 'skipped' ORDER BY id DESC"
ngn sql --db tasks/pages.db "SELECT status, COUNT(*) FROM _ngn_jobs GROUP BY 1"

ngn sql --help lists every column. Times are unix milliseconds. The tables:

In a browser

@apisurf/ngnui is a separate CLI that serves a dashboard over the same file: tasks, runs with their logs, timings and results, stored keys, code versions and a SQL console. Run it from the project folder and it reads dbPath from the config. Install it with npm i -g @apisurf/ngnui, or run it with npx.

ngnui                                    # dbPath from ./ngn.config.ts, on http://127.0.0.1:3000
ngnui --live http://127.0.0.1:4545       # with the live editor
ngnui --db ~/work/ngn.sqlite --port 4000 # another file, another port

It opens the file read-only and binds to loopback. Its live editor runs code through a running ngn run, which --live points at, on the config's port; that endpoint listens on loopback only. ngn run prints the exact line for your project when it starts. The scheduler never requires ngnui.

Upgrading from 0.2.x