How this note was made.

Grove’s research agent (0.2.0) drafted it on using local Gemma 4 (31B Dense) model , served by Ollama on a Mac M3 Max.

It read 40 sources: 30 web pages, the research brief and nine of Grove’s own documents. From them it kept 128 quotes, each checked word for word against a stored copy of its source.

I have approved the draft on 7 November 2026. Claude currently acting as an orchestrator without intelligence in the absence of interface and GPT helped me format for links, tables, and graphics used in this post shared.

Every factual statement below rests on one of those quotes, cited as [En] and listed under Sources at the end. Where the sources are silent, the note says so.

[All web pages were retrieved on 7 November 2026]

Executive summary

Research Brief and Architectural Context.

I have asked for the evidence behind “our case of leveraging TypeScript SDK” to back on my decision as well as cross verifying before final call. It was to go beyond “developer community fondness and support” and cover the current status of support, future efforts and, “most importantly current trade offs” [E12].

Grove’s recorded position is that it is TypeScript-first, with language-neutral contracts so that parts of it can later move to Python [E13][E1].

The case, beyond fondness.

The evidence gives four reasons:

1. Types plus run-time schemas. Both Claude SDKs and four of the five other agent toolkits in the run pair TypeScript types with a schema library that also checks data at run time, usually Zod (section 2.2).

2. Types as contracts. DeepSeek typed its harness’s wire protocol so that “server drift breaks compilation, not clients” [E70]. Grove’s plan makes bypassing its gateway, router or recorder a compile error [E3].

3. Agent-assisted coding. GitHub links TypeScript’s rise to “agent-assisted coding” [E107]. A 2025 paper uses type systems to constrain model-generated code, and extends the method to TypeScript [E128].

4. The SDKs exist and are actively released.

   • Both of Claude’s SDK layers have TypeScript editions.

   • Anthropic’s API SDK for TypeScript shipped at least nine releases between 15 September and 5 November 2026, including 1.0.0 on 21 October 2026 [E56].

   • DeepSeek’s design note of 27 July 2026 records building a TypeScript SDK for its harness [E68].

   • Five other agent toolkits ship for TypeScript (section 5).

Future efforts

TypeScript 7.0, a native port written in Go, reached beta on 21 April 2026. It is “often about 10 times faster than TypeScript 6.0” [E93], and it’s release was planned “within the next two months” [E94]. Whether it has shipped is not in the evidence. Node.js now runs TypeScript directly by stripping types, a feature its documentation marks as stable [E99].

Limits of the case.

• Quality evidence is mixed. The one direct study found better code quality and understandability in TypeScript applications than in JavaScript ones, but not fewer bugs or faster fixes [E120][E121].

• Python leads in AI. Python “remains dominant for AI and data science workloads” [E108].

• Popularity measures disagree. TypeScript is first on GitHub [E110] but sixth in the 2026 Stack Overflow survey [E116].

• Types vanish at run time. Node.js checks no types when it runs TypeScript [E101], and running it directly rules out some syntax [E103][E104].

• DeepSeek’s TypeScript SDK is not an API client. It drives the DeepSeek Harness [E64]. For its model API, DeepSeek’s documentation points to OpenAI’s and Anthropic’s SDKs [E75].

• The SDK comparison is thin. The run’s evidence on Claude’s Python Agent SDK shows it following the TypeScript one on three points [E38][E39][E40], and holds no fuller comparison.

For Grove.

The evidence supports TypeScript-first with language-neutral contracts. It contradicts nothing in Grove’s recorded position. Most of what it suggests adopting or avoiding is already settled by Grove’s decisions: one schema serving as both the type and the run-time check; a separate type check; Python only for the parts that move, behind a planned subprocess protocol with cancellation and a timeout. Five points are open or only partly settled (section 10):

• the cost of a full runtime per child process;

• the syntax limits of running TypeScript directly;

• how Claude’s Agent SDK would sit beside Grove’s own runtime;

• whether DeepSeek’s API works through Grove’s existing OpenAI-compatible route;

• keeping use of any low.

1. The Research Brief and the positions being tested

I have noted that Claude and DeepSeek have both developed TypeScript SDKs, and asked to deep dive beyond the “obvious reason” of community fondness to establish four things:

• why it is good to use a TypeScript SDK;

• the current status of support;

• future efforts from the communities;

• most importantly, the current trade-offs [E12].

Grove’s own records state the position being tested:

• Grove’s decision record on language-neutral contracts says: “Grove is TypeScript-first.” It names the team’s machine-learning codebase as Python, and as the place where an in-house language model is planned to evolve [E13].

• Grove’s implementation plan records the my own decision that some Grove modules will move into that codebase later, so Grove “stays TypeScript with language-neutral contracts” [E1]. Its decision log marks portability as decided on the same terms [E6].

These are stated positions, not findings. Sections 2 to 8 set out what the external sources say. Section 9 tests Grove’s position against them, and section 10 draws out what Grove should adopt or avoid.

2. Why TypeScript for agents, beyond community fondness

2.1 What static types are argued to buy

The 2022 paper “To Type or Not to Type? A Systematic Comparison of the Software Quality of JavaScript and TypeScript Applications on GitHub“ reports what proponents of static typing claim. They say it “allows early detection of some programming errors” and leads to “better documentation in the form of type signatures” [E119]. The paper’s own background section is more cautious. For smaller applications, dynamic typing’s flexibility “could be preferable”, while larger systems “might be more sustainable with static typing” [E125]. What the same paper measured is in section 8.1.

2.2 Schemas: types that also check data at run time

TypeScript’s types do not exist when the program runs (section 8.2). The sources show agent SDKs pairing them with schema libraries that validate data at run time:

• The Claude Agent SDK’s tool() takes a Zod schema for a tool’s input, and types the handler’s arguments from that schema [E32].

• The Claude API’s TypeScript SDK describes tool inputs with “Zod schemas or JSON Schemas” [E45].

• The Vercel AI SDK’s README generates structured model output against a Zod schema [E84].

• The OpenAI Agents SDK’s install command adds zod beside the SDK [E80].

• Google’s Agent Development Kit for TypeScript lists “Zod schema validation” among its features [E92].

• The MCP TypeScript SDK accepts any Standard Schema library: “bring Zod v4, Valibot, ArkType, or any compatible library” [E77].

• Grove’s plan uses zod for its schemas and for their JSON Schema export [E4].

2.3 Types as contracts between components and processes

Three sources use compile-time types to hold a boundary:

• DeepSeek’s wire protocol. DeepSeek’s design note for its TypeScript SDK, dated 27 July 2026, describes its protocol package as “the wire made shared and nominal”: one module names every payload the server speaks [E69]. Because the server’s notify() calls are typed against those names, “server drift breaks compilation, not clients” [E70].

• Grove’s dependency rule. In Grove’s plan, every library depends only on Grove’s public contracts package, and the applications are the only place implementations meet. Package dependencies and a dependency checker enforce the rule. Agents see only the contracts package, “so bypassing the gateway, router or recorder is a compile error” [E3].

• Typed tool results. Version 0.3.292 of the Claude Agent SDK for TypeScript added typed sections and notes to a tool’s result, “so hosts can list agents without parsing its text” [E26].

2.4 Types that flow from a definition to its users

The Vercel AI SDK’s README derives an agent’s message type directly from the agent’s definition [E86]. It also types a UI component’s tool invocation from the tool’s own return type [E87]. The examples show the mechanism. No source in the run measures its benefit.

2.5 Models writing code against types

• A 2025 paper on constrained decoding. “Type-Constrained Code Generation with Language Models“ introduces “a type-constrained decoding approach that leverages type systems to guide code generation”. The method is meant to “enforce well-typedness on LLM-generated code”. The authors formalise it on a simply-typed language, then “extend it to TypeScript to demonstrate practicality” [E128].

• GitHub’s 2025 reading. GitHub’s Octoverse 2025 report says TypeScript’s rise “illustrates how developers are shifting toward typed languages that make agent-assisted coding more reliable in production” [E107].

• GitHub’s 2026 data. Octoverse 2026 found TypeScript to be the primary language in 35% of repositories created in 2026 that showed an agent signal, against 24% of those without one [E114]. GitHub reads this as suggesting that “developers who work with agents gravitate toward languages where agent output is easier to check” [E115]. That reading is GitHub’s interpretation of a correlation, not a measured cause.

2.6 Where the code runs

Each SDK states its own runtimes. These are properties of the SDKs, not of the language.

3. The status of Claude’s TypeScript SDKs

Anthropic’s overview of its SDKs and libraries says that “Claude Code, the Claude Agent SDK, and Claude Managed Agents work at a higher level, providing the agent loop, tool execution, and runtime” [E41]. Below that level sits the Claude API’s own client SDK. Each layer has a TypeScript SDK: the Agent SDK (section 3.1) and the API SDK (section 3.2). The documentation pages are undated. The Agent SDK changelogs give version numbers without dates, and the API SDK changelog gives both.

3.1 The Claude Agent SDK for TypeScript

What it offers.

• Purpose. Its README says it lets developers “programmatically build AI agents with Claude Code’s capabilities”: agents that can “understand codebases, edit files, run commands, and execute complex workflows” [E17].

• Bundled binary. It bundles a native Claude Code binary for the platform as an optional dependency, such as @anthropic-ai/claude-agent-sdk-darwin-arm64. “Most installs need no separate Claude Code install.” [E28]

• Streaming. query() is the primary function. It “creates an async generator that streams messages as they arrive”, and takes either a string or an async iterable of user messages as its prompt [E31].

• Tools and MCP. tool() defines a typed tool from a Zod schema [E32]. createSdkMcpServer() defines an MCP server from a name, a version, instructions and such tools [E35].

• Sessions. It provides session helpers, including listSessions() and getSessionMessages() [E36][E37].

• Prewarming. prewarm() returns a spare process [E34]. It is alpha, was added to the /core entry in version 0.3.314 [E18], and requires Agent SDK v0.3.282 or later [E33].

• Bundling. An application that bundles the SDK with its own dependencies should import the /core entry, which needs Agent SDK v0.3.282 or later, and TypeScript 5.0 or later for its types [E30]. The root entry inlines its own copies of zod and the MCP SDK. /core takes them from the application’s own node_modules instead, so a bundle does not carry a second copy [E29].

Recent changes (in the newest 720 lines of the changelog, which carries no dates):

How actively it is released. The cited entries run from version 0.3.292 to 0.3.319, and the newest tracks Claude Code v2.1.319 [E21]. Because the changelog has no dates, the evidence cannot say how often these versions shipped.

3.2 The Claude API’s TypeScript SDK

What it offers.

• Types. TypeScript definitions “for all request parameters and response fields” [E43].

• Streaming and batches. Streaming responses over Server-Sent Events [E44], and batch processing under client.messages.batches [E49].

• Tool helpers. Tools are described with Zod schemas or JSON Schemas and run with client.beta.messages.toolRunner(), a beta method [E45]. A tool can throw a ToolError, which, unlike a plain Error, can carry images or other structured content back to the model [E46].

• MCP helpers. Helpers convert MCP tools, prompts and resources to Claude API types [E47]. They throw UnsupportedMCPValueError for values the API does not support, such as an unsupported content or MIME type, or a non-HTTP/HTTPS resource link [E48].

• File uploads. Uploads accept a File, a fetch Response, an fs.ReadStream or the output of toFile [E50].

• Errors. A connection failure or a 4xx or 5xx response throws a subclass of APIError [E51].

• Runtimes. The runtimes listed in section 2.6 [E42].

How actively it is released. The changelog is dated:

The cited entries alone show nine releases in about seven weeks, and a 1.0.0 release tied to a new API version. This is the most precisely dated evidence of release activity in the run. The fetch truncated this changelog and the run read only its first 720 lines, so older history is not covered.

3.3 Compared with the Python SDKs

Three entries in the changelog of the Claude Agent SDK for Python describe changes made to match the TypeScript SDK:

• 0.2.160. The SDK now listens for the CLI’s session_state_changed messages and keeps stdin open until the CLI reports idle, “matching the TypeScript SDK’s behavior” [E38].

• 0.2.140. A new forward_subagent_text option forwards a subagent’s text and thinking blocks into the stream. It “Matches the TypeScript SDK’s forwardSubagentText“ [E39].

• 0.2.126. ResultMessage.model_usage is now typed with a ModelUsage TypedDict that “mirrors the TypeScript SDK’s shape” [E40].

On these three points the TypeScript SDK had the behaviour first. The Python changelog is undated.

The evidence holds no full comparison of the two languages’ SDKs, and nothing on where the Python SDKs lead. The run fetched the Python Agent SDK’s README and the Claude API’s Python SDK page, but kept no quote from either (section 11).

4. The status of DeepSeek’s TypeScript SDK

4.1 What DeepSeek Harness is

DeepSeek Harness (dsh) is “an open-source agent harness developed by DeepSeek AI”. It is built on an “everything-is-a-plugin” architecture and powered by Cordis [E62]. Its README‘s quick start is to install Node.js and run it with npx [E63]. The README is undated.

4.2 The TypeScript SDK family

The SDK “lets another process drive a complete DeepSeek Harness runtime over newline-delimited JSON-RPC” [E64]:

• its protocol package defines the public messages;

• the TypeScript client launches dsh with a named profile and ordered patches;

• the server accepts SDK requests over stdio.

It has three packages [E65]:

PackageRoleprotocol/The newline-delimited JSON-RPC transport, and the named request, result and notification typesclient/A TypeScript client that spawns a runtime subprocess and drives agent turns through high-level and protocol-level APIsserver/A jsonrpc plugin that serves out-of-process SDK clients over stdio

DeepSeek’s architecture document adds that the TypeScript SDK resolves a dsh dependency of the same version and selects the sdk profile. Custom plugin composition remains “a profile plus ordered patch files, not another executable or inline application tree” [E73].

DeepSeek’s TypeScript SDK is therefore not a client for DeepSeek’s model API. It drives the harness as a subprocess [E64]. For the model API, see section 4.6.

4.3 It’s Python counterpart

The Python SDK “speaks the same protocol and ships a bundled runtime” [E66]. It is a “Python subprocess SDK for driving DeepSeek Harness over newline-delimited JSON-RPC on stdio” [E67]. It “follows the same application architecture”: its runtime wheel packages the normal dsh CLI, and by default the client launches dsh --profile sdk with an explicit Harness home [E74].

4.4 Why it was built

DeepSeek’s design note, dated 27 July 2026, gives the reason. “TypeScript consumers wanting the same drive-a-harness-as-a-subprocess capability — repo tests, automation, and above all a subagent backend whose child is a complete harness runtime rather than a generic ACP agent — had nothing to import” [E68]. The design made the wire shared and nominal [E69], so that server drift breaks compilation rather than clients [E70].

4.5 The limits its own documents state

The design note lists two costs of the design:

• A full runtime per child. “the SDK backend boots a complete plugin tree per child (heavier per-run than an ACP child; pooling remains future work, same as ACP)” [E71].

• No cancel method. “the wire still has no cancel method, so both the SDK’s RequestTimeoutError and the backend’s dispose settle locally while the server-side turn runs on until process teardown” [E72].

The evidence does not say whether either cost has been addressed since. The run also fetched DeepSeek Harness’s safety notice, but kept no verified quote from it. This note therefore cannot report the limits that notice states (section 11).

4.6 What DeepSeek’s API documentation says about client libraries

“The DeepSeek API uses an API format compatible with OpenAI/Anthropic. By modifying the configuration, you can use the OpenAI/Anthropic SDK or softwares compatible with the OpenAI/Anthropic API to access the DeepSeek API.“ [E75] The page is undated. The quoted passage names no DeepSeek client library.

5. The wider TypeScript agent ecosystem

Each project is described from its own README. The READMEs are undated, and no source in the run ranks the projects against each other.

Three more points from these sources:

• OpenAI states: “We’re committed to building the Agents SDK as an open source framework so others in the community can expand on our approach.” [E81]

• GitHub’s Octoverse 2025 report notes that the Model Context Protocol “hit 37k stars in just eight months”, though it is not on the report’s ranked lists [E109].

• Four of the five projects name Zod, or a schema standard that includes it. Two describe themselves as provider-agnostic [E78][E82].

6. Future efforts

6.1 TypeScript 7: the native compiler

The March 2025 plan. “A 10x Faster TypeScript“ (11 March 2025) announced a native port of the compiler and tools. The port would “drastically improve editor startup, reduce most build times by 10x, and substantially reduce memory usage” [E96]. Its plan at the time was:

• a preview of a native tsc for command-line type checking by mid-2025;

• a feature-complete solution for project builds and a language service by the end of 2025 [E97];

• a release as TypeScript 7.0 once the native codebase “has reached sufficient parity” [E98].

This was the plan as of March 2025, not a record of what happened.

The April 2026 beta. “Announcing TypeScript 7.0 Beta“ (21 April 2026) reports that the codebase had been ported from TypeScript to Go over the past year. With native code speed and shared-memory parallelism, “TypeScript 7.0 is often about 10 times faster than TypeScript 6.0” [E93]. The post said: “Our current plan is to release TypeScript 7.0 within the next two months” [E94]. Two months from the post would be about late June 2026. A stable programmatic API is planned “for TypeScript 7.1 or later” [E95].

What is not known. Whether TypeScript 7.0 has shipped as a stable release is not in the evidence. The newest dated evidence of a compiler in use is Anthropic’s API SDK building its sub-packages with TypeScript 6.0.3, in version 0.127.0 of 18 September 2026 [E61].

6.2 Node.js runs TypeScript directly

Node.js documentation states: “Type stripping is now stable.” [E99] Its version history records this for Node.js v25.2.0 and v24.12.0.

• How it works. By default Node.js “will execute TypeScript files that contains only erasable TypeScript syntax”. It replaces that syntax with whitespace, “and no type checking is performed” [E101].

• Why it is light. The feature is designed to be lightweight. It deliberately leaves out syntax that needs JavaScript code generation, so Node.js runs TypeScript without source maps [E102].

• The two options. Node.js documents two ways to run TypeScript. A third-party package gives “full support of all of TypeScript’s syntax and features, including using any version of TypeScript”. The built-in type stripping gives “lightweight support” [E100].

The limits this brings are in section 8.3.

6.3 The SDKs’ stated directions

The evidence holds no roadmap for any SDK beyond these statements:

• OpenAI commits to the Agents SDK as an open-source framework [E81].

• DeepSeek lists pooling of SDK children as “future work” [E71].

• The Claude Agent SDK tracks Claude Code: its latest entry is “Updated to parity with Claude Code v2.1.319” [E21].

• The Claude API SDK reached 1.0.0 with a new API version on 21 October 2026 [E56].

• TypeScript plans a stable programmatic API for 7.1 or later [E95].

7. Community evidence

Further points from these sources:

• AI involvement in pull requests. Octoverse 2026 reports that TypeScript, Python and Rust, “the fastest-growing communities, have the highest shares, about 40% to 44% of merged pull requests” [E113]. In the report this is each language’s share of merged pull requests with detected AI involvement, not TypeScript’s share of all pull requests.

• Python’s place. The same 2025 report that ranks TypeScript first adds: “Python remains dominant for AI and data science workloads, while the JavaScript/TypeScript ecosystem still accounts for more overall activity than Python alone” [E108].

• The same split in 2021. The 2022 paper cited the 2021 Stack Overflow survey for JavaScript being the most widely used language [E123], and Octoverse for TypeScript’s growing popularity [E124].

• Why the measures disagree. Octoverse counts contributors on GitHub [E110], while the survey counts respondents [E117]. They measure different populations, so they can disagree without either being wrong.

8. The trade-offs

I have called the trade-offs the most important part of the question [E12]. The evidence supports nine. Each ends with its bearing on Grove.

8.1 Better code quality, but not fewer bugs

The 2022 paper “To Type or Not to Type? A Systematic Comparison of the Software Quality of JavaScript and TypeScript Applications on GitHub“ compared JavaScript and TypeScript applications. Its abstract reports three findings:

• Quality. “TS applications exhibit significantly better code quality and understandability than JS applications” [E120].

• Bugs. “Contrary to expectations, however, bug proneness and bug resolution time of our TS sample were not significantly lower than for JS.” The mean bug-fix commit ratio of the TypeScript projects was more than 60% larger (0.206 against 0.126), and they took on average more than a day longer to fix bugs (33.04 against 31.86 days) [E121].

• The any type. Using any less “appears to be beneficial”: its frequency correlated significantly with every metric except bug proneness [E122].

That is the real tension in the evidence: better quality and understandability, but no lower bug proneness, in the same study [E120][E121].

The paper’s related-work section describes earlier studies, with their own samples, that lean the other way:

• Zhang et al. (2020): “Within their sample, dynamically typed languages consumed 59.5% more time for bug resolution than statically typed ones.” [E126]

• Gao et al. (2017): using TypeScript “would have prevented 60 of the 400 bugs they examined (15%)” [E127].

These are different studies, not more data from the 2022 paper’s sample. No source in the run studies agent harnesses or SDK codebases specifically.

For Grove: the evidence supports types for code quality and understandability. It does not promise fewer bugs.

8.2 Types are erased at run time

In the 2022 paper’s description, “TS performs type checking at transpile-time, while JS performs type checking at run-time” [E118]. Node.js’s type stripping replaces TypeScript syntax with whitespace and performs no type checking [E101]. Two things follow:

• Data that crosses a boundary, such as model output, tool input or a message between processes, needs a run-time schema (section 2.2).

• The type check has to run as a step of its own.

For Grove: it does both. Every persisted or exchanged structure is a zod schema [E14], and the type check runs across all packages [E11] and in CI [E8].

8.3 Running TypeScript directly has syntax limits

Type stripping runs only erasable syntax [E101]:

• Transformed features fail. Enum declarations, namespaces with runtime code, parameter properties and import aliases need transformation rather than stripping [E103].

• Decorators fail. They are not transformed and cause a parser error. Node.js “will not support decorators until they are supported natively in JavaScript” [E104].

• Type imports need the type keyword. Without it Node.js treats a type import as a value import, which is a run-time error [E105].

• Full syntax needs a package. Full TypeScript syntax, or a chosen TypeScript version, needs a third-party package [E100].

For Grove: its first milestone report says there is no build step, because Node runs the TypeScript directly [E9]. Its plan lists tsx and tsup for development and build [E4]. The evidence does not say which mechanism Grove’s direct run uses, so whether these limits bind Grove is a gap.

8.4 Python’s place in AI and machine learning

• Dominance. Python “remains dominant for AI and data science workloads” [E108].

• Contributors. It is second on GitHub [E110], added 3.23 million monthly contributors in a year [E111], and is forecast to reach about the same size as TypeScript by September 2027 [E112].

• Survey use. 58.0% of the 2026 survey’s respondents use it, against 43.8% for TypeScript [E117].

The evidence states Python’s dominance in AI in one sentence. No source compares the two languages’ machine-learning libraries.

For Grove: its records already keep a Python boundary. Its machine-learning codebase is Python [E13], and three ports may be re-implemented in Python: the model provider, the context source and the scorer [E16].

8.5 Where the Python SDKs lead or differ

The only comparison in the evidence is the three Python changes made to match the TypeScript Agent SDK [E38][E39][E40]. Nothing in the evidence shows where Python leads.

For Grove: the evidence neither confirms nor rules out a Python-only capability that Grove would miss.

8.6 The cost of keeping two languages

• DeepSeek keeps two SDKs over one protocol [E66][E67][E74]. It built the TypeScript one because TypeScript consumers “had nothing to import” [E68].

• Anthropic keeps the Python Agent SDK in step with the TypeScript one by matching its behaviour, options and types one change at a time [E38][E39][E40].

In both cases a single shared protocol or shape holds the two languages together. No source counts what that costs.

For Grove: it avoids a second SDK by design:

• one TypeScript implementation;

• contracts any language can read, exported as JSON Schema [E14];

• digests any language can reproduce [E15];

• a subprocess protocol only for the ports that move [E16].

8.7 Subprocess SDKs cost runtime and need cancellation

DeepSeek’s design boots a complete plugin tree for each child, and has no pooling yet [E71]. Its wire has no cancel method, so a timed-out turn keeps running on the server until the process is torn down [E72].

For Grove: its planned subprocess protocol maps cancellation to SIGTERM and has a timeout [E16]. Section 10 covers the open part.

8.8 Popularity measures disagree

TypeScript is first on GitHub by contributors [E110] and sixth in the 2026 Stack Overflow survey [E116][E117]. Popularity alone does not settle the case either way.

For Grove: the case rests better on sections 2 to 4 than on rankings.

8.9 Compiler speed and stability are in transition

The TypeScript 7.0 beta is reported as often about ten times faster than 6.0 [E93]. A stable programmatic API comes only with 7.1 or later [E95], and the evidence does not show that 7.0 has shipped (section 6.1).

For Grove: no Grove decision in the evidence addresses TypeScript 7.

9. Grove’s recorded position against the evidence

9.1 What Grove has recorded

• TypeScript-first. Grove is TypeScript-first [E13], and stays TypeScript with language-neutral contracts because some modules will later move to the Python machine-learning codebase [E1][E6].

• One schema per structure. Every persisted or exchanged structure is defined once as a zod schema in Grove’s public contracts package, and exported as JSON Schema with a schema version [E14][E2][E10]. This covers run records, run events, artifacts, provenance, memory entries, model selection, messages, errors and configuration [E14].

• Portable digests. Digests use canonical JSON (keys sorted recursively, undefined values dropped) followed by sha256, “so any language reproduces them” [E15].

• A subprocess protocol, planned. Ports that may be re-implemented in Python will get a local subprocess protocol: NDJSON request and response over stdio, schema-validated messages, cancellation mapped to SIGTERM, and a timeout [E16]. The plan’s risk register lists the TypeScript–Python split as an architecture risk. Its mitigation is that protocol plus a schema conformance script [E7].

• Checks. Grove’s traceability table marks its TypeScript-first typed contracts as tested, by a unit test and by the type check in CI [E8]. The first milestone report lists the type check across all packages, and a lint step for the dependency rule [E11].

• Models. Grove’s private expert is a local OpenAI-compatible model server (Ollama on CPU-only hosts, vLLM on a GPU host), reached through Grove’s OpenAI-compatible provider at a configured address [E5]. The plan lists Anthropic’s TypeScript SDK and the AWS Bedrock runtime client as optional dependencies of Grove’s model router [E4].

9.2 Where the evidence supports it

• Static types and compile-time boundaries. These are argued for by proponents [E119], used by DeepSeek [E69][E70], and used in Grove’s own dependency rule [E3].

• Schema-first contracts. They match how the Claude SDKs, the Vercel AI SDK, the OpenAI Agents SDK, Google’s ADK and the MCP SDK handle schemas [E32][E45][E84][E80][E92][E77].

• TypeScript SDKs exist and are maintained. The Claude API SDK shows frequent dated releases (section 3.2). On three points, the Python Agent SDK followed the TypeScript one [E38][E39][E40].

• DeepSeek also puts a protocol between languages. It runs one wire protocol with TypeScript and Python SDKs on top [E64][E66][E67], the approach Grove plans for the ports that move to Python [E16].

• Agent use. GitHub links TypeScript’s rise to agent-assisted coding, and finds it more common in repositories with agent signals. These are GitHub’s interpretation [E107][E114][E115].

9.3 Where the evidence qualifies it

• Mixed quality evidence. Code quality and understandability are better, but bug proneness is not lower [E120][E121]. Limiting any appears to help [E122].

• Python’s lead in AI. Python is dominant for AI and data science [E108]. Grove’s records already keep a Python boundary for the machine-learning codebase [E13].

• Erased types. Types are erased at run time [E101], so Grove’s run-time schemas are necessary, not optional.

• Direct-run syntax limits. Running TypeScript directly limits the syntax that can be used [E103][E104][E105]. Whether this binds Grove is a gap (section 8.3).

• Popularity. The popularity measures disagree [E110][E116][E117].

• Subprocess costs. Subprocess designs carry runtime and cancellation costs [E71][E72].

9.4 Where the evidence contradicts it

Nothing in the evidence contradicts Grove’s recorded position outright. The nearest is the 2022 paper’s finding that TypeScript projects did not have fewer bugs [E121]. That qualifies the general argument for static types [E119], not Grove’s design.

10. What Grove should adopt or avoid

Each point below is drawn only from the evidence above. The last column says whether a Grove decision quoted in the evidence already settles it.

No row proposes new components. Rows 6, 7, 10 and 12 need an owner decision, or evidence the run did not gather. Row 8 needs a test.

11. Conflicts, gaps and coverage limits

11.1 Conflicts

• The real tension. Within the 2022 study, TypeScript applications had better code quality and understandability [E120], but bug proneness and bug resolution time were not lower [E121].

• Not a conflict: how TypeScript is checked and run. The 2022 paper describes TypeScript as transpiled by tsc, with types checked at transpile time [E118]. Node.js can run erasable TypeScript directly, with no type checking [E101][E102]. That is one way a project can have no build step, as Grove reports [E9], with type checking as a separate step [E11].

• Not a conflict: the 59.5% figure. Zhang et al.’s result [E126] comes from a different study and sample than the 2022 paper’s own [E121].

• Not a conflict: the popularity rankings. GitHub and Stack Overflow measure different populations (section 7).

11.2 Gaps: what the sources do not say

1. Whether TypeScript 7.0 has shipped as a stable release. The newest evidence is the 21 April 2026 plan [E94].

2. A full feature comparison of Claude’s TypeScript and Python SDKs, or where the Python SDKs lead.

3. Release dates for the Claude Agent SDK’s TypeScript and Python versions. Both changelogs give versions only.

4. The limits stated in DeepSeek Harness’s safety notice. It was fetched, but no verified quote was kept.

5. Whether DeepSeek has addressed the per-child cost or the missing cancel method since July 2026 [E71][E72].

6. A comparison of TypeScript’s and Python’s machine-learning ecosystems, beyond one sentence [E108].

7. Evidence on software quality in agent harnesses or SDK codebases specifically. The studies cover general GitHub applications [E120][E121][E126][E127], and type-constrained code generation [E128].

8. The cost of maintaining SDKs in two languages.

9. Which mechanism Grove’s direct run of TypeScript uses [E9][E4].

10. Whether Grove’s subprocess protocol has been built [E16][E6], and whether DeepSeek’s API has been tried through Grove’s OpenAI-compatible provider [E5][E75].

11. Roadmaps for any of the SDKs, beyond the statements in section 6.3.

11.3 Coverage limits

• Long pages were read only in part. The run read the first 720 lines (six windows of 120 lines) of each long page. The pages affected are:

  • the Claude Agent SDK changelogs for TypeScript and Python;

  • the Agent SDK’s TypeScript reference;

  • the Claude API’s TypeScript and Python SDK pages;

  • the Anthropic TypeScript SDK changelog;

  • both Octoverse reports;

  • the 2022 paper, all but its last five lines.

  Older changelog history and later sections were not read.

• Two pages were truncated. The fetch truncated the Agent SDK’s TypeScript reference and the Anthropic TypeScript SDK changelog, so only their start was examined.

• Eight Grove documents were not read. The run reads at most ten local sources. Five decision records, two milestone reports and an operations note ranked below the top ten and were not read.

• Some sources gave no quotes.

  • Five of the nine Grove documents read yielded no quote.

  • Four fetched web pages gave no verified quote: the Claude Agent SDK overview, the Python Agent SDK README, the Claude API’s Python SDK page and DeepSeek Harness’s safety notice.

• Rejected quotes are a coverage limit, not missing material. 54 candidate quotes were rejected because the model paraphrased instead of copying. The rejected quotes are not used here, and nothing is missing from the sources.

Sources

Each cited id maps to one source below. Grove’s own documents are described by their role. Web pages were retrieved on 7 November 2026.

Research brief and Grove’s own documents

1. Research brief

1. Owner’s brief: the case for TypeScript SDKs in Grove

Evidence: E12
Role: My request for this deep dive, recorded word for word
Date: 7 November 2026

2. Grove v0.1 — discovery, implementation and traceability

2. Grove v0.1 — Phase 0 discovery report and implementation plan

Evidence: E1–E7
Role: Grove’s plan, decision log and risk register
Date: Last updated 20 October 2026

3. Grove v0.1 traceability

Evidence: E8
Role: Grove’s table mapping each component to its code and its tests
Date: Undated; read as of 7 November 2026

4. Grove v0.1 — first milestone report (agent foundation)

Evidence: E9–E11
Role: Grove’s first milestone report
Date: 20 October 2026

3. Language-neutral architecture and portability

5. Language-neutral contracts for portability to the machine-learning codebase

Evidence: E13–E16
Role: Grove’s decision record on language-neutral contracts
Date: 20 October 2026

Web sources

All web pages were retrieved on 7 November 2026. Evidence IDs in brackets correspond to citations in the article.

1. Anthropic — Claude SDKs

1. Claude Agent SDK (README)

Evidence: E17
Date: Undated

2. Claude Agent SDK for TypeScript: Changelog

Evidence: E18–E27
Date: Undated; versions 0.3.292 to 0.3.319

3. Agent SDK reference — TypeScript

Evidence: E28–E37
Date: Undated

4. Claude Agent SDK for Python: Changelog

Evidence: E38–E40
Date: Undated; versions 0.2.126 to 0.2.160

5. SDKs, CLI, and libraries

Evidence: E41
Date: Undated

6. TypeScript SDK — Claude API documentation

Evidence: E42–E51
Date: Undated

7. Anthropic TypeScript SDK: Changelog

Evidence: E52–E61
Date: Entries dated 15 September to 5 November 2026

2. DeepSeek — Harness and SDK architecture

8. DeepSeek Harness (README)

Evidence: E62–E63
Date: Undated

9. sdk/ — drive a Harness runtime from another process

Evidence: E64–E66
Date: Undated

10. DeepSeek Harness Python SDK

Evidence: E67
Date: Undated

11. Agent Note: TypeScript SDK client and the SDK subagent backend

Evidence: E68–E72
Date: 27 July 2026; archived 4 September 2026

12. DeepSeek Harness Architecture

Evidence: E73–E74
Date: Undated

13. DeepSeek API Docs: Your First API Call

Evidence: E75
Date: Undated

3. Other TypeScript agent SDKs and toolkits

14. MCP TypeScript SDK (README)

Evidence: E76–E77
Date: Undated

15. OpenAI Agents SDK (JavaScript/TypeScript) (README)

Evidence: E78–E81
Date: Undated

16. AI SDK (Vercel) (README)

Evidence: E82–E87
Date: Undated

17. Mastra (README)

Evidence: E88–E90
Date: Undated

18. Agent Development Kit (ADK) for TypeScript (README)

Evidence: E91–E92
Date: Undated

4. TypeScript and Node.js — language and runtime

19. Announcing TypeScript 7.0 Beta

Evidence: E93–E95
Date: 21 April 2026

20. A 10x Faster TypeScript

Evidence: E96–E98
Date: 11 March 2025

21. Modules: TypeScript — Node.js documentation

Evidence: E99–E105
Date: Undated; type stripping marked stable in v25.2.0 and v24.12.0

5. Developer community and language adoption

22. Octoverse: A new developer joins GitHub every second as AI leads TypeScript to #1

Evidence: E106–E109
Date: 28 October 2025; updated 28 February 2026

23. Octoverse: Commits, pushes, reach new highs in 2026 as developers build with agents

Evidence: E110–E115
Date: 28 October 2026; updated 6 November 2026

24. Stack Overflow Developer Survey 2026: Technology

Evidence: E116–E117
Date: 2026 survey; no publication date on the page

6. Research papers — software quality and type-constrained code generation

25. To Type or Not to Type? A Systematic Comparison of the Software Quality of JavaScript and TypeScript Applications on GitHub — Bogner and Merkel

Evidence: E118–E127
Date: 2022; arXiv v1 of 21 March 2022

26. Type-Constrained Code Generation with Language Models — Mündler et al.

Evidence: E128
Date: 2025; submitted 12 April 2025, revised 8 May 2025

Kodagu is often spoken of through its landscapes. Its coffee plantations, mist-covered hills, forests, winding roads and beautiful homes are familiar images that have come to represent this part of Karnataka.

But there is another Kodagu that reveals itself when we slow down and look beyond the landscape.

It is found in its streets, neighbourhoods, homes and marketplaces. In the conversations that happen outside shops, the daily routines of people at work, the warmth of an unexpected smile, and the quiet moments that might otherwise pass unnoticed.

During my time in Virajpet, and exploring around Coorg, I found myself drawn to these everyday encounters. Not necessarily to extraordinary moments, but to ordinary ones that, when observed closely, begin to tell us something about a place and the people who inhabit it.

The town

Every town has its own character. Sometimes it is found in its architecture, sometimes in its streets, and often in the little details we tend to overlook. Virajpet reveals itself gradually, through familiar corners and everyday movement.

People

Perhaps the most beautiful way to understand a place is through its people. A glance, a smile, a fleeting expression or simply the willingness to pause for a photograph. Every face carries a story, even when we know very little about it.

At home

Beyond the streets and marketplaces are quieter moments of everyday life. Families, children, older generations and the familiar spaces they inhabit. These are intimate glimpses into lives that rarely find their way into photographs of a destination.

Work and movement

Much of a town’s rhythm comes from the people who keep it moving. Those who work, travel, carry, build, sell and contribute in ways that often go unnoticed. Their everyday routines are as much a part of Kodagu’s identity as its landscapes.

Market life

Markets have always fascinated me. They are lively meeting places where people, produce, conversations and livelihoods come together. There is an unmistakable rhythm here — a little chaotic, wonderfully colourful and full of life. Spend enough time in a market and you begin to see much more than a place of trade.

Shared spaces

What I find equally interesting about Kodagu is its diversity. Different communities, religious traditions, languages and ways of life exist alongside one another. This diversity is not always something that announces itself. Often, it is simply part of the everyday fabric of the place.

You see it in children going to school, families outside their homes, shopkeepers serving their customers and people sharing the same streets and neighbourhoods. Their lives may be very different, but the town brings them together in ways that are both ordinary and meaningful.

I don’t think a handful of photographs can explain this coexistence, nor should they attempt to. But they can offer glimpses of it. They can encourage us to look at the people around us with a little more curiosity and perhaps a little more understanding.

There is also something to be said about the pace of life here. Not everything seems to revolve around urgency. The marketplace has its own rhythm, conversations take their time, and everyday interactions often carry a warmth that is easy to miss when we are in a hurry.

Perhaps this is what drew me to these photographs in the first place. The small gestures, expressions and interactions that make up the texture of everyday life.

Not every photograph contains a smile, and that is intentional. A smile can be an expression, but it can also be a feeling associated with a place — its people, their openness, their resilience and the simple humanity of everyday encounters.

This collection is my attempt to explore that feeling through photography.

I am happy to have these photographs become part of Kalaa Utsava Kodagu 2026, the tenth edition of the art festival, bringing together different forms of artistic expression in Virajpet. The exhibition offers an opportunity to share these observations with people beyond the camera and perhaps invite them to see Kodagu through a slightly different lens.

If you happen to visit, I hope you will spend a little time with these photographs. Look beyond the faces and see the relationships between people and their surroundings, the details of everyday life and the many stories that coexist within a small town.

Because sometimes, to understand a place, we need to look beyond its landscapes.

We need to meet its people.

Kalaa Utsava Kodagu 2026

10th Edition | 11 November – 31 December 2026

Location 01: Chombuka Art & Farm, Virajpet, Kodagu, Karnataka.

Photography is where technology meets art.

To break the rules, we must first learn to master them.

Over the past few months, I have been writing about my experience of photographing Hokkaido, Japan. Each story began with a particular visual curiosity — colour, monochrome, the behaviour of light, or the unexpected ways in which a camera can transform a landscape.

With Vibrant Hokkaido, I explored how custom camera profiles could translate an imagined visual language into photographs. What began as an attempt to recreate the atmosphere I associated with Japanese films and anime gradually led me towards a deeper understanding of luminance and the subtle tonal relationships hidden within snow.

Traditional Japan (Monochrome) took me in another direction. By removing colour, I discovered how form, texture, atmosphere and negative space could communicate a sense of timelessness.

Then came Silence in Three Movements, where I began noticing something I had not anticipated when I travelled to Hokkaido: landscapes that appeared to separate into distinct visual planes, almost as if multiple images had been arranged within a single frame.

These were different experiments, but they shared a common starting point.

I was learning to understand not just what my camera could capture, but how its technology could influence the way I saw the world.

Somewhere along the way, however, another question began to emerge.

That question is where Art of Seeing begins.

First, learn the rules

I have always been fascinated by the intersection of technology and art.

My background in technology has naturally influenced the way I approach photography. I enjoy understanding how things work, experimenting with settings, studying the behaviour of light and finding ways to make equipment perform according to a particular intention.

A camera is an extraordinary technological instrument. Aperture, shutter speed, focal length, focus, exposure, picture profiles and post-processing all influence the final outcome.

Understanding these fundamentals matters.

A shallow depth of field can isolate a subject. A long focal length can compress a landscape. A slower shutter speed can transform movement into a gesture. Exposure and tonal control can determine whether snow appears as a featureless white surface or a landscape filled with subtle patterns and depth.

These are not merely technical decisions. Each one changes how a photograph communicates.

For me, the first stage of learning photography was about understanding these possibilities and developing the ability to use them deliberately.

But mastering a technique is not the same as making a meaningful photograph.

Knowing how to create a blurred image does not explain why the blur should exist. Understanding exposure does not tell us what deserves to remain visible. Learning composition does not automatically reveal what a frame should make someone feel.

Technique gives us the vocabulary.

What we choose to say with it is another matter.

I am beginning to realise that the more comfortable I become with the technical side of photography, the more freedom I have to move beyond it.

Perhaps that is what it means to learn the rules before breaking them.

Not to abandon the rules, but to reach a point where they no longer dictate every decision.

When the subject is no longer enough

Wildlife photography often encourages us to pursue a familiar objective: find the subject, achieve focus, capture the behaviour and make the animal stand out against its surroundings.

There is nothing wrong with this. A technically accomplished portrait of a wild animal can be beautiful, powerful and important.

But I have increasingly found myself drawn towards photographs in which the animal is not necessarily the only thing that matters.

Sometimes, it is the relationship between the subject and the landscape that interests me. Sometimes, it is the way light separates one form from another. At other times, it is an imperfect, fleeting gesture that could never be reproduced in a conventional portrait.

The subject remains important, but the photograph begins to mean something beyond its identity.

_{A bird can become a line within a composition.}

_{A landscape can become a sequence of tones.}

_{A moment of movement can become an abstract gesture.}

And an apparently incomplete image can invite the viewer to participate in completing it.

These possibilities do not necessarily require elaborate manipulation or complicated equipment. Sometimes, they emerge simply by changing the way I look through the viewfinder.

The question gradually shifts from What am I photographing? → What is it about this moment that I want to communicate?

That shift has been particularly noticeable in the photographs I began making during my winter journey through Hokkaido.

Searching: discovering the image within the landscape

One of the most unexpected discoveries during the trip was how strongly a winter landscape could resemble an arrangement of separate visual layers.

Snowfields, belts of vegetation and distant woodland seemed to occupy distinct planes. Differences in tone, texture and atmosphere created boundaries that were not immediately obvious when looking at the physical landscape.

Through the camera, however, these relationships became remarkably pronounced.

I found myself looking at the scene differently. Rather than treating the landscape as a continuous environment, I began to see it as a composition of shapes, tonal regions and intervals.

The wildlife subject became part of this arrangement.

In Searching, a solitary crane occupies a distant ridge, surrounded by layers of winter woodland and snow. The bird is small within the frame, and that is precisely what makes the photograph interesting to me.

The landscape does not simply provide context for the animal. It determines how the animal is discovered.

The tonal transitions create the impression of three distinct visual planes, almost like separate images brought together within a single composition. The viewer’s attention moves across these layers before settling on the crane.

I did not set out to create an image about perception. I was initially interested in the landscape itself, however, the crane become merged made it further interesting.

Reviewing the photograph made me realise that the act of finding the subject had become part of the experience I wanted to convey.

The image asks for a little patience.

It rewards looking rather than merely glancing.

And that, for me, is an important distinction between documenting a subject and telling a story through it.

Reconstruction: allowing the viewer to participate

Another photograph emerged from a very different way of looking.

In Reconstruction, I approached the crane as though I were making an artistic study of its form rather than a conventional wildlife portrait.

The head, body and other elements do not resolve into one immediately recognisable, unified figure. Instead, they occupy different areas of the frame, with generous white space separating them.

The image invites the viewer to assemble these fragments mentally.

I am particularly interested in how our minds respond to incomplete information. We instinctively search for patterns, establish relationships and attempt to make sense of what we see.

A conventional portrait presents the subject as a complete form.

Here, I wanted to explore what might happen if I gave different parts of that form their own visual importance, almost as if I were sketching a portrait through separate studies rather than drawing it in a single continuous gesture.

The result sits somewhere between wildlife photography and abstraction.

The crane is still recognisable, but recognition is no longer the only objective. The spaces between the forms become as important as the forms themselves.

This was not an attempt to correct reality or improve the subject through technology.

It was an attempt to reinterpret what I had seen.

And that distinction matters to me. Photography need not always be a literal representation of the world. It can also be an expression of how an individual experiences and understands it.

Between presence and absence

Some photographs become interesting because of what they conceal.

In Between Presence and Absence, two cranes occupy the same snowy environment, but their forms are rendered differently. One is relatively distinct; the other dissolves into a soft, indistinct presence.

The contrast creates a tension between recognition and uncertainty.

A symmetry, yet not quite!

We know that another bird is there, yet we cannot experience both subjects in the same way. One commands attention through its clarity. The other becomes an impression, something we perceive without fully resolving.

I find this relationship more interesting than a photograph in which both birds are rendered with equal clarity.

The difference creates a visual dialogue between presence and absence, between what is immediately available to the eye and what remains just beyond its grasp.

It also reminds me that the photographic frame is never a neutral window onto reality.

Focus, depth of field, perspective and timing all influence what becomes prominent and what recedes. Even when I am photographing an entirely natural moment, the resulting image reflects a series of choices.

The photograph is a record of what was there, but it is also a particular way of experiencing it.

Emergence: when the environment becomes the subject

In Emergence, deer appear within a landscape of frost-covered branches and drifting fog.

The scene is not built around the immediate recognition of the animals. Instead, the surrounding woodland initially dominates the experience. Branches, tonal variations and mist create layers through which the deer gradually become visible.

The longer I look at the photograph, the more the relationship between subject and environment interests me.

The deer are not isolated from their surroundings. Their presence is inseparable from the atmosphere that partially conceals them.

This is another reason I have become increasingly interested in negative space and restrained compositions.

What we leave unresolved can be as important as what we reveal.

A photograph does not always need to announce its subject immediately. Sometimes, allowing the viewer to discover it creates a more engaging experience.

It introduces time into the act of looking.

The viewer has to pause, search and establish relationships within the frame.

And perhaps, in doing so, begins to experience the landscape a little differently.

Winter Calligraphy: when technique becomes expression

Of all the directions I explored, intentional motion blur has perhaps offered the clearest example of technique becoming a creative language.

In Winter Calligraphy, the movement of a bird is transformed into a flowing gesture against a pale winter background.

The resulting form is not intended to describe every feather or preserve the conventional sharpness expected of a wildlife photograph. Instead, the movement itself becomes the subject.

The bird leaves behind an impression, almost like a brushstroke across an otherwise quiet canvas.

This is where I find the relationship between technology and art particularly fascinating.

Shutter speed is a technical parameter. Motion blur is a physical consequence of how movement is recorded during an exposure.

But the expressive possibilities of that blur depend on intention.

A blurred photograph caused by an accident and a blurred photograph created to convey movement may share a technical characteristic, yet they do not necessarily share the same artistic purpose.

The difference lies in the decision behind the image.

I am not suggesting that every technical imperfection should be reclassified as art. Nor do I believe that abstraction automatically makes a photograph more meaningful.

What interests me is the freedom to choose a different visual outcome because it serves the story I want to tell.

Once I understand what a technique does, I can begin asking what else it might express.

That is when experimentation becomes more than an exercise.

It becomes a way of seeing.

Ghost of the North: the landscape completes the story

The red fox crossing the blue-toned snow offered another lesson.

Initially, the contrast between the warm colour of the fox and the cold blue landscape seemed to provide the photograph’s defining characteristic.

But as I spent more time studying the image, my attention shifted towards the snow.

The wind-shaped patterns across its surface created pathways through the frame. Subtle tonal differences established depth and direction. The lighter and darker areas guided the eye towards the fox.

The snow was doing more than providing a background.

It was creating the composition.

I had explored this idea earlier in Vibrant Hokkaido, where I wrote about what I came to think of as Snow Luminance Control: preserving the tonal relationships that give winter landscapes their depth and character.

With Ghost of the North, that technical exploration became part of a broader artistic consideration.

The fox did not need to dominate the frame.

Its presence was enough to complete a visual arrangement that the landscape had already established.

This is an idea I want to carry forward into my photography: the most important element in a photograph is not necessarily the largest, sharpest or most visually striking one.

Sometimes, it is the element that brings everything else together.

The Watchers: turning the gaze around

Photography is often described as an act of observation. We look at the world, select a moment and decide how to frame it.

But what happens when the act of looking becomes part of the photograph itself?

In The Watchers, the people observing wildlife become part of the visual narrative. Their attention, posture and relationship with the unseen subject tell a story that extends beyond the animal they have come to see.

The photograph becomes less about a single encounter with wildlife and more about the human desire to witness it.

We travel to remote places. We wait for encounters. We raise our cameras and try to preserve something of an experience that may last only a few seconds.

Yet the photograph can reveal more than the intended subject. It can reveal our own presence within the experience.

This has made me increasingly conscious of the relationship between the photographer, the subject and the eventual viewer.

Every photograph begins with someone looking.

Every frame includes certain things and excludes others.

And every viewer brings their own memories, assumptions and imagination to the finished image.

The photograph, in that sense, is never entirely complete when I press the shutter.

Its interpretation continues long after the moment has passed.

From photographs to stories

Looking back at these images, I can see that they do not all belong to the same visual style.

Some are minimalist. Some rely on atmospheric layers. Others explore fragmentation, movement or the relationship between humans and wildlife.

What connects them is not a common technique.

It is a common question.

That question has begun to change how I approach the entire process.

Earlier, I might have started with a technical objective: create a particular colour palette, experiment with monochrome, preserve the tonal detail in snow or achieve a specific depth of field.

I still value those objectives. They remain essential tools.

But increasingly, I want to begin with a feeling, a question or an idea.

I want to understand why a scene holds my attention before deciding how to photograph it. I want to explore whether a conventional portrait is the right response, or whether the subject might be better expressed through abstraction, negative space, movement or its relationship with the environment.

And I want to spend more time considering how individual photographs might relate to one another.

A single photograph can be compelling. A carefully considered sequence can create something different: a narrative that develops as the viewer moves from one image to the next.

This is where titles, sequencing and the relationships between photographs become important.

A title can direct attention without prescribing an interpretation. A sequence can introduce an idea, complicate it and eventually resolve it. A collection can become more than the sum of its individual images.

These are possibilities I am only beginning to explore seriously.

But they represent an exciting next step in my photographic journey.

Learning to break the rules

I often return to a simple thought:

Technology gives us the means to control the image. Art gives us reasons to make those choices.

Neither exists in isolation.

Without understanding the camera(tool), it can be difficult to translate an intention into a photograph. Without an artistic intention, technical mastery can become an end in itself.

For me, the challenge is to marrying two together.

I want to continue learning the technical foundations of photography, experimenting with equipment and understanding the science behind the images I make. But I also want to become more comfortable questioning familiar expectations.

_{Why must a wildlife portrait always be sharp?}

_{Why must the subject dominate the frame?}

_{Why should a landscape always convey depth in the conventional sense?}

_{Why must a photograph explain everything it contains?}

There are no universal answers to these questions, and I am not interested in breaking rules simply for the sake of doing so.

I want to understand the rules well enough to recognise when following them serves the image — and when departing from them might reveal something I would otherwise miss.

That, to me, is the difference between applying a technique and developing a personal visual language.

An ongoing journey

Art of Seeing is not a finished body of work, nor is it an attempt to define a single style of photography.

It is an evolving exploration of how I look at the world and how that experience might be translated into images.

Hokkaido provided the setting for many of these discoveries, but the questions extend well beyond Japan and winter wildlife. I want to explore them in other landscapes, with other subjects and under entirely different conditions.

I also want to continue bringing these photographs together as a coherent body of work — through essays, carefully sequenced series and, eventually, the possibility of an exhibition.

More importantly, I want to resist the temptation to settle too quickly into a familiar way of making images.

There will always be more techniques to learn, more equipment to understand and more ways to process a photograph. But I suspect the more interesting journey lies in discovering what to do with that knowledge.

Perhaps the next stage of photography is not about learning to control every aspect of the image.

Perhaps it is about knowing when to let the image become something unexpected.

I began this journey wanting to make better photographs.

I now find myself asking a different question:

What kind of stories can I tell through the way I see?

That is the question I hope to keep exploring.

And that is what Art of Seeing means to me.

Some more attempts which are yet to be fine tuned either by revisualing or reprocessing

Witnesses, making the Snow monkey enjoying celebrity status with people around her with cameras, tripods or just simply amazed to watch her

A celebrity, indeed!

In between the line(s)..!

An engineering work-in-progress note

This note documents an early development and experimentation phase of Grove v0.1. The implementation and experiments described here were conducted in a contained, local development environment, using Ollama and locally hosted models Gemma 4:31B Dense model on an Apple Mac (M3 Max).

The architecture, model configurations, runtime parameters, agent workflows and performance observations represent the state of experimentation at the time of writing. They are not production benchmarks, final specifications or commitments to a particular implementation.

Grove is actively evolving. These configurations, architectural decisions and experimental observations may change as development progresses, further testing is conducted, and additional models and capabilities are introduced. The purpose of publishing this note is to share the engineering process transparently, including the decisions, observations, limitations and unresolved questions.

Most importantly the journey, evolution and learning

1. The Agent Harness as a First Principle

1.1.What Is an Agent Harness?

A language model can understand instructions, reason through problems and generate responses. But building a useful AI agent requires more than connecting an application to a model. The agent also needs a way to manage its context, use tools, execute tasks, handle errors and decide what to do next.

This surrounding system is called the agent harness.

In simple terms, think of the model as the engine and the harness as the machinery that allows the engine to do useful work. The model provides the intelligence; the harness provides the structure, controls and environment in which that intelligence operates.

A useful way to express this relationship is:

Some of its key architectural ideas include:

• Session management: Maintains an event log of the interaction and derives the conversation history that is presented to the model.

• System prompt assembly: Brings together tool descriptions and prompt sections to construct the model’s working instructions.

• Tool execution: Provides a scoped tool registry and a guarded execution pipeline.

• Agent and execution loop: Defines how agents are registered and how their execution is driven.

• Capability seams: Creates boundaries that allow underlying providers to be replaced without having to redesign the capabilities that depend on them.

The important idea is not simply that the system supports plugins. It is that the harness itself is designed to be modular, replaceable and extensible.

1.3. How Grove Interprets This

Grove follows the same broader principle: the harness should not be an incidental layer hidden inside an agent implementation. It should be an independent, first-class part of the architecture.

Our interpretation is that agents should define what work needs to be done, models should provide the intelligence, and the harness should govern how that work is carried out.

This separation allows Grove to introduce new agents and experiment with different models without having to rebuild the underlying runtime each time. The harness provides the common execution foundation, while individual agents bring their own capabilities, workflows and domain-specific instructions.

For Grove, this is both an architectural principle and an ongoing engineering direction. The objective is to build a foundation where models and agents can evolve independently, while the harness provides consistency, control and extensibility across the system.

A useful working formulation for understanding modern AI agents is:

In this framework, the language model provides the underlying intelligence, while the harness serves as the runtime that converts that intelligence into actual work.

The harness constitutes the infrastructure that couples an LLM to the world through a loop, tools, orchestration, safety controls, context management, and extension surfaces. Specifically, the harness is critical because it mediates trajectory control, action execution, tool use, observation, and the interpretation of feedback.

Grove adopts the philosophy by treating its harness as both a first principle—the governing rules every agent run must obey regardless of the model’s behaviour—and a first-class citizen.

This is reflected in the implementation: Grove maintains the harness as its own distinct component within @grove/runtime, where it manages the run lifecycle, deadlines, cancellation, error policies, and provides a recorder with redaction for observability.

The distinction matters. The model should be replaceable without fundamentally changing the runtime that governs how an agent operates.

2. Pluggable and Composable Architecture: The DeepSeek Harness Case Study

Research into dynamic composition indicates that modern software, including evolving agent harnesses, increasingly requires composability.

A primary example is the DeepSeek Harness (dsh), which employs an “everything-is-a-plugin” design.

DeepSeek Harness architecture

The dsh avoids a privileged core; instead, it is extended by mounting plugins that can be unwound upon unloading.

Its architecture is segmented into packages owning specific context keys:

• core/session: Owns the ctx.sessions append-only SessionEvent log.

• core/system-prompt: Manages tool-schema and prompt-section assembly (ctx.systemPrompt).

• core/tools: Handles the guarded execution pipeline and scoped tool registry (ctx.tools).

• core/agent and core/agent-loop: Define the Agent interface, live registry, and default driver (ctx.agents, ctx.agentLoop).

Further architectural details are available in the DeepSeek Harness architecture documentation and its core subsystem documentation.

Key technical features of dsh

Session log

The session log is the primary source for model context. It uses deriveMessages() to project history; notably, it retains failed, cancelled, or retried attempts in assistant/attempt without adding them to the model’s history.

Reference: DeepSeek Harness – Session subsystem.

Capability seams

dsh utilises “seams” that allow for provider swaps without requiring provider forks. For example, shifting filesystem and subprocess providers to a remote sandbox moves Bash, LSP, and PTY collectively.

Comparison with Grove

While the available evidence does not detail every architectural component of Grove’s harness, it shows that Grove similarly implements pluggable agents via an AgentDefinition contract in @grove/sdk, ensuring that registering a new agent does not change the runtime.

Like dsh, Grove separates the runtime from the agent implementations.

The comparison here is architectural, rather than a claim that both systems have identical implementations.

3. Current Implementation: Grove Local Testing

Grove’s harness is currently being hosted and tested on a local Mac using Ollama.

• For testing purposes, a Gemma 4 31B dense model has been run locally using the gemma4:31b model identifier.

• Additionally, smoke tests have been conducted on an Apple M3 Max (CPU only) using qwen2.5:3b via Docker Desktop.

The local-inference infrastructure and its configurations are documented in the Grove local-inference README.

These are development and experimentation environments, not production deployment configurations.

4. Initial Release Model Strategy

The initial release involves consideration of the following models:

• Gemma (latest version)

• LLama Muse

• Qwen 3.8

• GLM 5.3

• Kimi 3.1

The intention is to use a model router to manage these multiple models. The design of this router or the planned implementation is using Mixture of Experts (MoE).

5. Roadmap: Agentic Pipeline Evolution

Grove’s agentic pipeline is intended to evolve in the following order.

5.1 Research Agent

Intended for evidence-grounded research, literature reviews, and white papers.

It utilises a staged pipeline with human review gates and specific roles for extraction, drafting, and verification.

The current Research Agent implementation can be explored in the Grove Research Agent source.

5.2 Documentation Agent

Aimed at documentation drafting and maintenance.

This agent is described in the project’s documentation but has not yet been registered or built in v0.1.

5.3 Coding Agent

Designed as a repository-aware engineering assistant for Nx consolidation.

Its goal is to move from a polyrepo to a modular monolith with an application layer encompassing the backend, frontend (with future mobile/desktop apps), and shared libraries.

It will utilise a “read-only analysis first” approach followed by gated patch proposals in a sandbox.

These three agents represent the intended evolution of Grove’s agentic pipeline, not a claim that all three are already implemented.

6. Interface and Distribution

Grove v0.1’s interface is interpreted to consist of a CLI, which serves as the composition root to wire the system together.

The technology stack for the harness and SDK is TypeScript.

Grove is a potential open-source release under the name Parjanya by Phagyul AI Systems Pvt Ltd.

The interface and distribution strategy remain subject to further development.

7. Experimental Record: Gemma 4 31B on Local Ollama

This section documents the experimental configuration and observations from running Grove’s research agent locally.

The run was hosted and tried on an Apple Mac M3 Max using Ollama running gemma4:31b.

The following records cover token consumption, observed performance, research-agent limits, sampling parameters, engine-level settings, and related environment configuration.

7.1 Token Usage and Performance

Individual model calls

The process involved varying input and output token counts across different sequences.

These are selected individual model-call observations from the experimental run.

Total usage

Across 38 model calls, the total token usage for writing the first post draft was:

Performance records

The recorded performance observations were:

• Prompt evaluation speed: Ranged from approximately 79.60 tokens per second (seq 1531) to 118.39 tokens per second (seq 0).

• Generation speed: Ranged between 8.49 tokens per second (seq 946) and 13.61 tokens per second (seq 925).

These figures are observations from the recorded run and should not be interpreted as controlled or generalised model benchmarks.

7.2 Research Agent Limits and Thresholds

The following limits and thresholds were used for the research agent.

The timeout and retry values are defined in the Gemma expert configuration. The context budget is defined in the research-agent harness configuration, and the default deadline is defined in the Research Agent implementation.

Relevant implementation reference: Grove Research Agent.

7.3 Ollama Sampling Parameters

The sampling settings recorded for the gemma4:31b model were as follows:

Both observations are retained here rather than choosing one value without further verification.

For parameter definitions and their behaviour, refer to the official Ollama Modelfile documentation.

7.4 Model Engine and Environment Settings

llama-server parameters

The following engine-level parameters were recorded when Ollama started the llama-server for this run.

These are the recorded llama-server settings from the model-load log.

Related Ollama environment configuration

The following values are present in Grove’s Docker Compose configuration for local inference:

Important distinction: These values are documented in Grove’s Docker Compose configuration. The Gemma experiment described above used the Ollama application on the Mac, not the Docker Compose Ollama service. Therefore, these environment variables should not be interpreted as confirmed settings for that specific experiment.

Configuration reference: Grove local-inference Docker Compose.

For additional context on Ollama environment variables and context-window configuration, refer to the official Ollama FAQ.

7.5 Conflicts and Uncertainties in the Experimental Configuration

There are two configuration discrepancies worth documenting.

1. Conflict in top_p

There is a conflict between the llama-server load log, which specifies top_p as 1.0, and the Ollama model parameters, which list it as 0.95.

2. Uncertainty in context length

It is unclear which setting took precedence during the run:

• The llama-server was started with a context size of 32,768 tokens.

• The OLLAMA_CONTEXT_LENGTH environment variable in the related Docker configuration was set to 16,384.

Since the Docker configuration was not used for the documented Mac experiment, the latter value does not establish the effective context length for that run.

The sources do not specify any other limits or parameters beyond those listed above.

8. Evolving Grove: Areas of Continued Development

Grove v0.1 is an initial foundation, not a finished system. As the architecture evolves and additional models are integrated, several areas will receive further attention, experimentation and refinement.

Some of the areas we intend to explore include:

• Multi-model intelligence and routing: Expanding model support and developing the model router, including the exploration of a Mixture of Experts (MoE) approach to direct tasks to suitable models based on their capabilities.

• Multimodal research capabilities: Further developing image-processing workflows, including handling high-bit-depth RAW files, image analysis and more effective filtering and evidence extraction.

• Documentation automation: Extending the Documentation Agent to support practical engineering and publishing workflows, including blog posts, READMEs, technical documentation and white papers, with integration into the relevant model services.

• Interfaces and accessibility: Exploring interfaces beyond the initial CLI as Grove’s runtime and agent capabilities mature.

• Performance and model evaluation: Comparing models across different hardware and execution environments, with a focus on inference speed, token consumption, reliability and task quality.

These are evolving areas of engineering work rather than fixed specifications. Their implementation and priorities will be shaped by experimentation, practical usage and feedback as Grove develops.

For a more detailed view of ongoing work, planned enhancements and implementation priorities, refer to the Parjanya Engineering Issues and Roadmap.

For shipped improvements and version-specific changes, follow the Parjanya Release Notes.

9. References and Evidence

The following public references provide context for the architecture, implementation and experimental configuration.

Architecture and agent harness

1. DeepSeek Harness – Official Repository

2. DeepSeek Harness – Architecture

3. DeepSeek Harness – Core Subsystems

4. DeepSeek Harness – Session Subsystem

Grove implementation

5. Grove – Local Inference README

6. Grove – Local Inference Docker Compose

7. Grove – Research Agent Implementation

Model and runtime documentation

8. Ollama – Modelfile Reference

9. Ollama – FAQ

10. Google – Gemma 4 Model Card

Internal experimental evidence

The experimental figures in this note are based on the local run records, Ollama model-load logs, request logs, Grove configuration and run records captured during the experiment.

These are internal development artifacts and are not publicly accessible through the links above. The original evidence export records the associated source files, line ranges, quoted passages and file fingerprints.

The local logs and configuration snapshots are retained as the supporting engineering record.

Grove v0.1 is a work in progress. This note captures an early point in its development, not a final destination. The architecture will evolve, the models may change, and the experimental parameters will be revisited as the system matures. We are sharing the process as it happens, including what works, what remains uncertain, and what we learn along the way.

The GPUs were taking too long to start.

Fix the startup time, make the model faster, and the problem would be solved.

It turned out to be much more interesting than that.

Over the past two months, I ended up digging through cold starts, GPU behaviour, model loading, storage bottlenecks, inference paths, numerical differences between GPUs, and even the way a tiny change in floating-point computation can alter a photographic verdict.

More importantly, it reminded me of something I have learned repeatedly while building products:

The hardest engineering problems are often not the ones that look technically complicated. They are the ones where speed, cost, correctness and user experience collide.

First, what actually happens when you upload photographs?

When you upload a shoot to Parjanya, two things happen.

First, a quick technical check runs immediately.

Then each photograph goes to a GPU machine running our vision-language model. That model produces the critique, identifies areas for improvement, and eventually gives the photograph a Keeper / Review / Skip verdict.

The interesting part is that those GPU machines aren’t running all the time.

At our current scale, keeping GPUs running continuously would mean paying for machines that spend most of their time doing nothing.

So we turn them off when there is no work.

That gives us a very attractive infrastructure cost profile — but introduces a cold start.

When your upload arrives and nothing is running, we have to:

1. Request a GPU machine.

2. Boot it.

3. Download the software.

4. Download the AI models.

5. Copy the models to local storage.

6. Load the models into GPU memory.

7. Process your first photograph.

At the end of July, that entire journey could take up to nine minutes.

For someone trying Parjanya for the first time, nine minutes doesn’t feel like “AI infrastructure is warming up.”

It feels like the product is broken.

That was the problem I wanted to solve.

The numbers before and after

Here is what we measured.

Note: The 27 September numbers are from single cold starts; I’ll keep measuring this as more real-world uploads arrive. But getting there wasn’t one optimisation.

It was a series of relatively small discoveries.

And the first discovery was perhaps the most embarrassing one.

1. I couldn’t optimise what I wasn’t measuring

The first thing I discovered was that roughly four minutes of every cold start wasn’t being measured at all.

Our logs started measuring the model’s work only after the machine had already:

• booted,

• downloaded the software,

• copied the model,

• and reached the point where inference could begin.

Everything before that was invisible.

There was another problem.

Our machine logs were identified using network addresses.

Those addresses get reused.

So when multiple machines started around the same time, it wasn’t always possible to tell which log belonged to which machine.

In other words, I was trying to optimise a system without having a reliable stopwatch.

So the first optimisation was not an optimisation at all.

We changed the observability.

Every machine now has a unique identity in the logs, and the boot process records timestamps for every major stage — from “machine running” all the way to “model loaded.”

And I changed another rule internally:

A performance change doesn’t get a tick simply because the code was merged. It gets a tick only after it is deployed and measured in the Production.

That sounds obvious.

In practice, it changes how you work.

2. Wake the GPU when the work arrives

The next surprise was the queue.

Our machines were being started based on the length of the work queue.

That sounds sensible.

It wasn’t fast.

The queue publishes its length periodically. Another mechanism checks that number periodically. Only then does the system decide that it needs another machine.

We measured the delay.

2 minutes 44 seconds.

Before the GPU had even started doing anything, almost three minutes had disappeared.

So we changed the trigger.

The same operation that puts the photograph into the queue now directly requests a machine.

On the first cold start after that change, the machine was requested 15 seconds after the first photograph arrived.

That was a much more direct relationship:

Work arrived → ask for compute.

We also changed how many machines we started.

Previously, the system essentially reacted to “is anything waiting?”

That meant a 13-photo upload could potentially wake every available machine.

Now the fleet sizes itself against the backlog.

3. If two things don’t depend on each other, why run them sequentially?

A new machine needs to download two large things:

• our analysis software — roughly a 5 GB container image

• the model weights

Previously, these happened one after another.

There was no real dependency between them.

So we made them happen simultaneously.

The machine now waits for both operations to complete before loading the model.

That saved approximately 110 seconds on each of the two machines we measured.

There is a small engineering detail here that I think is worth mentioning.

Once you start overlapping operations, measurement becomes harder.

If you only record the end time, you lose the ability to understand where the time actually went.

So we record the beginning and end of each stage.

That way, when operations overlap, we can still reconstruct the timeline.

And if a download fails, the machine stops cleanly rather than continuing with a half-ready environment.

Fast is useful.

Fast and observable is much more useful.

4. The fastest download tool couldn’t solve a disk bottleneck

By mid-September, we had reached another bottleneck.

Model copying had become the slowest part of the boot.

The obvious idea was:

Can we download it faster?

So we tried six different ways of copying the same 21 GB.

The result?

They were all within about 5% of one another.

That was the clue.

We separated network performance from disk performance.

The network could deliver around 780 MB/s.

The local disk could write only around 165 MB/s.

The network wasn’t the bottleneck.

The disk was.

That meant no amount of cleverness in the download tool was going to solve the fundamental problem.

So we asked a different question:

Can we move less data?

That worked.

Our main model runs in a compressed 4-bit representation.

Previously, every machine downloaded the full 17 GB model and compressed it during startup.

Instead, we now compress it once, store the compressed version, and download approximately 6 GB.

That changed:

• model loading: 51s → 20s

• copying: 164s → 102s

• machine readiness: approximately 90 seconds sooner

We also verified something that mattered enormously to me:

The stored weights were byte-for-byte identical to the weights each machine previously generated for itself.

And a production photograph re-analysed with them produced exactly the same result.

This was a good reminder of a general optimisation principle:

When the disk is the bottleneck, reducing bytes can be more valuable than making the pipe faster.

5. Then I started looking inside the model

Once the infrastructure was reasonably healthy, I turned to the actual photo analysis.

Parjanya currently uses two GPU types:

• NVIDIA T4 — older, cheaper

• NVIDIA A10G — newer, faster

A photograph took roughly 106 seconds on a T4.

So we profiled the model layer by layer.

The first major discovery was a particular operation that cuts the photograph into small patches for the model to process.

That single layer was taking around 31 seconds.

The T4 doesn’t have hardware support for the number format that layer was using — bfloat16 — so it was falling back to a very slow path.

Running that layer in standard 32-bit precision took only milliseconds.

That brought the photo from roughly:

106 seconds → 75 seconds

Then we found another optimisation in the attention mechanism.

Switching the language part of the model to a simpler attention implementation brought the photo down to around:

56 seconds

Our first production T4 run measured 65 seconds from start to finish.

At this point, it looked like a straightforward win.

And then things became much more interesting.

6. Faster didn’t necessarily mean the same answer

This was the part I didn’t expect.

The speed improvements changed numerical behaviour.

Not dramatically.

Not obviously.

But enough.

In the first optimisation, only 527 out of 3.7 million values moved by the smallest step representable by the format.

That sounds insignificant.

For most photographs, it was.

But for photographs sitting close to a decision boundary, it could change the final verdict.

So we did something that became one of the most important experiments in this entire exercise.

We ran the same photographs through both GPU types.

Each GPU was perfectly repeatable.

Same photograph.

Same GPU.

Same answer.

Every time.

But the two GPUs did not always agree with each other.

Across 64 varied production photographs, they agreed on 45.

And 15 photographs received a Keeper on one GPU and Skip on the other.

That changed how I thought about performance optimisation.

The question wasn’t:

“Did the model get faster?”

It became:

“Did the model get faster while preserving the thing the user actually cares about?”

In our case, that thing isn’t milliseconds.

It’s the verdict.

7. We chose consistency over raw speed

That led us to three decisions.

First: stability before speed

Across 159 photographs, the faster T4 path agreed with the A10G about as often as the old T4 path did:

122 vs 125.

Within the noise of the experiment, speed hadn’t meaningfully damaged consistency.

That gave us confidence to keep the optimisation.

Second: prefer one GPU type for new work

We now start T4 machines first and use the A10G only when a T4 isn’t available.

Since August, T4 launches succeeded 47 out of 48 times.

That means almost every new photograph is now analysed on the same class of GPU.

Third: make the curation rule less sensitive to a single tone issue

Our curation layer turns model findings into the final verdict.

We found that a single severe tone-related issue — exposure, dynamic range or noise — was one of the places where the GPU differences mattered most.

So we changed the rule:

A single severe tone problem by itself now counts as moderate rather than immediately sending the photograph to Skip.

Importantly, we didn’t rewrite history.

Photographs analysed before these changes retain the verdict they were given.

The new rules apply to new analyses.

That is deliberate.

If someone has already acted on a verdict, silently changing it underneath them isn’t something I want to do.

8. Some of the best engineering decisions were the things we didn’t ship

One of my favourite parts of this exercise is not actually the list of things we improved.

It is the list of things we tried, measured, and deliberately decided not to ship.

That distinction matters.

When you’re building a product, especially one involving AI infrastructure, it is very easy to fall into a particular pattern:

“This technology is faster, therefore we should use it.”

Or:

“This optimisation is supposed to help, therefore let’s implement it.”

But an engineering optimisation is only useful if it improves the system under our actual constraints.

For Parjanya, those constraints aren’t just latency.

We also care about:

• GPU cost

• availability

• reliability

• consistency of AI verdicts

• privacy of photographers’ work

• operational complexity

• and whether an improvement is actually meaningful to the person using the product

That led to several experiments that ended with a very satisfying outcome:

We didn’t ship them.

Here are some of them.

1. “Can we just download the model faster?”

This was the most obvious idea when model copying became the slowest part of the boot.

We were moving a lot of data, so the natural assumption was:

If downloading is slow, find a faster way to download.

So we tried six different approaches to copying the same 21 GB.

The result was surprisingly boring.

All six were within roughly 5% of one another.

At first, that was frustrating.

Then it became useful.

We separated the network from the local disk and measured both independently.

The network could deliver approximately 780 MB/s.

The local disk could write only around 165 MB/s.

That made the bottleneck obvious.

The network was capable of delivering data much faster than the machine could actually write it.

So changing the download tool wasn’t going to solve the problem.

We could spend days making the network path more sophisticated and still be limited by the disk.

The lesson was simple:

Before optimising a pipeline, find the slowest component in the pipeline.

In our case, the answer wasn’t “download faster.”

It was:

Move less data.

That led to the pre-compressed 4-bit model and saved roughly 90 seconds of machine startup.

2. “What if we put the software image on the fast disk too?”

Once we discovered that the local disk was the bottleneck, another idea naturally came up.

The machine has a fast local disk.

Why not put more of the boot assets there?

The problem was that the fast disk was already doing the work of receiving and storing the model.

Adding the software image would mean more data being written to the same bottleneck.

In other words, we would have been trying to fix a traffic jam by adding more cars to the road.

There was also another subtle point.

Moving the software image to the fast disk could improve one stage while making another stage slower because of the additional writes.

So we decided against it.

This is a useful pattern in performance engineering:

A faster component doesn’t automatically make every workload faster.

Sometimes the right optimisation is to reduce the amount of work that reaches the fast component in the first place.

3. “Can we load both models at the same time?”

Parjanya uses more than one model during the analysis process.

So another obvious question was:

If we load them sequentially today, can we load them simultaneously?

Normally, parallelising independent work is a good optimisation.

And we did test it.

But by the time we had already pre-compressed the main model, the combined loading time of the two models had fallen to roughly 22 seconds.

At that point, parallelising them didn’t leave us much to win.

This is an important distinction between a technically valid optimisation and a worthwhile optimisation.

Yes, we could make two operations overlap.

But if the total time we’re trying to eliminate is already small, the complexity isn’t justified.

So we left it alone.

Not every measurable optimisation is worth implementing.

4. “Should we warm up the model with a fake photograph?”

This one came from an observation that initially looked quite convincing.

We thought the first photograph processed by a newly started GPU might be dramatically slower than the photographs that followed.

If that were true, we could start the machine, process a dummy photograph, throw away the result, and then give the user the “real” warmed-up inference.

It sounded reasonable.

So we measured it properly.

The result?

The first photograph on the same GPU type was at most around 12 seconds slower than subsequent photographs.

That wasn’t remotely large enough to justify processing an extra photograph every time a machine started.

And there is a deeper reason why this matters.

A cold start already happens because a real photograph is waiting.

If we process a fake photograph first, we haven’t eliminated the work.

We’ve simply moved it earlier in the sequence.

So instead of:

Start → analyse user’s photo

we would have:

Start → analyse fake photo → analyse user’s photo

The user would still be waiting.

We decided not to do it.

This was a good reminder that sometimes the best optimisation is recognising that a cost cannot actually be removed — only moved around.

5. “Can we cache the fixed part of the prompt?”

This one looked much more promising.

Every photograph we send to the model comes with a fairly long set of instructions.

Most of those instructions are identical from photograph to photograph.

That immediately suggests caching.

If the instructions don’t change, why make the model process them repeatedly?

So we investigated it.

The surprise was where the time was actually going.

Only around 490 tokens come before the photograph in the current prompt structure.

That’s roughly a second of work.

The much larger component we had mentally categorised as “prompt processing” was actually the vision side of the model processing the photograph.

So caching the fixed text wasn’t going to give us the large improvement we had imagined.

There was another complication: separating and computing that part independently changed the model outputs.

So we would have been taking on additional complexity and potentially changing verdict behaviour for a relatively small performance gain.

We didn’t ship it.

However, this investigation did reveal a more interesting possibility.

If we reorder the instructions so that the reusable part comes before the photograph, a much larger portion of the prompt could potentially become cacheable.

That is now a separate future experiment.

And, like every model optimisation we’re considering, it will have to pass the same verdict-consistency tests before it goes anywhere near production.

6. “What about FlashAttention 2?”

If you’ve spent any time optimising transformer inference, FlashAttention is an obvious thing to investigate.

We did.

The problem was much simpler than benchmarking:

It doesn’t run on the T4.

Our infrastructure deliberately uses the T4 because it is cheaper and generally available for our workload.

So an optimisation that requires hardware we aren’t primarily running on isn’t really an optimisation for Parjanya.

It might be a useful path if our hardware strategy changes.

For the current system, it isn’t.

7. “Why don’t we just run everything on the faster A10G?”

This is probably the most tempting option.

The A10G is faster.

If the goal is simply to reduce inference time, why not use it everywhere?

Because the GPU isn’t free.

The A10G costs more per photo.

It is also harder to obtain consistently.

In one of our three zones, it isn’t offered at all.

And in 3 of 54 recent scale-ups, an A10G wasn’t available in any zone.

So we would be paying more for a resource that isn’t always available, while our cheaper T4 machines were giving us sufficiently good performance and, importantly, consistent verdicts.

There is also a product consideration here.

A photograph isn’t just a unit of GPU work.

It is someone’s photograph.

If changing hardware can change a borderline Keeper / Review / Skip decision, then “faster GPU” isn’t automatically better.

For our current workload, the T4 gives us a combination of:

reasonable inference time + lower cost + much better availability + predictable behaviour.

That’s a trade-off we’re comfortable with today.

It may not be the right trade-off forever.

The pattern behind all these decisions

Looking back, there is a common thread in almost every experiment we abandoned.

We started with a plausible hypothesis.

Then we measured it.

Sometimes the measurement disproved the hypothesis.

Sometimes the improvement was real but too small.

Sometimes the improvement solved one problem while making another worse.

And sometimes the technology simply didn’t fit our actual infrastructure.

That is why I increasingly think of optimisation as a constraint-solving exercise, rather than a race to make a benchmark number smaller.

For example:

And that leads to perhaps my favourite lesson from this entire exercise:

An optimisation isn’t valuable simply because it makes something faster.

It is valuable when it improves the product without creating a worse trade-off somewhere else.

For a developer, that might mean not introducing complexity for a 2% gain.

For a founder, it might mean not doubling infrastructure costs to shave a few seconds off a workflow that users don’t actually care about.

And for a photographer using the product, it might mean accepting a slightly slower GPU if that gives us a more consistent verdict and lets us keep the economics sustainable.

The engineering work is therefore not simply:

Make it faster.

It is:

Make the right thing faster, at the right cost, without compromising what the product is supposed to do.

That distinction has probably been the most valuable outcome of this entire cold-start exercise.

9. Why aren’t we just keeping a GPU warm?

This is probably the obvious question.

If cold starts are the problem, why not simply keep a GPU running?

Because infrastructure economics matter.

Today, Parjanya sees well under one cold start per day.

Most uploads are batches, which means hundreds of photographs (sometimes 500-1000 batches and multiple users) can share the cost of a single cold start.

Keeping a GPU running 24×7 to eliminate a wait that happens occasionally isn’t obviously a better product decision.

There are several possible versions of a warm pool:

Stopped pool

Boot machines once, put the software and models on them, then stop them.

Starting one should take roughly 1.5–2.5 minutes rather than around four.

Warm during working hours

Keep one machine running during defined hours.

Always on

Keep one GPU running continuously.

Then the first verdict would mostly be limited to inference time — around a minute on a T4.

The last option is obviously the most expensive.

So we haven’t done it.

Yet.

What would change my mind?

Several cold starts per day.

Several accounts uploading regularly.

Faster user growth.

Users abandoning the product after their first upload.

Or a product promise that requires a genuinely fast first result.

In other words:

The right infrastructure architecture depends on the workload, not on what looks technically elegant.

10. There are bigger optimisations ahead

The current work made the infrastructure we already have faster.

The next improvements require more fundamental changes.

One possibility is reordering the model instructions so that the reusable portion can be cached.

Today, much of the prompt comes after the photograph.

Because the photograph changes every time, the model cannot simply reuse everything.

Moving the stable instructions before the photograph could make much more of them cacheable.

But this needs to be tested carefully.

Even tiny numerical changes can move borderline verdicts.

So we would run the same verdict-agreement tests before shipping it.

What about vLLM?

Another possibility is moving from the standard Python model runtime to a dedicated serving engine such as vLLM.

These systems are designed to make better use of GPUs by:

• interleaving requests,

• managing model memory,

• caching repeated prompt text,

• and improving throughput.

It is tempting to assume that this will automatically make Parjanya dramatically faster.

I don’t.

Our machines already analyse roughly one photograph per minute, and we overlap fetching the next photograph with analysing the current one.

So I expect a potential double-digit percentage improvement in photos per hour, rather than some magical 10× transformation.

Before adopting it, I’d want to prove five things:

1. Our exact vision-language model works correctly.

2. The 4-bit weights work correctly.

3. Throughput improves on identical photographs.

4. Verdicts remain consistent.

5. Cold-start time doesn’t get worse.

The last one is particularly important.

A serving engine that makes inference faster but makes startup slower might not actually improve the experience.

11. And what about a hosted AI API?

This is another obvious architectural option.

Send the photograph to a hosted model API.

No GPU cold starts.

No idle machines.

No GPU fleet management.

From a purely operational perspective, that sounds attractive.

But there is another consideration that matters much more for Parjanya.

Photographers upload real work.

A lot of it is unpublished client work.

Sometimes it is work covered by contractual obligations.

So “processed on infrastructure we control” is not merely an infrastructure preference for us.

It is part of the product promise.

A hosted API would mean sending photographs to another company.

That makes data handling the first question, not performance.

Even if that question were resolved, there are other trade-offs:

• per-call costs grow with volume,

• external outages become part of our pipeline,

• rate limits become another dependency,

• and a provider could change its model and potentially change verdicts.

I can see a hosted model eventually playing a role as a fallback for bursts or GPU shortages.

But I don’t see it replacing the infrastructure we control without a much stronger reason.

12. The less glamorous work underneath all of this

There is also some groundwork we need before those bigger architectural changes become easy.

Today, the analysis code builds the model directly.

I’d rather have a clean interface:

photograph + instructions → structured analysis

Then today’s model becomes one backend.

A future serving engine becomes another.

A hosted API becomes another.

And we can run the same photographs through two backends and compare their verdicts.

Because for Parjanya, the backend is not the product.

The verdict is the product.

So a backend doesn’t get shipped simply because its benchmark is faster.

It has to preserve the behaviour that users depend on.

What I learned from this

After spending two months inside this problem, these are the lessons I am taking away.

1. Measure before optimising

The “mysterious four minutes” wasn’t mysterious.

We simply weren’t measuring it.

A proper boot profile answered the question in a day.

2. Make every optimisation explain itself in numbers

Twice, a number that initially looked like a regression or an opportunity turned out to be a mixture of different GPU types.

We now compare before and after on the same hardware.

3. Be willing to correct yourself publicly

One of our early figures claimed 221 seconds saved.

It was wrong.

We had omitted a stage that had become slower.

The actual saving was 132 seconds.

We corrected the number.

I think that’s an important part of building in public.

The point isn’t to always have the right answer.

It’s to make the process visible enough that you can correct the wrong one.

4. When the disk is the bottleneck, move less data

We tried faster ways to move data.

The better solution was to have less data to move.

5. Test the actual production system

One version of our attention optimisation accidentally changed the vision part of the model as well, because of how the model library propagated configuration settings.

Our tests passed.

A short startup census caught the mistake on the first development boot.

We then added a guard that prevents the model from starting if any component unexpectedly falls back to the CPU.

That kind of defensive engineering isn’t glamorous.

But it is the difference between a benchmark and a reliable system.

6. Speed and correctness are connected

This was probably the biggest lesson.

A performance optimisation can change numerical behaviour.

Numerical behaviour can change a borderline classification.

And a classification can change what a photographer does with their work.

So from now on, every model performance change has two measurements:

How much faster is it?

and

How often does it still give the same verdict?

So what does this mean for someone using Parjanya?

Practically, not much has changed in the workflow.

You upload your photographs.

You can close the browser and the analysis continues.

On a quiet day, the first verdict currently arrives roughly six to seven minutes after the upload finishes.

While the machines start, the gallery shows:

Warming up the AI analysis

After that, each machine processes roughly one photograph per minute, and larger batches get more machines.

Most importantly, new photographs now go through a more consistent analysis path, with the updated curation rules.

The bigger picture

When I started this work, I thought the goal was simple:

Make Parjanya faster.

I ended up with a more nuanced definition.

The goal is not to make an AI system as fast as technically possible.

It is to find the right point on the curve between:

speed × cost × consistency × privacy × product experience.

For a young product, those trade-offs are very real.

I could make the first verdict faster by keeping GPUs running.

I could make inference faster by using more expensive hardware.

I could remove much of the infrastructure complexity by using a hosted model API.

But each of those choices moves a different part of the equation.

So for now, we’re taking the slightly less glamorous path:

measure carefully, optimise what matters, keep the infrastructure under our control, and change architecture only when the workload justifies it.

And perhaps the most satisfying outcome isn’t that we got from nine minutes to five.

It is that we now understand why those five minutes exist.

That makes the next five minutes much easier to attack.

Building in the open

All of this work is tracked publicly.

The cold-start programme lives in our public tracker, and every release is documented in the release notes.

https://github.com/JagadeeshRampam/parjanya-issues/issues/49

https://parjanya.phagyul.ai/release-notes

If you’re a developer interested in GPU infrastructure, a photographer curious about what happens after you upload a shoot, or a founder working through the same uncomfortable trade-offs between cost, speed and reliability, I hope this was useful.

I’m particularly interested in hearing from others who have solved similar cold-start problems differently.

What would you like to see optimised next?

A service crashes.
A file refuses to open.
A test turns red.

And then there are bugs that are much more dangerous: the system works exactly as designed, produces a perfectly reasonable answer, and is still completely wrong.

I recently ran into one of those while building Parjanya 2.0, our image QA platform for professional photographers.

A photographer uploaded a batch of scanned medium-format negatives. Twenty-six images eventually turned out to be affected. Some were rejected as blank before they ever reached AI analysis. Others went through the pipeline and received what looked like perfectly plausible photographic criticism.

The interesting part wasn’t just the TIFF bug.

It was everything around it: how I found it, how the first hypothesis was wrong, how a correct rejection concealed an upstream defect, how two corrupted images escaped detection because they were not quite blank enough, and how fixing one line of code uncovered two more bugs waiting behind it.

That made this incident worth writing down.

The platform I was building

Parjanya 2.0 is designed around a simple idea: before asking an AI model to judge a photograph, I should first make sure I am actually giving it a photograph.

The platform ingests batches of images and runs each one through two stages.

The first is a fast technical gate. It doesn’t need a vision model. It checks whether the file can be decoded, whether its resolution is sane, whether it actually contains photographic content rather than a blank rectangle, and whether it is a duplicate.

Only after an image survives those checks does it reach the second stage, where a vision-language model evaluates things such as composition, sharpness, exposure and other characteristics that matter to a working photographer.

Most of the images we see are JPEGs and RAW camera files and supported 34 formats

Some are TIFFs.

And TIFF is one of those formats where the filename tells you much less than you think.

A TIFF can represent a surprisingly large number of combinations of bit depth, channels, byte order, tone conventions and other metadata.

This incident came from one particular combination:

Chapter 1: The rejection was telling the truth

The first signal was straightforward.

A batch of TIFFs was being rejected within seconds of upload, before any AI analysis ran.

That timing mattered.

Our technical gate was doing exactly what it was supposed to do: decode the image, check its dimensions and determine whether there was meaningful visual content.

The blank-frame detector uses image statistics such as luminance entropy and standard deviation. If both are sufficiently low, the image is considered blank and rejected before we spend GPU time on it.

Upon inspecting one of the rejected files and verified the source directly.

It was not blank.

It had a wide tonal range. The pixel data was fully populated. It was a perfectly legitimate photograph!

Then I ran our own preview-generation code against that exact file.

And there it was.

A completely white rectangle - blank frame.

The source image was fine.

Our preview was not.

The blank detector was measuring the preview, not the original. So from its perspective, the rejection was completely correct.

That distinction turned out to be the entire incident.

The gate wasn’t wrong about the preview. The preview was wrong about the photograph.

That is a particularly nasty class of bug.

A rejection with a technically correct reason can still be evidence of a failure upstream.

Chapter 2: 65,535 is not 255

To understand what happened, I had to go back to two fairly basic imaging concepts: bit depth and channels.

A normal 8-bit image stores each pixel between 0 and 255.

A 16-bit image stores each pixel between 0 and 65,535.

That’s 256 times more possible tonal values.

You don’t need that precision because your monitor can display 65,536 individual shades.

You need it because the file is often an intermediate master.

That is particularly important in film scanning (archivals).

A photographer or archivist can scan a negative at 16-bit precision and preserve a tremendous amount of tonal information. The actual interpretation—shadow recovery, highlight adjustment, grading, dodging and burning—can happen later without throwing away information prematurely.

Then there are channels.

A colour image normally has three channels: red, green and blue.

A grayscale image needs only one.

That isn’t a limitation. It is the correct representation when there is no colour information.

So the file that broke our pipeline was not strange at all.

It was actually a very reasonable professional archival file:

16-bit + 1-channel grayscale TIFF.

In fact, the file carried an embedded profile named EPSON Gray – Gamma 2.2, along with metadata identifying the editing software used.

This looked exactly like the kind of file I should expect from a real scan-to-digital workflow.

And that was precisely why our assumption about “supported TIFF” wasn’t good enough.

Chapter 3: My first hypothesis was wrong

My first thought was obvious:

Maybe 16-bit TIFFs are the problem.

Then production gave me the counterexample.

The same batch contained both:

Same bit depth.

Same encoder.

Same photographer.

Same upload session.

The variable that changed was the channel count.

That immediately narrowed the problem down.

It wasn’t simply “16-bit TIFF.”

It was the combination:

16-bit + single-channel grayscale.

That was the first important lesson for me:

When production gives you naturally occurring controlled experiments, use them.

I didn’t need to invent a synthetic test case to prove that 16-bit wasn’t the problem. The customer’s batch had already done the experiment for me.

Chapter 4: The one-line conversion that wasn’t harmless

Once I followed the image through the preview pipeline, the mechanism became surprisingly small.

Our imaging library automatically rescales 16-bit samples when it decodes certain multi-channel colour images.

For single-channel 16-bit data, however, it returned the raw 16-bit values.

The next step in our pipeline was an unconditional conversion to standard 8-bit RGB.

That conversion was safe for the image modes we had exercised.

It wasn’t safe for this one.

Instead of rescaling 16-bit values into the 8-bit range, it effectively clipped them.

Those two operations sound similar.

They are not.

What should happen is:

16-bit 200    → 8-bit 1
16-bit 32768  → 8-bit 128
16-bit 65535  → 8-bit 255

What actually happened was closer to:

16-bit 200    → 8-bit 200
16-bit 256    → 8-bit 255
16-bit 65535  → 8-bit 255

Everything above 255 was flattened to 255.

And 255 is white.

The arithmetic makes the severity obvious.

An 8-bit conversion preserves only 255 values below the maximum out of a possible 65,535.

That’s roughly 0.39% of the original tonal range.

For a typical scanned negative using a broad portion of the 16-bit range, almost everything therefore became white.

The photograph hadn’t disappeared.

We had mathematically crushed it into a white rectangle.

Chapter 5: The bug was even more interesting than I first thought

At this point I had an explanation for the obviously rejected images.

But something didn’t quite add up.

Two larger TIFFs in the same batch had apparently processed successfully.

That initially looked like evidence that the bug affected some narrower scanner-specific variant.

So I went back and measured them instead of trusting the earlier interpretation.

That changed the story.

They weren’t actually healthy.

They were also 16-bit single-channel grayscale files.

Their previews were approximately 98% and 96% solid white.

They had simply escaped our blank-frame detector.

Why?

Our blank check uses two conditions: low entropy and low variance.

Those two images had slightly more surviving shadow information, because they originated from digital captures converted to black-and-white rather than from analog film scans.

That tiny amount of surviving information was enough to push their variance above the rejection threshold.

So the images were not rejected.

They went to the AI stage.

And the AI did exactly what I had asked it to do.

It looked at a nearly blank white rectangle and produced a confident photographic assessment—essentially describing it as severely overexposed.

That was much more concerning than a simple rejection.

A visibly broken file gets investigated.

A corrupted file that passes the gate and receives a plausible AI explanation can look completely legitimate.

The original automated log suggested 24 affected images.

The actual number was 26.

Two corrupted images had successfully made it far enough downstream to receive convincing-looking AI output.

That changed how I thought about remediation.

I couldn’t say:

“Find all the files that were rejected and fix them.”

I had to say:

“Find every file that has the defect.”

The difference sounds subtle.

Operationally, it is enormous.

So we identified remediation candidates using the actual container metadata—bit depth and channel count—not the symptom produced by the broken pipeline.

Chapter 6: The obvious fix took ten minutes. Shipping it took three days.

The immediate fix looked almost embarrassingly simple:

Rescale the 16-bit values to 8-bit before converting to RGB.

I could write that in minutes.

I deliberately didn’t ship it.

Because once you touch a central image-conversion path, the question isn’t:

“Does this fix the file that broke?”

The question is:

“What else passes through this code?”

So I built synthetic files covering sixteen different variants.

Different byte orders.

Inverted-tone scans.

Packed sub-byte formats.

Signed samples.

Floating-point samples.

Four-channel print files.

Different combinations of formats that a real imaging library can encounter.

And that exercise immediately found two more problems.

Chapter 7: Fixing the bug almost created a photographic negative

Some scanners use an inverted tone convention.

In those files, zero can represent white rather than black.

At 8-bit depth, the imaging library was silently correcting for that.

At 16-bit depth, it wasn’t.

Previously, our clipping bug had destroyed the image so completely that this difference was invisible.

Once I fixed the clipping, it suddenly mattered.

A naive fix would have transformed some of those scans into photographic negatives.

In other words:

the first bug had been masking the second bug.

That is one of the reasons I have become much more cautious about declaring a conversion problem “fixed” after a single successful test.

Removing one failure mode can expose another one that was hiding behind it.

Chapter 8: Then I found a metadata bug hiding somewhere else

The next surprise wasn’t even in the conversion code.

Our orientation-correction step creates a fresh image object.

That fresh object didn’t retain all of the original technical metadata.

That mattered because the fix needed to know information such as bit depth and tone convention.

If I inspected that metadata early enough in the pipeline, it was available.

If I inspected it after the orientation step, it was gone.

This was a useful reminder that image-processing pipelines aren’t just a sequence of transformations.

They are also a sequence of information loss.

Every transformation potentially changes:

• pixel values

• dimensions

• colour space

• metadata

• orientation

• bit depth

• tone interpretation

And once information disappears, a later stage can’t magically recover it.

So the correct place to make a decision isn’t necessarily where the decision is easiest to implement.

It is where the required information still exists.

Chapter 9: I rejected an attractive shortcut

There was another tempting idea.

TIFF files can contain metadata describing the brightest value actually used in the image.

It seemed reasonable to use that value as the denominator when converting 16-bit to 8-bit.

Why scale against a theoretical maximum if I know the maximum value actually present?

Because that would subtly change the photograph.

Imagine a perfectly normal photograph whose brightest pixel is 55,000 rather than 65,535.

Scaling against 55,000 would brighten the entire image.

That isn’t a bit-depth conversion anymore.

It is an automatic contrast adjustment.

And the photographer didn’t ask me to change their exposure.

So we deliberately rejected that shortcut.

The conversion is based on the declared bit depth, not whatever happened to be present in one particular image.

I also locked that decision into a regression test.

Chapter 10: The better fix wasn’t just a better conversion

The most durable change I made wasn’t actually the 16-bit fix.

It was adding a self-checking preview writer.

Instead of assuming:

“The conversion function returned successfully, therefore the preview is good.”

the preview writer now compares meaningful tonal variation before and after the transformation.

If a photograph goes in with substantial tonal variation and comes out essentially flat, the preview generation itself fails.

It refuses to save the preview as if everything succeeded.

That distinction is important.

The system doesn’t need to know that the bug is TIFF.

It doesn’t need to know that the input is 16-bit.

It doesn’t need to know about grayscale channels.

It only needs to recognise something more fundamental:

a photograph went in, and almost nothing came out.

On our known-good conversions, the input/output variation ratio clustered around 1.0.

The broken examples were 0.35 or lower.

The original file that started this investigation was exactly 0.00.

That gives us a generic trip-wire for future bugs we haven’t imagined yet.

And that is much more valuable than a test that only knows about today’s failure.

Chapter 11: The AI wasn’t the problem. The pipeline was.

There is another lesson here that I think matters beyond image processing.

It would have been easy to describe this as an “AI quality” problem.

It wasn’t.

The VLM was downstream of a corrupted image.

It received a nearly blank frame and produced a plausible interpretation of what it saw.

The failure happened earlier.

That distinction becomes increasingly important as we build systems where AI is only one component in a much larger automated pipeline.

If the data entering the model is wrong, a more capable model doesn’t necessarily make the system safer.

It may make the failure more convincing.

That is something I now think about whenever I design an automated workflow:

Before asking whether the model’s answer is correct, ask whether the model received the thing I thought I gave it.

Chapter 12: The second AI session had already fixed it

There was one more twist.

By the time the fix was deployed and I was ready to reprocess the affected images, I discovered that the reprocessing had apparently already happened.

Another session had been working on the same incident in parallel.

Nothing had explicitly told me it was complete.

It was just... done.

The tempting responses were:

1. Trust that it was already completed.

2. Run it again just to be safe.

I didn’t want to do either.

Trusting silence isn’t verification.

Blindly rerunning a potentially expensive job isn’t verification either.

So I checked the artefact.

First, I ran the reprocessing tool in dry-run mode.

It still identified the same affected files from their actual container metadata, and all of them now showed as successfully processed.

Then I checked the record store independently.

The timestamps lined up with the deployment window.

Finally, the infrastructure logs showed a burst of preview-generation activity immediately after deployment that was much higher than normal traffic.

Three independent signals.

Only then did I consider the remediation complete.

This is becoming increasingly relevant as engineering moves from one developer sitting in one terminal to multiple people—and increasingly multiple AI agents—working concurrently on the same system.

When two workers might be acting on the same incident, the right question isn’t:

“Did someone say they did it?”

It is:

“Can I independently verify the artifact that proves it happened?”

What actually happened?

The final numbers made the incident much clearer.

Before the fix:

The customer’s batch returned to a normal curation mix: roughly 60% accepted and 40% rejected for genuine photographic reasons such as blur or exposure.

Most importantly, the original source files were never damaged.

The corruption existed entirely in the generated previews.

So remediation was simply a matter of generating those previews correctly.

The entire cycle—from report to root cause, fix, deployment, verification and closure—took three days.

What I would tell another engineering team

1. “Supported” and “decodable” are not the same thing

Putting TIFF on a supported-format list doesn’t mean you’ve supported every real-world TIFF.

TIFF is a container.

Inside that container are many combinations of properties that your pipeline may never have exercised.

A format can be technically supported and operationally untested.

That gap is where this bug lived.

2. A correct rejection doesn’t prove the system is working

Our blank detector was correct.

The preview it examined was wrong.

When an automated check reports a technically accurate failure, I now trace upstream before assuming the input is at fault.

The last check in a pipeline can be completely correct while everything feeding it is broken.

3. Remediate based on the defect, not the symptom

The two images that escaped our rejection logic would have remained corrupted if I had searched only for rejected files.

I had to identify the actual format signature:

16-bit + single-channel grayscale.

The symptom was “blank.”

The defect was something else.

4. The blast radius of a fix can be larger than the blast radius of the bug

The original incident was about clipping.

Testing the fix uncovered inverted-tone behaviour and metadata loss.

If I had stopped after the first successful test, I would have introduced new image corruption while fixing the old corruption.

5. Build checks that don’t know what tomorrow’s bug looks like

A format-specific regression test protects against known failures.

A structural fidelity check can catch unknown ones.

That is why I like the idea of measuring the transformation itself:

Did meaningful image information survive the operation?

6. Verify the artefact, not the silence

Whether the other worker is a colleague on another shift or another AI session running in parallel, “nothing complained” is not evidence of completion.

Look at the output.

Look at the records.

Look at an independent signal.

Then close the incident.

The broader lesson

I started this investigation thinking I had a TIFF problem.

What I actually found was a systems problem.

A professional photographer supplied a perfectly legitimate archival file.

Our software accepted the format.

The file decoded.

The preview-generation function returned successfully.

The automated quality gate correctly detected a blank image.

The AI then produced a plausible assessment on two corrupted images.

Every individual component behaved reasonably according to the information it had.

And yet the system as a whole was wrong.

That’s the part I find most interesting.

As we build more automated products, especially products where traditional software, image processing and AI models sit in the same pipeline, I think we need to become much more interested in fidelity between stages.

Not just:

Did the function return successfully?

But:

Did the meaning survive the transformation?

For Parjanya, that means being increasingly careful about the long tail of real-world photography formats.

For developers, it means testing the combinations that production will eventually discover for you if you don’t.

For photographers, it is a reminder that “supported format” doesn’t necessarily mean “faithfully interpreted format.”

And for founders, it is a reminder that automation doesn’t remove responsibility.

It moves the responsibility upstream.

The system needs to know not only when it can produce an answer, but when it should refuse to trust its own answer.

That is probably the more important fix I took away from this incident.

Parjanya is the image QA platform we’re building as part of Phagyul eco-system for professional photographers. If you’re working on heterogeneous image ingestion, archival photography, AI-assisted review pipelines, or reliability problems in AI-driven systems, I’d love to compare notes.

Related engineering incident: GitHub Issue #87 — 16-bit grayscale TIFF preview corruption

_{••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••}

Today it is open to everyone. This post is the release notes: what Parjanya does, who tested it and what they found, everything that changed between the July baseline and now, and what is coming next. The engineering detail is here too, because this blog has always been the place for it.

What Parjanya does, in two passes

You drop a folder or files from your card or hard drive straight into the browser, and the files go from the browser directly into private storage over signed, resumable transfers. The moment the upload says complete you can close your browser or session: the analysis picks the work up on its own, and every upload and every verdict is recorded as a replayable intent, so nothing is lost to a dropped connection or a busy GPU. (The upload path and the reconciliation model behind it, TBIE, in-house research paper successfully verified for our usecase and evolved and tested for 18+ usecases.) Parjanya makes fast previews from every file (majority is RAW files across DSLR, mirrorless, mobile phones and Analog converted to Digital) and then checks each frame twice, in a fixed order.

• Pass one is a technical gate. Cheap, strict, and deliberately narrow: unreadable or damaged files, images whose pixel size disagrees with what the file claims, blank frames, near-solid exposures (a frame is set aside only when almost every pixel is pure white or pure black, so brackets and silhouettes pass), duplicates by perceptual fingerprint (frames within ten seconds on the same body are a burst, never duplicates), and screenshots or exports in formats no camera writes will be filtered out. Anything set aside lands in Skip with the reason attached. Nothing is deleted.

• Pass two is VLM enrichment. A vision-language model running on our own GPUs reads the frame and writes what you see on the detail page: what the photo is doing, how it is composed, what is holding it back and how strongly, plus the tags search runs on. A deterministic rule engine turns those findings into Keeper, Review or Skip with the reason attached. There is no score out of ten, on purpose: the same findings give the same verdict every time. (our v2.0 major change proved score engine doesn’t hold true and a deterministic rule engine does the best for our usecase)

You curate in a three-tab gallery where any verdict can be overruled, search the shoot in your own words (”leopard in dappled light”, “long exposure, silky water”, “symmetry, blue hour”), and after 500 uploads you get Parjanya Vision: a report card written like a coach rather than a mark sheet.

The people who tested it

The pilot group was small enough that every one of them changed something. A wildlife shooter pressed Stop on a 479-image batch expecting a pause; that single click is why Pause and Resume exist. A documentary photographer was first to point out that a Review/Skip line drawn only on technique penalises intentional motion and grain, which is now an open design question on the tracker (#58). Someone on their first camera uploaded iPhone photos that were accepted and never analysed. An archivist uploaded 16-bit grayscale scans that came back as blank white frames(#87). A landscape photographer with a body released this year sent RAW files our decoder had never seen, and they came back in false colour until it learned to fall back to the camera’s own embedded preview. And a mix of legacy camera bodies, manual lens and more.

Several of the pilot group are credited on the public tracker against the issues they raised; where someone preferred to stay unnamed, the issue says “pilot feedback” and nothing more. The onboarding guide itself, the ten-minute walkthrough new users get today, was recommended by one of the pilot experts.

Thank you, all of you. The list of fixes below is mostly your list.

https://github.com/JagadeeshRampam/parjanya-issues/issues

What changed since the July baseline

The v2.0.0 baseline was cut on 11 July after a 12,000-image validation run. Between then and today, 34 items on the public tracker were closed. Grouped by what you will notice:

Uploads and formats

• iPhone photos process now. HEIC and HEIF were on the supported list and were accepted at upload, then stranded silently with no preview and no analysis. The decoder library was installed the whole time; the one line that registers it with the imaging library had been lost in a refactor. Fixed and verified on real iPhone uploads (#40).

• 16-bit grayscale scans no longer come back white. Single-channel 16-bit TIFFs were clipped to solid white during preview generation and then, correctly, set aside as blank. The fix rescales by bit depth and validates every preview it writes. All 28 affected images were reprocessed (#87).

• Canon CR3 previews lost their magenta cast and dark border (#16).

• Filenames with spaces render previews again (#69), and camera and lens names no longer carry trailing padding from fixed-width metadata fields, which had been splitting one lens into two entries in filters (#48).

• Upload progress no longer flashes “Uploading: 0” mid-transfer (#3), and switching accounts in the same tab no longer shows the previous account’s upload bar (#33).

• When an upload is blocked, by a trial cap or an account hold, the app now says why and what to do, instead of a generic error (#24).

Processing you can control

• Pause and Resume. Stop was permanent, and a photographer found that out on a 479-image batch. Pause parks queued work and Resume picks it up exactly where it left off; a frame already being analysed when you pause finishes and keeps its result (#36). Stop still exists and now sticks end to end, including for work already in flight (#32).

• “Warming up the AI analysis.” The GPU fleet starts on demand. When it is starting, the gallery now says so, so a cold start reads as a cold start and not as a broken product (#62).

• Screenshots are turned away before they reach the GPU. Nine screenshots uploaded in August all passed the technical checks and three were accepted as good photographs. A metadata gate now rejects non-photographic content first (#64).

Reading the analysis

• Critique and improvements are two cards now. The critique is served exactly as written. The one or two actions worth taking were previously the last sentence of an italic paragraph most readers had stopped reading; they are now their own card, and they never repeat what the critique already said (#93).

• Burst groups show a per-frame breakdown when the frames in a burst got different verdicts (#1).

• The gallery badge showed “Pending” for every auto-curated image because two parts of the system spelled the verdict differently. Fixed (#90).

Gallery and search

• Newest first. The gallery was not listing your newest uploads first, which meant your own upload could be invisible to you. It reads from a recency index now (#66), and the three tabs agree on what “newest” means (#68).

• Semantic re-ranking using SigLIP 2 image embeddings is integrated behind a flag, and search now says visibly when it has fallen back to keyword ranking (#28, #30). Content filters for scene, lighting and mood are still hidden until every image carries those tags (#72, #25).

Parjanya Vision

• The report card shipped on 9 September. It started life as a per-shoot summary idea (#85) and became a tenant-wide card: what you shoot, which camera bodies and formats, how far you zoom, where your Review pile clusters, how you curate (rescued versus demoted), and what to try. Every rate travels with its count. Copy is coach voice by rule; there is a test that fails the build if words like mistake or grade appear (#98).

• Two same-day fixes after the first photographers opened it: bars that overflowed their card, and a horizontal fill bar that read like a progress meter for what is really two counts being compared. It is a vertical bar chart now (#99).

Account, trial and email

• Sign-up provisions correctly (#22) and confirmation codes deliver (#23), which sounds like table stakes and was the last launch dependency to close.

• Every account email exists now: three days before a trial ends, the day it ends, a follow-up a week later, when you are at 90% of a cap, when a cap is reached, and one for every action we take on an account, including a past-tense notice after a deletion completes (#88, #89, #104).

Where your photos live

Confidently, because it was audited before launch: your originals go from your browser straight into private, encrypted storage in Mumbai, in a space that belongs only to your account, with public access blocked at the bucket level and every connection over HTTPS. Every request is checked against your signed-in identity before it touches a record; there is no path from one photographer’s library to another’s. Every access to stored photos is written to an audit trail with alarms on any human or unexpected access. The vision model runs on GPUs we operate, so your photos are never sent to a third-party AI service and are never used to train models. And nothing is deleted by us: when a trial ends, uploads pause and everything already there stays readable and downloadable.

Search, in your own words

Search reads the critique, the composition notes and the closed-vocabulary tags the model writes for every frame (subject, scene, lighting, mood, dominant colours), so queries like elephant herd at a waterhole, backlit, dew on a spider web, ridgeline at dawn, layered haze or stairwell, geometric shadows work today. Two more pieces are built and switching on as the whole catalogue gets its tags and embeddings: filter chips for scene, lighting and mood, and a look-alike mode that ranks by visual similarity using SigLIP 2 image embeddings (#11, #25, #72).

Three things the pilot taught us

1. Stop is not Pause, and a UI should know the difference. The 479-image incident was not a bug; Stop did what it said. The bug was offering only an irreversible action for a reversible intent. Resume had one subtle requirement worth writing down: flip the records back to pending before re-queueing them. Queue first and a worker can pick the message up, still read “paused”, and drop the job on the floor.

2. Small uploads should not summon the whole fleet. GPU scaling used to step up on a timer whenever the queue was non-empty. A 13-image batch could scale to the maximum fleet, most of which arrived after the work was done, and the fleet then took 69 minutes to scale back down. Scaling now targets an exact capacity from visible-plus-in-flight queue depth; the same drain takes about four minutes. The accepted side effect is that scale-in occasionally interrupts a job mid-inference, which the queue’s visibility timeout retries (about 1% of images, zero lost) (#41).

3. “Supported” is a promise about acceptance, not decoding. The HEIC and 16-bit TIFF cases were the same class of problem: a format we would accept at upload and then fail on quietly. The legacy and edge-case list from the pilot is now long and mostly handled: RAW from bodies newer than the decoder’s database (falls back to the camera’s embedded preview instead of false colour), 16-bit and grayscale scans (rescaled by bit depth), iPhone HEIC, 25 RAW formats including Canon CRW, Sony SRF/SR2, Hasselblad 3FR/FFF, Phase One IIQ and Sigma X3F, filenames with spaces, manufacturer-padded camera names, bracketed exposures (pass the gate), bursts (stack, best frame ranked). Still open: older bodies and phone JPEGs that record no capture time can have burst frames flagged as duplicates (#70). Of the 34 formats, 18 are proven end to end in production; the next one we want a real file for is Fujifilm RAF (#42, #47). If you shoot Fuji, your first upload is a favour to us.

Numbers from the closed beta

• 20+ photographer accounts, from first-camera to full-time professional

• 18,884 images enriched, across 18+ file formats

• 104 issues filed on the public tracker since July; 34 closed, every one with what actually shipped

• Launch-day verification: 100 human-checked images, 51 Keeper / 47 Skip / 2 Review, zero stranded

• GPU drain after a small upload: 69 minutes → about 4 minutes

• Cold start after an idle period: 5 min 32 s today; 90 s is the target

• 28 archival scans and 261 verdict records repaired in place, nothing deleted

The trial

30 days, 1,500 photos, 30 GB of storage including previews. Whichever runs out first ends the trial. Uploads pause at that point and nothing is removed; your gallery, critiques and downloads keep working. Ask and we can raise a cap on your existing account. No card is needed to start.

What is next

In roughly the order we expect to ship:

• A “your shoot is ready” notification, email and in-app, so you can close the laptop after the upload completes and be told when the analysis is done. Today the first verdicts after an idle period wait about five and a half minutes for the AI to warm up; the target is 90 seconds (#105). Broader in-app notifications follow (#8).

• Near-duplicate detection, burst-aware, on top of the exact-duplicate check that is live (#4)

• Curation that learns from you. Every Keeper and Skip on a Review frame is already recorded; the model of your taste that reads them is next (#5)

• Content filters in search once every image carries scene, lighting and mood tags (#72), and a relevance floor so weak semantic matches stop looking strong (#74)

• Parjanya Vision phase 4: gear-specific advice that unlocks only when your own photos provide the evidence (#100), and phase 5, genre for your older photos from stored embeddings without reprocessing (#101)

• Self-serve payment (#9), and burst covers that show the best frame rather than the middle one (#92)

• The onboarding guide in more formats: a video walkthrough with voice-over is next (#53)

Everything, including the items we have not fixed yet, is public at github.com/JagadeeshRampam/parjanya-issues. Open and closed alike. The people who tested this deserved to see the list, and so do you.

Start

Onboarding guide, ten minutes start to finish: parjanya.phagyul.ai/guide.html

Create your account: parjanya.phagyul.ai/signup

For the pilot group of Experts across the world and my buddies: thank you again! 🙏

Jagadeesh

Engineering appendix (for readers who came for the systems)

Short entries; each has a longer post either published or drafted.

• Why there is no score. The rule engine reads distortion type and severity (motion blur: heavy) rather than a numeric quality score. Scores compressed incompatible defects onto one axis and gave photographers nothing to act on. The model still emits one; the gate ignores it.

• Caching that never existed. The dashboards were designed against a response cache that was never provisioned in production, so every dashboard call recomputed. They now serve from a snapshot store warmed by an in-process loop; the admin pane, which took up to 29 seconds per range, answers in under a second (#34).

• The month that weighed nothing. A field rename mid-July left usage attribution reading the old name and projecting only the old name from the database, so every image uploaded for a month contributed zero bytes to per-tenant attribution. The headline figure stayed right because it preferred a storage-measured value, which is exactly why nobody noticed. Lesson written down: a rename has to land in the projection too, and a test fixture that only builds one spelling cannot catch a rename (#81).

• Idempotent email. Trial emails failed silently for three weeks after the flag went live because the execution role never received send permission, and a failed send was marked as delivered before it was attempted, so it could never retry. The rewrite is claim → send → confirm, with release on failure, and it is the helper every new notification now uses (#88, #89, #104).

• Dedup that could eat a whole cluster. Symmetric concurrent rejection let every member of a duplicate cluster defer to another member that was also being discarded. Twelve images got no analysis. Survivors are now chosen by a total order, earliest wins (#97).

• TBIE. The reconciliation model behind all of this, Truth, Belief, Intent, Execution, has its own post: From Pipeline to Platform. The white paper is in draft.

• From localhost friction to production-shaped architecture

  ---

1. The morning I looked at the bill

On May 3rd, I ran a cost audit on Parjanya’s AWS account because the number I’d been carrying in my head — “it’s cheap, it’s a GPU worker and some Lambdas” — didn’t match what Billing was showing me.

It wasn’t cheap.

The account was running at roughly $100/day, and GPU EC2 accounted for most of that at the time: 57–65% of the bill. Two g4dn.2xlarge instances were running 24/7 on 100% on-demand capacity for our Qwen3-VL-8B image-quality worker.

Then I checked the launch template.

"OnDemandPercentageAboveBaseCapacity": 100

A hundred percent.

The Terraform module already supported Spot allocation. The mixed_instances_policy had been there since March. I simply hadn’t enabled it.

That is where a typical cloud-cost story might end: find the waste, flip the switch, move on.

But that only answered one question:

Which pool should an instance come from?

It took me much longer to ask the second:

How many instances do I actually need?

That second question turned out to matter even more.

2. What I believed about Spot capacity — and what the data showed

I had an assumption I’d never re-tested: GPU Spot capacity in Mumbai wasn’t reliable enough to build around.

That assumption was the reason the ASG had remained on-demand-only since March. It wasn’t really an oversight. It was an old decision that I hadn’t revisited.

So before changing anything, I pulled seven days of Spot price history across the GPU instance types we could realistically use:

The assumption hadn’t been unreasonable when I made it. It was simply stale.

By May, g4dn.xlarge Spot capacity was looking healthy across all three availability zones. g5.xlarge was a different story: better GPU capacity, but available in only two AZs.

Rather than rely entirely on historical pricing, I ran a live canary.

I switched the ASG to Spot, terminated two on-demand instances, and sent the same workload through both configurations.

The measured result was a 61% reduction in cost per unit of work.

The important lesson wasn’t just about Spot.

The thing standing between me and that saving wasn’t a technical limitation. It was an assumption I had stopped questioning.

3. Why I didn’t go 100% Spot

Spot capacity can disappear with little warning.

Our worker also has a significant warm-up cost: pulling the container image and loading a roughly 17GB model. If an instance is reclaimed during warm-up, it produces no useful work.

So I didn’t want the entire fleet to depend on Spot availability.

The configuration became a combination of on-demand and Spot:

gpu_instance_types       = ["g5.xlarge", "g4dn.xlarge"]
on_demand_base_capacity  = 4
on_demand_percentage     = 0
spot_allocation_strategy = "price-capacity-optimized"

The idea is simple:

• Keep a small on-demand base as insurance.

• Use Spot for everything above that base.

• Let the fleet scale with demand without paying the full on-demand price for every instance.

The on-demand base provides a floor that doesn’t depend on Spot availability. The Spot fleet provides most of the cost efficiency.

That combination turned out to be much more useful than thinking of Spot as an all-or-nothing decision.

4. Widening the pool — then narrowing it again

The next experiment was about the instance pool itself.

On July 9th, I saw several g4dn.xlarge Spot instances get reclaimed during warm-up. A pool containing only g5.xlarge and g4dn.xlarge was too narrow to absorb that pressure.

So I temporarily added:

• g4dn.2xlarge

• g5.2xlarge

These instances had the same GPUs as their xlarge counterparts but more CPU and RAM than the workload really needed.

I wasn’t buying additional GPU throughput.

I was buying capacity insurance.

The idea was that price-capacity-optimized could use the larger instances only when the cheaper pools were under pressure. In that situation, paying a little more for an instance that actually becomes available can be better than waiting for a cheaper instance that doesn’t exist.

I calculated the worst-case premium before enabling it. Across a 12,000-image batch, the additional cost would have been roughly $3.

That was an acceptable temporary trade-off.

On July 28th, I removed the 2xlarge types again.

The reason was straightforward: as a permanent configuration, paying for extra CPU and RAM that the GPU workload didn’t use wasn’t attractive.

The temporary insurance had done its job. Once the capacity pressure eased, the simpler pool made more sense.

This also exposed an important operational principle:

A good configuration isn’t necessarily a permanent configuration.

Sometimes the right response to a capacity problem is to widen your options temporarily, then narrow them again when the underlying conditions change.

5. Two Spot capacity droughts

The experiments also exposed the limits of Spot.

July 9th

Mumbai’s g4dn Spot capacity went dry across the region.

The fleet dropped from five instances to one during a 450-image batch.

No allocation strategy could solve that. There simply wasn’t enough Spot capacity available.

The on-demand base was what kept the workload alive, so I increased the base from one to three instances that afternoon.

August 5th–7th

This time the problem was g5.xlarge.

There was no Spot support in one of its two available AZs, while the other experienced an on-demand capacity gap.

There were three launch failures over three days.

price-capacity-optimized was able to route around the problem by falling back to g4dn.xlarge, although the fallback introduced additional latency.

These two incidents changed how I think about Spot availability.

The answer isn’t simply that availability is “better” or “worse.”

It is better on average for some pools, but individual instance families can still experience significant capacity constraints.

Spot is a probability and capacity-management problem, not just a pricing problem.

6. What changed when I moved to price-capacity-optimized

There are two strategies that are easy to confuse:

capacity-optimized chooses the Spot pool with the deepest available capacity. Price isn’t part of the primary decision.

price-capacity-optimized considers both capacity availability and price, choosing among pools that have a reasonable likelihood of remaining available.

The distinction matters.

The May canary that produced the 61% cost reduction used capacity-optimized. I later moved to price-capacity-optimized on July 9th.

I don’t have a controlled, back-to-back experiment proving the exact dollar improvement from PCO. So I don’t want to manufacture one.

The decision was based on the allocation model and the observed capacity conditions rather than a clean A/B test.

That distinction matters when discussing infrastructure results:

Measured results and engineering judgments are not the same thing.

There was another useful discovery in the same change.

The Terraform module had an override issue that was silently alphabetizing the instance-type list rather than preserving the intended priority order.

So the configuration had effectively been saying one thing while the infrastructure was doing another.

It was a good reminder that infrastructure configuration needs the same level of testing and observability as application code.

7. The bigger question: how many instances?

This is where the more interesting problem appeared.

Until July 27th, both scale-up and scale-down used SimpleScaling.

The logic was effectively:

Is the queue empty?

If yes, scale down.

If no, scale up.

The problem is that this treats a queue containing 13 messages exactly like a queue containing 1,300 messages.

The system knew whether work existed.

It didn’t know how much work existed.

On July 27th, a small 13-image upload caused the fleet to scale to its maximum. Three of six instances processed nothing, and the timer-based scale-down took 69 minutes.

The post-incident calculation showed approximately 4.6 instance-hours spent for around 20 minutes of actual work.

The fix was to make scaling backlog-aware.

Instead of looking only at whether the queue was non-empty, the scaling logic now considers the combined backlog — visible and in-flight messages — and maps it to an explicit target fleet size:

backlog     target

1–25        → 1 instance
25–100      → 2
100–500     → 4
500+        → maximum

The policy also changed from incremental scaling to ExactCapacity.

Instead of saying:

Add one instance.

it says:

The fleet should contain four instances.

That difference is important.

Each scaling decision can converge directly on the desired fleet size rather than repeatedly increasing or decreasing the fleet one instance at a time.

In a later measured run, the same class of drain dropped from 69 minutes to 4 minutes.

But I made another mistake.

On August 8th, I reduced every scaling band by half.

That worked for small batches. It also meant that a 108-image production run on August 11th stayed on one instance for roughly 50 minutes when a second instance could have provided useful parallelism after its warm-up period.

Three days later, I adjusted the bands again rather than reverting the entire change.

The lesson was bigger than the individual thresholds:

Scaling needs to understand both price and workload depth.

Choosing the right Spot pool answers:

Which instance should I buy?

Autoscaling answers:

How many instances should I buy?

Those are different optimization problems.

And I had spent months tuning the first one before realizing the second one was costing me more.

8. Why this isn’t just a startup problem

It’s easy to think of backlog-blind scaling as an early-stage infrastructure problem.

I don’t think it is.

Any system where demand arrives in bursts can have the same problem.

Consider:

• Batch customer onboarding

• Large data imports

• End-of-month reporting

• Media processing

• Scheduled analytics workloads

• Seasonal traffic spikes

• Any SaaS workload with uneven usage

A presence-based alarm can tell you:

There is work waiting.

It cannot tell you:

There are 20 messages versus 20,000 messages.

Those situations require completely different capacity decisions.

The mechanism is general:

Scaling decisions should consider the amount of work waiting, not just the existence of work.

The exact thresholds will vary by system, but the principle applies broadly.

9. What actually changed

Here are the results I can measure and reproduce:

• 61% lower cost per unit of work in the Spot vs. on-demand canary.

• 69 minutes → 4 minutes for a comparable production drain after moving to depth-aware ExactCapacity scaling.

• ~4.6 instance-hours consumed by an inefficient scaling response to a small workload.

• ~$3 maximum premium calculated for temporarily widening the Spot pool for a 12,000-image batch.

• 3 launch failures over 3 days during the August g5.xlarge capacity issue.

The longer-term Cost Explorer data tells a similar story.

Across the GPU hours recorded this year, on-demand averaged roughly $0.64/hour, while Spot averaged roughly $0.30/hour — about a 53% discount.

For the 785 Spot hours recorded year-to-date, paying the blended on-demand rate would have cost approximately $505. Actual Spot spend was about $235.

That’s roughly $270 in measured savings from those Spot hours alone.

GPU EC2 has cost $967.85 year-to-date against $6,637.43 in total account spend through August 22 — roughly 15% of total spend.

The important point isn’t the exact percentage.

It is that the optimization moved from a simple:

“Can I get cheaper GPU instances?”

to a much broader question:

“Am I buying the right capacity, from the right pool, for the amount of work actually waiting?”

That is a much more useful infrastructure question.

10. The bigger takeaway: price and capacity must be optimized together

The biggest lesson from this exercise isn’t simply that Spot is cheaper than on-demand.

It is that price and capacity need to be considered together.

price-capacity-optimized matters because the cheapest Spot instance isn’t necessarily the cheapest option if that capacity isn’t available — or is likely to be reclaimed. A slightly more expensive pool with better capacity can be the better economic choice when the alternative is waiting for capacity that may never arrive.

That is the real value of price-capacity optimization:

Don’t optimize for the lowest price. Optimize for the lowest viable cost of reliable capacity.

The distinction became particularly clear during the capacity droughts I encountered. No allocation strategy can create capacity that doesn’t exist, which is why the on-demand base remained important. But when multiple Spot pools were available, price-capacity-optimized gave the fleet a better way to balance availability and cost instead of treating either one in isolation.

There is, however, a second layer that is just as important.

price-capacity-optimized answers:

Which capacity should I use?

Autoscaling answers:

How much capacity do I need?

Those are separate problems.

The configuration that ultimately worked for me brought both together:

1. Keep an on-demand base as protection against a complete Spot capacity shortage.

2. Use price-capacity-optimized Spot allocation to balance price and capacity across viable pools.

3. Keep multiple instance options available when capacity is constrained, but avoid permanently paying for resources the workload doesn’t actually use.

4. Scale based on backlog depth, rather than simply checking whether work exists.

5. Use ExactCapacity targets so the fleet converges directly on the capacity the workload needs.

This changed how I think about compute optimization.

The goal isn’t to find the cheapest instance.

It isn’t even to find the cheapest Spot pool.

The goal is to find the right amount of reliable capacity at the lowest practical cost.

That is what price-capacity-optimized ultimately taught me: price optimization without capacity awareness is incomplete, and capacity optimization without workload awareness can still waste money.

The real optimization happens when all three line up:

11. The connection to context engineering

This is where this experiment connects with the earlier post on Context Engineering and Context Debt.

The two problems look different, but the underlying optimization pattern is surprisingly similar.

In context management, you have to decide:

Which information belongs in the context?

Then separately:

How much context should the model actually carry?

The GPU fleet has the same two layers.

Price-capacity-optimized answers:

Which pool should the instance come from?

Depth-aware scaling answers:

How many instances does the workload justify?

Getting the first decision right doesn’t automatically make the second one right.

That was the mistake I made.

I optimized the price and availability of each unit without first asking whether I needed that many units.

The same pattern appears in context management: a good retrieval strategy can still become expensive if the context budget itself isn’t being managed.

The broader lesson for me is simple:

Optimization is rarely one decision. It is usually a stack of decisions.

You can optimize the individual resource and still waste money because you’re using too much of it.

You can optimize the model and still waste tokens because you’re sending too much context.

You can optimize the database query and still waste resources because you’re running it too many times.

The useful question is therefore not just:

“What’s the cheapest way to do this?”

It is:

“What is the minimum amount of the right resource needed to do this reliably?”

That is the question I should have asked much earlier.

Configuration referenced in this post reflects the production setup as of August 12, 2026. Cost figures are based on a Cost Explorer pull covering January 1 through August 22, 2026.

The N+1 query problem is what happens when fetching N items costs you N+1 round trips to a data store instead of one — and it is, without much competition, one of the most common performance bugs I see in software that talks to a database.

I want to write about it here not because we hit the textbook version. We didn’t. What we ran into while building a new feature for Parjanya were two close cousins of N+1, caught at different points in the same week.

What made the experience worth writing about was the pattern-matching involved. Once I started asking “how many times does this data actually need to leave the system?” rather than just “does this code work?”, both problems became much easier to see.

I think that is a useful engineering habit for anyone building on top of an ORM, a NoSQL table, or a cloud API with a batch endpoint nobody on the team has used yet.

Why N+1 Gets Its Own Name

Picture the canonical example, the one I have either written or fixed more times than I’d like to admit: I load a list of blog posts, then loop over them to print each author’s name.

posts = Post.objects.all()          # 1 query
for post in posts:
    print(post.author.name)         # N queries — one per post

One query becomes N+1 queries.

With 10 posts, that’s mildly wasteful. With 10,000 posts loaded on an admin page, it can be the difference between a 200ms response and a request that never comes back.

Once I started looking for the shape rather than the syntax, I found it everywhere: a GraphQL resolver fetching a Book per Author node instead of batching with a DataLoader; a Rails view calling .comments.count inside an each block; a microservice calling GET /users/:id once per row instead of POST /users/batch.

Different stacks, same root cause — a per-item operation standing in for a single query or batch operation the data layer was perfectly capable of running.

What makes N+1 worth having a name of its own, rather than simply calling it “a slow endpoint,” is how it hides.

The loop looks correct in code review. It is correct — just expensive.

Unit tests don’t necessarily expose it either. A fixture with three rows produces three extra queries, but nobody notices when the entire test takes 40ms.

It usually becomes visible only with real data volume: a production dashboard gets slower, a support ticket says a page “used to be fast,” and by then the same pattern has often been copied into two other endpoints.

That’s why I care about catching this class of problem early. The fix can be a five-line change when I catch it during design or review. Once the data has grown into the architecture, the same fix can become a multi-file refactor with a production incident attached.

Our Version Wasn’t the Textbook Case — Which Is Exactly the Point

At Parjanya, we don’t run a traditional ORM — no ActiveRecord, no Django QuerySets, no lazy-loaded relationships waiting to surprise us.

Our data layer is DynamoDB, queried directly, with explicit asyncio.gather calls instead of implicit lazy loading.

At first glance, that seems like it should rule out N+1 entirely. Nothing is silently issuing a query behind a property access.

It doesn’t.

It just changes the shape of the problem.

We found two variants in the same week while building the same feature, and neither looked anything like the classic blog-posts-and-authors example.

Cousin #1: The Same Partition, Fetched Twice by Two Honest Code Paths

In mid-July, we built GET /api/v1/admin/dashboard — a cross-tenant operations view for Parjanya: pipeline health, queue depth, GPU fleet state, reconciler activity, and a per-tenant usage/cost table, all on one screen for whoever’s on call.

Two of those blocks needed the same underlying data.

The pipeline-health block needed every tenant’s image records to compute stale-pending buckets.

The per-tenant table needed the same tenant’s image records to compute that tenant’s usage numbers — through UsageMetricsService.build_dashboard, a method that already existed and already queried DynamoDB for exactly this.

The straightforward implementation would have called build_dashboard once per tenant, which queries that tenant’s image partition, and then separately queried the same partition again to build the pipeline block.

For T tenants, that meant 2T DynamoDB queries against partitions we only needed to read once each.

There was no obvious for tenant_id in tenant_ids: await fetch(tenant_id) loop. Nothing screamed “N+1.”

Instead, I had two well-intentioned pieces of code, written for two different sections of the same page, each doing the right thing in isolation and the wrong thing together.

We caught it before it merged by asking the same question the textbook N+1 case teaches us to ask:

How many times does this data actually need to leave DynamoDB?

The answer was once.

So we fetched it once:

# One image-field fetch per tenant, shared by the pipeline block AND
# the per-tenant usage dashboards (build_dashboard accepts prefetched
# images) — avoids double-charging the DDB partitions.
images_by_tenant = await self._fetch_images(tenant_ids, attention)

results = await asyncio.gather(
    self._pipeline_block(images_by_tenant, days, now),
    ...
    self._tenants_block(tenant_configs, images_by_tenant, days, now, email_by_tenant),
    return_exceptions=True,
)

Then I changed build_dashboard so it could accept data the caller had already fetched instead of insisting on fetching it again:

async def build_dashboard(
    self,
    tenant_id: str,
    days: int,
    images: list[dict[str, Any]] | None = None,
) -> UsageDashboardResponse:
    """Build the dashboard; ``images`` lets callers that already fetched
    the tenant's image fields (admin dashboard) avoid a second query."""
    if images is None:
        images = await self.db.query_image_usage_fields(tenant_id)

One parameter. One shared dictionary threaded through two call sites.

2T queries became T.

There is an important detail here that I think is easy to miss.

We didn’t make the T queries sequential. They still run concurrently through asyncio.gather, one per tenant.

That distinction matters because concurrency can fool me into thinking I’ve fixed the problem.

Concurrency hides N+1’s latency cost.

T parallel queries can sometimes return in roughly the time of one query.

But concurrency does nothing to reduce the volume cost.

I’m still paying for T reads against T partitions. I’m still consuming T times the read capacity. And I’m still one slow or misbehaving tenant away from holding up the batch.

A parallel N+1 is faster to run and just as expensive to pay for.

The fix that matters is cutting the fetch count, not merely parallelizing it.

Cousin #2: An API With a Batch Endpoint You Forgot Existed

The same dashboard also needed CloudWatch metrics — reconciler activity across five pipeline stages times four counters each, giving us 20 data series, SQS queue ages for up to three queues, and GPU fleet throughput.

The naive implementation would have made a get_metric_data call per series.

That’s 20 calls for the reconciler block alone, plus one per queue, plus one for fleet throughput.

Every one of those is a network round trip. Every one is a CloudWatch API call subject to its own rate limits. Every one has its own chance to fail independently in the middle of the request.

This is probably the most common shape of N+1’s cousin that I now look for: not a data-modeling problem, but a case where I forgot that the SDK already has a native batch primitive for exactly what I’m trying to do.

boto3‘s CloudWatch client accepts a list of MetricDataQueries in a single get_metric_data call — up to 500 of them — and returns the results together.

So I built one small helper around that and used it everywhere we needed metrics:

async def _get_metric_data(
    self, queries: list[dict[str, Any]], start: datetime, end: datetime
) -> dict[str, float]:
    """One batched GetMetricData; returns {query_id: latest/sum value}."""
    resp = self.cloudwatch.get_metric_data(
        MetricDataQueries=queries, StartTime=start, EndTime=end,
        ScanBy="TimestampDescending",
    )
    ...

The reconciler block builds all 20 queries — 5 stages × 4 metrics — as one list and makes one call.

The queue-age lookup builds one query per queue and makes one call.

Fleet throughput reuses the same helper.

Twenty-some potential round trips collapsed into three calls for the whole dashboard build: one per logical block, not one per metric.

And I didn’t want the batching optimization to compromise resilience.

We kept the per-tenant and per-block exception isolation through asyncio.gather(..., return_exceptions=True).

That means one broken AWS permission or one tenant’s bad data can still degrade that panel to a signal_unavailable note instead of taking down the entire request.

For me, that’s an important architectural point: batching and graceful degradation aren’t in tension. I just have to design for both instead of treating “make it one call” as permission to let one failure take down everything.

Both of these changes shipped in the same commit, feat(admin): per-environment admin dashboard API, alongside 11 new tests.

Neither would have shown up on a code-review checklist that only asked, “Is there a loop calling the database?”

The first problem was two separate, individually-correct code paths.

The second was a missed SDK capability.

The habit that catches both is the same habit that catches classic N+1:

Before I merge anything that talks to a data store or external API more than once, I want to know exactly how many round trips it costs and whether that number scales with something that’s about to grow.

The Sequel: Batched Isn’t the Same as Cached

This is actually the part of the story I find most useful, because from the outside it looked like we’d already solved the problem.

Four days after that commit, I was checking why admin dashboard loads still felt sluggish when we found this:

Verified live: no ElastiCache exists in any deployed environment (zero clusters, no VALKEY_* env vars), so ValkeyCache silently no-ops and every dashboard load was a full rebuild (~29s admin builds, a Cost Explorer call per view).

We had designed the endpoint to be cached.

There was a ValkeyCache layer wired in from the start, a 3-minute TTL, and the whole shape of what looked like a normal caching story.

What we hadn’t verified was the one thing that mattered:

Did the cache actually exist in any running environment?

It didn’t.

ElastiCache had never been provisioned.

Every cache read quietly missed, every single time.

That meant every dashboard view was re-running the entire fan-out we’d just finished optimizing: T parallel DynamoDB queries, three batched-but-not-free CloudWatch calls, and a Cost Explorer call none of us wanted running on every page load.

We had reduced the number of round trips per build from something like 2T+20 down to T+3.

And then we paid that reduced-but-still-real cost on every request because nothing was actually skipping the build.

That was the moment the distinction became very clear to me:

Fixing N+1-shaped fan-out is necessary, but it isn’t sufficient if the whole expensive operation still runs on every request.

A cache I haven’t verified in a running environment is, for all practical purposes, no cache.

And an unverified cache in front of a well-batched N+1 fix just means I’m repeatedly paying a smaller bill instead of a larger one.

The real fix didn’t require adding another managed service.

Instead, we used what we already had — the images table and the long-running ECS task — rather than depending on infrastructure that had quietly never existed:

• A SnapshotStore writes gzip’d JSON dashboard snapshots into DynamoDB itself, with a read-time expires_at and a fail-open contract. Any DynamoDB hiccup degrades to a cache miss, which is just today’s behavior, never worse.

• A background warmer, running in the API process’s own lifespan, rebuilds the admin snapshot on a roughly 4-minute cadence and each tenant’s usage snapshot every 8 hours. We chose those cadences based on how often the underlying signals actually change, rather than picking arbitrary TTLs.

• The Cost Explorer block gets its own 8-hour snapshot inside the costs block, since Cost Explorer data itself only refreshes a few times a day. That takes it from roughly one call per view to roughly three calls a day.

• Both /usage/dashboard and /admin/dashboard now serve snapshots by default, with an explicit refresh=true escape hatch for anyone who genuinely needs the live number.

Nobody waits on a cold build anymore.

And, just as importantly, nobody is silently re-running a 29-second, multi-block AWS fan-out every time someone opens a browser tab.

Why Catching This Early Mattered More Than Usual, Here

Parjanya launched to its first self-serve trial cohort on 1st August.

Both fixes — the shared-fetch deduplication and batched metrics, followed by the snapshot cache that replaced a cache that never actually existed — landed before that date, while tenant counts were still near zero and nobody outside the team was looking at the admin dashboard.

That timing is the whole argument for dealing with N+1-shaped problems early.

At near-zero scale, 2T queries and T queries cost about the same amount of nothing.

A 29-second rebuild on every page view is merely annoying when I’m the only engineer clicking refresh.

None of this was a production incident.

But I could see exactly how it would become one.

As tenant count and dashboard traffic grew, the fan-out cost would scale linearly with exactly the number a growing SaaS product is supposed to grow.

The phantom-cache cost would scale linearly with exactly the traffic a real launch is supposed to bring.

We found both by asking a very simple question while the answer was still cheap to change:

How many round trips does this cost, and does that number grow with our own success?

I’d much rather answer that question before customers force me to.

What I’d Tell a Team Building This Kind of Thing

Ask the round-trip question before merge, not after a slow-page complaint.

For any code that touches a database or an external API more than once per request, I ask explicitly: how many round trips does this cost today, and what does that number scale with?

If the answer is “the number of tenants” or “the number of rows on this page,” that’s the moment to fix it — not a note for later.

A loop isn’t required for N+1’s cousins to exist.

Our duplicate-fetch case had no for tenant_id in tenant_ids: await fetch(tenant_id) shape at all.

It was two separately-written, individually-reasonable code paths that happened to want the same data.

I now watch for that pattern specifically: two features, built at different times, that each independently fetch “this tenant’s records” without either one knowing the other exists.

Check whether your SDK already has a batch verb before writing the loop.

CloudWatch’s GetMetricData takes a list.

So does DynamoDB’s BatchGetItem.

So do many well-designed cloud APIs from the last decade.

The fix for “N calls instead of one” is very often not custom code.

It is reading one more section of the SDK documentation.

Concurrency is not the same fix as batching.

asyncio.gather over N calls is faster than N sequential calls, and it is still N calls.

It hides the latency symptom without touching the cost or capacity symptom.

I don’t want a parallel version of the bug to pass code review as if it were the batched version.

A cache is a claim, not a fact, until I’ve verified it in every running environment.

The most expensive line in this whole story wasn’t a query.

It was an unprovisioned ElastiCache cluster that let a caching layer silently no-op for days while looking, in every log line and every code review, exactly like it was working.

If I can’t point to the cache-hit rate on a dashboard, I don’t want to assume the cache is working.

I want to verify it.

Small scale is when this is cheap to fix, not evidence that it doesn’t need fixing.

The version of this bug that costs nothing to run also costs almost nothing to fix.

The version that ships to production and grows with user count is the same bug, several engineer-days more expensive, with a support ticket attached.

Parjanya is the multi-tenant image QA platform we’re building at Phagyul for professional photographers. If your team has its own N+1 cousins — duplicate fetches across features that don’t know about each other, unbatched calls to an API that quietly supports batching, caches that looked wired but weren’t — I’d love to compare notes.

A few months ago, I wrote a short note arguing that developer-experience platforms are excellent at velocity, but that AI systems demand a different kind of architectural thinking.

My argument was that convenient shortcuts can eventually accumulate into what I called AI debt: runaway costs, reproducibility gaps, and debugging opacity.

That argument was written from conviction.

This one is written from production.

I launched Parjanya v2.0, our multi-tenant SaaS platform for professional photographers, as a self-serve trial on 1st August 2026.

The system has processed individual photography sessions containing 12,000 images and roughly 450 GB in a single pass, and today serves tens of thousands of requests a week.

Between the first sprint in March and the July launch, I went through four complete architecture generations.

Almost every generation boundary came down to the same question:

Do I buy the platform, or do I own the primitive?

Sometimes I adopted the managed service. Sometimes I replaced it. And sometimes I came very close to replacing a primitive with a managed service, only to run the numbers and walk away.

The four headline decisions were:

1. SageMaker vs. Bedrock vs. raw AWS primitives for VLM and SigLIP 2 inference

2. Vercel vs. S3 + CloudFront for the React + TypeScript frontend

3. OpenRouter vs. a self-hosted model gateway

4. Clerk vs. Cognito + SES for authentication and transactional email

There were also three less glamorous decisions that ended up moving the bill more than some of the headline ones:

• Spot and Graviton

• OpenSearch and Step Functions

• CloudWatch, networking and storage classes

For each decision, I wanted to understand not only what the comparison tables said, but what actually happened after the architecture met production.

That distinction turned out to matter enormously.

My operating rule today is simple:

Buy managed services for their operations, not their defaults — and leave them when their cost floor or cold start starts fighting your workload.

1. SageMaker vs. Bedrock vs. AWS Primitives

The decision at the center of Parjanya

The heart of Parjanya is batch visual inference.

A photographer uploads a session — routinely 4,000 or more images, and sometimes 12,000. Every image that passes a cheap deterministic technical-quality gate is scored by the VLM and embedded using SigLIP 2.

The workload characteristics are important because they tell me almost immediately where the different approaches will fit.

The workload is bursty, not steady.

A queue can go from zero to thousands of images and then return to zero. There can be days when almost nothing arrives.

It is also batch tolerant.

Nobody needs a score in 200 milliseconds. What matters is that 4,000 images are processed by morning.

It is privacy sensitive.

These are often unreleased professional photographs belonging to clients. Keeping image bytes inside our VPC is not merely an architectural preference; it is something I can actually explain to a customer.

And finally, the economics are dominated by GPU time.

Before optimization, GPU compute represented well over half of the entire bill.

That combination — bursty, batch-oriented, private and GPU-heavy — became the single biggest predictor of which architecture made sense.

I started with SageMaker

The obvious managed options were Bedrock and SageMaker.

Bedrock represented the pure developer-experience end of the spectrum: API calls, no infrastructure, no cold-start management and managed models.

SageMaker sat somewhere in the middle: managed infrastructure around our own container and model.

Raw AWS primitives were the opposite end: EC2 GPU instances, Auto Scaling, SQS, SNS, CloudWatch and an AMI with the model baked in.

I used all three lenses, and two of them in anger.

Generation 2 ran on SageMaker.

I started with a BYOC real-time endpoint and then moved to Batch Transform.

SageMaker did exactly what I expected it to do: it got me to a working GPU pipeline quickly.

That is the promise of DX, and in this case, it delivered.

Then two things happened.

Quantization changed the economics more than the platform did

Qwen3-VL-8B in BF16 wants roughly 16 GB of VRAM.

A T4 has exactly 16 GB — and does not support BF16.

Moving to NF4 4-bit quantization using bitsandbytes brought VRAM usage down to roughly 5 GB, which made an 8B model practical on a T4.

More importantly, it produced a roughly 13× per-image speedup.

Our batch cost collapsed from approximately $500 to $40–50.

That was my first major lesson from self-hosting:

The model configuration can be a much bigger cost lever than the hosting substrate.

A managed platform doesn’t automatically pull that lever for you.

Then SageMaker’s operational shape started fighting the workload

The first problem was cold starts.

The managed endpoint took on the order of ten minutes to become useful, largely because of model-weight loading.

Our replacement — an AMI/NVMe-staged model running on a raw EC2 GPU worker — loads roughly 17 GB of weights in under two minutes.

For a workload that scales from zero, warmup is latency.

The second problem was the cost floor.

A real-time endpoint costs money every hour it exists, whether a single image arrives or not.

One GPU endpoint represented a standing tax of roughly $500+ per month.

At one point, I discovered a forgotten endpoint costing almost $870/month.

Our worst single day was roughly $1,000, around three to four times our typical monthly development spend at the time. The root cause was endpoint provisioning and extended runs.

The most important early optimization was almost embarrassingly simple: a scheduled shutdown Lambda.

We kept the endpoint warm only during a seven-hour nightly window, which reduced its cost by roughly 70%, from about $540 to $160 per month.

That taught me something I now use as a general diagnostic:

When your best optimization for a managed platform is a cron job that turns the platform off, your workload and the platform’s default posture probably disagree.

The third problem was debuggability.

When something went wrong on primitives, I could inspect a queue, an alarm, an Auto Scaling event and a container log.

Everything was directly readable.

On the managed platform, the same investigation often disappeared behind an abstraction that I could not see through as clearly.

That was ultimately what made the decision for me.

I moved to EC2 Spot

In April, Generation 3 deprecated SageMaker entirely.

I moved to EC2 Spot GPU workers — primarily the g4dn family with NVIDIA T4s — inside an Auto Scaling group driven by SQS queue depth.

The fleet genuinely scales to zero.

The minimum size is zero.

A scheduler Lambda watches the queue every two minutes, and the scale-in rule only fires after both visible and in-flight messages have been zero for five consecutive minutes.

For Spot interruptions, I added a five-second metadata poll.

The worker finishes the current image, checkpoints the state to DynamoDB and exits cleanly.

After months in production, we have lost zero images to Spot interruption.

The result was substantially better than our early expectations.

Our largest autonomous run drained roughly 2,400 images in under eight hours on four Spot GPUs, scaled itself back to zero three minutes after the queue emptied, and required no operator intervention.

The total bill was approximately $20.

Pure GPU inference now costs roughly $0.002–0.005 per image, or around $0.005–0.008 all-in.

Compared with the SageMaker era, that is an order-of-magnitude difference.

Overall platform cost fell by roughly 65–75% from its pre-optimization peak.

At our worst, we were spending around $80–90/day.

Under load today, it is closer to $20–30/day, with an idle floor around $5/day.

Spot itself delivered a measured 56–61% saving against on-demand, roughly $0.26–0.29/hour versus about $0.55/hour for a T4 in Mumbai.

I also ran an A/B test of going 100% Spot.

It beat its own forecast.

Drain time roughly halved, cost per message fell by around 60%, and we experienced zero interruptions.

The written verdict was simply:

Stay 100% Spot.

There was another Spot lesson that was less obvious.

I now treat quota increases as spend risk, not simply performance capacity.

The cost is in instance-hours, not instance count. Increasing a GPU quota to “improve throughput” also increases the maximum possible blast radius on the bill.

I manage the on-demand/Spot quota ratio as deliberately as I manage the fleet itself.

What about Bedrock?

I did not dismiss Bedrock.

I benchmarked it.

A managed vision API came in at roughly $0.01/image.

Our Spot fleet sits around $0.002–0.005/image.

That is a 2–5× gap, but it exists because I am doing three things simultaneously:

• scaling to zero

• running a quantized model

• using Spot capacity

If I ran the same fleet on-demand and kept it always on (warm), Bedrock would win.

I keep that Bedrock number as a standing benchmark.

The day the fleet stops being well run, I want the spreadsheet to tell me so.

There are also two reasons I would keep self-hosting even if the economics reached parity:

1. Image bytes never leave our VPC.

2. I control model version pinning completely.

Those are not spreadsheet variables.

What the industry says

The broader industry experience broadly supports this progression.

Discord describes a deliberate strategy of prototyping against managed frontier APIs to validate whether current-generation models can solve the product problem, then moving to self-hosted open models once scale justifies the operational burden. https://discord.com/blog/developing-rapidly-with-generative-ai

Shopify built its Merlin ML platform on open-source primitives rather than committing to a managed ML platform, largely because its teams had conflicting requirements and needed a frictionless prototype-to-production path.

https://shopify.engineering/merlin-shopify-machine-learning-platform

Independent comparisons generally place the self-hosting break-even around 10–20k requests/day, with much larger savings at higher volumes.

And AWS itself frames Bedrock versus SageMaker as a genuine trade-off between serverless convenience and control.

My experience added a third fork:

sometimes the right answer is neither.

Sometimes the right answer is EC2, SQS and an AMI.

What production taught me that comparison tables didn’t

The first lesson was that primitives compose — but not freely.

Our warm-pool design passed architecture review and failed during

terraform apply.

AWS does not allow a warm pool on an Auto Scaling group with a mixed-instances policy.

The DIY tax isn’t necessarily writing more Terraform.

It is discovering the composition rules the hard way.

The second lesson was that design numbers are not production numbers.

Our AMI-bake design promised roughly 90-second model loads.

Fresh instances actually took 30–40 minutes.

Lazy EBS restore meant a new instance reading 17 GB of untouched blocks crawled at single-digit MB/s.

The solution was NVMe staging.

It brought the 17 GB load down to under two minutes — measured in production.

I now put a date and configuration hash next to every performance number in our documentation.

Benchmark rot is a debt class.

The third lesson was about deterministic gates.

Before anything reaches a GPU, a sub-cent Lambda gate rejects technically unusable files.

That rewrite replaced an earlier ML-based scorer, reducing that stage’s cost by roughly 90%, from around $13 to less than $1/month, while also reducing its cold start from about 15 seconds to roughly two seconds.

More importantly, the deterministic rule engine owns the accept/reject decision.

If I change the curation policy, I can replay stored model outputs through the new rules in under a minute for pennies.

I do not need to run the GPU again.

The cheapest inference is the inference I skip. The second cheapest is the inference I never have to redo.

And then there was the prompt.

The prompt is infrastructure.

Our worst GPU incident was a CUDA out-of-memory crash caused not by the model or the fleet, but by prompt creep.

The scoring prompt had quietly grown beyond 4,000 tokens.

Instead of buying a larger GPU at twice the price, I audited the prompt and cut it by roughly 60%.

A lot of the removed instruction was teaching the model to produce outputs that the deterministic rule engine downstream deliberately ignored.

I was literally paying VRAM to generate information the system threw away.

On a 16 GB card, token budgets are a debt ceiling.

2. Vercel vs. S3 + CloudFront

The frontend is a React 19 + TypeScript + Vite SPA.

It contains the gallery, curation workflow, uploads and administration interface, and serves photographers globally while the FastAPI backend runs in Mumbai.

During launch, I needed fast global static delivery, many deployments a day, instant rollback and strict security headers.

The default answer in 2026 is Vercel.

And I understand why.

Git-push deployments, preview environments, an edge network and almost no infrastructure knowledge required.

But I chose S3 + CloudFront.

The deployment pipeline is intentionally boring:

Vite build → S3 sync → CloudFront invalidation

GitHub Actions handles it through OIDC.

Merging to main deploys development.

Production remains a manual, human-triggered workflow.

The cache design does most of the work.

Hashed assets under /assets/* are immutable and carry a one-year TTL.

index.html gets a 60-second TTL.

That makes the cache policy itself the deployment lever.

A release becomes globally visible within about a minute, and rollback is simply redeploying the previous build.

Security headers — HSTS with preload, frame-deny and nosniff — are handled by a CloudFront response-headers policy.

The cost is generally in the single-digit dollars per month and, most months, sits inside the free tier.

The reason I chose it was not that Vercel is bad.

It was that the frontend was not where I needed Vercel’s strengths.

Cached SPA loads from London, New York and Sydney are already measured in tens of milliseconds.

What users actually feel is the dynamic API round trip to Mumbai.

A page making three to five sequential API calls can add more than a second from North America or Europe.

No edge frontend platform can fix that.

The answer is API design: batching, caching and eventually regional read replicas.

That reinforced the thesis I started with:

The edge is a traffic director, not a compute executor.

Static assets and routing belong at the edge.

Authentication, model resolution, inference and telemetry remain centralized.

For a Vite SPA, the DX delta was also surprisingly small.

Vercel’s real advantage appears in SSR, ISR, preview deployments and framework-integrated serverless functions.

I wasn’t using most of those features.

For me, the DX advantage amounted largely to one GitHub Action.

The third consideration was cost topology.

S3 + CloudFront scales predictably with traffic.

Platform function pricing scales with invocations — precisely the axis I hope will explode as Parjanya grows.

I didn’t reject Vercel.

I simply didn’t want to pay a platform premium for capabilities my architecture wasn’t using.

3. OpenRouter vs. a Self-Hosted Gateway

Every AI product eventually has to answer four questions:

• Which model serves this request?

• What happens when it becomes unavailable?

• Who tracks the spend?

• Whose infrastructure sees the payload?

Managed gateways such as OpenRouter answer these questions across hundreds of hosted models.

One API key.

Automatic fallback.

Unified billing.

For Parjanya, however, there was a twist.

Our production models are self-hosted open weights:

• Qwen3-VL-8B

• SigLIP 2

They run on our own GPUs.

There is no third-party inference endpoint sitting in the hot path.

So I asked a slightly different question:

What plays the role of the gateway when the models are mine?

I ended up decomposing the gateway into primitives.

Model resolution is a three-tier loading ladder on the GPU worker.

Tier 1: weights staged on local NVMe from the AMI bake.

Tier 2: S3 synchronization fallback.

Tier 3: Hugging Face download.

Every load records the tier.

In healthy production, I should only ever see Tier 1.

A Tier 2 or Tier 3 log line is therefore an alert.

I also made the upstream dependency deliberately boring.

Hugging Face is contacted exactly once per model version, during AMI baking in CI.

Production instances never download models from it.

A scheduled job compares the upstream model commit hash with our baked AMI tags and tells a human when a rebake is worth considering.

That gives me something I care about enormously in an inference system:

model versions are pinned by construction.

Fallback is handled by the mixed-instances policy across GPU families and the loading ladder.

Spend accounting comes from CloudWatch and tagged infrastructure.

Every inference dollar can be attributed to an EC2 line item.

For our particular architecture, a managed gateway would introduce three structural costs:

1. A per-request economic layer.

2. A data-path layer where client payloads transit someone else’s infrastructure.

3. A rate-limit layer where our ceiling becomes somebody else’s policy.

OpenRouter’s published model, for example, includes roughly a 5% platform fee on provider pass-through pricing.

For a self-hosted, single-model, VPC-contained workload, I don’t get enough routing benefit to justify those layers.

But there is an important concession here.

If I were calling frontier hosted models, I would absolutely use a gateway.

And I would probably use a managed one first.

Standing up LiteLLM with a database, cache and pager before I have meaningful multi-provider traffic is premature infrastructure.

The gateway decision comes after the hosting decision.

It is not independent of it.

The lesson from our fallback

A fallback I have never exercised is not a fallback.

For a while, our Tier 2 S3 fallback was configuration theater.

The environment variables existed in the design.

They were never actually injected into the container.

The fallback existed on paper and nowhere else.

I fixed it by deliberately breaking Tier 1 and watching what happened.

That is one of the biggest operational differences between managed and primitive systems:

if I decompose a managed capability, I inherit the obligation to rehearse every part of it.

4. Clerk vs. Cognito + SES

Authentication was the closest call.

Parjanya needs multi-tenant signup, email/password authentication, JWTs carrying an immutable tenant identifier, trial limits and transactional email for verification and password resets.

Clerk is extremely attractive here.

Polished UI.

Session management.

A much nicer developer experience.

And, critically, email that simply works.

The alternative was Cognito + SES.

I chose Cognito + SES.

Why

First, economics.

At our current scale, Cognito costs us essentially nothing under its applicable MAU threshold and remains materially cheaper through the growth range we are targeting.

For a product whose unit economics depend on very high gross margins on per-image pricing, a permanent per-MAU platform fee is a margin haircut on every future user.

Second, and more importantly:

the token is the tenancy model.

Our JWTs carry an immutable tenant_id claim.

The backend verifies it using RS256 against Cognito’s JWKS, deliberately avoiding a signing secret in the backend.

Every S3 bucket policy and DynamoDB access pattern keys off that claim.

The backend issues short-lived, tag-scoped STS credentials per tenant.

Buckets are per-tenant.

KMS keys are per-tenant.

Cross-tenant access is explicitly denied at the policy layer.

Auth is therefore not just a login box.

It is the root of the isolation tree.

I wanted that root inside the same Terraform, account boundary and audit trail as everything it protects.

A hosted authentication provider can issue my token.

But it cannot itself be the IAM principal that my storage policies deny against.

Third, there is the privacy argument.

One bill.

One IAM boundary.

No additional identity-data processor.

For a privacy-positioned product, that is a sales answer as much as it is an architecture answer.

And then DIY sent me the bill

This is where I think infrastructure writing is often too polite.

The DIY option has a cost.

I want to list mine.

Cognito immutable attributes really are immutable.

I verified that in production.

Get your custom claims right the first time.

Schema changes can also be pool-destroying.

Adding a new schema attribute through Terraform can force user-pool replacement and therefore destroy every user account.

I added a role claim through the CLI instead, knowingly accepting permanent drift for that resource.

Cognito also does not support cross-account user-pool migration.

When we reorganized into a multi-account AWS organization, the pool’s location became a one-way door.

Either users would re-register or we would face a mass password reset.

Then there were session bugs.

Our frontend token cache survived logout.

User two could briefly see user one’s data until I added an explicit cache clear at every login boundary.

Presigned upload URLs had another subtle problem.

They inherit the STS credential’s expiry rather than the nominal URL lifetime.

An upload presigned at minute zero failed at minute 61 with a signature that looked valid.

These are precisely the integration bugs that a mature hosted authentication SDK has probably already encountered.

And then there was email.

Cognito’s default email channel has a 50-email/day per-pool cap and is best effort.

During launch, password-reset emails were silently dropped.

The API returned success.

CloudTrail recorded the event.

No email arrived.

That is one of the worst failure modes in distributed systems:

The lying green checkmark.

The fix was SES with a verified domain, DKIM and developer sending enabled through Cognito.

That led into SES production-access review.

Our first request was denied, almost certainly because our submission said too little about bounce handling.

SES sandbox status became the sole external blocker on our release lock.

We eventually documented the ordering constraint in red:

auto-confirm cannot be switched off while SES is sandboxed, or every signup breaks.

This is exactly where Clerk’s pitch becomes compelling.

You don’t have to think about any of this.

And during launch week, that matters.

DIY auth is paid for in calendar time and attention.

Those were the two currencies I had the least of during launch.

My verdict

Cognito + SES was right for Parjanya.

But it was the closest decision of the four.

The SES saga is actually the strongest argument for DX in this entire post.

A managed platform’s ability to make an entire class of launch-week failures impossible has real value.

If authentication were not the root of our isolation model, I would have been much more comfortable paying a per-MAU platform fee at seed scale.

5. The Decisions That Don’t Make Blog Titles

Some of the decisions that affected the bill the most had nothing to do with DX platforms.

Spot — and learning when a lever isn’t a lever

The Spot decision gets most of the attention because the savings are obvious.

We measured roughly 56–61% savings versus on-demand.

Our A/B test of 100% Spot beat its forecast. There are additional optimisations like cost-performance optimisation by AWS, which I will detail in upcoming posts.

Interruption handling has lost zero images.

We even measured Spot pricing within the region and found that availability-zone prices for the same instance type could differ by more than 50%.

So the fleet biases toward the cheaper zone.

But some fashionable optimizations simply failed the arithmetic.

Multi-region Spot arbitrage died because inter-region data transfer consumed most of the projected savings.

A neighboring region also turned out not to have GPU capacity to arbitrage in the first place.

The more interesting story was Graviton.

Our first Graviton audit found that only one of four Lambda functions could even run on ARM64.

Native dependency wheels, CUDA and unbenchmarked inference libraries blocked the rest.

The projected saving was approximately 0.4% of the total bill — about a quarter of a dollar per month.

Meanwhile, quantization, Spot and scheduling were saving 40–60%.

So I wrote down the conclusion and walked away:

Sometimes the best architecture decision is knowing where not to spend your engineering time.

Months later, a dependency shipped ARM64 wheels.

The blocker disappeared.

We revisited the decision and migrated.

The measured Lambda saving was approximately 20%, almost exactly what we had projected.

Both halves of the story matter.

I needed the discipline to defer a fashionable optimization.

But I also needed the paper trail that allowed me to revisit it later without starting the research from scratch.

My rule now is simple:

A two-hour ARM benchmark costs pennies and beats two weeks of debugging a premature migration.

And I never flip Terraform to ARM64 before the image has actually run there.

6. OpenSearch and Step Functions

Two other managed services lost to arithmetic.

Search

I kept search in-process.

No OpenSearch.

No external vector database.

At our scale, the smallest useful OpenSearch domain costs more per month than our entire reconciliation control plane costs per year.

Rather than guessing, I measured the ceiling.

Today, search latency is around two seconds at a few thousand rows and degrades toward the load balancer timeout somewhere beyond 100k rows.

So I wrote down the trigger:

roughly 30 tenants.

That is when OpenSearch becomes a yes.

SigLIP 2 vectors are already ready for a proper vector store when that trigger fires.

Orchestration

I also rejected Step Functions and workflow engines.

Our orchestration is a roughly 200-line reconciler.

DynamoDB Streams provide the fast path.

An hourly sweep provides the backstop.

The entire control plane costs a few cents a month.

I made the same arithmetic-driven decision around AWS Config and commercial compliance tooling.

A weekly EventBridge + Lambda audit covers the rules we actually need at effectively zero cost.

There is a famous industry parallel here: Amazon’s Prime Video team moved a monitoring service away from Step Functions and Lambda to a plain ECS monolith and reported roughly 90% lower infrastructure cost.

But DIY orchestration has its own failure modes.

Our replay Lambda once lost a single IAM permission and logged AccessDenied on every invocation for half a day.

The dashboards stayed green.

Hundreds of images stalled.

Nothing drained.

Then, on our launch-baseline day, the reconciler itself became the incident.

A read-amplification bug caused it to consume tens of millions of DynamoDB read units in 24 hours.

The bill was roughly $15/day.

The pipeline itself was almost entirely healthy.

The repair mechanism had become the damage.

I added targeted query guards and reduced that spend by roughly 85–90%.

Both incidents ended as code.

An alarm on silence.

A conditional-write guard.

A query guard.

That is an important property of primitives:

Institutional memory can become code.

7. Where the Bill Actually Hides

Three numbers changed how I think about infrastructure.

The first was CloudWatch.

Ungoverned CloudWatch can quietly consume 20–40% of a small platform’s operating cost.

A single chatty DEBUG-level Lambda can generate roughly $20+/month in log ingestion — more than our entire early operating budget for that stage.

Embedded Metric Format instead of per-call metric APIs saved around $15/month by itself.

Composite alarms reduced alarm costs by roughly 70%.

I also stopped creating per-tenant dashboards.

At $3/dashboard/month, that becomes a fixed cost that scales in exactly the wrong direction.

With those changes, observability fell back to a few dollars per month.

The second number was networking.

While investigating that roughly $20 autonomous GPU drain run, I discovered that almost half the bill wasn’t GPU time.

It was NAT gateway egress from around 180 GB of S3 reads.

The GPU wasn’t the expensive part.

The data path was.

VPC gateway endpoints for S3 and DynamoDB eliminated most of that cost and had already saved roughly $30–45/month of standing NAT expense.

A handful of interface endpoints cut the remaining NAT data by another ~80%.

That was another reminder that no SageMaker-vs-EC2 comparison table is going to tell you that your data path may cost more than your compute.

The third number was storage.

At photography scale, storage-class economics matter.

I rejected S3 Intelligent-Tiering for previews because its per-1,000-object monitoring fee becomes meaningful when you have millions of tiny files.

Instead, previews go directly into Infrequent Access, around 46% cheaper than Standard in our region.

Originals step down to Glacier Instant Retrieval, around 83% cheaper.

Rejected images move to Deep Archive after a grace period.

At 10 TB of rejected images, that’s roughly $10/month.

The result is about 3.7× cheaper than naïvely keeping everything in Standard.

The broader industry has demonstrated this at enormous scale.

Canva, for example, moved tens of billions of objects to Glacier Instant Retrieval after measuring that access collapses after the first couple of weeks, saving millions annually.

The meta-lesson for me was even simpler.

Our weekly audit Lambda costs effectively nothing.

It caught roughly 7,500 objects sitting in the wrong storage class for months.

It also found an S3 versioning configuration where the console suggested around 2,000 objects while the bucket actually contained more than 100,000 hidden version objects.

Left unchecked, that could have grown into millions of noncurrent versions, terabytes of invisible storage and a 5–10× inflated storage bill.

On primitives, audit is the platform.

8. The Incident Ledger

A fair DIY-versus-DX discussion has to include the incident ledger.

The operational burden that managed platforms sell against is real.

Between the first sprint and the post-launch study window, I logged 17 incidents across 12 classes.

Here are a few.

Terraform drift nearly became a self-inflicted outage.

After weeks of console-side firefighting, a routine Terraform plan came back with:

173 to add.
127 to destroy.

A blind apply would have deleted the NAT gateway, the DynamoDB table containing every VLM verdict and the Cognito user pool containing every login.

Getting back to a clean 0/0/0 state took roughly four focused hours spread across four days.

A managed DX platform wouldn’t have given me that state file.

But it also wouldn’t have given me a state file proving exactly what production was.

There was a six-hour preview outage caused by deleting the wrong “stale” grant.

What looked like a leftover origin-access policy was actually the live one.

Every preview returned 403 until I restored it.

There was a burst-photography incident caused by a silently failing install.

A shell || echo swallowed both a 404 and a missing archive.

We shipped a worker image without its EXIF tool.

Near-identical burst frames then started being falsely rejected as duplicates.

Launch went out with exact-match-only deduplication as the safe setting.

There was a wrong-bucket incident where hundreds of images hung on repeated download errors because queue payloads contained a valid key pointing to the wrong bucket.

And there was a weekend bulk import where messages silently expired from SQS because queue retention was shorter than the weekend.

Retention is now an invariant with a name.

Not a default.

These incidents sound like arguments against primitives.

I actually think they demonstrate the opposite.

Every incident ended as code:

• a Terraform validation block

• a conditional write

• a CI grep

• a CloudWatch alarm

• a queue invariant

Institutional memory became executable.

And across all 17 incidents, recurrence after adding the guardrails is currently zero.

That is the bargain I am making with primitives.

I accept the initial operational burden in exchange for owning the system completely enough to turn every lesson into a permanent constraint.

The multi-account reorganization was another example.

It was triggered partly by a deployment script accidentally pushing to production from the wrong local context.

The reorganization added roughly $10/month in duplicated audit tooling and moved around 135 GB of data for pennies.

In return, it dramatically reduced blast radius.

That kind of control is difficult to purchase from a single-tenant DX platform’s pricing page.

One uncomfortable AI footnote

Parjanya was substantially AI-assisted in its construction.

Two of the nastiest silent failures in the incident ledger — the swallowed exiftool installation and a bare-except registry writer — were both machine-written and machine-reviewed code.

Their common property was silence.

The answer wasn’t to use less automation.

It was to write better contracts:

• CI greps for exception-swallowing patterns

• log-line contracts for every pipeline tier

• alarms on absence

• explicit invariants

The same discipline that makes primitives operable also makes AI-written code operable.

9. The Repatriation Debate, From the Cheap Seats

All of this sits inside the much larger cloud-versus-hardware debate.

37signals has made the case for leaving the cloud, with a reported reduction from roughly $3.2M to $1.3M annually and projected savings of more than $10M over five years.

Ahrefs has argued that owned hardware saved them hundreds of millions compared with AWS list pricing.

Canva has taken almost the opposite position: stay in the cloud, but become extremely good at engineering the bill.

At Parjanya’s scale, I’m much closer to Canva’s school.

But I think there is an important distinction.

I haven’t left the cloud.

I also haven’t left managed services.

DynamoDB, SQS, Lambda, CloudFront and Cognito are still managed services in our architecture.

I keep them because I want their operations.

What I left were the managed services whose cost floors and cold starts fought the workload.

A real-time inference endpoint billing through the night for a queue that is empty by midnight isn’t an operations benefit.

It is a subscription to someone else’s default posture.

That is the distinction I think the broader repatriation debate sometimes misses.

It isn’t really:

cloud vs. metal

or:

platform vs. primitives

It is:

What is the right economic and operational shape for each individual workload?

And that answer should be re-run when the workload changes.

10. The Decision Framework I Use Now

After five months and four architecture generations, I have reduced the decision process to four questions.

1. Is the expensive part of the hot path bursty?

If yes, the platform’s cost floor may be your enemy.

Scale-to-zero primitives are probably worth the setup cost.

If the load is steady, the cost floor becomes less relevant and the managed platform’s operational value may be worth buying.

That one question explains why I left SageMaker, benchmarked and declined Bedrock, declined Vercel and kept DynamoDB on-demand.

2. When it breaks at 2 a.m., can I read the failure surface?

This ultimately mattered more to me than cost.

A queue.

An alarm.

An Auto Scaling event.

A container log.

These are things I can read.

If my team cannot or does not want to own that failure surface, the managed abstraction is doing real work for us.

I should pay for it.

3. What currency does the DIY option cost — and do I have that currency right now?

Cognito + SES cost me launch-week calendar time.

That was the scarcest resource I had.

The exact same decision at another point in the company’s life would have cost almost nothing.

So I no longer think only in dollars.

I ask:

What resource am I actually short of?

Money?

Engineering hours?

Attention?

Launch-week calendar time?

Operational confidence?

The right architecture can change depending on the answer.

4. Have I written down the condition that would reverse my decision?

This is probably the most useful practice I have adopted.

A “no” is not a permanent architectural belief.

It is a decision with a trigger.

For example:

• vLLM becomes a yes around 500 active users or sustained 100+ images/minute.

• OpenSearch becomes a yes around 30 tenants.

• The monorepo decision changes with team growth.

Every rejected option gets an un-rejection trigger.

That turns architecture from ideology into a portfolio that I can rebalance.

The Operating Sentence

The note that started this journey argued that high-performing platforms should choose DX where it helps, and control where it matters.

After a launch, four architecture generations, seventeen logged incidents and a roughly two-thirds reduction in our platform bill, I would sharpen that sentence.

This is the rule I now keep at the top of our architecture decision register:

Buy managed services for their operations, not their defaults. Leave them when their cost floor or cold start fights your workload. And put deterministic gates around expensive models — the cheapest inference is the inference you skip.

Parjanya v2.0 is now live as a self-serve trial.

The production system runs Qwen3-VL-8B in NF4 and SigLIP 2 So400M on scale-to-zero Spot GPU workers in ap-south-1.

Everything sits behind a deterministic rule engine that owns every accept/reject decision.

The model observes.

The system decides.

And, perhaps more importantly, I now have a much clearer idea of when I want a platform to make the hard parts disappear — and when I would rather own the hard parts myself.

The original note documented how a combination of EC2 GPU warm pools, Auto Scaling Groups, SQS, EventBridge, DynamoDB Streams, and pre-baked GPU AMIs enabled Parjanya to eliminate SageMaker cold-start penalties, achieve scale-to-zero economics, and build a replay-native control plane capable of self-repair and reconciliation. Rather than treating GPU inference as a model-serving problem, the architecture reframed it as an orchestration and reconciliation problem, resulting in dramatically lower startup latency, parallel execution, improved replay safety, and reduced operational cost.

This article expands on that foundation by examining the complete system architecture, the TBIE (Truth, Belief, Intent, Execution) operating model, the move from polling to event-driven reconciliation, and the real production incidents that ultimately shaped the platform’s reliability model.

EXECUTIVE SUMMARY

Parjanya v2.0 is Phagyul AI’s image ingestion and GPU-based visual scoring, later evolved as curation platform by replacing score engine to rule engine.

Its nightly VLM inference pipeline was originally built on Amazon SageMaker. The pipeline processed photographer uploads using Qwen3-8B-VLM and worked — but it revealed a fundamental architectural mismatch over time. Wall-clock time regularly stretched to ~11 hours. Cold starts ate into every nightly window. Replay handling for failed enrichments grew fragile. Sequential execution prevented scale-out. And persistent managed-platform overhead stopped making financial sense for burst-only workloads.
The core insight that drove the migration was a reframing of the problem. The question was no longer

‘how do we run a model?’

It became:

‘how do we orchestrate burst-scale inference reliably, replay safely, and cost-efficiently?’

That reframing led to retiring SageMaker and rebuilding the pipeline entirely from native AWS primitives — EC2 GPU warm pools, Auto Scaling Groups, SQS-driven orchestration, EventBridge-triggered scale-out, and pre-baked GPU AMIs with Qwen3-8B-VLM weights embedded.

This is a detailed internal engineering deep dive into that migration. It covers the original architecture and its failure modes, the design decisions behind each new component, how TBIE (Truth, Belief, Intent, Execution) reconciliation patterns were applied to make the system replay-safe and operationally resilient, the specific production incidents encountered and how each shaped the architecture, and the guardrails now embedded to prevent regression.
Key outcome: the rebuilt system achieved near-zero or sub-minute startup latency, parallel GPU workers replacing sequential processing, event-driven replay handling, scale-to-demand orchestration, and drastically reduced idle GPU waste. More importantly, it became a system that could reason about and repair its own state — a reconciliation system, not just a pipeline.

BACKGROUND: WHAT IS PARJANYA V2.0?

Parjanya v2.0 is the image ingestion and GPU-powered visual scoring platform at Phagyul AI Systems. Photographers upload raw images; the platform extracts technical metadata (EXIF, resolution, format), performs visual-language model inference to produce content quality scores, enforces content policy, and writes enriched metadata back to the workflow database.

The data flow is:

• Photographers upload images to per-tenant S3 buckets via browser-based presigned URLs → An S3 event triggers a Graviton Lambda for EXIF extraction and technical validation.

• On successful validation, a DynamoDB workflow row is written with status pending_vlm_enrichment.

• A GPU worker consumes this work, runs Qwen3-8B-VLM inference, and writes scored metadata back(later recuration) to DynamoDB.

• A control plane (replay Lambda, reconciliation loop) monitors for stalled or missing work and re-orchestrates as needed.

• The platform spans immutable uploads (S3), workflow state (DynamoDB), intent queues (SQS), Lambda orchestration, GPU autoscaling (ASG + EC2), browser-side uploads with CORS policies, content policy enforcement, and tenant isolation.

THE ORIGINAL SAGEMAKER PIPELINE

Parjanya’s first VLM inference implementation used Amazon SageMaker as the compute platform. A nightly batch job was triggered on a schedule. SageMaker endpoints were provisioned, images were submitted for inference in sequence, results were collected, and endpoints were torn down.

The original system worked. It produced correct results. But it had structural problems that compounded as the platform grew:

• Cold-start tax: Every nightly run incurred GPU cold starts before the first image could be processed.

• Sequential execution: Image inference ran sequentially across the batch; there was no parallelism at the worker level.

• Fragile replay: Failed enrichments required increasingly fragile custom replay logic layered on top of SageMaker’s job model.

• Idle cost: The platform was billed for managed infrastructure overhead even during windows with no work to do.

• Throughput ceiling: Wall-clock time regularly stretched to approximately 11 hours per nightly run.

THE ARCHITECTURAL REFRAMING

The decision to migrate was not triggered by a single catastrophic failure. It was triggered by a change in how the problem was understood.

Once the problem was stated that way, SageMaker was the wrong tool. SageMaker is an excellent managed platform for running models. It is not designed to be a burst-scale GPU orchestration layer with event-driven replay. Those properties had to be bolted on from outside, which is why the replay logic was fragile and why cold start latency was unavoidable.
Native AWS primitives offered a better fit for the actual problem: EC2 GPU instances could be pre-baked with model weights, Auto Scaling Groups could scale to demand rather than a fixed schedule, SQS provided durable, replayable intent, and EventBridge could trigger orchestration the moment new work appeared. The result is not a simpler system — it is a more intentional one.

THE NEW ARCHITECTURE: NATIVE AWS- PRIMITIVES

EC2 GPU WARM POOLS

Rather than provisioning GPU instances from scratch each time a job runs, Parjanya v2.0 uses EC2 warm pools attached to the Auto Scaling Group. Instances are pre-initialised and held in a stopped or standby state. When the ASG scales out, warm pool instances start in seconds rather than minutes, eliminating the cold start latency that dominated the original SageMaker pipeline.

Warm pool design note: instances in the warm pool must be pre-baked with model weights already on disk. Do not rely on model download during instance boot — download failures become silent launch failures. The AMI is the contract between infrastructure and runtime.

PRE-BAKED GPU AMIs WITH QWEN3-8B-VLM WEIGHTS

Each GPU AMI is built with the Qwen3-8B-VLM model weights, runtime dependencies, and worker application code baked in. The AMI is immutable. When a new model version or dependency is needed, a new AMI is built and the launch template is updated. This makes the execution environment reproducible and testable before deployment.

Key constraints on AMI design:

All model weights must be present at AMI build time — no remote download at boot.
Python environment, CUDA libraries, and worker scripts must be included in the AMI.

The launch template image tag and the AMI must be kept in sync; stale launch templates are a production risk (see Incident 6).

Container and model artifact versions are a compatibility pair — they must be validated together.

AUTO SCALING GROUPS (ASG)

An Auto Scaling Group manages the GPU worker fleet. The ASG is configured with a minimum of 0 instances (scale-to-zero for cost efficiency) and a maximum that reflects the burst capacity needed during active processing windows. Desired capacity is driven by queue depth rather than a schedule.

ASG configuration decisions:

1. Scale-out is triggered by the replay Lambda when SQS queue depth exceeds threshold.

2. Scale-in is time-based to prevent premature termination of in-flight workers.

3. Instance type selection balances GPU memory requirements (Qwen3-8B-VLM needs sufficient VRAM) against cost per inference.

4. Warm pool instances buffer the latency between a scale-out trigger and first inference.

SQS-DRIVEN ORCHESTRATION

SQS is the durable intent layer. Each message represents a single VLM inference job for one image. Messages are not ephemeral transport — they are the replayable journal of work that must happen. This is a TBIE design principle: intent must survive transient failure.

SQS configuration decisions:

• Visibility timeout is set to exceed the maximum realistic inference time to prevent duplicate processing.

• Queue retention is set to exceed the maximum expected outage window (e.g., weekend + holiday gap) so intent survives scale-to-zero periods.

• A dead-letter queue captures messages that exceed the maximum receive count, enabling manual triage of terminal failures.

• Message attributes carry routing metadata: tenant ID, image key, source bucket, expected output location.

• Queue retention is not a generic operational setting. It is a resilience parameter. If the GPU fleet is at zero for 3 days, messages must still be present when the fleet wakes up. Set retention to match your worst-case outage window, not a default.

EVENTBRIDGE-TRIGGERED SCALE-OUT

Scale-out is event-driven via EventBridge and DynamoDB Streams. When a DynamoDB row reaches pending_vlm_enrichment, the stream emits an event. A stream processor filters and batches records, then invokes the replay Lambda. The replay Lambda evaluates queue state (Belief) against database state (Truth), emits SQS messages (Intent), and increases ASG desired capacity (Execution trigger).
This replaces a polling model where a scheduled timer would periodically scan for work. The event-driven approach gives lower latency, eliminates wasted invocations during quiet periods, and makes the system’s reaction to new work causally clear:

Truth changes → control plane notices → Intent is created → Execution follows.

REPLAY LAMBDA (CONTROL PLANE)

The replay Lambda is the reconciliation control plane. It is invoked by DynamoDB Streams and by periodic safety-net schedules. Its job is to compare Truth (DynamoDB workflow state) against Intent (SQS queue depth) and repair any divergence: emitting missing messages, resetting stale state, and triggering ASG scale-out when work is present.
The replay Lambda is a first-class reliability surface, not a background helper. If it fails silently, every other safety net becomes weaker. Its observability must be stronger than ordinary worker observability.
Monitor its error rate, execution duration, and whether it successfully reads queue attributes. A lambda that runs but cannot see queue state is functionally blind.

TBIE: TRUTH, BELIEF, INTENT, EXECUTION

TBIE is the operating model used to design, debug, and evolve Parjanya v2.0. It decomposes a distributed system into four conceptually distinct layers. The model is most valuable as a diagnostic tool: when the system behaves unexpectedly, TBIE provides a structured vocabulary for identifying which layer has diverged.

TRUTH

Truth is the durable, authoritative record of what is actually true. It must survive worker restarts, deploys, transient outages, and infrastructure drift.
In Parjanya v2.0, Truth lives in:

S3 — immutable image uploads and preview artifacts. If an object is in S3, it exists.

DynamoDB — workflow rows recording status, EXIF metadata, VLM scores, curation state, and hashes.

Derived committed state — VLM output written durably after successful inference.

Truth answers questions like:

• What objects exist?

• What stage is this image in?

• What has been computed?

• What is the authoritative record for this batch?

BELIEF

Belief is the operational view of reality. It is derived from Truth and current signals, but it may lag, be partial, or be noisy. Belief is what operators see first — but it must never be confused with source-of-record Truth.

In Parjanya, Belief includes:

1. SQS queue depth and in-flight message counts.

2. ASG desired capacity, running instance count, and warm pool state.

3. Worker health signals and CloudWatch alarms.

4. GPU availability and instance lifecycle state.

A healthy Belief is not the same as a healthy system. The queue can appear empty while Truth still holds a large pending backlog. That gap is a reconciliation failure, not an idle system.

INTENT

Intent is the durable record of work that has been requested. It is the replayable journal of ‘please do this work.’

The critical property of Intent is replayability: if Intent can be lost too early, transient failures become permanent failures requiring manual rescue.

In Parjanya, Intent is:

• SQS messages — each representing a single VLM inference job.

• Replay jobs — reconstructed Intent emitted for rows that had pending Truth without corresponding queue messages.

• Regrade jobs — Intent emitted for historical reprocessing after policy changes.

EXECUTION

Execution is the stateless work that turns Intent into side effects and updated Truth. Execution is where work happens, but in a resilient system it must never be allowed to erase the record of what was supposed to happen.

In Parjanya, Execution includes:

• Graviton Lambdas for EXIF extraction and technical validation.

• GPU workers running Qwen3-8B-VLM inference and writing scored results to DynamoDB.

• Replay Lambda emitting Intent and triggering ASG scale-out.

• ASG and EC2 actions that instantiate the worker fleet.

TBIE AS A DIAGNOSTIC MODEL

The power of TBIE is operational. Once internalised, it changes the first question asked during any incident. The question is no longer ‘is the queue healthy?’ or ‘are the GPU workers running?’ Those are Belief signals. The real question is:

Is the system waiting because Intent is missing — or because Intent is present but Execution cannot realise it?

That single question narrows the search space dramatically and prevents wasted time on the wrong layer.

EVENT-DRIVEN RECONCILIATION

Parjanya initially used a scheduled polling model. A timer would wake the replay Lambda periodically. The Lambda would scan for pending work and, if appropriate, emit SQS messages and trigger GPU autoscaling. This worked but introduced avoidable latency, wasted invocations, and operational ambiguity.
The move to DynamoDB Streams changed the architecture fundamentally. Now, when a workflow row reaches pending_vlm_enrichment, the stream emits an event immediately. An event source mapping filters for relevant writes, batches records briefly, and invokes the replay Lambda. The Lambda discovers missing or retriable work, emits Intent into SQS, and kicks the GPU ASG when necessary.

The net effect:

Reconciliation latency dropped from O(poll_interval) to O(stream_propagation) — typically seconds.

• Wasted Lambda invocations during idle periods eliminated.

• Causal chain made explicit: Truth changes → stream emits → Lambda reacts → Intent created → Execution follows.

• Operators can reason about system state without wondering ‘has the next poll happened yet?’

• A reconciliation system does not rely on humans noticing a stalled queue and manually fixing it.

It detects, explains, and repairs the divergence between Truth and Intent on its own. Event-driven streaming is what makes that self-repair fast enough to be useful in production.

PRODUCTION FAILURE MODES & LESSONS

The following incidents occurred during the development and operation of Parjanya v2.0. Each is documented here not as a post-mortem but as a design lesson — how the failure shaped the architecture, and what guardrail was added as a result. TBIE is used throughout as the diagnostic lens.

Incident 1: Transient GPU Failure Destroying Intent

GPU workers were deleting SQS messages before the retry window had been exhausted. A worker could encounter a temporary S3 download failure or an OOM condition, write a failure status to DynamoDB, and still delete the queue message. At that point, Truth said the work had failed, but Intent no longer existed — the system had no automatic path to retry.

TBIE diagnosis: Execution destroyed Intent prematurely.

Resolution:

Workers now return explicit outcome signals: retry, terminal_failure, or completed.

retry: preserve the message. Visibility timeout and receive count govern re-attempts.

terminal_failure: write durable failed Truth, then delete the message.

completed: write final Truth, then delete the message.

That separation preserves replayability while still allowing durable failure when necessary. The message deletion decision became a function of execution outcome, not a side effect of worker exit.

Worker outcome protocol

if outcome == ‘retry’:
# Do NOT delete. Let visibility timeout expire.
return # SQS will re-deliver
elif outcome == ‘terminal_failure’:
write_durable_failure(db, image_id)
sqs.delete_message(receipt_handle)
elif outcome == ‘completed’:
write_enriched_truth(db, image_id, scores)
sqs.delete_message(receipt_handle)

Incident 2: Pending Truth Without Intent (Reconciliation Gap)

DynamoDB rows remained in pending_vlm_enrichment status, but no corresponding SQS messages existed. Queue depth showed zero. The system appeared idle. The backlog still existed in Truth but was invisible to Execution.

TBIE diagnosis: Truth was pending while Intent never existed. Classic reconciliation gap.

Resolution:

• The replay Lambda now scans for pending rows without corresponding queue messages.

• For each orphaned pending row, it regenerates a queue message and kicks the GPU ASG.

• DynamoDB Streams ensure this scan is triggered promptly rather than waiting for a polling cycle.

• The key insight: this is not a worker bug. Workers cannot fix work that was never queued. The control plane must own the responsibility of keeping Truth and Intent aligned.

Incident 3: Retry vs. Replay Distinction

Some failures — S3 download errors, temporary GPU OOM — are retryable once the underlying condition is fixed. But simply re-enqueueing the message is insufficient when the workflow row contains partial or stale derived fields from the failed attempt.

TBIE clarification:

a retry is another attempt at the same execution. A replay is a deliberate reset to a pre-execution boundary, removal of stale derived state, and emission of fresh Intent.

Resolution:

Retry: re-attempt on the existing message, preserving current workflow state.

Replay: reset the DynamoDB row to a clean pre-VLM boundary (clear stale score fields, VLM status, partial outputs), then emit a fresh SQS message.

Replay is a controlled reconstruction, not just another attempt on top of corrupted state. The distinction prevents accumulating corrupted derived fields across multiple failed attempts.

Incident 4: Wrong Bucket, Right Key (Deterministic Misrouting)

An SQS payload pointed to a valid preview key but in the wrong bucket. The worker repeatedly failed with S3 download errors even though the file existed — just not at that address. This was deterministic misrouting, not flaky execution.

TBIE diagnosis: Intent encoded the wrong target. Not an Execution failure — an Intent construction failure.

Resolution:

Preview keys now canonically use the uploads bucket as the authoritative source.
Intent construction validates bucket routing before emitting to SQS.
Eligibility checks were tightened: only objects confirmed present in the correct bucket are enqueued.
The lesson generalises: Intent must encode the correct target, not just some valid key. A message that is ‘almost correct’ will fail deterministically and indefinitely until the Intent itself is repaired.

Incident 5: Policy Change Requiring Historical Replay

Content policy was updated to more strictly reject non-photographic content: screenshots, banners, infographics, social cards, slides, illustrations, and mockups. Historical images scored under the old policy needed to be re-evaluated.

TBIE framing: policy change = controlled replay, not a patching exercise.

Resolution:

Affected rows were identified by policy version marker in DynamoDB.
Each affected row was reset to a pre-VLM boundary: stale score fields cleared, VLM status reset to pending.
Fresh Intent was emitted into SQS so GPU workers re-scored under the new policy.
This is one of the most powerful properties of a replay-native system: policy evolution does not require one-off correction scripts. It becomes a controlled reprocessing event with clear before-and-after semantics. The same replay machinery used for failure recovery also handles deliberate re-evaluation.

Incident 6: Execution Drift Below the Application Layer

Two distinct incidents exposed failures where application code was correct but the execution environment was stale.

6a: Stale Launch Template (Image Tag)

A launch template continued to reference an old image tag that no longer existed. Cloud-init failed when pulling the nonexistent container image, and the GPU instance came up without ever starting the worker. The worker log group was empty — the application layer never ran.
Complete silence in the GPU log group is a critical signal. It usually means execution failed before the application layer started: image pull failure, IAM denial during bootstrap, or cloud-init crash.

6b: Incomplete AMI (Missing Model Artifacts)

A separate incident involved an AMI that was missing the Python files required by Qwen3-8B-VLM (a model that relies on remote code). The container started successfully, but the model loader crashed because local artifacts were incomplete.

Resolution — both incidents:

• Launch templates now include version assertions checked at deployment time.
  AMI build pipelines validate the presence and integrity of all model artifacts before publishing.

• Container version and model artifact version are treated as a compatibility pair, validated together.

• Deployment tooling refreshes launch templates automatically when image tags change.

Incident 7: CORS and Browser Upload Split-State Failure

The backend could write a pending upload record and generate a presigned URL, but the browser’s PUT could fail because of CORS misconfiguration. In that case, Truth at the API layer looked correct while Execution at the browser layer never completed.
Investigation revealed multiple independent configuration failures could co-exist: a bucket rejecting wildcard origins under restricted public bucket settings, while the client simultaneously used a global S3 endpoint that returned redirects that browsers would not follow on OPTIONS preflight.

TBIE diagnosis:

Intent existed (presigned URL generated, pending row written). Execution was blocked by infrastructure configuration drift. Two independent misconfiguration layers were both wrong simultaneously.

Resolution:

• CORS policies are version-controlled and validated in CI before deployment.

• Bucket CORS and public access settings are treated as a matched pair, not configured independently.

• Upload path includes client-side detection of presigned URL PUT failure with structured error reporting.

• Pending rows created without a completed upload are detected by the replay Lambda and age-gated before re-emission.

Incident 8: Control Plane Blindness (Replay Lambda IAM Failure)

The replay Lambda lacked permission to read SQS queue attributes. The function continued to be scheduled and invoked, but it crashed before it could inspect Belief (queue depth) or emit Intent. From the outside, everything appeared enabled. In reality, the control plane had gone blind.

This is the most dangerous failure mode in the system. A silently failing control plane means every other safety net weakens simultaneously. Worker failures that should self-repair accumulate without rescue. Operators see green infrastructure dashboards while reconciliation has effectively stopped.

Resolution:

• The replay Lambda’s IAM policy now explicitly grants sqs:GetQueueAttributes, sqs:GetQueueUrl, and sqs:SendMessage.

• IAM policy drift is checked in deployment assertions — the Lambda will not deploy if required permissions are absent.

• The Lambda now emits a structured heartbeat metric on every successful invocation, including after inspecting queue state.

• Alarm on absence of heartbeat metric: silence means the control plane has likely failed.

The operational philosophy shift from this incident: worker failures are recoverable noise. Control-plane blindness is a systemic risk. Observability on the control plane must be stronger than on individual workers.

Incident 9: Dynamic Tenant Discovery and Queue Retention

As the platform grew, static tenant lists became a bottleneck. Reconciliation could not depend on manually editing infrastructure configuration every time a new tenant was onboarded. Additionally, when the GPU fleet was scaled to zero for cost savings, queue retention was set too short — messages expired before the fleet woke up.

Resolution:

The control plane moved to dynamic tenant discovery from DynamoDB. Tenant registry markers are written by the upload path and discovered by the replay Lambda at runtime.
Queue retention policy was extended to match the maximum expected outage window, not an idealised constant.
New tenant onboarding now automatically creates the required DynamoDB registry marker without manual infrastructure changes.

OPERATIONAL MODEL AND RUNBOOK THINKING

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. The starting question is no longer ‘is the queue healthy?’ or ‘are GPU workers running?’ — those are Belief signals. Every incident investigation now begins with a TBIE classification:

THE DIAGNOSTIC SEQUENCE

• Step 1: Check Truth

Is DynamoDB accumulating rows in pending_vlm_enrichment? If so, the platform believes unfinished work exists, regardless of what the queue shows. A growing Truth backlog is the authoritative signal that the system is behind.

• Step 2: Check Intent

Does SQS contain replayable work items? High Truth backlog + near-zero queue depth = reconciliation failure, not execution failure. The replay Lambda itself becomes suspect. Either it stopped emitting Intent, or it lost the ability to discover the work.

• Step 3: Check Execution

If Intent exists but the backlog still does not move, shift to Execution health. GPU worker logs become critical. Check for:

1. Deterministic S3 download errors (wrong bucket routing — see Incident 4).
   Repeated OOM crashes (model too large for instance type).

2. Complete log group silence (execution failed before the application layer — check cloud-init).

3. Stale launch template (image tag no longer exists — check ASG launch configuration).

• Step 4: Check the Control Plane

If queue visibility is unchanged while pending Truth continues growing, the replay Lambda itself is the focus. Check: Is it being invoked? Is it completing successfully? Can it read sqs:GetQueueAttributes? Is it emitting its heartbeat metric?

Operational summary of diagnostic signals:

• Truth backlog growing → unfinished work exists

• Intent near zero despite Truth backlog → reconciliation failure

• Intent present but Execution failing → runtime or routing failure

• GPU log silence → bootstrap failure (cloud-init, image pull, IAM at boot)

• Replay Lambda silence → control-plane failure (IAM, crash, permission gap)

• No heartbeat metric → control plane has gone blind

INCIDENT RESPONSE DECISION TREE

At 02:00 AM, when the queue looks empty but customers are waiting:

1. Check Truth: pull DynamoDB pending count. If non-zero, the system is behind regardless of queue state.

2. Check replay Lambda logs: has it run in the last cycle? Did it complete? Did it emit messages?

3. Check SQS: are messages present? If Truth is high and SQS is low, replay Lambda is suspect.

4. Check ASG desired capacity: did the Lambda set it correctly? Check CloudWatch ASG activity log.

5. Check GPU worker logs: any entries? Silence means boot failure. Errors usually indicate routing or model issues.

6. Check IAM: can the Lambda call sqs:GetQueueAttributes? Can the worker read the correct S3 bucket?

7. Check launch template: does the image tag still exist? Pull the AMI manifest.

8. OUTCOMES
   The migration from SageMaker to native AWS primitives, combined with TBIE reconciliation, changed both the cost profile and the operational character of the pipeline.

OPERATIONAL IMPROVEMENTS

• Latency: Startup latency: near-zero or sub-minute (down from 20–40 minutes cold start on SageMaker).

• Parallelism: Workers process images in parallel rather than sequentially — throughput scales with fleet size.

• Throughput: Wall-clock time reduced dramatically from the ~11 hour nightly baseline.

• Replay: Event-driven replay handles failures automatically without manual intervention for most failure classes.

• Cost: Scale-to-zero when no work is present; burst capacity available within seconds via warm pools.

ARCHITECTURAL IMPROVEMENTS

The system can repair its own state: Truth without Intent is detected and corrected by the control plane.
Policy changes are handled as replay events — no custom migration scripts needed.
Tenant onboarding is dynamic — no infrastructure changes required per tenant.
Execution environments are immutable and reproducible via pre-baked AMIs.
Control plane observability now triggers independent alerting paths from worker failures.

THE BROADER LESSON

Not every AI workload needs a managed ML platform. Sometimes the real engineering challenge sits in orchestration, reconciliation, and operational resilience — not in model serving. When the problem is ‘how do we run a model?’, SageMaker is the right answer. When the problem is ‘how do we orchestrate burst-scale inference reliably, replay safely, and cost efficiently?’, native primitives and a clear reconciliation model are the right answer.

GUARDRAILS, MAINTENANCE, AND OPERATIONAL CHECKLIST

Parjanya’s reliability is not a property of its initial design. It is the accumulated result of incidents converted into permanent controls. The following checklist encodes what the system has already learned.

INTENT PRESERVATION

• Workers never delete SQS messages unless the outcome is terminal_failure or completed.

• Visibility timeout exceeds maximum realistic inference time.

• Queue retention exceeds maximum expected outage window.

• Dead-letter queue is monitored and reviewed regularly.

EXECUTION ENVIRONMENT INTEGRITY

Launch template image tags are validated against existing AMIs at deployment time.

AMI build pipelines assert presence of all model artifacts before publishing.

Container version and model artifact version are validated as a compatibility pair.

Launch templates are refreshed automatically when image tags change.

Python lockfiles are used to prevent dependency drift inside worker environments.

CONTROL PLANE HEALTH

Replay Lambda emits a structured heartbeat metric on every successful execution.

Alarm fires if heartbeat is absent for more than one poll cycle.

IAM policy grants for the Lambda are validated in deployment assertions.

Lambda error rate and duration are independently alarmed.

Control-plane CloudWatch logs are retained and searchable for incident forensics.

RECONCILIATION INTEGRITY

Truth backlog (pending DynamoDB rows) is monitored as a primary SLI.
Intent gap (pending Truth with no corresponding queue message) is checked by the Lambda on each invocation.

Replay logic resets Truth cleanly to a pre-execution boundary before re-emitting Intent.

Tenant discovery is dynamic — no manual configuration changes needed on onboarding.

Queue retention policy change requires explicit engineering review.

INFRASTRUCTURE DRIFT PREVENTION

CORS policies are version-controlled and validated in CI.

Bucket CORS and public access settings are configured as a matched pair.

API schema and frontend types have drift detection in CI.

S3 bucket routing is validated at Intent construction time, not only at worker download time.

POLICY AND REGRADE

Content policy version is stored per image in DynamoDB.

Policy changes trigger replay jobs for images scored under previous policy versions.

Regrade jobs use the same replay machinery as failure recovery — no custom tooling.

IMPLICATIONS FOR AI PLATFORM DESIGN

TBIE generalises beyond Parjanya. Any AI platform that spans storage, metadata, asynchronous work, GPU execution, and policy evolution benefits from the same model. The key is to stop thinking of the system as a one-way pipeline and treat it as a reconciliation loop that continuously restores alignment between durable truth and intended work.

That shift changes concrete architecture choices:

Queues: Queues become journals of intent rather than temporary transport.

Replay: Replay becomes a first-class product capability, not an ops workaround.

Autoscaling: Autoscaling becomes a consequence of work existing, not a heuristic guess.

Model packaging: Model packaging becomes part of execution correctness, not just deployment.

Browser uploads: Browser upload behaviour becomes part of the end-to-end reliability model.

Policy change: Policy change becomes a replay event rather than a manual repair task.

The right mental model for modern AI infrastructure is not ‘build a pipeline.’ It is ‘build a system that can continuously reconcile truth and intent until intended work is actually realised.’

APPENDIX A: COMPONENT MAPPING TABLE

APPENDIX B: KEY TERMS

The framework emerged from a simple observation:

Distributed systems rarely fail through catastrophic outages.

More often, they fail through divergence.

Truth says one thing.
Execution does another.
Intent disappears.
Belief becomes detached from reality.

TBIE gave me a vocabulary for understanding those failures.

What I did not fully appreciate at the time was the architectural consequence of taking TBIE seriously.

I thought I was building an AI pipeline.

I ended up building a platform.

Its primary responsibility was preserving correctness.

That realization fundamentally changed how I thought about reliability.

The hard problems in Parjanya v2.0 were not model problems.

They were coordination problems.

• Not GPU problems.

• Not inference problems.

• Not even scaling problems.

The hardest problems emerged whenever the system lost alignment between:

• what was true,

• what work should exist,

• what execution was currently happening,

• and what the system believed was happening.

The original architecture looked deceptively simple.

Images landed in S3.

Metadata was written into DynamoDB.

Validation ran through Lambda functions.

GPU workers performed VLM enrichment and IQA processing.

Results were written back into DynamoDB.

Autoscaling reacted to queue depth.

At first glance, it looked like a fairly conventional event-driven AI pipeline.

Production quickly proved otherwise.

Workers crashed.

Queues drained without progress.

Pending images remained pending forever.

Retries became impossible.

Infrastructure drifted away from declared state.

The system could remain technically healthy while doing absolutely nothing useful.

That was the moment I realised we were solving the wrong problem.

The challenge was not moving images through a pipeline.

The challenge was continuously reconciling distributed state.

Once viewed through that lens, the architecture began evolving in a very different direction.

The replay engine stopped being a utility.

It became the control plane.

Queues stopped being transport mechanisms.

They became durable Intent.

Autoscaling stopped being a scaling feature.

It became a realization layer for Intent.

And the system itself stopped behaving like a pipeline.

It started behaving like a platform.

This article is the story of that evolution.

Preface

Distributed systems rarely fail in the way architecture diagrams suggest.

The boxes stay alive. The queues still exist. The databases continue accepting writes. The autoscaling groups look healthy. The dashboards even remain green.

And yet the system silently stops doing useful work.

An image sits in a pending state forever. A queue drains but no GPU ever runs. A worker crashes and deletes the only replayable work item. A presigned upload URL looks valid but browser execution never begins. A GPU instance launches correctly while the actual container inside it continuously crashloops.

Most production failures are not total outages. They are reconciliation failures.

Something the system believes should happen never actually happens. Or something already completed continues to look pending. Or a control plane silently loses the ability to coordinate execution.

Parjanya v2.0 forced us to confront this repeatedly.

Initially, the architecture looked straightforward:

• Images upload into S3

• Metadata lands in DynamoDB

• Lambdas perform validation and enrichment

• GPU workers run VLM and IQA workloads

• Results are written back into DynamoDB

• Autoscaling follows queue depth

At first glance, this looked like a normal AI pipeline.

In practice, it evolved into something else.

The deeper challenge was not inference. The challenge was maintaining alignment between:

the durable truth of the system,
the operational understanding of the system,
the work the system intended to perform,
and the execution layer actually performing it.

Truth → Belief → Intent → Execution

TBIE was never intended as branding. It emerged as an operational language for reasoning about reliability.

The model eventually shaped:

the event-driven reconciliation architecture,
the autoscaling strategy,
the replay model,
the control plane,
failure recovery,
multi-tenant orchestration,
policy regrading,
infrastructure debugging,
and eventually the broader direction of Parjanya itself.

This article is a deep dive into that journey.

It is not a summary. It is a detailed architectural walkthrough of:

how TBIE emerged,
why event-driven reconciliation mattered,
what failed in production,
how those failures were diagnosed,
and why reconciliation patterns matter far more than most AI infrastructure discussions acknowledge.

The goal is not merely to explain Parjanya.

The goal is to explain how resilient AI systems are actually built.

1. The Real Problem in Distributed AI Systems

Most modern AI infrastructure discussions focus heavily on models.

Which VLM?
Which embedding strategy?
Which vector database?
Which GPU?
Which inference framework?
Which quantization?

These questions matter.

But they are not usually what breaks systems.

The real production problems tend to look different:

Work disappears.
Retries become impossible.
Workers process stale state.
Uploads partially succeed.
Queues drain without progress.
Control planes silently stop coordinating.
Autoscaling behaves correctly against the wrong metric.
Infrastructure layers drift away from declared source-of-truth.

In other words:

The problem is not merely computation. The problem is reconciliation.

Distributed systems are fundamentally coordination systems.

A resilient architecture needs to answer:

What is the durable truth?
What work should exist?
What execution is currently happening?
Can the system reconstruct work after failures?
Can the system distinguish transient failure from terminal failure?
Can infrastructure drift be detected?
Can the system replay work safely?
Can policy changes be re-applied deterministically?

Most architectures answer these implicitly.

TBIE attempts to answer them explicitly.

2. TBIE — Truth, Belief, Intent, Execution

The TBIE model separates distributed systems into four operational layers.

The model becomes powerful because it forces explicit boundaries.

Truth

Truth represents the authoritative record.

Truth must be:

durable,
reconstructable,
queryable,
replayable,
and independent of transient execution state.

In Parjanya:

S3 object bytes are Truth.
DynamoDB workflow fields are Truth.
Pipeline status transitions are Truth.
Curation outputs are Truth.
Hashes and metadata are Truth.

Truth is what remains after workers crash.

Truth is what survives retries.

Truth is what operators use to reconstruct system state.

Belief

Belief is operational understanding.

Belief is not always perfectly accurate.

Examples:

• Queue depth

• Autoscaling desired capacity

• In-flight SQS messages

• Worker health

• Current GPU availability

• Visibility timeout state

Belief may lag reality. Belief may be partial. Belief may even temporarily be wrong.

But Belief matters because orchestration depends on it.

Importantly:

Belief should be derivable from Truth + Intent.

That distinction becomes critical later.

Intent

Intent represents durable declarations of work.

Intent answers:

“What should the system attempt next?”

In Parjanya:

SQS messages are Intent.
Replay queues are Intent.
Regrade operations are Intent.
Retry workflows are Intent.

Intent is one of the most important concepts in the entire architecture.

Execution should never destroy Intent casually.

Why?

Because Intent is replayability.

Without replayable Intent:

transient failures become permanent,
retries become manual,
reconciliation becomes impossible,
and the system loses self-healing capability.

Execution

Execution is the stateless machinery that realizes work.

Execution includes:

Lambdas,
GPU workers,
autoscaling groups,
validation services,
EXIF processors,
and inference pipelines.

Execution should ideally be:

disposable,
reproducible,
restartable,
and stateless.

If an execution worker disappears, the system should still be able to recover from Truth + Intent.

That principle drove many later architectural decisions.

3. Mapping TBIE onto Parjanya v2.0

Once TBIE became the operational lens, the architecture became easier to reason about.

Truth Layer

Truth in Parjanya consisted primarily of:

S3

• original uploads,

• RAW image payloads,

• generated previews,

• derived thumbnails,

• metadata artifacts.

DynamoDB

Workflow state:

• batch_status

• pipeline_status

• curation

• quality scores

• VLM outputs

• EXIF metadata

• technical validation

• image hashes

• routing information

Truth represented the durable state of every image.

Intent Layer

Intent existed primarily in SQS.

Examples:

• ml-large-jobs

• replay queues

• regrade operations

• retry workflows

Each Intent payload encoded:

• bucket,

• object key,

• tenant,

• image id,

• execution metadata.

Intent was intentionally replayable.

That design decision later prevented multiple catastrophic replay-loss scenarios.

Belief Layer

Belief included:

• ASG desired capacity,

• queue visibility counts,

• in-flight message counts,

• CloudWatch metrics,

• worker availability,

• queue backlog,

• GPU scale state.

Belief drove orchestration.

For example:

• queue depth controlled autoscaling,

• replay Lambdas checked queue visibility,

• scaling decisions relied on approximate message counts.

Belief was operationally useful but never fully authoritative.

Execution Layer

Execution included:

Lambda Workers

• EXIF extraction

• technical validation

• replay orchestration

• autoscaling control-plane logic

GPU Workers

• Qwen VLM inference

• IQA processing

• curation scoring

• natural-language enrichment

Execution workers remained stateless wherever possible.

That allowed replay to remain safe.

4. Why Reconciliation Matters

A distributed system does not stay healthy merely because components exist.

Health depends on reconciliation.

Consider this state:

DynamoDB row says pending_vlm_enrichment
SQS queue is empty
GPU ASG is scaled to zero

Nothing is technically “down.”

And yet the system is stuck.

Truth says work should exist. Intent does not exist. Execution never begins. Belief incorrectly suggests the system is idle.

This is a reconciliation failure.

TBIE made this visible.

The system needed a mechanism that continuously aligned:

Truth ↔ Intent ↔ Execution

That eventually led to the event-driven reconciliation architecture.

5. From Polling to Event-Driven Reconciliation

The original architecture used scheduled polling.

Every few minutes:

EventBridge triggered a Lambda,
the Lambda scanned DynamoDB,
pending work was discovered,
SQS messages were emitted,
GPU autoscaling was triggered.

This worked.

But it introduced fundamental limitations.

Problems with Polling

Latency

An image could wait:

0–15 minutes before the next poll cycle.

GPU cold starts already consumed several minutes. Adding polling latency made the UX significantly worse.

Waste

Most polling invocations found no work.

The control plane continuously woke up merely to rediscover emptiness.

Operational Ambiguity

Polling blurred the distinction between:

“there is no work,”
and
“the system has not discovered the work yet.”

That ambiguity complicated debugging.

Poor Reactive Scaling

Autoscaling became tied to polling frequency.

Queue discovery latency directly affected GPU startup latency.

6. DynamoDB Streams Changed the Architecture

The major architectural shift came from moving reconciliation into an event-driven model.

Instead of periodically checking for state changes:

The system reacted immediately when Truth changed.

The New Model

DynamoDB row transitions to pending_vlm_enrichment
DynamoDB Streams emits an event
Event source mapping filters relevant writes
Replay Lambda triggers immediately
SQS Intent is emitted
GPU ASG is kicked from zero
GPU workers process work
Results are written back into Truth

The architecture became reactive.

7. Anatomy of the Event-Driven TBIE

Architecture

DynamoDB Streams

The table emits:

• INSERT events

• MODIFY events

Filtered specifically for:

batch_status = pending_vlm_enrichment

This is extremely important.

The system does not react to every write. It reacts only when Truth crosses a workflow boundary requiring Intent creation.

Event Source Mapping

The stream mapping aggregates:

up to 50 records,
or up to 5 seconds.

This balances:

latency,
throughput,
Lambda cost,
queue pressure.

The system effectively performs micro-batching naturally.

Replay Lambda

The replay Lambda became the core reconciliation engine.

Responsibilities:

• inspect stream events,

• discover pending rows,

• identify retriable failures,

• generate replayable Intent,

• avoid duplicate queueing,

• trigger autoscaling.

This Lambda evolved into the operational heart of the platform.

Queue Backlog Guard

One subtle but extremely important design decision:

The replay Lambda checks queue visibility before re-enqueueing pending work.

Why?

Because replay itself can become destructive if uncontrolled.

Without a backlog guard:

the control plane could continuously duplicate Intent,
queue pressure would inflate,
workers would process stale state,
autoscaling would overreact.

This is a classic reconciliation trap.

The backlog guard prevented it.

GPU ASG Kick

The replay Lambda also became responsible for scaling GPU workers.

This matters because scale-to-zero architectures create a hidden problem:

The queue may contain work while compute remains asleep.

The replay Lambda solved this by:

emitting Intent,
then immediately setting desired ASG capacity.

That turned the replay engine into both:

a reconciliation layer,
and a control-plane orchestrator.

8. Latency Transformation

The shift from polling to event-driven reconciliation fundamentally changed responsiveness.

Before

Upload → wait for poll → queue → ASG scale → GPU boot

Total latency:

~3–20 minutes

depending on polling window timing.

After

Upload → stream event → immediate reconciliation → queue → ASG scale

Total latency:

~4–6 minutes

with remaining delay dominated almost entirely by:

EC2 startup,
model load,
GPU warm-up.

This is a very different operational profile.

The system stopped feeling batch-oriented.

It began behaving like a reactive platform.

9. The First Major TBIE Failure

Intent Destroyed Too Early

One of the earliest major incidents exposed why TBIE mattered.

Symptom

GPU workers experienced transient:

• S3 download failures,

• temporary consistency races,

• OOM conditions.

But the worker still deleted the SQS message.

At the same time:

DynamoDB was updated with:

vlm_failed

The result:

Intent disappeared,
Truth claimed terminal failure,
automatic replay became impossible.

This was a severe reconciliation violation.

The Core TBIE Violation

Execution destroyed Intent before retries were exhausted.

That single decision broke replayability.

TBIE clarified the failure immediately.

The problem was not merely worker instability.

The problem was:

Execution invalidated Intent prematurely.

The Fix

The worker execution model changed.

Instead of binary success/failure, workers returned:

retry

• do NOT delete SQS message

• do NOT write terminal Truth

terminal_failure

• write durable failure state

• delete Intent

completed

• write success state

• delete Intent

This restored replay semantics.

Why This Matters Beyond Parjanya

Many distributed systems accidentally conflate:

“execution failed once” with
“execution is permanently impossible.”

Those are not equivalent.

TBIE forced the architecture to encode that distinction explicitly.

10. Truth Pending, Intent Missing

Another class of failures appeared repeatedly.

Symptom

Rows remained:

pending_vlm_enrichment

But:

• queues were empty,

• GPU workers never started,

• no replay existed.

The system appeared healthy.

But nothing moved.

What TBIE Revealed

Truth and Intent were decoupled.

The architecture had assumed:

“If Truth says pending, Intent must already exist.”

That assumption was false.

Reconciliation cannot be assumed. It must be continuously enforced.

The Replay Architecture

This led to:

• replay scripts,

• scheduled replay Lambdas,

• retriable failure reset workflows,

• queue reconstruction logic.

The replay Lambda evolved into a continuously running reconciler.

This was the moment the architecture stopped being merely a pipeline.

It became a platform.

11. Wrong Bucket, Right Key

A Perfect Distributed Systems Failure

One of the most revealing incidents involved bucket mismatch.

Symptom

GPU workers repeatedly failed with:

s3_download_error

Yet:

• queue depth was growing,

• autoscaling worked,

• workers were alive,

• previews existed.

Everything looked operational.

Root Cause

The SQS payload contained:

the correct object key,
but the wrong bucket.

The preview existed.

Just not in the referenced bucket.

Why This Was Architecturally Important

The system was reliably executing the wrong instruction.

This is one of the hardest classes of distributed failures.

Nothing is technically broken.

The system is functioning exactly as instructed.

The instruction itself is invalid.

TBIE framed this correctly:

Intent must not merely exist. Intent must be correct and replayable.

The Fix

The architecture introduced:

• bucket source-of-truth rules,

• preview-aware routing,

• enqueue eligibility tightening,

• RAW-aware preview validation,

• replay resets for retriable failures.

The fix was operationally deeper than changing one field.

It hardened the meaning of Intent itself.

12. CORS Failures and Split Truth

One of the most subtle incidents occurred at the ingestion boundary.

Symptom

Frontend uploads failed with browser CORS errors.

But backend APIs:

• generated presigned URLs,

• wrote pending upload records,

• and returned success responses.

Truth looked correct.

Execution never happened.

What Actually Failed

The browser failed during:

OPTIONS preflight.

Execution never reached:

PUT upload.

The TBIE Interpretation

Truth existed. Intent existed. Execution failed before realization.

This became an important diagnostic heuristic:

When work is stuck:

Ask whether:

• Truth is missing,

• Intent is missing,

• or Execution cannot realize Intent.

That framing simplified debugging enormously.

13. Event-Driven Architecture Exposes Infrastructure Drift Faster

As the architecture became more reactive, infrastructure drift became easier to detect.

This was unexpected.

Example: Stale Worker Image Tag

The ASG launch template still referenced:

:v20

while the actual system had moved to:

:v23

The ASG launched correctly. Cloud-init executed. Docker attempted to pull. The image no longer existed.

Workers never started.

The Deeper Problem

Terraform ignored user_data changes.

This created silent divergence between:

declared source-of-truth,
and actual runtime execution.

TBIE exposed this as an Execution drift problem.

Truth, Belief, and Intent were all correct.

Execution itself had drifted away from reality.

The Architectural Lesson

Infrastructure reproducibility is not optional.

Execution environments must remain tied to durable source-of-truth.

Otherwise replay becomes dangerous because:

the same Intent may produce different outcomes over time.

14. AMI Drift and Runtime Incompatibility

Another failure emerged at the AMI layer.

Symptom

Containers started. But model loading crashed.

The model weights existed. The processors did not.

The AMI bake omitted Python files required by:

trust_remote_code=True

The worker entered a crashloop.

Why This Was Important

Again:

Truth was correct. Intent was correct. Belief suggested workers existed.

Execution failed due to stale artifacts.

This is precisely why replayable systems require reproducible execution environments.

TBIE repeatedly reinforced this architectural truth.

15. The Replay Lambda Became the Control Plane

Originally, the replay Lambda was operational glue.

Eventually, it became the platform control plane.

Responsibilities expanded to include:

• pending reconciliation,

• retriable failure resets,

• tenant discovery,

• queue backlog management,

• autoscaling coordination,

• replay orchestration,

• dynamic routing.

The replay layer became central infrastructure.

This was a critical architectural evolution.

16. The IAM Incident

When the Control Plane Goes Blind

One of the most dangerous incidents involved a missing IAM permission.

Symptom

The replay Lambda continuously crashed on:

sqs:GetQueueAttributes

for nearly 12 hours.

But:

• EventBridge still invoked it,

• the schedule looked healthy,

• the infrastructure appeared alive.

Meanwhile:

• pending rows accumulated,

• queues stopped reconciling,

• autoscaling logic failed.

The TBIE Lesson

The control plane had lost Belief.

Without Belief:

the replay engine could not safely emit Intent,
queue guards could not function,
orchestration collapsed.

The system looked operational while the reconciler was effectively dead.

The Operational Insight

A scheduled Lambda is not necessarily a healthy Lambda.

Control-plane observability matters more than worker observability.

Why?

Because worker failures are recoverable.

Control-plane blindness disables recovery itself.

17. Dynamic Tenant Discovery

As Parjanya moved toward beta onboarding, static tenant configuration became a bottleneck.

Originally:

The replay Lambda used static tenant lists.

Every new tenant required:

• Terraform changes,

• environment updates,

• redeploys.

This violated the self-service model.

The Evolution

The architecture introduced:

• tenant registry writes during upload completion,

• runtime tenant discovery,

• dynamic reconciliation.

The control plane became data-driven rather than configuration-driven.

This is a major architectural distinction.

Reactive systems scale operationally only when discovery becomes dynamic.

18. Regrading and Policy Replay

One of the most powerful consequences of TBIE was deterministic replay.

This became especially valuable during policy changes.

Example

The IQA policy evolved to reject:

• screenshots,

• banners,

• posters,

• slides,

• social cards,

• infographics,

• illustrations.

The architecture needed to:

• reset VLM-derived Truth,

• regenerate Intent,

• rerun GPU enrichment,

• rewrite curation outputs.

This was not merely a retry.

It was a deterministic policy replay.

Why This Matters

Most ML systems struggle with:

“How do we reprocess historical data safely?”

TBIE already contained the answer.

Truth could be reset to a replay boundary. Intent could be regenerated. Execution remained stateless.

The system became replay-native.

This is a profound operational capability.

19. Queue Retention and Weekend Gaps

Another subtle operational issue:

Scale-to-zero architectures create temporal gaps.

Scenario

Users upload images Friday evening. GPU workers remain offline over the weekend.

If queue retention is too short:

Intent disappears before execution resumes.

Truth remains pending.

Intent vanishes.

This is catastrophic reconciliation drift.

The Fix

Queue retention was extended.

This ensured Intent outlived temporary execution dormancy.

An important TBIE lesson emerged:

Intent durability must exceed execution availability windows.

20. Why Event-Driven Reconciliation Matters

Beyond AI

Although Parjanya is an AI platform, the architectural lessons generalize broadly.

The same TBIE principles apply to:

distributed media systems,
payment processing,
ETL pipelines,
logistics systems,
asynchronous workflows,
microservice orchestration,
large-scale ingestion platforms.

The core challenge is universal:

How do you maintain alignment between:

durable truth,
intended work,
operational state,
and actual execution?

TBIE provides a vocabulary for reasoning about that alignment.

21. The Shift from Pipeline to Platform

This was perhaps the most important realization.

A pipeline moves data.

A platform:

• survives retries,

• supports replay,

• reconciles state continuously,

• tolerates execution drift,

• evolves policy safely,

• supports operational introspection,

• and remains debuggable under failure.

The event-driven TBIE architecture pushed Parjanya decisively toward the latter.

The replay Lambda stopped being a utility.

It became the reconciliation brain of the system.

The architecture stopped being:

“process uploaded images.”

It became:

“maintain durable correctness while asynchronous distributed execution continuously changes.”

That is a very different engineering problem.

22. The Operator’s Mental Model

TBIE becomes most useful when it changes how incidents are diagnosed.

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. Operators no longer began by asking whether “the queue is healthy” or whether “the GPU workers are running.” Those signals still mattered, but they were no longer treated as authoritative.

Instead, every investigation began with a much more precise question:

Is the system waiting because Intent is missing, or is Intent present but Execution cannot realize it?

That distinction dramatically narrowed the search space during incidents.

If Truth showed a growing backlog while Intent remained near zero, the issue was usually in reconciliation itself. Replay Lambda failure, queue emission failure, stale tenant discovery, or a blind control plane could all create this pattern. The queue might look empty and the autoscaling group might remain at zero, yet the real failure would not be in the workers. It would be in the system’s ability to regenerate Intent from Truth.

If Intent existed but Execution failed repeatedly, the investigation shifted toward realization problems instead: bucket routing mismatches, stale AMIs, launch-template drift, IAM gaps, runtime incompatibilities, model-loading failures, endpoint routing problems, or browser-side execution failures.

That diagnostic framing changed operations significantly. Incidents became easier to classify because TBIE separated the architecture into meaningful operational boundaries. Instead of treating the platform as one opaque pipeline, operators could isolate whether the failure belonged to Truth, Belief, Intent, or Execution.

The model also made one uncomfortable reality obvious: dashboards alone are not enough. A queue can look healthy while Truth continues accumulating backlog. Autoscaling can look correct while workers continuously crashloop. A scheduled Lambda can appear operational while silently failing every invocation because of an IAM permission gap.

TBIE did not eliminate debugging complexity. What it did was make debugging systematic.

23. When the System Hangs at 02:00 AM

Architectural diagrams are useful during design reviews.

Runbooks matter during incidents.

One of the most important evolutions in Parjanya v2.0 was realizing that reconciliation needed to exist not only in architecture, but also in operational practice. TBIE only became truly useful once there was a repeatable process for recovering the system under pressure.

The first step during any “hung VLM” investigation became identifying the TBIE boundary itself.

Truth consisted of the DynamoDB image rows and their workflow state. Intent consisted of SQS messages representing GPU work. Execution consisted of GPU workers consuming that Intent and writing enriched Truth back into DynamoDB. Belief consisted of queue depth, autoscaling state, visibility counts, worker health, and operational telemetry.

Once those boundaries were established, incident triage became much more structured.

Operators first checked Truth backlog. Were rows accumulating in pending_vlm_enrichment? If yes, the system still believed work should happen.

Next came Intent backlog. Did SQS contain replayable work items? If Truth backlog was high while Intent remained near zero, then reconciliation itself had failed.

Then came Execution health. Were GPU workers alive? Were containers actually starting? Were model loads succeeding? Were workers consuming messages or continuously failing before acknowledgement?

Several operational shortcuts emerged over time.

If the GPU log group was empty while ASG instances existed, the problem was often below the application layer entirely. Cloud-init failures, stale image tags, or broken launch templates frequently surfaced first through EC2 console output rather than through CloudWatch worker logs.

If the replay Lambda appeared scheduled but no queue activity existed, IAM permissions became the first thing to inspect. One of the most dangerous incidents in the system occurred precisely because the reconciliation Lambda could no longer read queue attributes, effectively blinding the control plane while the infrastructure superficially appeared healthy.

If rows repeatedly failed with S3 download errors while workers remained alive, the issue was often deterministic addressing rather than transient instability. Wrong-bucket references, preview routing mismatches, or endpoint configuration drift could all create workers that failed consistently while still appearing operational.

Over time, this changed the operational mindset of the platform. The goal stopped being “restart the workers and hope.” The goal became understanding exactly where reconciliation had broken down and then repairing the appropriate boundary.

That distinction is subtle, but extremely important.

A resilient platform is not the one that avoids incidents. It is the one that can recover from them methodically.

24. Guardrails and Proactive Controls

One of the clearest architectural patterns that emerged from Parjanya was that every major incident eventually needed to produce a permanent guardrail.

Without that discipline, incidents remain stories. With it, incidents become operational evolution.

The architecture gradually accumulated a large number of proactive controls:

Transient worker failures no longer delete Intent prematurely. Replay Lambdas continuously reconcile pending Truth. Reproducible lockfiles prevent dependency drift. Terraform validations reject invalid wildcard CORS patterns. Tenant discovery became dynamic rather than operator-maintained. Queue retention was extended to match real outage windows rather than idealized assumptions.

Many of these controls were not introduced because the architecture diagram required them. They emerged because production failures revealed where reconciliation could silently drift.

This is one of the strongest long-term lessons from the TBIE model.

Resilience is not achieved through one large design decision. It is accumulated gradually through operational guardrails.

The mature version of the system looked very different from the original one not because the core architecture changed dramatically, but because every important failure eventually became:

a validation rule,
a replay strategy,
a Terraform constraint,
a CI assertion,
a queue safeguard,
or an operational diagnostic.

That is how resilient systems evolve in practice.

They convert incidents into institutional memory.

25. Maintenance as a Reliability Discipline

One of the easiest mistakes in distributed systems is assuming resilience is something implemented once.

In reality, resilience is maintained continuously.

By the later stages of Parjanya v2.0, the team had accumulated a growing operational checklist precisely because many failures were caused not by dramatic outages, but by small forms of drift.

Replay logic and worker behavior needed to evolve together. GPU image tags needed to stay aligned with launch templates. Lockfiles had to remain committed when dependencies changed. Tenant buckets required CORS validation before onboarding. Queue retention needed to reflect actual business downtime windows. Backend API contracts and frontend types had to remain synchronized.

None of these tasks sound individually sophisticated.

But distributed systems rarely collapse from a single catastrophic event. They erode through gradual inconsistency.

That realization changed how the platform was maintained.

Operational discipline became part of the architecture itself.

The maintenance checklist was therefore not merely documentation. It was a recognition that reconciliation must continue throughout the entire lifecycle of the platform.

26. Why TBIE Matters Beyond AI

Although Parjanya is an AI platform, the architectural lessons generalize broadly.

The same TBIE principles apply to:

distributed media systems,
payment processing,
ETL pipelines,
logistics systems,
asynchronous workflows,
microservice orchestration,
large-scale ingestion platforms.

The core challenge is universal:

How do you maintain alignment between:

durable truth,
intended work,
operational state,
and actual execution?

TBIE provides a vocabulary for reasoning about that alignment.

27. The Shift from Pipeline to Platform

1. Truth must remain durable and replayable

If Truth cannot reconstruct the system after failures, the architecture becomes operationally fragile.

2. Intent is more important than most systems realize

Intent is replayability.

Destroying Intent prematurely destroys recovery.

3. Event-driven systems expose hidden drift faster

Reactive architectures surface infrastructure inconsistencies earlier.

This is painful initially.

But healthier long-term.

4. Execution must remain reproducible

Replay without reproducibility becomes dangerous.

5. Control-plane observability matters more than worker observability

Worker crashes are recoverable. Blind reconcilers are not.

6. Queue depth alone is not truth

Belief metrics require interpretation.

Approximate counts are operational signals, not durable truth.

7. Policy replay should be designed intentionally

Eventually every ML platform needs deterministic reprocessing.

Replay-native architectures handle this gracefully.

8. Distributed systems fail by divergence

Not by total outage.

The hardest bugs were:

wrong bucket,
stale AMI,
hidden CORS mismatch,
invisible IAM gap,
frozen replay,
missing Intent.

TBIE helped make those failures explainable.

28. The Future Direction

The current architecture already points toward several future evolutions:

• multi-stage event-driven pipelines,

• adaptive batching,

• warm GPU pools,

• richer metrics export,

• dynamic replay orchestration,

• per-stage autoscaling,

• event-sourced workflow history,

• tenant-aware media delivery,

• replayable policy evolution.

Importantly:

These evolutions do not require rewriting the architecture.

They emerge naturally because reconciliation is already foundational.

That is one of the strongest signs that the architecture direction is correct.

Closing Thoughts

TBIE did not begin as an attempt to create a framework.

It emerged because distributed AI infrastructure repeatedly exposed the same class of operational failures:

work disappearing,
retries becoming impossible,
control planes drifting,
execution environments diverging,
queues losing meaning,
infrastructure behaving correctly against stale assumptions.

The model became useful because it created clarity.

Truth. Belief. Intent. Execution.

Once those boundaries became explicit:

replay became easier,
debugging became faster,
autoscaling became safer,
policy evolution became manageable,
and infrastructure failures became explainable.

The most important lesson from Parjanya v2.0 is not that event-driven architectures are faster.

The deeper lesson is:

Resilient distributed systems are fundamentally reconciliation systems.

The system that survives is not necessarily the one with the fastest workers.

It is the one that can continuously reconstruct:

what is true,
what work should exist,
what execution already happened,
and what still needs to be reconciled.

That is what ultimately transformed Parjanya from a pipeline into a platform.

In May 2026, I found myself staring at a familiar engineering problem.

Parjanya’s Image Quality Assessment pipeline was approaching the limits of what our infrastructure could comfortably handle.

Inference workloads were growing.

Batch sizes were increasing.

Memory utilization was climbing.

And eventually, the system hit a wall.

A larger image.

A slightly bigger batch.

A little more context.

And then:

CUDA Out of Memory.

The obvious answer was straightforward.

Upgrade the hardware.

Move to larger GPUs.

Increase the budget.

Scale vertically.

That’s what I almost did.

Instead, I opened the prompt.

And that decision ended up saving far more money than a hardware upgrade would have.

The Assumption I Didn’t Question

By this point in the Parjanya v2.0 journey, I had spent months thinking about infrastructure.

The previous two posts in this series covered:

• Multi-account AWS architecture

• Terraform and operational discipline

• CI/CD pipelines

• IAM boundaries

• Infrastructure scaling

So when inference started becoming expensive, my instinct was predictable.

I looked at the hardware.

That’s what engineers do.

We assume the bottleneck lives where the graphs are red.

GPU memory usage was high.

Throughput was flattening.

Batch sizes were constrained.

The diagnosis seemed obvious.

But infrastructure metrics were only showing symptoms.

They weren’t showing the cause.

The Audit That Changed Everything

Instead of pricing larger GPUs, I decided to audit the prompt itself.

At that point our VLM-based IQA system had evolved through multiple versions.

Each release added new capabilities:

• Better composition evaluation

• Improved wildlife classification

• More robust defect detection

• Additional structured outputs

• More nuanced confidence handling

Every version added knowledge.

Almost nothing removed knowledge.

The prompt had become a living document.

Like most living documents, it accumulated debt.

When I finally measured it, the numbers surprised me.

The system prompt plus user context had grown to roughly:

4,150 tokens

Not because we needed 4,150 tokens.

Because nobody had challenged whether we still needed them.

What I Found

The easiest way to describe it is this:

The prompt looked less like a carefully engineered system and more like a codebase that hadn’t been refactored in years.

There was useful logic.

There was historical baggage.

And there was a lot of dead code.

Unused Outputs

One section calculated an overall quality score.

The model spent tokens reasoning about it.

The downstream system ignored it completely.

The actual acceptance and rejection decisions came from deterministic rules elsewhere in the pipeline.

The score existed because an earlier version needed it.

The current version didn’t.

Yet the prompt still carried the instructions.

Features That Never Shipped

I found sections generating:

• search tags

• mood descriptions

• discovery metadata

These outputs were originally intended for future content discovery features.

Those features never launched.

The instructions remained.

Every inference paid the cost.

Every single time.

Teaching the Model Things It Already Knew

This was the biggest surprise.

Over time we’d added examples.

Then more examples.

Then explanations for the examples.

Then explanations explaining the explanations.

The prompt contained long sections describing:

• composition techniques

• confidence scoring

• framing recognition

• classification behavior

At one point those explanations were useful.

But the model wasn’t learning from scratch anymore.

The explanatory scaffolding had become informational overhead.

The rules mattered.

The essays explaining the rules did not.

I Started Thinking About It Like Technical Debt

The moment everything clicked was when I stopped calling it prompt engineering.

This wasn’t prompt engineering.

This was refactoring.

The exact same process we apply to software.

In software we ask:

• Is this function still used?

• Does this dependency still matter?

• Is this abstraction providing value?

• Can this code be simplified?

I realized prompts deserve the same scrutiny.

A prompt is infrastructure.

It consumes resources.

It affects latency.

It affects throughput.

It affects cost.

And like any infrastructure, it accumulates debt.

I started calling this:

Prompt Debt.

What Stayed

This is the part that surprised me most.

Almost all of the intelligence survived.

The capabilities remained intact.

The model still knew how to:

• recognize deliberate composition

• distinguish artistic choices from technical defects

• apply confidence-aware wildlife classification

• detect focus issues

• identify exposure problems

• generate structured outputs

• evaluate image quality

We didn’t remove intelligence.

We removed repetition.

We removed explanation.

We removed historical residue.

The distinction matters.

The Numbers

After the audit:

System Prompt

~700 tokens → ~360 tokens

User Prompt

~3,450 tokens → ~890 tokens

Total Context

~4,150 tokens → ~1,250 tokens

Approximately:

70% reduction

without changing:

• the model

• the hardware

• the infrastructure

• the deployment architecture

Only the prompt changed.

Why 70% Matters

It’s tempting to see this as an academic optimization.

It wasn’t.

Those tokens had real operational consequences.

Every unnecessary token consumes:

• memory

• compute

• latency budget

• throughput capacity

On paper, a token looks small.

At scale, thousands of unnecessary tokens become infrastructure.

And infrastructure costs money.

Before the cleanup, our batching limits were constrained by context size.

After the cleanup, we could process roughly twice as many images within similar memory budgets.

The result was effectively:

2x throughput without changing hardware.

The kind of gain most teams expect from an expensive infrastructure project.

We got it from deleting text.

Every Token Has a Cost

This became the biggest lesson of the entire exercise.

Engineers often think about infrastructure costs.

Few think about information costs.

Every token carries a burden.

A token occupies memory.

A token increases inference time.

A token competes for attention.

A token introduces complexity.

The question isn’t:

“Can the model read this?”

The question is:

“Should the model be reading this at all?”

Those are very different questions.

This was a practical reminder of a lesson I explored earlier in Grounding Is Not a Prompt. Models don’t become more faithful simply because we provide more instructions. What matters is the quality and relevance of information available at inference time. A smaller, denser prompt often outperforms a larger one filled with historical baggage and explanatory noise.

The Broader Pattern

The more I reflected on the exercise, the more it felt familiar.

This pattern appears everywhere.

Organizations accumulate meetings.

Software accumulates dependencies.

Infrastructure accumulates services.

Prompts accumulate instructions.

Over time, complexity grows through addition.

Rarely through subtraction.

Most systems become slower because nobody asks what can be removed.

Prompts are no different.

While reviewing the prompt, I kept thinking about a principle we previously discussed in Databases, Caches and Queues: A Universal Rule for Distributed Systems. Systems become resilient when information has clear ownership and purpose. The same principle applies to prompts. Every instruction should exist for a reason. If nobody consumes it, it becomes operational debt.

The Connection to Inferencing

This experience fundamentally changed how I think about inference optimization.

Most conversations focus on:

• larger models

• larger context windows

• larger GPUs

• more sophisticated hardware

Those matter.

But they’re not always the first lever.

Sometimes the most effective optimization isn’t computational.

It’s informational.

Before changing the model, review the prompt.

Before upgrading the GPU, review the prompt.

Before increasing context windows, review the prompt.

The cheapest token is the one you never generate.

What Comes Next

Prompt reduction was only one layer.

The next frontier involves serving optimization:

• prompt prefix caching

• KV cache reuse

• vLLM batching improvements

• quantization strategies

• memory-efficient inference

Those areas still offer meaningful gains.

But they sit on top of a more fundamental principle.

Optimize information before optimizing infrastructure.

The Takeaway

When the system first ran out of memory, I assumed we had a hardware problem.

What we actually had was an information problem.

The model wasn’t struggling because it lacked capacity.

It was struggling because we were making it carry unnecessary baggage.

That distinction changed how I think about AI systems.

We often assume progress requires adding something.

A larger model.

A larger GPU.

A larger context window.

A larger budget.

In this case, the breakthrough came from subtraction.

The model already knew enough.

The system simply needed less noise.

And that’s a lesson that extends far beyond prompts.

It’s a lesson about engineering itself.

This concludes the initial Parjanya v2.0 engineering series. Across these three posts—from AWS account boundaries, to Terraform discipline, to prompt debt—the common theme has been surprisingly consistent: systems scale best when boundaries are clear, assumptions are explicit, and unnecessary complexity is removed. The challenge is rarely adding more. The challenge is knowing what to keep.

Further Reading

If this article resonated with you, these related essays expand on the ideas discussed here:

• Grounding Is Not a Prompt — Why better information retrieval often matters more than adding more instructions to a model.

• VLM Batch Inference - 4 bit quantisation depth — The Hidden Cost of “Almost Fitting”

• Context Engineering and Context Debt — a data investigation that revealed ~77% of my token spend was context accumulation waste, not productive reasoning. This is the framework I built to fix it — and the problem has a name: Context Debt and the fix is: Context Engineering .

• Databases, Caches and Queues: A Universal Rule for Distributed Systems — The Truth, Belief and Intent framework that inspired how we think about information flow across systems.

• TBIE in Practice: Designing Resilient AI Pipelines That Recover, Reconcile, and Re-run - Reconciliation loop and TBIE model with case studies with Parjanya v2.0

• Terraform at Scale: What Actually Worked, What Didn’t, and the Cost of Trusting AI with Infrastructure — The previous post in this series, exploring operational discipline and infrastructure boundaries.

• Why We Use Three AWS Accounts (Not Two) for Parjanya v2.0 — The architectural foundation that enabled the optimizations described here.

Introducing the Inferencing Series

This article marks the beginning of a new series exploring inferencing as a first-class engineering discipline. Much of the AI conversation today revolves around models, benchmarks, and training. Yet in production systems, the economics and behaviour of AI are often determined long after training is complete—during inference. The choices we make around prompts, context, quantization, batching, memory management, caching, retrieval, and serving architecture frequently have a greater impact on cost, latency, and reliability than changing the model itself.

Throughout this series, I’ll share lessons learned while building and operating Parjanya’s Visual Language Model (VLM) pipelines. Rather than focusing on LLM chat applications, the emphasis will be on real-world image understanding systems: image quality assessment, wildlife classification, structured extraction, multimodal reasoning, and the engineering decisions required to run them efficiently at scale. The goal is not simply to reduce costs, but to understand how information flows through an AI system and where meaningful optimization actually lives.

Why Start With Prompt Debt?

Prompt Debt is an ideal starting point because it reveals a common misconception about AI systems: that performance improvements come primarily from adding more. More instructions. More examples. More context. More hardware. In practice, many mature VLM systems accumulate informational debt in the same way software accumulates technical debt—unused outputs, redundant instructions, abandoned features, and historical assumptions that continue consuming resources long after they stop creating value.

Future articles in the Inferencing Series will build on this foundation by exploring quantization, context engineering, grounding strategies, serving architectures, prompt caching, KV-cache optimization, batching, multimodal pipelines, and the operational realities of running VLM workloads in production. Prompt Debt is simply the first lens. The broader question—and the one that interests me most—is how to maximize signal while minimizing computational waste across the entire inferencing stack.

When people talk about Terraform, they often talk about Infrastructure as Code.

What I learned while building Parjanya v2.0 is that Terraform is really about something else:

forcing operational discipline.

Terraform didn’t magically simplify our infrastructure. It exposed assumptions, surfaced hidden dependencies, and forced us to formalize decisions we had previously gotten away with making informally.

As Parjanya evolved from a collection of scripts, console clicks, and one-off configurations into a multi-tenant image processing platform, Terraform became the backbone that allowed us to scale infrastructure without losing control.

It also introduced new challenges.

And when I layered AI-assisted engineering into the workflow, I discovered an entirely different category of problems worth documenting.

This post is the implementation companion to my previous article on why we chose a three-account AWS architecture for Parjanya v2.0.

That post explained the boundaries.

This one explains how we turned those boundaries into code.

The Reality of Operating Across Five Repositories

Parjanya today spans five repositories.

Each exists for a reason.

• parjanya-ml handles inference and model serving.

• parjanya-ops owns infrastructure and platform operations.

• parjanya-backend manages orchestration and APIs.

• parjanya-frontend serves tenant-facing experiences.

• parjanya-shared-libraries contains contracts and shared abstractions.

The separation works well organizationally.

ML engineers don’t need to understand IAM condition keys.

Infrastructure engineers don’t need to navigate PyTorch dependencies.

Backend engineers don’t need to modify Terraform every time they ship a feature.

But Terraform exposed a challenge that few people talk about:

cross-repository infrastructure dependencies.

The moment one repository depends on infrastructure outputs defined in another, coordination becomes critical.

A renamed Terraform output in one repository can quietly break another repository that references it through remote state.

I’ve been bitten by this more than once.

Terraform didn’t prevent those mistakes.

What it did provide was visibility.

And visibility is often the difference between a recoverable issue and a production incident.

What Terraform Actually Improved

A lot of Terraform discussions stay theoretical.

Here’s what it changed in practice for us.

ECS and Warm Pools

One of the first problems we tackled was cold-start latency.

Inference workloads running on ECS-backed EC2 instances would occasionally wait several minutes for capacity to become available.

To solve this, we implemented Auto Scaling Group warm pools.

The result wasn’t just faster startup times.

It gave us a repeatable way to define capacity behavior for different tenant tiers.

Instead of manually tuning infrastructure across environments, we could express those decisions through variables and deploy them consistently.

That may sound mundane.

But consistency becomes incredibly valuable once multiple environments, tenants, and deployment pipelines are involved.

Multi-Tenant S3 Isolation

Parjanya is a multi-tenant platform.

Tenant isolation isn’t optional.

Terraform allowed us to codify S3 prefix-level isolation directly into IAM policies.

The most important outcome wasn’t the policy itself.

It was the reviewability.

A missing condition that could have exposed tenant object metadata became visible during code review instead of after deployment.

That’s one of Terraform’s underrated strengths:

It turns invisible infrastructure assumptions into visible diffs.

DynamoDB Tenant Boundaries

The same pattern applied to DynamoDB.

Every tenant’s data is partitioned using tenant identifiers, with IAM enforcing access boundaries.

Could application code enforce those rules?

Yes.

Should application code be the only layer enforcing them?

Absolutely not.

Terraform helped us move those guarantees into infrastructure itself.

Even if application logic fails, IAM remains the final gatekeeper.

Event-Driven Infrastructure

Parjanya’s processing pipeline relies heavily on EventBridge and SQS.

Before Terraform, environment drift was a recurring problem.

A queue configuration changed in Production but not Development.

An EventBridge rule existed in one environment but not another.

Over time these differences accumulate.

Terraform gave us repeatability.

The same modules, deployed consistently, dramatically reduced configuration divergence.

The Three-Account Migration

The biggest Terraform project in Parjanya v2.0 wasn’t a service deployment.

It was infrastructure reorganization.

We migrated from a single AWS account into:

• Management / Tooling

• Development

• Production

Terraform made that migration possible.

It did not make it easy.

Making State Account-Aware

Our original infrastructure assumed everything lived in one account.

Provider configurations were implicit.

Cross-account roles didn’t exist.

Account boundaries didn’t exist.

The first step was making account ownership explicit.

That sounds simple until you realize every Terraform resource suddenly needs to know which AWS account it belongs to.

The work wasn’t intellectually difficult.

It was painstaking.

And this is where I learned an important lesson about AI-assisted engineering:

AI is surprisingly bad at repetitive infrastructure migrations unless it has complete context.

Migrating State Safely

One of the most valuable Terraform commands I encountered during this process was:

terraform state mv

State migration sounds boring.

Until you realize a single mistake can cause Terraform to believe a production resource should be destroyed and recreated.

The migration process became an exercise in discipline:

Move state.

Run a plan.

Review carefully.

Repeat.

Slowly.

The temptation to batch changes is enormous.

Resisting that temptation saved us multiple times.

Cross-Account IAM

The three-account architecture introduced a new category of complexity.

Trust relationships.

The Management account hosts shared tooling.

Development and Production need access to those resources.

Terraform became the mechanism for expressing those trust boundaries consistently.

What surprised me wasn’t the complexity of the IAM policies.

It was how much architectural thinking is required before writing any Terraform.

The tooling faithfully implements your design.

It does not validate whether the design itself is sensible.

Centralized State Management

One decision I remain happy with is keeping Terraform state centralized within the Management account.

Development engineers can operate Development.

Production operators can manage Production.

But neither automatically gains access to every state file.

This separation aligns with the overall philosophy behind the architecture:

Boundaries first.

Convenience second.

Where AI Helped — And Where It Didn’t

Throughout this migration I relied heavily on AI-assisted development.

Claude became part of the workflow.

Over time I learned that different models excel at very different tasks.

The results were revealing.

Haiku: Fast But Dangerous for Infrastructure

Haiku was excellent at generating boilerplate.

Variable files.

Module outputs.

Simple scaffolding.

But infrastructure isn’t just syntax.

Infrastructure is reasoning.

I repeatedly found that Haiku could generate valid Terraform that contained subtle security flaws.

The code looked correct.

The logic wasn’t.

That distinction matters.

A lot.

My conclusion became simple:

Haiku should never be trusted with security-sensitive infrastructure decisions.

Sonnet: The Daily Driver

Most of our Terraform work happened with Sonnet.

For implementation tasks inside a repository, it performed extremely well.

Module structures.

Provider configurations.

Refactors.

General Terraform development.

Where it struggled was context.

A repository rename might affect multiple repositories.

A queue change might impact systems it couldn’t see.

The problem wasn’t model intelligence.

The problem was visibility.

The model can only reason about what you show it.

Opus: Better at Architecture Than Implementation

I brought Opus into the process whenever decisions became architectural.

Questions like:

• Should this responsibility live in Management or Production?

• How should artifacts flow between accounts?

• What’s the cleanest trust model?

These are trade-off questions.

Opus consistently handled them better than implementation-focused tasks.

The irony was that the more strategic the question became, the more useful the model became.

The Biggest Problem Wasn’t Terraform

It Was Context.

This became one of the clearest lessons from Parjanya v2.0.

The hardest infrastructure problems weren’t provider bugs.

They weren’t Terraform limitations.

They weren’t AWS limitations.

They were context problems.

Infrastructure lived in one repository.

ML workloads lived in another.

Shared interfaces lived somewhere else.

Every repository understood part of the system.

No repository understood the entire system.

That’s why we started building supporting tooling and internal agents focused on context discovery and verification.

The challenge wasn’t generating code.

The challenge was understanding the system before generating code.

What Terraform Didn’t Solve

It’s important to be honest about this.

Terraform isn’t a cure-all.

We still accumulated module sprawl.

We still dealt with drift.

We still had to manage secrets separately.

We still needed human judgment when reviewing large infrastructure changes.

Terraform gave us control.

It did not eliminate responsibility.

The Practical Lessons

If I were starting again today, these are the lessons I’d carry forward.

Use Terraform to enforce boundaries, not just provision resources.

Treat account architecture as an IAM problem first and a Terraform problem second.

Invest in cross-repository visibility before investing in more automation.

Choose AI models based on task type, not convenience.

Review infrastructure plans like code. Because they are.

Prioritize understanding the system before changing it.

Most infrastructure failures aren’t caused by bad tooling.

They’re caused by incomplete context.

Looking Ahead

The three-account architecture gave us the boundaries.

Terraform gave us the implementation.

But neither addressed another challenge that emerged as Parjanya’s inference workloads grew:

Cost.

Specifically, inference cost.

While we were optimizing infrastructure, another inefficiency was hiding in plain sight.

Our VLM system prompt had quietly grown to over 4,000 tokens.

Most of it wasn’t doing useful work.

Tomorrow’s post moves from infrastructure into inference engineering.

I’ll share how we reduced prompt size by nearly 70%, doubled throughput on the same hardware, and discovered that some of the most impactful optimizations don’t involve changing models at all.

The Takeaway

Terraform didn’t make Parjanya v2.0 successful.

What it did was force us to be explicit.

Explicit about ownership.

Explicit about trust boundaries.

Explicit about infrastructure decisions.

And perhaps most importantly, explicit about the assumptions we were making.

That’s the real value of Infrastructure as Code.

Not automation.

Clarity.

And clarity scales far better than heroics.

This is part of my ongoing Parjanya v2.0 field notes, documenting the architectural, operational, and engineering decisions behind building a multi-tenant AI platform. In the next post, I’ll move beyond infrastructure and into inference optimization—where a prompt review ended up delivering more value than a hardware upgrade.

One AWS account.

Maybe two if you’re feeling disciplined.

One for development. One for production.

It sounds clean. It sounds sufficient.

For a while, I thought so too.

As Parjanya evolved into a multi-tenant image processing platform with real workloads, real customers, and increasingly complex infrastructure, I started asking a different question:

What does the minimum viable production-grade AWS architecture actually look like?

The answer wasn’t one account.

It wasn’t even two.

It was three.

Today Parjanya v2.0 runs on a three-account AWS structure:

• Management / Tooling

• Development

• Production

It added complexity upfront.

It also removed entire categories of risk that I didn’t want to discover during an incident.

This post explains why I made that decision, what problems it solved, what I intentionally chose not to build yet, and how I think other SaaS teams should approach the same trade-offs.

The Setup We Started With

Like most startups, we began with a single AWS account.

Everything lived there:

• Development resources

• Production workloads

• CI/CD pipelines

• Container registries

• Shared infrastructure

The setup was fast.

The problem with fast is that it often hides risk.

Nothing feels wrong until the day an experiment, a permission mistake, or an infrastructure change unexpectedly affects production.

As we started formalising Parjanya v2.0, I realised I didn’t want to keep relying on discipline alone.

I wanted structural safety.

That’s when the three-account model emerged.

The Third Account Isn’t Staging

Whenever I mention three accounts, people assume I mean:

• Dev

• Staging

• Prod

That’s not what I built.

The structure is:

Management / Tooling Account

This account owns the operational plumbing:

• Container registry

• Terraform state

• CI/CD definitions

• Cross-account IAM roles

• Billing aggregation

• Audit aggregation

Importantly, it never runs application workloads.

Development Account

This is where experimentation happens.

Infrastructure changes.

New features.

Testing.

Things break here by design.

Production Account

This runs live tenant workloads and customer-facing infrastructure.

Nothing experimental belongs here.

The critical insight is that the third account isn’t another environment.

It’s neutral territory.

It owns the things both Dev and Prod need, without forcing either environment to depend directly on the other.

The First Problem It Solves: Artifact Ownership

Consider a simple CI/CD pipeline.

It builds a Docker image.

Where should that image live?

If the registry sits in Dev, then Production depends on Development.

If the registry sits in Prod, Development depends on Production.

Neither feels right.

The Management account solves this neatly.

CI builds once.

Pushes once.

Both Dev and Prod pull the exact same artifact.

No rebuilds.

No copies.

No divergence.

One image.

One source of truth.

Multiple promotions.

That sounds like a small detail until you’re debugging an issue and need absolute confidence that Production is running the same artifact that passed validation earlier.

The Real Reason: Blast Radius

This is the reason that mattered most to me.

AWS accounts are the strongest isolation boundary AWS provides.

A broken IAM policy.

An accidental deletion.

A Terraform mistake.

A bad deployment.

These things cannot casually cross account boundaries.

That matters because Development should be chaotic.

People test things.

Infrastructure changes.

Experiments happen.

Failures are expected.

Production should not inherit that chaos.

With separate accounts, Dev can fail spectacularly and Prod remains untouched.

That’s a much stronger guarantee than “everyone promises to be careful.”

Credentials Become Safer by Default

A surprisingly useful side effect is credential isolation.

When someone is working in the Dev account, their credentials only see Dev resources.

They aren’t merely discouraged from touching Production.

Production is outside their scope.

The same applies to automation.

The deployment pipeline for Dev assumes different roles than the deployment pipeline for Prod.

The separation isn’t cultural.

It’s structural.

And structural controls tend to survive stressful situations much better than process documents.

Cost Visibility Becomes Obvious

Another benefit I underestimated was cost attribution.

Using AWS Organizations means every account naturally reports its own spending.

Development spend is visible.

Production spend is visible.