Today it is open to everyone. This post is the release notes: what Parjanya does, who tested it and what they found, everything that changed between the July baseline and now, and what is coming next. The engineering detail is here too, because this blog has always been the place for it.

What Parjanya does, in two passes

You drop a folder or files from your card or hard drive straight into the browser, and the files go from the browser directly into private storage over signed, resumable transfers. The moment the upload says complete you can close your browser or session: the analysis picks the work up on its own, and every upload and every verdict is recorded as a replayable intent, so nothing is lost to a dropped connection or a busy GPU. (The upload path and the reconciliation model behind it, TBIE, in-house research paper successfully verified for our usecase and evolved and tested for 18+ usecases.) Parjanya makes fast previews from every file (majority is RAW files across DSLR, mirrorless, mobile phones and Analog converted to Digital) and then checks each frame twice, in a fixed order.

• Pass one is a technical gate. Cheap, strict, and deliberately narrow: unreadable or damaged files, images whose pixel size disagrees with what the file claims, blank frames, near-solid exposures (a frame is set aside only when almost every pixel is pure white or pure black, so brackets and silhouettes pass), duplicates by perceptual fingerprint (frames within ten seconds on the same body are a burst, never duplicates), and screenshots or exports in formats no camera writes will be filtered out. Anything set aside lands in Skip with the reason attached. Nothing is deleted.

• Pass two is VLM enrichment. A vision-language model running on our own GPUs reads the frame and writes what you see on the detail page: what the photo is doing, how it is composed, what is holding it back and how strongly, plus the tags search runs on. A deterministic rule engine turns those findings into Keeper, Review or Skip with the reason attached. There is no score out of ten, on purpose: the same findings give the same verdict every time. (our v2.0 major change proved score engine doesn’t hold true and a deterministic rule engine does the best for our usecase)

You curate in a three-tab gallery where any verdict can be overruled, search the shoot in your own words (”leopard in dappled light”, “long exposure, silky water”, “symmetry, blue hour”), and after 500 uploads you get Parjanya Vision: a report card written like a coach rather than a mark sheet.

The people who tested it

The pilot group was small enough that every one of them changed something. A wildlife shooter pressed Stop on a 479-image batch expecting a pause; that single click is why Pause and Resume exist. A documentary photographer was first to point out that a Review/Skip line drawn only on technique penalises intentional motion and grain, which is now an open design question on the tracker (#58). Someone on their first camera uploaded iPhone photos that were accepted and never analysed. An archivist uploaded 16-bit grayscale scans that came back as blank white frames(#87). A landscape photographer with a body released this year sent RAW files our decoder had never seen, and they came back in false colour until it learned to fall back to the camera’s own embedded preview. And a mix of legacy camera bodies, manual lens and more.

Several of the pilot group are credited on the public tracker against the issues they raised; where someone preferred to stay unnamed, the issue says “pilot feedback” and nothing more. The onboarding guide itself, the ten-minute walkthrough new users get today, was recommended by one of the pilot experts.

Thank you, all of you. The list of fixes below is mostly your list.

https://github.com/JagadeeshRampam/parjanya-issues/issues

What changed since the July baseline

The v2.0.0 baseline was cut on 11 July after a 12,000-image validation run. Between then and today, 34 items on the public tracker were closed. Grouped by what you will notice:

Uploads and formats

• iPhone photos process now. HEIC and HEIF were on the supported list and were accepted at upload, then stranded silently with no preview and no analysis. The decoder library was installed the whole time; the one line that registers it with the imaging library had been lost in a refactor. Fixed and verified on real iPhone uploads (#40).

• 16-bit grayscale scans no longer come back white. Single-channel 16-bit TIFFs were clipped to solid white during preview generation and then, correctly, set aside as blank. The fix rescales by bit depth and validates every preview it writes. All 28 affected images were reprocessed (#87).

• Canon CR3 previews lost their magenta cast and dark border (#16).

• Filenames with spaces render previews again (#69), and camera and lens names no longer carry trailing padding from fixed-width metadata fields, which had been splitting one lens into two entries in filters (#48).

• Upload progress no longer flashes “Uploading: 0” mid-transfer (#3), and switching accounts in the same tab no longer shows the previous account’s upload bar (#33).

• When an upload is blocked, by a trial cap or an account hold, the app now says why and what to do, instead of a generic error (#24).

Processing you can control

• Pause and Resume. Stop was permanent, and a photographer found that out on a 479-image batch. Pause parks queued work and Resume picks it up exactly where it left off; a frame already being analysed when you pause finishes and keeps its result (#36). Stop still exists and now sticks end to end, including for work already in flight (#32).

• “Warming up the AI analysis.” The GPU fleet starts on demand. When it is starting, the gallery now says so, so a cold start reads as a cold start and not as a broken product (#62).

• Screenshots are turned away before they reach the GPU. Nine screenshots uploaded in August all passed the technical checks and three were accepted as good photographs. A metadata gate now rejects non-photographic content first (#64).

Reading the analysis

• Critique and improvements are two cards now. The critique is served exactly as written. The one or two actions worth taking were previously the last sentence of an italic paragraph most readers had stopped reading; they are now their own card, and they never repeat what the critique already said (#93).

• Burst groups show a per-frame breakdown when the frames in a burst got different verdicts (#1).

• The gallery badge showed “Pending” for every auto-curated image because two parts of the system spelled the verdict differently. Fixed (#90).

Gallery and search

• Newest first. The gallery was not listing your newest uploads first, which meant your own upload could be invisible to you. It reads from a recency index now (#66), and the three tabs agree on what “newest” means (#68).

• Semantic re-ranking using SigLIP 2 image embeddings is integrated behind a flag, and search now says visibly when it has fallen back to keyword ranking (#28, #30). Content filters for scene, lighting and mood are still hidden until every image carries those tags (#72, #25).

Parjanya Vision

• The report card shipped on 9 September. It started life as a per-shoot summary idea (#85) and became a tenant-wide card: what you shoot, which camera bodies and formats, how far you zoom, where your Review pile clusters, how you curate (rescued versus demoted), and what to try. Every rate travels with its count. Copy is coach voice by rule; there is a test that fails the build if words like mistake or grade appear (#98).

• Two same-day fixes after the first photographers opened it: bars that overflowed their card, and a horizontal fill bar that read like a progress meter for what is really two counts being compared. It is a vertical bar chart now (#99).

Account, trial and email

• Sign-up provisions correctly (#22) and confirmation codes deliver (#23), which sounds like table stakes and was the last launch dependency to close.

• Every account email exists now: three days before a trial ends, the day it ends, a follow-up a week later, when you are at 90% of a cap, when a cap is reached, and one for every action we take on an account, including a past-tense notice after a deletion completes (#88, #89, #104).

Where your photos live

Confidently, because it was audited before launch: your originals go from your browser straight into private, encrypted storage in Mumbai, in a space that belongs only to your account, with public access blocked at the bucket level and every connection over HTTPS. Every request is checked against your signed-in identity before it touches a record; there is no path from one photographer’s library to another’s. Every access to stored photos is written to an audit trail with alarms on any human or unexpected access. The vision model runs on GPUs we operate, so your photos are never sent to a third-party AI service and are never used to train models. And nothing is deleted by us: when a trial ends, uploads pause and everything already there stays readable and downloadable.

Search, in your own words

Search reads the critique, the composition notes and the closed-vocabulary tags the model writes for every frame (subject, scene, lighting, mood, dominant colours), so queries like elephant herd at a waterhole, backlit, dew on a spider web, ridgeline at dawn, layered haze or stairwell, geometric shadows work today. Two more pieces are built and switching on as the whole catalogue gets its tags and embeddings: filter chips for scene, lighting and mood, and a look-alike mode that ranks by visual similarity using SigLIP 2 image embeddings (#11, #25, #72).

Three things the pilot taught us

1. Stop is not Pause, and a UI should know the difference. The 479-image incident was not a bug; Stop did what it said. The bug was offering only an irreversible action for a reversible intent. Resume had one subtle requirement worth writing down: flip the records back to pending before re-queueing them. Queue first and a worker can pick the message up, still read “paused”, and drop the job on the floor.

2. Small uploads should not summon the whole fleet. GPU scaling used to step up on a timer whenever the queue was non-empty. A 13-image batch could scale to the maximum fleet, most of which arrived after the work was done, and the fleet then took 69 minutes to scale back down. Scaling now targets an exact capacity from visible-plus-in-flight queue depth; the same drain takes about four minutes. The accepted side effect is that scale-in occasionally interrupts a job mid-inference, which the queue’s visibility timeout retries (about 1% of images, zero lost) (#41).

3. “Supported” is a promise about acceptance, not decoding. The HEIC and 16-bit TIFF cases were the same class of problem: a format we would accept at upload and then fail on quietly. The legacy and edge-case list from the pilot is now long and mostly handled: RAW from bodies newer than the decoder’s database (falls back to the camera’s embedded preview instead of false colour), 16-bit and grayscale scans (rescaled by bit depth), iPhone HEIC, 25 RAW formats including Canon CRW, Sony SRF/SR2, Hasselblad 3FR/FFF, Phase One IIQ and Sigma X3F, filenames with spaces, manufacturer-padded camera names, bracketed exposures (pass the gate), bursts (stack, best frame ranked). Still open: older bodies and phone JPEGs that record no capture time can have burst frames flagged as duplicates (#70). Of the 34 formats, 18 are proven end to end in production; the next one we want a real file for is Fujifilm RAF (#42, #47). If you shoot Fuji, your first upload is a favour to us.

Numbers from the closed beta

• 20+ photographer accounts, from first-camera to full-time professional

• 18,884 images enriched, across 18+ file formats

• 104 issues filed on the public tracker since July; 34 closed, every one with what actually shipped

• Launch-day verification: 100 human-checked images, 51 Keeper / 47 Skip / 2 Review, zero stranded

• GPU drain after a small upload: 69 minutes → about 4 minutes

• Cold start after an idle period: 5 min 32 s today; 90 s is the target

• 28 archival scans and 261 verdict records repaired in place, nothing deleted

The trial

30 days, 1,500 photos, 30 GB of storage including previews. Whichever runs out first ends the trial. Uploads pause at that point and nothing is removed; your gallery, critiques and downloads keep working. Ask and we can raise a cap on your existing account. No card is needed to start.

What is next

In roughly the order we expect to ship:

• A “your shoot is ready” notification, email and in-app, so you can close the laptop after the upload completes and be told when the analysis is done. Today the first verdicts after an idle period wait about five and a half minutes for the AI to warm up; the target is 90 seconds (#105). Broader in-app notifications follow (#8).

• Near-duplicate detection, burst-aware, on top of the exact-duplicate check that is live (#4)

• Curation that learns from you. Every Keeper and Skip on a Review frame is already recorded; the model of your taste that reads them is next (#5)

• Content filters in search once every image carries scene, lighting and mood tags (#72), and a relevance floor so weak semantic matches stop looking strong (#74)

• Parjanya Vision phase 4: gear-specific advice that unlocks only when your own photos provide the evidence (#100), and phase 5, genre for your older photos from stored embeddings without reprocessing (#101)

• Self-serve payment (#9), and burst covers that show the best frame rather than the middle one (#92)

• The onboarding guide in more formats: a video walkthrough with voice-over is next (#53)

Everything, including the items we have not fixed yet, is public at github.com/JagadeeshRampam/parjanya-issues. Open and closed alike. The people who tested this deserved to see the list, and so do you.

Start

Onboarding guide, ten minutes start to finish: parjanya.phagyul.ai/guide.html

Create your account: parjanya.phagyul.ai/signup

For the pilot group of Experts across the world and my buddies: thank you again! 🙏

Jagadeesh

Engineering appendix (for readers who came for the systems)

Short entries; each has a longer post either published or drafted.

• Why there is no score. The rule engine reads distortion type and severity (motion blur: heavy) rather than a numeric quality score. Scores compressed incompatible defects onto one axis and gave photographers nothing to act on. The model still emits one; the gate ignores it.

• Caching that never existed. The dashboards were designed against a response cache that was never provisioned in production, so every dashboard call recomputed. They now serve from a snapshot store warmed by an in-process loop; the admin pane, which took up to 29 seconds per range, answers in under a second (#34).

• The month that weighed nothing. A field rename mid-July left usage attribution reading the old name and projecting only the old name from the database, so every image uploaded for a month contributed zero bytes to per-tenant attribution. The headline figure stayed right because it preferred a storage-measured value, which is exactly why nobody noticed. Lesson written down: a rename has to land in the projection too, and a test fixture that only builds one spelling cannot catch a rename (#81).

• Idempotent email. Trial emails failed silently for three weeks after the flag went live because the execution role never received send permission, and a failed send was marked as delivered before it was attempted, so it could never retry. The rewrite is claim → send → confirm, with release on failure, and it is the helper every new notification now uses (#88, #89, #104).

• Dedup that could eat a whole cluster. Symmetric concurrent rejection let every member of a duplicate cluster defer to another member that was also being discarded. Twelve images got no analysis. Survivors are now chosen by a total order, earliest wins (#97).

• TBIE. The reconciliation model behind all of this, Truth, Belief, Intent, Execution, has its own post: From Pipeline to Platform. The white paper is in draft.

• From localhost friction to production-shaped architecture

  ---

1. The morning I looked at the bill

On May 3rd, I ran a cost audit on Parjanya’s AWS account because the number I’d been carrying in my head — “it’s cheap, it’s a GPU worker and some Lambdas” — didn’t match what Billing was showing me.

It wasn’t cheap.

The account was running at roughly $100/day, and GPU EC2 accounted for most of that at the time: 57–65% of the bill. Two g4dn.2xlarge instances were running 24/7 on 100% on-demand capacity for our Qwen3-VL-8B image-quality worker.

Then I checked the launch template.

"OnDemandPercentageAboveBaseCapacity": 100

A hundred percent.

The Terraform module already supported Spot allocation. The mixed_instances_policy had been there since March. I simply hadn’t enabled it.

That is where a typical cloud-cost story might end: find the waste, flip the switch, move on.

But that only answered one question:

Which pool should an instance come from?

It took me much longer to ask the second:

How many instances do I actually need?

That second question turned out to matter even more.

2. What I believed about Spot capacity — and what the data showed

I had an assumption I’d never re-tested: GPU Spot capacity in Mumbai wasn’t reliable enough to build around.

That assumption was the reason the ASG had remained on-demand-only since March. It wasn’t really an oversight. It was an old decision that I hadn’t revisited.

So before changing anything, I pulled seven days of Spot price history across the GPU instance types we could realistically use:

The assumption hadn’t been unreasonable when I made it. It was simply stale.

By May, g4dn.xlarge Spot capacity was looking healthy across all three availability zones. g5.xlarge was a different story: better GPU capacity, but available in only two AZs.

Rather than rely entirely on historical pricing, I ran a live canary.

I switched the ASG to Spot, terminated two on-demand instances, and sent the same workload through both configurations.

The measured result was a 61% reduction in cost per unit of work.

The important lesson wasn’t just about Spot.

The thing standing between me and that saving wasn’t a technical limitation. It was an assumption I had stopped questioning.

3. Why I didn’t go 100% Spot

Spot capacity can disappear with little warning.

Our worker also has a significant warm-up cost: pulling the container image and loading a roughly 17GB model. If an instance is reclaimed during warm-up, it produces no useful work.

So I didn’t want the entire fleet to depend on Spot availability.

The configuration became a combination of on-demand and Spot:

gpu_instance_types       = ["g5.xlarge", "g4dn.xlarge"]
on_demand_base_capacity  = 4
on_demand_percentage     = 0
spot_allocation_strategy = "price-capacity-optimized"

The idea is simple:

• Keep a small on-demand base as insurance.

• Use Spot for everything above that base.

• Let the fleet scale with demand without paying the full on-demand price for every instance.

The on-demand base provides a floor that doesn’t depend on Spot availability. The Spot fleet provides most of the cost efficiency.

That combination turned out to be much more useful than thinking of Spot as an all-or-nothing decision.

4. Widening the pool — then narrowing it again

The next experiment was about the instance pool itself.

On July 9th, I saw several g4dn.xlarge Spot instances get reclaimed during warm-up. A pool containing only g5.xlarge and g4dn.xlarge was too narrow to absorb that pressure.

So I temporarily added:

• g4dn.2xlarge

• g5.2xlarge

These instances had the same GPUs as their xlarge counterparts but more CPU and RAM than the workload really needed.

I wasn’t buying additional GPU throughput.

I was buying capacity insurance.

The idea was that price-capacity-optimized could use the larger instances only when the cheaper pools were under pressure. In that situation, paying a little more for an instance that actually becomes available can be better than waiting for a cheaper instance that doesn’t exist.

I calculated the worst-case premium before enabling it. Across a 12,000-image batch, the additional cost would have been roughly $3.

That was an acceptable temporary trade-off.

On July 28th, I removed the 2xlarge types again.

The reason was straightforward: as a permanent configuration, paying for extra CPU and RAM that the GPU workload didn’t use wasn’t attractive.

The temporary insurance had done its job. Once the capacity pressure eased, the simpler pool made more sense.

This also exposed an important operational principle:

A good configuration isn’t necessarily a permanent configuration.

Sometimes the right response to a capacity problem is to widen your options temporarily, then narrow them again when the underlying conditions change.

5. Two Spot capacity droughts

The experiments also exposed the limits of Spot.

July 9th

Mumbai’s g4dn Spot capacity went dry across the region.

The fleet dropped from five instances to one during a 450-image batch.

No allocation strategy could solve that. There simply wasn’t enough Spot capacity available.

The on-demand base was what kept the workload alive, so I increased the base from one to three instances that afternoon.

August 5th–7th

This time the problem was g5.xlarge.

There was no Spot support in one of its two available AZs, while the other experienced an on-demand capacity gap.

There were three launch failures over three days.

price-capacity-optimized was able to route around the problem by falling back to g4dn.xlarge, although the fallback introduced additional latency.

These two incidents changed how I think about Spot availability.

The answer isn’t simply that availability is “better” or “worse.”

It is better on average for some pools, but individual instance families can still experience significant capacity constraints.

Spot is a probability and capacity-management problem, not just a pricing problem.

6. What changed when I moved to price-capacity-optimized

There are two strategies that are easy to confuse:

capacity-optimized chooses the Spot pool with the deepest available capacity. Price isn’t part of the primary decision.

price-capacity-optimized considers both capacity availability and price, choosing among pools that have a reasonable likelihood of remaining available.

The distinction matters.

The May canary that produced the 61% cost reduction used capacity-optimized. I later moved to price-capacity-optimized on July 9th.

I don’t have a controlled, back-to-back experiment proving the exact dollar improvement from PCO. So I don’t want to manufacture one.

The decision was based on the allocation model and the observed capacity conditions rather than a clean A/B test.

That distinction matters when discussing infrastructure results:

Measured results and engineering judgments are not the same thing.

There was another useful discovery in the same change.

The Terraform module had an override issue that was silently alphabetizing the instance-type list rather than preserving the intended priority order.

So the configuration had effectively been saying one thing while the infrastructure was doing another.

It was a good reminder that infrastructure configuration needs the same level of testing and observability as application code.

7. The bigger question: how many instances?

This is where the more interesting problem appeared.

Until July 27th, both scale-up and scale-down used SimpleScaling.

The logic was effectively:

Is the queue empty?

If yes, scale down.

If no, scale up.

The problem is that this treats a queue containing 13 messages exactly like a queue containing 1,300 messages.

The system knew whether work existed.

It didn’t know how much work existed.

On July 27th, a small 13-image upload caused the fleet to scale to its maximum. Three of six instances processed nothing, and the timer-based scale-down took 69 minutes.

The post-incident calculation showed approximately 4.6 instance-hours spent for around 20 minutes of actual work.

The fix was to make scaling backlog-aware.

Instead of looking only at whether the queue was non-empty, the scaling logic now considers the combined backlog — visible and in-flight messages — and maps it to an explicit target fleet size:

backlog     target

1–25        → 1 instance
25–100      → 2
100–500     → 4
500+        → maximum

The policy also changed from incremental scaling to ExactCapacity.

Instead of saying:

Add one instance.

it says:

The fleet should contain four instances.

That difference is important.

Each scaling decision can converge directly on the desired fleet size rather than repeatedly increasing or decreasing the fleet one instance at a time.

In a later measured run, the same class of drain dropped from 69 minutes to 4 minutes.

But I made another mistake.

On August 8th, I reduced every scaling band by half.

That worked for small batches. It also meant that a 108-image production run on August 11th stayed on one instance for roughly 50 minutes when a second instance could have provided useful parallelism after its warm-up period.

Three days later, I adjusted the bands again rather than reverting the entire change.

The lesson was bigger than the individual thresholds:

Scaling needs to understand both price and workload depth.

Choosing the right Spot pool answers:

Which instance should I buy?

Autoscaling answers:

How many instances should I buy?

Those are different optimization problems.

And I had spent months tuning the first one before realizing the second one was costing me more.

8. Why this isn’t just a startup problem

It’s easy to think of backlog-blind scaling as an early-stage infrastructure problem.

I don’t think it is.

Any system where demand arrives in bursts can have the same problem.

Consider:

• Batch customer onboarding

• Large data imports

• End-of-month reporting

• Media processing

• Scheduled analytics workloads

• Seasonal traffic spikes

• Any SaaS workload with uneven usage

A presence-based alarm can tell you:

There is work waiting.

It cannot tell you:

There are 20 messages versus 20,000 messages.

Those situations require completely different capacity decisions.

The mechanism is general:

Scaling decisions should consider the amount of work waiting, not just the existence of work.

The exact thresholds will vary by system, but the principle applies broadly.

9. What actually changed

Here are the results I can measure and reproduce:

• 61% lower cost per unit of work in the Spot vs. on-demand canary.

• 69 minutes → 4 minutes for a comparable production drain after moving to depth-aware ExactCapacity scaling.

• ~4.6 instance-hours consumed by an inefficient scaling response to a small workload.

• ~$3 maximum premium calculated for temporarily widening the Spot pool for a 12,000-image batch.

• 3 launch failures over 3 days during the August g5.xlarge capacity issue.

The longer-term Cost Explorer data tells a similar story.

Across the GPU hours recorded this year, on-demand averaged roughly $0.64/hour, while Spot averaged roughly $0.30/hour — about a 53% discount.

For the 785 Spot hours recorded year-to-date, paying the blended on-demand rate would have cost approximately $505. Actual Spot spend was about $235.

That’s roughly $270 in measured savings from those Spot hours alone.

GPU EC2 has cost $967.85 year-to-date against $6,637.43 in total account spend through August 22 — roughly 15% of total spend.

The important point isn’t the exact percentage.

It is that the optimization moved from a simple:

“Can I get cheaper GPU instances?”

to a much broader question:

“Am I buying the right capacity, from the right pool, for the amount of work actually waiting?”

That is a much more useful infrastructure question.

10. The bigger takeaway: price and capacity must be optimized together

The biggest lesson from this exercise isn’t simply that Spot is cheaper than on-demand.

It is that price and capacity need to be considered together.

price-capacity-optimized matters because the cheapest Spot instance isn’t necessarily the cheapest option if that capacity isn’t available — or is likely to be reclaimed. A slightly more expensive pool with better capacity can be the better economic choice when the alternative is waiting for capacity that may never arrive.

That is the real value of price-capacity optimization:

Don’t optimize for the lowest price. Optimize for the lowest viable cost of reliable capacity.

The distinction became particularly clear during the capacity droughts I encountered. No allocation strategy can create capacity that doesn’t exist, which is why the on-demand base remained important. But when multiple Spot pools were available, price-capacity-optimized gave the fleet a better way to balance availability and cost instead of treating either one in isolation.

There is, however, a second layer that is just as important.

price-capacity-optimized answers:

Which capacity should I use?

Autoscaling answers:

How much capacity do I need?

Those are separate problems.

The configuration that ultimately worked for me brought both together:

1. Keep an on-demand base as protection against a complete Spot capacity shortage.

2. Use price-capacity-optimized Spot allocation to balance price and capacity across viable pools.

3. Keep multiple instance options available when capacity is constrained, but avoid permanently paying for resources the workload doesn’t actually use.

4. Scale based on backlog depth, rather than simply checking whether work exists.

5. Use ExactCapacity targets so the fleet converges directly on the capacity the workload needs.

This changed how I think about compute optimization.

The goal isn’t to find the cheapest instance.

It isn’t even to find the cheapest Spot pool.

The goal is to find the right amount of reliable capacity at the lowest practical cost.

That is what price-capacity-optimized ultimately taught me: price optimization without capacity awareness is incomplete, and capacity optimization without workload awareness can still waste money.

The real optimization happens when all three line up:

11. The connection to context engineering

This is where this experiment connects with the earlier post on Context Engineering and Context Debt.

The two problems look different, but the underlying optimization pattern is surprisingly similar.

In context management, you have to decide:

Which information belongs in the context?

Then separately:

How much context should the model actually carry?

The GPU fleet has the same two layers.

Price-capacity-optimized answers:

Which pool should the instance come from?

Depth-aware scaling answers:

How many instances does the workload justify?

Getting the first decision right doesn’t automatically make the second one right.

That was the mistake I made.

I optimized the price and availability of each unit without first asking whether I needed that many units.

The same pattern appears in context management: a good retrieval strategy can still become expensive if the context budget itself isn’t being managed.

The broader lesson for me is simple:

Optimization is rarely one decision. It is usually a stack of decisions.

You can optimize the individual resource and still waste money because you’re using too much of it.

You can optimize the model and still waste tokens because you’re sending too much context.

You can optimize the database query and still waste resources because you’re running it too many times.

The useful question is therefore not just:

“What’s the cheapest way to do this?”

It is:

“What is the minimum amount of the right resource needed to do this reliably?”

That is the question I should have asked much earlier.

Configuration referenced in this post reflects the production setup as of August 12, 2026. Cost figures are based on a Cost Explorer pull covering January 1 through August 22, 2026.

The N+1 query problem is what happens when fetching N items costs you N+1 round trips to a data store instead of one — and it is, without much competition, one of the most common performance bugs I see in software that talks to a database.

I want to write about it here not because we hit the textbook version. We didn’t. What we ran into while building a new feature for Parjanya were two close cousins of N+1, caught at different points in the same week.

What made the experience worth writing about was the pattern-matching involved. Once I started asking “how many times does this data actually need to leave the system?” rather than just “does this code work?”, both problems became much easier to see.

I think that is a useful engineering habit for anyone building on top of an ORM, a NoSQL table, or a cloud API with a batch endpoint nobody on the team has used yet.

Why N+1 Gets Its Own Name

Picture the canonical example, the one I have either written or fixed more times than I’d like to admit: I load a list of blog posts, then loop over them to print each author’s name.

posts = Post.objects.all()          # 1 query
for post in posts:
    print(post.author.name)         # N queries — one per post

One query becomes N+1 queries.

With 10 posts, that’s mildly wasteful. With 10,000 posts loaded on an admin page, it can be the difference between a 200ms response and a request that never comes back.

Once I started looking for the shape rather than the syntax, I found it everywhere: a GraphQL resolver fetching a Book per Author node instead of batching with a DataLoader; a Rails view calling .comments.count inside an each block; a microservice calling GET /users/:id once per row instead of POST /users/batch.

Different stacks, same root cause — a per-item operation standing in for a single query or batch operation the data layer was perfectly capable of running.

What makes N+1 worth having a name of its own, rather than simply calling it “a slow endpoint,” is how it hides.

The loop looks correct in code review. It is correct — just expensive.

Unit tests don’t necessarily expose it either. A fixture with three rows produces three extra queries, but nobody notices when the entire test takes 40ms.

It usually becomes visible only with real data volume: a production dashboard gets slower, a support ticket says a page “used to be fast,” and by then the same pattern has often been copied into two other endpoints.

That’s why I care about catching this class of problem early. The fix can be a five-line change when I catch it during design or review. Once the data has grown into the architecture, the same fix can become a multi-file refactor with a production incident attached.

Our Version Wasn’t the Textbook Case — Which Is Exactly the Point

At Parjanya, we don’t run a traditional ORM — no ActiveRecord, no Django QuerySets, no lazy-loaded relationships waiting to surprise us.

Our data layer is DynamoDB, queried directly, with explicit asyncio.gather calls instead of implicit lazy loading.

At first glance, that seems like it should rule out N+1 entirely. Nothing is silently issuing a query behind a property access.

It doesn’t.

It just changes the shape of the problem.

We found two variants in the same week while building the same feature, and neither looked anything like the classic blog-posts-and-authors example.

Cousin #1: The Same Partition, Fetched Twice by Two Honest Code Paths

In mid-July, we built GET /api/v1/admin/dashboard — a cross-tenant operations view for Parjanya: pipeline health, queue depth, GPU fleet state, reconciler activity, and a per-tenant usage/cost table, all on one screen for whoever’s on call.

Two of those blocks needed the same underlying data.

The pipeline-health block needed every tenant’s image records to compute stale-pending buckets.

The per-tenant table needed the same tenant’s image records to compute that tenant’s usage numbers — through UsageMetricsService.build_dashboard, a method that already existed and already queried DynamoDB for exactly this.

The straightforward implementation would have called build_dashboard once per tenant, which queries that tenant’s image partition, and then separately queried the same partition again to build the pipeline block.

For T tenants, that meant 2T DynamoDB queries against partitions we only needed to read once each.

There was no obvious for tenant_id in tenant_ids: await fetch(tenant_id) loop. Nothing screamed “N+1.”

Instead, I had two well-intentioned pieces of code, written for two different sections of the same page, each doing the right thing in isolation and the wrong thing together.

We caught it before it merged by asking the same question the textbook N+1 case teaches us to ask:

How many times does this data actually need to leave DynamoDB?

The answer was once.

So we fetched it once:

# One image-field fetch per tenant, shared by the pipeline block AND
# the per-tenant usage dashboards (build_dashboard accepts prefetched
# images) — avoids double-charging the DDB partitions.
images_by_tenant = await self._fetch_images(tenant_ids, attention)

results = await asyncio.gather(
    self._pipeline_block(images_by_tenant, days, now),
    ...
    self._tenants_block(tenant_configs, images_by_tenant, days, now, email_by_tenant),
    return_exceptions=True,
)

Then I changed build_dashboard so it could accept data the caller had already fetched instead of insisting on fetching it again:

async def build_dashboard(
    self,
    tenant_id: str,
    days: int,
    images: list[dict[str, Any]] | None = None,
) -> UsageDashboardResponse:
    """Build the dashboard; ``images`` lets callers that already fetched
    the tenant's image fields (admin dashboard) avoid a second query."""
    if images is None:
        images = await self.db.query_image_usage_fields(tenant_id)

One parameter. One shared dictionary threaded through two call sites.

2T queries became T.

There is an important detail here that I think is easy to miss.

We didn’t make the T queries sequential. They still run concurrently through asyncio.gather, one per tenant.

That distinction matters because concurrency can fool me into thinking I’ve fixed the problem.

Concurrency hides N+1’s latency cost.

T parallel queries can sometimes return in roughly the time of one query.

But concurrency does nothing to reduce the volume cost.

I’m still paying for T reads against T partitions. I’m still consuming T times the read capacity. And I’m still one slow or misbehaving tenant away from holding up the batch.

A parallel N+1 is faster to run and just as expensive to pay for.

The fix that matters is cutting the fetch count, not merely parallelizing it.

Cousin #2: An API With a Batch Endpoint You Forgot Existed

The same dashboard also needed CloudWatch metrics — reconciler activity across five pipeline stages times four counters each, giving us 20 data series, SQS queue ages for up to three queues, and GPU fleet throughput.

The naive implementation would have made a get_metric_data call per series.

That’s 20 calls for the reconciler block alone, plus one per queue, plus one for fleet throughput.

Every one of those is a network round trip. Every one is a CloudWatch API call subject to its own rate limits. Every one has its own chance to fail independently in the middle of the request.

This is probably the most common shape of N+1’s cousin that I now look for: not a data-modeling problem, but a case where I forgot that the SDK already has a native batch primitive for exactly what I’m trying to do.

boto3‘s CloudWatch client accepts a list of MetricDataQueries in a single get_metric_data call — up to 500 of them — and returns the results together.

So I built one small helper around that and used it everywhere we needed metrics:

async def _get_metric_data(
    self, queries: list[dict[str, Any]], start: datetime, end: datetime
) -> dict[str, float]:
    """One batched GetMetricData; returns {query_id: latest/sum value}."""
    resp = self.cloudwatch.get_metric_data(
        MetricDataQueries=queries, StartTime=start, EndTime=end,
        ScanBy="TimestampDescending",
    )
    ...

The reconciler block builds all 20 queries — 5 stages × 4 metrics — as one list and makes one call.

The queue-age lookup builds one query per queue and makes one call.

Fleet throughput reuses the same helper.

Twenty-some potential round trips collapsed into three calls for the whole dashboard build: one per logical block, not one per metric.

And I didn’t want the batching optimization to compromise resilience.

We kept the per-tenant and per-block exception isolation through asyncio.gather(..., return_exceptions=True).

That means one broken AWS permission or one tenant’s bad data can still degrade that panel to a signal_unavailable note instead of taking down the entire request.

For me, that’s an important architectural point: batching and graceful degradation aren’t in tension. I just have to design for both instead of treating “make it one call” as permission to let one failure take down everything.

Both of these changes shipped in the same commit, feat(admin): per-environment admin dashboard API, alongside 11 new tests.

Neither would have shown up on a code-review checklist that only asked, “Is there a loop calling the database?”

The first problem was two separate, individually-correct code paths.

The second was a missed SDK capability.

The habit that catches both is the same habit that catches classic N+1:

Before I merge anything that talks to a data store or external API more than once, I want to know exactly how many round trips it costs and whether that number scales with something that’s about to grow.

The Sequel: Batched Isn’t the Same as Cached

This is actually the part of the story I find most useful, because from the outside it looked like we’d already solved the problem.

Four days after that commit, I was checking why admin dashboard loads still felt sluggish when we found this:

Verified live: no ElastiCache exists in any deployed environment (zero clusters, no VALKEY_* env vars), so ValkeyCache silently no-ops and every dashboard load was a full rebuild (~29s admin builds, a Cost Explorer call per view).

We had designed the endpoint to be cached.

There was a ValkeyCache layer wired in from the start, a 3-minute TTL, and the whole shape of what looked like a normal caching story.

What we hadn’t verified was the one thing that mattered:

Did the cache actually exist in any running environment?

It didn’t.

ElastiCache had never been provisioned.

Every cache read quietly missed, every single time.

That meant every dashboard view was re-running the entire fan-out we’d just finished optimizing: T parallel DynamoDB queries, three batched-but-not-free CloudWatch calls, and a Cost Explorer call none of us wanted running on every page load.

We had reduced the number of round trips per build from something like 2T+20 down to T+3.

And then we paid that reduced-but-still-real cost on every request because nothing was actually skipping the build.

That was the moment the distinction became very clear to me:

Fixing N+1-shaped fan-out is necessary, but it isn’t sufficient if the whole expensive operation still runs on every request.

A cache I haven’t verified in a running environment is, for all practical purposes, no cache.

And an unverified cache in front of a well-batched N+1 fix just means I’m repeatedly paying a smaller bill instead of a larger one.

The real fix didn’t require adding another managed service.

Instead, we used what we already had — the images table and the long-running ECS task — rather than depending on infrastructure that had quietly never existed:

• A SnapshotStore writes gzip’d JSON dashboard snapshots into DynamoDB itself, with a read-time expires_at and a fail-open contract. Any DynamoDB hiccup degrades to a cache miss, which is just today’s behavior, never worse.

• A background warmer, running in the API process’s own lifespan, rebuilds the admin snapshot on a roughly 4-minute cadence and each tenant’s usage snapshot every 8 hours. We chose those cadences based on how often the underlying signals actually change, rather than picking arbitrary TTLs.

• The Cost Explorer block gets its own 8-hour snapshot inside the costs block, since Cost Explorer data itself only refreshes a few times a day. That takes it from roughly one call per view to roughly three calls a day.

• Both /usage/dashboard and /admin/dashboard now serve snapshots by default, with an explicit refresh=true escape hatch for anyone who genuinely needs the live number.

Nobody waits on a cold build anymore.

And, just as importantly, nobody is silently re-running a 29-second, multi-block AWS fan-out every time someone opens a browser tab.

Why Catching This Early Mattered More Than Usual, Here

Parjanya launched to its first self-serve trial cohort on 1st August.

Both fixes — the shared-fetch deduplication and batched metrics, followed by the snapshot cache that replaced a cache that never actually existed — landed before that date, while tenant counts were still near zero and nobody outside the team was looking at the admin dashboard.

That timing is the whole argument for dealing with N+1-shaped problems early.

At near-zero scale, 2T queries and T queries cost about the same amount of nothing.

A 29-second rebuild on every page view is merely annoying when I’m the only engineer clicking refresh.

None of this was a production incident.

But I could see exactly how it would become one.

As tenant count and dashboard traffic grew, the fan-out cost would scale linearly with exactly the number a growing SaaS product is supposed to grow.

The phantom-cache cost would scale linearly with exactly the traffic a real launch is supposed to bring.

We found both by asking a very simple question while the answer was still cheap to change:

How many round trips does this cost, and does that number grow with our own success?

I’d much rather answer that question before customers force me to.

What I’d Tell a Team Building This Kind of Thing

Ask the round-trip question before merge, not after a slow-page complaint.

For any code that touches a database or an external API more than once per request, I ask explicitly: how many round trips does this cost today, and what does that number scale with?

If the answer is “the number of tenants” or “the number of rows on this page,” that’s the moment to fix it — not a note for later.

A loop isn’t required for N+1’s cousins to exist.

Our duplicate-fetch case had no for tenant_id in tenant_ids: await fetch(tenant_id) shape at all.

It was two separately-written, individually-reasonable code paths that happened to want the same data.

I now watch for that pattern specifically: two features, built at different times, that each independently fetch “this tenant’s records” without either one knowing the other exists.

Check whether your SDK already has a batch verb before writing the loop.

CloudWatch’s GetMetricData takes a list.

So does DynamoDB’s BatchGetItem.

So do many well-designed cloud APIs from the last decade.

The fix for “N calls instead of one” is very often not custom code.

It is reading one more section of the SDK documentation.

Concurrency is not the same fix as batching.

asyncio.gather over N calls is faster than N sequential calls, and it is still N calls.

It hides the latency symptom without touching the cost or capacity symptom.

I don’t want a parallel version of the bug to pass code review as if it were the batched version.

A cache is a claim, not a fact, until I’ve verified it in every running environment.

The most expensive line in this whole story wasn’t a query.

It was an unprovisioned ElastiCache cluster that let a caching layer silently no-op for days while looking, in every log line and every code review, exactly like it was working.

If I can’t point to the cache-hit rate on a dashboard, I don’t want to assume the cache is working.

I want to verify it.

Small scale is when this is cheap to fix, not evidence that it doesn’t need fixing.

The version of this bug that costs nothing to run also costs almost nothing to fix.

The version that ships to production and grows with user count is the same bug, several engineer-days more expensive, with a support ticket attached.

Parjanya is the multi-tenant image QA platform we’re building at Phagyul for professional photographers. If your team has its own N+1 cousins — duplicate fetches across features that don’t know about each other, unbatched calls to an API that quietly supports batching, caches that looked wired but weren’t — I’d love to compare notes.

A few months ago, I wrote a short note arguing that developer-experience platforms are excellent at velocity, but that AI systems demand a different kind of architectural thinking.

My argument was that convenient shortcuts can eventually accumulate into what I called AI debt: runaway costs, reproducibility gaps, and debugging opacity.

That argument was written from conviction.

This one is written from production.

I launched Parjanya v2.0, our multi-tenant SaaS platform for professional photographers, as a self-serve trial on 1st August 2026.

The system has processed individual photography sessions containing 12,000 images and roughly 450 GB in a single pass, and today serves tens of thousands of requests a week.

Between the first sprint in March and the July launch, I went through four complete architecture generations.

Almost every generation boundary came down to the same question:

Do I buy the platform, or do I own the primitive?

Sometimes I adopted the managed service. Sometimes I replaced it. And sometimes I came very close to replacing a primitive with a managed service, only to run the numbers and walk away.

The four headline decisions were:

1. SageMaker vs. Bedrock vs. raw AWS primitives for VLM and SigLIP 2 inference

2. Vercel vs. S3 + CloudFront for the React + TypeScript frontend

3. OpenRouter vs. a self-hosted model gateway

4. Clerk vs. Cognito + SES for authentication and transactional email

There were also three less glamorous decisions that ended up moving the bill more than some of the headline ones:

• Spot and Graviton

• OpenSearch and Step Functions

• CloudWatch, networking and storage classes

For each decision, I wanted to understand not only what the comparison tables said, but what actually happened after the architecture met production.

That distinction turned out to matter enormously.

My operating rule today is simple:

Buy managed services for their operations, not their defaults — and leave them when their cost floor or cold start starts fighting your workload.

1. SageMaker vs. Bedrock vs. AWS Primitives

The decision at the center of Parjanya

The heart of Parjanya is batch visual inference.

A photographer uploads a session — routinely 4,000 or more images, and sometimes 12,000. Every image that passes a cheap deterministic technical-quality gate is scored by the VLM and embedded using SigLIP 2.

The workload characteristics are important because they tell me almost immediately where the different approaches will fit.

The workload is bursty, not steady.

A queue can go from zero to thousands of images and then return to zero. There can be days when almost nothing arrives.

It is also batch tolerant.

Nobody needs a score in 200 milliseconds. What matters is that 4,000 images are processed by morning.

It is privacy sensitive.

These are often unreleased professional photographs belonging to clients. Keeping image bytes inside our VPC is not merely an architectural preference; it is something I can actually explain to a customer.

And finally, the economics are dominated by GPU time.

Before optimization, GPU compute represented well over half of the entire bill.

That combination — bursty, batch-oriented, private and GPU-heavy — became the single biggest predictor of which architecture made sense.

I started with SageMaker

The obvious managed options were Bedrock and SageMaker.

Bedrock represented the pure developer-experience end of the spectrum: API calls, no infrastructure, no cold-start management and managed models.

SageMaker sat somewhere in the middle: managed infrastructure around our own container and model.

Raw AWS primitives were the opposite end: EC2 GPU instances, Auto Scaling, SQS, SNS, CloudWatch and an AMI with the model baked in.

I used all three lenses, and two of them in anger.

Generation 2 ran on SageMaker.

I started with a BYOC real-time endpoint and then moved to Batch Transform.

SageMaker did exactly what I expected it to do: it got me to a working GPU pipeline quickly.

That is the promise of DX, and in this case, it delivered.

Then two things happened.

Quantization changed the economics more than the platform did

Qwen3-VL-8B in BF16 wants roughly 16 GB of VRAM.

A T4 has exactly 16 GB — and does not support BF16.

Moving to NF4 4-bit quantization using bitsandbytes brought VRAM usage down to roughly 5 GB, which made an 8B model practical on a T4.

More importantly, it produced a roughly 13× per-image speedup.

Our batch cost collapsed from approximately $500 to $40–50.

That was my first major lesson from self-hosting:

The model configuration can be a much bigger cost lever than the hosting substrate.

A managed platform doesn’t automatically pull that lever for you.

Then SageMaker’s operational shape started fighting the workload

The first problem was cold starts.

The managed endpoint took on the order of ten minutes to become useful, largely because of model-weight loading.

Our replacement — an AMI/NVMe-staged model running on a raw EC2 GPU worker — loads roughly 17 GB of weights in under two minutes.

For a workload that scales from zero, warmup is latency.

The second problem was the cost floor.

A real-time endpoint costs money every hour it exists, whether a single image arrives or not.

One GPU endpoint represented a standing tax of roughly $500+ per month.

At one point, I discovered a forgotten endpoint costing almost $870/month.

Our worst single day was roughly $1,000, around three to four times our typical monthly development spend at the time. The root cause was endpoint provisioning and extended runs.

The most important early optimization was almost embarrassingly simple: a scheduled shutdown Lambda.

We kept the endpoint warm only during a seven-hour nightly window, which reduced its cost by roughly 70%, from about $540 to $160 per month.

That taught me something I now use as a general diagnostic:

When your best optimization for a managed platform is a cron job that turns the platform off, your workload and the platform’s default posture probably disagree.

The third problem was debuggability.

When something went wrong on primitives, I could inspect a queue, an alarm, an Auto Scaling event and a container log.

Everything was directly readable.

On the managed platform, the same investigation often disappeared behind an abstraction that I could not see through as clearly.

That was ultimately what made the decision for me.

I moved to EC2 Spot

In April, Generation 3 deprecated SageMaker entirely.

I moved to EC2 Spot GPU workers — primarily the g4dn family with NVIDIA T4s — inside an Auto Scaling group driven by SQS queue depth.

The fleet genuinely scales to zero.

The minimum size is zero.

A scheduler Lambda watches the queue every two minutes, and the scale-in rule only fires after both visible and in-flight messages have been zero for five consecutive minutes.

For Spot interruptions, I added a five-second metadata poll.

The worker finishes the current image, checkpoints the state to DynamoDB and exits cleanly.

After months in production, we have lost zero images to Spot interruption.

The result was substantially better than our early expectations.

Our largest autonomous run drained roughly 2,400 images in under eight hours on four Spot GPUs, scaled itself back to zero three minutes after the queue emptied, and required no operator intervention.

The total bill was approximately $20.

Pure GPU inference now costs roughly $0.002–0.005 per image, or around $0.005–0.008 all-in.

Compared with the SageMaker era, that is an order-of-magnitude difference.

Overall platform cost fell by roughly 65–75% from its pre-optimization peak.

At our worst, we were spending around $80–90/day.

Under load today, it is closer to $20–30/day, with an idle floor around $5/day.

Spot itself delivered a measured 56–61% saving against on-demand, roughly $0.26–0.29/hour versus about $0.55/hour for a T4 in Mumbai.

I also ran an A/B test of going 100% Spot.

It beat its own forecast.

Drain time roughly halved, cost per message fell by around 60%, and we experienced zero interruptions.

The written verdict was simply:

Stay 100% Spot.

There was another Spot lesson that was less obvious.

I now treat quota increases as spend risk, not simply performance capacity.

The cost is in instance-hours, not instance count. Increasing a GPU quota to “improve throughput” also increases the maximum possible blast radius on the bill.

I manage the on-demand/Spot quota ratio as deliberately as I manage the fleet itself.

What about Bedrock?

I did not dismiss Bedrock.

I benchmarked it.

A managed vision API came in at roughly $0.01/image.

Our Spot fleet sits around $0.002–0.005/image.

That is a 2–5× gap, but it exists because I am doing three things simultaneously:

• scaling to zero

• running a quantized model

• using Spot capacity

If I ran the same fleet on-demand and kept it always on (warm), Bedrock would win.

I keep that Bedrock number as a standing benchmark.

The day the fleet stops being well run, I want the spreadsheet to tell me so.

There are also two reasons I would keep self-hosting even if the economics reached parity:

1. Image bytes never leave our VPC.

2. I control model version pinning completely.

Those are not spreadsheet variables.

What the industry says

The broader industry experience broadly supports this progression.

Discord describes a deliberate strategy of prototyping against managed frontier APIs to validate whether current-generation models can solve the product problem, then moving to self-hosted open models once scale justifies the operational burden. https://discord.com/blog/developing-rapidly-with-generative-ai

Shopify built its Merlin ML platform on open-source primitives rather than committing to a managed ML platform, largely because its teams had conflicting requirements and needed a frictionless prototype-to-production path.

https://shopify.engineering/merlin-shopify-machine-learning-platform

Independent comparisons generally place the self-hosting break-even around 10–20k requests/day, with much larger savings at higher volumes.

And AWS itself frames Bedrock versus SageMaker as a genuine trade-off between serverless convenience and control.

My experience added a third fork:

sometimes the right answer is neither.

Sometimes the right answer is EC2, SQS and an AMI.

What production taught me that comparison tables didn’t

The first lesson was that primitives compose — but not freely.

Our warm-pool design passed architecture review and failed during

terraform apply.

AWS does not allow a warm pool on an Auto Scaling group with a mixed-instances policy.

The DIY tax isn’t necessarily writing more Terraform.

It is discovering the composition rules the hard way.

The second lesson was that design numbers are not production numbers.

Our AMI-bake design promised roughly 90-second model loads.

Fresh instances actually took 30–40 minutes.

Lazy EBS restore meant a new instance reading 17 GB of untouched blocks crawled at single-digit MB/s.

The solution was NVMe staging.

It brought the 17 GB load down to under two minutes — measured in production.

I now put a date and configuration hash next to every performance number in our documentation.

Benchmark rot is a debt class.

The third lesson was about deterministic gates.

Before anything reaches a GPU, a sub-cent Lambda gate rejects technically unusable files.

That rewrite replaced an earlier ML-based scorer, reducing that stage’s cost by roughly 90%, from around $13 to less than $1/month, while also reducing its cold start from about 15 seconds to roughly two seconds.

More importantly, the deterministic rule engine owns the accept/reject decision.

If I change the curation policy, I can replay stored model outputs through the new rules in under a minute for pennies.

I do not need to run the GPU again.

The cheapest inference is the inference I skip. The second cheapest is the inference I never have to redo.

And then there was the prompt.

The prompt is infrastructure.

Our worst GPU incident was a CUDA out-of-memory crash caused not by the model or the fleet, but by prompt creep.

The scoring prompt had quietly grown beyond 4,000 tokens.

Instead of buying a larger GPU at twice the price, I audited the prompt and cut it by roughly 60%.

A lot of the removed instruction was teaching the model to produce outputs that the deterministic rule engine downstream deliberately ignored.

I was literally paying VRAM to generate information the system threw away.

On a 16 GB card, token budgets are a debt ceiling.

2. Vercel vs. S3 + CloudFront

The frontend is a React 19 + TypeScript + Vite SPA.

It contains the gallery, curation workflow, uploads and administration interface, and serves photographers globally while the FastAPI backend runs in Mumbai.

During launch, I needed fast global static delivery, many deployments a day, instant rollback and strict security headers.

The default answer in 2026 is Vercel.

And I understand why.

Git-push deployments, preview environments, an edge network and almost no infrastructure knowledge required.

But I chose S3 + CloudFront.

The deployment pipeline is intentionally boring:

Vite build → S3 sync → CloudFront invalidation

GitHub Actions handles it through OIDC.

Merging to main deploys development.

Production remains a manual, human-triggered workflow.

The cache design does most of the work.

Hashed assets under /assets/* are immutable and carry a one-year TTL.

index.html gets a 60-second TTL.

That makes the cache policy itself the deployment lever.

A release becomes globally visible within about a minute, and rollback is simply redeploying the previous build.

Security headers — HSTS with preload, frame-deny and nosniff — are handled by a CloudFront response-headers policy.

The cost is generally in the single-digit dollars per month and, most months, sits inside the free tier.

The reason I chose it was not that Vercel is bad.

It was that the frontend was not where I needed Vercel’s strengths.

Cached SPA loads from London, New York and Sydney are already measured in tens of milliseconds.

What users actually feel is the dynamic API round trip to Mumbai.

A page making three to five sequential API calls can add more than a second from North America or Europe.

No edge frontend platform can fix that.

The answer is API design: batching, caching and eventually regional read replicas.

That reinforced the thesis I started with:

The edge is a traffic director, not a compute executor.

Static assets and routing belong at the edge.

Authentication, model resolution, inference and telemetry remain centralized.

For a Vite SPA, the DX delta was also surprisingly small.

Vercel’s real advantage appears in SSR, ISR, preview deployments and framework-integrated serverless functions.

I wasn’t using most of those features.

For me, the DX advantage amounted largely to one GitHub Action.

The third consideration was cost topology.

S3 + CloudFront scales predictably with traffic.

Platform function pricing scales with invocations — precisely the axis I hope will explode as Parjanya grows.

I didn’t reject Vercel.

I simply didn’t want to pay a platform premium for capabilities my architecture wasn’t using.

3. OpenRouter vs. a Self-Hosted Gateway

Every AI product eventually has to answer four questions:

• Which model serves this request?

• What happens when it becomes unavailable?

• Who tracks the spend?

• Whose infrastructure sees the payload?

Managed gateways such as OpenRouter answer these questions across hundreds of hosted models.

One API key.

Automatic fallback.

Unified billing.

For Parjanya, however, there was a twist.

Our production models are self-hosted open weights:

• Qwen3-VL-8B

• SigLIP 2

They run on our own GPUs.

There is no third-party inference endpoint sitting in the hot path.

So I asked a slightly different question:

What plays the role of the gateway when the models are mine?

I ended up decomposing the gateway into primitives.

Model resolution is a three-tier loading ladder on the GPU worker.

Tier 1: weights staged on local NVMe from the AMI bake.

Tier 2: S3 synchronization fallback.

Tier 3: Hugging Face download.

Every load records the tier.

In healthy production, I should only ever see Tier 1.

A Tier 2 or Tier 3 log line is therefore an alert.

I also made the upstream dependency deliberately boring.

Hugging Face is contacted exactly once per model version, during AMI baking in CI.

Production instances never download models from it.

A scheduled job compares the upstream model commit hash with our baked AMI tags and tells a human when a rebake is worth considering.

That gives me something I care about enormously in an inference system:

model versions are pinned by construction.

Fallback is handled by the mixed-instances policy across GPU families and the loading ladder.

Spend accounting comes from CloudWatch and tagged infrastructure.

Every inference dollar can be attributed to an EC2 line item.

For our particular architecture, a managed gateway would introduce three structural costs:

1. A per-request economic layer.

2. A data-path layer where client payloads transit someone else’s infrastructure.

3. A rate-limit layer where our ceiling becomes somebody else’s policy.

OpenRouter’s published model, for example, includes roughly a 5% platform fee on provider pass-through pricing.

For a self-hosted, single-model, VPC-contained workload, I don’t get enough routing benefit to justify those layers.

But there is an important concession here.

If I were calling frontier hosted models, I would absolutely use a gateway.

And I would probably use a managed one first.

Standing up LiteLLM with a database, cache and pager before I have meaningful multi-provider traffic is premature infrastructure.

The gateway decision comes after the hosting decision.

It is not independent of it.

The lesson from our fallback

A fallback I have never exercised is not a fallback.

For a while, our Tier 2 S3 fallback was configuration theater.

The environment variables existed in the design.

They were never actually injected into the container.

The fallback existed on paper and nowhere else.

I fixed it by deliberately breaking Tier 1 and watching what happened.

That is one of the biggest operational differences between managed and primitive systems:

if I decompose a managed capability, I inherit the obligation to rehearse every part of it.

4. Clerk vs. Cognito + SES

Authentication was the closest call.

Parjanya needs multi-tenant signup, email/password authentication, JWTs carrying an immutable tenant identifier, trial limits and transactional email for verification and password resets.

Clerk is extremely attractive here.

Polished UI.

Session management.

A much nicer developer experience.

And, critically, email that simply works.

The alternative was Cognito + SES.

I chose Cognito + SES.

Why

First, economics.

At our current scale, Cognito costs us essentially nothing under its applicable MAU threshold and remains materially cheaper through the growth range we are targeting.

For a product whose unit economics depend on very high gross margins on per-image pricing, a permanent per-MAU platform fee is a margin haircut on every future user.

Second, and more importantly:

the token is the tenancy model.

Our JWTs carry an immutable tenant_id claim.

The backend verifies it using RS256 against Cognito’s JWKS, deliberately avoiding a signing secret in the backend.

Every S3 bucket policy and DynamoDB access pattern keys off that claim.

The backend issues short-lived, tag-scoped STS credentials per tenant.

Buckets are per-tenant.

KMS keys are per-tenant.

Cross-tenant access is explicitly denied at the policy layer.

Auth is therefore not just a login box.

It is the root of the isolation tree.

I wanted that root inside the same Terraform, account boundary and audit trail as everything it protects.

A hosted authentication provider can issue my token.

But it cannot itself be the IAM principal that my storage policies deny against.

Third, there is the privacy argument.

One bill.

One IAM boundary.

No additional identity-data processor.

For a privacy-positioned product, that is a sales answer as much as it is an architecture answer.

And then DIY sent me the bill

This is where I think infrastructure writing is often too polite.

The DIY option has a cost.

I want to list mine.

Cognito immutable attributes really are immutable.

I verified that in production.

Get your custom claims right the first time.

Schema changes can also be pool-destroying.

Adding a new schema attribute through Terraform can force user-pool replacement and therefore destroy every user account.

I added a role claim through the CLI instead, knowingly accepting permanent drift for that resource.

Cognito also does not support cross-account user-pool migration.

When we reorganized into a multi-account AWS organization, the pool’s location became a one-way door.

Either users would re-register or we would face a mass password reset.

Then there were session bugs.

Our frontend token cache survived logout.

User two could briefly see user one’s data until I added an explicit cache clear at every login boundary.

Presigned upload URLs had another subtle problem.

They inherit the STS credential’s expiry rather than the nominal URL lifetime.

An upload presigned at minute zero failed at minute 61 with a signature that looked valid.

These are precisely the integration bugs that a mature hosted authentication SDK has probably already encountered.

And then there was email.

Cognito’s default email channel has a 50-email/day per-pool cap and is best effort.

During launch, password-reset emails were silently dropped.

The API returned success.

CloudTrail recorded the event.

No email arrived.

That is one of the worst failure modes in distributed systems:

The lying green checkmark.

The fix was SES with a verified domain, DKIM and developer sending enabled through Cognito.

That led into SES production-access review.

Our first request was denied, almost certainly because our submission said too little about bounce handling.

SES sandbox status became the sole external blocker on our release lock.

We eventually documented the ordering constraint in red:

auto-confirm cannot be switched off while SES is sandboxed, or every signup breaks.

This is exactly where Clerk’s pitch becomes compelling.

You don’t have to think about any of this.

And during launch week, that matters.

DIY auth is paid for in calendar time and attention.

Those were the two currencies I had the least of during launch.

My verdict

Cognito + SES was right for Parjanya.

But it was the closest decision of the four.

The SES saga is actually the strongest argument for DX in this entire post.

A managed platform’s ability to make an entire class of launch-week failures impossible has real value.

If authentication were not the root of our isolation model, I would have been much more comfortable paying a per-MAU platform fee at seed scale.

5. The Decisions That Don’t Make Blog Titles

Some of the decisions that affected the bill the most had nothing to do with DX platforms.

Spot — and learning when a lever isn’t a lever

The Spot decision gets most of the attention because the savings are obvious.

We measured roughly 56–61% savings versus on-demand.

Our A/B test of 100% Spot beat its forecast. There are additional optimisations like cost-performance optimisation by AWS, which I will detail in upcoming posts.

Interruption handling has lost zero images.

We even measured Spot pricing within the region and found that availability-zone prices for the same instance type could differ by more than 50%.

So the fleet biases toward the cheaper zone.

But some fashionable optimizations simply failed the arithmetic.

Multi-region Spot arbitrage died because inter-region data transfer consumed most of the projected savings.

A neighboring region also turned out not to have GPU capacity to arbitrage in the first place.

The more interesting story was Graviton.

Our first Graviton audit found that only one of four Lambda functions could even run on ARM64.

Native dependency wheels, CUDA and unbenchmarked inference libraries blocked the rest.

The projected saving was approximately 0.4% of the total bill — about a quarter of a dollar per month.

Meanwhile, quantization, Spot and scheduling were saving 40–60%.

So I wrote down the conclusion and walked away:

Sometimes the best architecture decision is knowing where not to spend your engineering time.

Months later, a dependency shipped ARM64 wheels.

The blocker disappeared.

We revisited the decision and migrated.

The measured Lambda saving was approximately 20%, almost exactly what we had projected.

Both halves of the story matter.

I needed the discipline to defer a fashionable optimization.

But I also needed the paper trail that allowed me to revisit it later without starting the research from scratch.

My rule now is simple:

A two-hour ARM benchmark costs pennies and beats two weeks of debugging a premature migration.

And I never flip Terraform to ARM64 before the image has actually run there.

6. OpenSearch and Step Functions

Two other managed services lost to arithmetic.

Search

I kept search in-process.

No OpenSearch.

No external vector database.

At our scale, the smallest useful OpenSearch domain costs more per month than our entire reconciliation control plane costs per year.

Rather than guessing, I measured the ceiling.

Today, search latency is around two seconds at a few thousand rows and degrades toward the load balancer timeout somewhere beyond 100k rows.

So I wrote down the trigger:

roughly 30 tenants.

That is when OpenSearch becomes a yes.

SigLIP 2 vectors are already ready for a proper vector store when that trigger fires.

Orchestration

I also rejected Step Functions and workflow engines.

Our orchestration is a roughly 200-line reconciler.

DynamoDB Streams provide the fast path.

An hourly sweep provides the backstop.

The entire control plane costs a few cents a month.

I made the same arithmetic-driven decision around AWS Config and commercial compliance tooling.

A weekly EventBridge + Lambda audit covers the rules we actually need at effectively zero cost.

There is a famous industry parallel here: Amazon’s Prime Video team moved a monitoring service away from Step Functions and Lambda to a plain ECS monolith and reported roughly 90% lower infrastructure cost.

But DIY orchestration has its own failure modes.

Our replay Lambda once lost a single IAM permission and logged AccessDenied on every invocation for half a day.

The dashboards stayed green.

Hundreds of images stalled.

Nothing drained.

Then, on our launch-baseline day, the reconciler itself became the incident.

A read-amplification bug caused it to consume tens of millions of DynamoDB read units in 24 hours.

The bill was roughly $15/day.

The pipeline itself was almost entirely healthy.

The repair mechanism had become the damage.

I added targeted query guards and reduced that spend by roughly 85–90%.

Both incidents ended as code.

An alarm on silence.

A conditional-write guard.

A query guard.

That is an important property of primitives:

Institutional memory can become code.

7. Where the Bill Actually Hides

Three numbers changed how I think about infrastructure.

The first was CloudWatch.

Ungoverned CloudWatch can quietly consume 20–40% of a small platform’s operating cost.

A single chatty DEBUG-level Lambda can generate roughly $20+/month in log ingestion — more than our entire early operating budget for that stage.

Embedded Metric Format instead of per-call metric APIs saved around $15/month by itself.

Composite alarms reduced alarm costs by roughly 70%.

I also stopped creating per-tenant dashboards.

At $3/dashboard/month, that becomes a fixed cost that scales in exactly the wrong direction.

With those changes, observability fell back to a few dollars per month.

The second number was networking.

While investigating that roughly $20 autonomous GPU drain run, I discovered that almost half the bill wasn’t GPU time.

It was NAT gateway egress from around 180 GB of S3 reads.

The GPU wasn’t the expensive part.

The data path was.

VPC gateway endpoints for S3 and DynamoDB eliminated most of that cost and had already saved roughly $30–45/month of standing NAT expense.

A handful of interface endpoints cut the remaining NAT data by another ~80%.

That was another reminder that no SageMaker-vs-EC2 comparison table is going to tell you that your data path may cost more than your compute.

The third number was storage.

At photography scale, storage-class economics matter.

I rejected S3 Intelligent-Tiering for previews because its per-1,000-object monitoring fee becomes meaningful when you have millions of tiny files.

Instead, previews go directly into Infrequent Access, around 46% cheaper than Standard in our region.

Originals step down to Glacier Instant Retrieval, around 83% cheaper.

Rejected images move to Deep Archive after a grace period.

At 10 TB of rejected images, that’s roughly $10/month.

The result is about 3.7× cheaper than naïvely keeping everything in Standard.

The broader industry has demonstrated this at enormous scale.

Canva, for example, moved tens of billions of objects to Glacier Instant Retrieval after measuring that access collapses after the first couple of weeks, saving millions annually.

The meta-lesson for me was even simpler.

Our weekly audit Lambda costs effectively nothing.

It caught roughly 7,500 objects sitting in the wrong storage class for months.

It also found an S3 versioning configuration where the console suggested around 2,000 objects while the bucket actually contained more than 100,000 hidden version objects.

Left unchecked, that could have grown into millions of noncurrent versions, terabytes of invisible storage and a 5–10× inflated storage bill.

On primitives, audit is the platform.

8. The Incident Ledger

A fair DIY-versus-DX discussion has to include the incident ledger.

The operational burden that managed platforms sell against is real.

Between the first sprint and the post-launch study window, I logged 17 incidents across 12 classes.

Here are a few.

Terraform drift nearly became a self-inflicted outage.

After weeks of console-side firefighting, a routine Terraform plan came back with:

173 to add.
127 to destroy.

A blind apply would have deleted the NAT gateway, the DynamoDB table containing every VLM verdict and the Cognito user pool containing every login.

Getting back to a clean 0/0/0 state took roughly four focused hours spread across four days.

A managed DX platform wouldn’t have given me that state file.

But it also wouldn’t have given me a state file proving exactly what production was.

There was a six-hour preview outage caused by deleting the wrong “stale” grant.

What looked like a leftover origin-access policy was actually the live one.

Every preview returned 403 until I restored it.

There was a burst-photography incident caused by a silently failing install.

A shell || echo swallowed both a 404 and a missing archive.

We shipped a worker image without its EXIF tool.

Near-identical burst frames then started being falsely rejected as duplicates.

Launch went out with exact-match-only deduplication as the safe setting.

There was a wrong-bucket incident where hundreds of images hung on repeated download errors because queue payloads contained a valid key pointing to the wrong bucket.

And there was a weekend bulk import where messages silently expired from SQS because queue retention was shorter than the weekend.

Retention is now an invariant with a name.

Not a default.

These incidents sound like arguments against primitives.

I actually think they demonstrate the opposite.

Every incident ended as code:

• a Terraform validation block

• a conditional write

• a CI grep

• a CloudWatch alarm

• a queue invariant

Institutional memory became executable.

And across all 17 incidents, recurrence after adding the guardrails is currently zero.

That is the bargain I am making with primitives.

I accept the initial operational burden in exchange for owning the system completely enough to turn every lesson into a permanent constraint.

The multi-account reorganization was another example.

It was triggered partly by a deployment script accidentally pushing to production from the wrong local context.

The reorganization added roughly $10/month in duplicated audit tooling and moved around 135 GB of data for pennies.

In return, it dramatically reduced blast radius.

That kind of control is difficult to purchase from a single-tenant DX platform’s pricing page.

One uncomfortable AI footnote

Parjanya was substantially AI-assisted in its construction.

Two of the nastiest silent failures in the incident ledger — the swallowed exiftool installation and a bare-except registry writer — were both machine-written and machine-reviewed code.

Their common property was silence.

The answer wasn’t to use less automation.

It was to write better contracts:

• CI greps for exception-swallowing patterns

• log-line contracts for every pipeline tier

• alarms on absence

• explicit invariants

The same discipline that makes primitives operable also makes AI-written code operable.

9. The Repatriation Debate, From the Cheap Seats

All of this sits inside the much larger cloud-versus-hardware debate.

37signals has made the case for leaving the cloud, with a reported reduction from roughly $3.2M to $1.3M annually and projected savings of more than $10M over five years.

Ahrefs has argued that owned hardware saved them hundreds of millions compared with AWS list pricing.

Canva has taken almost the opposite position: stay in the cloud, but become extremely good at engineering the bill.

At Parjanya’s scale, I’m much closer to Canva’s school.

But I think there is an important distinction.

I haven’t left the cloud.

I also haven’t left managed services.

DynamoDB, SQS, Lambda, CloudFront and Cognito are still managed services in our architecture.

I keep them because I want their operations.

What I left were the managed services whose cost floors and cold starts fought the workload.

A real-time inference endpoint billing through the night for a queue that is empty by midnight isn’t an operations benefit.

It is a subscription to someone else’s default posture.

That is the distinction I think the broader repatriation debate sometimes misses.

It isn’t really:

cloud vs. metal

or:

platform vs. primitives

It is:

What is the right economic and operational shape for each individual workload?

And that answer should be re-run when the workload changes.

10. The Decision Framework I Use Now

After five months and four architecture generations, I have reduced the decision process to four questions.

1. Is the expensive part of the hot path bursty?

If yes, the platform’s cost floor may be your enemy.

Scale-to-zero primitives are probably worth the setup cost.

If the load is steady, the cost floor becomes less relevant and the managed platform’s operational value may be worth buying.

That one question explains why I left SageMaker, benchmarked and declined Bedrock, declined Vercel and kept DynamoDB on-demand.

2. When it breaks at 2 a.m., can I read the failure surface?

This ultimately mattered more to me than cost.

A queue.

An alarm.

An Auto Scaling event.

A container log.

These are things I can read.

If my team cannot or does not want to own that failure surface, the managed abstraction is doing real work for us.

I should pay for it.

3. What currency does the DIY option cost — and do I have that currency right now?

Cognito + SES cost me launch-week calendar time.

That was the scarcest resource I had.

The exact same decision at another point in the company’s life would have cost almost nothing.

So I no longer think only in dollars.

I ask:

What resource am I actually short of?

Money?

Engineering hours?

Attention?

Launch-week calendar time?

Operational confidence?

The right architecture can change depending on the answer.

4. Have I written down the condition that would reverse my decision?

This is probably the most useful practice I have adopted.

A “no” is not a permanent architectural belief.

It is a decision with a trigger.

For example:

• vLLM becomes a yes around 500 active users or sustained 100+ images/minute.

• OpenSearch becomes a yes around 30 tenants.

• The monorepo decision changes with team growth.

Every rejected option gets an un-rejection trigger.

That turns architecture from ideology into a portfolio that I can rebalance.

The Operating Sentence

The note that started this journey argued that high-performing platforms should choose DX where it helps, and control where it matters.

After a launch, four architecture generations, seventeen logged incidents and a roughly two-thirds reduction in our platform bill, I would sharpen that sentence.

This is the rule I now keep at the top of our architecture decision register:

Buy managed services for their operations, not their defaults. Leave them when their cost floor or cold start fights your workload. And put deterministic gates around expensive models — the cheapest inference is the inference you skip.

Parjanya v2.0 is now live as a self-serve trial.

The production system runs Qwen3-VL-8B in NF4 and SigLIP 2 So400M on scale-to-zero Spot GPU workers in ap-south-1.

Everything sits behind a deterministic rule engine that owns every accept/reject decision.

The model observes.

The system decides.

And, perhaps more importantly, I now have a much clearer idea of when I want a platform to make the hard parts disappear — and when I would rather own the hard parts myself.

The original note documented how a combination of EC2 GPU warm pools, Auto Scaling Groups, SQS, EventBridge, DynamoDB Streams, and pre-baked GPU AMIs enabled Parjanya to eliminate SageMaker cold-start penalties, achieve scale-to-zero economics, and build a replay-native control plane capable of self-repair and reconciliation. Rather than treating GPU inference as a model-serving problem, the architecture reframed it as an orchestration and reconciliation problem, resulting in dramatically lower startup latency, parallel execution, improved replay safety, and reduced operational cost.

This article expands on that foundation by examining the complete system architecture, the TBIE (Truth, Belief, Intent, Execution) operating model, the move from polling to event-driven reconciliation, and the real production incidents that ultimately shaped the platform’s reliability model.

EXECUTIVE SUMMARY

Parjanya v2.0 is Phagyul AI’s image ingestion and GPU-based visual scoring, later evolved as curation platform by replacing score engine to rule engine.

Its nightly VLM inference pipeline was originally built on Amazon SageMaker. The pipeline processed photographer uploads using Qwen3-8B-VLM and worked — but it revealed a fundamental architectural mismatch over time. Wall-clock time regularly stretched to ~11 hours. Cold starts ate into every nightly window. Replay handling for failed enrichments grew fragile. Sequential execution prevented scale-out. And persistent managed-platform overhead stopped making financial sense for burst-only workloads.
The core insight that drove the migration was a reframing of the problem. The question was no longer

‘how do we run a model?’

It became:

‘how do we orchestrate burst-scale inference reliably, replay safely, and cost-efficiently?’

That reframing led to retiring SageMaker and rebuilding the pipeline entirely from native AWS primitives — EC2 GPU warm pools, Auto Scaling Groups, SQS-driven orchestration, EventBridge-triggered scale-out, and pre-baked GPU AMIs with Qwen3-8B-VLM weights embedded.

This is a detailed internal engineering deep dive into that migration. It covers the original architecture and its failure modes, the design decisions behind each new component, how TBIE (Truth, Belief, Intent, Execution) reconciliation patterns were applied to make the system replay-safe and operationally resilient, the specific production incidents encountered and how each shaped the architecture, and the guardrails now embedded to prevent regression.
Key outcome: the rebuilt system achieved near-zero or sub-minute startup latency, parallel GPU workers replacing sequential processing, event-driven replay handling, scale-to-demand orchestration, and drastically reduced idle GPU waste. More importantly, it became a system that could reason about and repair its own state — a reconciliation system, not just a pipeline.

BACKGROUND: WHAT IS PARJANYA V2.0?

Parjanya v2.0 is the image ingestion and GPU-powered visual scoring platform at Phagyul AI Systems. Photographers upload raw images; the platform extracts technical metadata (EXIF, resolution, format), performs visual-language model inference to produce content quality scores, enforces content policy, and writes enriched metadata back to the workflow database.

The data flow is:

• Photographers upload images to per-tenant S3 buckets via browser-based presigned URLs → An S3 event triggers a Graviton Lambda for EXIF extraction and technical validation.

• On successful validation, a DynamoDB workflow row is written with status pending_vlm_enrichment.

• A GPU worker consumes this work, runs Qwen3-8B-VLM inference, and writes scored metadata back(later recuration) to DynamoDB.

• A control plane (replay Lambda, reconciliation loop) monitors for stalled or missing work and re-orchestrates as needed.

• The platform spans immutable uploads (S3), workflow state (DynamoDB), intent queues (SQS), Lambda orchestration, GPU autoscaling (ASG + EC2), browser-side uploads with CORS policies, content policy enforcement, and tenant isolation.

THE ORIGINAL SAGEMAKER PIPELINE

Parjanya’s first VLM inference implementation used Amazon SageMaker as the compute platform. A nightly batch job was triggered on a schedule. SageMaker endpoints were provisioned, images were submitted for inference in sequence, results were collected, and endpoints were torn down.

The original system worked. It produced correct results. But it had structural problems that compounded as the platform grew:

• Cold-start tax: Every nightly run incurred GPU cold starts before the first image could be processed.

• Sequential execution: Image inference ran sequentially across the batch; there was no parallelism at the worker level.

• Fragile replay: Failed enrichments required increasingly fragile custom replay logic layered on top of SageMaker’s job model.

• Idle cost: The platform was billed for managed infrastructure overhead even during windows with no work to do.

• Throughput ceiling: Wall-clock time regularly stretched to approximately 11 hours per nightly run.

THE ARCHITECTURAL REFRAMING

The decision to migrate was not triggered by a single catastrophic failure. It was triggered by a change in how the problem was understood.

Once the problem was stated that way, SageMaker was the wrong tool. SageMaker is an excellent managed platform for running models. It is not designed to be a burst-scale GPU orchestration layer with event-driven replay. Those properties had to be bolted on from outside, which is why the replay logic was fragile and why cold start latency was unavoidable.
Native AWS primitives offered a better fit for the actual problem: EC2 GPU instances could be pre-baked with model weights, Auto Scaling Groups could scale to demand rather than a fixed schedule, SQS provided durable, replayable intent, and EventBridge could trigger orchestration the moment new work appeared. The result is not a simpler system — it is a more intentional one.

THE NEW ARCHITECTURE: NATIVE AWS- PRIMITIVES

EC2 GPU WARM POOLS

Rather than provisioning GPU instances from scratch each time a job runs, Parjanya v2.0 uses EC2 warm pools attached to the Auto Scaling Group. Instances are pre-initialised and held in a stopped or standby state. When the ASG scales out, warm pool instances start in seconds rather than minutes, eliminating the cold start latency that dominated the original SageMaker pipeline.

Warm pool design note: instances in the warm pool must be pre-baked with model weights already on disk. Do not rely on model download during instance boot — download failures become silent launch failures. The AMI is the contract between infrastructure and runtime.

PRE-BAKED GPU AMIs WITH QWEN3-8B-VLM WEIGHTS

Each GPU AMI is built with the Qwen3-8B-VLM model weights, runtime dependencies, and worker application code baked in. The AMI is immutable. When a new model version or dependency is needed, a new AMI is built and the launch template is updated. This makes the execution environment reproducible and testable before deployment.

Key constraints on AMI design:

All model weights must be present at AMI build time — no remote download at boot.
Python environment, CUDA libraries, and worker scripts must be included in the AMI.

The launch template image tag and the AMI must be kept in sync; stale launch templates are a production risk (see Incident 6).

Container and model artifact versions are a compatibility pair — they must be validated together.

AUTO SCALING GROUPS (ASG)

An Auto Scaling Group manages the GPU worker fleet. The ASG is configured with a minimum of 0 instances (scale-to-zero for cost efficiency) and a maximum that reflects the burst capacity needed during active processing windows. Desired capacity is driven by queue depth rather than a schedule.

ASG configuration decisions:

1. Scale-out is triggered by the replay Lambda when SQS queue depth exceeds threshold.

2. Scale-in is time-based to prevent premature termination of in-flight workers.

3. Instance type selection balances GPU memory requirements (Qwen3-8B-VLM needs sufficient VRAM) against cost per inference.

4. Warm pool instances buffer the latency between a scale-out trigger and first inference.

SQS-DRIVEN ORCHESTRATION

SQS is the durable intent layer. Each message represents a single VLM inference job for one image. Messages are not ephemeral transport — they are the replayable journal of work that must happen. This is a TBIE design principle: intent must survive transient failure.

SQS configuration decisions:

• Visibility timeout is set to exceed the maximum realistic inference time to prevent duplicate processing.

• Queue retention is set to exceed the maximum expected outage window (e.g., weekend + holiday gap) so intent survives scale-to-zero periods.

• A dead-letter queue captures messages that exceed the maximum receive count, enabling manual triage of terminal failures.

• Message attributes carry routing metadata: tenant ID, image key, source bucket, expected output location.

• Queue retention is not a generic operational setting. It is a resilience parameter. If the GPU fleet is at zero for 3 days, messages must still be present when the fleet wakes up. Set retention to match your worst-case outage window, not a default.

EVENTBRIDGE-TRIGGERED SCALE-OUT

Scale-out is event-driven via EventBridge and DynamoDB Streams. When a DynamoDB row reaches pending_vlm_enrichment, the stream emits an event. A stream processor filters and batches records, then invokes the replay Lambda. The replay Lambda evaluates queue state (Belief) against database state (Truth), emits SQS messages (Intent), and increases ASG desired capacity (Execution trigger).
This replaces a polling model where a scheduled timer would periodically scan for work. The event-driven approach gives lower latency, eliminates wasted invocations during quiet periods, and makes the system’s reaction to new work causally clear:

Truth changes → control plane notices → Intent is created → Execution follows.

REPLAY LAMBDA (CONTROL PLANE)

The replay Lambda is the reconciliation control plane. It is invoked by DynamoDB Streams and by periodic safety-net schedules. Its job is to compare Truth (DynamoDB workflow state) against Intent (SQS queue depth) and repair any divergence: emitting missing messages, resetting stale state, and triggering ASG scale-out when work is present.
The replay Lambda is a first-class reliability surface, not a background helper. If it fails silently, every other safety net becomes weaker. Its observability must be stronger than ordinary worker observability.
Monitor its error rate, execution duration, and whether it successfully reads queue attributes. A lambda that runs but cannot see queue state is functionally blind.

TBIE: TRUTH, BELIEF, INTENT, EXECUTION

TBIE is the operating model used to design, debug, and evolve Parjanya v2.0. It decomposes a distributed system into four conceptually distinct layers. The model is most valuable as a diagnostic tool: when the system behaves unexpectedly, TBIE provides a structured vocabulary for identifying which layer has diverged.

TRUTH

Truth is the durable, authoritative record of what is actually true. It must survive worker restarts, deploys, transient outages, and infrastructure drift.
In Parjanya v2.0, Truth lives in:

S3 — immutable image uploads and preview artifacts. If an object is in S3, it exists.

DynamoDB — workflow rows recording status, EXIF metadata, VLM scores, curation state, and hashes.

Derived committed state — VLM output written durably after successful inference.

Truth answers questions like:

• What objects exist?

• What stage is this image in?

• What has been computed?

• What is the authoritative record for this batch?

BELIEF

Belief is the operational view of reality. It is derived from Truth and current signals, but it may lag, be partial, or be noisy. Belief is what operators see first — but it must never be confused with source-of-record Truth.

In Parjanya, Belief includes:

1. SQS queue depth and in-flight message counts.

2. ASG desired capacity, running instance count, and warm pool state.

3. Worker health signals and CloudWatch alarms.

4. GPU availability and instance lifecycle state.

A healthy Belief is not the same as a healthy system. The queue can appear empty while Truth still holds a large pending backlog. That gap is a reconciliation failure, not an idle system.

INTENT

Intent is the durable record of work that has been requested. It is the replayable journal of ‘please do this work.’

The critical property of Intent is replayability: if Intent can be lost too early, transient failures become permanent failures requiring manual rescue.

In Parjanya, Intent is:

• SQS messages — each representing a single VLM inference job.

• Replay jobs — reconstructed Intent emitted for rows that had pending Truth without corresponding queue messages.

• Regrade jobs — Intent emitted for historical reprocessing after policy changes.

EXECUTION

Execution is the stateless work that turns Intent into side effects and updated Truth. Execution is where work happens, but in a resilient system it must never be allowed to erase the record of what was supposed to happen.

In Parjanya, Execution includes:

• Graviton Lambdas for EXIF extraction and technical validation.

• GPU workers running Qwen3-8B-VLM inference and writing scored results to DynamoDB.

• Replay Lambda emitting Intent and triggering ASG scale-out.

• ASG and EC2 actions that instantiate the worker fleet.

TBIE AS A DIAGNOSTIC MODEL

The power of TBIE is operational. Once internalised, it changes the first question asked during any incident. The question is no longer ‘is the queue healthy?’ or ‘are the GPU workers running?’ Those are Belief signals. The real question is:

Is the system waiting because Intent is missing — or because Intent is present but Execution cannot realise it?

That single question narrows the search space dramatically and prevents wasted time on the wrong layer.

EVENT-DRIVEN RECONCILIATION

Parjanya initially used a scheduled polling model. A timer would wake the replay Lambda periodically. The Lambda would scan for pending work and, if appropriate, emit SQS messages and trigger GPU autoscaling. This worked but introduced avoidable latency, wasted invocations, and operational ambiguity.
The move to DynamoDB Streams changed the architecture fundamentally. Now, when a workflow row reaches pending_vlm_enrichment, the stream emits an event immediately. An event source mapping filters for relevant writes, batches records briefly, and invokes the replay Lambda. The Lambda discovers missing or retriable work, emits Intent into SQS, and kicks the GPU ASG when necessary.

The net effect:

Reconciliation latency dropped from O(poll_interval) to O(stream_propagation) — typically seconds.

• Wasted Lambda invocations during idle periods eliminated.

• Causal chain made explicit: Truth changes → stream emits → Lambda reacts → Intent created → Execution follows.

• Operators can reason about system state without wondering ‘has the next poll happened yet?’

• A reconciliation system does not rely on humans noticing a stalled queue and manually fixing it.

It detects, explains, and repairs the divergence between Truth and Intent on its own. Event-driven streaming is what makes that self-repair fast enough to be useful in production.

PRODUCTION FAILURE MODES & LESSONS

The following incidents occurred during the development and operation of Parjanya v2.0. Each is documented here not as a post-mortem but as a design lesson — how the failure shaped the architecture, and what guardrail was added as a result. TBIE is used throughout as the diagnostic lens.

Incident 1: Transient GPU Failure Destroying Intent

GPU workers were deleting SQS messages before the retry window had been exhausted. A worker could encounter a temporary S3 download failure or an OOM condition, write a failure status to DynamoDB, and still delete the queue message. At that point, Truth said the work had failed, but Intent no longer existed — the system had no automatic path to retry.

TBIE diagnosis: Execution destroyed Intent prematurely.

Resolution:

Workers now return explicit outcome signals: retry, terminal_failure, or completed.

retry: preserve the message. Visibility timeout and receive count govern re-attempts.

terminal_failure: write durable failed Truth, then delete the message.

completed: write final Truth, then delete the message.

That separation preserves replayability while still allowing durable failure when necessary. The message deletion decision became a function of execution outcome, not a side effect of worker exit.

Worker outcome protocol

if outcome == ‘retry’:
# Do NOT delete. Let visibility timeout expire.
return # SQS will re-deliver
elif outcome == ‘terminal_failure’:
write_durable_failure(db, image_id)
sqs.delete_message(receipt_handle)
elif outcome == ‘completed’:
write_enriched_truth(db, image_id, scores)
sqs.delete_message(receipt_handle)

Incident 2: Pending Truth Without Intent (Reconciliation Gap)

DynamoDB rows remained in pending_vlm_enrichment status, but no corresponding SQS messages existed. Queue depth showed zero. The system appeared idle. The backlog still existed in Truth but was invisible to Execution.

TBIE diagnosis: Truth was pending while Intent never existed. Classic reconciliation gap.

Resolution:

• The replay Lambda now scans for pending rows without corresponding queue messages.

• For each orphaned pending row, it regenerates a queue message and kicks the GPU ASG.

• DynamoDB Streams ensure this scan is triggered promptly rather than waiting for a polling cycle.

• The key insight: this is not a worker bug. Workers cannot fix work that was never queued. The control plane must own the responsibility of keeping Truth and Intent aligned.

Incident 3: Retry vs. Replay Distinction

Some failures — S3 download errors, temporary GPU OOM — are retryable once the underlying condition is fixed. But simply re-enqueueing the message is insufficient when the workflow row contains partial or stale derived fields from the failed attempt.

TBIE clarification:

a retry is another attempt at the same execution. A replay is a deliberate reset to a pre-execution boundary, removal of stale derived state, and emission of fresh Intent.

Resolution:

Retry: re-attempt on the existing message, preserving current workflow state.

Replay: reset the DynamoDB row to a clean pre-VLM boundary (clear stale score fields, VLM status, partial outputs), then emit a fresh SQS message.

Replay is a controlled reconstruction, not just another attempt on top of corrupted state. The distinction prevents accumulating corrupted derived fields across multiple failed attempts.

Incident 4: Wrong Bucket, Right Key (Deterministic Misrouting)

An SQS payload pointed to a valid preview key but in the wrong bucket. The worker repeatedly failed with S3 download errors even though the file existed — just not at that address. This was deterministic misrouting, not flaky execution.

TBIE diagnosis: Intent encoded the wrong target. Not an Execution failure — an Intent construction failure.

Resolution:

Preview keys now canonically use the uploads bucket as the authoritative source.
Intent construction validates bucket routing before emitting to SQS.
Eligibility checks were tightened: only objects confirmed present in the correct bucket are enqueued.
The lesson generalises: Intent must encode the correct target, not just some valid key. A message that is ‘almost correct’ will fail deterministically and indefinitely until the Intent itself is repaired.

Incident 5: Policy Change Requiring Historical Replay

Content policy was updated to more strictly reject non-photographic content: screenshots, banners, infographics, social cards, slides, illustrations, and mockups. Historical images scored under the old policy needed to be re-evaluated.

TBIE framing: policy change = controlled replay, not a patching exercise.

Resolution:

Affected rows were identified by policy version marker in DynamoDB.
Each affected row was reset to a pre-VLM boundary: stale score fields cleared, VLM status reset to pending.
Fresh Intent was emitted into SQS so GPU workers re-scored under the new policy.
This is one of the most powerful properties of a replay-native system: policy evolution does not require one-off correction scripts. It becomes a controlled reprocessing event with clear before-and-after semantics. The same replay machinery used for failure recovery also handles deliberate re-evaluation.

Incident 6: Execution Drift Below the Application Layer

Two distinct incidents exposed failures where application code was correct but the execution environment was stale.

6a: Stale Launch Template (Image Tag)

A launch template continued to reference an old image tag that no longer existed. Cloud-init failed when pulling the nonexistent container image, and the GPU instance came up without ever starting the worker. The worker log group was empty — the application layer never ran.
Complete silence in the GPU log group is a critical signal. It usually means execution failed before the application layer started: image pull failure, IAM denial during bootstrap, or cloud-init crash.

6b: Incomplete AMI (Missing Model Artifacts)

A separate incident involved an AMI that was missing the Python files required by Qwen3-8B-VLM (a model that relies on remote code). The container started successfully, but the model loader crashed because local artifacts were incomplete.

Resolution — both incidents:

• Launch templates now include version assertions checked at deployment time.
  AMI build pipelines validate the presence and integrity of all model artifacts before publishing.

• Container version and model artifact version are treated as a compatibility pair, validated together.

• Deployment tooling refreshes launch templates automatically when image tags change.

Incident 7: CORS and Browser Upload Split-State Failure

The backend could write a pending upload record and generate a presigned URL, but the browser’s PUT could fail because of CORS misconfiguration. In that case, Truth at the API layer looked correct while Execution at the browser layer never completed.
Investigation revealed multiple independent configuration failures could co-exist: a bucket rejecting wildcard origins under restricted public bucket settings, while the client simultaneously used a global S3 endpoint that returned redirects that browsers would not follow on OPTIONS preflight.

TBIE diagnosis:

Intent existed (presigned URL generated, pending row written). Execution was blocked by infrastructure configuration drift. Two independent misconfiguration layers were both wrong simultaneously.

Resolution:

• CORS policies are version-controlled and validated in CI before deployment.

• Bucket CORS and public access settings are treated as a matched pair, not configured independently.

• Upload path includes client-side detection of presigned URL PUT failure with structured error reporting.

• Pending rows created without a completed upload are detected by the replay Lambda and age-gated before re-emission.

Incident 8: Control Plane Blindness (Replay Lambda IAM Failure)

The replay Lambda lacked permission to read SQS queue attributes. The function continued to be scheduled and invoked, but it crashed before it could inspect Belief (queue depth) or emit Intent. From the outside, everything appeared enabled. In reality, the control plane had gone blind.

This is the most dangerous failure mode in the system. A silently failing control plane means every other safety net weakens simultaneously. Worker failures that should self-repair accumulate without rescue. Operators see green infrastructure dashboards while reconciliation has effectively stopped.

Resolution:

• The replay Lambda’s IAM policy now explicitly grants sqs:GetQueueAttributes, sqs:GetQueueUrl, and sqs:SendMessage.

• IAM policy drift is checked in deployment assertions — the Lambda will not deploy if required permissions are absent.

• The Lambda now emits a structured heartbeat metric on every successful invocation, including after inspecting queue state.

• Alarm on absence of heartbeat metric: silence means the control plane has likely failed.

The operational philosophy shift from this incident: worker failures are recoverable noise. Control-plane blindness is a systemic risk. Observability on the control plane must be stronger than on individual workers.

Incident 9: Dynamic Tenant Discovery and Queue Retention

As the platform grew, static tenant lists became a bottleneck. Reconciliation could not depend on manually editing infrastructure configuration every time a new tenant was onboarded. Additionally, when the GPU fleet was scaled to zero for cost savings, queue retention was set too short — messages expired before the fleet woke up.

Resolution:

The control plane moved to dynamic tenant discovery from DynamoDB. Tenant registry markers are written by the upload path and discovered by the replay Lambda at runtime.
Queue retention policy was extended to match the maximum expected outage window, not an idealised constant.
New tenant onboarding now automatically creates the required DynamoDB registry marker without manual infrastructure changes.

OPERATIONAL MODEL AND RUNBOOK THINKING

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. The starting question is no longer ‘is the queue healthy?’ or ‘are GPU workers running?’ — those are Belief signals. Every incident investigation now begins with a TBIE classification:

THE DIAGNOSTIC SEQUENCE

• Step 1: Check Truth

Is DynamoDB accumulating rows in pending_vlm_enrichment? If so, the platform believes unfinished work exists, regardless of what the queue shows. A growing Truth backlog is the authoritative signal that the system is behind.

• Step 2: Check Intent

Does SQS contain replayable work items? High Truth backlog + near-zero queue depth = reconciliation failure, not execution failure. The replay Lambda itself becomes suspect. Either it stopped emitting Intent, or it lost the ability to discover the work.

• Step 3: Check Execution

If Intent exists but the backlog still does not move, shift to Execution health. GPU worker logs become critical. Check for:

1. Deterministic S3 download errors (wrong bucket routing — see Incident 4).
   Repeated OOM crashes (model too large for instance type).

2. Complete log group silence (execution failed before the application layer — check cloud-init).

3. Stale launch template (image tag no longer exists — check ASG launch configuration).

• Step 4: Check the Control Plane

If queue visibility is unchanged while pending Truth continues growing, the replay Lambda itself is the focus. Check: Is it being invoked? Is it completing successfully? Can it read sqs:GetQueueAttributes? Is it emitting its heartbeat metric?

Operational summary of diagnostic signals:

• Truth backlog growing → unfinished work exists

• Intent near zero despite Truth backlog → reconciliation failure

• Intent present but Execution failing → runtime or routing failure

• GPU log silence → bootstrap failure (cloud-init, image pull, IAM at boot)

• Replay Lambda silence → control-plane failure (IAM, crash, permission gap)

• No heartbeat metric → control plane has gone blind

INCIDENT RESPONSE DECISION TREE

At 02:00 AM, when the queue looks empty but customers are waiting:

1. Check Truth: pull DynamoDB pending count. If non-zero, the system is behind regardless of queue state.

2. Check replay Lambda logs: has it run in the last cycle? Did it complete? Did it emit messages?

3. Check SQS: are messages present? If Truth is high and SQS is low, replay Lambda is suspect.

4. Check ASG desired capacity: did the Lambda set it correctly? Check CloudWatch ASG activity log.

5. Check GPU worker logs: any entries? Silence means boot failure. Errors usually indicate routing or model issues.

6. Check IAM: can the Lambda call sqs:GetQueueAttributes? Can the worker read the correct S3 bucket?

7. Check launch template: does the image tag still exist? Pull the AMI manifest.

8. OUTCOMES
   The migration from SageMaker to native AWS primitives, combined with TBIE reconciliation, changed both the cost profile and the operational character of the pipeline.

OPERATIONAL IMPROVEMENTS

• Latency: Startup latency: near-zero or sub-minute (down from 20–40 minutes cold start on SageMaker).

• Parallelism: Workers process images in parallel rather than sequentially — throughput scales with fleet size.

• Throughput: Wall-clock time reduced dramatically from the ~11 hour nightly baseline.

• Replay: Event-driven replay handles failures automatically without manual intervention for most failure classes.

• Cost: Scale-to-zero when no work is present; burst capacity available within seconds via warm pools.

ARCHITECTURAL IMPROVEMENTS

The system can repair its own state: Truth without Intent is detected and corrected by the control plane.
Policy changes are handled as replay events — no custom migration scripts needed.
Tenant onboarding is dynamic — no infrastructure changes required per tenant.
Execution environments are immutable and reproducible via pre-baked AMIs.
Control plane observability now triggers independent alerting paths from worker failures.

THE BROADER LESSON

Not every AI workload needs a managed ML platform. Sometimes the real engineering challenge sits in orchestration, reconciliation, and operational resilience — not in model serving. When the problem is ‘how do we run a model?’, SageMaker is the right answer. When the problem is ‘how do we orchestrate burst-scale inference reliably, replay safely, and cost efficiently?’, native primitives and a clear reconciliation model are the right answer.

GUARDRAILS, MAINTENANCE, AND OPERATIONAL CHECKLIST

Parjanya’s reliability is not a property of its initial design. It is the accumulated result of incidents converted into permanent controls. The following checklist encodes what the system has already learned.

INTENT PRESERVATION

• Workers never delete SQS messages unless the outcome is terminal_failure or completed.

• Visibility timeout exceeds maximum realistic inference time.

• Queue retention exceeds maximum expected outage window.

• Dead-letter queue is monitored and reviewed regularly.

EXECUTION ENVIRONMENT INTEGRITY

Launch template image tags are validated against existing AMIs at deployment time.

AMI build pipelines assert presence of all model artifacts before publishing.

Container version and model artifact version are validated as a compatibility pair.

Launch templates are refreshed automatically when image tags change.

Python lockfiles are used to prevent dependency drift inside worker environments.

CONTROL PLANE HEALTH

Replay Lambda emits a structured heartbeat metric on every successful execution.

Alarm fires if heartbeat is absent for more than one poll cycle.

IAM policy grants for the Lambda are validated in deployment assertions.

Lambda error rate and duration are independently alarmed.

Control-plane CloudWatch logs are retained and searchable for incident forensics.

RECONCILIATION INTEGRITY

Truth backlog (pending DynamoDB rows) is monitored as a primary SLI.
Intent gap (pending Truth with no corresponding queue message) is checked by the Lambda on each invocation.

Replay logic resets Truth cleanly to a pre-execution boundary before re-emitting Intent.

Tenant discovery is dynamic — no manual configuration changes needed on onboarding.

Queue retention policy change requires explicit engineering review.

INFRASTRUCTURE DRIFT PREVENTION

CORS policies are version-controlled and validated in CI.

Bucket CORS and public access settings are configured as a matched pair.

API schema and frontend types have drift detection in CI.

S3 bucket routing is validated at Intent construction time, not only at worker download time.

POLICY AND REGRADE

Content policy version is stored per image in DynamoDB.

Policy changes trigger replay jobs for images scored under previous policy versions.

Regrade jobs use the same replay machinery as failure recovery — no custom tooling.

IMPLICATIONS FOR AI PLATFORM DESIGN

TBIE generalises beyond Parjanya. Any AI platform that spans storage, metadata, asynchronous work, GPU execution, and policy evolution benefits from the same model. The key is to stop thinking of the system as a one-way pipeline and treat it as a reconciliation loop that continuously restores alignment between durable truth and intended work.

That shift changes concrete architecture choices:

Queues: Queues become journals of intent rather than temporary transport.

Replay: Replay becomes a first-class product capability, not an ops workaround.

Autoscaling: Autoscaling becomes a consequence of work existing, not a heuristic guess.

Model packaging: Model packaging becomes part of execution correctness, not just deployment.

Browser uploads: Browser upload behaviour becomes part of the end-to-end reliability model.

Policy change: Policy change becomes a replay event rather than a manual repair task.

The right mental model for modern AI infrastructure is not ‘build a pipeline.’ It is ‘build a system that can continuously reconcile truth and intent until intended work is actually realised.’

APPENDIX A: COMPONENT MAPPING TABLE

APPENDIX B: KEY TERMS

The framework emerged from a simple observation:

Distributed systems rarely fail through catastrophic outages.

More often, they fail through divergence.

Truth says one thing.
Execution does another.
Intent disappears.
Belief becomes detached from reality.

TBIE gave me a vocabulary for understanding those failures.

What I did not fully appreciate at the time was the architectural consequence of taking TBIE seriously.

I thought I was building an AI pipeline.

I ended up building a platform.

Its primary responsibility was preserving correctness.

That realization fundamentally changed how I thought about reliability.

The hard problems in Parjanya v2.0 were not model problems.

They were coordination problems.

• Not GPU problems.

• Not inference problems.

• Not even scaling problems.

The hardest problems emerged whenever the system lost alignment between:

• what was true,

• what work should exist,

• what execution was currently happening,

• and what the system believed was happening.

The original architecture looked deceptively simple.

Images landed in S3.

Metadata was written into DynamoDB.

Validation ran through Lambda functions.

GPU workers performed VLM enrichment and IQA processing.

Results were written back into DynamoDB.

Autoscaling reacted to queue depth.

At first glance, it looked like a fairly conventional event-driven AI pipeline.

Production quickly proved otherwise.

Workers crashed.

Queues drained without progress.

Pending images remained pending forever.

Retries became impossible.

Infrastructure drifted away from declared state.

The system could remain technically healthy while doing absolutely nothing useful.

That was the moment I realised we were solving the wrong problem.

The challenge was not moving images through a pipeline.

The challenge was continuously reconciling distributed state.

Once viewed through that lens, the architecture began evolving in a very different direction.

The replay engine stopped being a utility.

It became the control plane.

Queues stopped being transport mechanisms.

They became durable Intent.

Autoscaling stopped being a scaling feature.

It became a realization layer for Intent.

And the system itself stopped behaving like a pipeline.

It started behaving like a platform.

This article is the story of that evolution.

Preface

Distributed systems rarely fail in the way architecture diagrams suggest.

The boxes stay alive. The queues still exist. The databases continue accepting writes. The autoscaling groups look healthy. The dashboards even remain green.

And yet the system silently stops doing useful work.

An image sits in a pending state forever. A queue drains but no GPU ever runs. A worker crashes and deletes the only replayable work item. A presigned upload URL looks valid but browser execution never begins. A GPU instance launches correctly while the actual container inside it continuously crashloops.

Most production failures are not total outages. They are reconciliation failures.

Something the system believes should happen never actually happens. Or something already completed continues to look pending. Or a control plane silently loses the ability to coordinate execution.

Parjanya v2.0 forced us to confront this repeatedly.

Initially, the architecture looked straightforward:

• Images upload into S3

• Metadata lands in DynamoDB

• Lambdas perform validation and enrichment

• GPU workers run VLM and IQA workloads

• Results are written back into DynamoDB

• Autoscaling follows queue depth

At first glance, this looked like a normal AI pipeline.

In practice, it evolved into something else.

The deeper challenge was not inference. The challenge was maintaining alignment between:

the durable truth of the system,
the operational understanding of the system,
the work the system intended to perform,
and the execution layer actually performing it.

Truth → Belief → Intent → Execution

TBIE was never intended as branding. It emerged as an operational language for reasoning about reliability.

The model eventually shaped:

the event-driven reconciliation architecture,
the autoscaling strategy,
the replay model,
the control plane,
failure recovery,
multi-tenant orchestration,
policy regrading,
infrastructure debugging,
and eventually the broader direction of Parjanya itself.

This article is a deep dive into that journey.

It is not a summary. It is a detailed architectural walkthrough of:

how TBIE emerged,
why event-driven reconciliation mattered,
what failed in production,
how those failures were diagnosed,
and why reconciliation patterns matter far more than most AI infrastructure discussions acknowledge.

The goal is not merely to explain Parjanya.

The goal is to explain how resilient AI systems are actually built.

1. The Real Problem in Distributed AI Systems

Most modern AI infrastructure discussions focus heavily on models.

Which VLM?
Which embedding strategy?
Which vector database?
Which GPU?
Which inference framework?
Which quantization?

These questions matter.

But they are not usually what breaks systems.

The real production problems tend to look different:

Work disappears.
Retries become impossible.
Workers process stale state.
Uploads partially succeed.
Queues drain without progress.
Control planes silently stop coordinating.
Autoscaling behaves correctly against the wrong metric.
Infrastructure layers drift away from declared source-of-truth.

In other words:

The problem is not merely computation. The problem is reconciliation.

Distributed systems are fundamentally coordination systems.

A resilient architecture needs to answer:

What is the durable truth?
What work should exist?
What execution is currently happening?
Can the system reconstruct work after failures?
Can the system distinguish transient failure from terminal failure?
Can infrastructure drift be detected?
Can the system replay work safely?
Can policy changes be re-applied deterministically?

Most architectures answer these implicitly.

TBIE attempts to answer them explicitly.

2. TBIE — Truth, Belief, Intent, Execution

The TBIE model separates distributed systems into four operational layers.

The model becomes powerful because it forces explicit boundaries.

Truth

Truth represents the authoritative record.

Truth must be:

durable,
reconstructable,
queryable,
replayable,
and independent of transient execution state.

In Parjanya:

S3 object bytes are Truth.
DynamoDB workflow fields are Truth.
Pipeline status transitions are Truth.
Curation outputs are Truth.
Hashes and metadata are Truth.

Truth is what remains after workers crash.

Truth is what survives retries.

Truth is what operators use to reconstruct system state.

Belief

Belief is operational understanding.

Belief is not always perfectly accurate.

Examples:

• Queue depth

• Autoscaling desired capacity

• In-flight SQS messages

• Worker health

• Current GPU availability

• Visibility timeout state

Belief may lag reality. Belief may be partial. Belief may even temporarily be wrong.

But Belief matters because orchestration depends on it.

Importantly:

Belief should be derivable from Truth + Intent.

That distinction becomes critical later.

Intent

Intent represents durable declarations of work.

Intent answers:

“What should the system attempt next?”

In Parjanya:

SQS messages are Intent.
Replay queues are Intent.
Regrade operations are Intent.
Retry workflows are Intent.

Intent is one of the most important concepts in the entire architecture.

Execution should never destroy Intent casually.

Why?

Because Intent is replayability.

Without replayable Intent:

transient failures become permanent,
retries become manual,
reconciliation becomes impossible,
and the system loses self-healing capability.

Execution

Execution is the stateless machinery that realizes work.

Execution includes:

Lambdas,
GPU workers,
autoscaling groups,
validation services,
EXIF processors,
and inference pipelines.

Execution should ideally be:

disposable,
reproducible,
restartable,
and stateless.

If an execution worker disappears, the system should still be able to recover from Truth + Intent.

That principle drove many later architectural decisions.

3. Mapping TBIE onto Parjanya v2.0

Once TBIE became the operational lens, the architecture became easier to reason about.

Truth Layer

Truth in Parjanya consisted primarily of:

S3

• original uploads,

• RAW image payloads,

• generated previews,

• derived thumbnails,

• metadata artifacts.

DynamoDB

Workflow state:

• batch_status

• pipeline_status

• curation

• quality scores

• VLM outputs

• EXIF metadata

• technical validation

• image hashes

• routing information

Truth represented the durable state of every image.

Intent Layer

Intent existed primarily in SQS.

Examples:

• ml-large-jobs

• replay queues

• regrade operations

• retry workflows

Each Intent payload encoded:

• bucket,

• object key,

• tenant,

• image id,

• execution metadata.

Intent was intentionally replayable.

That design decision later prevented multiple catastrophic replay-loss scenarios.

Belief Layer

Belief included:

• ASG desired capacity,

• queue visibility counts,

• in-flight message counts,

• CloudWatch metrics,

• worker availability,

• queue backlog,

• GPU scale state.

Belief drove orchestration.

For example:

• queue depth controlled autoscaling,

• replay Lambdas checked queue visibility,

• scaling decisions relied on approximate message counts.

Belief was operationally useful but never fully authoritative.

Execution Layer

Execution included:

Lambda Workers

• EXIF extraction

• technical validation

• replay orchestration

• autoscaling control-plane logic

GPU Workers

• Qwen VLM inference

• IQA processing

• curation scoring

• natural-language enrichment

Execution workers remained stateless wherever possible.

That allowed replay to remain safe.

4. Why Reconciliation Matters

A distributed system does not stay healthy merely because components exist.

Health depends on reconciliation.

Consider this state:

DynamoDB row says pending_vlm_enrichment
SQS queue is empty
GPU ASG is scaled to zero

Nothing is technically “down.”

And yet the system is stuck.

Truth says work should exist. Intent does not exist. Execution never begins. Belief incorrectly suggests the system is idle.

This is a reconciliation failure.

TBIE made this visible.

The system needed a mechanism that continuously aligned:

Truth ↔ Intent ↔ Execution

That eventually led to the event-driven reconciliation architecture.

5. From Polling to Event-Driven Reconciliation

The original architecture used scheduled polling.

Every few minutes:

EventBridge triggered a Lambda,
the Lambda scanned DynamoDB,
pending work was discovered,
SQS messages were emitted,
GPU autoscaling was triggered.

This worked.

But it introduced fundamental limitations.

Problems with Polling

Latency

An image could wait:

0–15 minutes before the next poll cycle.

GPU cold starts already consumed several minutes. Adding polling latency made the UX significantly worse.

Waste

Most polling invocations found no work.

The control plane continuously woke up merely to rediscover emptiness.

Operational Ambiguity

Polling blurred the distinction between:

“there is no work,”
and
“the system has not discovered the work yet.”

That ambiguity complicated debugging.

Poor Reactive Scaling

Autoscaling became tied to polling frequency.

Queue discovery latency directly affected GPU startup latency.

6. DynamoDB Streams Changed the Architecture

The major architectural shift came from moving reconciliation into an event-driven model.

Instead of periodically checking for state changes:

The system reacted immediately when Truth changed.

The New Model

DynamoDB row transitions to pending_vlm_enrichment
DynamoDB Streams emits an event
Event source mapping filters relevant writes
Replay Lambda triggers immediately
SQS Intent is emitted
GPU ASG is kicked from zero
GPU workers process work
Results are written back into Truth

The architecture became reactive.

7. Anatomy of the Event-Driven TBIE

Architecture

DynamoDB Streams

The table emits:

• INSERT events

• MODIFY events

Filtered specifically for:

batch_status = pending_vlm_enrichment

This is extremely important.

The system does not react to every write. It reacts only when Truth crosses a workflow boundary requiring Intent creation.

Event Source Mapping

The stream mapping aggregates:

up to 50 records,
or up to 5 seconds.

This balances:

latency,
throughput,
Lambda cost,
queue pressure.

The system effectively performs micro-batching naturally.

Replay Lambda

The replay Lambda became the core reconciliation engine.

Responsibilities:

• inspect stream events,

• discover pending rows,

• identify retriable failures,

• generate replayable Intent,

• avoid duplicate queueing,

• trigger autoscaling.

This Lambda evolved into the operational heart of the platform.

Queue Backlog Guard

One subtle but extremely important design decision:

The replay Lambda checks queue visibility before re-enqueueing pending work.

Why?

Because replay itself can become destructive if uncontrolled.

Without a backlog guard:

the control plane could continuously duplicate Intent,
queue pressure would inflate,
workers would process stale state,
autoscaling would overreact.

This is a classic reconciliation trap.

The backlog guard prevented it.

GPU ASG Kick

The replay Lambda also became responsible for scaling GPU workers.

This matters because scale-to-zero architectures create a hidden problem:

The queue may contain work while compute remains asleep.

The replay Lambda solved this by:

emitting Intent,
then immediately setting desired ASG capacity.

That turned the replay engine into both:

a reconciliation layer,
and a control-plane orchestrator.

8. Latency Transformation

The shift from polling to event-driven reconciliation fundamentally changed responsiveness.

Before

Upload → wait for poll → queue → ASG scale → GPU boot

Total latency:

~3–20 minutes

depending on polling window timing.

After

Upload → stream event → immediate reconciliation → queue → ASG scale

Total latency:

~4–6 minutes

with remaining delay dominated almost entirely by:

EC2 startup,
model load,
GPU warm-up.

This is a very different operational profile.

The system stopped feeling batch-oriented.

It began behaving like a reactive platform.

9. The First Major TBIE Failure

Intent Destroyed Too Early

One of the earliest major incidents exposed why TBIE mattered.

Symptom

GPU workers experienced transient:

• S3 download failures,

• temporary consistency races,

• OOM conditions.

But the worker still deleted the SQS message.

At the same time:

DynamoDB was updated with:

vlm_failed

The result:

Intent disappeared,
Truth claimed terminal failure,
automatic replay became impossible.

This was a severe reconciliation violation.

The Core TBIE Violation

Execution destroyed Intent before retries were exhausted.

That single decision broke replayability.

TBIE clarified the failure immediately.

The problem was not merely worker instability.

The problem was:

Execution invalidated Intent prematurely.

The Fix

The worker execution model changed.

Instead of binary success/failure, workers returned:

retry

• do NOT delete SQS message

• do NOT write terminal Truth

terminal_failure

• write durable failure state

• delete Intent

completed

• write success state

• delete Intent

This restored replay semantics.

Why This Matters Beyond Parjanya

Many distributed systems accidentally conflate:

“execution failed once” with
“execution is permanently impossible.”

Those are not equivalent.

TBIE forced the architecture to encode that distinction explicitly.

10. Truth Pending, Intent Missing

Another class of failures appeared repeatedly.

Symptom

Rows remained:

pending_vlm_enrichment

But:

• queues were empty,

• GPU workers never started,

• no replay existed.

The system appeared healthy.

But nothing moved.

What TBIE Revealed

Truth and Intent were decoupled.

The architecture had assumed:

“If Truth says pending, Intent must already exist.”

That assumption was false.

Reconciliation cannot be assumed. It must be continuously enforced.

The Replay Architecture

This led to:

• replay scripts,

• scheduled replay Lambdas,

• retriable failure reset workflows,

• queue reconstruction logic.

The replay Lambda evolved into a continuously running reconciler.

This was the moment the architecture stopped being merely a pipeline.

It became a platform.

11. Wrong Bucket, Right Key

A Perfect Distributed Systems Failure

One of the most revealing incidents involved bucket mismatch.

Symptom

GPU workers repeatedly failed with:

s3_download_error

Yet:

• queue depth was growing,

• autoscaling worked,

• workers were alive,

• previews existed.

Everything looked operational.

Root Cause

The SQS payload contained:

the correct object key,
but the wrong bucket.

The preview existed.

Just not in the referenced bucket.

Why This Was Architecturally Important

The system was reliably executing the wrong instruction.

This is one of the hardest classes of distributed failures.

Nothing is technically broken.

The system is functioning exactly as instructed.

The instruction itself is invalid.

TBIE framed this correctly:

Intent must not merely exist. Intent must be correct and replayable.

The Fix

The architecture introduced:

• bucket source-of-truth rules,

• preview-aware routing,

• enqueue eligibility tightening,

• RAW-aware preview validation,

• replay resets for retriable failures.

The fix was operationally deeper than changing one field.

It hardened the meaning of Intent itself.

12. CORS Failures and Split Truth

One of the most subtle incidents occurred at the ingestion boundary.

Symptom

Frontend uploads failed with browser CORS errors.

But backend APIs:

• generated presigned URLs,

• wrote pending upload records,

• and returned success responses.

Truth looked correct.

Execution never happened.

What Actually Failed

The browser failed during:

OPTIONS preflight.

Execution never reached:

PUT upload.

The TBIE Interpretation

Truth existed. Intent existed. Execution failed before realization.

This became an important diagnostic heuristic:

When work is stuck:

Ask whether:

• Truth is missing,

• Intent is missing,

• or Execution cannot realize Intent.

That framing simplified debugging enormously.

13. Event-Driven Architecture Exposes Infrastructure Drift Faster

As the architecture became more reactive, infrastructure drift became easier to detect.

This was unexpected.

Example: Stale Worker Image Tag

The ASG launch template still referenced:

:v20

while the actual system had moved to:

:v23

The ASG launched correctly. Cloud-init executed. Docker attempted to pull. The image no longer existed.

Workers never started.

The Deeper Problem

Terraform ignored user_data changes.

This created silent divergence between:

declared source-of-truth,
and actual runtime execution.

TBIE exposed this as an Execution drift problem.

Truth, Belief, and Intent were all correct.

Execution itself had drifted away from reality.

The Architectural Lesson

Infrastructure reproducibility is not optional.

Execution environments must remain tied to durable source-of-truth.

Otherwise replay becomes dangerous because:

the same Intent may produce different outcomes over time.

14. AMI Drift and Runtime Incompatibility

Another failure emerged at the AMI layer.

Symptom

Containers started. But model loading crashed.

The model weights existed. The processors did not.

The AMI bake omitted Python files required by:

trust_remote_code=True

The worker entered a crashloop.

Why This Was Important

Again:

Truth was correct. Intent was correct. Belief suggested workers existed.

Execution failed due to stale artifacts.

This is precisely why replayable systems require reproducible execution environments.

TBIE repeatedly reinforced this architectural truth.

15. The Replay Lambda Became the Control Plane

Originally, the replay Lambda was operational glue.

Eventually, it became the platform control plane.

Responsibilities expanded to include:

• pending reconciliation,

• retriable failure resets,

• tenant discovery,

• queue backlog management,

• autoscaling coordination,

• replay orchestration,

• dynamic routing.

The replay layer became central infrastructure.

This was a critical architectural evolution.

16. The IAM Incident

When the Control Plane Goes Blind

One of the most dangerous incidents involved a missing IAM permission.

Symptom

The replay Lambda continuously crashed on:

sqs:GetQueueAttributes

for nearly 12 hours.

But:

• EventBridge still invoked it,

• the schedule looked healthy,

• the infrastructure appeared alive.

Meanwhile:

• pending rows accumulated,

• queues stopped reconciling,

• autoscaling logic failed.

The TBIE Lesson

The control plane had lost Belief.

Without Belief:

the replay engine could not safely emit Intent,
queue guards could not function,
orchestration collapsed.

The system looked operational while the reconciler was effectively dead.

The Operational Insight

A scheduled Lambda is not necessarily a healthy Lambda.

Control-plane observability matters more than worker observability.

Why?

Because worker failures are recoverable.

Control-plane blindness disables recovery itself.

17. Dynamic Tenant Discovery

As Parjanya moved toward beta onboarding, static tenant configuration became a bottleneck.

Originally:

The replay Lambda used static tenant lists.

Every new tenant required:

• Terraform changes,

• environment updates,

• redeploys.

This violated the self-service model.

The Evolution

The architecture introduced:

• tenant registry writes during upload completion,

• runtime tenant discovery,

• dynamic reconciliation.

The control plane became data-driven rather than configuration-driven.

This is a major architectural distinction.

Reactive systems scale operationally only when discovery becomes dynamic.

18. Regrading and Policy Replay

One of the most powerful consequences of TBIE was deterministic replay.

This became especially valuable during policy changes.

Example

The IQA policy evolved to reject:

• screenshots,

• banners,

• posters,

• slides,

• social cards,

• infographics,

• illustrations.

The architecture needed to:

• reset VLM-derived Truth,

• regenerate Intent,

• rerun GPU enrichment,

• rewrite curation outputs.

This was not merely a retry.

It was a deterministic policy replay.

Why This Matters

Most ML systems struggle with:

“How do we reprocess historical data safely?”

TBIE already contained the answer.

Truth could be reset to a replay boundary. Intent could be regenerated. Execution remained stateless.

The system became replay-native.

This is a profound operational capability.

19. Queue Retention and Weekend Gaps

Another subtle operational issue:

Scale-to-zero architectures create temporal gaps.

Scenario

Users upload images Friday evening. GPU workers remain offline over the weekend.

If queue retention is too short:

Intent disappears before execution resumes.

Truth remains pending.

Intent vanishes.

This is catastrophic reconciliation drift.

The Fix

Queue retention was extended.

This ensured Intent outlived temporary execution dormancy.

An important TBIE lesson emerged:

Intent durability must exceed execution availability windows.

20. Why Event-Driven Reconciliation Matters

Beyond AI

Although Parjanya is an AI platform, the architectural lessons generalize broadly.

The same TBIE principles apply to:

distributed media systems,
payment processing,
ETL pipelines,
logistics systems,
asynchronous workflows,
microservice orchestration,
large-scale ingestion platforms.

The core challenge is universal:

How do you maintain alignment between:

durable truth,
intended work,
operational state,
and actual execution?

TBIE provides a vocabulary for reasoning about that alignment.

21. The Shift from Pipeline to Platform

This was perhaps the most important realization.

A pipeline moves data.

A platform:

• survives retries,

• supports replay,

• reconciles state continuously,

• tolerates execution drift,

• evolves policy safely,

• supports operational introspection,

• and remains debuggable under failure.

The event-driven TBIE architecture pushed Parjanya decisively toward the latter.

The replay Lambda stopped being a utility.

It became the reconciliation brain of the system.

The architecture stopped being:

“process uploaded images.”

It became:

“maintain durable correctness while asynchronous distributed execution continuously changes.”

That is a very different engineering problem.

22. The Operator’s Mental Model

TBIE becomes most useful when it changes how incidents are diagnosed.

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. Operators no longer began by asking whether “the queue is healthy” or whether “the GPU workers are running.” Those signals still mattered, but they were no longer treated as authoritative.

Instead, every investigation began with a much more precise question:

Is the system waiting because Intent is missing, or is Intent present but Execution cannot realize it?

That distinction dramatically narrowed the search space during incidents.

If Truth showed a growing backlog while Intent remained near zero, the issue was usually in reconciliation itself. Replay Lambda failure, queue emission failure, stale tenant discovery, or a blind control plane could all create this pattern. The queue might look empty and the autoscaling group might remain at zero, yet the real failure would not be in the workers. It would be in the system’s ability to regenerate Intent from Truth.

If Intent existed but Execution failed repeatedly, the investigation shifted toward realization problems instead: bucket routing mismatches, stale AMIs, launch-template drift, IAM gaps, runtime incompatibilities, model-loading failures, endpoint routing problems, or browser-side execution failures.

That diagnostic framing changed operations significantly. Incidents became easier to classify because TBIE separated the architecture into meaningful operational boundaries. Instead of treating the platform as one opaque pipeline, operators could isolate whether the failure belonged to Truth, Belief, Intent, or Execution.

The model also made one uncomfortable reality obvious: dashboards alone are not enough. A queue can look healthy while Truth continues accumulating backlog. Autoscaling can look correct while workers continuously crashloop. A scheduled Lambda can appear operational while silently failing every invocation because of an IAM permission gap.

TBIE did not eliminate debugging complexity. What it did was make debugging systematic.

23. When the System Hangs at 02:00 AM

Architectural diagrams are useful during design reviews.

Runbooks matter during incidents.

One of the most important evolutions in Parjanya v2.0 was realizing that reconciliation needed to exist not only in architecture, but also in operational practice. TBIE only became truly useful once there was a repeatable process for recovering the system under pressure.

The first step during any “hung VLM” investigation became identifying the TBIE boundary itself.

Truth consisted of the DynamoDB image rows and their workflow state. Intent consisted of SQS messages representing GPU work. Execution consisted of GPU workers consuming that Intent and writing enriched Truth back into DynamoDB. Belief consisted of queue depth, autoscaling state, visibility counts, worker health, and operational telemetry.

Once those boundaries were established, incident triage became much more structured.

Operators first checked Truth backlog. Were rows accumulating in pending_vlm_enrichment? If yes, the system still believed work should happen.

Next came Intent backlog. Did SQS contain replayable work items? If Truth backlog was high while Intent remained near zero, then reconciliation itself had failed.

Then came Execution health. Were GPU workers alive? Were containers actually starting? Were model loads succeeding? Were workers consuming messages or continuously failing before acknowledgement?

Several operational shortcuts emerged over time.

If the GPU log group was empty while ASG instances existed, the problem was often below the application layer entirely. Cloud-init failures, stale image tags, or broken launch templates frequently surfaced first through EC2 console output rather than through CloudWatch worker logs.

If the replay Lambda appeared scheduled but no queue activity existed, IAM permissions became the first thing to inspect. One of the most dangerous incidents in the system occurred precisely because the reconciliation Lambda could no longer read queue attributes, effectively blinding the control plane while the infrastructure superficially appeared healthy.

If rows repeatedly failed with S3 download errors while workers remained alive, the issue was often deterministic addressing rather than transient instability. Wrong-bucket references, preview routing mismatches, or endpoint configuration drift could all create workers that failed consistently while still appearing operational.

Over time, this changed the operational mindset of the platform. The goal stopped being “restart the workers and hope.” The goal became understanding exactly where reconciliation had broken down and then repairing the appropriate boundary.

That distinction is subtle, but extremely important.

A resilient platform is not the one that avoids incidents. It is the one that can recover from them methodically.

24. Guardrails and Proactive Controls

One of the clearest architectural patterns that emerged from Parjanya was that every major incident eventually needed to produce a permanent guardrail.

Without that discipline, incidents remain stories. With it, incidents become operational evolution.

The architecture gradually accumulated a large number of proactive controls:

Transient worker failures no longer delete Intent prematurely. Replay Lambdas continuously reconcile pending Truth. Reproducible lockfiles prevent dependency drift. Terraform validations reject invalid wildcard CORS patterns. Tenant discovery became dynamic rather than operator-maintained. Queue retention was extended to match real outage windows rather than idealized assumptions.

Many of these controls were not introduced because the architecture diagram required them. They emerged because production failures revealed where reconciliation could silently drift.

This is one of the strongest long-term lessons from the TBIE model.

Resilience is not achieved through one large design decision. It is accumulated gradually through operational guardrails.

The mature version of the system looked very different from the original one not because the core architecture changed dramatically, but because every important failure eventually became:

a validation rule,
a replay strategy,
a Terraform constraint,
a CI assertion,
a queue safeguard,
or an operational diagnostic.

That is how resilient systems evolve in practice.

They convert incidents into institutional memory.

25. Maintenance as a Reliability Discipline

One of the easiest mistakes in distributed systems is assuming resilience is something implemented once.

In reality, resilience is maintained continuously.

By the later stages of Parjanya v2.0, the team had accumulated a growing operational checklist precisely because many failures were caused not by dramatic outages, but by small forms of drift.

Replay logic and worker behavior needed to evolve together. GPU image tags needed to stay aligned with launch templates. Lockfiles had to remain committed when dependencies changed. Tenant buckets required CORS validation before onboarding. Queue retention needed to reflect actual business downtime windows. Backend API contracts and frontend types had to remain synchronized.

None of these tasks sound individually sophisticated.

But distributed systems rarely collapse from a single catastrophic event. They erode through gradual inconsistency.

That realization changed how the platform was maintained.

Operational discipline became part of the architecture itself.

The maintenance checklist was therefore not merely documentation. It was a recognition that reconciliation must continue throughout the entire lifecycle of the platform.

26. Why TBIE Matters Beyond AI

Although Parjanya is an AI platform, the architectural lessons generalize broadly.

The same TBIE principles apply to:

distributed media systems,
payment processing,
ETL pipelines,
logistics systems,
asynchronous workflows,
microservice orchestration,
large-scale ingestion platforms.

The core challenge is universal:

How do you maintain alignment between:

durable truth,
intended work,
operational state,
and actual execution?

TBIE provides a vocabulary for reasoning about that alignment.

27. The Shift from Pipeline to Platform

1. Truth must remain durable and replayable

If Truth cannot reconstruct the system after failures, the architecture becomes operationally fragile.

2. Intent is more important than most systems realize

Intent is replayability.

Destroying Intent prematurely destroys recovery.

3. Event-driven systems expose hidden drift faster

Reactive architectures surface infrastructure inconsistencies earlier.

This is painful initially.

But healthier long-term.

4. Execution must remain reproducible

Replay without reproducibility becomes dangerous.

5. Control-plane observability matters more than worker observability

Worker crashes are recoverable. Blind reconcilers are not.

6. Queue depth alone is not truth

Belief metrics require interpretation.

Approximate counts are operational signals, not durable truth.

7. Policy replay should be designed intentionally

Eventually every ML platform needs deterministic reprocessing.

Replay-native architectures handle this gracefully.

8. Distributed systems fail by divergence

Not by total outage.

The hardest bugs were:

wrong bucket,
stale AMI,
hidden CORS mismatch,
invisible IAM gap,
frozen replay,
missing Intent.

TBIE helped make those failures explainable.

28. The Future Direction

The current architecture already points toward several future evolutions:

• multi-stage event-driven pipelines,

• adaptive batching,

• warm GPU pools,

• richer metrics export,

• dynamic replay orchestration,

• per-stage autoscaling,

• event-sourced workflow history,

• tenant-aware media delivery,

• replayable policy evolution.

Importantly:

These evolutions do not require rewriting the architecture.

They emerge naturally because reconciliation is already foundational.

That is one of the strongest signs that the architecture direction is correct.

Closing Thoughts

TBIE did not begin as an attempt to create a framework.

It emerged because distributed AI infrastructure repeatedly exposed the same class of operational failures:

work disappearing,
retries becoming impossible,
control planes drifting,
execution environments diverging,
queues losing meaning,
infrastructure behaving correctly against stale assumptions.

The model became useful because it created clarity.

Truth. Belief. Intent. Execution.

Once those boundaries became explicit:

replay became easier,
debugging became faster,
autoscaling became safer,
policy evolution became manageable,
and infrastructure failures became explainable.

The most important lesson from Parjanya v2.0 is not that event-driven architectures are faster.

The deeper lesson is:

Resilient distributed systems are fundamentally reconciliation systems.

The system that survives is not necessarily the one with the fastest workers.

It is the one that can continuously reconstruct:

what is true,
what work should exist,
what execution already happened,
and what still needs to be reconciled.

That is what ultimately transformed Parjanya from a pipeline into a platform.

In May 2026, I found myself staring at a familiar engineering problem.

Parjanya’s Image Quality Assessment pipeline was approaching the limits of what our infrastructure could comfortably handle.

Inference workloads were growing.

Batch sizes were increasing.

Memory utilization was climbing.

And eventually, the system hit a wall.

A larger image.

A slightly bigger batch.

A little more context.

And then:

CUDA Out of Memory.

The obvious answer was straightforward.

Upgrade the hardware.

Move to larger GPUs.

Increase the budget.

Scale vertically.

That’s what I almost did.

Instead, I opened the prompt.

And that decision ended up saving far more money than a hardware upgrade would have.

The Assumption I Didn’t Question

By this point in the Parjanya v2.0 journey, I had spent months thinking about infrastructure.

The previous two posts in this series covered:

• Multi-account AWS architecture

• Terraform and operational discipline

• CI/CD pipelines

• IAM boundaries

• Infrastructure scaling

So when inference started becoming expensive, my instinct was predictable.

I looked at the hardware.

That’s what engineers do.

We assume the bottleneck lives where the graphs are red.

GPU memory usage was high.

Throughput was flattening.

Batch sizes were constrained.

The diagnosis seemed obvious.

But infrastructure metrics were only showing symptoms.

They weren’t showing the cause.

The Audit That Changed Everything

Instead of pricing larger GPUs, I decided to audit the prompt itself.

At that point our VLM-based IQA system had evolved through multiple versions.

Each release added new capabilities:

• Better composition evaluation

• Improved wildlife classification

• More robust defect detection

• Additional structured outputs

• More nuanced confidence handling

Every version added knowledge.

Almost nothing removed knowledge.

The prompt had become a living document.

Like most living documents, it accumulated debt.

When I finally measured it, the numbers surprised me.

The system prompt plus user context had grown to roughly:

4,150 tokens

Not because we needed 4,150 tokens.

Because nobody had challenged whether we still needed them.

What I Found

The easiest way to describe it is this:

The prompt looked less like a carefully engineered system and more like a codebase that hadn’t been refactored in years.

There was useful logic.

There was historical baggage.

And there was a lot of dead code.

Unused Outputs

One section calculated an overall quality score.

The model spent tokens reasoning about it.

The downstream system ignored it completely.

The actual acceptance and rejection decisions came from deterministic rules elsewhere in the pipeline.

The score existed because an earlier version needed it.

The current version didn’t.

Yet the prompt still carried the instructions.

Features That Never Shipped

I found sections generating:

• search tags

• mood descriptions

• discovery metadata

These outputs were originally intended for future content discovery features.

Those features never launched.

The instructions remained.

Every inference paid the cost.

Every single time.

Teaching the Model Things It Already Knew

This was the biggest surprise.

Over time we’d added examples.

Then more examples.

Then explanations for the examples.

Then explanations explaining the explanations.

The prompt contained long sections describing:

• composition techniques

• confidence scoring

• framing recognition

• classification behavior

At one point those explanations were useful.

But the model wasn’t learning from scratch anymore.

The explanatory scaffolding had become informational overhead.

The rules mattered.

The essays explaining the rules did not.

I Started Thinking About It Like Technical Debt

The moment everything clicked was when I stopped calling it prompt engineering.

This wasn’t prompt engineering.

This was refactoring.

The exact same process we apply to software.

In software we ask:

• Is this function still used?

• Does this dependency still matter?

• Is this abstraction providing value?

• Can this code be simplified?

I realized prompts deserve the same scrutiny.

A prompt is infrastructure.

It consumes resources.

It affects latency.

It affects throughput.

It affects cost.

And like any infrastructure, it accumulates debt.

I started calling this:

Prompt Debt.

What Stayed

This is the part that surprised me most.

Almost all of the intelligence survived.

The capabilities remained intact.

The model still knew how to:

• recognize deliberate composition

• distinguish artistic choices from technical defects

• apply confidence-aware wildlife classification

• detect focus issues

• identify exposure problems

• generate structured outputs

• evaluate image quality

We didn’t remove intelligence.

We removed repetition.

We removed explanation.

We removed historical residue.

The distinction matters.

The Numbers

After the audit:

System Prompt

~700 tokens → ~360 tokens

User Prompt

~3,450 tokens → ~890 tokens

Total Context

~4,150 tokens → ~1,250 tokens

Approximately:

70% reduction

without changing:

• the model

• the hardware

• the infrastructure

• the deployment architecture

Only the prompt changed.

Why 70% Matters

It’s tempting to see this as an academic optimization.

It wasn’t.

Those tokens had real operational consequences.

Every unnecessary token consumes:

• memory

• compute

• latency budget

• throughput capacity

On paper, a token looks small.

At scale, thousands of unnecessary tokens become infrastructure.

And infrastructure costs money.

Before the cleanup, our batching limits were constrained by context size.

After the cleanup, we could process roughly twice as many images within similar memory budgets.

The result was effectively:

2x throughput without changing hardware.

The kind of gain most teams expect from an expensive infrastructure project.

We got it from deleting text.

Every Token Has a Cost

This became the biggest lesson of the entire exercise.

Engineers often think about infrastructure costs.

Few think about information costs.

Every token carries a burden.

A token occupies memory.

A token increases inference time.

A token competes for attention.

A token introduces complexity.

The question isn’t:

“Can the model read this?”

The question is:

“Should the model be reading this at all?”

Those are very different questions.

This was a practical reminder of a lesson I explored earlier in Grounding Is Not a Prompt. Models don’t become more faithful simply because we provide more instructions. What matters is the quality and relevance of information available at inference time. A smaller, denser prompt often outperforms a larger one filled with historical baggage and explanatory noise.

The Broader Pattern

The more I reflected on the exercise, the more it felt familiar.

This pattern appears everywhere.

Organizations accumulate meetings.

Software accumulates dependencies.

Infrastructure accumulates services.

Prompts accumulate instructions.

Over time, complexity grows through addition.

Rarely through subtraction.

Most systems become slower because nobody asks what can be removed.

Prompts are no different.

While reviewing the prompt, I kept thinking about a principle we previously discussed in Databases, Caches and Queues: A Universal Rule for Distributed Systems. Systems become resilient when information has clear ownership and purpose. The same principle applies to prompts. Every instruction should exist for a reason. If nobody consumes it, it becomes operational debt.

The Connection to Inferencing

This experience fundamentally changed how I think about inference optimization.

Most conversations focus on:

• larger models

• larger context windows

• larger GPUs

• more sophisticated hardware

Those matter.

But they’re not always the first lever.

Sometimes the most effective optimization isn’t computational.

It’s informational.

Before changing the model, review the prompt.

Before upgrading the GPU, review the prompt.

Before increasing context windows, review the prompt.

The cheapest token is the one you never generate.

What Comes Next

Prompt reduction was only one layer.

The next frontier involves serving optimization:

• prompt prefix caching

• KV cache reuse

• vLLM batching improvements

• quantization strategies

• memory-efficient inference

Those areas still offer meaningful gains.

But they sit on top of a more fundamental principle.

Optimize information before optimizing infrastructure.

The Takeaway

When the system first ran out of memory, I assumed we had a hardware problem.

What we actually had was an information problem.

The model wasn’t struggling because it lacked capacity.

It was struggling because we were making it carry unnecessary baggage.

That distinction changed how I think about AI systems.

We often assume progress requires adding something.

A larger model.

A larger GPU.

A larger context window.

A larger budget.

In this case, the breakthrough came from subtraction.

The model already knew enough.

The system simply needed less noise.

And that’s a lesson that extends far beyond prompts.

It’s a lesson about engineering itself.

This concludes the initial Parjanya v2.0 engineering series. Across these three posts—from AWS account boundaries, to Terraform discipline, to prompt debt—the common theme has been surprisingly consistent: systems scale best when boundaries are clear, assumptions are explicit, and unnecessary complexity is removed. The challenge is rarely adding more. The challenge is knowing what to keep.

Further Reading

If this article resonated with you, these related essays expand on the ideas discussed here:

• Grounding Is Not a Prompt — Why better information retrieval often matters more than adding more instructions to a model.

• VLM Batch Inference - 4 bit quantisation depth — The Hidden Cost of “Almost Fitting”

• Context Engineering and Context Debt — a data investigation that revealed ~77% of my token spend was context accumulation waste, not productive reasoning. This is the framework I built to fix it — and the problem has a name: Context Debt and the fix is: Context Engineering .

• Databases, Caches and Queues: A Universal Rule for Distributed Systems — The Truth, Belief and Intent framework that inspired how we think about information flow across systems.

• TBIE in Practice: Designing Resilient AI Pipelines That Recover, Reconcile, and Re-run - Reconciliation loop and TBIE model with case studies with Parjanya v2.0

• Terraform at Scale: What Actually Worked, What Didn’t, and the Cost of Trusting AI with Infrastructure — The previous post in this series, exploring operational discipline and infrastructure boundaries.

• Why We Use Three AWS Accounts (Not Two) for Parjanya v2.0 — The architectural foundation that enabled the optimizations described here.

Introducing the Inferencing Series

This article marks the beginning of a new series exploring inferencing as a first-class engineering discipline. Much of the AI conversation today revolves around models, benchmarks, and training. Yet in production systems, the economics and behaviour of AI are often determined long after training is complete—during inference. The choices we make around prompts, context, quantization, batching, memory management, caching, retrieval, and serving architecture frequently have a greater impact on cost, latency, and reliability than changing the model itself.

Throughout this series, I’ll share lessons learned while building and operating Parjanya’s Visual Language Model (VLM) pipelines. Rather than focusing on LLM chat applications, the emphasis will be on real-world image understanding systems: image quality assessment, wildlife classification, structured extraction, multimodal reasoning, and the engineering decisions required to run them efficiently at scale. The goal is not simply to reduce costs, but to understand how information flows through an AI system and where meaningful optimization actually lives.

Why Start With Prompt Debt?

Prompt Debt is an ideal starting point because it reveals a common misconception about AI systems: that performance improvements come primarily from adding more. More instructions. More examples. More context. More hardware. In practice, many mature VLM systems accumulate informational debt in the same way software accumulates technical debt—unused outputs, redundant instructions, abandoned features, and historical assumptions that continue consuming resources long after they stop creating value.

Future articles in the Inferencing Series will build on this foundation by exploring quantization, context engineering, grounding strategies, serving architectures, prompt caching, KV-cache optimization, batching, multimodal pipelines, and the operational realities of running VLM workloads in production. Prompt Debt is simply the first lens. The broader question—and the one that interests me most—is how to maximize signal while minimizing computational waste across the entire inferencing stack.

When people talk about Terraform, they often talk about Infrastructure as Code.

What I learned while building Parjanya v2.0 is that Terraform is really about something else:

forcing operational discipline.

Terraform didn’t magically simplify our infrastructure. It exposed assumptions, surfaced hidden dependencies, and forced us to formalize decisions we had previously gotten away with making informally.

As Parjanya evolved from a collection of scripts, console clicks, and one-off configurations into a multi-tenant image processing platform, Terraform became the backbone that allowed us to scale infrastructure without losing control.

It also introduced new challenges.

And when I layered AI-assisted engineering into the workflow, I discovered an entirely different category of problems worth documenting.

This post is the implementation companion to my previous article on why we chose a three-account AWS architecture for Parjanya v2.0.

That post explained the boundaries.

This one explains how we turned those boundaries into code.

The Reality of Operating Across Five Repositories

Parjanya today spans five repositories.

Each exists for a reason.

• parjanya-ml handles inference and model serving.

• parjanya-ops owns infrastructure and platform operations.

• parjanya-backend manages orchestration and APIs.

• parjanya-frontend serves tenant-facing experiences.

• parjanya-shared-libraries contains contracts and shared abstractions.

The separation works well organizationally.

ML engineers don’t need to understand IAM condition keys.

Infrastructure engineers don’t need to navigate PyTorch dependencies.

Backend engineers don’t need to modify Terraform every time they ship a feature.

But Terraform exposed a challenge that few people talk about:

cross-repository infrastructure dependencies.

The moment one repository depends on infrastructure outputs defined in another, coordination becomes critical.

A renamed Terraform output in one repository can quietly break another repository that references it through remote state.

I’ve been bitten by this more than once.

Terraform didn’t prevent those mistakes.

What it did provide was visibility.

And visibility is often the difference between a recoverable issue and a production incident.

What Terraform Actually Improved

A lot of Terraform discussions stay theoretical.

Here’s what it changed in practice for us.

ECS and Warm Pools

One of the first problems we tackled was cold-start latency.

Inference workloads running on ECS-backed EC2 instances would occasionally wait several minutes for capacity to become available.

To solve this, we implemented Auto Scaling Group warm pools.

The result wasn’t just faster startup times.

It gave us a repeatable way to define capacity behavior for different tenant tiers.

Instead of manually tuning infrastructure across environments, we could express those decisions through variables and deploy them consistently.

That may sound mundane.

But consistency becomes incredibly valuable once multiple environments, tenants, and deployment pipelines are involved.

Multi-Tenant S3 Isolation

Parjanya is a multi-tenant platform.

Tenant isolation isn’t optional.

Terraform allowed us to codify S3 prefix-level isolation directly into IAM policies.

The most important outcome wasn’t the policy itself.

It was the reviewability.

A missing condition that could have exposed tenant object metadata became visible during code review instead of after deployment.

That’s one of Terraform’s underrated strengths:

It turns invisible infrastructure assumptions into visible diffs.

DynamoDB Tenant Boundaries

The same pattern applied to DynamoDB.

Every tenant’s data is partitioned using tenant identifiers, with IAM enforcing access boundaries.

Could application code enforce those rules?

Yes.

Should application code be the only layer enforcing them?

Absolutely not.

Terraform helped us move those guarantees into infrastructure itself.

Even if application logic fails, IAM remains the final gatekeeper.

Event-Driven Infrastructure

Parjanya’s processing pipeline relies heavily on EventBridge and SQS.

Before Terraform, environment drift was a recurring problem.

A queue configuration changed in Production but not Development.

An EventBridge rule existed in one environment but not another.

Over time these differences accumulate.

Terraform gave us repeatability.

The same modules, deployed consistently, dramatically reduced configuration divergence.

The Three-Account Migration

The biggest Terraform project in Parjanya v2.0 wasn’t a service deployment.

It was infrastructure reorganization.

We migrated from a single AWS account into:

• Management / Tooling

• Development

• Production

Terraform made that migration possible.

It did not make it easy.

Making State Account-Aware

Our original infrastructure assumed everything lived in one account.

Provider configurations were implicit.

Cross-account roles didn’t exist.

Account boundaries didn’t exist.

The first step was making account ownership explicit.

That sounds simple until you realize every Terraform resource suddenly needs to know which AWS account it belongs to.

The work wasn’t intellectually difficult.

It was painstaking.

And this is where I learned an important lesson about AI-assisted engineering:

AI is surprisingly bad at repetitive infrastructure migrations unless it has complete context.

Migrating State Safely

One of the most valuable Terraform commands I encountered during this process was:

terraform state mv

State migration sounds boring.

Until you realize a single mistake can cause Terraform to believe a production resource should be destroyed and recreated.

The migration process became an exercise in discipline:

Move state.

Run a plan.

Review carefully.

Repeat.

Slowly.

The temptation to batch changes is enormous.

Resisting that temptation saved us multiple times.

Cross-Account IAM

The three-account architecture introduced a new category of complexity.

Trust relationships.

The Management account hosts shared tooling.

Development and Production need access to those resources.

Terraform became the mechanism for expressing those trust boundaries consistently.

What surprised me wasn’t the complexity of the IAM policies.

It was how much architectural thinking is required before writing any Terraform.

The tooling faithfully implements your design.

It does not validate whether the design itself is sensible.

Centralized State Management

One decision I remain happy with is keeping Terraform state centralized within the Management account.

Development engineers can operate Development.

Production operators can manage Production.

But neither automatically gains access to every state file.

This separation aligns with the overall philosophy behind the architecture:

Boundaries first.

Convenience second.

Where AI Helped — And Where It Didn’t

Throughout this migration I relied heavily on AI-assisted development.

Claude became part of the workflow.

Over time I learned that different models excel at very different tasks.

The results were revealing.

Haiku: Fast But Dangerous for Infrastructure

Haiku was excellent at generating boilerplate.

Variable files.

Module outputs.

Simple scaffolding.

But infrastructure isn’t just syntax.

Infrastructure is reasoning.

I repeatedly found that Haiku could generate valid Terraform that contained subtle security flaws.

The code looked correct.

The logic wasn’t.

That distinction matters.

A lot.

My conclusion became simple:

Haiku should never be trusted with security-sensitive infrastructure decisions.

Sonnet: The Daily Driver

Most of our Terraform work happened with Sonnet.

For implementation tasks inside a repository, it performed extremely well.

Module structures.

Provider configurations.

Refactors.

General Terraform development.

Where it struggled was context.

A repository rename might affect multiple repositories.

A queue change might impact systems it couldn’t see.

The problem wasn’t model intelligence.

The problem was visibility.

The model can only reason about what you show it.

Opus: Better at Architecture Than Implementation

I brought Opus into the process whenever decisions became architectural.

Questions like:

• Should this responsibility live in Management or Production?

• How should artifacts flow between accounts?

• What’s the cleanest trust model?

These are trade-off questions.

Opus consistently handled them better than implementation-focused tasks.

The irony was that the more strategic the question became, the more useful the model became.

The Biggest Problem Wasn’t Terraform

It Was Context.

This became one of the clearest lessons from Parjanya v2.0.

The hardest infrastructure problems weren’t provider bugs.

They weren’t Terraform limitations.

They weren’t AWS limitations.

They were context problems.

Infrastructure lived in one repository.

ML workloads lived in another.

Shared interfaces lived somewhere else.

Every repository understood part of the system.

No repository understood the entire system.

That’s why we started building supporting tooling and internal agents focused on context discovery and verification.

The challenge wasn’t generating code.

The challenge was understanding the system before generating code.

What Terraform Didn’t Solve

It’s important to be honest about this.

Terraform isn’t a cure-all.

We still accumulated module sprawl.

We still dealt with drift.

We still had to manage secrets separately.

We still needed human judgment when reviewing large infrastructure changes.

Terraform gave us control.

It did not eliminate responsibility.

The Practical Lessons

If I were starting again today, these are the lessons I’d carry forward.

Use Terraform to enforce boundaries, not just provision resources.

Treat account architecture as an IAM problem first and a Terraform problem second.

Invest in cross-repository visibility before investing in more automation.

Choose AI models based on task type, not convenience.

Review infrastructure plans like code. Because they are.

Prioritize understanding the system before changing it.

Most infrastructure failures aren’t caused by bad tooling.

They’re caused by incomplete context.

Looking Ahead

The three-account architecture gave us the boundaries.

Terraform gave us the implementation.

But neither addressed another challenge that emerged as Parjanya’s inference workloads grew:

Cost.

Specifically, inference cost.

While we were optimizing infrastructure, another inefficiency was hiding in plain sight.

Our VLM system prompt had quietly grown to over 4,000 tokens.

Most of it wasn’t doing useful work.

Tomorrow’s post moves from infrastructure into inference engineering.

I’ll share how we reduced prompt size by nearly 70%, doubled throughput on the same hardware, and discovered that some of the most impactful optimizations don’t involve changing models at all.

The Takeaway

Terraform didn’t make Parjanya v2.0 successful.

What it did was force us to be explicit.

Explicit about ownership.

Explicit about trust boundaries.

Explicit about infrastructure decisions.

And perhaps most importantly, explicit about the assumptions we were making.

That’s the real value of Infrastructure as Code.

Not automation.

Clarity.

And clarity scales far better than heroics.

This is part of my ongoing Parjanya v2.0 field notes, documenting the architectural, operational, and engineering decisions behind building a multi-tenant AI platform. In the next post, I’ll move beyond infrastructure and into inference optimization—where a prompt review ended up delivering more value than a hardware upgrade.

One AWS account.

Maybe two if you’re feeling disciplined.

One for development. One for production.

It sounds clean. It sounds sufficient.

For a while, I thought so too.

As Parjanya evolved into a multi-tenant image processing platform with real workloads, real customers, and increasingly complex infrastructure, I started asking a different question:

What does the minimum viable production-grade AWS architecture actually look like?

The answer wasn’t one account.

It wasn’t even two.

It was three.

Today Parjanya v2.0 runs on a three-account AWS structure:

• Management / Tooling

• Development

• Production

It added complexity upfront.

It also removed entire categories of risk that I didn’t want to discover during an incident.

This post explains why I made that decision, what problems it solved, what I intentionally chose not to build yet, and how I think other SaaS teams should approach the same trade-offs.

The Setup We Started With

Like most startups, we began with a single AWS account.

Everything lived there:

• Development resources

• Production workloads

• CI/CD pipelines

• Container registries

• Shared infrastructure

The setup was fast.

The problem with fast is that it often hides risk.

Nothing feels wrong until the day an experiment, a permission mistake, or an infrastructure change unexpectedly affects production.

As we started formalising Parjanya v2.0, I realised I didn’t want to keep relying on discipline alone.

I wanted structural safety.

That’s when the three-account model emerged.

The Third Account Isn’t Staging

Whenever I mention three accounts, people assume I mean:

• Dev

• Staging

• Prod

That’s not what I built.

The structure is:

Management / Tooling Account

This account owns the operational plumbing:

• Container registry

• Terraform state

• CI/CD definitions

• Cross-account IAM roles

• Billing aggregation

• Audit aggregation

Importantly, it never runs application workloads.

Development Account

This is where experimentation happens.

Infrastructure changes.

New features.

Testing.

Things break here by design.

Production Account

This runs live tenant workloads and customer-facing infrastructure.

Nothing experimental belongs here.

The critical insight is that the third account isn’t another environment.

It’s neutral territory.

It owns the things both Dev and Prod need, without forcing either environment to depend directly on the other.

The First Problem It Solves: Artifact Ownership

Consider a simple CI/CD pipeline.

It builds a Docker image.

Where should that image live?

If the registry sits in Dev, then Production depends on Development.

If the registry sits in Prod, Development depends on Production.

Neither feels right.

The Management account solves this neatly.

CI builds once.

Pushes once.

Both Dev and Prod pull the exact same artifact.

No rebuilds.

No copies.

No divergence.

One image.

One source of truth.

Multiple promotions.

That sounds like a small detail until you’re debugging an issue and need absolute confidence that Production is running the same artifact that passed validation earlier.

The Real Reason: Blast Radius

This is the reason that mattered most to me.

AWS accounts are the strongest isolation boundary AWS provides.

A broken IAM policy.

An accidental deletion.

A Terraform mistake.

A bad deployment.

These things cannot casually cross account boundaries.

That matters because Development should be chaotic.

People test things.

Infrastructure changes.

Experiments happen.

Failures are expected.

Production should not inherit that chaos.

With separate accounts, Dev can fail spectacularly and Prod remains untouched.

That’s a much stronger guarantee than “everyone promises to be careful.”

Credentials Become Safer by Default

A surprisingly useful side effect is credential isolation.

When someone is working in the Dev account, their credentials only see Dev resources.

They aren’t merely discouraged from touching Production.

Production is outside their scope.

The same applies to automation.

The deployment pipeline for Dev assumes different roles than the deployment pipeline for Prod.

The separation isn’t cultural.

It’s structural.

And structural controls tend to survive stressful situations much better than process documents.

Cost Visibility Becomes Obvious

Another benefit I underestimated was cost attribution.

Using AWS Organizations means every account naturally reports its own spending.

Development spend is visible.

Production spend is visible.

There’s no tagging strategy to maintain.

No allocation spreadsheets.

No cost archaeology.

When the AWS bill changes, I immediately know whether it was caused by:

• a production feature,

• increased tenant activity,

• or a development experiment that ran longer than intended.

That clarity is surprisingly valuable.

What Lives Where

The distribution is straightforward.

Management Account

• Container registry

• Terraform backend

• CI/CD pipelines

• Cross-account IAM roles

• Billing aggregation

• Audit logging

Development Account

• Full application stack

• Synthetic tenant data

• Reduced-scale services

• Fast feedback loops

• Manual automation triggers

Production Account

• Live tenant workloads

• Customer-facing services

• Scheduled automation

• Alerting

• Full-scale infrastructure

The key principle is simple:

No application workloads run in the Management account.

It exists to operate the platform, not host it.

The Deployment Flow

The deployment path now looks like this:

1. Code lands on the main branch.

2. CI/CD runs in the Management account.

3. The pipeline assumes a deployment role in Dev.

4. Integration tests run.

5. The same artifact is promoted into Prod.

Production never builds code.

Production never owns source repositories.

Production simply runs validated artifacts.

That separation removes a surprising amount of operational risk.

What I Chose Not to Build Yet

Architecture decisions are often more about what you don’t build than what you do.

No Staging Environment

Yet.

A true staging environment means maintaining a production-like replica.

That’s useful.

It’s also expensive.

At our current scale, good automated testing in Dev provides far more value than maintaining another full environment.

The architecture already supports a staging account.

I simply chose not to pay for it yet.

No Tenant-Level VPC Isolation

Today tenant isolation is logical rather than physical.

Data boundaries are enforced at the application and storage layers.

Per-tenant VPCs or accounts add stronger isolation, but also significant complexity.

Most SaaS companies don’t need that level of separation until enterprise customers explicitly require it.

We’re not there yet.

No Dedicated Security Account

AWS would ideally have:

• Management

• Security

• Dev

• Prod

with centralized immutable audit storage.

I agree with that architecture.

I also believe infrastructure maturity should follow business reality.

We’ll add a dedicated security account when compliance requirements justify it.

Not before.

My Rule of Thumb for Startups

If you’re building a SaaS product and you’ve moved beyond proof-of-concept, my recommendation is straightforward:

Start with Three Accounts

Management.

Development.

Production.

The setup overhead is measured in hours.

The long-term safety benefits are measured in years.

Don’t Rush Into Staging

Build staging when your release process genuinely requires it.

Not because every architecture diagram on the internet includes one.

Use AWS Organizations Early

Retrofitting Organizations later is painful.

Starting with it is easy.

Future-you will be grateful.

Treat the Management Account as Permanent Infrastructure

Products evolve.

Tooling evolves.

The operational foundation should remain stable.

The Management account owns the plumbing, not the application.

Looking Ahead

The current structure gives me a clear path forward.

When the platform grows, I can add:

• A staging account

• A dedicated security account

• Enterprise tenant accounts

Each becomes an extension of the existing architecture rather than a redesign.

That’s exactly what I wanted.

The Takeaway

The strongest boundary in AWS is the account boundary.

Use it deliberately.

For me, three accounts turned out to be the minimum structure that delivered:

• blast-radius isolation,

• clean credential scoping,

• predictable deployments,

• and clear cost attribution.

Not because it’s elegant on a diagram.

Because it makes operational mistakes dramatically less expensive.

Parjanya v2.0 is being built with longevity in mind.

The three-account architecture is one of the quiet decisions that makes everything else possible.

The architecture looks clean on a diagram.

Reality was messier.

Moving Parjanya from a single AWS account to a three-account model meant refactoring Terraform state, redesigning IAM trust relationships, rebuilding deployment pipelines, and discovering that AI-generated infrastructure code is only as good as the context you give it.

Tomorrow, I’ll share the implementation story—what broke, what surprised us, and why the hardest part of infrastructure isn’t Terraform itself, but managing context across systems, repositories, and teams.

This is part of my ongoing field notes from building Parjanya v2.0 — a multi-tenant image QA platform. I write about the architectural decisions, trade-offs, mistakes, and lessons learned while building it in public.

A few weeks ago, one of the photographs from this series, Vamana, was selected as a finalist in the Black & White category of the ReFocus Awards.

The image sparked an unexpected response. Friends who had seen it over the past several months kept asking the same questions:

I promised to write about it when I first shared the image shortly after returning from Antarctica in October 2025. Like many unfinished promises to myself, it remained buried beneath work, travel, writing and countless photographs waiting to be processed.

The ReFocus recognition felt like the right moment to finally tell the story.

Because these photographs were never planned as a project.

They emerged gradually during the journey.

And like many of my favourite images, they began with a problem.

The Problem of Scale

Antarctica, South Georgia and the Falkland Islands repeatedly confronted me with a challenge that a single photograph struggled to solve.

Scale.

A portrait could show an individual bird.

A landscape could show the colony.

But neither could convey what it felt like to stand there.

One of the earliest moments came on Steeple Jason Island in the Falklands.

Before landing, we had been briefed that we would be visiting one of the world’s largest Black-browed Albatross colonies.

I knew the numbers.

I had seen the photographs.

But numbers and photographs do not prepare you for the experience.

As I climbed the slope and looked across the colony, thousands upon thousands of albatrosses occupied the hillsides in every direction.

My instinct was to photograph an individual bird.

My second instinct was to photograph the colony.

Neither image felt complete.

The bird lacked context.

The landscape lacked intimacy.

I wanted both.

That was the moment the experiment began.

Discovering Multiple Exposure

The Canon R5C includes an in-camera multiple exposure mode.

It is a feature that many photographers know exists but few use regularly.

I had experimented with it before, but mostly as a creative exercise.

This time it felt useful.

Instead of creating a portrait and a landscape separately, I wondered whether I could combine them into a single frame.

Not digitally.

Not later in Photoshop.

But in-camera.

The goal wasn’t artistic abstraction.

At least not initially.

The goal was documentation.

I wanted to document how the place felt.

My settings remained remarkably simple throughout the project:

• Canon R5C

• RF 100-500mm lens

• Multiple Exposure mode

• Continuous shooting

• Additive/Luminance blending

• Mostly 2 exposures per image (3 exposures only for Vamana)

Luminance blending proved especially important.

It allowed brighter areas from one frame to merge naturally with darker regions from another, creating layered images that retained structure without becoming visually chaotic.

The process quickly became addictive.

Instead of searching for a decisive moment, I began searching for relationships between moments.

Rajahamsa

The first successful image from the experiment became what I later titled Rajahamsa.

A Black-browed Albatross emerged from within its own colony.

The bird was simultaneously an individual and a multitude.

The more I looked at the image, the less it felt like a documentary photograph.

Something symbolic had appeared.

At the time I couldn’t fully explain it.

I only knew the image felt different.

It seemed to contain an idea rather than merely a subject.

Looking back now, I realize that was the first indication that the project was heading somewhere unexpected.

From Observation to Archetype

The transformation happened gradually.

Not during editing.

Not even during shooting.

It happened while looking.

Many of the stories we encounter in childhood never truly leave us.

They remain dormant somewhere beneath conscious thought.

Standing among vast colonies of birds and seals at the edge of the world, those stories began resurfacing.

Not literally.

Symbolically.

The animals remained animals.

But they also began to resemble archetypes.

The same way clouds occasionally resemble faces.

Or mountains resemble sleeping giants.

The photographs were becoming bridges between observation and imagination.

Vamana

The turning point arrived with a lone penguin(Aďelé) standing alone an ice scape and there near by an immense colony of Adele and Rockhopper (I recall!)

Unlike the other images in the series, Vamana required three exposures.

An individual penguin.

The colony below.

The surrounding landscape.

When the image appeared on the camera screen, something immediately felt familiar.

A small figure.

An immeasurable world.

A paradox of scale.

It reminded me of Vamana, the dwarf incarnation who expands beyond expectation and measures the universe itself.

The connection was not deliberate.

It emerged after the photograph existed.

The mythology wasn’t guiding the image.

The image was awakening the mythology.

For the first time I began wondering whether these photographs might belong together as a series.

Jatayu

If there is a photograph that convinced me the project was real, it was Jatayu.

Captured among the king penguin colonies of South Georgia, the image combines a chick’s portrait with thousands of penguins spread across a glacial valley.

The resulting figure feels monumental.

Protective.

Almost watchful.

The glacier behind it only amplifies that sensation.

I remember staring at the rear LCD and immediately thinking:

“That looks like Jatayu.”

Not because it resembled any traditional depiction.

But because it carried the emotional weight I associated with the character.

Courage.

Guardianship.

Sacrifice.

For the first time, I stopped thinking about multiple exposure as a technique.

And started thinking about it as a language.

When Wildlife Becomes Myth

One question I am often asked is whether these photographs were intended to illustrate Hindu mythology.

The answer is no.

At least not consciously.

I wasn’t searching for Vamana.

I wasn’t searching for Jatayu.

I wasn’t searching for Rajahamsa.

I was searching for ways to describe experiences that exceeded the limits of a single frame.

The mythology appeared later.

What fascinates me now is that this happened almost automatically.

Faced with extraordinary wildlife spectacles, my mind reached for archetypes I had inherited through stories.

Perhaps that is what myths have always done.

They give us a language for experiences that feel larger than ordinary reality.

Technical Notes

All photographs in this series were created entirely in-camera using the Canon R5C.

No compositing was performed in post-processing.

The images rely on:

• Multiple Exposure mode

• Continuous shooting

• Luminance blending

• Telephoto compression using the RF 100-500mm lens

Most images use two exposures.

Vamana uses three exposures.

What appears complex is actually the result of repeated experimentation in the field and a willingness to embrace unpredictability.

The final image often revealed itself only after the shutter sequence was complete.

Looking Back

When I boarded the expedition vessel, I expected wildlife photography.

I did not expect a conversation with memory.

Yet that is ultimately what this project became.

The Southern Ocean presented extraordinary animals, extraordinary landscapes and extraordinary scale.

Multiple exposure allowed those experiences to coexist within a single frame.

And somewhere between observation and imagination, the birds became archetypes.

Not because they ceased being wildlife.

But because they became something more.

A reminder that every act of seeing is shaped by the stories we carry with us.

more images from the series arrived unexpectedly,

Matsya → Preservation of life

While photographing a Gentoo penguin colony in Antarctica, I created a multiple exposure that combined the landscape, the colony, and a solitary penguin. Later, as I revisited the frame, it reminded me of Matsya, the fish incarnation of Vishnu in Hindu mythology.

In the ancient story, Matsya appears during a great deluge and guides a vessel carrying life through the flood, ensuring continuity when the world stands on the edge of destruction. In this photograph, the expedition ship (our own Ortelius MV) rests in the distant background while the penguin emerges as a quiet, symbolic guide in the foreground. The resemblance was not planned in the field, but revealed itself later—one of those moments where photography becomes less about recording what is seen and more about discovering meaning within what was witnessed.

Whether coincidence or subconscious association, the image felt like a reminder that stories often exist before we recognize them. The camera records light; imagination completes the journey.

Varaha → Restoration of the world

By the end of the expedition, I had stopped looking for specific characters. Instead, I simply observed and waited for forms to emerge on their own. Varaha appeared unexpectedly—in the shape of ice, snow, and a colony scattered across the shoreline. In Hindu mythology, Varaha rescues the Earth from cosmic waters and restores balance to the world. Standing in Antarctica, a place that constantly reminds us of both fragility and endurance, I found the symbolism difficult to ignore. It became a fitting final archetype: not a story of survival alone, but of renewal.

I will be Continuing the search for new archetypes in new landscapes in the future

What began as an experiment has quietly become an ongoing creative pursuit. I now find myself actively searching for new characters and visual metaphors wherever I travel. They may emerge from different landscapes, different species, and different cultures, but the underlying idea remains the same: finding stories that exist somewhere between reality and imagination. More than anything, this project reminded me how much joy there is in playful exploration. I had an incredible amount of fun creating these images, and I look forward to discovering where the next character appears. You can follow my work here

The challenge was never collecting information.

The challenge was remembering it, connecting it and turning it into something useful.

Modern AI is remarkably capable.

Yet most systems remain surprisingly forgetful.

Every conversation begins from scratch.

Every project requires context to be reintroduced.

Every insight risks being lost in another folder, notebook or document.

The more I worked across research, photography, software, storytelling and archival work, the more obvious the problem became:

Knowledge was accumulating faster than understanding.

Instead of building another assistant, I started exploring a different idea.

What would a genuine second brain look like?

Not a chatbot.

Not a search engine.

Not a note-taking application.

A system that could:

Observe → Remember → Understand → Create → Preserve

A system capable of connecting ideas across years rather than conversations.

A system that becomes more valuable as its memory grows.

From Information to Understanding

Most AI systems operate on prompts.

A second brain should operate on context.

Every note should strengthen understanding.

Every project should enrich memory.

Every experiment should improve future decisions.

The objective is not simply to retrieve information.

The objective is to build accumulated intelligence.

The Emergence of Smriti

This thinking eventually evolved into a concept I now call:

Parjanya’s Second Brain

At its core sits Smriti.

Smriti is not simply another product.

It is the memory and intelligence layer that connects everything else.

A system designed to preserve context, connect knowledge and build long-term understanding.

Around this core are four complementary domains:

Parjanya

Visual Intelligence

Image quality assessment, visual understanding, benchmarking and insight generation.

WilderhoodTV

Creative Intelligence

Storytelling, production workflows, media creation and distribution.

Smriti

Rooted Intelligence

Memory, context, reflection and accumulated knowledge.

Prakriti

Archival Intelligence

Documenting and preserving wildlife, nature, heritage and field knowledge.

Together they form a connected intelligence ecosystem rather than a collection of independent products.

Why Context Matters

The AI industry is understandably focused on models.

Bigger models.

Smarter models.

Faster models.

But I increasingly believe the next frontier may be something else.

Context.

Models will continue to improve.

What remains unique is the context we accumulate.

Our notes.

Our observations.

Our projects.

Our experiences.

Our decisions.

Our understanding of the world.

Without context, intelligence becomes generic.

With context, intelligence becomes personal.

Looking Ahead

This is still an evolving idea.

The architecture will change.

The tools will change.

The models will change.

What remains constant is the underlying belief:

Knowledge should not disappear. It should compound.

The goal is not to automate thought.

The goal is to augment it.

To build a system that helps transform information into understanding, understanding into wisdom and wisdom into meaningful work.

Guiding Principle

Observe → Remember → Understand → Create → Preserve.

DISCLAIMER: This article is part of my personal learning journey and is meant purely for educational and informational purposes. I have drawn insights from Prof. Aswath Damodaran’s publicly available teachings and financial information available in the public domain.

While preparing this write-up, I referred to platforms such as Screener and AI-based study tools to better understand the concepts and validate publicly available data such as company filings and information available for investors from the company domain. The interpretations and conclusions shared here are entirely my own, and any errors or misunderstandings are solely my responsibility.

This is not investment advice or a recommendation to buy, sell, or hold any securities. I encourage readers to do their own research and consult a SEBI-registered investment advisor before making any investment decisions.

Key Takeaways

• Info Edge is best understood as two businesses under one roof: a dominant internet classifieds franchise and a highly successful venture investment engine.

• Traditional valuation multiples can be misleading because a significant portion of the company’s value comes from listed and unlisted investments.

• A sum-of-the-parts framework, inspired by Aswath Damodaran’s valuation approach, is the most appropriate way to analyze the company.

• The core operating business appears to be a high-quality, cash-generative franchise with strong margins and durable competitive advantages.

• At current prices, the stock appears to be trading around fair value rather than at a significant discount.

• Future returns will likely depend on both the continued strength of the Naukri ecosystem and the success of the next generation of startup investments.

When I look at Info Edge, I do not see a normal listed company. I see two businesses stitched together: a dominant, cash-generating internet classifieds franchise led by Naukri, and a venture-style investment engine that has produced extraordinary value through stakes such as Eternal and PB Fintech.

That is exactly why I think Info Edge is one of the best Indian case studies for applying Professor Aswath Damodaran's framework. Damodaran has repeatedly argued that cash, cross-holdings, and other non-operating assets should be valued separately from the operating business. In his discussions on holding companies and cross-holdings, he notes that the "sum of the parts" can often be greater than the whole when these assets are not properly accounted for. For a company like Info Edge, where listed and unlisted investments contribute materially to shareholder value, a sum-of-the-parts approach is not optional—it is essential.1

In Info Edge’s case, that distinction is not academic. As of June 2026, the company itself had a market capitalization of roughly ₹65,000 crore, while a substantial part of its value sat in listed investments and a startup portfolio that management valued at nearly ₹37,000 crore.

So the right question is not, “Is Naukri expensive on earnings?”

The right question is:

How much am I paying for the core franchise after adjusting for Eternal, PB Fintech, and the next cohort of private investments?

Why Info Edge Is a Perfect case study for understanding Prof. Damodaran’s lens

Damodaran’s valuation philosophy begins with a story and then forces the numbers to be consistent with that story.

My story for Info Edge is simple.

The first engine is a high-quality operating business with a durable moat in recruitment classifieds, strong operating leverage, and real free-cash-flow discipline.

The second engine is a capital allocator with a demonstrated ability to identify, back, and hold asymmetric winners for long periods.

That combination makes Info Edge unusual.

Many companies are either operating businesses or investment vehicles.

Info Edge is both.

Naukri remains the core cash machine, but the market’s view of the company is also shaped by what happened with Zomato—now Eternal—and with Policybazaar through PB Fintech.

This also explains why plain P/E ratios can mislead. Reported earnings often include large amounts of investment-related income and mark-to-market effects, creating accounting noise that distorts headline valuation multiples.

The Operating Story I Am Telling Myself

When I strip away the investment book, I am left with a business that is significantly better than a typical Indian internet company.

FY25 standalone revenue was approximately ₹2,654 crore, operating profit was around ₹973 crore, and the starting EBIT margin in my model was roughly 36.7%.

That is not what a fragile, capital-hungry internet business looks like.

It looks like a platform business with real pricing power and low incremental capital intensity.

Info Edge’s own disclosures increasingly describe Naukri as evolving from a traditional job-search platform into a broader talent ecosystem. That matters because deeper workflow relevance typically strengthens both customer stickiness and monetization potential.

The cash-flow quality is equally important.

Free-cash-flow conversion remains strong, and cash generation has consistently supported both internal reinvestment and external venture investments.

That cash discipline sits at the center of my thesis.

In a Damodaran framework, earnings are only useful if they can be converted into cash and distributed or reinvested intelligently.

Info Edge passes that test far better than many businesses trading on technology narratives.

The Second Engine: Capital Allocation

The other half of the story is where Info Edge becomes truly distinctive.

Management disclosed that total invested capital across its startup platform stood at roughly ₹4,000 crore, while the fair market value of those investments approached ₹37,000 crore.

Those are extraordinary numbers.

More importantly, they fundamentally change how I think about the balance sheet.

This is not idle treasury cash.

This is a portfolio that has already produced two listed winners and still carries meaningful optionality from the rest of the investment book.

The listed winners are the easiest to value.

Using June 2026 market values, Info Edge’s stakes in Eternal and PB Fintech together were worth approximately ₹39,500 crore.

That single number explains why this company must be valued in pieces.

If the market capitalization is roughly ₹65,000 crore and the two listed holdings alone are worth nearly ₹40,000 crore, then the market is implicitly assigning the remaining value to the core operating franchises plus the unlisted portfolio.

How I Split the Business

Damodaran’s treatment of cash and cross-holdings provides the right starting point.

First, value the parent company’s operating assets.

Second, value the holdings separately.

Third, decide whether those holdings deserve to be added at full value or at a discount because of taxes, opacity, or governance considerations.

For Info Edge, I split the company into three buckets:

• The core operating business: Naukri, 99acres, Jeevansathi, Shiksha, and other controlled operating assets.

• The listed investment book: Eternal and PB Fintech.

• The private portfolio: the broader startup portfolio, valued through probability-weighted assumptions and appropriate haircuts.

Once I do that, the valuation becomes much clearer.

I no longer have to force one multiple to explain two fundamentally different economic engines.

My Core-Business DCF

I built three versions of the core-business valuation.

The first was a conventional FCFF model.

The second was a stricter Damodaran-style build with a higher cost of capital and explicit return-on-capital fade.

The third was an owner-earnings model that treats Info Edge more like a high-ROIC platform business with low maintenance reinvestment requirements.

The conventional FCFF model produced an intrinsic value range of:

₹805.83 to ₹1,173.05 based on Bear to Bull scenario

The stricter Damodaran build produced:

₹624.07-₹675.71-₹722.74(Bear-Base-Bull)

The owner-earnings model produced:

Bear₹856.76, Base₹978.00, Bull₹1,150.75

I do not treat any one of these as the truth.

I treat them as a range generated by different but intellectually honest stories about the business.

That is very much in the spirit of Damodaran: valuation is a disciplined conversation between narrative and numbers, not a hunt for a fake precision point estimate.

What the Three Models Are Really Saying

Taken together, these models say something simple.

Info Edge is not obviously cheap.

But it is also not wildly expensive if I believe the core operating franchise deserves a premium for moat, cash conversion, and capital allocation quality.

The overlap zone between my more realistic base cases falls roughly between ₹950 and ₹1,000 per share.

That sits remarkably close to where the market currently values the stock.

My conclusion is not that the stock is a bargain.

My conclusion is that the stock is trading around intrinsic value, with modest upside or downside depending on how optimistic I choose to be about the operating business and how much of the private investment book I am willing to recognize.

Why the Private Portfolio Cannot Be Ignored

A lot of investors stop after marking Eternal and PB Fintech.

I think that is incomplete.

It is easy to value listed stakes and pretend the rest of the startup book is noise.

Management’s disclosures suggest otherwise.

At the same time, I do not want to value the private portfolio lazily.

For early-stage companies, the right framework resembles venture-capital valuation rather than traditional DCF analysis.

Estimate a plausible exit value.

Assign a probability of success.

Adjust for dilution.

Discount back at a sufficiently high required return.

That is why, in my own sum-of-the-parts framework, I do not add the startup portfolio at full value by default.

I use scenario-based weights and meaningful haircuts rather than assuming every fair-value mark will eventually translate into realized gains.

Promoter Skin, Moat, and Cash Discipline

My three favourite lenses for evaluating businesses are:

1. Moat

2. Promoter skin in the game

3. Free-cash-flow discipline

Info Edge scores well on all three.

Promoter ownership remains meaningful at approximately 37.5%, with founder Sanjeev Bikhchandani individually holding about 24% of the company. This level of ownership creates strong alignment between management and minority shareholders, particularly in a business where long-term capital allocation is a major driver of value creation.

When founders continue to own a meaningful portion of the business after decades of operation, I generally have greater confidence that capital allocation decisions are being made with a long-term mindset.

On moat, the easiest evidence is visible in the economics of the core business.

A business generating EBIT margins north of 35%, producing strong free cash flow, and maintaining leadership in a category for years is rarely doing so by accident.

Those economics are often downstream manifestations of:

• Brand strength

• Network effects

• Embedded workflows

• Customer inertia

• Rational competition

Naukri benefits from several of these simultaneously.

Employers want access to the largest candidate pool.

Candidates want access to the largest number of recruiters.

That creates a reinforcing flywheel that becomes increasingly difficult for competitors to replicate.

On cash-flow discipline, the story becomes even stronger.

The company has remained largely debt free while continuing to:

• Fund organic growth

• Invest in adjacent businesses

• Build a venture portfolio

• Maintain balance-sheet flexibility

That combination is surprisingly rare.

Many firms with venture portfolios lack a strong operating engine.

Many cash-generative firms struggle to identify high-return external investments.

Info Edge has demonstrated an ability to do both.

The Buy-Below Levels I Would Use

Because the stock appears to be trading around fair value, I think price discipline matters more than enthusiasm.

Valuation should influence position sizing.

Quality tells me what to buy.

Valuation tells me when to buy.

My framework currently looks like this:

ZoneInterpretationBelow ₹850Strong Buy₹850–₹930Buy₹930–₹1,030Hold / WatchAbove ₹1,100Avoid aggressive additions

Below ₹850

At these levels, even relatively conservative valuation frameworks begin to show a meaningful margin of safety.

The investment case becomes less dependent on perfect execution and more dependent on simply avoiding major mistakes.

That is usually where I prefer to deploy capital.

₹850–₹930

The valuation starts tilting in my favor while still allowing room for execution risk.

The core franchise remains attractive and the optionality from investments is available at a more reasonable price.

₹930–₹1,030

This is approximately fair-value territory.

At these levels I would be comfortable holding existing positions but would not feel a strong urge to increase exposure.

Above ₹1,100

At these valuations, I would need a substantially more optimistic view regarding:

• Core business growth

• Margin expansion

• The next generation of startup winners

• Long-term value creation from the investment portfolio

Without that conviction, future returns become increasingly dependent on continued multiple expansion rather than fundamental value creation.

What Could Break My Thesis

No valuation is complete without understanding what can go wrong.

There are four developments that would make me materially less constructive on Info Edge.

1. Naukri Loses Pricing Power

Naukri remains the economic anchor of the entire group.

If recruiters begin reducing spend, switching platforms, or finding better alternatives, the valuation of the entire business changes.

The investment portfolio may be exciting, but the operating franchise pays the bills.

A weakening recruitment moat would force me to revisit almost every assumption in my model.

2. Adjacent Businesses Fail to Improve Economics

The long-term narrative depends on businesses such as 99acres and other operating segments continuing to improve efficiency and monetization.

If these businesses fail to progress toward stronger economics, some of the optionality embedded in my valuation disappears.

The result would be lower growth assumptions and reduced confidence in margin expansion.

3. Capital Allocation Discipline Deteriorates

One of the biggest strengths of Info Edge has been its investment discipline.

Successful startup investors often face a dangerous temptation after early wins.

Past success can encourage larger bets, weaker underwriting standards, or excessive optimism.

If management begins prioritizing valuation marks over governance and business quality, I would immediately apply a larger discount to the investment portfolio.

The track record has earned credibility.

But credibility must continually be re-earned.

4. A Prolonged Decline in Eternal or PB Fintech

The market currently recognizes Info Edge not only as an operating company but also as a successful allocator of capital.

If Eternal and PB Fintech experience significant and prolonged declines, investors may stop assigning value to that allocator identity.

In that scenario, the market could increasingly focus only on the operating business while assigning lower value to the investment portfolio.

That would compress valuation multiples even if the underlying businesses remain fundamentally sound.

The Bigger Lesson

One reason I enjoy studying Info Edge is that it demonstrates how valuation is often more about structure than forecasting.

Most investors spend their time debating whether revenue growth will be 12% or 15%.

In many situations, that is not where the real insight lies.

The real insight is understanding what exactly you are valuing.

Info Edge forces that discipline.

You cannot treat it as a simple internet stock.

You cannot treat it as a venture capital fund.

You cannot treat it as a holding company.

It is all three at the same time.

And that complexity creates both confusion and opportunity.

This is precisely the type of situation where Damodaran’s framework shines.

Separate the pieces.

Value each piece independently.

Then bring them back together.

Simple in principle.

Difficult in practice.

My Final Verdict

When I put everything together, I keep arriving at the same conclusion.

Info Edge is a high-quality business with:

• A genuine moat

• Meaningful promoter alignment

• Strong free-cash-flow generation

• Exceptional capital-allocation history

• Significant optionality through its startup portfolio

Those are characteristics I actively seek in long-term compounders.

However, quality alone does not make something cheap.

At roughly ₹1,010 per share, I believe the market already recognizes much of what makes Info Edge special.

The Naukri franchise is appreciated.

The Eternal success story is appreciated.

The PB Fintech investment is appreciated.

A significant portion of the optionality is already reflected in the price.

That leaves me with a conclusion that may sound less exciting but is probably more useful:

Info Edge is not a deep-value opportunity today.

It is a high-quality business trading around intrinsic value.

Would I own it?

Yes.

Would I chase it aggressively?

Probably not.

Would I become significantly more interested if the market offered it closer to ₹850–₹900?

Absolutely.

If I were forced to summarize my view in a single sentence, it would be this:

Info Edge is one of the clearest examples in India of how Aswath Damodaran’s framework helps separate a great operating business from a great capital allocator living inside the same stock.

And that is precisely what makes it such a fascinating company to study.

Disclosure

This article is intended solely for educational and research purposes and should not be construed as investment advice. The valuation frameworks, assumptions, and conclusions presented here reflect my personal analysis and may prove incorrect. Investors should conduct their own due diligence and consider their risk tolerance before making any investment decisions.

The crimson crowns of cranes against fresh snow. The blue hues of winter mornings. The warm glow of sunrise spilling across frozen landscapes.

Naturally, those scenes became the foundation of C1 — Vibrant Hokkaido.

But as the journey unfolded, I found myself increasingly drawn to something else.

Not colour.

Time.

The more I travelled through Hokkaido, the more I felt that many of its landscapes, traditions, and wildlife existed outside the rhythm of the modern world. A crane standing motionless in the snow, a sea eagle gliding across a winter valley, steam rising through an outdoor onsen—these scenes felt as though they could belong to today, fifty years ago, or perhaps even longer.

That feeling became the inspiration for C2.

Why Monochrome?

For me, monochrome was never about removing colour.

It was about removing time.

Colour often anchors a photograph to a specific moment. We recognise the palette of a season, the look of a particular era, or the visual characteristics of modern life.

Monochrome strips away those references.

What remains are the essential elements of a photograph:

• Light

• Form

• Texture

• Atmosphere

• Emotion

More importantly, it allows a scene to feel timeless.

In Hokkaido, monochrome became the most natural way to express what I was experiencing. The enduring elegance of Japanese culture, the quiet beauty of winter landscapes, and the sense that many traditions continue largely unchanged despite the rapid pace of the world around them.

Rather than documenting Japan as it appeared before me, I wanted to create images that felt suspended between past and present.

Building C2

The objective was not to create dramatic black-and-white photographs.

Instead, I wanted to create images that felt calm, restrained, and enduring.

The settings were designed to emphasize tonal separation rather than contrast alone. Snow remained soft and luminous. Shadows retained detail. Blacks provided structure without overwhelming the frame.

The result was a visual language inspired less by spectacle and more by permanence.

A crane became a silhouette against endless snow.

An eagle became a dark brushstroke against mountain ridges.

Steam transformed a crowded onsen into something almost dreamlike.

The photographs began to feel less like records of a journey and more like fragments of memory.

What I Found

The most unexpected outcome was how much attention shifted from the subject to the feeling of the scene.

Without colour competing for attention, subtle details became more important:

The curve of a crane’s neck.

The delicate footprints pressed into fresh snow.

The layers of mist drifting through an onsen.

The patterns of winter forests beneath a soaring eagle.

Many photographs became simpler, yet somehow more evocative.

The absence of colour encouraged me to focus on what initially drew me to the frame rather than what made it visually striking.

The Images

The red-crowned cranes became symbols of grace rather than colour.

Against the snow, they appeared almost sculptural—reduced to elegant lines, shapes, and movement.

The sea eagle soaring across Hokkaido’s mountains felt timeless, its silhouette echoing the rugged landscape beneath it.

The onsen scene revealed a different side of Japan. Visitors gathered quietly in rising steam while nature enveloped the entire experience, blurring the distinction between people and place.

Even moments of intentional blur and abstraction began to feel more meaningful in monochrome, emphasizing atmosphere over description.

Each image seemed less concerned with documenting what was there and more interested in conveying what it felt like to stand there.

What C2 Taught Me

Photography often encourages us to capture moments.

Monochrome encouraged me to capture permanence.

It reminded me that some places possess a character that transcends a single season, a single journey, or even a single generation.

Hokkaido’s wildlife, landscapes, and traditions carry that quality.

By removing colour, I wasn’t simplifying the story.

I was revealing the part of it that felt enduring.

C2 — Traditional Japan (Monochrome) is my attempt to explore the timeless side of Hokkaido—a visual journey through landscapes, wildlife, and traditions that feel as relevant today as they might have decades ago, expressed through a monochrome palette that celebrates simplicity, atmosphere, and the enduring beauty of Japan.

I wasn’t researching locations.

I wasn’t building shot lists.

I wasn’t even looking at wildlife photography.

Instead, I was revisiting old Japanese films and spending far too much time pausing anime scenes.

As someone who grew up watching anime, I had always been fascinated by how Japanese artists treated landscapes. Snow felt different. Forests felt different. Even ordinary roads seemed to carry a sense of atmosphere that was difficult to describe but immediately recognizable.

The colours were often richer than reality, yet never felt artificial.

The landscapes felt vibrant, but never loud.

Somewhere between memory and observation, a distinct visual language emerged.

Without realizing it, I had already started naming it in my head.

Vibrant Hokkaido.

At the time, I thought it was a colour problem.

By the end of the trip, I realized it was a luminance problem.

And that distinction changed everything.

Designing Hokkaido Before Seeing Hokkaido

Long before boarding a flight to Japan, I started building two custom shooting profiles on my Canon EOS R5 Mark II.

The first became C1: Vibrant Hokkaido.

The second became C2: Traditional Japan.

The intention behind C1 was relatively straightforward. I wanted a profile that embraced the visual qualities I associated with modern Japanese landscape art and anime backgrounds—strong colour separation, rich blues, warm wildlife subjects and an overall sense of atmosphere.

The foundation was Canon’s Landscape Picture Style, but pushed further through additional contrast, saturation and colour tuning.

I wasn’t trying to reproduce exactly what I saw.

I was trying to reproduce what Hokkaido felt like in my imagination after years of consuming Japanese visual culture.

C2 was built for a completely different reason and deserves its own article, but in short, it was inspired by older Japanese cinema, monochrome photography and traditional aesthetics where form, shape and negative space matter more than colour.

At the time, both profiles felt complete.

I thought I had already solved the problem.

Hokkaido had other plans.

The Surprise Waiting in the Snow

One of the assumptions I carried into the trip was that colour would be the defining characteristic of the landscape.

After all, C1 was built around colour.

Yet once I started photographing winter scenes, I noticed something unexpected.

The strongest images weren’t necessarily the most colourful ones.

In fact, many of them contained remarkably little colour.

The fox crossing an icy road became one of those moments.

Initially, what attracted me to the scene was obvious.

A warm-toned fox against a cold blue landscape.

Classic colour contrast.

The kind of scene that fits perfectly within the original idea of Vibrant Hokkaido.

But as I reviewed the images later, I found myself paying less attention to the fox and more attention to the snow.

The snow wasn’t acting as background.

It was acting as structure.

The wind patterns across the frozen surface were creating visual pathways.

The lighter areas pulled the eye forward.

The darker areas acted as leading lines.

The fox wasn’t creating the composition.

The snow already had.

The fox was simply completing it.

That realization became one of the most important lessons of the trip.

Snow Isn’t White

Photographers often talk about exposing snow correctly.

What I discovered in Hokkaido was slightly different.

The challenge wasn’t exposure.

The challenge was preserving luminance relationships.

When we think about snow, we instinctively think about whiteness.

But Hokkaido snow rarely appears truly white.

It contains subtle blues.

Grey tones.

Cyan shifts.

Tiny variations that create depth and dimension.

Push everything toward pure white and the landscape becomes flatter.

Neutralize the colour too aggressively and winter loses its atmosphere.

The more I photographed, the more I realized that the visual identity of Hokkaido wasn’t being created by colour alone.

It was being created by the way snow carries light.

That became the foundation of what I eventually started calling Snow Luminance Control.

Not as a formal technique.

Not as a preset.

Simply as a way of seeing.

The objective became preserving those tonal transitions so that the snow itself could guide the viewer through the frame.

The fox remained important.

But the snow became the protagonist.

The Lightroom Problem

Ironically, the next lesson arrived after I returned from the field.

Opening the RAW files inside Lightroom was a shock.

The images didn’t look anything like what I had seen on the back of the camera.

The carefully tuned colours from C1 seemed to disappear.

The atmosphere felt different.

The tonal relationships felt different.

For a while, I assumed the profile simply wasn’t working as intended.

That confusion continued until a conversation with Rahul Sachdev.

Rahul, a Canon Ambassador and someone whose field experience I deeply respect, pointed me toward something surprisingly simple.

Match Camera.

Once Lightroom and Photoshop were instructed to honour the in-camera rendering through Canon’s camera-matching profiles, the images immediately started resembling what I had designed and seen in the field.

It was one of those moments that seems obvious in hindsight.

I had spent months creating a visual language inside the camera, only to discover that my software workflow was quietly replacing it.

That lesson alone probably saved me countless hours of unnecessary editing.

The Limitations of Custom Modes

As useful as the C1 and C2 workflow became, the trip also exposed some practical limitations.

One of the biggest frustrations involved crop settings.

Certain shooting configurations—particularly 1.6x crop workflows that proved useful for wildlife—did not always behave the way I expected once stored inside the custom modes.

Moving between C1, C2 and C3 often required additional verification to ensure the camera was actually operating the way I intended.

It wasn’t a deal-breaker.

But it was a reminder that custom modes are powerful shortcuts rather than perfect state-management systems.

The creative intent remains consistent.

The technical details still need supervision.

Looking Back

What started as an attempt to recreate a visual style inspired by anime eventually became something far more interesting.

I arrived in Hokkaido thinking about colour.

I left thinking about luminance.

The preset I designed before the trip survived.

But the reasoning behind it evolved.

Vibrant Hokkaido was never really about making colours stronger.

It was about allowing winter to remain expressive.

About preserving the subtle pathways hidden inside snow.

About understanding that a landscape can lead the eye long before a wildlife subject enters the frame.

The fox helped me recognize that.

The snow taught me why.

In the next post, I’ll explore the other half of this experiment—C2: Traditional Japan—a profile inspired not by modern anime aesthetics, but by older Japanese cinema, monochrome photography and the idea that sometimes removing colour reveals more than adding it ever could.

I was drawn to the contrast between Hokkaido’s vivid winter character — cranes, foxes, snowfields, isolated color, landscapes — and the quieter monochromatic side of Japan that emerges through mist, overcast skies, bare forests, and restrained tonal palettes.

What I did not anticipate was discovering an entirely different visual phenomenon altogether: what I now think of as layered winter landscapes.

Somewhere during the trip, while photographing frozen wetlands and distant forests through a telephoto lens, I began noticing that the landscape no longer behaved as a single continuous scene. Instead, it appeared to separate itself into distinct visual planes — snowfields, reed belts, dense trunks, atmospheric canopy — each carrying its own tonal identity.

The effect was subtle in reality but became remarkably pronounced through the viewfinder.

That discovery was unexpected.

I had travelled searching for color and monochrome as emotional themes, but Hokkaido quietly revealed a third visual language to me — one built not around color, but around compression, atmosphere, tonal separation, and layered perception.

The more time I spent with these scenes, the more they began feeling less like traditional wildlife photographs and more like studies in visual structure.

What fascinated me most was that this separation only became obvious through the camera itself.

Standing in the landscape, I did not experience the scene as strongly layered in this way. But once framed through a long lens, the environment transformed. The world flattened, distance compressed, and the landscape started behaving almost like a composed arrangement rather than a naturally receding space.

It was as though the camera was not only recording the landscape — but reorganizing it.

Understanding the Layers

The more I photographed these scenes, the more I realized that several visual ideas were quietly working together inside the frame.

Telephoto compression was one of them. Long focal lengths visually compress distance, making separate parts of the landscape appear stacked closer together. Instead of feeling deep and continuous, the world begins separating into flatter layers.

There was also an element of Gestalt perception at play. Human vision naturally groups similar tones, textures, and patterns together. Snowfields, reed belts, forests, and mist each become their own visual regions, which is why a single photograph can sometimes feel like multiple images stacked together.

And finally, there is visual rhythm — the repetition of trunks, rocks, reeds, and negative space across the frame. Winter simplifies the landscape enough for these repeating forms to become more noticeable, almost musical in their spacing and flow.

Individually, these ideas are subtle. But together, they create the layered winter landscapes I unexpectedly found myself drawn toward in Hokkaido.

Searching for a Visual Parallel

Interestingly, none of this was intentional.

I did not travel to Hokkaido inspired by any particular art movement or visual tradition. The connection appeared only later, after returning from the trip and trying to understand why these images felt so visually separated and structured.

That curiosity eventually led me toward ukiyo-e.

While researching whether similar spatial phenomena had existed in historical art, I found myself repeatedly drawn to Japanese landscape prints — particularly the quieter atmospheric works of Utagawa Hiroshige.

What resonated was not necessarily the subject matter, but the way space behaved within those images.

Many ukiyo-e landscapes organize the frame through layers and intervals rather than strong linear depth. Snow, water, mist, forests, and sky often exist as separate visual regions while still remaining part of a unified composition. The eye drifts across the scene rather than moving directly into it.

That sensation felt remarkably close to what I had been experiencing through the viewfinder in Hokkaido.

The similarity was especially striking in winter-oriented prints where forests compress into texture, snow becomes open negative space, and atmosphere quietly separates one plane from another.

What fascinated me was that photography and ukiyo-e arrive at this layered perception through entirely different means.

Ukiyo-e achieved it through artistic interpretation and compositional stylization.

The photographs achieve it through:

• optical compression,

• atmospheric flattening,

• tonal simplification,

• and perceptual grouping.

The overlap, therefore, is not imitation.

It is convergence.

Different mediums.
Different centuries.
Yet arriving at a strangely similar way of organizing space.

That realization changed how I began thinking about these images. I stopped seeing them purely as wildlife or winter photographs and started viewing them as studies in perception — moments where optics quietly transform reality into layered abstraction.

Winter as an Abstraction Engine

I suspect winter may be uniquely suited for this kind of visual experience.

Snow removes clutter.

Color palettes collapse into restraint.
Foreground distractions disappear.
Shadows soften beneath overcast skies.
The world simplifies itself into line, tone, rhythm, and silence.

Under those conditions, landscapes stop behaving purely as physical environments and begin functioning almost like visual structures.

The camera intensifies this transformation further.

The moment the frame edges isolate the scene, the eye stops navigating physical space and instead begins reading intervals:
foreground,
transition,
structure,
atmosphere.

That was the most unexpected lesson Hokkaido offered me.

I arrived searching for vibrant winter and monochrome quietness.
I left discovering layered perception.

Looking Ahead

What began as a winter wildlife trip has quietly reshaped how I now look at landscapes.

I am increasingly interested in searching for places where this phenomenon might emerge again — environments where atmosphere, distance, weather, and optics work together to create layered spatial separation.

Not necessarily dramatic landscapes.
Not necessarily famous locations.

Just places where the world simplifies enough for its internal structure to reveal itself.

Perhaps that is what photography continues to teach me most:
sometimes the camera does not merely document a place.

Sometimes it changes the way we perceive space itself.

The number that matters is this one: how many of your Lambda functions can actually run on ARM64 today — not in theory, not on a clean greenfield stack, but with your actual dependencies, your actual Python version, and your actual C extensions.

For Parjanya, an ML-powered image quality pipeline, the answer started at 1 out of 4. That gap — between what Graviton promises and what your stack allows — is almost entirely explained by a single, under appreciated concept: the ARM wheel.

This article explains what ARM wheels are, why they are the gatekeeper to Graviton savings, how the Parjanya re-architecture navigated them, and how to build a framework for evaluating Graviton readiness in any product.

First, an Honest Framing of the 34% Number

AWS’s stated price-performance advantage for Graviton Lambda comes from two compounding factors:

Pricing: ARM64 Lambda is priced approximately 20% lower per GB-second than equivalent x86 Lambda. This is a direct line-item discount.

Performance: For SIMD-heavy workloads — image processing, compression, media encoding — ARM NEON instruction sets often complete work faster than x86 AVX-512 for the same memory allocation, which effectively reduces billed duration.

In Parjanya’s $10 POC (running PIL-based image processing against Graviton4 and Intel x86), the empirical result was a 24% cost delta — Intel x86 cost 24% more for identical or worse performance. AWS’s 34% claim holds at the higher end because it accounts for the full price-performance package including EC2 and container workloads.

The savings are real. The caveat is that they only materialise if your software stack is compatible.

What Is an ARM Wheel?

Before evaluating Graviton readiness, you need to understand what a Python wheel is and why the ARM variant specifically matters.

A wheel (.whl) is a pre-built binary distribution of a Python package. When you run pip install numpy, pip looks for a wheel that matches your platform. If it finds one, it downloads and installs it directly — no compilation required. If it does not, pip falls back to downloading source code and compiling it locally.

For pure Python packages, this distinction is irrelevant. The bytecode runs anywhere.

For packages with native C/C++ extensions — NumPy, Pillow, OpenCV, PyTorch, SciPy, and most of the ML ecosystem — a wheel is architecture-specific. A wheel built for x86_64 will not install on aarch64. The package must publish a separate wheel for ARM64.

The naming convention tells you exactly what architecture a wheel targets:

numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl   # x86
numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl  # ARM64

The aarch64 suffix is what you are looking for. If it exists, installation is clean. If it does not, you are in source-build territory — and source builds in Lambda container images introduce a layer of complexity that breaks most “just change the architecture flag” migration plans.

The Four States of a Dependency

When evaluating a Python service for ARM64 migration, every dependency falls into one of four categories:

This matrix is the actual output of a Graviton readiness assessment. Not a benchmark. Not a cost model. The wheel matrix.

The Parjanya Case Study: From 1/4 to 3/4

Parjanya’s pipeline had four compute stages, each containerised and isolated. When the first Graviton migration attempt was made, only one — the deduplication stage — made it onto ARM64. The other three were blocked.

Here is what happened at each stage:

Stage 1: RAW Processing (Blocked → Eventually Bypassed)

Camera RAW files require binary parsing to extract embedded JPEGs. The dependency chain pulled in native image parsers without clean aarch64 wheels. Compilation from source failed or produced unstable binaries in the Lambda container environment.

Resolution: The stage architecture changed. Rather than fight a fragile native dependency chain, the pipeline design shifted to extract JPEG previews at ingest — earlier in the flow — reducing the surface area where native RAW parsing was required.

Stage 2: CLIP-IQA Scoring (Blocked → Replaced)

CLIP-IQA was being used for visual quality scoring. It had two problems:

1. Wheel problem: The ARM64 wheel chain for its dependencies was incomplete. Installing it in a Graviton Lambda container was unreliable.

2. Model problem: CLIP-IQA was trained on stock photos, not professional RAW captures. It accepted only ~1% of professional images, making it useless for the actual curation use case.

Resolution: The entire stage was replaced with a rule-based technical validator — exposure clipping detection, blur scoring, sharpness scoring, and perceptual hashing. No ML forward pass. No compiled extensions in the hot path. The new validator ran in pure Python, deployed cleanly on ARM64, and executed in under 100ms per image.

This is the single most important insight from the Parjanya migration:

The fastest path to ARM64 is sometimes not to migrate the workload. It is to replace the workload.

The new parjanya-technical_validator runs on arm64 with Python 3.12 and 512 MB memory. It is not a compromise — it is measurably better: faster, cheaper, more correct, and production-verified.

Stage 3: GPU Inference (Intentional Hard Boundary)

The VLM inference stage runs Qwen3-VL-8B-Instruct and requires CUDA. CUDA is the hard architectural boundary.

CUDA is NVIDIA’s parallel computing platform. It runs on NVIDIA GPUs, which are paired with x86_64 hosts in most production setups including AWS g5 instances. There is no ARM64 CUDA story in Lambda.

Resolution: The GPU inference stage was not migrated. Instead, the control plane was separated from execution:

• A lightweight Lambda — ARM64 — handles scheduling logic: when to scale the GPU fleet, when to drain it

• The actual VLM inference runs on EC2 g5 instances in an Auto Scaling Group

• The Lambda no longer needs ML libraries, so it migrated to Graviton without any dependency issues

This is architectural isolation — moving the ARM64-compatible logic (scheduling, orchestration) onto Graviton and leaving the ARM64-incompatible logic (GPU inference) exactly where it belongs.

Stage 4: Deduplication (Original Success)

Small, CPU-bound, dependency-light. Perceptual hashing with no native C extensions in the hot path. This was the function that migrated cleanly from day one, and it remains the model for what “Graviton-ready” looks like: pure logic, minimal native dependencies, well-supported Python ecosystem.

The Current State

Three of four Lambda functions on Graviton (+1 scheduler). The remaining x86 workload is the one that genuinely belongs there.

Why ARM Wheels Fail: The Root Causes

Understanding why a package lacks an ARM64 wheel helps you predict and resolve failures faster.

1. The package maintainer hasn’t built it yet

Many packages added aarch64 wheel support in 2022–2024 as AWS Graviton gained adoption. Packages that haven’t been updated since then may still not publish ARM64 wheels. This is solvable by checking the current release on PyPI — support often appears in later versions.

2. GLIBC version mismatch

Lambda’s ARM64 runtime uses Amazon Linux 2023. A wheel built against an older GLIBC version (manylinux2014) may not install cleanly in that environment. When you see installation errors that reference GLIBC, this is the cause. The fix is to build your container image from the correct ARM64 base image and verify wheel compatibility inside it — not on your development machine.

3. Transitive dependencies break the chain

Even if the top-level package publishes an ARM64 wheel, its transitive dependencies may not. OpenCV is a canonical example: opencv-python may install, but a transitive dependency on a specific image codec library may not have an ARM64 wheel. The failure surface is the entire dependency tree, not just the packages you explicitly listed.

4. ABI incompatibility

Some packages have version-specific ABI contracts. A package pinned to an older version may have published an x86_64 wheel for that version but not an ARM64 one. When you pin for stability and the pinned version predates ARM64 support, you are stuck — until you unpin.

5. Build-time assumptions in C extensions

Some C extensions were written with x86 assumptions baked into SIMD code paths, memory alignment behaviour, or endianness handling. These may compile on ARM64 but behave incorrectly at runtime. This is rare but dangerous — it is the category where correctness testing matters most.

A Practical Evaluation Framework for Any Product

The following process translates the Parjanya experience into a repeatable assessment for any Lambda-based system.

Phase 1: Dependency Audit (Before Touching Infrastructure)

For every Lambda or containerised service, generate the full dependency tree including transitive dependencies:

pip install pipdeptree
pipdeptree --warn silence

Then, for every package with a native C extension (you can identify these by checking if the installed package contains .so files):

find /path/to/site-packages -name "*.so" | sort

Classify each package against the four-state matrix above. The output is your wheel matrix — one row per native dependency, one column per state.

Phase 2: Build Environment Test (Before Assuming Local Results)

Do not test ARM64 compatibility on your development machine. Build inside an ARM64 environment:

# Docker on an Apple Silicon Mac or a Graviton EC2 instance
docker build --platform linux/arm64 -f Dockerfile.arm64 .

The Lambda execution environment is Amazon Linux 2023 on aarch64. Your build environment should match this exactly. A dependency that installs cleanly in a generic ARM64 Docker image may still fail in the Lambda runtime — validate inside the correct base image.

Phase 3: Runtime Validation (Beyond Installation)

Successful installation is not the same as correct runtime behaviour. For packages that made it through the build:

• Run your actual workload, not a smoke test

• Measure latency and memory usage under ARM64 — they may differ from x86 even when the package installs cleanly

• For any package in the “source build required” category, verify that the compiled binary links against the correct system libraries at runtime

Phase 4: The Three-Question Decision Gate

For each service in your stack, answer these three questions:

1. Does every native dependency have an ARM64 wheel for my Python version and target runtime?

   • If yes: proceed to migration

   • If no: go to question 2

2. Can I build from source reliably in CI, in the correct ARM64 environment, with the correct GLIBC version?

   • If yes: proceed with source-build path; add a CI gate that fails if compilation regresses

   • If no: go to question 3

3. Can I replace the dependency or the entire stage with something ARM64-compatible?

   • If yes: replace it — this is often the better outcome anyway

   • If no: this stage stays on x86; isolate it intentionally rather than blocking the rest of the migration

This decision tree is what the Parjanya migration followed, even if it was not formalised until the second iteration.

Phase 5: Architectural Boundary Declaration

For every stage that cannot move to ARM64, declare the boundary explicitly in your architecture documentation:

• What is the technical reason it stays on x86 (CUDA dependency, missing wheel, untested behaviour)?

• What would need to change for it to become migratable?

• Is the x86 stage fully isolated so that the ARM64 migration of other stages is not blocked by it?

The goal is intentional architecture, not incomplete migration. A GPU inference stage that stays on x86 by design is not a failure. It is a correct architectural boundary.

What This Means for New Products: Building for ARM Portability

If you are building a new product and want to default to Graviton for cost efficiency, the considerations are different from migration. The choices you make at the start of your architecture will determine how much of the system can run on ARM64.

Choose dependency-light stages where possible

The deduplication stage in Parjanya worked on Graviton from day one because it had minimal native dependencies. When designing a new service, ask whether a pure-Python or well-supported-wheel implementation can solve the problem before reaching for a heavier ML or native library.

Treat CUDA as a hard architectural boundary from the start

If a stage requires GPU inference, isolate it behind a clean interface from day one. The scheduling logic, the orchestration, the result handling — all of that can run on ARM64. Only the GPU execution itself needs x86. Design for that split explicitly rather than letting the GPU requirement contaminate surrounding stages.

Version-pin with ARM64 wheel availability in mind

When locking dependency versions for reproducibility, verify that your pinned versions ship manylinux_aarch64 wheels. A version that was pinned for stability before the package added ARM64 support will block migration silently — you will not know until you try to build.

Build in CI on ARM64 from day one

Adding ARM64 build validation to your CI pipeline when you start a project costs almost nothing. Adding it after 18 months of x86-first development costs weeks of debugging. Many packages add and remove ARM64 wheel support across versions; having a CI gate that verifies ARM64 builds on every dependency update catches regressions immediately.

Use the 20% Lambda pricing delta as a design signal

AWS prices ARM64 Lambda at approximately 20% less than x86. That is not just an operational saving — it is a signal about where to route compute. If a function is trivially portable (pure Python, well-supported wheels), choosing x86 without evaluating ARM64 first is a deliberate decision to pay a 20% premium for no benefit.

The AWS-Specific Caveat

Everything in this post is calibrated to AWS. The cost numbers, the Lambda architecture flag, the Graviton family of processors, the manylinux_aarch64 wheel availability for AWS runtimes — these are AWS-specific.

Other cloud providers offer ARM compute. Azure has Ampere A1-based VMs. GCP has Tau T2A. But:

• Serverless ARM support (equivalent to Lambda arm64) is not uniformly available or mature across providers

• The degree of AWS-specific optimisation in the Python scientific and ML ecosystem — particularly for Graviton NEON instructions — is not replicated elsewhere

• Cost structures differ; the 20% pricing delta is an AWS Lambda figure and does not translate directly to other billing models

If you are building a multi-cloud or cloud-agnostic system, treat Graviton optimisation as an AWS-specific tuning layer rather than a foundational architectural decision. The portability of your ARM64-compatible stages will transfer. The AWS-specific performance characteristics will not.

The Real Cost Model

The standard framing of Graviton savings is “20% cheaper Lambda.” That is accurate but incomplete. The full savings model for an architecture like Parjanya’s has four components:

1. Lower per-invocation Lambda cost on ARM64 (~20% pricing delta, compounding with every invocation)

2. Avoided NAT data-path spend — VPC gateway endpoints route S3 and DynamoDB traffic without NAT tax; this is infrastructure rather than ARM64, but it compounds with the migration

3. Reduced ML over-processing — a rule-based pre-filter like parjanya-technical_validator rejects bad images before they reach the GPU queue; every rejected frame is a GPU invocation that never happens

4. Lower operational overhead — fragile native builds create deployment failures, cold start issues, and CI complexity; removing them reduces engineering time

For a high-volume pipeline, component 3 often dominates. The ARM64 savings from the validator are real. But the savings from not running expensive GPU inference on images that would have been rejected anyway are an order of magnitude larger.

This is the correct way to think about Graviton adoption: not as a compute cost reduction in isolation, but as an opportunity to reconsider stage design, dependency choices, and processing flow in ways that compound the savings.

Summary

The ARM wheel is the gatekeeper to Graviton savings. Not the hardware, not AWS support, not your application code. The compiled binary distribution of your Python dependencies — and whether it exists for aarch64 — is what determines whether a given Lambda function can run on Graviton.

The practical conclusions from Parjanya’s full migration arc:

• Audit your wheel matrix before you plan your migration. The output tells you which functions migrate, which need source builds, and which need architectural changes.

• Build in an ARM64 environment that matches your Lambda runtime. Local results on x86 do not predict ARM64 install success.

• Replacement is often faster than migration. A fragile ML stage replaced by a dependency-light rule-based implementation solves the ARM64 problem, the reliability problem, and often the correctness problem simultaneously.

• Isolate x86 hard boundaries explicitly. CUDA is the cleanest example — design around it rather than pretending it is temporary.

• Partial adoption is architecture, not failure. Running 3 of 4 Lambda functions on Graviton with an intentional x86 boundary at the GPU stage is a correct outcome.

Graviton adoption is not a hardware decision. The hardware is already ready. It is an ecosystem decision — and the ARM wheel matrix is where that decision lives.

Further Reading

• AWS Graviton Getting Started – Python — The canonical reference for why Python version and GLIBC version matter for wheel compatibility

• Migrating AWS Lambda functions to ARM-based Graviton2 — AWS’s official guide; read between the lines for what is assumed about dependency simplicity

• AWS Graviton Technical Guide — Deep reference on runtime support and ecosystem maturity by language

• Why We Could Only Use AWS Graviton for 1 of Our 4 ML Lambdas — First Parjanya post: the ecosystem reality check

• From 1/4 to 3/4: Re-architecting an ML Pipeline for Graviton — Second Parjanya post: the re-architecture and cost analysis

• AWS Graviton4 vs Intel x86: From $10 POC to Production Validation — The original POC that established the cost and performance baseline

  ---

  

Abstract

Resilient AI infrastructure is not defined only by model quality, throughput, or scaling efficiency. In production, the harder problem is reconciliation: keeping durable state, requested work, and actual execution aligned when workers fail, queues drift, storage paths change, policies evolve, and control planes become blind. Let me present TBIE, a practical operating model for distributed systems: Truth, Belief, Intent, and Execution. TBIE separates the authoritative record of what is true from the operational view of what seems to be happening, the replayable journal of work that should occur, and the stateless workers that realize side effects.

I have used Parjanya v2.0, an image ingestion and GPU-based visual scoring platform, as a detailed case study. Parjanya’s architecture spans immutable uploads, DynamoDB workflow state, SQS-based intent, event-driven replay, GPU autoscaling, browser-side uploads, policy regrading, and infrastructure drift. Across these layers, TBIE proved useful not as a slogan but as a diagnostic and design framework. It clarified why transient worker failure should not destroy replayable work, why pending state without corresponding intent is a reconciliation gap, why wrong-bucket routing can be deterministically wrong, why policy changes should be handled as controlled replay, and why control-plane blindness is more dangerous than isolated worker failure.

The central claim of this is that resilient AI platforms should be designed as reconciliation systems rather than one-way pipelines. A pipeline moves data forward. A reconciliation system continuously repairs the alignment between truth and work until intended outcomes are actually realized. That distinction changes how we design queues, retries, autoscaling, observability, IAM boundaries, tenant onboarding, model packaging, browser uploads, and recovery runbooks.

1. Introduction

Modern AI systems are often described in terms of model choice, hardware selection, vector stores, and inference efficiency. Those concerns are important, but they are not usually what breaks production systems. In practice, systems fail because they lose work, duplicate work, misroute work, or misinterpret what is actually happening. A queue may look healthy while the underlying work has already vanished. A worker may crash without leaving a replayable record. A browser upload may fail after the backend has already committed a pending state. An autoscaling group may be configured correctly while the reconciliation control plane that should activate it is blind.

These failures are not unusual edge cases. They are the normal shape of distributed systems under stress. The question, then, is not whether a system will experience drift between state and execution, but whether the architecture is built to detect, explain, and repair that drift. TBIE was developed as a response to that question.

TBIE stands for Truth, Belief, Intent, and Execution. It is a practical decomposition of distributed systems that helps separate durable facts from operational signals, requested work from realized work, and recoverable failures from terminal ones. The model is especially useful in asynchronous AI pipelines, where storage, metadata, queues, workers, and control planes all operate independently and can fail independently.

Parjanya v2.0 is an ideal case study because it combines all of those surfaces in a single system. Images are uploaded to S3. Metadata is written to DynamoDB. GPU work is performed asynchronously through SQS. Lambda functions handle technical validation and replay orchestration. GPU workers consume intent, perform visual-language inference, and write enriched truth back to the database. Autoscaling responds to queue depth. Browser uploads interact with tenant-specific buckets and CORS policies. Policy changes require historical reprocessing. Infrastructure drift can occur in launch templates, AMIs, model packages, and endpoint configuration.

Let’s deep dive how TBIE behaves in practice.

2. The TBIE Model

TBIE divides a distributed system into four layers that must remain conceptually distinct.

2.1 Truth

Truth is the durable record of what is actually true. It should survive worker restarts, deploys, transient outages, and runtime drift. In Parjanya, Truth lives in immutable S3 objects, DynamoDB workflow rows, hashes, status fields, curation metadata, and derived state that has been durably committed. Truth answers questions such as: What object exists? What stage of the workflow is this image in? What has already been computed? What is the authoritative record of the tenant, batch, or image?

2.2 Belief

Belief is the operational view of reality. It is derived from Truth and current signals, but it may lag, be partial, or be noisy. In Parjanya, Belief includes queue depth, in-flight message counts, worker health, GPU availability, desired ASG capacity, scaling activity, and alarm state. Belief is what operators see first, but it must not be confused with source-of-record truth.

2.3 Intent

Intent is the durable record of work that has been requested. It is the journal of “please do this work.” In Parjanya, SQS messages represent Intent. Replay jobs, regrade jobs, and retry workflows are all forms of Intent. The critical property of Intent is replayability. If Intent can be lost too early, then transient failures become permanent failures and manual rescue becomes necessary.

2.4 Execution

Execution is the stateless work that turns Intent into side effects and updated Truth. In Parjanya, Execution includes Graviton Lambdas for EXIF extraction and technical validation, GPU workers for VLM inferencing and image quality analysis, replay jobs that reconstruct missing Intent, and autoscaling actions that wake the worker fleet when work appears. Execution is where work happens, but in a resilient system it must never be allowed to erase the record of what was supposed to happen.

3. Why Reconciliation Matters

The key insight behind TBIE is that a distributed system is not healthy merely because its components are running. Health depends on reconciliation. A system can be superficially alive while still being functionally stuck. A database row may say pending_vlm_enrichment while no SQS message exists. A queue may be empty while worker work is still pending in truth. A control plane may be scheduled while unable to read the signals it needs to act.

Reconciliation is the process of bringing Truth, Intent, and Execution back into alignment. In the TBIE model, reconciliation is not a background nicety; it is the core operating principle. Systems should not rely on humans noticing a stale queue and manually fixing it. They should be able to repair the missing relationship between truth and work on their own.

This is why the distinction between a pipeline and a reconciliation system matters. A pipeline assumes a mostly linear progression from input to output. A reconciliation system assumes that progression can be interrupted, delayed, partially completed, or invalidated by failure. It therefore builds explicit mechanisms to recover the intended outcome rather than merely continue from the last known step.

4. Event-Driven Reconciliation in Parjanya v2.0

Parjanya initially used a scheduled polling model. A timer would wake the replay Lambda periodically, the Lambda would scan for pending work, and if appropriate it would emit SQS messages and trigger GPU autoscaling. This worked, but it introduced avoidable latency, wasted invocations, and operational ambiguity. The system could not react immediately when Truth changed, and operators had to reason about whether the queue was empty because the system had genuinely caught up or because the next poll had not yet occurred.

The move to DynamoDB Streams changed that architecture fundamentally. Now, when a workflow row reaches pending_vlm_enrichment, the stream emits an event immediately. An event source mapping filters for relevant writes, batches records briefly, and invokes the replay Lambda. The Lambda discovers missing or retriable work, emits Intent into SQS, and kicks the GPU ASG when necessary. This turns reconciliation from periodic discovery into event-driven reaction.

The result is a lower-latency, more honest architecture. Truth changes, the control plane notices, Intent is created, and Execution follows. The system no longer waits for a timer to rediscover what should already be known.

5. Parjanya v2.0 as a TBIE Case Study

Parjanya’s architecture maps neatly onto TBIE. S3 holds immutable uploads and preview artifacts. DynamoDB holds workflow state, curation metadata, and hashes. SQS holds replayable intent. Lambdas handle validation, replay, and orchestration. GPU workers perform inference and write enriched state back to DynamoDB. Autoscaling follows queue pressure and worker demand.

What makes the case study valuable is not the existence of those components, but the failure modes they exposed. Parjanya demonstrated that many production issues are really reconciliation issues in disguise. Some failures were caused by deleting intent too early. Others were caused by truth existing without intent. Some were caused by the wrong bucket being referenced correctly. Others were caused by the control plane being unable to read queue state. Still others came from stale image tags, missing model files, CORS misconfiguration, or API schema drift. TBIE made those failures legible.

5.1 Transient GPU failure should not destroy Intent

One of the earliest and most important incidents involved GPU workers deleting SQS messages before the retry window had been exhausted. A worker could encounter a temporary S3 download failure or an out-of-memory condition, write a failure state, and still remove the only replayable message. At that point, Truth said the work had failed, but Intent no longer existed and the system had no automatic path to try again.

TBIE made the problem obvious. Execution had destroyed Intent prematurely. The fix was to return explicit execution outcomes: retry, terminal_failure, or completed. A retry preserves the message and lets visibility timeout and receive count determine whether the work should be attempted again. A terminal failure writes durable failed truth and then deletes the message. Success writes final truth and deletes the message. That separation preserves replayability while still allowing durable failure when necessary.

5.2 Truth can be pending while Intent never existed

Another recurring failure mode was the mirror image of the first. Rows in DynamoDB would remain in a pending VLM state, but no corresponding queue message had been created. Queue depth fell to zero and the system appeared idle, yet the backlog still existed in Truth.

This is a reconciliation gap, not a worker bug. The response was to introduce replay mechanisms that scan for pending rows without Intent, regenerate queue messages, and kick the GPU fleet when work is emitted. In other words, the platform needed a control plane that could restore missing Intent from Truth automatically.

5.3 Replaying retryable failures requires resetting Truth deliberately

Not every failure should be treated as terminal. Some failures, such as S3 download errors or temporary GPU OOM events, are retryable if the underlying condition is corrected. But a simple re-enqueue is not enough when the workflow row itself still contains partial or stale derived fields.

TBIE clarifies the difference between retry and replay. A retry is another attempt at the same execution. A replay is a deliberate reset to a pre-execution boundary, removal of stale derived state, and emission of fresh Intent. This distinction matters because the second form is a controlled reconstruction of the workflow, not merely another attempt on top of corrupted state.

5.4 Wrong bucket, right key

One of the most instructive incidents involved an SQS payload that pointed to a valid preview key but the wrong bucket. The worker repeatedly failed with S3 download errors, even though the preview existed. The issue was not flaky execution. It was deterministic misrouting.

This is a classic distributed systems failure: the object key is correct, but the address is wrong. TBIE makes that failure visible because Intent is not just “some work exists.” Intent must encode the correct target. The fix was to force preview keys to use the uploads bucket as the source of truth and to tighten eligibility so only objects suitable for GPU work are enqueued.

5.5 Policy changes should be handled as replay, not patching

When the content policy changed to more strictly reject non-photographic content such as screenshots, banners, infographics, social cards, slides, illustrations, and mockups, the platform needed to reprocess historical images under the new rules. TBIE made the solution obvious: reset the affected Truth fields to a pre-VLM boundary, set the rows back to pending, and emit new Intent so GPU workers can re-grade under the revised policy.

This is one of the most powerful properties of replay-native systems. They allow policy evolution without manual data surgery. A change in judgment does not require a one-off correction script. It becomes a controlled reprocessing event with clear before-and-after semantics.

5.6 Execution drift can occur below the application layer

Parjanya also surfaced failures where the application code was correct but the execution environment was stale. One launch template continued to point to an old image tag. Cloud-init failed while pulling the nonexistent container image, and the GPU instance came up without ever starting the worker.

A separate incident involved an AMI missing the Python files required by a model that relied on remote code. The container started, but the model loader crashed because the local artifact was incomplete. These are both Execution-layer failures, but they occur at different depths: one at instance boot, the other at model runtime. TBIE is useful because it keeps both cases in the same conceptual category: the execution environment drifted away from the declared source of truth.

5.7 CORS and browser execution are part of the resilience story

The ingestion path introduced another class of split-state failure. The backend could write a pending upload record and generate a presigned URL, but the browser’s PUT could still fail because the bucket CORS policy or endpoint routing was incorrect. In that case, Truth at the API layer looked correct while Execution at the browser layer never completed.

Later investigations revealed that multiple independent layers could be wrong at once. A bucket might reject wildcard origins under restricted public bucket settings while the client simultaneously used a global S3 endpoint that returned a redirect browsers would not follow on OPTIONS preflight. TBIE helps here by forcing the diagnostic question: is Intent missing, or is Execution unable to realize it? In this case, Intent existed. Execution was blocked by infrastructure and configuration drift.

5.8 The control plane itself can fail silently

One of the most dangerous incidents in the system occurred when the replay Lambda lacked the permission it needed to read queue attributes. The function was still scheduled and invoked, but it could not inspect Belief and therefore crashed before it could emit Intent or kick the GPU ASG. From the outside, everything looked enabled. In reality, the control plane had gone blind.

This is a crucial TBIE lesson. The component that repairs work is itself a first-class reliability surface. If the control plane silently fails, then every other safety net becomes weaker. That is why control-plane observability must be stronger than ordinary worker observability.

5.9 Dynamic tenant discovery and queue retention are part of reconciliation

As the platform grew, static tenant lists became a bottleneck. Reconciliation could not depend on manually editing Terraform every time a new tenant was onboarded. The control plane therefore moved toward dynamic tenant discovery from DynamoDB, with tenant registry markers written by the upload path and discovered by the replay Lambda at runtime.

Queue retention was another important operational detail. If the GPU fleet is scaled to zero for cost savings, Intent must survive long enough for work to be processed after a weekend or holiday gap. Retention policy is therefore part of resilience design, not a generic queue setting.

6. Operational Model and Runbook Thinking

TBIE becomes truly useful only when it changes how incidents are diagnosed.

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. Operators no longer began by asking whether “the queue is healthy” or whether “the GPU workers are running.” Those signals mattered, but they were no longer treated as authoritative.

Instead, every investigation began with a simpler and much more precise question:

Is the system waiting because Intent is missing, or is Intent present but Execution cannot realize it?

That distinction dramatically narrowed the search space during incidents.

If Truth showed a growing backlog while Intent remained near zero, the issue was usually in reconciliation: replay Lambda failure, queue emission failure, stale tenant discovery, or control-plane blindness.

If Intent existed but Execution failed repeatedly, the problem shifted toward realization: bucket routing, IAM permissions, runtime drift, stale launch templates, model incompatibility, or endpoint mismatch.

The architecture became easier to reason about because TBIE transformed debugging from improvisation into classification.

7. Guardrails and Maintenance as Part of the Architecture

Parjanya’s history shows that resilient systems are built by converting incidents into guardrails. Transient worker failures should not destroy intent. Replay logic should reset truth cleanly before re-emitting work. Dependency drift should be prevented with lockfiles and build assertions. CORS origin drift should be rejected at plan time. Tenant discovery should be dynamic rather than manually maintained. Launch templates should refresh when image tags change. Model artifacts and container versions should be treated as a compatibility pair. API contracts and frontend types should not be allowed to drift silently. Queue retention should match the actual outage window, not an idealized one.

This is the practical meaning of reliability engineering. Incidents become permanent controls. The architecture evolves by accumulating constraints, validations, and replay paths that encode what the system has already learned.

8. Implications for AI Platform Design

TBIE generalizes beyond Parjanya. Any AI platform that spans storage, metadata, asynchronous work, GPU execution, and policy evolution can benefit from the same model. The key is to stop thinking of the system as a one-way pipeline and instead treat it as a reconciliation loop that continuously restores alignment between durable truth and intended work.

That shift affects architecture choices. Queues become journals of intent rather than temporary transport. Replay becomes a first-class product capability rather than an ops workaround. Autoscaling becomes a consequence of work existing, not a heuristic guess. Model packaging becomes part of execution correctness. Browser upload behavior becomes part of the end-to-end reliability model. And policy change becomes a replay event rather than a manual repair task.

When the System Hangs at 02:00 AM

At 02:00 AM, the queue may look empty while customers are still waiting for images to process. GPU instances may exist while no workers are actually running. A replay Lambda may still appear scheduled while silently crashing on every invocation. Dashboards may continue showing green infrastructure while the control plane itself has already gone blind.

This is where TBIE stopped being an architectural abstraction and became an operational model.

The first step was no longer restarting workers or increasing autoscaling limits. The first step became identifying which reconciliation boundary had failed.

The investigation usually began with Truth.

Was DynamoDB accumulating rows in pending_vlm_enrichment? If so, the platform still believed work should exist, regardless of what the queue appeared to show. Truth backlog became the first signal because Truth is the authoritative record of unfinished work.

The next step was Intent.

Did SQS actually contain replayable work items? A high Truth backlog combined with near-zero queue depth usually indicated a reconciliation failure rather than an execution failure. In those cases, the replay Lambda itself became suspect. Either the control plane had stopped emitting Intent, or it had lost the ability to discover the work that needed to exist.

That distinction mattered enormously because the operational response changed completely depending on the answer.

If Intent existed but the backlog still did not move, attention shifted toward Execution health. GPU worker logs became critical. Sometimes the workers were healthy but repeatedly failing with deterministic download errors caused by bucket routing drift. Sometimes GPU instances launched successfully while containers never started because a stale launch template pointed to a nonexistent image tag. In other cases, the instance booted correctly but the model loader crashed because required runtime artifacts were missing from the AMI.

One of the most revealing operational signals turned out to be complete silence.

An empty GPU log group was rarely a good sign. More often, it meant execution had failed before the application layer even started. Cloud-init failures, image pull failures, IAM denial during bootstrap, or launch-template drift frequently surfaced first as absence rather than explicit failure.

The replay Lambda introduced another category of silence.

If queue visibility remained unchanged while pending Truth continued growing, the control plane itself became the focus. One of the most dangerous incidents in the system occurred when the replay Lambda lost permission to read queue attributes. The function was still scheduled. It was still being invoked. But it crashed before emitting Intent or triggering autoscaling. From the outside, everything looked operational while reconciliation had effectively stopped.

That incident permanently changed the operational philosophy of the platform.

Worker failures became treated as recoverable noise.

Control-plane blindness became treated as a systemic risk.

Eventually, the debugging process itself evolved into a TBIE classification exercise:

• Truth backlog revealed whether unfinished work existed.

• Intent backlog revealed whether reconciliation was functioning.

• Queue visibility revealed whether work was flowing.

• GPU log silence revealed bootstrap or execution drift.

• Replay Lambda silence revealed control-plane failure.

• IAM checks validated whether orchestration still had authority to act.

• Launch-template validation confirmed whether execution environments still matched declared infrastructure state.

The system stopped being debugged as “a pipeline.”

It began being debugged as a continuously reconciling distributed system.

That distinction changed how incidents were understood, how recovery was automated, and ultimately how the platform itself evolved.

9. Conclusion

Parjanya v2.0 demonstrates that resilient AI infrastructure depends on more than good models or fast workers. It depends on a clear model for distinguishing what is true, what is believed, what work is intended, and what execution can actually accomplish. TBIE provides that model.

The value of TBIE is not just conceptual. It changes how systems are built, how retries are designed, how control planes are monitored, how infrastructure drift is prevented, and how policy changes are reapplied. Most importantly, it turns production failure from a confusing mystery into a structured reconciliation problem.

That is why the right mental model for modern AI infrastructure is not merely “build a pipeline.” It is “build a system that can continuously reconcile truth and intent until intended work is actually realized.”

Appendix A. Maintenance Checklist

That checklist is not a substitute for architecture. It is how architecture remains trustworthy over time.

1. The Observation That Started This

A few weeks ago I noticed something in my Claude Code session trace: Haiku 4.5 was being invoked as a subagent while my main session was running on Opus 4.7.

My first instinct was: is this personalisation? Has Claude learned my patterns? Is this a new adaptive router?

It was none of those things. But chasing the answer led me into a deep audit of my actual usage data — and what I found changed how I think about AI session economics entirely.

The short answer: Claude Code has a built-in subagent architecture where the main orchestrator (Opus 4.7 in my case) delegates read-only codebase work to the Explore subagent, which runs on Haiku 4.5 by default. It is deterministic product behaviour, not personalisation. The model matching is happening at the subagent dispatch boundary, not through learned routing.

But the longer answer — what I found when I actually pulled my usage data — is the more important story.

2. What the Data Actually Showed

Running /cost in my Claude Code terminal revealed the real picture, however it is only 3% of usage, 33 sessions, complete details are found using ccusage, still showing the important metrics:

Favorite model:  Opus 4.7        Total tokens: 22.0M
Sessions: 33                     Longest session: 7d 2h 33m

● Opus 4.7  (53.3%)   In: 118.2k  · Out: 11.6M
● Opus 4.6  (18.2%)   In: 118.3k  · Out: 3.9M
● Haiku 4.5 (20.6%)   In: 363.5k  · Out: 4.2M
● Sonnet 4.6  (7.9%)  In:  63.0k  · Out: 1.7M

Combined Opus usage: 71.5% of all tokens.

Sonnet — the model I publicly advocated as the benchmark — was only 7.9% of my actual usage.

Running ccusage session revealed the concentration problem:

Two sessions accounted for 50.9% of all costs.

The small sessions with $0.16–0.22/M cost-per-token are my manual Haiku sessions — git commits, README updates, simple lookups. Haiku routing was working correctly there.

The two expensive sessions are a different story entirely. And the cost-per-million ratio is the key signal: at $0.53–0.63/M blended, those sessions were Opus-orchestrated with some subagent delegation, but the token volume itself was the problem — not the model choice.

3. Naming the Problem: Context Debt

Before getting to solutions, I want to name what I observed, because it needs a name.

Context Debt is the accumulation of tokens in a session that compound the cost of every future turn without adding proportional value to the reasoning quality of those turns.

It behaves like technical debt in one critical way: it is invisible when it is forming, and expensive when you finally notice it. Unlike technical debt, it has an immediate dollar cost that compounds within the same session.

Here is the mechanics:

Turn 1:   Opus reads inference_config.yaml    → context: 3k tokens
Turn 10:  Opus reads vlm_pipeline.py          → context: 40k tokens
Turn 30:  Opus reads cuda_kernels/, manifests  → context: 180k tokens
Turn 50+: Every API call sends 180k input     → paying full price
          even if your prompt is 50 words

At Opus pricing (~$15/M input tokens), 200 subsequent turns × 180k context = 36M input tokens = $540 in input alone, before a single output token is generated.

This is what those two sessions were doing. It was not that Opus was the wrong model. It was that the session carried everything it had ever read into every subsequent turn — and paid Opus prices for all of it, repeatedly.

Three forces compound context debt:

Prompt cache expiry. Claude caches the static prefix of your context for 5 minutes (up to 1 hour for some configurations). In a session spanning hours or days, that cache expired repeatedly. Each expiry means paying full Opus input price for the entire accumulated context again. My 7-day session paid full price for a 400k+ context many times over.

Compaction tax. When context compacts (the default fires at ~95% capacity), Claude writes a summary and prepends it to all future turns. In a very long session with multiple compactions, those summaries stack — each subsequent turn carries all of them as mandatory context.

Speculative file reads. Opus tends to read files “to understand the broader context” before answering, even when the current task doesn’t require them. In an ML infrastructure repo, this means model configs, environment files, pipeline definitions, deployment scripts, and CUDA configurations all end up in context — and stay there.

4. The Constraint I Cannot Engineer Away

Before presenting the framework, an honest acknowledgment of constraints.

My primary workload is VLM (Vision-Language Model) inferencing, infra-heavy automation, and ML optimisation pipelines. For this work:

• Minimum viable context is ~300k tokens. Multiple model configs, inference scripts, pipeline definitions, and infrastructure manifests need to be simultaneously present for coherent cross-file reasoning.

• Sonnet 4.6 has only recently gained a 1M context window (now GA, worth testing for execution phases). But adaptive thinking — structurally required for deep VLM reasoning — was absent from Sonnet until recently and is still maturing.

• For the reasoning-heavy phases of this work, Opus 4.7 at 1M context is the correct model. This is a technical requirement, not a preference.

The insight that changes everything: the model being correct does not mean the context footprint needs to be what it currently is.

Opus 4.7 is the right reasoning engine. Sending it 400k tokens of raw source code when it only needs 40k tokens of structured summaries is waste. That is the distinction the framework targets.

5. Context Engineering: The Framework

Context Engineering is the practice of deliberately managing what enters an AI session’s context window, when it enters, and in what form — to maximise reasoning quality per token rather than total tokens.

It has four mechanisms.

Mechanism 1: Pre-Summarisation (Haiku as Reader)

The default pattern:

Opus opens session
Opus reads 30 files to orient itself
All 30 files sit in context for the rest of the session
Cost: 150k input tokens × every subsequent turn

The engineered pattern:

Before session opens:
  @ml-context-loader reads all 30 files (Haiku, separate context)
  Returns: 12k token structured dependency map

Opus main session receives:
  The 12k map, not the 150k raw files
  Opus never directly holds the source files
Cost: 12k input tokens × every subsequent turn

The work output is identical. The context footprint is ~12x smaller.

Subagent configuration (~/.claude/agents/ml-context-loader.md):

---
name: ml-context-loader
description: Pre-read ML pipeline files, model configs, inference scripts,
  and VLM configurations before complex optimisation work. Returns a
  structured dependency map for the main session. Invoke this BEFORE
  starting any ML or infra work in the main session.
model: haiku
tools: Read, Grep, Glob
---
You read ML infrastructure files and produce a structured summary covering:

1. Model architecture and config paths
2. Inference pipeline entry points and key functions
3. CUDA/GPU configuration and memory constraints
4. Cross-file dependencies relevant to the stated task
5. Files that need to change to accomplish the goal

Return a structured map. Do not return raw file contents.
Maximum output: 15k tokens. Be precise and omit irrelevant files.

Paired subagent for infrastructure (~/.claude/agents/infra-scanner.md):

---
name: infra-scanner
description: Scan infrastructure configs, Terraform files, CI/CD pipelines,
  deployment manifests, and environment configs before optimisation work.
  Returns dependency map and change surface for the stated task.
model: haiku
tools: Read, Grep, Glob, Bash
---
Read infra files and return:
- Resource dependencies and environment configs
- Critical paths that affect the stated task
- Files that need to change
- Current state of relevant resources

Compact output only. Omit files irrelevant to the task.

Mechanism 2: Targeted Loading via CLAUDE.md

Even with pre-summarisation, Opus will add raw files to context during execution if not explicitly instructed otherwise. CLAUDE.md is the right place to install session-level context discipline.

Project-level .claude/CLAUDE.md additions:

## Context Discipline

Before reading any file, state why it is needed for the current task.
Never read files speculatively to "understand the broader context."

Use @ml-context-loader before starting any ML or infra work.
Use @infra-scanner before starting any infrastructure work.
Use @explorer for any file discovery or codebase search.

If a file has been summarised by a subagent this session, 
use that summary rather than re-reading the raw file.

Maximum 5 files in direct context at any point during implementation.
For architecture decisions, request a structured summary first.

When asked to fix a bug, read only the file containing the bug
and its direct imports. Do not read the broader codebase unless
the fix requires understanding something not in those files.

This changes Opus’s default behaviour from breadth-first file reading to targeted loading. In practice, this alone reduces speculative reads by 50–60% in long sessions.

Mechanism 3: Session Chunking by Reasoning Type

The most expensive sessions — the two that consumed 50.9% of total costs — mixed two fundamentally different cognitive modes in a single long-running context:

• Reasoning mode: Architecture decisions, bottleneck analysis, choosing optimisation strategies. Needs Opus depth. Short turns, modest context.

• Execution mode: Implementing the decided approach across many files. Needs large context. Lighter per-turn reasoning.

When mixed, the session context grows to serve both needs simultaneously — and every turn pays the full cost of the combined footprint.

Separated:

# Session 1: Architecture planning
# Opus, focused context (summary only), ~15–20 turns
claude --model opus
# Prompt: "Given this summary [paste ml-context-loader output],
#   design the memory optimisation strategy for the VLM dataloader.
#   Output a structured implementation plan."
# → Session ends with a plan document. Context: ~30k tokens throughout.

# Session 2: Implementation
# Sonnet 4.6 at 1M (or Opus if VLM-specific reasoning needed)
# Longer, but following a decided plan — execution, not discovery
claude --model sonnet
# Prompt: "Implement this plan [paste plan].
#   Start with [specific file]. Read only the files listed in the plan."
# → Context grows with implementation files, but no re-discovery overhead.

The Opus session stays short because it worked from a summary, not raw files. The implementation session can run on Sonnet 4.6 at 1M context if the per-turn reasoning does not require Opus depth — which it often does not, once the architecture decision is made.

The practical test for session split: If your current turn is deciding something, you are in reasoning mode. If your current turn is doing something already decided, you are in execution mode. Different sessions.

Mechanism 4: Cache Warming on Re-entry

The single most expensive moment in a long session is returning after a break.

Prompt cache TTL is 5 minutes by default for most Claude Code request types (up to 1 hour in some configurations). When you step away for 30 minutes and return to a 300k-token context, the first turn back pays full Opus input price for all 300k tokens. If your session had 20 such re-entries across a workday, you paid for that 300k context twenty times.

Before taking any break longer than a few minutes:

/compact

This forces a clean compaction summary before the cache goes cold. The next turn back pays for a 5–10k summary re-warm, not a 300k context re-warm.

Prevent the problem structurally by setting aggressive autocompaction:

In .claude/settings.json:

{
  "env": {
    "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "60"
  }
}

Compacting at 60% context capacity means the session never accumulates a context large enough to be expensive to re-warm. The compaction summary costs a few thousand tokens once. The alternative is paying full input price for a massive context every time the cache expires.

6. The Escalation Model

With context engineering in place, the tier routing becomes:

The key principle: Tier selection is determined by reasoning depth required per turn, not by task complexity overall. A complex ML task still has Haiku turns (reading files), Sonnet turns (writing implementations), and Opus turns (designing the approach). The same task should not require every turn to be Opus.

Environment defaults that enforce this without manual switching:

# In ~/.zshrc or ~/.bashrc

# Subagents default to Sonnet, not parent model
export CLAUDE_CODE_SUBAGENT_MODEL="claude-sonnet-4-6"

# Haiku slot pinned explicitly
export ANTHROPIC_DEFAULT_HAIKU_MODEL="claude-haiku-4-5-20251001"

7. Complete Subagent Stack

For teams adopting this framework, the full subagent configuration:

~/.claude/agents/explorer.md — Universal read-only search

---
name: explorer
description: Search files, grep for symbols, read code structure, find
  usages, locate definitions. Use for any read-only investigation
  before editing. Never use for file modification.
model: haiku
tools: Read, Grep, Glob, Bash
---
You are a fast read-only code explorer. Search thoroughly and return
a concise structured summary. Never modify files. Optimise for finding
the minimum set of files relevant to the stated question.

~/.claude/agents/git-helper.md — Repository operations

---
name: git-helper
description: Handle git commits, branch management, README updates,
  changelog entries, and documentation formatting. Use for all
  repository hygiene tasks.
model: haiku
tools: Bash, Read, Write
---
You handle git operations and documentation updates efficiently.
Use conventional commits format. Commit messages: type(scope): description.

~/.claude/agents/code-writer.md — Standard implementation

---
name: code-writer
description: Implement features, write new functions, refactor existing
  code, add tests. Use for code generation and editing tasks that
  do not require architectural reasoning.
model: sonnet
tools: Read, Write, Edit, Bash, Glob, Grep
---
You are a focused implementation engineer. Read the minimum required
files before editing. Make one logical change per invocation.
Follow the plan provided; do not redesign unless asked.

~/.claude/agents/ml-context-loader.md — ML pre-summarisation (See Mechanism 1 above)

~/.claude/agents/infra-scanner.md — Infrastructure pre-summarisation (See Mechanism 1 above)

~/.claude/agents/architect.md — High-stakes reasoning only

---
name: architect
description: Design system architecture, resolve complex multi-file bugs,
  make technology decisions, plan major refactors requiring cross-system
  understanding. Only invoke when deep reasoning is genuinely required.
  Receives summaries, not raw files.
model: opus
tools: Read, Grep, Glob
---
You make high-level decisions from structured summaries.
Think deeply before proposing changes.
Output a structured plan that code-writer can execute step by step.
Request specific raw files only when the summary is insufficient.

8. Before and After: The Numbers

Based on actual session data from this audit:

The two expensive sessions ($625.72 combined API equivalent):

Output tokens are unchanged because the actual work product — the code written, the analysis produced — is identical. The saving is entirely in the context overhead that was being sent to the Opus API on every turn without contributing to reasoning quality.

Across all sessions:

9. What Is Still Missing (Community Ask)

This framework works within current Claude Code capabilities, but several gaps remain that limit how far it can go without manual configuration:

GitHub #44976 — Auto model routing by task type. Currently, there is no native content-aware router in the main session. The routing that exists (Explore → Haiku) operates at the subagent dispatch boundary only. Automatic routing based on turn type — planning turns to Opus, execution turns to Sonnet — requires manual setup or explicit model switching. This feature request, if shipped, would make context engineering automatic rather than configured.

GitHub #43869 — Subagent model routing bugs. On some Max plan configurations, subagents silently run on the parent session’s model despite explicit per-agent configuration. If your Haiku-configured subagents appear to cost Opus rates, this ticket is the reference. Check env | grep CLAUDE_CODE_SUBAGENT_MODEL and verify Haiku is actually firing.

JSONL token counts are unreliable. The ccusage tool reads from JSONL files where input tokens are undercounted by 100–174x due to a streaming placeholder issue. For accurate token accounting, use /cost inside Claude Code (reads from the statusbar, which is accurate) and treat ccusage cost estimates as relative comparisons only.

Sonnet 4.6 at 1M + adaptive thinking. Both are now GA but recently so. For workloads requiring 300k+ context but not frontier VLM reasoning, Sonnet 4.6 at 1M is worth testing as a drop-in replacement for Opus in execution sessions. One session test against a known task will answer whether the quality holds for your specific workload.

10. Best Practices Summary

Before starting any ML or infra session:

• Run @ml-context-loader or @infra-scanner first

• Start with the plan document from a prior architecture session, not a blank slate

• Launch on Sonnet unless you know the first turn needs Opus-depth reasoning

During a session:

• /compact before any break longer than 5 minutes

• Never read files speculatively — enforce this in CLAUDE.md

• Split sessions when you shift from designing to implementing

For VLM and large-context ML work specifically:

• Opus 4.7 at 1M is still the correct model for reasoning-heavy phases

• The saving is in context footprint, not model choice

• Pre-summarise with Haiku; Opus only sees the structured output

On model selection:

• Default: Sonnet 4.6 (set CLAUDE_CODE_SUBAGENT_MODEL="claude-sonnet-4-6")

• Start sessions with claude --model sonnet unless the first task earns Opus

• Use opusplan for sessions that need Opus for planning and Sonnet for execution

On measuring progress:

• Use /cost for accurate token and model-split data

• Use ccusage session --breakdown to track model mix per session over time

• Watch cost-per-million-tokens ratio: $0.20/M = healthy Haiku/cache mix, $1.00+/M = context debt accumulating

Closing Thought

The observation that started this — Haiku 4.5 being spawned during an Opus 4.7 session — turned out to be Anthropic’s own answer to the problem I had been creating. The built-in subagent architecture already encodes the right instinct: read cheap, reason expensive, never conflate the two.

Context Debt is what happens when that separation breaks down at the session level. Context Engineering is the practice of maintaining it deliberately.

The framework is not about using cheaper models. It is about using the right model for each cognitive operation — and never making the expensive model carry the weight of the cheap one’s job.

Related posts:

• The Hidden Runtime Failure Behind Claude’s Recent Regression

• When Adaptive Thinking Goes Off the Rails

• Sonnet Becomes My New Benchmark

• Why you should chose Haiku as Default model and escalate if needed

• Building with Opus 4.6

Additional references:

• GitHub #27665 — Intelligent model routing / 93.8% Opus usage issue

• GitHub #43326 — Auto-select model and effort level based on task complexity

• GitHub #44077 — Skill model field not propagated to Agent sub-invocations

• GitHub #45228 — SendMessage ignores model specified at Agent creation

• GitHub #50574 — Auto model switch

I did not set out to replace a scoring engine with a rule engine. I set out to make the system cheaper, easier to explain, and less fragile as I was not keen on moving to g5.xlarge or g6.xlarge due to prompt sizes increased and would be lot of churn in terms of architectural and infra changes, comes with re testing every functionality and the increase in the costs both the dev costs as well as instances cost even more if non-availability of spot and on-demand, currently using 75/25 for spot and OD for g4.mdn.xlarge (and 2xlarge).

What I eventually learned was that the original design had an architectural flaw: it asked a single opaque model score to do too many jobs at once. It had to decide whether an image was technically valid, whether it was worth sending to the expensive VLM path, whether it should be accepted or rejected, and how to justify that decision later. That worked until it did not. So I split the pipeline into two deterministic layers: a Tier-0 technical validator before the model, and a rule engine after the model. That change dropped cost, improved explainability, and made policy iteration much faster.

The system I started with

Before Parjanya v2.0 architecture v5.3, the pipeline was basically this:

The problem was not that the model was “bad.” The problem was that I had turned a rich decision into a number. Two images with the same score could be fundamentally different: one might be a sharp wildlife shot with a minor distortion, while another might be a soft-focus landscape with no clear subject. The number hid the nuance. Prompt changes also shifted the score distribution, which meant calibration drift could silently change the policy. And when someone asked why an image was rejected, a floating-point score was not a useful answer.

What I changed

I made two structural changes.

How the new decision layer works

The rule engine is deliberately boring in the best possible way. It is not trying to be clever. It is trying to be consistent.

Here is the decision logic in plain language:

Every verdict also gets a version tag and a reason string, so I can audit what happened later. That is a huge shift from the old system, where the final answer was essentially “the score was 0.41.”

Why this mattered operationally

The first benefit was cost.

Here is the cost comparison I care about most:

On my typical tenant workload, GPU spend still dominates the bill, so every image I keep out of the GPU queue is real leverage. I also found one non-obvious cost: at single-tenant scale, about half of the per-drain cost was NAT Gateway egress, not GPU. In the measured v25 drain, NAT cost was actually larger than GPU cost. That means the rule engine improved the decision layer, but infrastructure tuning still matters separately.

The second benefit was iteration speed.

I moved the rule engine through several versions in days, not weeks. The workflow became: edit the rule file, test it on canaries, inspect the reasons, then run the full re-curation if it looked good. No GPU redeploy. No prompt-engine rebuild. No waiting for model infrastructure to churn. In practice, that made policy iteration feel like software engineering again instead of model ops. The v25 drain took 7 hours 41 minutes end to end, but the rule-engine pass over 3,427 rows took less than a minute.

The third benefit was explainability.

I finally had a decision layer that could say things like:

• “reject — subject-critical motion blur”

• “reject — 3 slight defects stack”

• “review — weak subject dominance; technically clean”

That may sound small, but it changes everything when a user asks why something was rejected. A reason string is much more useful than a numeric score.

The fourth benefit was robustness to prompt drift.

When the IQA prompt changed from v16 to v17, the score distribution shifted. Under the old design, that meant threshold recalibration and potential policy drift. Under the new design, the rule engine sat on top of structured fields whose meaning stayed stable across prompt versions. The model could change in ways that improved extraction without forcing me to rethink the whole decision layer.

The measured validation run

The clearest proof came from the v25 rescore.

I had 2,426 images drained through the pipeline, and then 3,427 rows reprocessed by the rule engine. This was not a synthetic benchmark. It was the first large enough live run to give me defensible numbers.

Pipeline results

What stood out to me was not just that the run completed, but that it completed cleanly. There was no message loss, the DLQ stayed at zero, and the autoscaler shut down correctly after the queue emptied.

Rule-engine results

The most interesting transitions were:

That last part mattered to me because it showed the rule engine was not trying to force certainty where none existed. It preserved uncertainty instead of masking it.

Why this architecture works

This pattern is not new in spirit. It is a cascade: a cheap stage rejects obvious failures, and the expensive stage only handles what survives. That reduces inference cost and keeps the scarce GPU path for cases that actually need it. The broader literature supports the same idea, especially for VLM and moderation workloads.

For me, the more important pattern was extract-then-decide. The VLM is the extractor. The rule engine is the decider. Once I separated those roles, the system became easier to reason about. The model could be probabilistic and flexible. The policy could remain deterministic and auditable.

What I watch for now

I did not trade away all the hard problems. I just moved them into better places.

That last item is still open: infrastructure cost can easily dominate the apparent model cost, so I do not want to fool myself into thinking the architecture alone solved everything.

What is next

The next steps are pretty clear to me. Dedup is still shipped but not yet used as a hard gate in production. The rule engine will likely get a v5 cleanup once the remaining fallback paths are collapsed. And I want a dashboard that breaks out cost by Tier-0 reject reason, because it would be useful to know exactly how much pure margin I am saving when corrupt files, blank frames, and blown exposures are filtered early.

The takeaway

I did not move from “AI” to “rules.” I moved from one opaque score to a more honest pipeline.

Now the model extracts structure, the rule engine makes the decision, and the cheap gate keeps obvious failures away from the GPU. That gave me lower cost, faster iteration, and a system I can actually explain to another human without hand-waving. For a production image pipeline, that is the architecture I trust more.

One thing I appreciated only after moving to the rule engine was how much the user experience improved once the system stopped behaving like a black-box scorer.

Previously, the output was essentially a number hidden behind a threshold. After the re-curation architecture, every image started carrying an explainable pipeline trail: technical validation status, VLM enrichment state, structured distortion analysis, composition signals, subject analysis, and finally a deterministic curation verdict with explicit reasoning. The gallery stopped feeling like “an AI guessed a score” and started feeling like a traceable review system.

In the image below, for example, the pipeline clearly shows the two-stage flow: Tier-0 technical validation first, followed by VLM enrichment and structured analysis. The system identifies slight sharpness and focus issues, explains where they occur (“subject lacks critical sharpness on head and eye area”), records composition attributes like framing and depth layers, and still accepts the image because the final rule evaluation determined the defects were minor enough relative to subject structure and overall composition quality.

That distinction matters. Under the old scoring engine, this image could easily have landed near a threshold boundary and become another opaque 0.62 vs 0.58 decision. Under the new architecture, the reasoning is visible, deterministic, and debuggable.

The biggest realization for me was this: the rule engine did not make the system less intelligent. It made the intelligence easier to operationalize.

The VLM still does the hard perceptual work — understanding composition, defects, subject prominence, and scene semantics. But the final decision layer is no longer probabilistic glue hidden behind one floating-point number. It is now an auditable policy system built on top of structured perception.

That separation ended up becoming the real architectural shift.

A few months ago, I wrote about why I believed in a Haiku-first strategy: start with the cheapest and fastest model possible, then escalate only when the task genuinely becomes harder. That idea still makes sense in principle.

But over the last few weeks, while working deeply on my VLM control plane / IQA platform, I realized something important:

The benchmark itself had changed. Probably, the models are sitting across 4.5, 4.6 and 4.7 versions and found it hard to compare (Haiku 4.5 vs Sonnet 4.6 vs Opus 4.7 1M), I meant, gap is widened!

For my current workload, Sonnet 4.6 has become the model I now measure everything against.

Not because it is the smartest model Anthropic offers. Not because it is the cheapest. But because it currently sits at the most practical intersection of:

• reasoning quality

• long-context reliability

• infra-heavy debugging

• ML-heavy orchestration

• coding consistency

• operational cost

• and sustained throughput

That combination matters a lot when you are building systems where the model is not simply “answering questions,” but continuously operating across orchestration, debugging, inferencing workflows, infrastructure decisions, and large-context reasoning.

The original Haiku-first thesis still holds

I still believe the core idea behind the earlier post was correct:

Default to the cheapest model that can reliably complete the task.

Anthropic’s pricing still makes that logic compelling.

At scale, these economics matter enormously.

If you are running:

• agentic workflows

• multi-step debugging

• recursive orchestration

• long-context analysis

• or VLM-heavy pipelines

…output-token costs become very real very quickly.

That is exactly why Haiku originally made so much sense.

But my workload changed

The problem is not that Haiku became “bad.” (also the gap across the models become evident due to version differences, Haiku is still at 4.5 with 200k context, whereas Sonnet and Opus at 4.6 and 4.7 with 1M Context)

The problem is that my workload evolved into something much more infra-heavy and ML-heavy.

The VLM control plane is no longer just lightweight orchestration or simple extraction work. The workflows now involve:

• larger context retention

• deeper architectural reasoning

• distributed infra debugging

• inferencing orchestration

• stateful workflows

• code + infra + ML reasoning in the same session

• long-running debugging loops

• and increasingly large operational context

This is where Sonnet 4.6 started to outperform Haiku dramatically in practical usefulness.

Why Sonnet 4.6 became my benchmark

Anthropic positions Sonnet 4.6 as the “best combination of intelligence, speed, and cost,” and honestly, that aligns closely with my own experience.

Sonnet 4.6 currently provides:

• 1M-token context window

• adaptive thinking

• fast latency

• significantly improved coding quality

• stronger instruction following

• much better long-context coherence

Anthropic also mentioned that in Claude Code testing:

• users preferred Sonnet 4.6 over Sonnet 4.5 roughly 70% of the time

• and even preferred it over Opus 4.5 in many workflows

That honestly tracks with what I have seen.

Sonnet 4.6 feels much more stable in:

• sustained coding sessions

• infra debugging

• architectural iteration

• and orchestration-heavy workflows

without becoming economically painful the way Opus can become.

Haiku 4.5: still useful, but no longer my default

This is probably the most important nuance in the entire post.

Haiku 4.5 is still:

• extremely fast

• extremely cheap

• very capable for its price tier

• excellent for parallelized execution

• strong for extraction/tagging/routing

• surprisingly good on coding benchmarks

Anthropic reports:

• 73.3% on SWE-bench Verified

• 40.21% / 41.75% on Terminal-Bench

Those are genuinely strong results.

But benchmark scores alone do not determine operational usefulness.

In my own experience, Haiku 4.5 now feels degraded for my specific workload.

Not degraded universally.

Degraded for this class of infra-heavy, ML-heavy systems work.

What I started noticing:

• unnecessary follow-up questions

• command thrashing locally

• context drift during debugging

• inefficient iteration loops

• loss of architectural continuity

• excessive operational churn

Sometimes it felt like the model was spending more effort “doing activity” than actually advancing the task.

That was the turning point for me.

Sonnet became the operational center of gravity

So the strategy evolved.

I no longer think in terms of:

“Haiku everywhere unless proven otherwise.”

Now the stack looks more like this:

This feels significantly more stable operationally.

Where Opus still matters

I still escalate aggressively to Opus for:

• architectural redesigns

• distributed systems debugging

• ML-heavy reasoning

• difficult infra changes

• large-context synthesis

• high-stakes production decisions

This is where the Opus models still separate themselves.

Anthropic’s published numbers are notable:

• Opus 4.6: 81.42% SWE-bench Verified

• 90.2% BigLaw Bench

• strong long-context retrieval

• better reasoning across hundreds of thousands of tokens