The original note documented how a combination of EC2 GPU warm pools, Auto Scaling Groups, SQS, EventBridge, DynamoDB Streams, and pre-baked GPU AMIs enabled Parjanya to eliminate SageMaker cold-start penalties, achieve scale-to-zero economics, and build a replay-native control plane capable of self-repair and reconciliation. Rather than treating GPU inference as a model-serving problem, the architecture reframed it as an orchestration and reconciliation problem, resulting in dramatically lower startup latency, parallel execution, improved replay safety, and reduced operational cost.

This article expands on that foundation by examining the complete system architecture, the TBIE (Truth, Belief, Intent, Execution) operating model, the move from polling to event-driven reconciliation, and the real production incidents that ultimately shaped the platform’s reliability model.

EXECUTIVE SUMMARY

Parjanya v2.0 is Phagyul AI’s image ingestion and GPU-based visual scoring, later evolved as curation platform by replacing score engine to rule engine.

Its nightly VLM inference pipeline was originally built on Amazon SageMaker. The pipeline processed photographer uploads using Qwen3-8B-VLM and worked — but it revealed a fundamental architectural mismatch over time. Wall-clock time regularly stretched to ~11 hours. Cold starts ate into every nightly window. Replay handling for failed enrichments grew fragile. Sequential execution prevented scale-out. And persistent managed-platform overhead stopped making financial sense for burst-only workloads.
The core insight that drove the migration was a reframing of the problem. The question was no longer

‘how do we run a model?’

It became:

‘how do we orchestrate burst-scale inference reliably, replay safely, and cost-efficiently?’

That reframing led to retiring SageMaker and rebuilding the pipeline entirely from native AWS primitives — EC2 GPU warm pools, Auto Scaling Groups, SQS-driven orchestration, EventBridge-triggered scale-out, and pre-baked GPU AMIs with Qwen3-8B-VLM weights embedded.

This is a detailed internal engineering deep dive into that migration. It covers the original architecture and its failure modes, the design decisions behind each new component, how TBIE (Truth, Belief, Intent, Execution) reconciliation patterns were applied to make the system replay-safe and operationally resilient, the specific production incidents encountered and how each shaped the architecture, and the guardrails now embedded to prevent regression.
Key outcome: the rebuilt system achieved near-zero or sub-minute startup latency, parallel GPU workers replacing sequential processing, event-driven replay handling, scale-to-demand orchestration, and drastically reduced idle GPU waste. More importantly, it became a system that could reason about and repair its own state — a reconciliation system, not just a pipeline.

BACKGROUND: WHAT IS PARJANYA V2.0?

Parjanya v2.0 is the image ingestion and GPU-powered visual scoring platform at Phagyul AI Systems. Photographers upload raw images; the platform extracts technical metadata (EXIF, resolution, format), performs visual-language model inference to produce content quality scores, enforces content policy, and writes enriched metadata back to the workflow database.

The data flow is:

• Photographers upload images to per-tenant S3 buckets via browser-based presigned URLs → An S3 event triggers a Graviton Lambda for EXIF extraction and technical validation.

• On successful validation, a DynamoDB workflow row is written with status pending_vlm_enrichment.

• A GPU worker consumes this work, runs Qwen3-8B-VLM inference, and writes scored metadata back(later recuration) to DynamoDB.

• A control plane (replay Lambda, reconciliation loop) monitors for stalled or missing work and re-orchestrates as needed.

• The platform spans immutable uploads (S3), workflow state (DynamoDB), intent queues (SQS), Lambda orchestration, GPU autoscaling (ASG + EC2), browser-side uploads with CORS policies, content policy enforcement, and tenant isolation.

THE ORIGINAL SAGEMAKER PIPELINE

Parjanya’s first VLM inference implementation used Amazon SageMaker as the compute platform. A nightly batch job was triggered on a schedule. SageMaker endpoints were provisioned, images were submitted for inference in sequence, results were collected, and endpoints were torn down.

The original system worked. It produced correct results. But it had structural problems that compounded as the platform grew:

• Cold-start tax: Every nightly run incurred GPU cold starts before the first image could be processed.

• Sequential execution: Image inference ran sequentially across the batch; there was no parallelism at the worker level.

• Fragile replay: Failed enrichments required increasingly fragile custom replay logic layered on top of SageMaker’s job model.

• Idle cost: The platform was billed for managed infrastructure overhead even during windows with no work to do.

• Throughput ceiling: Wall-clock time regularly stretched to approximately 11 hours per nightly run.

THE ARCHITECTURAL REFRAMING

The decision to migrate was not triggered by a single catastrophic failure. It was triggered by a change in how the problem was understood.

Once the problem was stated that way, SageMaker was the wrong tool. SageMaker is an excellent managed platform for running models. It is not designed to be a burst-scale GPU orchestration layer with event-driven replay. Those properties had to be bolted on from outside, which is why the replay logic was fragile and why cold start latency was unavoidable.
Native AWS primitives offered a better fit for the actual problem: EC2 GPU instances could be pre-baked with model weights, Auto Scaling Groups could scale to demand rather than a fixed schedule, SQS provided durable, replayable intent, and EventBridge could trigger orchestration the moment new work appeared. The result is not a simpler system — it is a more intentional one.

THE NEW ARCHITECTURE: NATIVE AWS- PRIMITIVES

EC2 GPU WARM POOLS

Rather than provisioning GPU instances from scratch each time a job runs, Parjanya v2.0 uses EC2 warm pools attached to the Auto Scaling Group. Instances are pre-initialised and held in a stopped or standby state. When the ASG scales out, warm pool instances start in seconds rather than minutes, eliminating the cold start latency that dominated the original SageMaker pipeline.

Warm pool design note: instances in the warm pool must be pre-baked with model weights already on disk. Do not rely on model download during instance boot — download failures become silent launch failures. The AMI is the contract between infrastructure and runtime.

PRE-BAKED GPU AMIs WITH QWEN3-8B-VLM WEIGHTS

Each GPU AMI is built with the Qwen3-8B-VLM model weights, runtime dependencies, and worker application code baked in. The AMI is immutable. When a new model version or dependency is needed, a new AMI is built and the launch template is updated. This makes the execution environment reproducible and testable before deployment.

Key constraints on AMI design:

All model weights must be present at AMI build time — no remote download at boot.
Python environment, CUDA libraries, and worker scripts must be included in the AMI.

The launch template image tag and the AMI must be kept in sync; stale launch templates are a production risk (see Incident 6).

Container and model artifact versions are a compatibility pair — they must be validated together.

AUTO SCALING GROUPS (ASG)

An Auto Scaling Group manages the GPU worker fleet. The ASG is configured with a minimum of 0 instances (scale-to-zero for cost efficiency) and a maximum that reflects the burst capacity needed during active processing windows. Desired capacity is driven by queue depth rather than a schedule.

ASG configuration decisions:

1. Scale-out is triggered by the replay Lambda when SQS queue depth exceeds threshold.

2. Scale-in is time-based to prevent premature termination of in-flight workers.

3. Instance type selection balances GPU memory requirements (Qwen3-8B-VLM needs sufficient VRAM) against cost per inference.

4. Warm pool instances buffer the latency between a scale-out trigger and first inference.

SQS-DRIVEN ORCHESTRATION

SQS is the durable intent layer. Each message represents a single VLM inference job for one image. Messages are not ephemeral transport — they are the replayable journal of work that must happen. This is a TBIE design principle: intent must survive transient failure.

SQS configuration decisions:

• Visibility timeout is set to exceed the maximum realistic inference time to prevent duplicate processing.

• Queue retention is set to exceed the maximum expected outage window (e.g., weekend + holiday gap) so intent survives scale-to-zero periods.

• A dead-letter queue captures messages that exceed the maximum receive count, enabling manual triage of terminal failures.

• Message attributes carry routing metadata: tenant ID, image key, source bucket, expected output location.

• Queue retention is not a generic operational setting. It is a resilience parameter. If the GPU fleet is at zero for 3 days, messages must still be present when the fleet wakes up. Set retention to match your worst-case outage window, not a default.

EVENTBRIDGE-TRIGGERED SCALE-OUT

Scale-out is event-driven via EventBridge and DynamoDB Streams. When a DynamoDB row reaches pending_vlm_enrichment, the stream emits an event. A stream processor filters and batches records, then invokes the replay Lambda. The replay Lambda evaluates queue state (Belief) against database state (Truth), emits SQS messages (Intent), and increases ASG desired capacity (Execution trigger).
This replaces a polling model where a scheduled timer would periodically scan for work. The event-driven approach gives lower latency, eliminates wasted invocations during quiet periods, and makes the system’s reaction to new work causally clear:

Truth changes → control plane notices → Intent is created → Execution follows.

REPLAY LAMBDA (CONTROL PLANE)

The replay Lambda is the reconciliation control plane. It is invoked by DynamoDB Streams and by periodic safety-net schedules. Its job is to compare Truth (DynamoDB workflow state) against Intent (SQS queue depth) and repair any divergence: emitting missing messages, resetting stale state, and triggering ASG scale-out when work is present.
The replay Lambda is a first-class reliability surface, not a background helper. If it fails silently, every other safety net becomes weaker. Its observability must be stronger than ordinary worker observability.
Monitor its error rate, execution duration, and whether it successfully reads queue attributes. A lambda that runs but cannot see queue state is functionally blind.

TBIE: TRUTH, BELIEF, INTENT, EXECUTION

TBIE is the operating model used to design, debug, and evolve Parjanya v2.0. It decomposes a distributed system into four conceptually distinct layers. The model is most valuable as a diagnostic tool: when the system behaves unexpectedly, TBIE provides a structured vocabulary for identifying which layer has diverged.

TRUTH

Truth is the durable, authoritative record of what is actually true. It must survive worker restarts, deploys, transient outages, and infrastructure drift.
In Parjanya v2.0, Truth lives in:

S3 — immutable image uploads and preview artifacts. If an object is in S3, it exists.

DynamoDB — workflow rows recording status, EXIF metadata, VLM scores, curation state, and hashes.

Derived committed state — VLM output written durably after successful inference.

Truth answers questions like:

• What objects exist?

• What stage is this image in?

• What has been computed?

• What is the authoritative record for this batch?

BELIEF

Belief is the operational view of reality. It is derived from Truth and current signals, but it may lag, be partial, or be noisy. Belief is what operators see first — but it must never be confused with source-of-record Truth.

In Parjanya, Belief includes:

1. SQS queue depth and in-flight message counts.

2. ASG desired capacity, running instance count, and warm pool state.

3. Worker health signals and CloudWatch alarms.

4. GPU availability and instance lifecycle state.

A healthy Belief is not the same as a healthy system. The queue can appear empty while Truth still holds a large pending backlog. That gap is a reconciliation failure, not an idle system.

INTENT

Intent is the durable record of work that has been requested. It is the replayable journal of ‘please do this work.’

The critical property of Intent is replayability: if Intent can be lost too early, transient failures become permanent failures requiring manual rescue.

In Parjanya, Intent is:

• SQS messages — each representing a single VLM inference job.

• Replay jobs — reconstructed Intent emitted for rows that had pending Truth without corresponding queue messages.

• Regrade jobs — Intent emitted for historical reprocessing after policy changes.

EXECUTION

Execution is the stateless work that turns Intent into side effects and updated Truth. Execution is where work happens, but in a resilient system it must never be allowed to erase the record of what was supposed to happen.

In Parjanya, Execution includes:

• Graviton Lambdas for EXIF extraction and technical validation.

• GPU workers running Qwen3-8B-VLM inference and writing scored results to DynamoDB.

• Replay Lambda emitting Intent and triggering ASG scale-out.

• ASG and EC2 actions that instantiate the worker fleet.

TBIE AS A DIAGNOSTIC MODEL

The power of TBIE is operational. Once internalised, it changes the first question asked during any incident. The question is no longer ‘is the queue healthy?’ or ‘are the GPU workers running?’ Those are Belief signals. The real question is:

Is the system waiting because Intent is missing — or because Intent is present but Execution cannot realise it?

That single question narrows the search space dramatically and prevents wasted time on the wrong layer.

EVENT-DRIVEN RECONCILIATION

Parjanya initially used a scheduled polling model. A timer would wake the replay Lambda periodically. The Lambda would scan for pending work and, if appropriate, emit SQS messages and trigger GPU autoscaling. This worked but introduced avoidable latency, wasted invocations, and operational ambiguity.
The move to DynamoDB Streams changed the architecture fundamentally. Now, when a workflow row reaches pending_vlm_enrichment, the stream emits an event immediately. An event source mapping filters for relevant writes, batches records briefly, and invokes the replay Lambda. The Lambda discovers missing or retriable work, emits Intent into SQS, and kicks the GPU ASG when necessary.

The net effect:

Reconciliation latency dropped from O(poll_interval) to O(stream_propagation) — typically seconds.

• Wasted Lambda invocations during idle periods eliminated.

• Causal chain made explicit: Truth changes → stream emits → Lambda reacts → Intent created → Execution follows.

• Operators can reason about system state without wondering ‘has the next poll happened yet?’

• A reconciliation system does not rely on humans noticing a stalled queue and manually fixing it.

It detects, explains, and repairs the divergence between Truth and Intent on its own. Event-driven streaming is what makes that self-repair fast enough to be useful in production.

PRODUCTION FAILURE MODES & LESSONS

The following incidents occurred during the development and operation of Parjanya v2.0. Each is documented here not as a post-mortem but as a design lesson — how the failure shaped the architecture, and what guardrail was added as a result. TBIE is used throughout as the diagnostic lens.

Incident 1: Transient GPU Failure Destroying Intent

GPU workers were deleting SQS messages before the retry window had been exhausted. A worker could encounter a temporary S3 download failure or an OOM condition, write a failure status to DynamoDB, and still delete the queue message. At that point, Truth said the work had failed, but Intent no longer existed — the system had no automatic path to retry.

TBIE diagnosis: Execution destroyed Intent prematurely.

Resolution:

Workers now return explicit outcome signals: retry, terminal_failure, or completed.

retry: preserve the message. Visibility timeout and receive count govern re-attempts.

terminal_failure: write durable failed Truth, then delete the message.

completed: write final Truth, then delete the message.

That separation preserves replayability while still allowing durable failure when necessary. The message deletion decision became a function of execution outcome, not a side effect of worker exit.

Worker outcome protocol

if outcome == ‘retry’:
# Do NOT delete. Let visibility timeout expire.
return # SQS will re-deliver
elif outcome == ‘terminal_failure’:
write_durable_failure(db, image_id)
sqs.delete_message(receipt_handle)
elif outcome == ‘completed’:
write_enriched_truth(db, image_id, scores)
sqs.delete_message(receipt_handle)

Incident 2: Pending Truth Without Intent (Reconciliation Gap)

DynamoDB rows remained in pending_vlm_enrichment status, but no corresponding SQS messages existed. Queue depth showed zero. The system appeared idle. The backlog still existed in Truth but was invisible to Execution.

TBIE diagnosis: Truth was pending while Intent never existed. Classic reconciliation gap.

Resolution:

• The replay Lambda now scans for pending rows without corresponding queue messages.

• For each orphaned pending row, it regenerates a queue message and kicks the GPU ASG.

• DynamoDB Streams ensure this scan is triggered promptly rather than waiting for a polling cycle.

• The key insight: this is not a worker bug. Workers cannot fix work that was never queued. The control plane must own the responsibility of keeping Truth and Intent aligned.

Incident 3: Retry vs. Replay Distinction

Some failures — S3 download errors, temporary GPU OOM — are retryable once the underlying condition is fixed. But simply re-enqueueing the message is insufficient when the workflow row contains partial or stale derived fields from the failed attempt.

TBIE clarification:

a retry is another attempt at the same execution. A replay is a deliberate reset to a pre-execution boundary, removal of stale derived state, and emission of fresh Intent.

Resolution:

Retry: re-attempt on the existing message, preserving current workflow state.

Replay: reset the DynamoDB row to a clean pre-VLM boundary (clear stale score fields, VLM status, partial outputs), then emit a fresh SQS message.

Replay is a controlled reconstruction, not just another attempt on top of corrupted state. The distinction prevents accumulating corrupted derived fields across multiple failed attempts.

Incident 4: Wrong Bucket, Right Key (Deterministic Misrouting)

An SQS payload pointed to a valid preview key but in the wrong bucket. The worker repeatedly failed with S3 download errors even though the file existed — just not at that address. This was deterministic misrouting, not flaky execution.

TBIE diagnosis: Intent encoded the wrong target. Not an Execution failure — an Intent construction failure.

Resolution:

Preview keys now canonically use the uploads bucket as the authoritative source.
Intent construction validates bucket routing before emitting to SQS.
Eligibility checks were tightened: only objects confirmed present in the correct bucket are enqueued.
The lesson generalises: Intent must encode the correct target, not just some valid key. A message that is ‘almost correct’ will fail deterministically and indefinitely until the Intent itself is repaired.

Incident 5: Policy Change Requiring Historical Replay

Content policy was updated to more strictly reject non-photographic content: screenshots, banners, infographics, social cards, slides, illustrations, and mockups. Historical images scored under the old policy needed to be re-evaluated.

TBIE framing: policy change = controlled replay, not a patching exercise.

Resolution:

Affected rows were identified by policy version marker in DynamoDB.
Each affected row was reset to a pre-VLM boundary: stale score fields cleared, VLM status reset to pending.
Fresh Intent was emitted into SQS so GPU workers re-scored under the new policy.
This is one of the most powerful properties of a replay-native system: policy evolution does not require one-off correction scripts. It becomes a controlled reprocessing event with clear before-and-after semantics. The same replay machinery used for failure recovery also handles deliberate re-evaluation.

Incident 6: Execution Drift Below the Application Layer

Two distinct incidents exposed failures where application code was correct but the execution environment was stale.

6a: Stale Launch Template (Image Tag)

A launch template continued to reference an old image tag that no longer existed. Cloud-init failed when pulling the nonexistent container image, and the GPU instance came up without ever starting the worker. The worker log group was empty — the application layer never ran.
Complete silence in the GPU log group is a critical signal. It usually means execution failed before the application layer started: image pull failure, IAM denial during bootstrap, or cloud-init crash.

6b: Incomplete AMI (Missing Model Artifacts)

A separate incident involved an AMI that was missing the Python files required by Qwen3-8B-VLM (a model that relies on remote code). The container started successfully, but the model loader crashed because local artifacts were incomplete.

Resolution — both incidents:

• Launch templates now include version assertions checked at deployment time.
  AMI build pipelines validate the presence and integrity of all model artifacts before publishing.

• Container version and model artifact version are treated as a compatibility pair, validated together.

• Deployment tooling refreshes launch templates automatically when image tags change.

Incident 7: CORS and Browser Upload Split-State Failure

The backend could write a pending upload record and generate a presigned URL, but the browser’s PUT could fail because of CORS misconfiguration. In that case, Truth at the API layer looked correct while Execution at the browser layer never completed.
Investigation revealed multiple independent configuration failures could co-exist: a bucket rejecting wildcard origins under restricted public bucket settings, while the client simultaneously used a global S3 endpoint that returned redirects that browsers would not follow on OPTIONS preflight.

TBIE diagnosis:

Intent existed (presigned URL generated, pending row written). Execution was blocked by infrastructure configuration drift. Two independent misconfiguration layers were both wrong simultaneously.

Resolution:

• CORS policies are version-controlled and validated in CI before deployment.

• Bucket CORS and public access settings are treated as a matched pair, not configured independently.

• Upload path includes client-side detection of presigned URL PUT failure with structured error reporting.

• Pending rows created without a completed upload are detected by the replay Lambda and age-gated before re-emission.

Incident 8: Control Plane Blindness (Replay Lambda IAM Failure)

The replay Lambda lacked permission to read SQS queue attributes. The function continued to be scheduled and invoked, but it crashed before it could inspect Belief (queue depth) or emit Intent. From the outside, everything appeared enabled. In reality, the control plane had gone blind.

This is the most dangerous failure mode in the system. A silently failing control plane means every other safety net weakens simultaneously. Worker failures that should self-repair accumulate without rescue. Operators see green infrastructure dashboards while reconciliation has effectively stopped.

Resolution:

• The replay Lambda’s IAM policy now explicitly grants sqs:GetQueueAttributes, sqs:GetQueueUrl, and sqs:SendMessage.

• IAM policy drift is checked in deployment assertions — the Lambda will not deploy if required permissions are absent.

• The Lambda now emits a structured heartbeat metric on every successful invocation, including after inspecting queue state.

• Alarm on absence of heartbeat metric: silence means the control plane has likely failed.

The operational philosophy shift from this incident: worker failures are recoverable noise. Control-plane blindness is a systemic risk. Observability on the control plane must be stronger than on individual workers.

Incident 9: Dynamic Tenant Discovery and Queue Retention

As the platform grew, static tenant lists became a bottleneck. Reconciliation could not depend on manually editing infrastructure configuration every time a new tenant was onboarded. Additionally, when the GPU fleet was scaled to zero for cost savings, queue retention was set too short — messages expired before the fleet woke up.

Resolution:

The control plane moved to dynamic tenant discovery from DynamoDB. Tenant registry markers are written by the upload path and discovered by the replay Lambda at runtime.
Queue retention policy was extended to match the maximum expected outage window, not an idealised constant.
New tenant onboarding now automatically creates the required DynamoDB registry marker without manual infrastructure changes.

OPERATIONAL MODEL AND RUNBOOK THINKING

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. The starting question is no longer ‘is the queue healthy?’ or ‘are GPU workers running?’ — those are Belief signals. Every incident investigation now begins with a TBIE classification:

THE DIAGNOSTIC SEQUENCE

• Step 1: Check Truth

Is DynamoDB accumulating rows in pending_vlm_enrichment? If so, the platform believes unfinished work exists, regardless of what the queue shows. A growing Truth backlog is the authoritative signal that the system is behind.

• Step 2: Check Intent

Does SQS contain replayable work items? High Truth backlog + near-zero queue depth = reconciliation failure, not execution failure. The replay Lambda itself becomes suspect. Either it stopped emitting Intent, or it lost the ability to discover the work.

• Step 3: Check Execution

If Intent exists but the backlog still does not move, shift to Execution health. GPU worker logs become critical. Check for:

1. Deterministic S3 download errors (wrong bucket routing — see Incident 4).
   Repeated OOM crashes (model too large for instance type).

2. Complete log group silence (execution failed before the application layer — check cloud-init).

3. Stale launch template (image tag no longer exists — check ASG launch configuration).

• Step 4: Check the Control Plane

If queue visibility is unchanged while pending Truth continues growing, the replay Lambda itself is the focus. Check: Is it being invoked? Is it completing successfully? Can it read sqs:GetQueueAttributes? Is it emitting its heartbeat metric?

Operational summary of diagnostic signals:

• Truth backlog growing → unfinished work exists

• Intent near zero despite Truth backlog → reconciliation failure

• Intent present but Execution failing → runtime or routing failure

• GPU log silence → bootstrap failure (cloud-init, image pull, IAM at boot)

• Replay Lambda silence → control-plane failure (IAM, crash, permission gap)

• No heartbeat metric → control plane has gone blind

INCIDENT RESPONSE DECISION TREE

At 02:00 AM, when the queue looks empty but customers are waiting:

1. Check Truth: pull DynamoDB pending count. If non-zero, the system is behind regardless of queue state.

2. Check replay Lambda logs: has it run in the last cycle? Did it complete? Did it emit messages?

3. Check SQS: are messages present? If Truth is high and SQS is low, replay Lambda is suspect.

4. Check ASG desired capacity: did the Lambda set it correctly? Check CloudWatch ASG activity log.

5. Check GPU worker logs: any entries? Silence means boot failure. Errors usually indicate routing or model issues.

6. Check IAM: can the Lambda call sqs:GetQueueAttributes? Can the worker read the correct S3 bucket?

7. Check launch template: does the image tag still exist? Pull the AMI manifest.

8. OUTCOMES
   The migration from SageMaker to native AWS primitives, combined with TBIE reconciliation, changed both the cost profile and the operational character of the pipeline.

OPERATIONAL IMPROVEMENTS

• Latency: Startup latency: near-zero or sub-minute (down from 20–40 minutes cold start on SageMaker).

• Parallelism: Workers process images in parallel rather than sequentially — throughput scales with fleet size.

• Throughput: Wall-clock time reduced dramatically from the ~11 hour nightly baseline.

• Replay: Event-driven replay handles failures automatically without manual intervention for most failure classes.

• Cost: Scale-to-zero when no work is present; burst capacity available within seconds via warm pools.

ARCHITECTURAL IMPROVEMENTS

The system can repair its own state: Truth without Intent is detected and corrected by the control plane.
Policy changes are handled as replay events — no custom migration scripts needed.
Tenant onboarding is dynamic — no infrastructure changes required per tenant.
Execution environments are immutable and reproducible via pre-baked AMIs.
Control plane observability now triggers independent alerting paths from worker failures.

THE BROADER LESSON

Not every AI workload needs a managed ML platform. Sometimes the real engineering challenge sits in orchestration, reconciliation, and operational resilience — not in model serving. When the problem is ‘how do we run a model?’, SageMaker is the right answer. When the problem is ‘how do we orchestrate burst-scale inference reliably, replay safely, and cost efficiently?’, native primitives and a clear reconciliation model are the right answer.

GUARDRAILS, MAINTENANCE, AND OPERATIONAL CHECKLIST

Parjanya’s reliability is not a property of its initial design. It is the accumulated result of incidents converted into permanent controls. The following checklist encodes what the system has already learned.

INTENT PRESERVATION

• Workers never delete SQS messages unless the outcome is terminal_failure or completed.

• Visibility timeout exceeds maximum realistic inference time.

• Queue retention exceeds maximum expected outage window.

• Dead-letter queue is monitored and reviewed regularly.

EXECUTION ENVIRONMENT INTEGRITY

Launch template image tags are validated against existing AMIs at deployment time.

AMI build pipelines assert presence of all model artifacts before publishing.

Container version and model artifact version are validated as a compatibility pair.

Launch templates are refreshed automatically when image tags change.

Python lockfiles are used to prevent dependency drift inside worker environments.

CONTROL PLANE HEALTH

Replay Lambda emits a structured heartbeat metric on every successful execution.

Alarm fires if heartbeat is absent for more than one poll cycle.

IAM policy grants for the Lambda are validated in deployment assertions.

Lambda error rate and duration are independently alarmed.

Control-plane CloudWatch logs are retained and searchable for incident forensics.

RECONCILIATION INTEGRITY

Truth backlog (pending DynamoDB rows) is monitored as a primary SLI.
Intent gap (pending Truth with no corresponding queue message) is checked by the Lambda on each invocation.

Replay logic resets Truth cleanly to a pre-execution boundary before re-emitting Intent.

Tenant discovery is dynamic — no manual configuration changes needed on onboarding.

Queue retention policy change requires explicit engineering review.

INFRASTRUCTURE DRIFT PREVENTION

CORS policies are version-controlled and validated in CI.

Bucket CORS and public access settings are configured as a matched pair.

API schema and frontend types have drift detection in CI.

S3 bucket routing is validated at Intent construction time, not only at worker download time.

POLICY AND REGRADE

Content policy version is stored per image in DynamoDB.

Policy changes trigger replay jobs for images scored under previous policy versions.

Regrade jobs use the same replay machinery as failure recovery — no custom tooling.

IMPLICATIONS FOR AI PLATFORM DESIGN

TBIE generalises beyond Parjanya. Any AI platform that spans storage, metadata, asynchronous work, GPU execution, and policy evolution benefits from the same model. The key is to stop thinking of the system as a one-way pipeline and treat it as a reconciliation loop that continuously restores alignment between durable truth and intended work.

That shift changes concrete architecture choices:

Queues: Queues become journals of intent rather than temporary transport.

Replay: Replay becomes a first-class product capability, not an ops workaround.

Autoscaling: Autoscaling becomes a consequence of work existing, not a heuristic guess.

Model packaging: Model packaging becomes part of execution correctness, not just deployment.

Browser uploads: Browser upload behaviour becomes part of the end-to-end reliability model.

Policy change: Policy change becomes a replay event rather than a manual repair task.

The right mental model for modern AI infrastructure is not ‘build a pipeline.’ It is ‘build a system that can continuously reconcile truth and intent until intended work is actually realised.’

APPENDIX A: COMPONENT MAPPING TABLE

APPENDIX B: KEY TERMS

The framework emerged from a simple observation:

Distributed systems rarely fail through catastrophic outages.

More often, they fail through divergence.

Truth says one thing.
Execution does another.
Intent disappears.
Belief becomes detached from reality.

TBIE gave me a vocabulary for understanding those failures.

What I did not fully appreciate at the time was the architectural consequence of taking TBIE seriously.

I thought I was building an AI pipeline.

I ended up building a platform.

Its primary responsibility was preserving correctness.

That realization fundamentally changed how I thought about reliability.

The hard problems in Parjanya v2.0 were not model problems.

They were coordination problems.

• Not GPU problems.

• Not inference problems.

• Not even scaling problems.

The hardest problems emerged whenever the system lost alignment between:

• what was true,

• what work should exist,

• what execution was currently happening,

• and what the system believed was happening.

The original architecture looked deceptively simple.

Images landed in S3.

Metadata was written into DynamoDB.

Validation ran through Lambda functions.

GPU workers performed VLM enrichment and IQA processing.

Results were written back into DynamoDB.

Autoscaling reacted to queue depth.

At first glance, it looked like a fairly conventional event-driven AI pipeline.

Production quickly proved otherwise.

Workers crashed.

Queues drained without progress.

Pending images remained pending forever.

Retries became impossible.

Infrastructure drifted away from declared state.

The system could remain technically healthy while doing absolutely nothing useful.

That was the moment I realised we were solving the wrong problem.

The challenge was not moving images through a pipeline.

The challenge was continuously reconciling distributed state.

Once viewed through that lens, the architecture began evolving in a very different direction.

The replay engine stopped being a utility.

It became the control plane.

Queues stopped being transport mechanisms.

They became durable Intent.

Autoscaling stopped being a scaling feature.

It became a realization layer for Intent.

And the system itself stopped behaving like a pipeline.

It started behaving like a platform.

This article is the story of that evolution.

Preface

Distributed systems rarely fail in the way architecture diagrams suggest.

The boxes stay alive. The queues still exist. The databases continue accepting writes. The autoscaling groups look healthy. The dashboards even remain green.

And yet the system silently stops doing useful work.

An image sits in a pending state forever. A queue drains but no GPU ever runs. A worker crashes and deletes the only replayable work item. A presigned upload URL looks valid but browser execution never begins. A GPU instance launches correctly while the actual container inside it continuously crashloops.

Most production failures are not total outages. They are reconciliation failures.

Something the system believes should happen never actually happens. Or something already completed continues to look pending. Or a control plane silently loses the ability to coordinate execution.

Parjanya v2.0 forced us to confront this repeatedly.

Initially, the architecture looked straightforward:

• Images upload into S3

• Metadata lands in DynamoDB

• Lambdas perform validation and enrichment

• GPU workers run VLM and IQA workloads

• Results are written back into DynamoDB

• Autoscaling follows queue depth

At first glance, this looked like a normal AI pipeline.

In practice, it evolved into something else.

The deeper challenge was not inference. The challenge was maintaining alignment between:

the durable truth of the system,
the operational understanding of the system,
the work the system intended to perform,
and the execution layer actually performing it.

Truth → Belief → Intent → Execution

TBIE was never intended as branding. It emerged as an operational language for reasoning about reliability.

The model eventually shaped:

the event-driven reconciliation architecture,
the autoscaling strategy,
the replay model,
the control plane,
failure recovery,
multi-tenant orchestration,
policy regrading,
infrastructure debugging,
and eventually the broader direction of Parjanya itself.

This article is a deep dive into that journey.

It is not a summary. It is a detailed architectural walkthrough of:

how TBIE emerged,
why event-driven reconciliation mattered,
what failed in production,
how those failures were diagnosed,
and why reconciliation patterns matter far more than most AI infrastructure discussions acknowledge.

The goal is not merely to explain Parjanya.

The goal is to explain how resilient AI systems are actually built.

1. The Real Problem in Distributed AI Systems

Most modern AI infrastructure discussions focus heavily on models.

Which VLM?
Which embedding strategy?
Which vector database?
Which GPU?
Which inference framework?
Which quantization?

These questions matter.

But they are not usually what breaks systems.

The real production problems tend to look different:

Work disappears.
Retries become impossible.
Workers process stale state.
Uploads partially succeed.
Queues drain without progress.
Control planes silently stop coordinating.
Autoscaling behaves correctly against the wrong metric.
Infrastructure layers drift away from declared source-of-truth.

In other words:

The problem is not merely computation. The problem is reconciliation.

Distributed systems are fundamentally coordination systems.

A resilient architecture needs to answer:

What is the durable truth?
What work should exist?
What execution is currently happening?
Can the system reconstruct work after failures?
Can the system distinguish transient failure from terminal failure?
Can infrastructure drift be detected?
Can the system replay work safely?
Can policy changes be re-applied deterministically?

Most architectures answer these implicitly.

TBIE attempts to answer them explicitly.

2. TBIE — Truth, Belief, Intent, Execution

The TBIE model separates distributed systems into four operational layers.

The model becomes powerful because it forces explicit boundaries.

Truth

Truth represents the authoritative record.

Truth must be:

durable,
reconstructable,
queryable,
replayable,
and independent of transient execution state.

In Parjanya:

S3 object bytes are Truth.
DynamoDB workflow fields are Truth.
Pipeline status transitions are Truth.
Curation outputs are Truth.
Hashes and metadata are Truth.

Truth is what remains after workers crash.

Truth is what survives retries.

Truth is what operators use to reconstruct system state.

Belief

Belief is operational understanding.

Belief is not always perfectly accurate.

Examples:

• Queue depth

• Autoscaling desired capacity

• In-flight SQS messages

• Worker health

• Current GPU availability

• Visibility timeout state

Belief may lag reality. Belief may be partial. Belief may even temporarily be wrong.

But Belief matters because orchestration depends on it.

Importantly:

Belief should be derivable from Truth + Intent.

That distinction becomes critical later.

Intent

Intent represents durable declarations of work.

Intent answers:

“What should the system attempt next?”

In Parjanya:

SQS messages are Intent.
Replay queues are Intent.
Regrade operations are Intent.
Retry workflows are Intent.

Intent is one of the most important concepts in the entire architecture.

Execution should never destroy Intent casually.

Why?

Because Intent is replayability.

Without replayable Intent:

transient failures become permanent,
retries become manual,
reconciliation becomes impossible,
and the system loses self-healing capability.

Execution

Execution is the stateless machinery that realizes work.

Execution includes:

Lambdas,
GPU workers,
autoscaling groups,
validation services,
EXIF processors,
and inference pipelines.

Execution should ideally be:

disposable,
reproducible,
restartable,
and stateless.

If an execution worker disappears, the system should still be able to recover from Truth + Intent.

That principle drove many later architectural decisions.

3. Mapping TBIE onto Parjanya v2.0

Once TBIE became the operational lens, the architecture became easier to reason about.

Truth Layer

Truth in Parjanya consisted primarily of:

S3

• original uploads,

• RAW image payloads,

• generated previews,

• derived thumbnails,

• metadata artifacts.

DynamoDB

Workflow state:

• batch_status

• pipeline_status

• curation

• quality scores

• VLM outputs

• EXIF metadata

• technical validation

• image hashes

• routing information

Truth represented the durable state of every image.

Intent Layer

Intent existed primarily in SQS.

Examples:

• ml-large-jobs

• replay queues

• regrade operations

• retry workflows

Each Intent payload encoded:

• bucket,

• object key,

• tenant,

• image id,

• execution metadata.

Intent was intentionally replayable.

That design decision later prevented multiple catastrophic replay-loss scenarios.

Belief Layer

Belief included:

• ASG desired capacity,

• queue visibility counts,

• in-flight message counts,

• CloudWatch metrics,

• worker availability,

• queue backlog,

• GPU scale state.

Belief drove orchestration.

For example:

• queue depth controlled autoscaling,

• replay Lambdas checked queue visibility,

• scaling decisions relied on approximate message counts.

Belief was operationally useful but never fully authoritative.

Execution Layer

Execution included:

Lambda Workers

• EXIF extraction

• technical validation

• replay orchestration

• autoscaling control-plane logic

GPU Workers

• Qwen VLM inference

• IQA processing

• curation scoring

• natural-language enrichment

Execution workers remained stateless wherever possible.

That allowed replay to remain safe.

4. Why Reconciliation Matters

A distributed system does not stay healthy merely because components exist.

Health depends on reconciliation.

Consider this state:

DynamoDB row says pending_vlm_enrichment
SQS queue is empty
GPU ASG is scaled to zero

Nothing is technically “down.”

And yet the system is stuck.

Truth says work should exist. Intent does not exist. Execution never begins. Belief incorrectly suggests the system is idle.

This is a reconciliation failure.

TBIE made this visible.

The system needed a mechanism that continuously aligned:

Truth ↔ Intent ↔ Execution

That eventually led to the event-driven reconciliation architecture.

5. From Polling to Event-Driven Reconciliation

The original architecture used scheduled polling.

Every few minutes:

EventBridge triggered a Lambda,
the Lambda scanned DynamoDB,
pending work was discovered,
SQS messages were emitted,
GPU autoscaling was triggered.

This worked.

But it introduced fundamental limitations.

Problems with Polling

Latency

An image could wait:

0–15 minutes before the next poll cycle.

GPU cold starts already consumed several minutes. Adding polling latency made the UX significantly worse.

Waste

Most polling invocations found no work.

The control plane continuously woke up merely to rediscover emptiness.

Operational Ambiguity

Polling blurred the distinction between:

“there is no work,”
and
“the system has not discovered the work yet.”

That ambiguity complicated debugging.

Poor Reactive Scaling

Autoscaling became tied to polling frequency.

Queue discovery latency directly affected GPU startup latency.

6. DynamoDB Streams Changed the Architecture

The major architectural shift came from moving reconciliation into an event-driven model.

Instead of periodically checking for state changes:

The system reacted immediately when Truth changed.

The New Model

DynamoDB row transitions to pending_vlm_enrichment
DynamoDB Streams emits an event
Event source mapping filters relevant writes
Replay Lambda triggers immediately
SQS Intent is emitted
GPU ASG is kicked from zero
GPU workers process work
Results are written back into Truth

The architecture became reactive.

7. Anatomy of the Event-Driven TBIE

Architecture

DynamoDB Streams

The table emits:

• INSERT events

• MODIFY events

Filtered specifically for:

batch_status = pending_vlm_enrichment

This is extremely important.

The system does not react to every write. It reacts only when Truth crosses a workflow boundary requiring Intent creation.

Event Source Mapping

The stream mapping aggregates:

up to 50 records,
or up to 5 seconds.

This balances:

latency,
throughput,
Lambda cost,
queue pressure.

The system effectively performs micro-batching naturally.

Replay Lambda

The replay Lambda became the core reconciliation engine.

Responsibilities:

• inspect stream events,

• discover pending rows,

• identify retriable failures,

• generate replayable Intent,

• avoid duplicate queueing,

• trigger autoscaling.

This Lambda evolved into the operational heart of the platform.

Queue Backlog Guard

One subtle but extremely important design decision:

The replay Lambda checks queue visibility before re-enqueueing pending work.

Why?

Because replay itself can become destructive if uncontrolled.

Without a backlog guard:

the control plane could continuously duplicate Intent,
queue pressure would inflate,
workers would process stale state,
autoscaling would overreact.

This is a classic reconciliation trap.

The backlog guard prevented it.

GPU ASG Kick

The replay Lambda also became responsible for scaling GPU workers.

This matters because scale-to-zero architectures create a hidden problem:

The queue may contain work while compute remains asleep.

The replay Lambda solved this by:

emitting Intent,
then immediately setting desired ASG capacity.

That turned the replay engine into both:

a reconciliation layer,
and a control-plane orchestrator.

8. Latency Transformation

The shift from polling to event-driven reconciliation fundamentally changed responsiveness.

Before

Upload → wait for poll → queue → ASG scale → GPU boot

Total latency:

~3–20 minutes

depending on polling window timing.

After

Upload → stream event → immediate reconciliation → queue → ASG scale

Total latency:

~4–6 minutes

with remaining delay dominated almost entirely by:

EC2 startup,
model load,
GPU warm-up.

This is a very different operational profile.

The system stopped feeling batch-oriented.

It began behaving like a reactive platform.

9. The First Major TBIE Failure

Intent Destroyed Too Early

One of the earliest major incidents exposed why TBIE mattered.

Symptom

GPU workers experienced transient:

• S3 download failures,

• temporary consistency races,

• OOM conditions.

But the worker still deleted the SQS message.

At the same time:

DynamoDB was updated with:

vlm_failed

The result:

Intent disappeared,
Truth claimed terminal failure,
automatic replay became impossible.

This was a severe reconciliation violation.

The Core TBIE Violation

Execution destroyed Intent before retries were exhausted.

That single decision broke replayability.

TBIE clarified the failure immediately.

The problem was not merely worker instability.

The problem was:

Execution invalidated Intent prematurely.

The Fix

The worker execution model changed.

Instead of binary success/failure, workers returned:

retry

• do NOT delete SQS message

• do NOT write terminal Truth

terminal_failure

• write durable failure state

• delete Intent

completed

• write success state

• delete Intent

This restored replay semantics.

Why This Matters Beyond Parjanya

Many distributed systems accidentally conflate:

“execution failed once” with
“execution is permanently impossible.”

Those are not equivalent.

TBIE forced the architecture to encode that distinction explicitly.

10. Truth Pending, Intent Missing

Another class of failures appeared repeatedly.

Symptom

Rows remained:

pending_vlm_enrichment

But:

• queues were empty,

• GPU workers never started,

• no replay existed.

The system appeared healthy.

But nothing moved.

What TBIE Revealed

Truth and Intent were decoupled.

The architecture had assumed:

“If Truth says pending, Intent must already exist.”

That assumption was false.

Reconciliation cannot be assumed. It must be continuously enforced.

The Replay Architecture

This led to:

• replay scripts,

• scheduled replay Lambdas,

• retriable failure reset workflows,

• queue reconstruction logic.

The replay Lambda evolved into a continuously running reconciler.

This was the moment the architecture stopped being merely a pipeline.

It became a platform.

11. Wrong Bucket, Right Key

A Perfect Distributed Systems Failure

One of the most revealing incidents involved bucket mismatch.

Symptom

GPU workers repeatedly failed with:

s3_download_error

Yet:

• queue depth was growing,

• autoscaling worked,

• workers were alive,

• previews existed.

Everything looked operational.

Root Cause

The SQS payload contained:

the correct object key,
but the wrong bucket.

The preview existed.

Just not in the referenced bucket.

Why This Was Architecturally Important

The system was reliably executing the wrong instruction.

This is one of the hardest classes of distributed failures.

Nothing is technically broken.

The system is functioning exactly as instructed.

The instruction itself is invalid.

TBIE framed this correctly:

Intent must not merely exist. Intent must be correct and replayable.

The Fix

The architecture introduced:

• bucket source-of-truth rules,

• preview-aware routing,

• enqueue eligibility tightening,

• RAW-aware preview validation,

• replay resets for retriable failures.

The fix was operationally deeper than changing one field.

It hardened the meaning of Intent itself.

12. CORS Failures and Split Truth

One of the most subtle incidents occurred at the ingestion boundary.

Symptom

Frontend uploads failed with browser CORS errors.

But backend APIs:

• generated presigned URLs,

• wrote pending upload records,

• and returned success responses.

Truth looked correct.

Execution never happened.

What Actually Failed

The browser failed during:

OPTIONS preflight.

Execution never reached:

PUT upload.

The TBIE Interpretation

Truth existed. Intent existed. Execution failed before realization.

This became an important diagnostic heuristic:

When work is stuck:

Ask whether:

• Truth is missing,

• Intent is missing,

• or Execution cannot realize Intent.

That framing simplified debugging enormously.

13. Event-Driven Architecture Exposes Infrastructure Drift Faster

As the architecture became more reactive, infrastructure drift became easier to detect.

This was unexpected.

Example: Stale Worker Image Tag

The ASG launch template still referenced:

:v20

while the actual system had moved to:

:v23

The ASG launched correctly. Cloud-init executed. Docker attempted to pull. The image no longer existed.

Workers never started.

The Deeper Problem

Terraform ignored user_data changes.

This created silent divergence between:

declared source-of-truth,
and actual runtime execution.

TBIE exposed this as an Execution drift problem.

Truth, Belief, and Intent were all correct.

Execution itself had drifted away from reality.

The Architectural Lesson

Infrastructure reproducibility is not optional.

Execution environments must remain tied to durable source-of-truth.

Otherwise replay becomes dangerous because:

the same Intent may produce different outcomes over time.

14. AMI Drift and Runtime Incompatibility

Another failure emerged at the AMI layer.

Symptom

Containers started. But model loading crashed.

The model weights existed. The processors did not.

The AMI bake omitted Python files required by:

trust_remote_code=True

The worker entered a crashloop.

Why This Was Important

Again:

Truth was correct. Intent was correct. Belief suggested workers existed.

Execution failed due to stale artifacts.

This is precisely why replayable systems require reproducible execution environments.

TBIE repeatedly reinforced this architectural truth.

15. The Replay Lambda Became the Control Plane

Originally, the replay Lambda was operational glue.

Eventually, it became the platform control plane.

Responsibilities expanded to include:

• pending reconciliation,

• retriable failure resets,

• tenant discovery,

• queue backlog management,

• autoscaling coordination,

• replay orchestration,

• dynamic routing.

The replay layer became central infrastructure.

This was a critical architectural evolution.

16. The IAM Incident

When the Control Plane Goes Blind

One of the most dangerous incidents involved a missing IAM permission.

Symptom

The replay Lambda continuously crashed on:

sqs:GetQueueAttributes

for nearly 12 hours.

But:

• EventBridge still invoked it,

• the schedule looked healthy,

• the infrastructure appeared alive.

Meanwhile:

• pending rows accumulated,

• queues stopped reconciling,

• autoscaling logic failed.

The TBIE Lesson

The control plane had lost Belief.

Without Belief:

the replay engine could not safely emit Intent,
queue guards could not function,
orchestration collapsed.

The system looked operational while the reconciler was effectively dead.

The Operational Insight

A scheduled Lambda is not necessarily a healthy Lambda.

Control-plane observability matters more than worker observability.

Why?

Because worker failures are recoverable.

Control-plane blindness disables recovery itself.

17. Dynamic Tenant Discovery

As Parjanya moved toward beta onboarding, static tenant configuration became a bottleneck.

Originally:

The replay Lambda used static tenant lists.

Every new tenant required:

• Terraform changes,

• environment updates,

• redeploys.

This violated the self-service model.

The Evolution

The architecture introduced:

• tenant registry writes during upload completion,

• runtime tenant discovery,

• dynamic reconciliation.

The control plane became data-driven rather than configuration-driven.

This is a major architectural distinction.

Reactive systems scale operationally only when discovery becomes dynamic.

18. Regrading and Policy Replay

One of the most powerful consequences of TBIE was deterministic replay.

This became especially valuable during policy changes.

Example

The IQA policy evolved to reject:

• screenshots,

• banners,

• posters,

• slides,

• social cards,

• infographics,

• illustrations.

The architecture needed to:

• reset VLM-derived Truth,

• regenerate Intent,

• rerun GPU enrichment,

• rewrite curation outputs.

This was not merely a retry.

It was a deterministic policy replay.

Why This Matters

Most ML systems struggle with:

“How do we reprocess historical data safely?”

TBIE already contained the answer.

Truth could be reset to a replay boundary. Intent could be regenerated. Execution remained stateless.

The system became replay-native.

This is a profound operational capability.

19. Queue Retention and Weekend Gaps

Another subtle operational issue:

Scale-to-zero architectures create temporal gaps.

Scenario

Users upload images Friday evening. GPU workers remain offline over the weekend.

If queue retention is too short:

Intent disappears before execution resumes.

Truth remains pending.

Intent vanishes.

This is catastrophic reconciliation drift.

The Fix

Queue retention was extended.

This ensured Intent outlived temporary execution dormancy.

An important TBIE lesson emerged:

Intent durability must exceed execution availability windows.

20. Why Event-Driven Reconciliation Matters

Beyond AI

Although Parjanya is an AI platform, the architectural lessons generalize broadly.

The same TBIE principles apply to:

distributed media systems,
payment processing,
ETL pipelines,
logistics systems,
asynchronous workflows,
microservice orchestration,
large-scale ingestion platforms.

The core challenge is universal:

How do you maintain alignment between:

durable truth,
intended work,
operational state,
and actual execution?

TBIE provides a vocabulary for reasoning about that alignment.

21. The Shift from Pipeline to Platform

This was perhaps the most important realization.

A pipeline moves data.

A platform:

• survives retries,

• supports replay,

• reconciles state continuously,

• tolerates execution drift,

• evolves policy safely,

• supports operational introspection,

• and remains debuggable under failure.

The event-driven TBIE architecture pushed Parjanya decisively toward the latter.

The replay Lambda stopped being a utility.

It became the reconciliation brain of the system.

The architecture stopped being:

“process uploaded images.”

It became:

“maintain durable correctness while asynchronous distributed execution continuously changes.”

That is a very different engineering problem.

22. The Operator’s Mental Model

TBIE becomes most useful when it changes how incidents are diagnosed.

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. Operators no longer began by asking whether “the queue is healthy” or whether “the GPU workers are running.” Those signals still mattered, but they were no longer treated as authoritative.

Instead, every investigation began with a much more precise question:

Is the system waiting because Intent is missing, or is Intent present but Execution cannot realize it?

That distinction dramatically narrowed the search space during incidents.

If Truth showed a growing backlog while Intent remained near zero, the issue was usually in reconciliation itself. Replay Lambda failure, queue emission failure, stale tenant discovery, or a blind control plane could all create this pattern. The queue might look empty and the autoscaling group might remain at zero, yet the real failure would not be in the workers. It would be in the system’s ability to regenerate Intent from Truth.

If Intent existed but Execution failed repeatedly, the investigation shifted toward realization problems instead: bucket routing mismatches, stale AMIs, launch-template drift, IAM gaps, runtime incompatibilities, model-loading failures, endpoint routing problems, or browser-side execution failures.

That diagnostic framing changed operations significantly. Incidents became easier to classify because TBIE separated the architecture into meaningful operational boundaries. Instead of treating the platform as one opaque pipeline, operators could isolate whether the failure belonged to Truth, Belief, Intent, or Execution.

The model also made one uncomfortable reality obvious: dashboards alone are not enough. A queue can look healthy while Truth continues accumulating backlog. Autoscaling can look correct while workers continuously crashloop. A scheduled Lambda can appear operational while silently failing every invocation because of an IAM permission gap.

TBIE did not eliminate debugging complexity. What it did was make debugging systematic.

23. When the System Hangs at 02:00 AM

Architectural diagrams are useful during design reviews.

Runbooks matter during incidents.

One of the most important evolutions in Parjanya v2.0 was realizing that reconciliation needed to exist not only in architecture, but also in operational practice. TBIE only became truly useful once there was a repeatable process for recovering the system under pressure.

The first step during any “hung VLM” investigation became identifying the TBIE boundary itself.

Truth consisted of the DynamoDB image rows and their workflow state. Intent consisted of SQS messages representing GPU work. Execution consisted of GPU workers consuming that Intent and writing enriched Truth back into DynamoDB. Belief consisted of queue depth, autoscaling state, visibility counts, worker health, and operational telemetry.

Once those boundaries were established, incident triage became much more structured.

Operators first checked Truth backlog. Were rows accumulating in pending_vlm_enrichment? If yes, the system still believed work should happen.

Next came Intent backlog. Did SQS contain replayable work items? If Truth backlog was high while Intent remained near zero, then reconciliation itself had failed.

Then came Execution health. Were GPU workers alive? Were containers actually starting? Were model loads succeeding? Were workers consuming messages or continuously failing before acknowledgement?

Several operational shortcuts emerged over time.

If the GPU log group was empty while ASG instances existed, the problem was often below the application layer entirely. Cloud-init failures, stale image tags, or broken launch templates frequently surfaced first through EC2 console output rather than through CloudWatch worker logs.

If the replay Lambda appeared scheduled but no queue activity existed, IAM permissions became the first thing to inspect. One of the most dangerous incidents in the system occurred precisely because the reconciliation Lambda could no longer read queue attributes, effectively blinding the control plane while the infrastructure superficially appeared healthy.

If rows repeatedly failed with S3 download errors while workers remained alive, the issue was often deterministic addressing rather than transient instability. Wrong-bucket references, preview routing mismatches, or endpoint configuration drift could all create workers that failed consistently while still appearing operational.

Over time, this changed the operational mindset of the platform. The goal stopped being “restart the workers and hope.” The goal became understanding exactly where reconciliation had broken down and then repairing the appropriate boundary.

That distinction is subtle, but extremely important.

A resilient platform is not the one that avoids incidents. It is the one that can recover from them methodically.

24. Guardrails and Proactive Controls

One of the clearest architectural patterns that emerged from Parjanya was that every major incident eventually needed to produce a permanent guardrail.

Without that discipline, incidents remain stories. With it, incidents become operational evolution.

The architecture gradually accumulated a large number of proactive controls:

Transient worker failures no longer delete Intent prematurely. Replay Lambdas continuously reconcile pending Truth. Reproducible lockfiles prevent dependency drift. Terraform validations reject invalid wildcard CORS patterns. Tenant discovery became dynamic rather than operator-maintained. Queue retention was extended to match real outage windows rather than idealized assumptions.

Many of these controls were not introduced because the architecture diagram required them. They emerged because production failures revealed where reconciliation could silently drift.

This is one of the strongest long-term lessons from the TBIE model.

Resilience is not achieved through one large design decision. It is accumulated gradually through operational guardrails.

The mature version of the system looked very different from the original one not because the core architecture changed dramatically, but because every important failure eventually became:

a validation rule,
a replay strategy,
a Terraform constraint,
a CI assertion,
a queue safeguard,
or an operational diagnostic.

That is how resilient systems evolve in practice.

They convert incidents into institutional memory.

25. Maintenance as a Reliability Discipline

One of the easiest mistakes in distributed systems is assuming resilience is something implemented once.

In reality, resilience is maintained continuously.

By the later stages of Parjanya v2.0, the team had accumulated a growing operational checklist precisely because many failures were caused not by dramatic outages, but by small forms of drift.

Replay logic and worker behavior needed to evolve together. GPU image tags needed to stay aligned with launch templates. Lockfiles had to remain committed when dependencies changed. Tenant buckets required CORS validation before onboarding. Queue retention needed to reflect actual business downtime windows. Backend API contracts and frontend types had to remain synchronized.

None of these tasks sound individually sophisticated.

But distributed systems rarely collapse from a single catastrophic event. They erode through gradual inconsistency.

That realization changed how the platform was maintained.

Operational discipline became part of the architecture itself.

The maintenance checklist was therefore not merely documentation. It was a recognition that reconciliation must continue throughout the entire lifecycle of the platform.

26. Why TBIE Matters Beyond AI

Although Parjanya is an AI platform, the architectural lessons generalize broadly.

The same TBIE principles apply to:

distributed media systems,
payment processing,
ETL pipelines,
logistics systems,
asynchronous workflows,
microservice orchestration,
large-scale ingestion platforms.

The core challenge is universal:

How do you maintain alignment between:

durable truth,
intended work,
operational state,
and actual execution?

TBIE provides a vocabulary for reasoning about that alignment.

27. The Shift from Pipeline to Platform

1. Truth must remain durable and replayable

If Truth cannot reconstruct the system after failures, the architecture becomes operationally fragile.

2. Intent is more important than most systems realize

Intent is replayability.

Destroying Intent prematurely destroys recovery.

3. Event-driven systems expose hidden drift faster

Reactive architectures surface infrastructure inconsistencies earlier.

This is painful initially.

But healthier long-term.

4. Execution must remain reproducible

Replay without reproducibility becomes dangerous.

5. Control-plane observability matters more than worker observability

Worker crashes are recoverable. Blind reconcilers are not.

6. Queue depth alone is not truth

Belief metrics require interpretation.

Approximate counts are operational signals, not durable truth.

7. Policy replay should be designed intentionally

Eventually every ML platform needs deterministic reprocessing.

Replay-native architectures handle this gracefully.

8. Distributed systems fail by divergence

Not by total outage.

The hardest bugs were:

wrong bucket,
stale AMI,
hidden CORS mismatch,
invisible IAM gap,
frozen replay,
missing Intent.

TBIE helped make those failures explainable.

28. The Future Direction

The current architecture already points toward several future evolutions:

• multi-stage event-driven pipelines,

• adaptive batching,

• warm GPU pools,

• richer metrics export,

• dynamic replay orchestration,

• per-stage autoscaling,

• event-sourced workflow history,

• tenant-aware media delivery,

• replayable policy evolution.

Importantly:

These evolutions do not require rewriting the architecture.

They emerge naturally because reconciliation is already foundational.

That is one of the strongest signs that the architecture direction is correct.

Closing Thoughts

TBIE did not begin as an attempt to create a framework.

It emerged because distributed AI infrastructure repeatedly exposed the same class of operational failures:

work disappearing,
retries becoming impossible,
control planes drifting,
execution environments diverging,
queues losing meaning,
infrastructure behaving correctly against stale assumptions.

The model became useful because it created clarity.

Truth. Belief. Intent. Execution.

Once those boundaries became explicit:

replay became easier,
debugging became faster,
autoscaling became safer,
policy evolution became manageable,
and infrastructure failures became explainable.

The most important lesson from Parjanya v2.0 is not that event-driven architectures are faster.

The deeper lesson is:

Resilient distributed systems are fundamentally reconciliation systems.

The system that survives is not necessarily the one with the fastest workers.

It is the one that can continuously reconstruct:

what is true,
what work should exist,
what execution already happened,
and what still needs to be reconciled.

That is what ultimately transformed Parjanya from a pipeline into a platform.

In May 2026, I found myself staring at a familiar engineering problem.

Parjanya’s Image Quality Assessment pipeline was approaching the limits of what our infrastructure could comfortably handle.

Inference workloads were growing.

Batch sizes were increasing.

Memory utilization was climbing.

And eventually, the system hit a wall.

A larger image.

A slightly bigger batch.

A little more context.

And then:

CUDA Out of Memory.

The obvious answer was straightforward.

Upgrade the hardware.

Move to larger GPUs.

Increase the budget.

Scale vertically.

That’s what I almost did.

Instead, I opened the prompt.

And that decision ended up saving far more money than a hardware upgrade would have.

The Assumption I Didn’t Question

By this point in the Parjanya v2.0 journey, I had spent months thinking about infrastructure.

The previous two posts in this series covered:

• Multi-account AWS architecture

• Terraform and operational discipline

• CI/CD pipelines

• IAM boundaries

• Infrastructure scaling

So when inference started becoming expensive, my instinct was predictable.

I looked at the hardware.

That’s what engineers do.

We assume the bottleneck lives where the graphs are red.

GPU memory usage was high.

Throughput was flattening.

Batch sizes were constrained.

The diagnosis seemed obvious.

But infrastructure metrics were only showing symptoms.

They weren’t showing the cause.

The Audit That Changed Everything

Instead of pricing larger GPUs, I decided to audit the prompt itself.

At that point our VLM-based IQA system had evolved through multiple versions.

Each release added new capabilities:

• Better composition evaluation

• Improved wildlife classification

• More robust defect detection

• Additional structured outputs

• More nuanced confidence handling

Every version added knowledge.

Almost nothing removed knowledge.

The prompt had become a living document.

Like most living documents, it accumulated debt.

When I finally measured it, the numbers surprised me.

The system prompt plus user context had grown to roughly:

4,150 tokens

Not because we needed 4,150 tokens.

Because nobody had challenged whether we still needed them.

What I Found

The easiest way to describe it is this:

The prompt looked less like a carefully engineered system and more like a codebase that hadn’t been refactored in years.

There was useful logic.

There was historical baggage.

And there was a lot of dead code.

Unused Outputs

One section calculated an overall quality score.

The model spent tokens reasoning about it.

The downstream system ignored it completely.

The actual acceptance and rejection decisions came from deterministic rules elsewhere in the pipeline.

The score existed because an earlier version needed it.

The current version didn’t.

Yet the prompt still carried the instructions.

Features That Never Shipped

I found sections generating:

• search tags

• mood descriptions

• discovery metadata

These outputs were originally intended for future content discovery features.

Those features never launched.

The instructions remained.

Every inference paid the cost.

Every single time.

Teaching the Model Things It Already Knew

This was the biggest surprise.

Over time we’d added examples.

Then more examples.

Then explanations for the examples.

Then explanations explaining the explanations.

The prompt contained long sections describing:

• composition techniques

• confidence scoring

• framing recognition

• classification behavior

At one point those explanations were useful.

But the model wasn’t learning from scratch anymore.

The explanatory scaffolding had become informational overhead.

The rules mattered.

The essays explaining the rules did not.

I Started Thinking About It Like Technical Debt

The moment everything clicked was when I stopped calling it prompt engineering.

This wasn’t prompt engineering.

This was refactoring.

The exact same process we apply to software.

In software we ask:

• Is this function still used?

• Does this dependency still matter?

• Is this abstraction providing value?

• Can this code be simplified?

I realized prompts deserve the same scrutiny.

A prompt is infrastructure.

It consumes resources.

It affects latency.

It affects throughput.

It affects cost.

And like any infrastructure, it accumulates debt.

I started calling this:

Prompt Debt.

What Stayed

This is the part that surprised me most.

Almost all of the intelligence survived.

The capabilities remained intact.

The model still knew how to:

• recognize deliberate composition

• distinguish artistic choices from technical defects

• apply confidence-aware wildlife classification

• detect focus issues

• identify exposure problems

• generate structured outputs

• evaluate image quality

We didn’t remove intelligence.

We removed repetition.

We removed explanation.

We removed historical residue.

The distinction matters.

The Numbers

After the audit:

System Prompt

~700 tokens → ~360 tokens

User Prompt

~3,450 tokens → ~890 tokens

Total Context

~4,150 tokens → ~1,250 tokens

Approximately:

70% reduction

without changing:

• the model

• the hardware

• the infrastructure

• the deployment architecture

Only the prompt changed.

Why 70% Matters

It’s tempting to see this as an academic optimization.

It wasn’t.

Those tokens had real operational consequences.

Every unnecessary token consumes:

• memory

• compute

• latency budget

• throughput capacity

On paper, a token looks small.

At scale, thousands of unnecessary tokens become infrastructure.

And infrastructure costs money.

Before the cleanup, our batching limits were constrained by context size.

After the cleanup, we could process roughly twice as many images within similar memory budgets.

The result was effectively:

2x throughput without changing hardware.

The kind of gain most teams expect from an expensive infrastructure project.

We got it from deleting text.

Every Token Has a Cost

This became the biggest lesson of the entire exercise.

Engineers often think about infrastructure costs.

Few think about information costs.

Every token carries a burden.

A token occupies memory.

A token increases inference time.

A token competes for attention.

A token introduces complexity.

The question isn’t:

“Can the model read this?”

The question is:

“Should the model be reading this at all?”

Those are very different questions.

This was a practical reminder of a lesson I explored earlier in Grounding Is Not a Prompt. Models don’t become more faithful simply because we provide more instructions. What matters is the quality and relevance of information available at inference time. A smaller, denser prompt often outperforms a larger one filled with historical baggage and explanatory noise.

The Broader Pattern

The more I reflected on the exercise, the more it felt familiar.

This pattern appears everywhere.

Organizations accumulate meetings.

Software accumulates dependencies.

Infrastructure accumulates services.

Prompts accumulate instructions.

Over time, complexity grows through addition.

Rarely through subtraction.

Most systems become slower because nobody asks what can be removed.

Prompts are no different.

While reviewing the prompt, I kept thinking about a principle we previously discussed in Databases, Caches and Queues: A Universal Rule for Distributed Systems. Systems become resilient when information has clear ownership and purpose. The same principle applies to prompts. Every instruction should exist for a reason. If nobody consumes it, it becomes operational debt.

The Connection to Inferencing

This experience fundamentally changed how I think about inference optimization.

Most conversations focus on:

• larger models

• larger context windows

• larger GPUs

• more sophisticated hardware

Those matter.

But they’re not always the first lever.

Sometimes the most effective optimization isn’t computational.

It’s informational.

Before changing the model, review the prompt.

Before upgrading the GPU, review the prompt.

Before increasing context windows, review the prompt.

The cheapest token is the one you never generate.

What Comes Next

Prompt reduction was only one layer.

The next frontier involves serving optimization:

• prompt prefix caching

• KV cache reuse

• vLLM batching improvements

• quantization strategies

• memory-efficient inference

Those areas still offer meaningful gains.

But they sit on top of a more fundamental principle.

Optimize information before optimizing infrastructure.

The Takeaway

When the system first ran out of memory, I assumed we had a hardware problem.

What we actually had was an information problem.

The model wasn’t struggling because it lacked capacity.

It was struggling because we were making it carry unnecessary baggage.

That distinction changed how I think about AI systems.

We often assume progress requires adding something.

A larger model.

A larger GPU.

A larger context window.

A larger budget.

In this case, the breakthrough came from subtraction.

The model already knew enough.

The system simply needed less noise.

And that’s a lesson that extends far beyond prompts.

It’s a lesson about engineering itself.

This concludes the initial Parjanya v2.0 engineering series. Across these three posts—from AWS account boundaries, to Terraform discipline, to prompt debt—the common theme has been surprisingly consistent: systems scale best when boundaries are clear, assumptions are explicit, and unnecessary complexity is removed. The challenge is rarely adding more. The challenge is knowing what to keep.

Further Reading

If this article resonated with you, these related essays expand on the ideas discussed here:

• Grounding Is Not a Prompt — Why better information retrieval often matters more than adding more instructions to a model.

• VLM Batch Inference - 4 bit quantisation depth — The Hidden Cost of “Almost Fitting”

• Context Engineering and Context Debt — a data investigation that revealed ~77% of my token spend was context accumulation waste, not productive reasoning. This is the framework I built to fix it — and the problem has a name: Context Debt and the fix is: Context Engineering .

• Databases, Caches and Queues: A Universal Rule for Distributed Systems — The Truth, Belief and Intent framework that inspired how we think about information flow across systems.

• TBIE in Practice: Designing Resilient AI Pipelines That Recover, Reconcile, and Re-run - Reconciliation loop and TBIE model with case studies with Parjanya v2.0

• Terraform at Scale: What Actually Worked, What Didn’t, and the Cost of Trusting AI with Infrastructure — The previous post in this series, exploring operational discipline and infrastructure boundaries.

• Why We Use Three AWS Accounts (Not Two) for Parjanya v2.0 — The architectural foundation that enabled the optimizations described here.

Introducing the Inferencing Series

This article marks the beginning of a new series exploring inferencing as a first-class engineering discipline. Much of the AI conversation today revolves around models, benchmarks, and training. Yet in production systems, the economics and behaviour of AI are often determined long after training is complete—during inference. The choices we make around prompts, context, quantization, batching, memory management, caching, retrieval, and serving architecture frequently have a greater impact on cost, latency, and reliability than changing the model itself.

Throughout this series, I’ll share lessons learned while building and operating Parjanya’s Visual Language Model (VLM) pipelines. Rather than focusing on LLM chat applications, the emphasis will be on real-world image understanding systems: image quality assessment, wildlife classification, structured extraction, multimodal reasoning, and the engineering decisions required to run them efficiently at scale. The goal is not simply to reduce costs, but to understand how information flows through an AI system and where meaningful optimization actually lives.

Why Start With Prompt Debt?

Prompt Debt is an ideal starting point because it reveals a common misconception about AI systems: that performance improvements come primarily from adding more. More instructions. More examples. More context. More hardware. In practice, many mature VLM systems accumulate informational debt in the same way software accumulates technical debt—unused outputs, redundant instructions, abandoned features, and historical assumptions that continue consuming resources long after they stop creating value.

Future articles in the Inferencing Series will build on this foundation by exploring quantization, context engineering, grounding strategies, serving architectures, prompt caching, KV-cache optimization, batching, multimodal pipelines, and the operational realities of running VLM workloads in production. Prompt Debt is simply the first lens. The broader question—and the one that interests me most—is how to maximize signal while minimizing computational waste across the entire inferencing stack.

When people talk about Terraform, they often talk about Infrastructure as Code.

What I learned while building Parjanya v2.0 is that Terraform is really about something else:

forcing operational discipline.

Terraform didn’t magically simplify our infrastructure. It exposed assumptions, surfaced hidden dependencies, and forced us to formalize decisions we had previously gotten away with making informally.

As Parjanya evolved from a collection of scripts, console clicks, and one-off configurations into a multi-tenant image processing platform, Terraform became the backbone that allowed us to scale infrastructure without losing control.

It also introduced new challenges.

And when I layered AI-assisted engineering into the workflow, I discovered an entirely different category of problems worth documenting.

This post is the implementation companion to my previous article on why we chose a three-account AWS architecture for Parjanya v2.0.

That post explained the boundaries.

This one explains how we turned those boundaries into code.

The Reality of Operating Across Five Repositories

Parjanya today spans five repositories.

Each exists for a reason.

• parjanya-ml handles inference and model serving.

• parjanya-ops owns infrastructure and platform operations.

• parjanya-backend manages orchestration and APIs.

• parjanya-frontend serves tenant-facing experiences.

• parjanya-shared-libraries contains contracts and shared abstractions.

The separation works well organizationally.

ML engineers don’t need to understand IAM condition keys.

Infrastructure engineers don’t need to navigate PyTorch dependencies.

Backend engineers don’t need to modify Terraform every time they ship a feature.

But Terraform exposed a challenge that few people talk about:

cross-repository infrastructure dependencies.

The moment one repository depends on infrastructure outputs defined in another, coordination becomes critical.

A renamed Terraform output in one repository can quietly break another repository that references it through remote state.

I’ve been bitten by this more than once.

Terraform didn’t prevent those mistakes.

What it did provide was visibility.

And visibility is often the difference between a recoverable issue and a production incident.

What Terraform Actually Improved

A lot of Terraform discussions stay theoretical.

Here’s what it changed in practice for us.

ECS and Warm Pools

One of the first problems we tackled was cold-start latency.

Inference workloads running on ECS-backed EC2 instances would occasionally wait several minutes for capacity to become available.

To solve this, we implemented Auto Scaling Group warm pools.

The result wasn’t just faster startup times.

It gave us a repeatable way to define capacity behavior for different tenant tiers.

Instead of manually tuning infrastructure across environments, we could express those decisions through variables and deploy them consistently.

That may sound mundane.

But consistency becomes incredibly valuable once multiple environments, tenants, and deployment pipelines are involved.

Multi-Tenant S3 Isolation

Parjanya is a multi-tenant platform.

Tenant isolation isn’t optional.

Terraform allowed us to codify S3 prefix-level isolation directly into IAM policies.

The most important outcome wasn’t the policy itself.

It was the reviewability.

A missing condition that could have exposed tenant object metadata became visible during code review instead of after deployment.

That’s one of Terraform’s underrated strengths:

It turns invisible infrastructure assumptions into visible diffs.

DynamoDB Tenant Boundaries

The same pattern applied to DynamoDB.

Every tenant’s data is partitioned using tenant identifiers, with IAM enforcing access boundaries.

Could application code enforce those rules?

Yes.

Should application code be the only layer enforcing them?

Absolutely not.

Terraform helped us move those guarantees into infrastructure itself.

Even if application logic fails, IAM remains the final gatekeeper.

Event-Driven Infrastructure

Parjanya’s processing pipeline relies heavily on EventBridge and SQS.

Before Terraform, environment drift was a recurring problem.

A queue configuration changed in Production but not Development.

An EventBridge rule existed in one environment but not another.

Over time these differences accumulate.

Terraform gave us repeatability.

The same modules, deployed consistently, dramatically reduced configuration divergence.

The Three-Account Migration

The biggest Terraform project in Parjanya v2.0 wasn’t a service deployment.

It was infrastructure reorganization.

We migrated from a single AWS account into:

• Management / Tooling

• Development

• Production

Terraform made that migration possible.

It did not make it easy.

Making State Account-Aware

Our original infrastructure assumed everything lived in one account.

Provider configurations were implicit.

Cross-account roles didn’t exist.

Account boundaries didn’t exist.

The first step was making account ownership explicit.

That sounds simple until you realize every Terraform resource suddenly needs to know which AWS account it belongs to.

The work wasn’t intellectually difficult.

It was painstaking.

And this is where I learned an important lesson about AI-assisted engineering:

AI is surprisingly bad at repetitive infrastructure migrations unless it has complete context.

Migrating State Safely

One of the most valuable Terraform commands I encountered during this process was:

terraform state mv

State migration sounds boring.

Until you realize a single mistake can cause Terraform to believe a production resource should be destroyed and recreated.

The migration process became an exercise in discipline:

Move state.

Run a plan.

Review carefully.

Repeat.

Slowly.

The temptation to batch changes is enormous.

Resisting that temptation saved us multiple times.

Cross-Account IAM

The three-account architecture introduced a new category of complexity.

Trust relationships.

The Management account hosts shared tooling.

Development and Production need access to those resources.

Terraform became the mechanism for expressing those trust boundaries consistently.

What surprised me wasn’t the complexity of the IAM policies.

It was how much architectural thinking is required before writing any Terraform.

The tooling faithfully implements your design.

It does not validate whether the design itself is sensible.

Centralized State Management

One decision I remain happy with is keeping Terraform state centralized within the Management account.

Development engineers can operate Development.

Production operators can manage Production.

But neither automatically gains access to every state file.

This separation aligns with the overall philosophy behind the architecture:

Boundaries first.

Convenience second.

Where AI Helped — And Where It Didn’t

Throughout this migration I relied heavily on AI-assisted development.

Claude became part of the workflow.

Over time I learned that different models excel at very different tasks.

The results were revealing.

Haiku: Fast But Dangerous for Infrastructure

Haiku was excellent at generating boilerplate.

Variable files.

Module outputs.

Simple scaffolding.

But infrastructure isn’t just syntax.

Infrastructure is reasoning.

I repeatedly found that Haiku could generate valid Terraform that contained subtle security flaws.

The code looked correct.

The logic wasn’t.

That distinction matters.

A lot.

My conclusion became simple:

Haiku should never be trusted with security-sensitive infrastructure decisions.

Sonnet: The Daily Driver

Most of our Terraform work happened with Sonnet.

For implementation tasks inside a repository, it performed extremely well.

Module structures.

Provider configurations.

Refactors.

General Terraform development.

Where it struggled was context.

A repository rename might affect multiple repositories.

A queue change might impact systems it couldn’t see.

The problem wasn’t model intelligence.

The problem was visibility.

The model can only reason about what you show it.

Opus: Better at Architecture Than Implementation

I brought Opus into the process whenever decisions became architectural.

Questions like:

• Should this responsibility live in Management or Production?

• How should artifacts flow between accounts?

• What’s the cleanest trust model?

These are trade-off questions.

Opus consistently handled them better than implementation-focused tasks.

The irony was that the more strategic the question became, the more useful the model became.

The Biggest Problem Wasn’t Terraform

It Was Context.

This became one of the clearest lessons from Parjanya v2.0.

The hardest infrastructure problems weren’t provider bugs.

They weren’t Terraform limitations.

They weren’t AWS limitations.

They were context problems.

Infrastructure lived in one repository.

ML workloads lived in another.

Shared interfaces lived somewhere else.

Every repository understood part of the system.

No repository understood the entire system.

That’s why we started building supporting tooling and internal agents focused on context discovery and verification.

The challenge wasn’t generating code.

The challenge was understanding the system before generating code.

What Terraform Didn’t Solve

It’s important to be honest about this.

Terraform isn’t a cure-all.

We still accumulated module sprawl.

We still dealt with drift.

We still had to manage secrets separately.

We still needed human judgment when reviewing large infrastructure changes.

Terraform gave us control.

It did not eliminate responsibility.

The Practical Lessons

If I were starting again today, these are the lessons I’d carry forward.

Use Terraform to enforce boundaries, not just provision resources.

Treat account architecture as an IAM problem first and a Terraform problem second.

Invest in cross-repository visibility before investing in more automation.

Choose AI models based on task type, not convenience.

Review infrastructure plans like code. Because they are.

Prioritize understanding the system before changing it.

Most infrastructure failures aren’t caused by bad tooling.

They’re caused by incomplete context.

Looking Ahead

The three-account architecture gave us the boundaries.

Terraform gave us the implementation.

But neither addressed another challenge that emerged as Parjanya’s inference workloads grew:

Cost.

Specifically, inference cost.

While we were optimizing infrastructure, another inefficiency was hiding in plain sight.

Our VLM system prompt had quietly grown to over 4,000 tokens.

Most of it wasn’t doing useful work.

Tomorrow’s post moves from infrastructure into inference engineering.

I’ll share how we reduced prompt size by nearly 70%, doubled throughput on the same hardware, and discovered that some of the most impactful optimizations don’t involve changing models at all.

The Takeaway

Terraform didn’t make Parjanya v2.0 successful.

What it did was force us to be explicit.

Explicit about ownership.

Explicit about trust boundaries.

Explicit about infrastructure decisions.

And perhaps most importantly, explicit about the assumptions we were making.

That’s the real value of Infrastructure as Code.

Not automation.

Clarity.

And clarity scales far better than heroics.

This is part of my ongoing Parjanya v2.0 field notes, documenting the architectural, operational, and engineering decisions behind building a multi-tenant AI platform. In the next post, I’ll move beyond infrastructure and into inference optimization—where a prompt review ended up delivering more value than a hardware upgrade.

One AWS account.

Maybe two if you’re feeling disciplined.

One for development. One for production.

It sounds clean. It sounds sufficient.

For a while, I thought so too.

As Parjanya evolved into a multi-tenant image processing platform with real workloads, real customers, and increasingly complex infrastructure, I started asking a different question:

What does the minimum viable production-grade AWS architecture actually look like?

The answer wasn’t one account.

It wasn’t even two.

It was three.

Today Parjanya v2.0 runs on a three-account AWS structure:

• Management / Tooling

• Development

• Production

It added complexity upfront.

It also removed entire categories of risk that I didn’t want to discover during an incident.

This post explains why I made that decision, what problems it solved, what I intentionally chose not to build yet, and how I think other SaaS teams should approach the same trade-offs.

The Setup We Started With

Like most startups, we began with a single AWS account.

Everything lived there:

• Development resources

• Production workloads

• CI/CD pipelines

• Container registries

• Shared infrastructure

The setup was fast.

The problem with fast is that it often hides risk.

Nothing feels wrong until the day an experiment, a permission mistake, or an infrastructure change unexpectedly affects production.

As we started formalising Parjanya v2.0, I realised I didn’t want to keep relying on discipline alone.

I wanted structural safety.

That’s when the three-account model emerged.

The Third Account Isn’t Staging

Whenever I mention three accounts, people assume I mean:

• Dev

• Staging

• Prod

That’s not what I built.

The structure is:

Management / Tooling Account

This account owns the operational plumbing:

• Container registry

• Terraform state

• CI/CD definitions

• Cross-account IAM roles

• Billing aggregation

• Audit aggregation

Importantly, it never runs application workloads.

Development Account

This is where experimentation happens.

Infrastructure changes.

New features.

Testing.

Things break here by design.

Production Account

This runs live tenant workloads and customer-facing infrastructure.

Nothing experimental belongs here.

The critical insight is that the third account isn’t another environment.

It’s neutral territory.

It owns the things both Dev and Prod need, without forcing either environment to depend directly on the other.

The First Problem It Solves: Artifact Ownership

Consider a simple CI/CD pipeline.

It builds a Docker image.

Where should that image live?

If the registry sits in Dev, then Production depends on Development.

If the registry sits in Prod, Development depends on Production.

Neither feels right.

The Management account solves this neatly.

CI builds once.

Pushes once.

Both Dev and Prod pull the exact same artifact.

No rebuilds.

No copies.

No divergence.

One image.

One source of truth.

Multiple promotions.

That sounds like a small detail until you’re debugging an issue and need absolute confidence that Production is running the same artifact that passed validation earlier.

The Real Reason: Blast Radius

This is the reason that mattered most to me.

AWS accounts are the strongest isolation boundary AWS provides.

A broken IAM policy.

An accidental deletion.

A Terraform mistake.

A bad deployment.

These things cannot casually cross account boundaries.

That matters because Development should be chaotic.

People test things.

Infrastructure changes.

Experiments happen.

Failures are expected.

Production should not inherit that chaos.

With separate accounts, Dev can fail spectacularly and Prod remains untouched.

That’s a much stronger guarantee than “everyone promises to be careful.”

Credentials Become Safer by Default

A surprisingly useful side effect is credential isolation.

When someone is working in the Dev account, their credentials only see Dev resources.

They aren’t merely discouraged from touching Production.

Production is outside their scope.

The same applies to automation.

The deployment pipeline for Dev assumes different roles than the deployment pipeline for Prod.

The separation isn’t cultural.

It’s structural.

And structural controls tend to survive stressful situations much better than process documents.

Cost Visibility Becomes Obvious

Another benefit I underestimated was cost attribution.

Using AWS Organizations means every account naturally reports its own spending.

Development spend is visible.

Production spend is visible.

There’s no tagging strategy to maintain.

No allocation spreadsheets.

No cost archaeology.

When the AWS bill changes, I immediately know whether it was caused by:

• a production feature,

• increased tenant activity,

• or a development experiment that ran longer than intended.

That clarity is surprisingly valuable.

What Lives Where

The distribution is straightforward.

Management Account

• Container registry

• Terraform backend

• CI/CD pipelines

• Cross-account IAM roles

• Billing aggregation

• Audit logging

Development Account

• Full application stack

• Synthetic tenant data

• Reduced-scale services

• Fast feedback loops

• Manual automation triggers

Production Account

• Live tenant workloads

• Customer-facing services

• Scheduled automation

• Alerting

• Full-scale infrastructure

The key principle is simple:

No application workloads run in the Management account.

It exists to operate the platform, not host it.

The Deployment Flow

The deployment path now looks like this:

1. Code lands on the main branch.

2. CI/CD runs in the Management account.

3. The pipeline assumes a deployment role in Dev.

4. Integration tests run.

5. The same artifact is promoted into Prod.

Production never builds code.

Production never owns source repositories.

Production simply runs validated artifacts.

That separation removes a surprising amount of operational risk.

What I Chose Not to Build Yet

Architecture decisions are often more about what you don’t build than what you do.

No Staging Environment

Yet.

A true staging environment means maintaining a production-like replica.

That’s useful.

It’s also expensive.

At our current scale, good automated testing in Dev provides far more value than maintaining another full environment.

The architecture already supports a staging account.

I simply chose not to pay for it yet.

No Tenant-Level VPC Isolation

Today tenant isolation is logical rather than physical.

Data boundaries are enforced at the application and storage layers.

Per-tenant VPCs or accounts add stronger isolation, but also significant complexity.

Most SaaS companies don’t need that level of separation until enterprise customers explicitly require it.

We’re not there yet.

No Dedicated Security Account

AWS would ideally have:

• Management

• Security

• Dev

• Prod

with centralized immutable audit storage.

I agree with that architecture.

I also believe infrastructure maturity should follow business reality.

We’ll add a dedicated security account when compliance requirements justify it.

Not before.

My Rule of Thumb for Startups

If you’re building a SaaS product and you’ve moved beyond proof-of-concept, my recommendation is straightforward:

Start with Three Accounts

Management.

Development.

Production.

The setup overhead is measured in hours.

The long-term safety benefits are measured in years.

Don’t Rush Into Staging

Build staging when your release process genuinely requires it.

Not because every architecture diagram on the internet includes one.

Use AWS Organizations Early

Retrofitting Organizations later is painful.

Starting with it is easy.

Future-you will be grateful.

Treat the Management Account as Permanent Infrastructure

Products evolve.

Tooling evolves.

The operational foundation should remain stable.

The Management account owns the plumbing, not the application.

Looking Ahead

The current structure gives me a clear path forward.

When the platform grows, I can add:

• A staging account

• A dedicated security account

• Enterprise tenant accounts

Each becomes an extension of the existing architecture rather than a redesign.

That’s exactly what I wanted.

The Takeaway

The strongest boundary in AWS is the account boundary.

Use it deliberately.

For me, three accounts turned out to be the minimum structure that delivered:

• blast-radius isolation,

• clean credential scoping,

• predictable deployments,

• and clear cost attribution.

Not because it’s elegant on a diagram.

Because it makes operational mistakes dramatically less expensive.

Parjanya v2.0 is being built with longevity in mind.

The three-account architecture is one of the quiet decisions that makes everything else possible.

The architecture looks clean on a diagram.

Reality was messier.

Moving Parjanya from a single AWS account to a three-account model meant refactoring Terraform state, redesigning IAM trust relationships, rebuilding deployment pipelines, and discovering that AI-generated infrastructure code is only as good as the context you give it.

Tomorrow, I’ll share the implementation story—what broke, what surprised us, and why the hardest part of infrastructure isn’t Terraform itself, but managing context across systems, repositories, and teams.

This is part of my ongoing field notes from building Parjanya v2.0 — a multi-tenant image QA platform. I write about the architectural decisions, trade-offs, mistakes, and lessons learned while building it in public.

A few weeks ago, one of the photographs from this series, Vamana, was selected as a finalist in the Black & White category of the ReFocus Awards.

The image sparked an unexpected response. Friends who had seen it over the past several months kept asking the same questions:

I promised to write about it when I first shared the image shortly after returning from Antarctica in October 2025. Like many unfinished promises to myself, it remained buried beneath work, travel, writing and countless photographs waiting to be processed.

The ReFocus recognition felt like the right moment to finally tell the story.

Because these photographs were never planned as a project.

They emerged gradually during the journey.

And like many of my favourite images, they began with a problem.

The Problem of Scale

Antarctica, South Georgia and the Falkland Islands repeatedly confronted me with a challenge that a single photograph struggled to solve.

Scale.

A portrait could show an individual bird.

A landscape could show the colony.

But neither could convey what it felt like to stand there.

One of the earliest moments came on Steeple Jason Island in the Falklands.

Before landing, we had been briefed that we would be visiting one of the world’s largest Black-browed Albatross colonies.

I knew the numbers.

I had seen the photographs.

But numbers and photographs do not prepare you for the experience.

As I climbed the slope and looked across the colony, thousands upon thousands of albatrosses occupied the hillsides in every direction.

My instinct was to photograph an individual bird.

My second instinct was to photograph the colony.

Neither image felt complete.

The bird lacked context.

The landscape lacked intimacy.

I wanted both.

That was the moment the experiment began.

Discovering Multiple Exposure

The Canon R5C includes an in-camera multiple exposure mode.

It is a feature that many photographers know exists but few use regularly.

I had experimented with it before, but mostly as a creative exercise.

This time it felt useful.

Instead of creating a portrait and a landscape separately, I wondered whether I could combine them into a single frame.

Not digitally.

Not later in Photoshop.

But in-camera.

The goal wasn’t artistic abstraction.

At least not initially.

The goal was documentation.

I wanted to document how the place felt.

My settings remained remarkably simple throughout the project:

• Canon R5C

• RF 100-500mm lens

• Multiple Exposure mode

• Continuous shooting

• Additive/Luminance blending

• Mostly 2 exposures per image (3 exposures only for Vamana)

Luminance blending proved especially important.

It allowed brighter areas from one frame to merge naturally with darker regions from another, creating layered images that retained structure without becoming visually chaotic.

The process quickly became addictive.

Instead of searching for a decisive moment, I began searching for relationships between moments.

Rajahamsa

The first successful image from the experiment became what I later titled Rajahamsa.

A Black-browed Albatross emerged from within its own colony.

The bird was simultaneously an individual and a multitude.

The more I looked at the image, the less it felt like a documentary photograph.

Something symbolic had appeared.

At the time I couldn’t fully explain it.

I only knew the image felt different.

It seemed to contain an idea rather than merely a subject.

Looking back now, I realize that was the first indication that the project was heading somewhere unexpected.

From Observation to Archetype

The transformation happened gradually.

Not during editing.

Not even during shooting.

It happened while looking.

Many of the stories we encounter in childhood never truly leave us.

They remain dormant somewhere beneath conscious thought.

Standing among vast colonies of birds and seals at the edge of the world, those stories began resurfacing.

Not literally.

Symbolically.

The animals remained animals.

But they also began to resemble archetypes.

The same way clouds occasionally resemble faces.

Or mountains resemble sleeping giants.

The photographs were becoming bridges between observation and imagination.

Vamana

The turning point arrived with a lone penguin(Aďelé) standing alone an ice scape and there near by an immense colony of Adele and Rockhopper (I recall!)

Unlike the other images in the series, Vamana required three exposures.

An individual penguin.

The colony below.

The surrounding landscape.

When the image appeared on the camera screen, something immediately felt familiar.

A small figure.

An immeasurable world.

A paradox of scale.

It reminded me of Vamana, the dwarf incarnation who expands beyond expectation and measures the universe itself.

The connection was not deliberate.

It emerged after the photograph existed.

The mythology wasn’t guiding the image.

The image was awakening the mythology.

For the first time I began wondering whether these photographs might belong together as a series.

Jatayu

If there is a photograph that convinced me the project was real, it was Jatayu.

Captured among the king penguin colonies of South Georgia, the image combines a chick’s portrait with thousands of penguins spread across a glacial valley.

The resulting figure feels monumental.

Protective.

Almost watchful.

The glacier behind it only amplifies that sensation.

I remember staring at the rear LCD and immediately thinking:

“That looks like Jatayu.”

Not because it resembled any traditional depiction.

But because it carried the emotional weight I associated with the character.

Courage.

Guardianship.

Sacrifice.

For the first time, I stopped thinking about multiple exposure as a technique.

And started thinking about it as a language.

When Wildlife Becomes Myth

One question I am often asked is whether these photographs were intended to illustrate Hindu mythology.

The answer is no.

At least not consciously.

I wasn’t searching for Vamana.

I wasn’t searching for Jatayu.

I wasn’t searching for Rajahamsa.

I was searching for ways to describe experiences that exceeded the limits of a single frame.

The mythology appeared later.

What fascinates me now is that this happened almost automatically.

Faced with extraordinary wildlife spectacles, my mind reached for archetypes I had inherited through stories.

Perhaps that is what myths have always done.

They give us a language for experiences that feel larger than ordinary reality.

Technical Notes

All photographs in this series were created entirely in-camera using the Canon R5C.

No compositing was performed in post-processing.

The images rely on:

• Multiple Exposure mode

• Continuous shooting

• Luminance blending

• Telephoto compression using the RF 100-500mm lens

Most images use two exposures.

Vamana uses three exposures.

What appears complex is actually the result of repeated experimentation in the field and a willingness to embrace unpredictability.

The final image often revealed itself only after the shutter sequence was complete.

Looking Back

When I boarded the expedition vessel, I expected wildlife photography.

I did not expect a conversation with memory.

Yet that is ultimately what this project became.

The Southern Ocean presented extraordinary animals, extraordinary landscapes and extraordinary scale.

Multiple exposure allowed those experiences to coexist within a single frame.

And somewhere between observation and imagination, the birds became archetypes.

Not because they ceased being wildlife.

But because they became something more.

A reminder that every act of seeing is shaped by the stories we carry with us.

more images from the series arrived unexpectedly,

Matsya → Preservation of life

While photographing a Gentoo penguin colony in Antarctica, I created a multiple exposure that combined the landscape, the colony, and a solitary penguin. Later, as I revisited the frame, it reminded me of Matsya, the fish incarnation of Vishnu in Hindu mythology.

In the ancient story, Matsya appears during a great deluge and guides a vessel carrying life through the flood, ensuring continuity when the world stands on the edge of destruction. In this photograph, the expedition ship (our own Ortelius MV) rests in the distant background while the penguin emerges as a quiet, symbolic guide in the foreground. The resemblance was not planned in the field, but revealed itself later—one of those moments where photography becomes less about recording what is seen and more about discovering meaning within what was witnessed.

Whether coincidence or subconscious association, the image felt like a reminder that stories often exist before we recognize them. The camera records light; imagination completes the journey.

Varaha → Restoration of the world

By the end of the expedition, I had stopped looking for specific characters. Instead, I simply observed and waited for forms to emerge on their own. Varaha appeared unexpectedly—in the shape of ice, snow, and a colony scattered across the shoreline. In Hindu mythology, Varaha rescues the Earth from cosmic waters and restores balance to the world. Standing in Antarctica, a place that constantly reminds us of both fragility and endurance, I found the symbolism difficult to ignore. It became a fitting final archetype: not a story of survival alone, but of renewal.

I will be Continuing the search for new archetypes in new landscapes in the future

What began as an experiment has quietly become an ongoing creative pursuit. I now find myself actively searching for new characters and visual metaphors wherever I travel. They may emerge from different landscapes, different species, and different cultures, but the underlying idea remains the same: finding stories that exist somewhere between reality and imagination. More than anything, this project reminded me how much joy there is in playful exploration. I had an incredible amount of fun creating these images, and I look forward to discovering where the next character appears. You can follow my work here

The challenge was never collecting information.

The challenge was remembering it, connecting it and turning it into something useful.

Modern AI is remarkably capable.

Yet most systems remain surprisingly forgetful.

Every conversation begins from scratch.

Every project requires context to be reintroduced.

Every insight risks being lost in another folder, notebook or document.

The more I worked across research, photography, software, storytelling and archival work, the more obvious the problem became:

Knowledge was accumulating faster than understanding.

Instead of building another assistant, I started exploring a different idea.

What would a genuine second brain look like?

Not a chatbot.

Not a search engine.

Not a note-taking application.

A system that could:

Observe → Remember → Understand → Create → Preserve

A system capable of connecting ideas across years rather than conversations.

A system that becomes more valuable as its memory grows.

From Information to Understanding

Most AI systems operate on prompts.

A second brain should operate on context.

Every note should strengthen understanding.

Every project should enrich memory.

Every experiment should improve future decisions.

The objective is not simply to retrieve information.

The objective is to build accumulated intelligence.

The Emergence of Smriti

This thinking eventually evolved into a concept I now call:

Parjanya’s Second Brain

At its core sits Smriti.

Smriti is not simply another product.

It is the memory and intelligence layer that connects everything else.

A system designed to preserve context, connect knowledge and build long-term understanding.

Around this core are four complementary domains:

Parjanya

Visual Intelligence

Image quality assessment, visual understanding, benchmarking and insight generation.

WilderhoodTV

Creative Intelligence

Storytelling, production workflows, media creation and distribution.

Smriti

Rooted Intelligence

Memory, context, reflection and accumulated knowledge.

Prakriti

Archival Intelligence

Documenting and preserving wildlife, nature, heritage and field knowledge.

Together they form a connected intelligence ecosystem rather than a collection of independent products.

Why Context Matters

The AI industry is understandably focused on models.

Bigger models.

Smarter models.

Faster models.

But I increasingly believe the next frontier may be something else.

Context.

Models will continue to improve.

What remains unique is the context we accumulate.

Our notes.

Our observations.

Our projects.

Our experiences.

Our decisions.

Our understanding of the world.

Without context, intelligence becomes generic.

With context, intelligence becomes personal.

Looking Ahead

This is still an evolving idea.

The architecture will change.

The tools will change.

The models will change.

What remains constant is the underlying belief:

Knowledge should not disappear. It should compound.

The goal is not to automate thought.

The goal is to augment it.

To build a system that helps transform information into understanding, understanding into wisdom and wisdom into meaningful work.

Guiding Principle

Observe → Remember → Understand → Create → Preserve.

DISCLAIMER: This article is part of my personal learning journey and is meant purely for educational and informational purposes. I have drawn insights from Prof. Aswath Damodaran’s publicly available teachings and financial information available in the public domain.

While preparing this write-up, I referred to platforms such as Screener and AI-based study tools to better understand the concepts and validate publicly available data such as company filings and information available for investors from the company domain. The interpretations and conclusions shared here are entirely my own, and any errors or misunderstandings are solely my responsibility.

This is not investment advice or a recommendation to buy, sell, or hold any securities. I encourage readers to do their own research and consult a SEBI-registered investment advisor before making any investment decisions.

Key Takeaways

• Info Edge is best understood as two businesses under one roof: a dominant internet classifieds franchise and a highly successful venture investment engine.

• Traditional valuation multiples can be misleading because a significant portion of the company’s value comes from listed and unlisted investments.

• A sum-of-the-parts framework, inspired by Aswath Damodaran’s valuation approach, is the most appropriate way to analyze the company.

• The core operating business appears to be a high-quality, cash-generative franchise with strong margins and durable competitive advantages.

• At current prices, the stock appears to be trading around fair value rather than at a significant discount.

• Future returns will likely depend on both the continued strength of the Naukri ecosystem and the success of the next generation of startup investments.

When I look at Info Edge, I do not see a normal listed company. I see two businesses stitched together: a dominant, cash-generating internet classifieds franchise led by Naukri, and a venture-style investment engine that has produced extraordinary value through stakes such as Eternal and PB Fintech.

That is exactly why I think Info Edge is one of the best Indian case studies for applying Professor Aswath Damodaran's framework. Damodaran has repeatedly argued that cash, cross-holdings, and other non-operating assets should be valued separately from the operating business. In his discussions on holding companies and cross-holdings, he notes that the "sum of the parts" can often be greater than the whole when these assets are not properly accounted for. For a company like Info Edge, where listed and unlisted investments contribute materially to shareholder value, a sum-of-the-parts approach is not optional—it is essential.1

In Info Edge’s case, that distinction is not academic. As of June 2026, the company itself had a market capitalization of roughly ₹65,000 crore, while a substantial part of its value sat in listed investments and a startup portfolio that management valued at nearly ₹37,000 crore.

So the right question is not, “Is Naukri expensive on earnings?”

The right question is:

How much am I paying for the core franchise after adjusting for Eternal, PB Fintech, and the next cohort of private investments?

Why Info Edge Is a Perfect case study for understanding Prof. Damodaran’s lens

Damodaran’s valuation philosophy begins with a story and then forces the numbers to be consistent with that story.

My story for Info Edge is simple.

The first engine is a high-quality operating business with a durable moat in recruitment classifieds, strong operating leverage, and real free-cash-flow discipline.

The second engine is a capital allocator with a demonstrated ability to identify, back, and hold asymmetric winners for long periods.

That combination makes Info Edge unusual.

Many companies are either operating businesses or investment vehicles.

Info Edge is both.

Naukri remains the core cash machine, but the market’s view of the company is also shaped by what happened with Zomato—now Eternal—and with Policybazaar through PB Fintech.

This also explains why plain P/E ratios can mislead. Reported earnings often include large amounts of investment-related income and mark-to-market effects, creating accounting noise that distorts headline valuation multiples.

The Operating Story I Am Telling Myself

When I strip away the investment book, I am left with a business that is significantly better than a typical Indian internet company.

FY25 standalone revenue was approximately ₹2,654 crore, operating profit was around ₹973 crore, and the starting EBIT margin in my model was roughly 36.7%.

That is not what a fragile, capital-hungry internet business looks like.

It looks like a platform business with real pricing power and low incremental capital intensity.

Info Edge’s own disclosures increasingly describe Naukri as evolving from a traditional job-search platform into a broader talent ecosystem. That matters because deeper workflow relevance typically strengthens both customer stickiness and monetization potential.

The cash-flow quality is equally important.

Free-cash-flow conversion remains strong, and cash generation has consistently supported both internal reinvestment and external venture investments.

That cash discipline sits at the center of my thesis.

In a Damodaran framework, earnings are only useful if they can be converted into cash and distributed or reinvested intelligently.

Info Edge passes that test far better than many businesses trading on technology narratives.

The Second Engine: Capital Allocation

The other half of the story is where Info Edge becomes truly distinctive.

Management disclosed that total invested capital across its startup platform stood at roughly ₹4,000 crore, while the fair market value of those investments approached ₹37,000 crore.

Those are extraordinary numbers.

More importantly, they fundamentally change how I think about the balance sheet.

This is not idle treasury cash.

This is a portfolio that has already produced two listed winners and still carries meaningful optionality from the rest of the investment book.

The listed winners are the easiest to value.

Using June 2026 market values, Info Edge’s stakes in Eternal and PB Fintech together were worth approximately ₹39,500 crore.

That single number explains why this company must be valued in pieces.

If the market capitalization is roughly ₹65,000 crore and the two listed holdings alone are worth nearly ₹40,000 crore, then the market is implicitly assigning the remaining value to the core operating franchises plus the unlisted portfolio.

How I Split the Business

Damodaran’s treatment of cash and cross-holdings provides the right starting point.

First, value the parent company’s operating assets.

Second, value the holdings separately.

Third, decide whether those holdings deserve to be added at full value or at a discount because of taxes, opacity, or governance considerations.

For Info Edge, I split the company into three buckets:

• The core operating business: Naukri, 99acres, Jeevansathi, Shiksha, and other controlled operating assets.

• The listed investment book: Eternal and PB Fintech.

• The private portfolio: the broader startup portfolio, valued through probability-weighted assumptions and appropriate haircuts.

Once I do that, the valuation becomes much clearer.

I no longer have to force one multiple to explain two fundamentally different economic engines.

My Core-Business DCF

I built three versions of the core-business valuation.

The first was a conventional FCFF model.

The second was a stricter Damodaran-style build with a higher cost of capital and explicit return-on-capital fade.

The third was an owner-earnings model that treats Info Edge more like a high-ROIC platform business with low maintenance reinvestment requirements.

The conventional FCFF model produced an intrinsic value range of:

₹805.83 to ₹1,173.05 based on Bear to Bull scenario

The stricter Damodaran build produced:

₹624.07-₹675.71-₹722.74(Bear-Base-Bull)

The owner-earnings model produced:

Bear₹856.76, Base₹978.00, Bull₹1,150.75

I do not treat any one of these as the truth.

I treat them as a range generated by different but intellectually honest stories about the business.

That is very much in the spirit of Damodaran: valuation is a disciplined conversation between narrative and numbers, not a hunt for a fake precision point estimate.

What the Three Models Are Really Saying

Taken together, these models say something simple.

Info Edge is not obviously cheap.

But it is also not wildly expensive if I believe the core operating franchise deserves a premium for moat, cash conversion, and capital allocation quality.

The overlap zone between my more realistic base cases falls roughly between ₹950 and ₹1,000 per share.

That sits remarkably close to where the market currently values the stock.

My conclusion is not that the stock is a bargain.

My conclusion is that the stock is trading around intrinsic value, with modest upside or downside depending on how optimistic I choose to be about the operating business and how much of the private investment book I am willing to recognize.

Why the Private Portfolio Cannot Be Ignored

A lot of investors stop after marking Eternal and PB Fintech.

I think that is incomplete.

It is easy to value listed stakes and pretend the rest of the startup book is noise.

Management’s disclosures suggest otherwise.

At the same time, I do not want to value the private portfolio lazily.

For early-stage companies, the right framework resembles venture-capital valuation rather than traditional DCF analysis.

Estimate a plausible exit value.

Assign a probability of success.

Adjust for dilution.

Discount back at a sufficiently high required return.

That is why, in my own sum-of-the-parts framework, I do not add the startup portfolio at full value by default.

I use scenario-based weights and meaningful haircuts rather than assuming every fair-value mark will eventually translate into realized gains.

Promoter Skin, Moat, and Cash Discipline

My three favourite lenses for evaluating businesses are:

1. Moat

2. Promoter skin in the game

3. Free-cash-flow discipline

Info Edge scores well on all three.

Promoter ownership remains meaningful at approximately 37.5%, with founder Sanjeev Bikhchandani individually holding about 24% of the company. This level of ownership creates strong alignment between management and minority shareholders, particularly in a business where long-term capital allocation is a major driver of value creation.

When founders continue to own a meaningful portion of the business after decades of operation, I generally have greater confidence that capital allocation decisions are being made with a long-term mindset.

On moat, the easiest evidence is visible in the economics of the core business.

A business generating EBIT margins north of 35%, producing strong free cash flow, and maintaining leadership in a category for years is rarely doing so by accident.

Those economics are often downstream manifestations of:

• Brand strength

• Network effects

• Embedded workflows

• Customer inertia

• Rational competition

Naukri benefits from several of these simultaneously.

Employers want access to the largest candidate pool.

Candidates want access to the largest number of recruiters.

That creates a reinforcing flywheel that becomes increasingly difficult for competitors to replicate.

On cash-flow discipline, the story becomes even stronger.

The company has remained largely debt free while continuing to:

• Fund organic growth

• Invest in adjacent businesses

• Build a venture portfolio

• Maintain balance-sheet flexibility

That combination is surprisingly rare.

Many firms with venture portfolios lack a strong operating engine.

Many cash-generative firms struggle to identify high-return external investments.

Info Edge has demonstrated an ability to do both.

The Buy-Below Levels I Would Use

Because the stock appears to be trading around fair value, I think price discipline matters more than enthusiasm.

Valuation should influence position sizing.

Quality tells me what to buy.

Valuation tells me when to buy.

My framework currently looks like this:

ZoneInterpretationBelow ₹850Strong Buy₹850–₹930Buy₹930–₹1,030Hold / WatchAbove ₹1,100Avoid aggressive additions

Below ₹850

At these levels, even relatively conservative valuation frameworks begin to show a meaningful margin of safety.

The investment case becomes less dependent on perfect execution and more dependent on simply avoiding major mistakes.

That is usually where I prefer to deploy capital.

₹850–₹930

The valuation starts tilting in my favor while still allowing room for execution risk.

The core franchise remains attractive and the optionality from investments is available at a more reasonable price.

₹930–₹1,030

This is approximately fair-value territory.

At these levels I would be comfortable holding existing positions but would not feel a strong urge to increase exposure.

Above ₹1,100

At these valuations, I would need a substantially more optimistic view regarding:

• Core business growth

• Margin expansion

• The next generation of startup winners

• Long-term value creation from the investment portfolio

Without that conviction, future returns become increasingly dependent on continued multiple expansion rather than fundamental value creation.

What Could Break My Thesis

No valuation is complete without understanding what can go wrong.

There are four developments that would make me materially less constructive on Info Edge.

1. Naukri Loses Pricing Power

Naukri remains the economic anchor of the entire group.

If recruiters begin reducing spend, switching platforms, or finding better alternatives, the valuation of the entire business changes.

The investment portfolio may be exciting, but the operating franchise pays the bills.

A weakening recruitment moat would force me to revisit almost every assumption in my model.

2. Adjacent Businesses Fail to Improve Economics

The long-term narrative depends on businesses such as 99acres and other operating segments continuing to improve efficiency and monetization.

If these businesses fail to progress toward stronger economics, some of the optionality embedded in my valuation disappears.

The result would be lower growth assumptions and reduced confidence in margin expansion.

3. Capital Allocation Discipline Deteriorates

One of the biggest strengths of Info Edge has been its investment discipline.

Successful startup investors often face a dangerous temptation after early wins.

Past success can encourage larger bets, weaker underwriting standards, or excessive optimism.

If management begins prioritizing valuation marks over governance and business quality, I would immediately apply a larger discount to the investment portfolio.

The track record has earned credibility.

But credibility must continually be re-earned.

4. A Prolonged Decline in Eternal or PB Fintech

The market currently recognizes Info Edge not only as an operating company but also as a successful allocator of capital.

If Eternal and PB Fintech experience significant and prolonged declines, investors may stop assigning value to that allocator identity.

In that scenario, the market could increasingly focus only on the operating business while assigning lower value to the investment portfolio.

That would compress valuation multiples even if the underlying businesses remain fundamentally sound.

The Bigger Lesson

One reason I enjoy studying Info Edge is that it demonstrates how valuation is often more about structure than forecasting.

Most investors spend their time debating whether revenue growth will be 12% or 15%.

In many situations, that is not where the real insight lies.

The real insight is understanding what exactly you are valuing.

Info Edge forces that discipline.

You cannot treat it as a simple internet stock.

You cannot treat it as a venture capital fund.

You cannot treat it as a holding company.

It is all three at the same time.

And that complexity creates both confusion and opportunity.

This is precisely the type of situation where Damodaran’s framework shines.

Separate the pieces.

Value each piece independently.

Then bring them back together.

Simple in principle.

Difficult in practice.

My Final Verdict

When I put everything together, I keep arriving at the same conclusion.

Info Edge is a high-quality business with:

• A genuine moat

• Meaningful promoter alignment

• Strong free-cash-flow generation

• Exceptional capital-allocation history

• Significant optionality through its startup portfolio

Those are characteristics I actively seek in long-term compounders.

However, quality alone does not make something cheap.

At roughly ₹1,010 per share, I believe the market already recognizes much of what makes Info Edge special.

The Naukri franchise is appreciated.

The Eternal success story is appreciated.

The PB Fintech investment is appreciated.

A significant portion of the optionality is already reflected in the price.

That leaves me with a conclusion that may sound less exciting but is probably more useful:

Info Edge is not a deep-value opportunity today.

It is a high-quality business trading around intrinsic value.

Would I own it?

Yes.

Would I chase it aggressively?

Probably not.

Would I become significantly more interested if the market offered it closer to ₹850–₹900?

Absolutely.

If I were forced to summarize my view in a single sentence, it would be this:

Info Edge is one of the clearest examples in India of how Aswath Damodaran’s framework helps separate a great operating business from a great capital allocator living inside the same stock.

And that is precisely what makes it such a fascinating company to study.

Disclosure

This article is intended solely for educational and research purposes and should not be construed as investment advice. The valuation frameworks, assumptions, and conclusions presented here reflect my personal analysis and may prove incorrect. Investors should conduct their own due diligence and consider their risk tolerance before making any investment decisions.

The crimson crowns of cranes against fresh snow. The blue hues of winter mornings. The warm glow of sunrise spilling across frozen landscapes.

Naturally, those scenes became the foundation of C1 — Vibrant Hokkaido.

But as the journey unfolded, I found myself increasingly drawn to something else.

Not colour.

Time.

The more I travelled through Hokkaido, the more I felt that many of its landscapes, traditions, and wildlife existed outside the rhythm of the modern world. A crane standing motionless in the snow, a sea eagle gliding across a winter valley, steam rising through an outdoor onsen—these scenes felt as though they could belong to today, fifty years ago, or perhaps even longer.

That feeling became the inspiration for C2.

Why Monochrome?

For me, monochrome was never about removing colour.

It was about removing time.

Colour often anchors a photograph to a specific moment. We recognise the palette of a season, the look of a particular era, or the visual characteristics of modern life.

Monochrome strips away those references.

What remains are the essential elements of a photograph:

• Light

• Form

• Texture

• Atmosphere

• Emotion

More importantly, it allows a scene to feel timeless.

In Hokkaido, monochrome became the most natural way to express what I was experiencing. The enduring elegance of Japanese culture, the quiet beauty of winter landscapes, and the sense that many traditions continue largely unchanged despite the rapid pace of the world around them.

Rather than documenting Japan as it appeared before me, I wanted to create images that felt suspended between past and present.

Building C2

The objective was not to create dramatic black-and-white photographs.

Instead, I wanted to create images that felt calm, restrained, and enduring.

The settings were designed to emphasize tonal separation rather than contrast alone. Snow remained soft and luminous. Shadows retained detail. Blacks provided structure without overwhelming the frame.

The result was a visual language inspired less by spectacle and more by permanence.

A crane became a silhouette against endless snow.

An eagle became a dark brushstroke against mountain ridges.

Steam transformed a crowded onsen into something almost dreamlike.

The photographs began to feel less like records of a journey and more like fragments of memory.

What I Found

The most unexpected outcome was how much attention shifted from the subject to the feeling of the scene.

Without colour competing for attention, subtle details became more important:

The curve of a crane’s neck.

The delicate footprints pressed into fresh snow.

The layers of mist drifting through an onsen.

The patterns of winter forests beneath a soaring eagle.

Many photographs became simpler, yet somehow more evocative.

The absence of colour encouraged me to focus on what initially drew me to the frame rather than what made it visually striking.

The Images

The red-crowned cranes became symbols of grace rather than colour.

Against the snow, they appeared almost sculptural—reduced to elegant lines, shapes, and movement.

The sea eagle soaring across Hokkaido’s mountains felt timeless, its silhouette echoing the rugged landscape beneath it.

The onsen scene revealed a different side of Japan. Visitors gathered quietly in rising steam while nature enveloped the entire experience, blurring the distinction between people and place.

Even moments of intentional blur and abstraction began to feel more meaningful in monochrome, emphasizing atmosphere over description.

Each image seemed less concerned with documenting what was there and more interested in conveying what it felt like to stand there.

What C2 Taught Me

Photography often encourages us to capture moments.

Monochrome encouraged me to capture permanence.

It reminded me that some places possess a character that transcends a single season, a single journey, or even a single generation.

Hokkaido’s wildlife, landscapes, and traditions carry that quality.

By removing colour, I wasn’t simplifying the story.

I was revealing the part of it that felt enduring.

C2 — Traditional Japan (Monochrome) is my attempt to explore the timeless side of Hokkaido—a visual journey through landscapes, wildlife, and traditions that feel as relevant today as they might have decades ago, expressed through a monochrome palette that celebrates simplicity, atmosphere, and the enduring beauty of Japan.

I wasn’t researching locations.

I wasn’t building shot lists.

I wasn’t even looking at wildlife photography.

Instead, I was revisiting old Japanese films and spending far too much time pausing anime scenes.

As someone who grew up watching anime, I had always been fascinated by how Japanese artists treated landscapes. Snow felt different. Forests felt different. Even ordinary roads seemed to carry a sense of atmosphere that was difficult to describe but immediately recognizable.

The colours were often richer than reality, yet never felt artificial.

The landscapes felt vibrant, but never loud.

Somewhere between memory and observation, a distinct visual language emerged.

Without realizing it, I had already started naming it in my head.

Vibrant Hokkaido.

At the time, I thought it was a colour problem.

By the end of the trip, I realized it was a luminance problem.

And that distinction changed everything.

Designing Hokkaido Before Seeing Hokkaido

Long before boarding a flight to Japan, I started building two custom shooting profiles on my Canon EOS R5 Mark II.

The first became C1: Vibrant Hokkaido.

The second became C2: Traditional Japan.

The intention behind C1 was relatively straightforward. I wanted a profile that embraced the visual qualities I associated with modern Japanese landscape art and anime backgrounds—strong colour separation, rich blues, warm wildlife subjects and an overall sense of atmosphere.

The foundation was Canon’s Landscape Picture Style, but pushed further through additional contrast, saturation and colour tuning.

I wasn’t trying to reproduce exactly what I saw.

I was trying to reproduce what Hokkaido felt like in my imagination after years of consuming Japanese visual culture.

C2 was built for a completely different reason and deserves its own article, but in short, it was inspired by older Japanese cinema, monochrome photography and traditional aesthetics where form, shape and negative space matter more than colour.

At the time, both profiles felt complete.

I thought I had already solved the problem.

Hokkaido had other plans.

The Surprise Waiting in the Snow

One of the assumptions I carried into the trip was that colour would be the defining characteristic of the landscape.

After all, C1 was built around colour.

Yet once I started photographing winter scenes, I noticed something unexpected.

The strongest images weren’t necessarily the most colourful ones.

In fact, many of them contained remarkably little colour.

The fox crossing an icy road became one of those moments.

Initially, what attracted me to the scene was obvious.

A warm-toned fox against a cold blue landscape.

Classic colour contrast.

The kind of scene that fits perfectly within the original idea of Vibrant Hokkaido.

But as I reviewed the images later, I found myself paying less attention to the fox and more attention to the snow.

The snow wasn’t acting as background.

It was acting as structure.

The wind patterns across the frozen surface were creating visual pathways.

The lighter areas pulled the eye forward.

The darker areas acted as leading lines.

The fox wasn’t creating the composition.

The snow already had.

The fox was simply completing it.

That realization became one of the most important lessons of the trip.

Snow Isn’t White

Photographers often talk about exposing snow correctly.

What I discovered in Hokkaido was slightly different.

The challenge wasn’t exposure.

The challenge was preserving luminance relationships.

When we think about snow, we instinctively think about whiteness.

But Hokkaido snow rarely appears truly white.

It contains subtle blues.

Grey tones.

Cyan shifts.

Tiny variations that create depth and dimension.

Push everything toward pure white and the landscape becomes flatter.

Neutralize the colour too aggressively and winter loses its atmosphere.

The more I photographed, the more I realized that the visual identity of Hokkaido wasn’t being created by colour alone.

It was being created by the way snow carries light.

That became the foundation of what I eventually started calling Snow Luminance Control.

Not as a formal technique.

Not as a preset.

Simply as a way of seeing.

The objective became preserving those tonal transitions so that the snow itself could guide the viewer through the frame.

The fox remained important.

But the snow became the protagonist.

The Lightroom Problem

Ironically, the next lesson arrived after I returned from the field.

Opening the RAW files inside Lightroom was a shock.

The images didn’t look anything like what I had seen on the back of the camera.

The carefully tuned colours from C1 seemed to disappear.

The atmosphere felt different.

The tonal relationships felt different.

For a while, I assumed the profile simply wasn’t working as intended.

That confusion continued until a conversation with Rahul Sachdev.

Rahul, a Canon Ambassador and someone whose field experience I deeply respect, pointed me toward something surprisingly simple.

Match Camera.

Once Lightroom and Photoshop were instructed to honour the in-camera rendering through Canon’s camera-matching profiles, the images immediately started resembling what I had designed and seen in the field.

It was one of those moments that seems obvious in hindsight.

I had spent months creating a visual language inside the camera, only to discover that my software workflow was quietly replacing it.

That lesson alone probably saved me countless hours of unnecessary editing.

The Limitations of Custom Modes

As useful as the C1 and C2 workflow became, the trip also exposed some practical limitations.

One of the biggest frustrations involved crop settings.

Certain shooting configurations—particularly 1.6x crop workflows that proved useful for wildlife—did not always behave the way I expected once stored inside the custom modes.

Moving between C1, C2 and C3 often required additional verification to ensure the camera was actually operating the way I intended.

It wasn’t a deal-breaker.

But it was a reminder that custom modes are powerful shortcuts rather than perfect state-management systems.

The creative intent remains consistent.

The technical details still need supervision.

Looking Back

What started as an attempt to recreate a visual style inspired by anime eventually became something far more interesting.

I arrived in Hokkaido thinking about colour.

I left thinking about luminance.

The preset I designed before the trip survived.

But the reasoning behind it evolved.

Vibrant Hokkaido was never really about making colours stronger.

It was about allowing winter to remain expressive.

About preserving the subtle pathways hidden inside snow.

About understanding that a landscape can lead the eye long before a wildlife subject enters the frame.

The fox helped me recognize that.

The snow taught me why.

In the next post, I’ll explore the other half of this experiment—C2: Traditional Japan—a profile inspired not by modern anime aesthetics, but by older Japanese cinema, monochrome photography and the idea that sometimes removing colour reveals more than adding it ever could.

I was drawn to the contrast between Hokkaido’s vivid winter character — cranes, foxes, snowfields, isolated color, landscapes — and the quieter monochromatic side of Japan that emerges through mist, overcast skies, bare forests, and restrained tonal palettes.

What I did not anticipate was discovering an entirely different visual phenomenon altogether: what I now think of as layered winter landscapes.

Somewhere during the trip, while photographing frozen wetlands and distant forests through a telephoto lens, I began noticing that the landscape no longer behaved as a single continuous scene. Instead, it appeared to separate itself into distinct visual planes — snowfields, reed belts, dense trunks, atmospheric canopy — each carrying its own tonal identity.

The effect was subtle in reality but became remarkably pronounced through the viewfinder.

That discovery was unexpected.

I had travelled searching for color and monochrome as emotional themes, but Hokkaido quietly revealed a third visual language to me — one built not around color, but around compression, atmosphere, tonal separation, and layered perception.

The more time I spent with these scenes, the more they began feeling less like traditional wildlife photographs and more like studies in visual structure.

What fascinated me most was that this separation only became obvious through the camera itself.

Standing in the landscape, I did not experience the scene as strongly layered in this way. But once framed through a long lens, the environment transformed. The world flattened, distance compressed, and the landscape started behaving almost like a composed arrangement rather than a naturally receding space.

It was as though the camera was not only recording the landscape — but reorganizing it.

Understanding the Layers

The more I photographed these scenes, the more I realized that several visual ideas were quietly working together inside the frame.

Telephoto compression was one of them. Long focal lengths visually compress distance, making separate parts of the landscape appear stacked closer together. Instead of feeling deep and continuous, the world begins separating into flatter layers.

There was also an element of Gestalt perception at play. Human vision naturally groups similar tones, textures, and patterns together. Snowfields, reed belts, forests, and mist each become their own visual regions, which is why a single photograph can sometimes feel like multiple images stacked together.

And finally, there is visual rhythm — the repetition of trunks, rocks, reeds, and negative space across the frame. Winter simplifies the landscape enough for these repeating forms to become more noticeable, almost musical in their spacing and flow.

Individually, these ideas are subtle. But together, they create the layered winter landscapes I unexpectedly found myself drawn toward in Hokkaido.

Searching for a Visual Parallel

Interestingly, none of this was intentional.

I did not travel to Hokkaido inspired by any particular art movement or visual tradition. The connection appeared only later, after returning from the trip and trying to understand why these images felt so visually separated and structured.

That curiosity eventually led me toward ukiyo-e.

While researching whether similar spatial phenomena had existed in historical art, I found myself repeatedly drawn to Japanese landscape prints — particularly the quieter atmospheric works of Utagawa Hiroshige.

What resonated was not necessarily the subject matter, but the way space behaved within those images.

Many ukiyo-e landscapes organize the frame through layers and intervals rather than strong linear depth. Snow, water, mist, forests, and sky often exist as separate visual regions while still remaining part of a unified composition. The eye drifts across the scene rather than moving directly into it.

That sensation felt remarkably close to what I had been experiencing through the viewfinder in Hokkaido.

The similarity was especially striking in winter-oriented prints where forests compress into texture, snow becomes open negative space, and atmosphere quietly separates one plane from another.

What fascinated me was that photography and ukiyo-e arrive at this layered perception through entirely different means.

Ukiyo-e achieved it through artistic interpretation and compositional stylization.

The photographs achieve it through:

• optical compression,

• atmospheric flattening,

• tonal simplification,

• and perceptual grouping.

The overlap, therefore, is not imitation.

It is convergence.

Different mediums.
Different centuries.
Yet arriving at a strangely similar way of organizing space.

That realization changed how I began thinking about these images. I stopped seeing them purely as wildlife or winter photographs and started viewing them as studies in perception — moments where optics quietly transform reality into layered abstraction.

Winter as an Abstraction Engine

I suspect winter may be uniquely suited for this kind of visual experience.

Snow removes clutter.

Color palettes collapse into restraint.
Foreground distractions disappear.
Shadows soften beneath overcast skies.
The world simplifies itself into line, tone, rhythm, and silence.

Under those conditions, landscapes stop behaving purely as physical environments and begin functioning almost like visual structures.

The camera intensifies this transformation further.

The moment the frame edges isolate the scene, the eye stops navigating physical space and instead begins reading intervals:
foreground,
transition,
structure,
atmosphere.

That was the most unexpected lesson Hokkaido offered me.

I arrived searching for vibrant winter and monochrome quietness.
I left discovering layered perception.

Looking Ahead

What began as a winter wildlife trip has quietly reshaped how I now look at landscapes.

I am increasingly interested in searching for places where this phenomenon might emerge again — environments where atmosphere, distance, weather, and optics work together to create layered spatial separation.

Not necessarily dramatic landscapes.
Not necessarily famous locations.

Just places where the world simplifies enough for its internal structure to reveal itself.

Perhaps that is what photography continues to teach me most:
sometimes the camera does not merely document a place.

Sometimes it changes the way we perceive space itself.

The number that matters is this one: how many of your Lambda functions can actually run on ARM64 today — not in theory, not on a clean greenfield stack, but with your actual dependencies, your actual Python version, and your actual C extensions.

For Parjanya, an ML-powered image quality pipeline, the answer started at 1 out of 4. That gap — between what Graviton promises and what your stack allows — is almost entirely explained by a single, under appreciated concept: the ARM wheel.

This article explains what ARM wheels are, why they are the gatekeeper to Graviton savings, how the Parjanya re-architecture navigated them, and how to build a framework for evaluating Graviton readiness in any product.

First, an Honest Framing of the 34% Number

AWS’s stated price-performance advantage for Graviton Lambda comes from two compounding factors:

Pricing: ARM64 Lambda is priced approximately 20% lower per GB-second than equivalent x86 Lambda. This is a direct line-item discount.

Performance: For SIMD-heavy workloads — image processing, compression, media encoding — ARM NEON instruction sets often complete work faster than x86 AVX-512 for the same memory allocation, which effectively reduces billed duration.

In Parjanya’s $10 POC (running PIL-based image processing against Graviton4 and Intel x86), the empirical result was a 24% cost delta — Intel x86 cost 24% more for identical or worse performance. AWS’s 34% claim holds at the higher end because it accounts for the full price-performance package including EC2 and container workloads.

The savings are real. The caveat is that they only materialise if your software stack is compatible.

What Is an ARM Wheel?

Before evaluating Graviton readiness, you need to understand what a Python wheel is and why the ARM variant specifically matters.

A wheel (.whl) is a pre-built binary distribution of a Python package. When you run pip install numpy, pip looks for a wheel that matches your platform. If it finds one, it downloads and installs it directly — no compilation required. If it does not, pip falls back to downloading source code and compiling it locally.

For pure Python packages, this distinction is irrelevant. The bytecode runs anywhere.

For packages with native C/C++ extensions — NumPy, Pillow, OpenCV, PyTorch, SciPy, and most of the ML ecosystem — a wheel is architecture-specific. A wheel built for x86_64 will not install on aarch64. The package must publish a separate wheel for ARM64.

The naming convention tells you exactly what architecture a wheel targets:

numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl   # x86
numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl  # ARM64

The aarch64 suffix is what you are looking for. If it exists, installation is clean. If it does not, you are in source-build territory — and source builds in Lambda container images introduce a layer of complexity that breaks most “just change the architecture flag” migration plans.

The Four States of a Dependency

When evaluating a Python service for ARM64 migration, every dependency falls into one of four categories:

This matrix is the actual output of a Graviton readiness assessment. Not a benchmark. Not a cost model. The wheel matrix.

The Parjanya Case Study: From 1/4 to 3/4

Parjanya’s pipeline had four compute stages, each containerised and isolated. When the first Graviton migration attempt was made, only one — the deduplication stage — made it onto ARM64. The other three were blocked.

Here is what happened at each stage:

Stage 1: RAW Processing (Blocked → Eventually Bypassed)

Camera RAW files require binary parsing to extract embedded JPEGs. The dependency chain pulled in native image parsers without clean aarch64 wheels. Compilation from source failed or produced unstable binaries in the Lambda container environment.

Resolution: The stage architecture changed. Rather than fight a fragile native dependency chain, the pipeline design shifted to extract JPEG previews at ingest — earlier in the flow — reducing the surface area where native RAW parsing was required.

Stage 2: CLIP-IQA Scoring (Blocked → Replaced)

CLIP-IQA was being used for visual quality scoring. It had two problems:

1. Wheel problem: The ARM64 wheel chain for its dependencies was incomplete. Installing it in a Graviton Lambda container was unreliable.

2. Model problem: CLIP-IQA was trained on stock photos, not professional RAW captures. It accepted only ~1% of professional images, making it useless for the actual curation use case.

Resolution: The entire stage was replaced with a rule-based technical validator — exposure clipping detection, blur scoring, sharpness scoring, and perceptual hashing. No ML forward pass. No compiled extensions in the hot path. The new validator ran in pure Python, deployed cleanly on ARM64, and executed in under 100ms per image.

This is the single most important insight from the Parjanya migration:

The fastest path to ARM64 is sometimes not to migrate the workload. It is to replace the workload.

The new parjanya-technical_validator runs on arm64 with Python 3.12 and 512 MB memory. It is not a compromise — it is measurably better: faster, cheaper, more correct, and production-verified.

Stage 3: GPU Inference (Intentional Hard Boundary)

The VLM inference stage runs Qwen3-VL-8B-Instruct and requires CUDA. CUDA is the hard architectural boundary.

CUDA is NVIDIA’s parallel computing platform. It runs on NVIDIA GPUs, which are paired with x86_64 hosts in most production setups including AWS g5 instances. There is no ARM64 CUDA story in Lambda.

Resolution: The GPU inference stage was not migrated. Instead, the control plane was separated from execution:

• A lightweight Lambda — ARM64 — handles scheduling logic: when to scale the GPU fleet, when to drain it

• The actual VLM inference runs on EC2 g5 instances in an Auto Scaling Group

• The Lambda no longer needs ML libraries, so it migrated to Graviton without any dependency issues

This is architectural isolation — moving the ARM64-compatible logic (scheduling, orchestration) onto Graviton and leaving the ARM64-incompatible logic (GPU inference) exactly where it belongs.

Stage 4: Deduplication (Original Success)

Small, CPU-bound, dependency-light. Perceptual hashing with no native C extensions in the hot path. This was the function that migrated cleanly from day one, and it remains the model for what “Graviton-ready” looks like: pure logic, minimal native dependencies, well-supported Python ecosystem.

The Current State

Three of four Lambda functions on Graviton (+1 scheduler). The remaining x86 workload is the one that genuinely belongs there.

Why ARM Wheels Fail: The Root Causes

Understanding why a package lacks an ARM64 wheel helps you predict and resolve failures faster.

1. The package maintainer hasn’t built it yet

Many packages added aarch64 wheel support in 2022–2024 as AWS Graviton gained adoption. Packages that haven’t been updated since then may still not publish ARM64 wheels. This is solvable by checking the current release on PyPI — support often appears in later versions.

2. GLIBC version mismatch

Lambda’s ARM64 runtime uses Amazon Linux 2023. A wheel built against an older GLIBC version (manylinux2014) may not install cleanly in that environment. When you see installation errors that reference GLIBC, this is the cause. The fix is to build your container image from the correct ARM64 base image and verify wheel compatibility inside it — not on your development machine.

3. Transitive dependencies break the chain

Even if the top-level package publishes an ARM64 wheel, its transitive dependencies may not. OpenCV is a canonical example: opencv-python may install, but a transitive dependency on a specific image codec library may not have an ARM64 wheel. The failure surface is the entire dependency tree, not just the packages you explicitly listed.

4. ABI incompatibility

Some packages have version-specific ABI contracts. A package pinned to an older version may have published an x86_64 wheel for that version but not an ARM64 one. When you pin for stability and the pinned version predates ARM64 support, you are stuck — until you unpin.

5. Build-time assumptions in C extensions

Some C extensions were written with x86 assumptions baked into SIMD code paths, memory alignment behaviour, or endianness handling. These may compile on ARM64 but behave incorrectly at runtime. This is rare but dangerous — it is the category where correctness testing matters most.

A Practical Evaluation Framework for Any Product

The following process translates the Parjanya experience into a repeatable assessment for any Lambda-based system.

Phase 1: Dependency Audit (Before Touching Infrastructure)

For every Lambda or containerised service, generate the full dependency tree including transitive dependencies:

pip install pipdeptree
pipdeptree --warn silence

Then, for every package with a native C extension (you can identify these by checking if the installed package contains .so files):

find /path/to/site-packages -name "*.so" | sort

Classify each package against the four-state matrix above. The output is your wheel matrix — one row per native dependency, one column per state.

Phase 2: Build Environment Test (Before Assuming Local Results)

Do not test ARM64 compatibility on your development machine. Build inside an ARM64 environment:

# Docker on an Apple Silicon Mac or a Graviton EC2 instance
docker build --platform linux/arm64 -f Dockerfile.arm64 .

The Lambda execution environment is Amazon Linux 2023 on aarch64. Your build environment should match this exactly. A dependency that installs cleanly in a generic ARM64 Docker image may still fail in the Lambda runtime — validate inside the correct base image.

Phase 3: Runtime Validation (Beyond Installation)

Successful installation is not the same as correct runtime behaviour. For packages that made it through the build:

• Run your actual workload, not a smoke test

• Measure latency and memory usage under ARM64 — they may differ from x86 even when the package installs cleanly

• For any package in the “source build required” category, verify that the compiled binary links against the correct system libraries at runtime

Phase 4: The Three-Question Decision Gate

For each service in your stack, answer these three questions:

1. Does every native dependency have an ARM64 wheel for my Python version and target runtime?

   • If yes: proceed to migration

   • If no: go to question 2

2. Can I build from source reliably in CI, in the correct ARM64 environment, with the correct GLIBC version?

   • If yes: proceed with source-build path; add a CI gate that fails if compilation regresses

   • If no: go to question 3

3. Can I replace the dependency or the entire stage with something ARM64-compatible?

   • If yes: replace it — this is often the better outcome anyway

   • If no: this stage stays on x86; isolate it intentionally rather than blocking the rest of the migration

This decision tree is what the Parjanya migration followed, even if it was not formalised until the second iteration.

Phase 5: Architectural Boundary Declaration

For every stage that cannot move to ARM64, declare the boundary explicitly in your architecture documentation:

• What is the technical reason it stays on x86 (CUDA dependency, missing wheel, untested behaviour)?

• What would need to change for it to become migratable?

• Is the x86 stage fully isolated so that the ARM64 migration of other stages is not blocked by it?

The goal is intentional architecture, not incomplete migration. A GPU inference stage that stays on x86 by design is not a failure. It is a correct architectural boundary.

What This Means for New Products: Building for ARM Portability

If you are building a new product and want to default to Graviton for cost efficiency, the considerations are different from migration. The choices you make at the start of your architecture will determine how much of the system can run on ARM64.

Choose dependency-light stages where possible

The deduplication stage in Parjanya worked on Graviton from day one because it had minimal native dependencies. When designing a new service, ask whether a pure-Python or well-supported-wheel implementation can solve the problem before reaching for a heavier ML or native library.

Treat CUDA as a hard architectural boundary from the start

If a stage requires GPU inference, isolate it behind a clean interface from day one. The scheduling logic, the orchestration, the result handling — all of that can run on ARM64. Only the GPU execution itself needs x86. Design for that split explicitly rather than letting the GPU requirement contaminate surrounding stages.

Version-pin with ARM64 wheel availability in mind

When locking dependency versions for reproducibility, verify that your pinned versions ship manylinux_aarch64 wheels. A version that was pinned for stability before the package added ARM64 support will block migration silently — you will not know until you try to build.

Build in CI on ARM64 from day one

Adding ARM64 build validation to your CI pipeline when you start a project costs almost nothing. Adding it after 18 months of x86-first development costs weeks of debugging. Many packages add and remove ARM64 wheel support across versions; having a CI gate that verifies ARM64 builds on every dependency update catches regressions immediately.

Use the 20% Lambda pricing delta as a design signal

AWS prices ARM64 Lambda at approximately 20% less than x86. That is not just an operational saving — it is a signal about where to route compute. If a function is trivially portable (pure Python, well-supported wheels), choosing x86 without evaluating ARM64 first is a deliberate decision to pay a 20% premium for no benefit.

The AWS-Specific Caveat

Everything in this post is calibrated to AWS. The cost numbers, the Lambda architecture flag, the Graviton family of processors, the manylinux_aarch64 wheel availability for AWS runtimes — these are AWS-specific.

Other cloud providers offer ARM compute. Azure has Ampere A1-based VMs. GCP has Tau T2A. But:

• Serverless ARM support (equivalent to Lambda arm64) is not uniformly available or mature across providers

• The degree of AWS-specific optimisation in the Python scientific and ML ecosystem — particularly for Graviton NEON instructions — is not replicated elsewhere

• Cost structures differ; the 20% pricing delta is an AWS Lambda figure and does not translate directly to other billing models

If you are building a multi-cloud or cloud-agnostic system, treat Graviton optimisation as an AWS-specific tuning layer rather than a foundational architectural decision. The portability of your ARM64-compatible stages will transfer. The AWS-specific performance characteristics will not.

The Real Cost Model

The standard framing of Graviton savings is “20% cheaper Lambda.” That is accurate but incomplete. The full savings model for an architecture like Parjanya’s has four components:

1. Lower per-invocation Lambda cost on ARM64 (~20% pricing delta, compounding with every invocation)

2. Avoided NAT data-path spend — VPC gateway endpoints route S3 and DynamoDB traffic without NAT tax; this is infrastructure rather than ARM64, but it compounds with the migration

3. Reduced ML over-processing — a rule-based pre-filter like parjanya-technical_validator rejects bad images before they reach the GPU queue; every rejected frame is a GPU invocation that never happens

4. Lower operational overhead — fragile native builds create deployment failures, cold start issues, and CI complexity; removing them reduces engineering time

For a high-volume pipeline, component 3 often dominates. The ARM64 savings from the validator are real. But the savings from not running expensive GPU inference on images that would have been rejected anyway are an order of magnitude larger.

This is the correct way to think about Graviton adoption: not as a compute cost reduction in isolation, but as an opportunity to reconsider stage design, dependency choices, and processing flow in ways that compound the savings.

Summary

The ARM wheel is the gatekeeper to Graviton savings. Not the hardware, not AWS support, not your application code. The compiled binary distribution of your Python dependencies — and whether it exists for aarch64 — is what determines whether a given Lambda function can run on Graviton.

The practical conclusions from Parjanya’s full migration arc:

• Audit your wheel matrix before you plan your migration. The output tells you which functions migrate, which need source builds, and which need architectural changes.

• Build in an ARM64 environment that matches your Lambda runtime. Local results on x86 do not predict ARM64 install success.

• Replacement is often faster than migration. A fragile ML stage replaced by a dependency-light rule-based implementation solves the ARM64 problem, the reliability problem, and often the correctness problem simultaneously.

• Isolate x86 hard boundaries explicitly. CUDA is the cleanest example — design around it rather than pretending it is temporary.

• Partial adoption is architecture, not failure. Running 3 of 4 Lambda functions on Graviton with an intentional x86 boundary at the GPU stage is a correct outcome.

Graviton adoption is not a hardware decision. The hardware is already ready. It is an ecosystem decision — and the ARM wheel matrix is where that decision lives.

Further Reading

• AWS Graviton Getting Started – Python — The canonical reference for why Python version and GLIBC version matter for wheel compatibility

• Migrating AWS Lambda functions to ARM-based Graviton2 — AWS’s official guide; read between the lines for what is assumed about dependency simplicity

• AWS Graviton Technical Guide — Deep reference on runtime support and ecosystem maturity by language

• Why We Could Only Use AWS Graviton for 1 of Our 4 ML Lambdas — First Parjanya post: the ecosystem reality check

• From 1/4 to 3/4: Re-architecting an ML Pipeline for Graviton — Second Parjanya post: the re-architecture and cost analysis

• AWS Graviton4 vs Intel x86: From $10 POC to Production Validation — The original POC that established the cost and performance baseline

Abstract

Resilient AI infrastructure is not defined only by model quality, throughput, or scaling efficiency. In production, the harder problem is reconciliation: keeping durable state, requested work, and actual execution aligned when workers fail, queues drift, storage paths change, policies evolve, and control planes become blind. Let me present TBIE, a practical operating model for distributed systems: Truth, Belief, Intent, and Execution. TBIE separates the authoritative record of what is true from the operational view of what seems to be happening, the replayable journal of work that should occur, and the stateless workers that realize side effects.

I have used Parjanya v2.0, an image ingestion and GPU-based visual scoring platform, as a detailed case study. Parjanya’s architecture spans immutable uploads, DynamoDB workflow state, SQS-based intent, event-driven replay, GPU autoscaling, browser-side uploads, policy regrading, and infrastructure drift. Across these layers, TBIE proved useful not as a slogan but as a diagnostic and design framework. It clarified why transient worker failure should not destroy replayable work, why pending state without corresponding intent is a reconciliation gap, why wrong-bucket routing can be deterministically wrong, why policy changes should be handled as controlled replay, and why control-plane blindness is more dangerous than isolated worker failure.

The central claim of this is that resilient AI platforms should be designed as reconciliation systems rather than one-way pipelines. A pipeline moves data forward. A reconciliation system continuously repairs the alignment between truth and work until intended outcomes are actually realized. That distinction changes how we design queues, retries, autoscaling, observability, IAM boundaries, tenant onboarding, model packaging, browser uploads, and recovery runbooks.

1. Introduction

Modern AI systems are often described in terms of model choice, hardware selection, vector stores, and inference efficiency. Those concerns are important, but they are not usually what breaks production systems. In practice, systems fail because they lose work, duplicate work, misroute work, or misinterpret what is actually happening. A queue may look healthy while the underlying work has already vanished. A worker may crash without leaving a replayable record. A browser upload may fail after the backend has already committed a pending state. An autoscaling group may be configured correctly while the reconciliation control plane that should activate it is blind.

These failures are not unusual edge cases. They are the normal shape of distributed systems under stress. The question, then, is not whether a system will experience drift between state and execution, but whether the architecture is built to detect, explain, and repair that drift. TBIE was developed as a response to that question.

TBIE stands for Truth, Belief, Intent, and Execution. It is a practical decomposition of distributed systems that helps separate durable facts from operational signals, requested work from realized work, and recoverable failures from terminal ones. The model is especially useful in asynchronous AI pipelines, where storage, metadata, queues, workers, and control planes all operate independently and can fail independently.

Parjanya v2.0 is an ideal case study because it combines all of those surfaces in a single system. Images are uploaded to S3. Metadata is written to DynamoDB. GPU work is performed asynchronously through SQS. Lambda functions handle technical validation and replay orchestration. GPU workers consume intent, perform visual-language inference, and write enriched truth back to the database. Autoscaling responds to queue depth. Browser uploads interact with tenant-specific buckets and CORS policies. Policy changes require historical reprocessing. Infrastructure drift can occur in launch templates, AMIs, model packages, and endpoint configuration.

Let’s deep dive how TBIE behaves in practice.

2. The TBIE Model

TBIE divides a distributed system into four layers that must remain conceptually distinct.

2.1 Truth

Truth is the durable record of what is actually true. It should survive worker restarts, deploys, transient outages, and runtime drift. In Parjanya, Truth lives in immutable S3 objects, DynamoDB workflow rows, hashes, status fields, curation metadata, and derived state that has been durably committed. Truth answers questions such as: What object exists? What stage of the workflow is this image in? What has already been computed? What is the authoritative record of the tenant, batch, or image?

2.2 Belief

Belief is the operational view of reality. It is derived from Truth and current signals, but it may lag, be partial, or be noisy. In Parjanya, Belief includes queue depth, in-flight message counts, worker health, GPU availability, desired ASG capacity, scaling activity, and alarm state. Belief is what operators see first, but it must not be confused with source-of-record truth.

2.3 Intent

Intent is the durable record of work that has been requested. It is the journal of “please do this work.” In Parjanya, SQS messages represent Intent. Replay jobs, regrade jobs, and retry workflows are all forms of Intent. The critical property of Intent is replayability. If Intent can be lost too early, then transient failures become permanent failures and manual rescue becomes necessary.

2.4 Execution

Execution is the stateless work that turns Intent into side effects and updated Truth. In Parjanya, Execution includes Graviton Lambdas for EXIF extraction and technical validation, GPU workers for VLM inferencing and image quality analysis, replay jobs that reconstruct missing Intent, and autoscaling actions that wake the worker fleet when work appears. Execution is where work happens, but in a resilient system it must never be allowed to erase the record of what was supposed to happen.

3. Why Reconciliation Matters

The key insight behind TBIE is that a distributed system is not healthy merely because its components are running. Health depends on reconciliation. A system can be superficially alive while still being functionally stuck. A database row may say pending_vlm_enrichment while no SQS message exists. A queue may be empty while worker work is still pending in truth. A control plane may be scheduled while unable to read the signals it needs to act.

Reconciliation is the process of bringing Truth, Intent, and Execution back into alignment. In the TBIE model, reconciliation is not a background nicety; it is the core operating principle. Systems should not rely on humans noticing a stale queue and manually fixing it. They should be able to repair the missing relationship between truth and work on their own.

This is why the distinction between a pipeline and a reconciliation system matters. A pipeline assumes a mostly linear progression from input to output. A reconciliation system assumes that progression can be interrupted, delayed, partially completed, or invalidated by failure. It therefore builds explicit mechanisms to recover the intended outcome rather than merely continue from the last known step.

4. Event-Driven Reconciliation in Parjanya v2.0

Parjanya initially used a scheduled polling model. A timer would wake the replay Lambda periodically, the Lambda would scan for pending work, and if appropriate it would emit SQS messages and trigger GPU autoscaling. This worked, but it introduced avoidable latency, wasted invocations, and operational ambiguity. The system could not react immediately when Truth changed, and operators had to reason about whether the queue was empty because the system had genuinely caught up or because the next poll had not yet occurred.

The move to DynamoDB Streams changed that architecture fundamentally. Now, when a workflow row reaches pending_vlm_enrichment, the stream emits an event immediately. An event source mapping filters for relevant writes, batches records briefly, and invokes the replay Lambda. The Lambda discovers missing or retriable work, emits Intent into SQS, and kicks the GPU ASG when necessary. This turns reconciliation from periodic discovery into event-driven reaction.

The result is a lower-latency, more honest architecture. Truth changes, the control plane notices, Intent is created, and Execution follows. The system no longer waits for a timer to rediscover what should already be known.

5. Parjanya v2.0 as a TBIE Case Study

Parjanya’s architecture maps neatly onto TBIE. S3 holds immutable uploads and preview artifacts. DynamoDB holds workflow state, curation metadata, and hashes. SQS holds replayable intent. Lambdas handle validation, replay, and orchestration. GPU workers perform inference and write enriched state back to DynamoDB. Autoscaling follows queue pressure and worker demand.

What makes the case study valuable is not the existence of those components, but the failure modes they exposed. Parjanya demonstrated that many production issues are really reconciliation issues in disguise. Some failures were caused by deleting intent too early. Others were caused by truth existing without intent. Some were caused by the wrong bucket being referenced correctly. Others were caused by the control plane being unable to read queue state. Still others came from stale image tags, missing model files, CORS misconfiguration, or API schema drift. TBIE made those failures legible.

5.1 Transient GPU failure should not destroy Intent

One of the earliest and most important incidents involved GPU workers deleting SQS messages before the retry window had been exhausted. A worker could encounter a temporary S3 download failure or an out-of-memory condition, write a failure state, and still remove the only replayable message. At that point, Truth said the work had failed, but Intent no longer existed and the system had no automatic path to try again.

TBIE made the problem obvious. Execution had destroyed Intent prematurely. The fix was to return explicit execution outcomes: retry, terminal_failure, or completed. A retry preserves the message and lets visibility timeout and receive count determine whether the work should be attempted again. A terminal failure writes durable failed truth and then deletes the message. Success writes final truth and deletes the message. That separation preserves replayability while still allowing durable failure when necessary.

5.2 Truth can be pending while Intent never existed

Another recurring failure mode was the mirror image of the first. Rows in DynamoDB would remain in a pending VLM state, but no corresponding queue message had been created. Queue depth fell to zero and the system appeared idle, yet the backlog still existed in Truth.

This is a reconciliation gap, not a worker bug. The response was to introduce replay mechanisms that scan for pending rows without Intent, regenerate queue messages, and kick the GPU fleet when work is emitted. In other words, the platform needed a control plane that could restore missing Intent from Truth automatically.

5.3 Replaying retryable failures requires resetting Truth deliberately

Not every failure should be treated as terminal. Some failures, such as S3 download errors or temporary GPU OOM events, are retryable if the underlying condition is corrected. But a simple re-enqueue is not enough when the workflow row itself still contains partial or stale derived fields.

TBIE clarifies the difference between retry and replay. A retry is another attempt at the same execution. A replay is a deliberate reset to a pre-execution boundary, removal of stale derived state, and emission of fresh Intent. This distinction matters because the second form is a controlled reconstruction of the workflow, not merely another attempt on top of corrupted state.

5.4 Wrong bucket, right key

One of the most instructive incidents involved an SQS payload that pointed to a valid preview key but the wrong bucket. The worker repeatedly failed with S3 download errors, even though the preview existed. The issue was not flaky execution. It was deterministic misrouting.

This is a classic distributed systems failure: the object key is correct, but the address is wrong. TBIE makes that failure visible because Intent is not just “some work exists.” Intent must encode the correct target. The fix was to force preview keys to use the uploads bucket as the source of truth and to tighten eligibility so only objects suitable for GPU work are enqueued.

5.5 Policy changes should be handled as replay, not patching

When the content policy changed to more strictly reject non-photographic content such as screenshots, banners, infographics, social cards, slides, illustrations, and mockups, the platform needed to reprocess historical images under the new rules. TBIE made the solution obvious: reset the affected Truth fields to a pre-VLM boundary, set the rows back to pending, and emit new Intent so GPU workers can re-grade under the revised policy.

This is one of the most powerful properties of replay-native systems. They allow policy evolution without manual data surgery. A change in judgment does not require a one-off correction script. It becomes a controlled reprocessing event with clear before-and-after semantics.

5.6 Execution drift can occur below the application layer

Parjanya also surfaced failures where the application code was correct but the execution environment was stale. One launch template continued to point to an old image tag. Cloud-init failed while pulling the nonexistent container image, and the GPU instance came up without ever starting the worker.

A separate incident involved an AMI missing the Python files required by a model that relied on remote code. The container started, but the model loader crashed because the local artifact was incomplete. These are both Execution-layer failures, but they occur at different depths: one at instance boot, the other at model runtime. TBIE is useful because it keeps both cases in the same conceptual category: the execution environment drifted away from the declared source of truth.

5.7 CORS and browser execution are part of the resilience story

The ingestion path introduced another class of split-state failure. The backend could write a pending upload record and generate a presigned URL, but the browser’s PUT could still fail because the bucket CORS policy or endpoint routing was incorrect. In that case, Truth at the API layer looked correct while Execution at the browser layer never completed.

Later investigations revealed that multiple independent layers could be wrong at once. A bucket might reject wildcard origins under restricted public bucket settings while the client simultaneously used a global S3 endpoint that returned a redirect browsers would not follow on OPTIONS preflight. TBIE helps here by forcing the diagnostic question: is Intent missing, or is Execution unable to realize it? In this case, Intent existed. Execution was blocked by infrastructure and configuration drift.

5.8 The control plane itself can fail silently

One of the most dangerous incidents in the system occurred when the replay Lambda lacked the permission it needed to read queue attributes. The function was still scheduled and invoked, but it could not inspect Belief and therefore crashed before it could emit Intent or kick the GPU ASG. From the outside, everything looked enabled. In reality, the control plane had gone blind.

This is a crucial TBIE lesson. The component that repairs work is itself a first-class reliability surface. If the control plane silently fails, then every other safety net becomes weaker. That is why control-plane observability must be stronger than ordinary worker observability.

5.9 Dynamic tenant discovery and queue retention are part of reconciliation

As the platform grew, static tenant lists became a bottleneck. Reconciliation could not depend on manually editing Terraform every time a new tenant was onboarded. The control plane therefore moved toward dynamic tenant discovery from DynamoDB, with tenant registry markers written by the upload path and discovered by the replay Lambda at runtime.

Queue retention was another important operational detail. If the GPU fleet is scaled to zero for cost savings, Intent must survive long enough for work to be processed after a weekend or holiday gap. Retention policy is therefore part of resilience design, not a generic queue setting.

6. Operational Model and Runbook Thinking

TBIE becomes truly useful only when it changes how incidents are diagnosed.

By the time Parjanya matured into an event-driven reconciliation system, the debugging approach itself had changed. Operators no longer began by asking whether “the queue is healthy” or whether “the GPU workers are running.” Those signals mattered, but they were no longer treated as authoritative.

Instead, every investigation began with a simpler and much more precise question:

Is the system waiting because Intent is missing, or is Intent present but Execution cannot realize it?

That distinction dramatically narrowed the search space during incidents.

If Truth showed a growing backlog while Intent remained near zero, the issue was usually in reconciliation: replay Lambda failure, queue emission failure, stale tenant discovery, or control-plane blindness.

If Intent existed but Execution failed repeatedly, the problem shifted toward realization: bucket routing, IAM permissions, runtime drift, stale launch templates, model incompatibility, or endpoint mismatch.

The architecture became easier to reason about because TBIE transformed debugging from improvisation into classification.

7. Guardrails and Maintenance as Part of the Architecture

Parjanya’s history shows that resilient systems are built by converting incidents into guardrails. Transient worker failures should not destroy intent. Replay logic should reset truth cleanly before re-emitting work. Dependency drift should be prevented with lockfiles and build assertions. CORS origin drift should be rejected at plan time. Tenant discovery should be dynamic rather than manually maintained. Launch templates should refresh when image tags change. Model artifacts and container versions should be treated as a compatibility pair. API contracts and frontend types should not be allowed to drift silently. Queue retention should match the actual outage window, not an idealized one.

This is the practical meaning of reliability engineering. Incidents become permanent controls. The architecture evolves by accumulating constraints, validations, and replay paths that encode what the system has already learned.

8. Implications for AI Platform Design

TBIE generalizes beyond Parjanya. Any AI platform that spans storage, metadata, asynchronous work, GPU execution, and policy evolution can benefit from the same model. The key is to stop thinking of the system as a one-way pipeline and instead treat it as a reconciliation loop that continuously restores alignment between durable truth and intended work.

That shift affects architecture choices. Queues become journals of intent rather than temporary transport. Replay becomes a first-class product capability rather than an ops workaround. Autoscaling becomes a consequence of work existing, not a heuristic guess. Model packaging becomes part of execution correctness. Browser upload behavior becomes part of the end-to-end reliability model. And policy change becomes a replay event rather than a manual repair task.

When the System Hangs at 02:00 AM

At 02:00 AM, the queue may look empty while customers are still waiting for images to process. GPU instances may exist while no workers are actually running. A replay Lambda may still appear scheduled while silently crashing on every invocation. Dashboards may continue showing green infrastructure while the control plane itself has already gone blind.

This is where TBIE stopped being an architectural abstraction and became an operational model.

The first step was no longer restarting workers or increasing autoscaling limits. The first step became identifying which reconciliation boundary had failed.

The investigation usually began with Truth.

Was DynamoDB accumulating rows in pending_vlm_enrichment? If so, the platform still believed work should exist, regardless of what the queue appeared to show. Truth backlog became the first signal because Truth is the authoritative record of unfinished work.

The next step was Intent.

Did SQS actually contain replayable work items? A high Truth backlog combined with near-zero queue depth usually indicated a reconciliation failure rather than an execution failure. In those cases, the replay Lambda itself became suspect. Either the control plane had stopped emitting Intent, or it had lost the ability to discover the work that needed to exist.

That distinction mattered enormously because the operational response changed completely depending on the answer.

If Intent existed but the backlog still did not move, attention shifted toward Execution health. GPU worker logs became critical. Sometimes the workers were healthy but repeatedly failing with deterministic download errors caused by bucket routing drift. Sometimes GPU instances launched successfully while containers never started because a stale launch template pointed to a nonexistent image tag. In other cases, the instance booted correctly but the model loader crashed because required runtime artifacts were missing from the AMI.

One of the most revealing operational signals turned out to be complete silence.

An empty GPU log group was rarely a good sign. More often, it meant execution had failed before the application layer even started. Cloud-init failures, image pull failures, IAM denial during bootstrap, or launch-template drift frequently surfaced first as absence rather than explicit failure.

The replay Lambda introduced another category of silence.

If queue visibility remained unchanged while pending Truth continued growing, the control plane itself became the focus. One of the most dangerous incidents in the system occurred when the replay Lambda lost permission to read queue attributes. The function was still scheduled. It was still being invoked. But it crashed before emitting Intent or triggering autoscaling. From the outside, everything looked operational while reconciliation had effectively stopped.

That incident permanently changed the operational philosophy of the platform.

Worker failures became treated as recoverable noise.

Control-plane blindness became treated as a systemic risk.

Eventually, the debugging process itself evolved into a TBIE classification exercise:

• Truth backlog revealed whether unfinished work existed.

• Intent backlog revealed whether reconciliation was functioning.

• Queue visibility revealed whether work was flowing.

• GPU log silence revealed bootstrap or execution drift.

• Replay Lambda silence revealed control-plane failure.

• IAM checks validated whether orchestration still had authority to act.

• Launch-template validation confirmed whether execution environments still matched declared infrastructure state.

The system stopped being debugged as “a pipeline.”

It began being debugged as a continuously reconciling distributed system.

That distinction changed how incidents were understood, how recovery was automated, and ultimately how the platform itself evolved.

9. Conclusion

Parjanya v2.0 demonstrates that resilient AI infrastructure depends on more than good models or fast workers. It depends on a clear model for distinguishing what is true, what is believed, what work is intended, and what execution can actually accomplish. TBIE provides that model.

The value of TBIE is not just conceptual. It changes how systems are built, how retries are designed, how control planes are monitored, how infrastructure drift is prevented, and how policy changes are reapplied. Most importantly, it turns production failure from a confusing mystery into a structured reconciliation problem.

That is why the right mental model for modern AI infrastructure is not merely “build a pipeline.” It is “build a system that can continuously reconcile truth and intent until intended work is actually realized.”

Appendix A. Maintenance Checklist

That checklist is not a substitute for architecture. It is how architecture remains trustworthy over time.

1. The Observation That Started This

A few weeks ago I noticed something in my Claude Code session trace: Haiku 4.5 was being invoked as a subagent while my main session was running on Opus 4.7.

My first instinct was: is this personalisation? Has Claude learned my patterns? Is this a new adaptive router?

It was none of those things. But chasing the answer led me into a deep audit of my actual usage data — and what I found changed how I think about AI session economics entirely.

The short answer: Claude Code has a built-in subagent architecture where the main orchestrator (Opus 4.7 in my case) delegates read-only codebase work to the Explore subagent, which runs on Haiku 4.5 by default. It is deterministic product behaviour, not personalisation. The model matching is happening at the subagent dispatch boundary, not through learned routing.

But the longer answer — what I found when I actually pulled my usage data — is the more important story.

2. What the Data Actually Showed

Running /cost in my Claude Code terminal revealed the real picture, however it is only 3% of usage, 33 sessions, complete details are found using ccusage, still showing the important metrics:

Favorite model:  Opus 4.7        Total tokens: 22.0M
Sessions: 33                     Longest session: 7d 2h 33m

● Opus 4.7  (53.3%)   In: 118.2k  · Out: 11.6M
● Opus 4.6  (18.2%)   In: 118.3k  · Out: 3.9M
● Haiku 4.5 (20.6%)   In: 363.5k  · Out: 4.2M
● Sonnet 4.6  (7.9%)  In:  63.0k  · Out: 1.7M

Combined Opus usage: 71.5% of all tokens.

Sonnet — the model I publicly advocated as the benchmark — was only 7.9% of my actual usage.

Running ccusage session revealed the concentration problem:

Two sessions accounted for 50.9% of all costs.

The small sessions with $0.16–0.22/M cost-per-token are my manual Haiku sessions — git commits, README updates, simple lookups. Haiku routing was working correctly there.

The two expensive sessions are a different story entirely. And the cost-per-million ratio is the key signal: at $0.53–0.63/M blended, those sessions were Opus-orchestrated with some subagent delegation, but the token volume itself was the problem — not the model choice.

3. Naming the Problem: Context Debt

Before getting to solutions, I want to name what I observed, because it needs a name.

Context Debt is the accumulation of tokens in a session that compound the cost of every future turn without adding proportional value to the reasoning quality of those turns.

It behaves like technical debt in one critical way: it is invisible when it is forming, and expensive when you finally notice it. Unlike technical debt, it has an immediate dollar cost that compounds within the same session.

Here is the mechanics:

Turn 1:   Opus reads inference_config.yaml    → context: 3k tokens
Turn 10:  Opus reads vlm_pipeline.py          → context: 40k tokens
Turn 30:  Opus reads cuda_kernels/, manifests  → context: 180k tokens
Turn 50+: Every API call sends 180k input     → paying full price
          even if your prompt is 50 words

At Opus pricing (~$15/M input tokens), 200 subsequent turns × 180k context = 36M input tokens = $540 in input alone, before a single output token is generated.

This is what those two sessions were doing. It was not that Opus was the wrong model. It was that the session carried everything it had ever read into every subsequent turn — and paid Opus prices for all of it, repeatedly.

Three forces compound context debt:

Prompt cache expiry. Claude caches the static prefix of your context for 5 minutes (up to 1 hour for some configurations). In a session spanning hours or days, that cache expired repeatedly. Each expiry means paying full Opus input price for the entire accumulated context again. My 7-day session paid full price for a 400k+ context many times over.

Compaction tax. When context compacts (the default fires at ~95% capacity), Claude writes a summary and prepends it to all future turns. In a very long session with multiple compactions, those summaries stack — each subsequent turn carries all of them as mandatory context.

Speculative file reads. Opus tends to read files “to understand the broader context” before answering, even when the current task doesn’t require them. In an ML infrastructure repo, this means model configs, environment files, pipeline definitions, deployment scripts, and CUDA configurations all end up in context — and stay there.

4. The Constraint I Cannot Engineer Away

Before presenting the framework, an honest acknowledgment of constraints.

My primary workload is VLM (Vision-Language Model) inferencing, infra-heavy automation, and ML optimisation pipelines. For this work:

• Minimum viable context is ~300k tokens. Multiple model configs, inference scripts, pipeline definitions, and infrastructure manifests need to be simultaneously present for coherent cross-file reasoning.

• Sonnet 4.6 has only recently gained a 1M context window (now GA, worth testing for execution phases). But adaptive thinking — structurally required for deep VLM reasoning — was absent from Sonnet until recently and is still maturing.

• For the reasoning-heavy phases of this work, Opus 4.7 at 1M context is the correct model. This is a technical requirement, not a preference.

The insight that changes everything: the model being correct does not mean the context footprint needs to be what it currently is.

Opus 4.7 is the right reasoning engine. Sending it 400k tokens of raw source code when it only needs 40k tokens of structured summaries is waste. That is the distinction the framework targets.

5. Context Engineering: The Framework

Context Engineering is the practice of deliberately managing what enters an AI session’s context window, when it enters, and in what form — to maximise reasoning quality per token rather than total tokens.

It has four mechanisms.

Mechanism 1: Pre-Summarisation (Haiku as Reader)

The default pattern:

Opus opens session
Opus reads 30 files to orient itself
All 30 files sit in context for the rest of the session
Cost: 150k input tokens × every subsequent turn

The engineered pattern:

Before session opens:
  @ml-context-loader reads all 30 files (Haiku, separate context)
  Returns: 12k token structured dependency map

Opus main session receives:
  The 12k map, not the 150k raw files
  Opus never directly holds the source files
Cost: 12k input tokens × every subsequent turn

The work output is identical. The context footprint is ~12x smaller.

Subagent configuration (~/.claude/agents/ml-context-loader.md):

---
name: ml-context-loader
description: Pre-read ML pipeline files, model configs, inference scripts,
  and VLM configurations before complex optimisation work. Returns a
  structured dependency map for the main session. Invoke this BEFORE
  starting any ML or infra work in the main session.
model: haiku
tools: Read, Grep, Glob
---
You read ML infrastructure files and produce a structured summary covering:

1. Model architecture and config paths
2. Inference pipeline entry points and key functions
3. CUDA/GPU configuration and memory constraints
4. Cross-file dependencies relevant to the stated task
5. Files that need to change to accomplish the goal

Return a structured map. Do not return raw file contents.
Maximum output: 15k tokens. Be precise and omit irrelevant files.

Paired subagent for infrastructure (~/.claude/agents/infra-scanner.md):

---
name: infra-scanner
description: Scan infrastructure configs, Terraform files, CI/CD pipelines,
  deployment manifests, and environment configs before optimisation work.
  Returns dependency map and change surface for the stated task.
model: haiku
tools: Read, Grep, Glob, Bash
---
Read infra files and return:
- Resource dependencies and environment configs
- Critical paths that affect the stated task
- Files that need to change
- Current state of relevant resources

Compact output only. Omit files irrelevant to the task.

Mechanism 2: Targeted Loading via CLAUDE.md

Even with pre-summarisation, Opus will add raw files to context during execution if not explicitly instructed otherwise. CLAUDE.md is the right place to install session-level context discipline.

Project-level .claude/CLAUDE.md additions:

## Context Discipline

Before reading any file, state why it is needed for the current task.
Never read files speculatively to "understand the broader context."

Use @ml-context-loader before starting any ML or infra work.
Use @infra-scanner before starting any infrastructure work.
Use @explorer for any file discovery or codebase search.

If a file has been summarised by a subagent this session, 
use that summary rather than re-reading the raw file.

Maximum 5 files in direct context at any point during implementation.
For architecture decisions, request a structured summary first.

When asked to fix a bug, read only the file containing the bug
and its direct imports. Do not read the broader codebase unless
the fix requires understanding something not in those files.

This changes Opus’s default behaviour from breadth-first file reading to targeted loading. In practice, this alone reduces speculative reads by 50–60% in long sessions.

Mechanism 3: Session Chunking by Reasoning Type

The most expensive sessions — the two that consumed 50.9% of total costs — mixed two fundamentally different cognitive modes in a single long-running context:

• Reasoning mode: Architecture decisions, bottleneck analysis, choosing optimisation strategies. Needs Opus depth. Short turns, modest context.

• Execution mode: Implementing the decided approach across many files. Needs large context. Lighter per-turn reasoning.

When mixed, the session context grows to serve both needs simultaneously — and every turn pays the full cost of the combined footprint.

Separated:

# Session 1: Architecture planning
# Opus, focused context (summary only), ~15–20 turns
claude --model opus
# Prompt: "Given this summary [paste ml-context-loader output],
#   design the memory optimisation strategy for the VLM dataloader.
#   Output a structured implementation plan."
# → Session ends with a plan document. Context: ~30k tokens throughout.

# Session 2: Implementation
# Sonnet 4.6 at 1M (or Opus if VLM-specific reasoning needed)
# Longer, but following a decided plan — execution, not discovery
claude --model sonnet
# Prompt: "Implement this plan [paste plan].
#   Start with [specific file]. Read only the files listed in the plan."
# → Context grows with implementation files, but no re-discovery overhead.

The Opus session stays short because it worked from a summary, not raw files. The implementation session can run on Sonnet 4.6 at 1M context if the per-turn reasoning does not require Opus depth — which it often does not, once the architecture decision is made.

The practical test for session split: If your current turn is deciding something, you are in reasoning mode. If your current turn is doing something already decided, you are in execution mode. Different sessions.

Mechanism 4: Cache Warming on Re-entry

The single most expensive moment in a long session is returning after a break.

Prompt cache TTL is 5 minutes by default for most Claude Code request types (up to 1 hour in some configurations). When you step away for 30 minutes and return to a 300k-token context, the first turn back pays full Opus input price for all 300k tokens. If your session had 20 such re-entries across a workday, you paid for that 300k context twenty times.

Before taking any break longer than a few minutes:

/compact

This forces a clean compaction summary before the cache goes cold. The next turn back pays for a 5–10k summary re-warm, not a 300k context re-warm.

Prevent the problem structurally by setting aggressive autocompaction:

In .claude/settings.json:

{
  "env": {
    "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "60"
  }
}

Compacting at 60% context capacity means the session never accumulates a context large enough to be expensive to re-warm. The compaction summary costs a few thousand tokens once. The alternative is paying full input price for a massive context every time the cache expires.

6. The Escalation Model

With context engineering in place, the tier routing becomes:

The key principle: Tier selection is determined by reasoning depth required per turn, not by task complexity overall. A complex ML task still has Haiku turns (reading files), Sonnet turns (writing implementations), and Opus turns (designing the approach). The same task should not require every turn to be Opus.

Environment defaults that enforce this without manual switching:

# In ~/.zshrc or ~/.bashrc

# Subagents default to Sonnet, not parent model
export CLAUDE_CODE_SUBAGENT_MODEL="claude-sonnet-4-6"

# Haiku slot pinned explicitly
export ANTHROPIC_DEFAULT_HAIKU_MODEL="claude-haiku-4-5-20251001"

7. Complete Subagent Stack

For teams adopting this framework, the full subagent configuration:

~/.claude/agents/explorer.md — Universal read-only search

---
name: explorer
description: Search files, grep for symbols, read code structure, find
  usages, locate definitions. Use for any read-only investigation
  before editing. Never use for file modification.
model: haiku
tools: Read, Grep, Glob, Bash
---
You are a fast read-only code explorer. Search thoroughly and return
a concise structured summary. Never modify files. Optimise for finding
the minimum set of files relevant to the stated question.

~/.claude/agents/git-helper.md — Repository operations

---
name: git-helper
description: Handle git commits, branch management, README updates,
  changelog entries, and documentation formatting. Use for all
  repository hygiene tasks.
model: haiku
tools: Bash, Read, Write
---
You handle git operations and documentation updates efficiently.
Use conventional commits format. Commit messages: type(scope): description.

~/.claude/agents/code-writer.md — Standard implementation

---
name: code-writer
description: Implement features, write new functions, refactor existing
  code, add tests. Use for code generation and editing tasks that
  do not require architectural reasoning.
model: sonnet
tools: Read, Write, Edit, Bash, Glob, Grep
---
You are a focused implementation engineer. Read the minimum required
files before editing. Make one logical change per invocation.
Follow the plan provided; do not redesign unless asked.

~/.claude/agents/ml-context-loader.md — ML pre-summarisation (See Mechanism 1 above)

~/.claude/agents/infra-scanner.md — Infrastructure pre-summarisation (See Mechanism 1 above)

~/.claude/agents/architect.md — High-stakes reasoning only

---
name: architect
description: Design system architecture, resolve complex multi-file bugs,
  make technology decisions, plan major refactors requiring cross-system
  understanding. Only invoke when deep reasoning is genuinely required.
  Receives summaries, not raw files.
model: opus
tools: Read, Grep, Glob
---
You make high-level decisions from structured summaries.
Think deeply before proposing changes.
Output a structured plan that code-writer can execute step by step.
Request specific raw files only when the summary is insufficient.

8. Before and After: The Numbers

Based on actual session data from this audit:

The two expensive sessions ($625.72 combined API equivalent):

Output tokens are unchanged because the actual work product — the code written, the analysis produced — is identical. The saving is entirely in the context overhead that was being sent to the Opus API on every turn without contributing to reasoning quality.

Across all sessions:

9. What Is Still Missing (Community Ask)

This framework works within current Claude Code capabilities, but several gaps remain that limit how far it can go without manual configuration:

GitHub #44976 — Auto model routing by task type. Currently, there is no native content-aware router in the main session. The routing that exists (Explore → Haiku) operates at the subagent dispatch boundary only. Automatic routing based on turn type — planning turns to Opus, execution turns to Sonnet — requires manual setup or explicit model switching. This feature request, if shipped, would make context engineering automatic rather than configured.

GitHub #43869 — Subagent model routing bugs. On some Max plan configurations, subagents silently run on the parent session’s model despite explicit per-agent configuration. If your Haiku-configured subagents appear to cost Opus rates, this ticket is the reference. Check env | grep CLAUDE_CODE_SUBAGENT_MODEL and verify Haiku is actually firing.

JSONL token counts are unreliable. The ccusage tool reads from JSONL files where input tokens are undercounted by 100–174x due to a streaming placeholder issue. For accurate token accounting, use /cost inside Claude Code (reads from the statusbar, which is accurate) and treat ccusage cost estimates as relative comparisons only.

Sonnet 4.6 at 1M + adaptive thinking. Both are now GA but recently so. For workloads requiring 300k+ context but not frontier VLM reasoning, Sonnet 4.6 at 1M is worth testing as a drop-in replacement for Opus in execution sessions. One session test against a known task will answer whether the quality holds for your specific workload.

10. Best Practices Summary

Before starting any ML or infra session:

• Run @ml-context-loader or @infra-scanner first

• Start with the plan document from a prior architecture session, not a blank slate

• Launch on Sonnet unless you know the first turn needs Opus-depth reasoning

During a session:

• /compact before any break longer than 5 minutes

• Never read files speculatively — enforce this in CLAUDE.md

• Split sessions when you shift from designing to implementing

For VLM and large-context ML work specifically:

• Opus 4.7 at 1M is still the correct model for reasoning-heavy phases

• The saving is in context footprint, not model choice

• Pre-summarise with Haiku; Opus only sees the structured output

On model selection:

• Default: Sonnet 4.6 (set CLAUDE_CODE_SUBAGENT_MODEL="claude-sonnet-4-6")

• Start sessions with claude --model sonnet unless the first task earns Opus

• Use opusplan for sessions that need Opus for planning and Sonnet for execution

On measuring progress:

• Use /cost for accurate token and model-split data

• Use ccusage session --breakdown to track model mix per session over time

• Watch cost-per-million-tokens ratio: $0.20/M = healthy Haiku/cache mix, $1.00+/M = context debt accumulating

Closing Thought

The observation that started this — Haiku 4.5 being spawned during an Opus 4.7 session — turned out to be Anthropic’s own answer to the problem I had been creating. The built-in subagent architecture already encodes the right instinct: read cheap, reason expensive, never conflate the two.

Context Debt is what happens when that separation breaks down at the session level. Context Engineering is the practice of maintaining it deliberately.

The framework is not about using cheaper models. It is about using the right model for each cognitive operation — and never making the expensive model carry the weight of the cheap one’s job.

Related posts:

• The Hidden Runtime Failure Behind Claude’s Recent Regression

• When Adaptive Thinking Goes Off the Rails

• Sonnet Becomes My New Benchmark

• Why you should chose Haiku as Default model and escalate if needed

• Building with Opus 4.6

Additional references:

• GitHub #27665 — Intelligent model routing / 93.8% Opus usage issue

• GitHub #43326 — Auto-select model and effort level based on task complexity

• GitHub #44077 — Skill model field not propagated to Agent sub-invocations

• GitHub #45228 — SendMessage ignores model specified at Agent creation

• GitHub #50574 — Auto model switch

I did not set out to replace a scoring engine with a rule engine. I set out to make the system cheaper, easier to explain, and less fragile as I was not keen on moving to g5.xlarge or g6.xlarge due to prompt sizes increased and would be lot of churn in terms of architectural and infra changes, comes with re testing every functionality and the increase in the costs both the dev costs as well as instances cost even more if non-availability of spot and on-demand, currently using 75/25 for spot and OD for g4.mdn.xlarge (and 2xlarge).

What I eventually learned was that the original design had an architectural flaw: it asked a single opaque model score to do too many jobs at once. It had to decide whether an image was technically valid, whether it was worth sending to the expensive VLM path, whether it should be accepted or rejected, and how to justify that decision later. That worked until it did not. So I split the pipeline into two deterministic layers: a Tier-0 technical validator before the model, and a rule engine after the model. That change dropped cost, improved explainability, and made policy iteration much faster.

The system I started with

Before Parjanya v2.0 architecture v5.3, the pipeline was basically this:

The problem was not that the model was “bad.” The problem was that I had turned a rich decision into a number. Two images with the same score could be fundamentally different: one might be a sharp wildlife shot with a minor distortion, while another might be a soft-focus landscape with no clear subject. The number hid the nuance. Prompt changes also shifted the score distribution, which meant calibration drift could silently change the policy. And when someone asked why an image was rejected, a floating-point score was not a useful answer.

What I changed

I made two structural changes.

How the new decision layer works

The rule engine is deliberately boring in the best possible way. It is not trying to be clever. It is trying to be consistent.

Here is the decision logic in plain language:

Every verdict also gets a version tag and a reason string, so I can audit what happened later. That is a huge shift from the old system, where the final answer was essentially “the score was 0.41.”

Why this mattered operationally

The first benefit was cost.

Here is the cost comparison I care about most:

On my typical tenant workload, GPU spend still dominates the bill, so every image I keep out of the GPU queue is real leverage. I also found one non-obvious cost: at single-tenant scale, about half of the per-drain cost was NAT Gateway egress, not GPU. In the measured v25 drain, NAT cost was actually larger than GPU cost. That means the rule engine improved the decision layer, but infrastructure tuning still matters separately.

The second benefit was iteration speed.

I moved the rule engine through several versions in days, not weeks. The workflow became: edit the rule file, test it on canaries, inspect the reasons, then run the full re-curation if it looked good. No GPU redeploy. No prompt-engine rebuild. No waiting for model infrastructure to churn. In practice, that made policy iteration feel like software engineering again instead of model ops. The v25 drain took 7 hours 41 minutes end to end, but the rule-engine pass over 3,427 rows took less than a minute.

The third benefit was explainability.

I finally had a decision layer that could say things like:

• “reject — subject-critical motion blur”

• “reject — 3 slight defects stack”

• “review — weak subject dominance; technically clean”

That may sound small, but it changes everything when a user asks why something was rejected. A reason string is much more useful than a numeric score.

The fourth benefit was robustness to prompt drift.

When the IQA prompt changed from v16 to v17, the score distribution shifted. Under the old design, that meant threshold recalibration and potential policy drift. Under the new design, the rule engine sat on top of structured fields whose meaning stayed stable across prompt versions. The model could change in ways that improved extraction without forcing me to rethink the whole decision layer.

The measured validation run

The clearest proof came from the v25 rescore.

I had 2,426 images drained through the pipeline, and then 3,427 rows reprocessed by the rule engine. This was not a synthetic benchmark. It was the first large enough live run to give me defensible numbers.

Pipeline results

What stood out to me was not just that the run completed, but that it completed cleanly. There was no message loss, the DLQ stayed at zero, and the autoscaler shut down correctly after the queue emptied.

Rule-engine results

The most interesting transitions were:

That last part mattered to me because it showed the rule engine was not trying to force certainty where none existed. It preserved uncertainty instead of masking it.

Why this architecture works

This pattern is not new in spirit. It is a cascade: a cheap stage rejects obvious failures, and the expensive stage only handles what survives. That reduces inference cost and keeps the scarce GPU path for cases that actually need it. The broader literature supports the same idea, especially for VLM and moderation workloads.

For me, the more important pattern was extract-then-decide. The VLM is the extractor. The rule engine is the decider. Once I separated those roles, the system became easier to reason about. The model could be probabilistic and flexible. The policy could remain deterministic and auditable.

What I watch for now

I did not trade away all the hard problems. I just moved them into better places.

That last item is still open: infrastructure cost can easily dominate the apparent model cost, so I do not want to fool myself into thinking the architecture alone solved everything.

What is next

The next steps are pretty clear to me. Dedup is still shipped but not yet used as a hard gate in production. The rule engine will likely get a v5 cleanup once the remaining fallback paths are collapsed. And I want a dashboard that breaks out cost by Tier-0 reject reason, because it would be useful to know exactly how much pure margin I am saving when corrupt files, blank frames, and blown exposures are filtered early.

The takeaway

I did not move from “AI” to “rules.” I moved from one opaque score to a more honest pipeline.

Now the model extracts structure, the rule engine makes the decision, and the cheap gate keeps obvious failures away from the GPU. That gave me lower cost, faster iteration, and a system I can actually explain to another human without hand-waving. For a production image pipeline, that is the architecture I trust more.

One thing I appreciated only after moving to the rule engine was how much the user experience improved once the system stopped behaving like a black-box scorer.

Previously, the output was essentially a number hidden behind a threshold. After the re-curation architecture, every image started carrying an explainable pipeline trail: technical validation status, VLM enrichment state, structured distortion analysis, composition signals, subject analysis, and finally a deterministic curation verdict with explicit reasoning. The gallery stopped feeling like “an AI guessed a score” and started feeling like a traceable review system.

In the image below, for example, the pipeline clearly shows the two-stage flow: Tier-0 technical validation first, followed by VLM enrichment and structured analysis. The system identifies slight sharpness and focus issues, explains where they occur (“subject lacks critical sharpness on head and eye area”), records composition attributes like framing and depth layers, and still accepts the image because the final rule evaluation determined the defects were minor enough relative to subject structure and overall composition quality.

That distinction matters. Under the old scoring engine, this image could easily have landed near a threshold boundary and become another opaque 0.62 vs 0.58 decision. Under the new architecture, the reasoning is visible, deterministic, and debuggable.

The biggest realization for me was this: the rule engine did not make the system less intelligent. It made the intelligence easier to operationalize.

The VLM still does the hard perceptual work — understanding composition, defects, subject prominence, and scene semantics. But the final decision layer is no longer probabilistic glue hidden behind one floating-point number. It is now an auditable policy system built on top of structured perception.

That separation ended up becoming the real architectural shift.

A few months ago, I wrote about why I believed in a Haiku-first strategy: start with the cheapest and fastest model possible, then escalate only when the task genuinely becomes harder. That idea still makes sense in principle.

But over the last few weeks, while working deeply on my VLM control plane / IQA platform, I realized something important:

The benchmark itself had changed. Probably, the models are sitting across 4.5, 4.6 and 4.7 versions and found it hard to compare (Haiku 4.5 vs Sonnet 4.6 vs Opus 4.7 1M), I meant, gap is widened!

For my current workload, Sonnet 4.6 has become the model I now measure everything against.

Not because it is the smartest model Anthropic offers. Not because it is the cheapest. But because it currently sits at the most practical intersection of:

• reasoning quality

• long-context reliability

• infra-heavy debugging

• ML-heavy orchestration

• coding consistency

• operational cost

• and sustained throughput

That combination matters a lot when you are building systems where the model is not simply “answering questions,” but continuously operating across orchestration, debugging, inferencing workflows, infrastructure decisions, and large-context reasoning.

The original Haiku-first thesis still holds

I still believe the core idea behind the earlier post was correct:

Default to the cheapest model that can reliably complete the task.

Anthropic’s pricing still makes that logic compelling.

At scale, these economics matter enormously.

If you are running:

• agentic workflows

• multi-step debugging

• recursive orchestration

• long-context analysis

• or VLM-heavy pipelines

…output-token costs become very real very quickly.

That is exactly why Haiku originally made so much sense.

But my workload changed

The problem is not that Haiku became “bad.” (also the gap across the models become evident due to version differences, Haiku is still at 4.5 with 200k context, whereas Sonnet and Opus at 4.6 and 4.7 with 1M Context)

The problem is that my workload evolved into something much more infra-heavy and ML-heavy.

The VLM control plane is no longer just lightweight orchestration or simple extraction work. The workflows now involve:

• larger context retention

• deeper architectural reasoning

• distributed infra debugging

• inferencing orchestration

• stateful workflows

• code + infra + ML reasoning in the same session

• long-running debugging loops

• and increasingly large operational context

This is where Sonnet 4.6 started to outperform Haiku dramatically in practical usefulness.

Why Sonnet 4.6 became my benchmark

Anthropic positions Sonnet 4.6 as the “best combination of intelligence, speed, and cost,” and honestly, that aligns closely with my own experience.

Sonnet 4.6 currently provides:

• 1M-token context window

• adaptive thinking

• fast latency

• significantly improved coding quality

• stronger instruction following

• much better long-context coherence

Anthropic also mentioned that in Claude Code testing:

• users preferred Sonnet 4.6 over Sonnet 4.5 roughly 70% of the time

• and even preferred it over Opus 4.5 in many workflows

That honestly tracks with what I have seen.

Sonnet 4.6 feels much more stable in:

• sustained coding sessions

• infra debugging

• architectural iteration

• and orchestration-heavy workflows

without becoming economically painful the way Opus can become.

Haiku 4.5: still useful, but no longer my default

This is probably the most important nuance in the entire post.

Haiku 4.5 is still:

• extremely fast

• extremely cheap

• very capable for its price tier

• excellent for parallelized execution

• strong for extraction/tagging/routing

• surprisingly good on coding benchmarks

Anthropic reports:

• 73.3% on SWE-bench Verified

• 40.21% / 41.75% on Terminal-Bench

Those are genuinely strong results.

But benchmark scores alone do not determine operational usefulness.

In my own experience, Haiku 4.5 now feels degraded for my specific workload.

Not degraded universally.

Degraded for this class of infra-heavy, ML-heavy systems work.

What I started noticing:

• unnecessary follow-up questions

• command thrashing locally

• context drift during debugging

• inefficient iteration loops

• loss of architectural continuity

• excessive operational churn

Sometimes it felt like the model was spending more effort “doing activity” than actually advancing the task.

That was the turning point for me.

Sonnet became the operational center of gravity

So the strategy evolved.

I no longer think in terms of:

“Haiku everywhere unless proven otherwise.”

Now the stack looks more like this:

This feels significantly more stable operationally.

Where Opus still matters

I still escalate aggressively to Opus for:

• architectural redesigns

• distributed systems debugging

• ML-heavy reasoning

• difficult infra changes

• large-context synthesis

• high-stakes production decisions

This is where the Opus models still separate themselves.

Anthropic’s published numbers are notable:

• Opus 4.6: 81.42% SWE-bench Verified

• 90.2% BigLaw Bench

• strong long-context retrieval

• better reasoning across hundreds of thousands of tokens

Opus 4.7 improves further:

• 13% improvement on a 93-task coding benchmark

• solved 4 tasks neither Sonnet 4.6 nor Opus 4.6 could solve

• 3× more production-task resolutions on Rakuten-SWE-Bench

• 21% fewer errors on OfficeQA Pro

You can actually feel some of these improvements during long debugging sessions.

Especially:

• reduced context drift

• better continuity

• fewer reasoning collapses

• stronger architectural consistency

Adaptive thinking changes the workflow significantly

One under-discussed improvement is adaptive thinking.

Sonnet 4.6 supports it.
Opus 4.6 supports it.
Opus 4.7 uses it exclusively.

This matters because:

• the model dynamically allocates reasoning effort

• easier tasks complete faster

• harder tasks receive deeper reasoning automatically

For infra-heavy workflows, this is a huge operational improvement.

Especially when the same session moves across:

• infra

• application logic

• orchestration

• ML behavior

• and debugging

without explicit model switching every few minutes.

Something interesting I noticed about Haiku switching automatically

One thing I still need to investigate further:

I occasionally noticed Claude seemingly switching into Haiku behavior automatically during sessions.

I initially thought this happened only at the beginning of sessions, but after more usage I suspect it can happen mid-session as well.

Anthropic’s Claude Code docs actually support this possibility:

• model switching can happen dynamically

• subagents can use Haiku automatically

• planning/execution layers may use different models internally

That would explain some of the behavior I observed:

• sudden increases in unnecessary questioning

• lightweight command loops

• simplified reasoning paths

• or execution-style interactions appearing inside longer workflows

I do not think this is inherently bad.
It actually makes sense architecturally.

But on infra-heavy workflows, it can sometimes become noticeable.

The timing of this shift matters for me personally

This transition is happening at an important moment.

I am currently preparing:

• dev + prod environment separation

• launch readiness

• infrastructure stabilization

• ML inferencing orchestration

• and final architectural refinement

ahead of launch in roughly two weeks.

That timing matters because the recent Anthropic infrastructure and limit improvements are actually having a practical operational effect.

Anthropic recently:

• removed dedicated 1M-context rate limits for supported models

• expanded higher-usage limits

• improved capacity availability

• and publicly discussed large-scale compute partnerships

I do not want to get into commercial details.

But practically speaking:

• peak-hour friction has reduced significantly

• longer debugging sessions feel more reliable

• large-context sessions are more usable

• and the additional headroom gives much more leverage for coding and debugging

That matters a lot when your workflows involve:

• long-context infra debugging

• VLM inferencing orchestration

• architecture synthesis

• and recursive engineering sessions

Bottom line

My original Haiku-first idea still solid and would try working on enhancing further in future versions. (Plan is to leverage, multiple models across the frontier models, both models like Claude, codex and free versions like Gemma4, Qwen3.6 etc.)

But the center of gravity moved.

Today:

• Haiku 4.5 is my narrow execution layer

• Sonnet 4.6 is my operational benchmark

• Opus 4.6 is my deep reasoning escalation

• and Opus 4.7 is the model I reserve for the hardest architectural and ML-heavy problems

The real lesson is not:

“Always use the smartest model.”

And it is also not:

“Always optimize for cost.”

The real lesson is:

The best model is the one that remains operationally useful under your actual workload.

For my current stack, Sonnet 4.6 is now that benchmark.

References & Further Reading

Anthropic Documentation & Official Notes

1. Anthropic Claude Models Overview
   Official model capabilities, context windows, adaptive thinking, latency tiers, and supported features for Haiku, Sonnet, and Opus.

2. Anthropic Pricing Documentation
   Current pricing for Haiku 4.5, Sonnet 4.6, Opus 4.6, Opus 4.7, prompt caching, batch processing, and 1M-context usage.

3. Claude Code — Costs & Model Usage Patterns
   Useful for understanding model switching, subagents, orchestration strategies, and practical guidance on when to use Haiku vs Sonnet vs Opus.

My Related Notes & Blogs

1. Why You Should Choose Haiku as Default
   My earlier thesis on why a Haiku-first strategy originally made sense economically and operationally.

2. Opus 4.7 — Initial Practical Observations
   Early notes on context retention, hallucinations, workflow behavior, and operational observations while testing Opus 4.7.

3. Jagadeesh Rampam — Substack Archive
   Broader writing on infra-heavy systems, AI engineering workflows, AWS architecture, ML systems, and long-context operational patterns.

Engineering & Long-Context Best Practices

1. OpenAI — Prompt Engineering Best Practices
   Strong reference for practical orchestration, structured prompting, decomposition, and reliability strategies across long workflows.

2. AWS Well-Architected Framework
   Essential reading for scalability, operational excellence, cost optimization, reliability, and architectural trade-offs in infra-heavy systems.

3. Google SRE Workbook
   One of the best operational references for debugging culture, monitoring, production reliability, incident management, and long-running system operations.

When I developed the Parjanya v2.0 frontend on localhost, I ran into a series of issues that only became obvious later in deployment: CORS and preflight failures, mixed-content blocks, proxy quirks, and security leaks. To solve this and get closer to production-parity, I moved toward a server-to-server pattern: serving the SPA through Amazon CloudFront (with an S3/CloudFront static origin or Lambda@Edge) and serving the API through an Application Load Balancer (ALB) in a VPC. This gives me global caching, TLS/WAF protection, single-origin consistency that removes CORS pain, and lower egress costs.

In this post, I go deep into the localhost development pitfalls, CORS mechanics and misconfigurations, localhost vs cloud-hosted trade-offs, and the migration plan I would follow, along with code snippets, architecture diagrams, tables, and references.

TL;DR

I originally wrote a primer of Parjanya v2.0’s move from localhost-first development to a production-shaped AWS architecture. This shorter repost is the key lesson: localhost is great for iteration, but it hides the boundaries that matter in production. CORS failures, presigned upload issues, and origin mismatch kept pointing to the same conclusion — the right model for modern cloud apps is server-to-server.

Serve the frontend from CloudFront, route backend services through ALB, and let the backend control S3 access and tenant isolation. Production-shaped architecture is not a nice-to-have; it is the difference between a system that merely works locally and one that is actually built to scale.

Background & Context: Parjanya v2.0

Parjanya v2.0 is a multi-tenant SaaS system where each tenant has its own storage and its own backend API. In development, the frontend runs on localhost while talking to AWS services for file uploads and API calls. In production, I intend to serve the SPA from a CDN (CloudFront) and the API behind an ALB.

That split introduced problems for me: local development deviated from production in origin and protocol, which led to CORS and mixed-content issues. The prod parity gap was wide. I often saw developers disable CORS checks or use wildcards on S3 buckets for convenience, but that broke strict production settings and weakened security. LocalStack even ignored S3 CORS rules in some versions, which meant bugs were not caught until deployment.

My setup here is that the Parjanya v2.0 backend is AWS-based (ECS/Fargate or EC2 behind an ALB, with S3, IAM, and related services), and that the frontend is a static app that can be hosted on S3 + CloudFront.

Localhost Development Pitfalls

Switching from localhost to AWS exposed several hidden traps for me. The main issues were:

CORS Preflight Failures

Browsers enforce the same-origin policy, so any cross-domain AJAX or fetch request triggers CORS checks. On localhost, I often used Access-Control-Allow-Origin: * or disabled checks (DISABLE_CORS_CHECKS=1 with LocalStack), which masked issues. In production, the SPA origin, such as https://<cloudfront-id>.cloudfront.net, must exactly match S3 bucket CORS rules; otherwise S3 returns 403 Forbidden.

For example, if only https://api.parjanya.com is in AllowedOrigins, a request from https://<cloudfront-id>.cloudfront.net gets blocked. The root cause is simple: dev used wildcards or localhost origins, while prod locked to the CDN domain.

Origin Drift (Dev vs Production)

A subtle pitfall is that the frontend origin differs in dev (http://localhost:3000) versus prod (CloudFront domain or a custom domain). Code that relies on window.location.origin or environment variables can compute the wrong URLs in production. This drift also means the correct CORS origin and cookie domains can be wrong in prod.

Mixed-Content Blocks (HTTPS vs HTTP)

Browsers on https:// pages block any http:// resources. On localhost, I might use plain HTTP without SSL, but once the SPA is on CloudFront over HTTPS, calls to an unsecured origin, or an S3 HTTP URL, are blocked as mixed content. My fix is to use HTTPS everywhere in AWS (ALB and CloudFront) and enforce redirect-to-HTTPS policies.

Dev Proxy vs True Environment

Many dev setups use a proxy, such as npm proxy settings or a simple NGINX layer, to avoid CORS. For example, NGINX on localhost:8001 forwarding /api/ to the real backend means the browser sees only one origin. That hides CORS, but only in development. In production, the CDN fronts the API with a different host unless I unify the route.

IAM / Bucket Policy Surprises

In dev, a single IAM role or default credential set can accidentally be used for all S3 uploads. That can hide cross-tenant access problems. In AWS, I need separate IAM roles and strict bucket policies per tenant. Bucket policies may also imply a deny if conditions are not met; for example, a presigned URL can 403 if the request is not over HTTPS or if the Host header does not match an allowed VPC condition. These issues usually do not appear until I hit real AWS.

LocalStack / Emulator Mismatch

Tools like LocalStack do not always fully honour CORS. In one case, even with CORS configured on the bucket, LocalStack ignored it, and DISABLE_CORS_CHECKS=1 had no effect. That means a workflow that appears to work in dev can fail in prod. LocalStack may also default to HTTP endpoints instead of HTTPS, which introduces mixed-content issues.

Credentials & Cookies

On localhost, cookies are often not sent cross-origin. If my app relies on cookie-based auth (fetch with credentials: 'include'), it can work locally but fail once domains differ. Browsers also forbid Access-Control-Allow-Origin: * when withCredentials is true, so production needs specific origins and Access-Control-Allow-Credentials: true.

Presigned URL Expiry and Caching

Some dev workflows use very short-lived presigned URLs or assume uploads happen instantly. In AWS, CloudFront may cache redirects by default unless configured otherwise, and ALB/CloudFront caches need invalidation after deploys. A misconfigured max-age or missing invalidation can make users see stale JS bundles. Presigned URLs can also expire during multi-part uploads if they are not refreshed.

Other Environmental Mismatches

Hardcoded environment values, such as REACT_APP_API_URL=http://localhost:8080, can cause build issues. Differences in Node versions, npm proxy settings, or TLS certificate trust on developer machines can also produce odd errors that I do not see in production.

Each of these issues comes from the same root problem: development on localhost is not the same as the deployed cloud environment. A major cause is the separate-origin problem. When the frontend and API live under different schemes or hosts, the browser’s SOP and CORS rules start to matter, and preflight requests fail if any method or header is not whitelisted.

Pitfall Comparison Table

CORS Deep Dive: Browser Enforcement & Misconfigurations

CORS (Cross-Origin Resource Sharing) is an HTTP header-based protocol that lets servers whitelist allowed origins for resource sharing. Browsers enforce it like this: when a script on https://frontend.com makes a cross-domain request, the browser adds an Origin: https://frontend.com header. If the request is simple, the browser checks the response for Access-Control-Allow-Origin. If it is missing or does not match, the browser blocks the response in JavaScript and shows a console error such as “CORS request blocked.”

For non-simple requests, like PUT, custom headers, or requests with Content-Type: application/json, browsers first send a preflight OPTIONS request with Origin and Access-Control-Request-Method/Headers. The server must return the correct Access-Control-Allow-* headers. If the preflight fails, the browser aborts without sending the real request.

A missing or mismatched header causes CORS errors that are often hard to debug. Because the browser hides details for security, the console often gives only a generic block message, so I need to inspect the network tab or devtools to see the headers.

Key Headers and Rules

• Access-Control-Allow-Origin: Specifies which origin or origins are allowed. It cannot be * if Access-Control-Allow-Credentials: true is required.

• Access-Control-Allow-Methods: Lists which HTTP methods the server accepts. If my SPA does a PUT but the server only allows GET and POST, preflight fails.

• Access-Control-Allow-Headers: Lists which custom request headers, such as Authorization, are allowed.

• Access-Control-Allow-Credentials: If true, cookies or credentials can be sent. Browsers do not allow * origin when this is true.

Common Misconfigurations

• Wrong Origin Whitelist: If the SPA is served at https://app.example.com but S3 CORS only includes http://localhost:3000, production will fail.

• Forgetting OPTIONS Handling: Some backends only handle GET/POST and ignore OPTIONS. Preflight then returns 403 or 404.

• Missing Response Headers: The resource may respond, but if it does not include CORS headers, the browser blocks it. CloudFront response headers or edge functions can help.

• Credentials & Wildcards: Enabling Access-Control-Allow-Origin: * globally and then using credentials will fail.

• Case or Slash Mismatch: Origins are exact strings, so protocol, port, and trailing slash matter.

• Tooling Oversights: Some clients send headers that trigger preflight. JSON and Authorization almost always do.

By understanding CORS this way, I can configure AWS correctly. For example, on S3 I can apply a bucket CORS configuration like this:

<CORSConfiguration>
 <CORSRule>
   <AllowedOrigin>https://d8sqejhsq49zo.cloudfront.net</AllowedOrigin>
   <AllowedMethod>GET</AllowedMethod>
   <AllowedMethod>PUT</AllowedMethod>
   <AllowedHeader>*</AllowedHeader>
   <ExposeHeader>ETag</ExposeHeader>
 </CORSRule>
</CORSConfiguration>

In practice, I whitelist the specific CloudFront or app domain in AllowedOrigins and include every method I need.

CloudFront also offers features that help with CORS. I can use a Response Header Policy that automatically adds Access-Control-Allow-Origin, or I can use a CloudFront Function to dynamically echo the Origin. The simplest pattern is still to use CloudFront + ALB with the same origin, so CORS becomes mostly unnecessary.

CloudFront + ALB Architecture: Pros & Cons

Moving the frontend to CloudFront and the API behind an ALB gives me a lot of benefits, but it also adds complexity.

What I Gain

• Global Performance & Latency: CloudFront has a global edge network, and static assets can be cached at the edge. That reduces TTFB and improves performance for distant clients.

• Security & Compliance: CloudFront gives me DDoS protection and a clean place to attach AWS WAF. I can also isolate the ALB inside a private VPC and allow access only from CloudFront.

• Single-Origin Domain: By serving the SPA from CloudFront and routing API calls through the same CloudFront distribution or domain, I remove most CORS issues entirely.

• Cost Optimisation: CloudFront can reduce outbound bandwidth costs by caching content closer to users and lowering load on the ALB and backend.

What I Trade Off

• Architecture Complexity: I now manage CloudFront, Lambda@Edge or CloudFront Functions if needed, the ALB, Route 53, ACM certificates, and deployment automation.

• Debug and Deploy Overhead: Changes are slower to test because I need deploys and cache invalidations.

• Cache Invalidation Hassles: SPA releases can leave stale files behind unless I use hashed filenames or invalidate the right paths.

• Session & Auth Handling: Cookie-based auth needs careful CloudFront forwarding rules and secure cookie settings.

• Extra Hop Latency: There is an extra network hop, but cache hits usually outweigh that cost.

• CDN Cost: CloudFront and related services have a cost, although caching and egress savings can offset it at scale.

High-Level Architecture Diagram

This is the architecture I want: the browser hits CloudFront, static content comes from S3 through CloudFront, and dynamic requests hit the ALB. Both S3 and ALB stay secured, and only CloudFront acts as the public entry point.

Migration Strategy & Checklist

Step 1: Preparation & Planning

I first audit the current deployment: all domains, SSL certificates, DNS records, WAF rules, IAM roles, and the existing frontend and API endpoints. Then I define the target state: a single CloudFront distribution, for example with an alternate domain like app.example.com, fronting an internal ALB and S3 origin. I also estimate the work and categorise risk so I can separate simple changes from business-critical flows.

Step 2: Provision Infrastructure

• Create the CloudFront distribution. If I use Terraform, I need a custom_origin_config for the ALB origin so it is not treated like an S3 origin.

• Configure cache behavior. I point routes to the ALB origin for API calls and to S3 for static routes. I forward the Origin header and the relevant CORS headers so the origin sees them.

• Set up TLS certificates. I use ACM for the frontend domain and attach the certificate to CloudFront. The ALB also needs HTTPS.

• Restrict ALB access. I create a security group that allows inbound 443 only from CloudFront IP ranges or a trusted origin header setup.

• Configure S3 CORS and bucket policy. I allow my frontend domain and enforce HTTPS via aws:SecureTransport.

• Update CI/CD. After uploading SPA artifacts to S3, I trigger a CloudFront invalidation for the changed paths.

Step 3: Testing & Validation

• Smoke tests in staging: I point a test domain to the new distribution and verify that static assets, API calls, and auth flows work.

• Performance and security verification: I check cache hit ratios, enable WAF metrics if used, and confirm that direct access to the old ALB is blocked.

• Staged DNS cutover: I move traffic gradually using Route 53 weighted records or a similar approach.

Step 4: Full Cutover and Rollback Plan

Once I am confident, I point the production app domain to CloudFront. I keep rollback simple by preserving the old config and having a direct path back to the ALB if needed. Because the ALB is restricted to CloudFront traffic, rollback needs to be planned carefully.

Code & Config Snippets

Amazon S3 Bucket CORS (example JSON)

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://your.frontend.domain"],
      "AllowedMethods": ["GET", "PUT", "POST", "DELETE", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag", "x-amz-version-id"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Terraform: AWS CloudFront Distribution (ALB origin)

resource "aws_cloudfront_distribution" "cdn" {
  origin {
    domain_name = aws_lb.api.dns_name
    origin_id   = "api-origin"
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  default_cache_behavior {
    target_origin_id         = "api-origin"
    viewer_protocol_policy   = "redirect-to-https"
    forwarded_values {
      headers = ["Origin", "Access-Control-Request-Method", "Access-Control-Request-Headers"]
      cookies { forward = "all" }
      query_string = true
    }
    allowed_methods = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods  = ["GET", "HEAD"]
  }

  viewer_certificate {
    acm_certificate_arn      = aws_acm_certificate.certificate.arn
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1.2_2021"
  }
}

NGINX Reverse Proxy (Dev Example)

server {
  listen      8001;
  server_name localhost;
  location / {
    proxy_pass http://127.0.0.1:3000;
  }
  location /api/ {
    proxy_pass http://127.0.0.1:8080/;
  }
}

CloudFront Function (or Lambda@Edge) to Add CORS

async function handler(event) {
  var request = event.request;
  var response = event.response;
  if (request.headers.origin && !response.headers['access-control-allow-origin']) {
    response.headers['access-control-allow-origin'] = { value: request.headers.origin.value };
  }
  return response;
}

Sample IAM Policy (Bucket HTTPS Enforce)

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "RequireTLS",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"],
    "Condition": {"Bool": {"aws:SecureTransport": "false"}}
  }]
}

Localhost vs Cloud-Hosted Frontend

Monitoring, Logging & Testing Tools

Once the system is live, I want to rely on:

• Metrics & Logs: CloudWatch metrics for CloudFront and ALB, plus access logs.

• Real-User Monitoring: CloudWatch RUM for client-side metrics and errors.

• Tracing: AWS X-Ray if the backend is complex.

• Error Tracking: A frontend JS error reporter such as Sentry.

• Testing Tools: Postman or curl with an Origin header for CORS validation, and load tools like Apache Bench or JMeter for performance testing.

Community Insights & References

AWS Official Guidance

• Optimizing application performance: CloudFront + ALB strategic benefits
  Covers CloudFront in front of ALB, performance improvements, origin protection, and migration considerations.

• Introducing Amazon CloudFront VPC Origins
  Explains private VPC origins, CloudFront-to-private-ALB architecture, and stronger origin isolation.

• Restrict access to Application Load Balancers from CloudFront only
  AWS guidance for protecting ALB origins using CloudFront custom headers and origin restrictions.

• CORS configuration through Amazon CloudFront
  Deep dive into forwarding CORS headers, OPTIONS caching, and CloudFront cache/origin request policies.

• Accelerate and protect dynamic workloads with CloudFront
  Focuses on performance, persistent connections, and origin offload for dynamic applications.

MDN Documentation

• MDN — Cross-Origin Resource Sharing (CORS) Guide
  The foundational browser-side explanation of CORS, preflight requests, and browser enforcement.

• MDN — CORS Errors Guide
  Excellent debugging reference for interpreting browser CORS failures.

• MDN — Access-Control-Allow-Origin Header
  Detailed explanation of the most important CORS response header.

• MDN — Practical CORS Configuration Guide
  Security-oriented implementation guidance for configuring CORS correctly.

• MDN — CORS Glossary
  Concise overview of same-origin policy and cross-origin access.

AWS Documentation

• Amazon S3 — Using CORS
  Official S3 documentation for configuring and understanding CORS.

• Amazon S3 — Troubleshooting CORS
  Extremely useful for diagnosing 403s, AllowedOrigins mismatches, and missing methods.

• Amazon S3 — Configuring CORS Examples
  Shows JSON examples for S3 bucket CORS policies.

• Amazon S3 — Elements of a CORS Configuration
  Detailed breakdown of AllowedOrigins, AllowedMethods, AllowedHeaders, etc.

• AWS re:Post — Resolve “No Access-Control-Allow-Origin” errors in CloudFront
  Useful troubleshooting walkthrough for CloudFront + CORS setups.

• AWS re:Post — Configure and confirm CORS in Amazon S3
  Practical debugging examples using curl and OPTIONS preflight checks.

Community Q&A

• Stack Overflow — S3 Access-Control-Allow-Origin Header discussion
  One of the classic threads on S3 CORS response handling.

• Stack Overflow — S3 not returning Access-Control-Allow-Origin headers
  Useful troubleshooting thread for bucket/object-level CORS confusion.

• Express.js CORS Middleware Docs
  Good explanation of browser-enforced CORS vs server-side behavior.

• AWS re:Post — Intermittent CORS errors even with Allow-Origin configured
  Helpful for understanding cache and header propagation edge cases.

Case Study Blogs & Practical Guides

• StormIT — CloudFront Distribution for EC2 & ALB
  Practical CloudFront + ALB deployment walkthrough.

• Using AWS CloudFront to improve performance, security & availability
  Covers caching, origin groups, availability, and performance patterns.

• Handling CORS Issues in Amazon S3
  Community-written debugging guide with real-world examples.

• Adding a CORS Policy to an S3 Bucket
  Lightweight and practical S3 bucket CORS reference.

• Prepare and run performance tests for CloudFront using RUM
  Great reference for validating CloudFront improvements in production.

Closing Thought

What I learned is simple: localhost is great for speed, but it is not production. If I want fewer surprises, fewer CORS headaches, and better security, I need to design the frontend and backend like they will live in production from the beginning.

Lessons from Parjanya v2.0 and the v5.4 architecture revision

I never treated localhost as the target architecture. It was simply the default starting point for local development, as it is for most full-stack systems. But as I built Parjanya v2.0, I kept running into a recurring class of failures: CORS errors, origin mismatch, presigned upload friction, and the widening gap between development-time assumptions and production-time behaviour. That experience forced me to step back and re-evaluate the system from first principles. What emerged was not a localhost fix, but a stronger production-shaped model: server-to-server architecture for cloud-based full-stack applications and distributed systems.

Why localhost stopped being enough

Local development is useful because it is fast. But speed can hide architecture debt.

In Parjanya v2.0, the real problem was not that localhost was broken. The problem was that localhost encouraged a development loop that was too far removed from the actual production trust boundary. The browser, S3, IAM, and backend services each enforce different rules, and local development often smooths those rules away just enough to create false confidence. Then the system reaches the real deployment path and the hidden assumptions come apart.

That became especially clear with browser-based uploads. A local setup may appear functional, but the moment the frontend starts speaking directly to tenant-specific buckets in S3, the browser becomes an active participant in the security model. The result is not just a technical inconvenience; it is a signal that the architecture itself needs to be shaped more deliberately.

The first real signal: CORS was exposing the architecture gap

CORS was one of the clearest signs that the local development experience was drifting away from the real system.

In theory, CORS is straightforward. In practice, it is one of the fastest ways to reveal that the frontend and backend are too loosely coupled. In Parjanya, browser uploads to S3 were not talking to a generic bucket. They were talking to tenant-scoped storage with strict policy boundaries. That meant every origin, method, and header mattered. A local proxy or permissive dev configuration might make the request succeed temporarily, but that success did not mean the architecture was correct. It only meant the browser had not yet been forced to tell the truth.

The architectural implication was obvious: if the browser has to fight the security model just to upload a file, the development workflow is revealing a deeper mismatch. In my case, that mismatch was not merely about CORS syntax. It was about the location of trust.

What Parjanya v2.0 architecture v5.4 actually changed

Architecture v5.4 is an architecture revision inside Parjanya v2.0, not a separate product. That revision was shaped by one principle: move trust out of the browser and into the backend.

The S3 architecture notes show the core design clearly: one bucket per tenant, IAM-scoped permissions, bucket policies, presigned URL generation in the backend, and lifecycle rules for cost control. The model is intentionally infrastructure-enforced. Tenant isolation is not left to frontend conventions or prefix hygiene. It is built into the storage boundary itself.

That design changes the system in a fundamental way. Instead of letting the frontend improvise access, the backend becomes the authority that validates identity, determines tenant ownership, constructs the object key, and issues a time-bound signed capability. The browser no longer owns authorization. It simply executes what the backend has already allowed.

Why server-to-server architecture made more sense

Once I looked at the problem that way, the answer became much clearer.

For a cloud-native application, especially a multi-tenant one, the frontend should not be the place where trust decisions live. The frontend should be delivered through a stable edge layer, the backend should own policy and identity, and object storage should be accessed through signed backend-issued capabilities. That is why I started favouring a server-to-server model for real production systems.

In practice, that means serving the frontend from CloudFront, exposing backend services through ALB, and letting the backend own S3 access through presigned URLs. The browser becomes a client, not a policy engine. The backend becomes the control point. The trust boundary becomes explicit. And the deployment path starts to look like the real system instead of a local approximation.

That shift also made debugging easier. When a request fails, there are fewer places to guess. Is it a policy issue? A bucket issue? A signature issue? A CORS issue? A routing issue? With a server-to-server design, the failure surface becomes narrower and more legible.

Multi-tenant storage is an infrastructure problem, not just an application problem

One of the strongest lessons from Parjanya’s architecture revision was that multi-tenant storage cannot be treated as an application afterthought.

The architecture notes show a per-tenant bucket model with explicit role and bucket policy enforcement. That is the right direction because it reduces ambiguity. Tenant isolation should not depend entirely on code discipline or naming conventions; it should be enforced by the infrastructure boundary itself.

This matters because multi-tenant systems fail in subtle ways when storage is shared too casually. A shared bucket can work, but it increases the burden on prefix logic, object ownership rules, and application-layer checks. A per-tenant bucket model is cleaner to reason about and easier to audit. It also aligns better with the way the rest of the Parjanya stack is already designed: backend validation, signed upload paths, explicit bucket policies, and lifecycle management.

Presigned URLs are capability tokens, not shortcuts

Presigned URLs were another major lesson.

A presigned URL is not just a convenience feature. It is a capability token that allows a very specific action for a very specific scope and time window. That makes it a very strong fit for Parjanya because the backend can validate the request first and then issue a constrained upload path. The browser does not need broad credentials. It only needs the signed permission to perform the exact operation the backend allowed.

That is the right security model for a SaaS system with tenant isolation. It reduces exposure, tightens the trust boundary, and lets the backend remain the source of truth for who can upload what and where. It also makes the system far easier to evolve because the upload flow is now a controlled protocol, not a loose browser-side improvisation.

Production-shaped development is the real goal

The biggest shift in my thinking was this: the goal is not to make localhost feel like production forever. The goal is to make local development reveal the same architectural boundaries that production will enforce.

That means treating origin boundaries, identity boundaries, and policy boundaries as first-class concerns from the beginning. It means avoiding development shortcuts that erase those boundaries too early. It means designing the system so that local development is still useful, but not misleading. And it means accepting that some friction is not a bug in the workflow — it is the architecture trying to tell you something.

In Parjanya v2.0, that lesson showed up everywhere: in CORS behavior, in presigned URL flows, in per-tenant bucket design, and in the decision to move toward a cleaner CloudFront + ALB + backend-controlled storage path. The architecture revision in v5.4 was not just an optimization. It was a correction toward a more honest production model.

What I would tell another engineering team

If I were explaining this to another team building a cloud-native product, I would say this:

localhost is fine for iteration, but it should not define the architecture. For distributed systems, especially multi-tenant SaaS, the real system lives in the boundaries: frontend delivery, backend authority, object storage policy, and signed capabilities. If those boundaries are blurred during development, the application will pay for it later in debugging time, security risk, and operational complexity.

The better pattern is production-shaped from the start:

the frontend should be served from CloudFront, backend services should sit behind ALB, the backend should generate presigned URLs, and S3 access should be governed by infrastructure-enforced tenant isolation. That design is cleaner, more secure, and much closer to how real cloud applications behave in production.

Closing thought

Parjanya v2.0 taught me that architecture is often revealed through friction.

CORS failures were not just browser noise. They were evidence that trust boundaries needed to be sharper. Presigned URLs were not just an implementation detail. They were the correct way to move capability into the backend. Per-tenant buckets were not just a storage choice. They were the cleanest way to enforce isolation. And localhost, while useful, was never the architecture — only the starting point.

The end result is simple: I now think in terms of production-shaped systems, not localhost-shaped workflows. For me, that is the difference between a prototype that merely works and a system that is actually ready to scale.

Good catch — here’s a clean, copy-paste ready References section with actual links (kept high-signal and aligned to your architecture themes).

📚 References & Further Reading

☁️ AWS (Core to this architecture)

• Multi-tenant S3 design patterns (must read)
  AWS Storage Blog
  https://aws.amazon.com/blogs/storage/design-patterns-for-multi-tenant-access-control-on-amazon-s3/

• Dynamic routing with CloudFront (multi-tenant edge patterns)
  AWS Architecture Blog
  https://aws.amazon.com/blogs/architecture/dynamic-request-routing-in-multi-tenant-systems-with-amazon-cloudfront/

• CloudFront + origin patterns (edge → backend)
  AWS Docs
  https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistS3AndCustomOrigins.html

• Restricting ALB access via CloudFront (edge-controlled backend)
  AWS Docs
  https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/restrict-access-to-load-balancer.html

• Presigned URLs (backend-controlled access model)
  AWS Docs
  https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html

• CORS troubleshooting (why browser uploads fail)
  AWS Docs
  https://docs.aws.amazon.com/AmazonS3/latest/userguide/cors-troubleshooting.html

• AWS Well-Architected Framework
  https://aws.amazon.com/architecture/well-architected/

These patterns are consistent across modern cloud-native systems:
move trust to the backend, enforce isolation at the infrastructure layer, and design for production boundaries early.

It wasn’t subtle.

It read less code.
It reasoned less deeply.
It stopped early.

And yet—nothing “official” had changed.

When Anthropic finally published their postmortem, they called it a set of bugs.

After digging through the data and my own usage, I don’t think that tells the full story.

Adaptive Thinking Gone Wrong

Why Claude’s Silence Broke My Trust

For months, Claude Code with Opus 4.6 was the backbone of my engineering workflow.

With Opus 4.7—and the rise of “adaptive thinking”—that same assistant began to feel unreliable, opaque, and, most concerning of all, quietly downgraded.

Anthropic’s April 23 postmortem confirmed something many of us already felt: Claude Code did get worse for weeks.

https://www.anthropic.com/engineering/april-23-postmortem

But the explanation—and the framing—left me with a deeper concern:

This wasn’t just a bug. It was a runtime failure of adaptive thinking.

This is a first-person account of what changed, what the data shows, and why this moment broke trust for me—even though I’m still not walking away.

1. When It Worked: Opus 4.6 as a Force Multiplier

My earlier experience with Opus 4.6 was very interesting.

Claude Code behaved like a senior engineer embedded in my workflow:

• It read deeply before editing

• It maintained context across long sessions

• It handled multi-step refactors with caution and coherence

Under the hood, this was already powered by adaptive thinking—a system that dynamically allocates reasoning effort per task.

But here’s the key:

It worked quietly and reliably.

2. When It Broke: Opus 4.7 Regression

With Opus 4.7, the experience flipped.

What I observed:

• Shallower reasoning

• More hallucinated structure

• Premature stopping or avoidance

• Worse tool usage

• Increased repetition

All while adaptive thinking was being positioned as the primary reasoning mode.

At the same time:

• Usage limits tightened

• Sessions became less predictable

From a user perspective:

It felt like a silent downgrade.

3. Anthropic’s Explanation

https://www.anthropic.com/engineering/april-23-postmortem

Anthropic attributes the regression to three issues:

Effort downgrade (March 4)

Default reasoning dropped from high → medium

Thinking cache bug (March 26)

Thinking history cleared after idle periods

Verbosity constraint (April 16)

Artificial limits reduced output quality

All three are now reverted.

Their position:

The model wasn’t degraded—only the application layer was.

4. What the Data Shows

Independent telemetry and analysis revealed:

• Files read before editing: 6.6 → 2.0

• Thinking length: ~2200 → ~600 characters

• Stop/avoidance behaviors: spiked significantly

These are not isolated anomalies.

They indicate systemic changes in:

• Reasoning depth

• Context retention

• Decision consistency

5. The Real Problem: A Runtime Failure

All three issues converge on one layer:

Adaptive thinking at runtime

• Lower effort → less reasoning

• Cache bug → lost reasoning

• Verbosity cap → suppressed reasoning

This creates a cascade:

Adaptive Thinking Failure Chain

6. Why the Framing Matters

Anthropic’s framing:

“A few issues, now fixed”

User reality:

“Core reasoning behaviour degraded for weeks”

That gap is where trust erodes.

Because for power users:

The runtime layer is the product

7. Bad Timing Made It Worse

This regression coincided with:

• Tighter usage limits

• Pricing pressure toward higher tiers

• Source-code leak incident

• Mythos leak

Together:

Reduced capability + reduced transparency + increased constraints

8. Why I’m Still Using Claude

I’m not leaving.

Because when it works:

It’s still one of the best engineering copilots available.

To Anthropic’s credit:

• Detailed postmortem

• Fixes implemented

• Public acknowledgment

But:

Benefit of the doubt ≠ restored trust

9. What Needs to Change

To rebuild trust, adaptive thinking must become:

Observable

Expose reasoning allocation

Controllable

Allow effort overrides

Stable

Test long-session behavior

Transparent

Treat runtime changes like model updates

Final Thought

Adaptive thinking isn’t just a feature.

It is the system between the user and the model.

When that system changes silently:

• Behaviour shifts

• Expectations break

• Trust erodes

I’ll keep using Claude.

But not as a black box anymore. In fact, exploring the options like Orchestrators like a battle-tested Cursor code and in future LiteLLMProxy along with Claude, gpt-4o, Gemma4:31b and Qwen-3.6 and/or Kimi k2.6

This is the simplest way to understand what actually broke:

What the Diagram Shows

At first glance, these look like three unrelated issues:

• Effort downgrade

• Cache bug

• Verbosity constraint

But they all hit the same underlying system:

How the model thinks, remembers, and explains

The Chain Reaction

Effort ↓ → Thinking ↓  
Cache Bug → Memory ↓  
Verbosity Cap → Visibility ↓  

= System Behavior Collapse

Each step compounds the next:

• Less effort → shallower reasoning

• Broken cache → lost continuity

• Lower verbosity → hidden reasoning

Individually manageable.
Together:

A collapse in perceived intelligence

Why This Matters

From a system design perspective, this is the real insight:

Reasoning quality is not just a model property.
It is an emergent property of the runtime.

That runtime includes:

• Effort allocation (adaptive thinking)

• Memory persistence (thinking cache)

• Output constraints (verbosity rules)

When all three degrade—even slightly—the system doesn’t fail gracefully.

It fails behaviourally.

The Key Takeaway

The model didn’t get worse.
The system deciding how it thinks did.

And that distinction is exactly why this episode matters.

References

1. Claude Code Regression Update: Anthropic’s April 23 Postmortem Changes the Story

2. Clients Were Right: Anthropic Admits Claude Code Got Dumber
   https://kingy.ai/ai/clients-were-right-anthropic-admits-claude-code-got-dumber-not-claude-post-mortem/

3. An update on recent Claude Code quality reports
   https://www.anthropic.com/engineering/april-23-postmortem

4. Deep News — Superlinear Academy
   https://yage.ai/share/

5. The Claude Code Nerf: An Invisible, Unilateral Downgrade
   https://yage.ai/share/claude-code-runtime-regression-en-20260407.html

6. Adaptive thinking - Amazon Bedrock
   https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-adaptive-thinking.html

7. Claude Opus 4.6 adaptive reasoning
   https://www.infoq.com/news/2026/03/opus-4-6-context-compaction/

8. Adaptive thinking - Claude API Docs
   https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking

9. Opus 4.7 overview
   https://artificialanalysis.ai/articles/opus-4-7-everything-you-need-to-know

10. Did Anthropic Secretly Nerf Claude?
    https://thedroidguy.com/anthropic-nerf-claude-quality-degradation-explained-1273110

11. Anthropic source code leak (NDTV)
    https://www.ndtv.com/feature/2026-just-got-crazy-internet-erupts-after-anthropics-claude-source-code-leak-shakes-ai-industry-11294628

12. Anthropic leaked its own Claude source code (Axios)
    https://www.axios.com/2026/03/31/anthropic-leaked-source-code-ai

13. Claude Code leak analysis (Dev.to)
    https://dev.to/varshithvhegde/the-great-claude-code-leak-of-2026-accident-incompetence-or-the-best-pr-stunt-in-ai-history-3igm

14. Claude Mythos leak https://absolutelyagentic.com/p/claude-mythos-the-ai-model-anthropic-won-t-release-to-the-public

15. Mythos leak coverage (Fortune)
    https://fortune.com/2026/03/26/anthropic-says-testing-mythos-powerful-new-ai-model-after-data-leak-reveals-its-existence-step-change-in-capabilities/

16. Mythos summary (Mashable)
    https://mashable.com/article/claude-mythos-ai-model-anthropic-leak

17. Anthropic second security lapse (Fortune)
    https://fortune.com/2026/03/31/anthropic-source-code-claude-code-data-leak-second-security-lapse-days-after-accidentally-revealing-mythos/

18. Reddit discussion
    https://www.reddit.com/r/ClaudeAI/comments/1s9ai45/anthropic_leaked_512000_lines_of_claude_code/

19. Alignment Forum discussion
    https://www.alignmentforum.org/posts/K8FxfK9GmJfiAhgcT/anthropic-repeatedly-accidentally-trained-against-the-cot

20. Claude Code leak breakdown
    https://www.felamity.com/post/claude-code-leak-2026-what-happened-and-why-it-matters-for-ai-security

21. Mythos video

1. Japanese coverage of quality regression
   https://innovatopia.jp/ai/ai-news/98588/

Three weeks ago,

I wrote about how Claude Opus 4.6—with its 1M context window—felt like a genuine breakthrough for my workflow:

That post built on a broader recommendation I had been advocating: start with lighter models and escalate only when needed. I wrote about that approach here:

That strategy worked when the model ladder felt consistent. But in practice today, the gap between versions has widened enough that this escalation model is far less reliable.

In my workflow—infra-heavy, ML-heavy, multi-repo—the bulk of the system was actually built by Opus 4.6 within 3–4 days. What followed were incremental improvements:

• Prompt updates for stricter IQA

• Docker build refinements

• CLIP-IQA deprecation

• Graviton migrations

• SageMaker removal → EC2 migration (Spot vs on-demand)

• Handling infra quirks (e.g., g4dn.xlarge unavailability in ap-south-1)

These are not “greenfield” tasks—they depend heavily on accurate recall of current system state and few of them were encountered earlier and have documented in the architecture review (23+ sections, Appendix of 6 detailed sections) with enough context and thorough deep dive and always handy at the root folder of workspace.

Since moving to Opus 4.7, that’s exactly where things started breaking.

For the past 18 days, I’ve been dealing with usage limits, context drift, and behavior changes I never asked for and now hit weekly limits to be completely locked out for the next 5 days —enough to pause my subscription and evaluate alternatives.

you can see from my usage chart, I tried leveraging different models based on the usecase/component/task and complexity

Going into the details…

The 4.6 Honeymoon: 1M Context Without the Drama

My setup with Opus 4.6:

• Multiple repos (ML, backend, frontend, shared libs, infra)

• Plan → execute workflows

• Heavy context, but usually <200k tokens

And yet, it behaved like a disciplined senior engineer:

• Stayed within plan boundaries

• Maintained architectural coherence

• Delivered steady, compounding progress

It hit the long-context sweet spot: large memory, without instability.

What Changed with Opus 4.7

With adaptive thinking enabled by default, the behaviour profile shifted significantly.

1. One-Prompt Runs That Burn an Hour

• Prompts run excessively long

• Usage limits hit faster

• Thinking increases, output doesn’t

It feels like over-analysis replacing execution.

2. Context Bloat and Losing the “Current State”

In long sessions:

• Old decisions override new ones

• Recent corrections get ignored

• Model behaves as if it’s working on stale code

In a poly-repo system, this is critical failure—not inconvenience.

3. Hallucinated Behaviour (Real Production Risk)

The most concerning issue: unsolicited behaviour changes

Example 1: Image Rotation

• Final testing stage

• Behaviour already agreed

• Suddenly: vertical images rotate

No prompt. No plan. No instruction.

Instead of reverting, the model:

• Expanded the problem

• Designed rotation logic

• Introduced new edge cases

Even explicit instructions like

“Never rotate images”
didn’t cleanly undo prior drift.

Example 2: Hardcoded Aspect Ratio Logic (Half an Hour Later)

A second example happened shortly after, while fixing the same issue.

The model proposed:

• Portrait → 1366×2048

• Landscape → 2048×1366

• “No rotation logic, just libraw defaults”

This sounds reasonable—but it’s wrong.

The real requirement is aspect-ratio-aware handling, because RAW files can be:

• 16:9

• 4:5

• 1:1

• others

Even though my current dataset (1000+ images) includes 1:1, the model ignored that variability and jumped to a hardcoded assumption.

These are “small” issues—but in production, they are trust breakers:

• They introduce silent regressions

• They require manual detection

• They increase review overhead before launch

And also not sure, how many of these issues (small/high/max) are persists, including IQA logic which is a ship-blocker, infra changes (burns my pocket!) etc.

This Isn’t a Single-Model Issue

One important detail: this is not happening in isolation.

I’ve actively used:

• Opus 4.6

• Opus 4.7

• Sonnet 4.6

• Haiku 4.5

The attached usage snapshot reflects this multi-model workflow.

That actually makes the issue more concerning—because:

• The regression isn’t tied to a single model

• Switching models doesn’t reliably stabilize behaviour

• The “escalation strategy” becomes inconsistent

What Others Are Seeing

Across blogs, Reddit, and developer communities:

• Prompt regressions vs 4.6

• Higher sensitivity to phrasing

• Increased hallucinations in coding workflows

• Faster quota exhaustion

There are positive reports—but the split is real.

Here are 7 solid, diverse sources (Reddit, GitHub, news, blogs, HN) that reinforce the exact patterns you’re describing—context drift, adaptive loops, hallucinations, and regression concerns. I’ve curated them so you can directly reference them in your appendix or when sharing with Anthropic.

🔴 1. Reddit – Developer discussion on adaptive thinking behavior

• Claude Opus 4.7 Changed How Thinking Works (Reddit)

Key takeaway:

• Developers discuss how reasoning flow changed, including issues around tracking context and understanding where reasoning goes wrong.

🔴 2. GitHub Issue – Adaptive thinking inconsistency

• Claude 4.7 Opus thinking inconsistency issue (#5153)

Key takeaway:

• 4.7 behaves differently from 4.6

• Thinking visibility missing / inconsistent in real integrations

• Confirms behavioural differences at system level, not just UX perception

🔴 3. GitHub (Anthropic / Claude Code) – Real workflow regression data

• Claude Code regression issue (complex workflows degraded)

Key takeaway (this one is gold for your thesis):

• Increase in reasoning loops and contradictions

• “Edits without reading context” jumped significantly

• Model choosing “simplest fix” instead of correct fix

• 12× increase in user interruptions

This directly maps to:

• Context drift

• Adaptive loops

• Hallucinated or low-quality edits

🔴 4. News (Business Insider) – Token burn + backlash

• Claude Opus 4.7 backlash and token usage issues

Key takeaway:

• Users report:

  • Higher token consumption

  • Slower / less useful outputs

  • Regressions vs earlier versions

• Adaptive reasoning cited as a possible cause

🔴 5. News (TechRadar / AMD AI Head criticism)

• AMD AI head: Claude cannot be trusted for complex engineering

Key takeaway:

• Large-scale real-world usage (6,800+ sessions)

• Reports:

  • Ignoring instructions

  • Contradictions

  • Reduced reliability

• Explicit claim: decline in engineering trustworthiness

🔴 6. Blog / Analysis – Token inflation + adaptive thinking shift

• Claude Opus 4.7 analysis (token inflation + adaptive thinking)

Key takeaway:

• New tokenizer → ~1.0–1.35× more tokens per task

• Shift to adaptive thinking instead of fixed budgets

• Explains:

  • Faster quota exhaustion

  • Less predictable compute behavior

🔴 7. Hacker News – Forced adaptive thinking model behaviour

• HN discussion: adaptive thinking mandatory in 4.7

Key takeaway:

• Adaptive thinking is no longer optional

• Traditional “controlled reasoning” approaches removed

• Developers experimenting with effort tuning to avoid issues

🧠 Blog / Analysis – performance + regression narrative

• Claude Opus 4.7 flaky performance analysis

• Reports:

  • Self-contradictions

  • Degraded reliability

Why Adaptive Thinking Might Be Backfiring

Conceptually:

Let the model decide when to think more.

In practice:

Unbounded Initiative

Model “improves” things you didn’t ask for

Token Hunger

High effort → excessive reasoning

Plan Violations

“Plan mode” isn’t strictly enforced

This creates a system that is:

• More powerful

• But less predictable

• And harder to trust

Where This Leaves Me

Over 18 days:

• Frequent limit hits

• Context instability

• Silent behaviour drift

Result:

• Progress stalled

• Subscription paused

• Alternatives being evaluated

• And completely locked out with weekly limits hit (I’m on Max plan: $200 + $20 top-up)

This is a deal-breaker for me!

It’s:

4.7 is powerful—but volatile for long-running workflows.

What I’d Love to See

1. Strict Execution Mode

No deviation from plan

2. Hard Thinking Budgets

Tokens + time caps + visibility

3. Context Hygiene Tools

Detect stale vs current state

4. Strong Plan Guarantees

“Plan only” must mean no execution

Closing Thought

Adaptive thinking is a sharp tool.

Right now, it feels like:

• Extremely capable

• But lacking guardrails

And for production systems:

Capability without predictability is risk.

Appendix: External Signals Worth Noting (with sources)

This is the factual trail I would attach when sharing this with Anthropic. These are not isolated anecdotes—they reflect consistent patterns across GitHub issues, news coverage, and independent analyses.

1. Adaptive thinking & token usage changes

• Opus 4.7 introduces adaptive thinking and removes earlier budget-based controls, replacing them with effort tuning and task budgets

• Token usage can increase significantly due to tokenizer changes (~1.0–1.35× for the same task)

👉 Relevance:
Explains why usage limits are hit faster and why compute behavior feels less predictable.

2. Higher quota burn and cost impact in real workflows

• Measured reports show Opus 4.7 consuming ~2.4× quota compared to 4.6 in real API usage scenarios

• News coverage confirms user complaints about higher token consumption and slower outputs

👉 Relevance:
Directly supports the adaptive loop + quota exhaustion issue.

3. Reasoning loops, shallow edits, and workflow degradation

• GitHub issue analysis shows models shifting toward:

  • “edit without reading”

  • “simplest fix instead of correct fix”

  • increased need for human intervention

👉 Relevance:
Matches:

• Adaptive loops

• Reduced execution quality

• Loss of disciplined workflow behavior

4. Silent failures and stuck execution loops

• Reports of sessions staying active without progress (no tool execution, no response advancement)

👉 Relevance:
Real-world evidence of non-converging loops—exactly what you observed with long-running prompts.

5. API and behavior inconsistencies across environments

• Errors due to thinking mode changes and removed parameters (e.g., thinking.type.enabled no longer supported)

• Regression reports where previously working setups fail in 4.7 environments

👉 Relevance:
Explains instability across toolchains (Bedrock, Claude Code, etc.)

6. Broader user backlash and reliability concerns

• Reports of:

  • poorer performance

  • inaccuracies

  • unexpected reasoning behavior

  • increased cost
    across Reddit and developer communities

• Coverage highlighting growing frustration among power users and perceived regressions

👉 Relevance:
Confirms your experience is not isolated

7. Mixed positioning: improved capability, but inconsistent execution

• Opus 4.7 is described as:

  • more capable in theory

  • better at long-running tasks

  • but still inconsistent in real-world workflows

👉 Relevance:
Captures the core tension:

More powerful ≠ more reliable

👉 Food for thought

The patterns described in this post—context drift, adaptive loops, and hallucinated behaviour changes—are not isolated observations.

They are consistently reflected across:

• GitHub issue logs

• API migration changes

• News reports

• Independent developer analyses

Which suggests this is not a prompt problem or a single workflow issue—

but a system-level behavioural shift in how the model operates under adaptive thinking.