A few months ago, I wrote a short note arguing that developer-experience platforms are excellent at velocity, but that AI systems demand a different kind of architectural thinking.
My argument was that convenient shortcuts can eventually accumulate into what I called AI debt: runaway costs, reproducibility gaps, and debugging opacity.
That argument was written from conviction.
This one is written from production.
I launched Parjanya v2.0, our multi-tenant SaaS platform for professional photographers, as a self-serve trial on 1st August 2026.
Under the hood, it runs a self-hosted 8B-parameter vision-language model, Qwen3-VL-8B in 4-bit quantization, alongside Google’s SigLIP 2 for semantic-search embeddings.
The system has processed individual photography sessions containing 12,000 images and roughly 450 GB in a single pass, and today serves tens of thousands of requests a week.
Between the first sprint in March and the July launch, I went through four complete architecture generations.
Almost every generation boundary came down to the same question:
Do I buy the platform, or do I own the primitive?
Sometimes I adopted the managed service. Sometimes I replaced it. And sometimes I came very close to replacing a primitive with a managed service, only to run the numbers and walk away.
The four headline decisions were:
SageMaker vs. Bedrock vs. raw AWS primitives for VLM and SigLIP 2 inference
Vercel vs. S3 + CloudFront for the React + TypeScript frontend
OpenRouter vs. a self-hosted model gateway
Clerk vs. Cognito + SES for authentication and transactional email
There were also three less glamorous decisions that ended up moving the bill more than some of the headline ones:
Spot and Graviton
OpenSearch and Step Functions
CloudWatch, networking and storage classes
For each decision, I wanted to understand not only what the comparison tables said, but what actually happened after the architecture met production.
That distinction turned out to matter enormously.
My operating rule today is simple:
Buy managed services for their operations, not their defaults — and leave them when their cost floor or cold start starts fighting your workload.
1. SageMaker vs. Bedrock vs. AWS Primitives
The decision at the center of Parjanya
The heart of Parjanya is batch visual inference.
A photographer uploads a session — routinely 4,000 or more images, and sometimes 12,000. Every image that passes a cheap deterministic technical-quality gate is scored by the VLM and embedded using SigLIP 2.
The workload characteristics are important because they tell me almost immediately where the different approaches will fit.
The workload is bursty, not steady.
A queue can go from zero to thousands of images and then return to zero. There can be days when almost nothing arrives.
It is also batch tolerant.
Nobody needs a score in 200 milliseconds. What matters is that 4,000 images are processed by morning.
It is privacy sensitive.
These are often unreleased professional photographs belonging to clients. Keeping image bytes inside our VPC is not merely an architectural preference; it is something I can actually explain to a customer.
And finally, the economics are dominated by GPU time.
Before optimization, GPU compute represented well over half of the entire bill.
That combination — bursty, batch-oriented, private and GPU-heavy — became the single biggest predictor of which architecture made sense.
I started with SageMaker
The obvious managed options were Bedrock and SageMaker.
Bedrock represented the pure developer-experience end of the spectrum: API calls, no infrastructure, no cold-start management and managed models.
SageMaker sat somewhere in the middle: managed infrastructure around our own container and model.
Raw AWS primitives were the opposite end: EC2 GPU instances, Auto Scaling, SQS, SNS, CloudWatch and an AMI with the model baked in.
I used all three lenses, and two of them in anger.
Generation 2 ran on SageMaker.
I started with a BYOC real-time endpoint and then moved to Batch Transform.
SageMaker did exactly what I expected it to do: it got me to a working GPU pipeline quickly.
That is the promise of DX, and in this case, it delivered.
When SageMaker Wasn't the Problem
As a follow-up to the original engineering note on Parjanya v2.0, this deep dive explores the architectural decisions, operational lessons, and production incidents that shaped the migration away from Amazon SageMaker toward a purpose-built AWS-native inference orchestration platform.
Then two things happened.
Quantization changed the economics more than the platform did
Qwen3-VL-8B in BF16 wants roughly 16 GB of VRAM.
A T4 has exactly 16 GB — and does not support BF16.
Moving to NF4 4-bit quantization using bitsandbytes brought VRAM usage down to roughly 5 GB, which made an 8B model practical on a T4.
More importantly, it produced a roughly 13× per-image speedup.
Our batch cost collapsed from approximately $500 to $40–50.
That was my first major lesson from self-hosting:
The model configuration can be a much bigger cost lever than the hosting substrate.
A managed platform doesn’t automatically pull that lever for you.
Then SageMaker’s operational shape started fighting the workload
The first problem was cold starts.
The managed endpoint took on the order of ten minutes to become useful, largely because of model-weight loading.
Our replacement — an AMI/NVMe-staged model running on a raw EC2 GPU worker — loads roughly 17 GB of weights in under two minutes.
For a workload that scales from zero, warmup is latency.
The second problem was the cost floor.
A real-time endpoint costs money every hour it exists, whether a single image arrives or not.
One GPU endpoint represented a standing tax of roughly $500+ per month.
At one point, I discovered a forgotten endpoint costing almost $870/month.
Our worst single day was roughly $1,000, around three to four times our typical monthly development spend at the time. The root cause was endpoint provisioning and extended runs.
The most important early optimization was almost embarrassingly simple: a scheduled shutdown Lambda.
We kept the endpoint warm only during a seven-hour nightly window, which reduced its cost by roughly 70%, from about $540 to $160 per month.
That taught me something I now use as a general diagnostic:
When your best optimization for a managed platform is a cron job that turns the platform off, your workload and the platform’s default posture probably disagree.
The third problem was debuggability.
When something went wrong on primitives, I could inspect a queue, an alarm, an Auto Scaling event and a container log.
Everything was directly readable.
On the managed platform, the same investigation often disappeared behind an abstraction that I could not see through as clearly.
That was ultimately what made the decision for me.
I moved to EC2 Spot
In April, Generation 3 deprecated SageMaker entirely.
I moved to EC2 Spot GPU workers — primarily the g4dn family with NVIDIA T4s — inside an Auto Scaling group driven by SQS queue depth.
The fleet genuinely scales to zero.
The minimum size is zero.
A scheduler Lambda watches the queue every two minutes, and the scale-in rule only fires after both visible and in-flight messages have been zero for five consecutive minutes.
For Spot interruptions, I added a five-second metadata poll.
The worker finishes the current image, checkpoints the state to DynamoDB and exits cleanly.
After months in production, we have lost zero images to Spot interruption.
The result was substantially better than our early expectations.
Our largest autonomous run drained roughly 2,400 images in under eight hours on four Spot GPUs, scaled itself back to zero three minutes after the queue emptied, and required no operator intervention.
The total bill was approximately $20.
Pure GPU inference now costs roughly $0.002–0.005 per image, or around $0.005–0.008 all-in.
Compared with the SageMaker era, that is an order-of-magnitude difference.
Overall platform cost fell by roughly 65–75% from its pre-optimization peak.
At our worst, we were spending around $80–90/day.
Under load today, it is closer to $20–30/day, with an idle floor around $5/day.
Spot itself delivered a measured 56–61% saving against on-demand, roughly $0.26–0.29/hour versus about $0.55/hour for a T4 in Mumbai.
I also ran an A/B test of going 100% Spot.
It beat its own forecast.
Drain time roughly halved, cost per message fell by around 60%, and we experienced zero interruptions.
The written verdict was simply:
Stay 100% Spot.
There was another Spot lesson that was less obvious.
I now treat quota increases as spend risk, not simply performance capacity.
The cost is in instance-hours, not instance count. Increasing a GPU quota to “improve throughput” also increases the maximum possible blast radius on the bill.
I manage the on-demand/Spot quota ratio as deliberately as I manage the fleet itself.
What about Bedrock?
I did not dismiss Bedrock.
I benchmarked it.
A managed vision API came in at roughly $0.01/image.
Our Spot fleet sits around $0.002–0.005/image.
That is a 2–5× gap, but it exists because I am doing three things simultaneously:
scaling to zero
running a quantized model
using Spot capacity
If I ran the same fleet on-demand and kept it always on (warm), Bedrock would win.
I keep that Bedrock number as a standing benchmark.
The day the fleet stops being well run, I want the spreadsheet to tell me so.
There are also two reasons I would keep self-hosting even if the economics reached parity:
Image bytes never leave our VPC.
I control model version pinning completely.
Those are not spreadsheet variables.
What the industry says
The broader industry experience broadly supports this progression.
Discord describes a deliberate strategy of prototyping against managed frontier APIs to validate whether current-generation models can solve the product problem, then moving to self-hosted open models once scale justifies the operational burden. https://discord.com/blog/developing-rapidly-with-generative-ai
Shopify built its Merlin ML platform on open-source primitives rather than committing to a managed ML platform, largely because its teams had conflicting requirements and needed a frictionless prototype-to-production path.
https://shopify.engineering/merlin-shopify-machine-learning-platform
Independent comparisons generally place the self-hosting break-even around 10–20k requests/day, with much larger savings at higher volumes.
And AWS itself frames Bedrock versus SageMaker as a genuine trade-off between serverless convenience and control.
My experience added a third fork:
sometimes the right answer is neither.
Sometimes the right answer is EC2, SQS and an AMI.
What production taught me that comparison tables didn’t
The first lesson was that primitives compose — but not freely.
Our warm-pool design passed architecture review and failed during
terraform apply.
AWS does not allow a warm pool on an Auto Scaling group with a mixed-instances policy.
The DIY tax isn’t necessarily writing more Terraform.
It is discovering the composition rules the hard way.
The second lesson was that design numbers are not production numbers.
Our AMI-bake design promised roughly 90-second model loads.
Fresh instances actually took 30–40 minutes.
Lazy EBS restore meant a new instance reading 17 GB of untouched blocks crawled at single-digit MB/s.
The solution was NVMe staging.
It brought the 17 GB load down to under two minutes — measured in production.
I now put a date and configuration hash next to every performance number in our documentation.
Benchmark rot is a debt class.
The third lesson was about deterministic gates.
Before anything reaches a GPU, a sub-cent Lambda gate rejects technically unusable files.
That rewrite replaced an earlier ML-based scorer, reducing that stage’s cost by roughly 90%, from around $13 to less than $1/month, while also reducing its cold start from about 15 seconds to roughly two seconds.
More importantly, the deterministic rule engine owns the accept/reject decision.
If I change the curation policy, I can replay stored model outputs through the new rules in under a minute for pennies.
I do not need to run the GPU again.
The cheapest inference is the inference I skip. The second cheapest is the inference I never have to redo.
And then there was the prompt.
The prompt is infrastructure.
Our worst GPU incident was a CUDA out-of-memory crash caused not by the model or the fleet, but by prompt creep.
The scoring prompt had quietly grown beyond 4,000 tokens.
Instead of buying a larger GPU at twice the price, I audited the prompt and cut it by roughly 60%.
A lot of the removed instruction was teaching the model to produce outputs that the deterministic rule engine downstream deliberately ignored.
I was literally paying VRAM to generate information the system threw away.
On a 16 GB card, token budgets are a debt ceiling.
2. Vercel vs. S3 + CloudFront
The frontend is a React 19 + TypeScript + Vite SPA.
It contains the gallery, curation workflow, uploads and administration interface, and serves photographers globally while the FastAPI backend runs in Mumbai.
During launch, I needed fast global static delivery, many deployments a day, instant rollback and strict security headers.
The default answer in 2026 is Vercel.
And I understand why.
Git-push deployments, preview environments, an edge network and almost no infrastructure knowledge required.
But I chose S3 + CloudFront.
The deployment pipeline is intentionally boring:
Vite build → S3 sync → CloudFront invalidation
GitHub Actions handles it through OIDC.
Merging to main deploys development.
Production remains a manual, human-triggered workflow.
The cache design does most of the work.
Hashed assets under /assets/* are immutable and carry a one-year TTL.
index.html gets a 60-second TTL.
That makes the cache policy itself the deployment lever.
A release becomes globally visible within about a minute, and rollback is simply redeploying the previous build.
Security headers — HSTS with preload, frame-deny and nosniff — are handled by a CloudFront response-headers policy.
The cost is generally in the single-digit dollars per month and, most months, sits inside the free tier.
The reason I chose it was not that Vercel is bad.
It was that the frontend was not where I needed Vercel’s strengths.
Cached SPA loads from London, New York and Sydney are already measured in tens of milliseconds.
What users actually feel is the dynamic API round trip to Mumbai.
A page making three to five sequential API calls can add more than a second from North America or Europe.
No edge frontend platform can fix that.
The answer is API design: batching, caching and eventually regional read replicas.
That reinforced the thesis I started with:
The edge is a traffic director, not a compute executor.
Static assets and routing belong at the edge.
Authentication, model resolution, inference and telemetry remain centralized.
For a Vite SPA, the DX delta was also surprisingly small.
Vercel’s real advantage appears in SSR, ISR, preview deployments and framework-integrated serverless functions.
I wasn’t using most of those features.
For me, the DX advantage amounted largely to one GitHub Action.
The third consideration was cost topology.
S3 + CloudFront scales predictably with traffic.
Platform function pricing scales with invocations — precisely the axis I hope will explode as Parjanya grows.
I didn’t reject Vercel.
I simply didn’t want to pay a platform premium for capabilities my architecture wasn’t using.
3. OpenRouter vs. a Self-Hosted Gateway
Every AI product eventually has to answer four questions:
Which model serves this request?
What happens when it becomes unavailable?
Who tracks the spend?
Whose infrastructure sees the payload?
Managed gateways such as OpenRouter answer these questions across hundreds of hosted models.
One API key.
Automatic fallback.
Unified billing.
For Parjanya, however, there was a twist.
Our production models are self-hosted open weights:
Qwen3-VL-8B
SigLIP 2
They run on our own GPUs.
There is no third-party inference endpoint sitting in the hot path.
So I asked a slightly different question:
What plays the role of the gateway when the models are mine?
I ended up decomposing the gateway into primitives.
Model resolution is a three-tier loading ladder on the GPU worker.
Tier 1: weights staged on local NVMe from the AMI bake.
Tier 2: S3 synchronization fallback.
Tier 3: Hugging Face download.
Every load records the tier.
In healthy production, I should only ever see Tier 1.
A Tier 2 or Tier 3 log line is therefore an alert.
I also made the upstream dependency deliberately boring.
Hugging Face is contacted exactly once per model version, during AMI baking in CI.
Production instances never download models from it.
A scheduled job compares the upstream model commit hash with our baked AMI tags and tells a human when a rebake is worth considering.
That gives me something I care about enormously in an inference system:
model versions are pinned by construction.
Fallback is handled by the mixed-instances policy across GPU families and the loading ladder.
Spend accounting comes from CloudWatch and tagged infrastructure.
Every inference dollar can be attributed to an EC2 line item.
For our particular architecture, a managed gateway would introduce three structural costs:
A per-request economic layer.
A data-path layer where client payloads transit someone else’s infrastructure.
A rate-limit layer where our ceiling becomes somebody else’s policy.
OpenRouter’s published model, for example, includes roughly a 5% platform fee on provider pass-through pricing.
For a self-hosted, single-model, VPC-contained workload, I don’t get enough routing benefit to justify those layers.
But there is an important concession here.
If I were calling frontier hosted models, I would absolutely use a gateway.
And I would probably use a managed one first.
Standing up LiteLLM with a database, cache and pager before I have meaningful multi-provider traffic is premature infrastructure.
The gateway decision comes after the hosting decision.
It is not independent of it.
The lesson from our fallback
A fallback I have never exercised is not a fallback.
For a while, our Tier 2 S3 fallback was configuration theater.
The environment variables existed in the design.
They were never actually injected into the container.
The fallback existed on paper and nowhere else.
I fixed it by deliberately breaking Tier 1 and watching what happened.
That is one of the biggest operational differences between managed and primitive systems:
if I decompose a managed capability, I inherit the obligation to rehearse every part of it.
4. Clerk vs. Cognito + SES
Authentication was the closest call.
Parjanya needs multi-tenant signup, email/password authentication, JWTs carrying an immutable tenant identifier, trial limits and transactional email for verification and password resets.
Clerk is extremely attractive here.
Polished UI.
Session management.
A much nicer developer experience.
And, critically, email that simply works.
The alternative was Cognito + SES.
I chose Cognito + SES.
Why
First, economics.
At our current scale, Cognito costs us essentially nothing under its applicable MAU threshold and remains materially cheaper through the growth range we are targeting.
For a product whose unit economics depend on very high gross margins on per-image pricing, a permanent per-MAU platform fee is a margin haircut on every future user.
Second, and more importantly:
the token is the tenancy model.
Our JWTs carry an immutable tenant_id claim.
The backend verifies it using RS256 against Cognito’s JWKS, deliberately avoiding a signing secret in the backend.
Every S3 bucket policy and DynamoDB access pattern keys off that claim.
The backend issues short-lived, tag-scoped STS credentials per tenant.
Buckets are per-tenant.
KMS keys are per-tenant.
Cross-tenant access is explicitly denied at the policy layer.
Auth is therefore not just a login box.
It is the root of the isolation tree.
I wanted that root inside the same Terraform, account boundary and audit trail as everything it protects.
A hosted authentication provider can issue my token.
But it cannot itself be the IAM principal that my storage policies deny against.
Third, there is the privacy argument.
One bill.
One IAM boundary.
No additional identity-data processor.
For a privacy-positioned product, that is a sales answer as much as it is an architecture answer.
And then DIY sent me the bill
This is where I think infrastructure writing is often too polite.
The DIY option has a cost.
I want to list mine.
Cognito immutable attributes really are immutable.
I verified that in production.
Get your custom claims right the first time.
Schema changes can also be pool-destroying.
Adding a new schema attribute through Terraform can force user-pool replacement and therefore destroy every user account.
I added a role claim through the CLI instead, knowingly accepting permanent drift for that resource.
Cognito also does not support cross-account user-pool migration.
When we reorganized into a multi-account AWS organization, the pool’s location became a one-way door.
Either users would re-register or we would face a mass password reset.
Then there were session bugs.
Our frontend token cache survived logout.
User two could briefly see user one’s data until I added an explicit cache clear at every login boundary.
Presigned upload URLs had another subtle problem.
They inherit the STS credential’s expiry rather than the nominal URL lifetime.
An upload presigned at minute zero failed at minute 61 with a signature that looked valid.
These are precisely the integration bugs that a mature hosted authentication SDK has probably already encountered.
And then there was email.
Cognito’s default email channel has a 50-email/day per-pool cap and is best effort.
During launch, password-reset emails were silently dropped.
The API returned success.
CloudTrail recorded the event.
No email arrived.
That is one of the worst failure modes in distributed systems:
The lying green checkmark.
The fix was SES with a verified domain, DKIM and developer sending enabled through Cognito.
That led into SES production-access review.
Our first request was denied, almost certainly because our submission said too little about bounce handling.
SES sandbox status became the sole external blocker on our release lock.
We eventually documented the ordering constraint in red:
auto-confirm cannot be switched off while SES is sandboxed, or every signup breaks.
This is exactly where Clerk’s pitch becomes compelling.
You don’t have to think about any of this.
And during launch week, that matters.
DIY auth is paid for in calendar time and attention.
Those were the two currencies I had the least of during launch.
My verdict
Cognito + SES was right for Parjanya.
But it was the closest decision of the four.
The SES saga is actually the strongest argument for DX in this entire post.
A managed platform’s ability to make an entire class of launch-week failures impossible has real value.
If authentication were not the root of our isolation model, I would have been much more comfortable paying a per-MAU platform fee at seed scale.
5. The Decisions That Don’t Make Blog Titles
Some of the decisions that affected the bill the most had nothing to do with DX platforms.
Spot — and learning when a lever isn’t a lever
The Spot decision gets most of the attention because the savings are obvious.
We measured roughly 56–61% savings versus on-demand.
Our A/B test of 100% Spot beat its forecast. There are additional optimisations like cost-performance optimisation by AWS, which I will detail in upcoming posts.
Interruption handling has lost zero images.
We even measured Spot pricing within the region and found that availability-zone prices for the same instance type could differ by more than 50%.
So the fleet biases toward the cheaper zone.
But some fashionable optimizations simply failed the arithmetic.
Multi-region Spot arbitrage died because inter-region data transfer consumed most of the projected savings.
A neighboring region also turned out not to have GPU capacity to arbitrage in the first place.
The more interesting story was Graviton.
Our first Graviton audit found that only one of four Lambda functions could even run on ARM64.
Native dependency wheels, CUDA and unbenchmarked inference libraries blocked the rest.
The projected saving was approximately 0.4% of the total bill — about a quarter of a dollar per month.
Meanwhile, quantization, Spot and scheduling were saving 40–60%.
So I wrote down the conclusion and walked away:
Sometimes the best architecture decision is knowing where not to spend your engineering time.
Months later, a dependency shipped ARM64 wheels.
The blocker disappeared.
We revisited the decision and migrated.
The measured Lambda saving was approximately 20%, almost exactly what we had projected.
Both halves of the story matter.
I needed the discipline to defer a fashionable optimization.
But I also needed the paper trail that allowed me to revisit it later without starting the research from scratch.
My rule now is simple:
A two-hour ARM benchmark costs pennies and beats two weeks of debugging a premature migration.
And I never flip Terraform to ARM64 before the image has actually run there.
6. OpenSearch and Step Functions
Two other managed services lost to arithmetic.
Search
I kept search in-process.
No OpenSearch.
No external vector database.
At our scale, the smallest useful OpenSearch domain costs more per month than our entire reconciliation control plane costs per year.
Rather than guessing, I measured the ceiling.
Today, search latency is around two seconds at a few thousand rows and degrades toward the load balancer timeout somewhere beyond 100k rows.
So I wrote down the trigger:
roughly 30 tenants.
That is when OpenSearch becomes a yes.
SigLIP 2 vectors are already ready for a proper vector store when that trigger fires.
Orchestration
I also rejected Step Functions and workflow engines.
Our orchestration is a roughly 200-line reconciler.
DynamoDB Streams provide the fast path.
An hourly sweep provides the backstop.
The entire control plane costs a few cents a month.
I made the same arithmetic-driven decision around AWS Config and commercial compliance tooling.
A weekly EventBridge + Lambda audit covers the rules we actually need at effectively zero cost.
There is a famous industry parallel here: Amazon’s Prime Video team moved a monitoring service away from Step Functions and Lambda to a plain ECS monolith and reported roughly 90% lower infrastructure cost.
But DIY orchestration has its own failure modes.
Our replay Lambda once lost a single IAM permission and logged AccessDenied on every invocation for half a day.
The dashboards stayed green.
Hundreds of images stalled.
Nothing drained.
Then, on our launch-baseline day, the reconciler itself became the incident.
A read-amplification bug caused it to consume tens of millions of DynamoDB read units in 24 hours.
The bill was roughly $15/day.
The pipeline itself was almost entirely healthy.
The repair mechanism had become the damage.
I added targeted query guards and reduced that spend by roughly 85–90%.
Both incidents ended as code.
An alarm on silence.
A conditional-write guard.
A query guard.
That is an important property of primitives:
Institutional memory can become code.
7. Where the Bill Actually Hides
Three numbers changed how I think about infrastructure.
The first was CloudWatch.
Ungoverned CloudWatch can quietly consume 20–40% of a small platform’s operating cost.
A single chatty DEBUG-level Lambda can generate roughly $20+/month in log ingestion — more than our entire early operating budget for that stage.
Embedded Metric Format instead of per-call metric APIs saved around $15/month by itself.
Composite alarms reduced alarm costs by roughly 70%.
I also stopped creating per-tenant dashboards.
At $3/dashboard/month, that becomes a fixed cost that scales in exactly the wrong direction.
With those changes, observability fell back to a few dollars per month.
The second number was networking.
While investigating that roughly $20 autonomous GPU drain run, I discovered that almost half the bill wasn’t GPU time.
It was NAT gateway egress from around 180 GB of S3 reads.
The GPU wasn’t the expensive part.
The data path was.
VPC gateway endpoints for S3 and DynamoDB eliminated most of that cost and had already saved roughly $30–45/month of standing NAT expense.
A handful of interface endpoints cut the remaining NAT data by another ~80%.
That was another reminder that no SageMaker-vs-EC2 comparison table is going to tell you that your data path may cost more than your compute.
The third number was storage.
At photography scale, storage-class economics matter.
I rejected S3 Intelligent-Tiering for previews because its per-1,000-object monitoring fee becomes meaningful when you have millions of tiny files.
Instead, previews go directly into Infrequent Access, around 46% cheaper than Standard in our region.
Originals step down to Glacier Instant Retrieval, around 83% cheaper.
Rejected images move to Deep Archive after a grace period.
At 10 TB of rejected images, that’s roughly $10/month.
The result is about 3.7× cheaper than naïvely keeping everything in Standard.
The broader industry has demonstrated this at enormous scale.
Canva, for example, moved tens of billions of objects to Glacier Instant Retrieval after measuring that access collapses after the first couple of weeks, saving millions annually.
The meta-lesson for me was even simpler.
Our weekly audit Lambda costs effectively nothing.
It caught roughly 7,500 objects sitting in the wrong storage class for months.
It also found an S3 versioning configuration where the console suggested around 2,000 objects while the bucket actually contained more than 100,000 hidden version objects.
Left unchecked, that could have grown into millions of noncurrent versions, terabytes of invisible storage and a 5–10× inflated storage bill.
On primitives, audit is the platform.
8. The Incident Ledger
A fair DIY-versus-DX discussion has to include the incident ledger.
The operational burden that managed platforms sell against is real.
Between the first sprint and the post-launch study window, I logged 17 incidents across 12 classes.
Here are a few.
Terraform drift nearly became a self-inflicted outage.
After weeks of console-side firefighting, a routine Terraform plan came back with:
173 to add.
127 to destroy.
A blind apply would have deleted the NAT gateway, the DynamoDB table containing every VLM verdict and the Cognito user pool containing every login.
Getting back to a clean 0/0/0 state took roughly four focused hours spread across four days.
A managed DX platform wouldn’t have given me that state file.
But it also wouldn’t have given me a state file proving exactly what production was.
There was a six-hour preview outage caused by deleting the wrong “stale” grant.
What looked like a leftover origin-access policy was actually the live one.
Every preview returned 403 until I restored it.
There was a burst-photography incident caused by a silently failing install.
A shell || echo swallowed both a 404 and a missing archive.
We shipped a worker image without its EXIF tool.
Near-identical burst frames then started being falsely rejected as duplicates.
Launch went out with exact-match-only deduplication as the safe setting.
There was a wrong-bucket incident where hundreds of images hung on repeated download errors because queue payloads contained a valid key pointing to the wrong bucket.
And there was a weekend bulk import where messages silently expired from SQS because queue retention was shorter than the weekend.
Retention is now an invariant with a name.
Not a default.
These incidents sound like arguments against primitives.
I actually think they demonstrate the opposite.
Every incident ended as code:
a Terraform validation block
a conditional write
a CI grep
a CloudWatch alarm
a queue invariant
Institutional memory became executable.
And across all 17 incidents, recurrence after adding the guardrails is currently zero.
That is the bargain I am making with primitives.
I accept the initial operational burden in exchange for owning the system completely enough to turn every lesson into a permanent constraint.
The multi-account reorganization was another example.
It was triggered partly by a deployment script accidentally pushing to production from the wrong local context.
The reorganization added roughly $10/month in duplicated audit tooling and moved around 135 GB of data for pennies.
In return, it dramatically reduced blast radius.
That kind of control is difficult to purchase from a single-tenant DX platform’s pricing page.
One uncomfortable AI footnote
Parjanya was substantially AI-assisted in its construction.
Two of the nastiest silent failures in the incident ledger — the swallowed exiftool installation and a bare-except registry writer — were both machine-written and machine-reviewed code.
Their common property was silence.
The answer wasn’t to use less automation.
It was to write better contracts:
CI greps for exception-swallowing patterns
log-line contracts for every pipeline tier
alarms on absence
explicit invariants
The same discipline that makes primitives operable also makes AI-written code operable.
9. The Repatriation Debate, From the Cheap Seats
All of this sits inside the much larger cloud-versus-hardware debate.
37signals has made the case for leaving the cloud, with a reported reduction from roughly $3.2M to $1.3M annually and projected savings of more than $10M over five years.
Ahrefs has argued that owned hardware saved them hundreds of millions compared with AWS list pricing.
Canva has taken almost the opposite position: stay in the cloud, but become extremely good at engineering the bill.
At Parjanya’s scale, I’m much closer to Canva’s school.
But I think there is an important distinction.
I haven’t left the cloud.
I also haven’t left managed services.
DynamoDB, SQS, Lambda, CloudFront and Cognito are still managed services in our architecture.
I keep them because I want their operations.
What I left were the managed services whose cost floors and cold starts fought the workload.
A real-time inference endpoint billing through the night for a queue that is empty by midnight isn’t an operations benefit.
It is a subscription to someone else’s default posture.
That is the distinction I think the broader repatriation debate sometimes misses.
It isn’t really:
cloud vs. metal
or:
platform vs. primitives
It is:
What is the right economic and operational shape for each individual workload?
And that answer should be re-run when the workload changes.
10. The Decision Framework I Use Now
After five months and four architecture generations, I have reduced the decision process to four questions.
1. Is the expensive part of the hot path bursty?
If yes, the platform’s cost floor may be your enemy.
Scale-to-zero primitives are probably worth the setup cost.
If the load is steady, the cost floor becomes less relevant and the managed platform’s operational value may be worth buying.
That one question explains why I left SageMaker, benchmarked and declined Bedrock, declined Vercel and kept DynamoDB on-demand.
2. When it breaks at 2 a.m., can I read the failure surface?
This ultimately mattered more to me than cost.
A queue.
An alarm.
An Auto Scaling event.
A container log.
These are things I can read.
If my team cannot or does not want to own that failure surface, the managed abstraction is doing real work for us.
I should pay for it.
3. What currency does the DIY option cost — and do I have that currency right now?
Cognito + SES cost me launch-week calendar time.
That was the scarcest resource I had.
The exact same decision at another point in the company’s life would have cost almost nothing.
So I no longer think only in dollars.
I ask:
What resource am I actually short of?
Money?
Engineering hours?
Attention?
Launch-week calendar time?
Operational confidence?
The right architecture can change depending on the answer.
4. Have I written down the condition that would reverse my decision?
This is probably the most useful practice I have adopted.
A “no” is not a permanent architectural belief.
It is a decision with a trigger.
For example:
vLLM becomes a yes around 500 active users or sustained 100+ images/minute.
OpenSearch becomes a yes around 30 tenants.
The monorepo decision changes with team growth.
Every rejected option gets an un-rejection trigger.
That turns architecture from ideology into a portfolio that I can rebalance.
The Operating Sentence
The note that started this journey argued that high-performing platforms should choose DX where it helps, and control where it matters.
After a launch, four architecture generations, seventeen logged incidents and a roughly two-thirds reduction in our platform bill, I would sharpen that sentence.
This is the rule I now keep at the top of our architecture decision register:
Buy managed services for their operations, not their defaults. Leave them when their cost floor or cold start fights your workload. And put deterministic gates around expensive models — the cheapest inference is the inference you skip.
Parjanya v2.0 is now live as a self-serve trial.
The production system runs Qwen3-VL-8B in NF4 and SigLIP 2 So400M on scale-to-zero Spot GPU workers in ap-south-1.
Everything sits behind a deterministic rule engine that owns every accept/reject decision.
The model observes.
The system decides.
And, perhaps more importantly, I now have a much clearer idea of when I want a platform to make the hard parts disappear — and when I would rather own the hard parts myself.
