Preface

The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay’s total UPI volume.

Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure.

This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on.

The First Version: Monolith and SQS

When we began building the UPI Switch, we weren’t thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately.

For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn’t need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL.

The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows:

1. Update NPCI with an API call
2. Send callbacks to merchants about payment status
3. Push structured data into our warehouse (AWS Redshift)

To handle this, we started bolting on AWS SNS plus SQS for fan-out. It worked technically, but it felt improvised. The fan-out was a chain of pipes we kept extending as new requirements surfaced. We were still delivering and nothing was on fire, but the architecture was growing sideways rather than upward, and every addition came with operational drag.

Moving to Event-driven Microservices

The next target was bold: 10,000 TPS. To get there, we had to rethink the fundamentals. We redesigned the system into Switch v2, moving from a monolith to an event-driven microservices architecture. We split the Switch into independently scalable components: customer and merchant services, VPA, payments, mandates, and others. Each service owned a specific domain and could scale horizontally without impacting the rest.

AWS SQS Standard had served us well, but it lacked two capabilities we now needed: true pub-sub behavior, and high-throughput with low-latency delivery. We wanted a system built for scale, and the answer pointed clearly toward Kafka. So we moved from AWS SQS to AWS MSK (Managed Kafka), which felt like the right fit for the new architecture.

The Decisions We Made

As Switch v2 took shape, we faced a series of architectural decisions. Each seemed reasonable, even elegant, at the time. Many of them would return later to cause real problems. The promises were genuine; the costs came later.

1. Choosing GoCloud Pub/Sub. We wanted to stay vendor-agnostic and avoid locking ourselves into Kafka or MSK. So instead of integrating with Kafka directly through a native client, we picked GoCloud Pub/Sub (gocloud.dev/pubsub), which gave us a common abstraction over multiple backends like SQS, Kafka, and NATS. The thinking was simple: if we ever wanted to move off Kafka, it should be easy.
2. MSK Connector and the outbox promise. We wanted atomicity between persisting a new payment and publishing the corresponding event, because writing to the DB but failing to publish the event would leave the system inconsistent. To solve this, we leaned on the Outbox Pattern, where the application writes the business data (such as a payment) and the event to an outbox table within the same database transaction, and an external component (like AWS MSK Connector) reads from that table and publishes to Kafka.
3. Betting on AWS MSK Serverless. Provisioning and scaling a Kafka cluster felt like unnecessary overhead. We wanted autoscaling with no broker-sizing decisions, support for unpredictable and elastic workloads, and pay-as-you-go pricing. So we picked AWS MSK Serverless over provisioned MSK clusters.
4. DDD-inspired event modeling. We used event-storming (a collaborative workshop technique for mapping the events that occur across a business domain) along with domain-driven design (DDD) to break down the problem space. Each bounded context emitted its own events, and to keep domains cleanly separated, we created a Kafka topic for every event type.

Individually, each decision seemed justified. Together, they shaped our architecture, sometimes for the better and sometimes in ways we would spend months undoing.

Where It Broke at Scale

1. Topic explosion: event-storming gone too far. We embraced DDD and event-storming seriously. Every domain emitted its own events, and every event type received its own Kafka topic. Then came retry topics and DLQ topics. On top of that, we segregated traffic per product line (Acquiring, TPAP, Turbo) to ensure isolation.

Total partitions ≈ (#topics) × (replication factor) × (retry + DLQ topics) × (#product lines)

The partition count ballooned quickly. Although Kafka is “infinitely scalable” in theory, each broker has a practical limit on how many partitions it can host. Every partition consumes CPU and memory for metadata and file handles, replica sync and network I/O, and thread-scheduling overhead. As partition counts grow, brokers spend more time on bookkeeping (ISR management, replica fetch, leader election) than on actually moving data. This drives CPU spikes and throttling long before throughput becomes the bottleneck. In practice, scalability was limited not by throughput but by partitions per broker.

2. MSK Connector: routing that didn’t scale. To achieve DB and Kafka atomicity, we wrote business records and events to a single database table, and MSK Connector would read the events table and publish to Kafka. However, MSK Connect’s source connectors read from a source system and write to a single configured topic; they don’t support multi-topic routing from a single source table. We tried to solve this by extending the connector with custom routing logic, dynamically pushing messages to different Kafka topics based on metadata inside the event row. This did not work well in practice, and the connector processes crashed frequently.

3. MSK Serverless instability and faulty client behavior. We selected MSK Serverless to avoid operational overhead, but it introduced critical limitations.

• We observed severe connection imbalance. Out of three brokers, one consistently accumulated a disproportionately high number of TCP connections and significantly higher CPU utilization. This hotspot contributed to instability and throttling under load.
• Unlike AWS Aurora RDS, which exposes a stable cluster endpoint and transparently handles node restarts, MSK Serverless only provides individual broker addresses, which means clients must handle broker restarts themselves.
• MSK performs scheduled monthly maintenance during which brokers are restarted. Our consumers failed to reconnect to Kafka automatically during these periods, and recovery required manually restarting application pods.
• The root cause was that gocloud.dev/pubsub did not re-resolve DNS on reconnect. It cached broker IPs at startup and kept using those stale addresses after maintenance, even when brokers came back up with new IPs. As a result, clients failed to reconnect until pods were restarted. This broke self-healing and caused rolling broker restarts to cascade into application outages, undermining Kafka’s high-availability guarantees.

4. The abstraction introduced by the GoCloud Pub/Sub layer also limited our ability to configure Kafka-level client parameters that could have improved connection management and distribution, which prevented us from mitigating the imbalance at the client layer.

5. IAM auth connection cap: the Asia Cup incident. Ahead of the Asia Cup 2025 surge, we scaled out our pods assuming we could absorb the expected traffic. We were using MSK IAM authentication, which enforces a limit of roughly 3,000 active TCP connections per broker. Unaware of this constraint, and combined with GoCloud Pub/Sub’s tendency to create excessive connections, we saturated the connection quota almost immediately.

6. How we fixed it

1. Migrating from GoCloud Pub/Sub to franz-go. The first and most important change was replacing gocloud.dev/pubsub with franz-go, a Kafka-native Go client. franz-go gave us full Kafka protocol support, DNS re-resolution on reconnect, efficient connection handling, better batching and compression support, fine-grained control over producers and consumers, and better instrumentation and metrics exposure.
2. The impact was immediate. DNS was re-resolved correctly, so clients auto-recovered after broker restarts. Broker load became evenly distributed across CPU and connection count. Connection count dropped from ~1,100 to ~300 per broker in production [VERIFY], and publish latency fell from ~250 ms to ~10 ms [VERIFY]. The client-library swap alone neutralized most of our critical pain points.
3. Consumer-side batch processing with worker pools. Partition count had become a limiting factor, since partitions define the upper bound of concurrency, and more partitions meant more broker overhead. To break this coupling, we introduced worker pools per consumer: messages are fetched sequentially per partition, processing is parallelized using goroutine workers, and a batch offset commit is issued after the workers finish. This preserved per-partition ordering guarantees while improving compute utilization.
4. Reconnection and retry handling. Even though franz-go handles consumer and producer reconnection internally, we added guard rails with additional retry and reconnect logic around consumer polling for defensive recovery. This made the consumer layer resilient to flap events, network jitter, and rolling restarts. The combination of native recovery and defensive logic made failures non-catastrophic and eliminated the need for manual pod restarts.
5. TCP pooling via a custom TLS dialer. Unlike SQL clients, Kafka clients usually do not provide built-in TCP connection pooling, and the default behavior often leads to excessive, short-lived TCP connections. We introduced a custom TLS dialer, a controlled TCP connection pool, and reuse of existing network connections.
6. Topic consolidation and infra isolation. The one-to-one mapping between event types and Kafka topics was the root cause of our partition explosion. We redesigned the event model: instead of a dedicated topic per event type, we grouped related events from the same domain into a single Kafka topic. For example, all events associated with the VPA and Bank Account modules (registrations, validations, deletions, updates) now go to just two Kafka topics, and consumer-side filtering on event-type metadata handles the differentiation within each topic.
7. Reducing topics alone is not enough if high-traffic flows can starve latency-sensitive ones. VPA validation, for instance, is a resource-intensive API call with external service latency of up to 15 seconds. Grouping it with financial flows like payments would let it monopolize partitions and consumers, with cascading effects on payment throughput. To address this, we split our gateway NPCI module (the service responsible for all egress to NPCI) into separate, independently deployable components: VPA_GwNPCI, Payments_GwNPCI, and Mandates_GwNPCI. Each component has its own dedicated Kafka topics, consumer groups, and compute resources. Non-financial flows like VPA validation and financial flows like payment processing are now isolated at the infrastructure level and cannot interfere with each other’s throughput or latency budgets.
8. Migrating to MSK Provisioned with SASL/SCRAM auth. Finally, we moved from MSK Serverless with IAM auth to MSK Provisioned with SASL/SCRAM, which does not impose the same strict limits on connection counts.

Individually, each optimization solved a specific issue: connection churn, uneven broker load, or limited concurrency. Together, they shifted us from reacting to failures to reliably operating at scale. Kafka stopped feeling like a constant negotiation and started acting like the backbone we needed.

The Benchmarks After the Rebuild

We ran extensive performance and load tests to validate the improvements, iterating on configuration tuning across producers, consumers, batching parameters, and network settings to find the right balance of throughput and stability. We benchmarked several flows individually and also ran mixed-load scenarios to simulate real production traffic patterns:

1. Intent payments
2. Mandate creation
3. VPA validation
4. Online refunds

Performance highlights:

These numbers showed that we had regained performance headroom in both throughput and broker capacity. The cluster was stable, predictable, and responsive even under mixed workloads.

What We Would Say to Other Teams Building This

A few lessons generalize beyond our setup.

1. Vendor-agnostic abstractions can hide the things you actually need to configure. GoCloud Pub/Sub gave us portability on paper, but it also hid Kafka-level client controls (DNS re-resolution, connection management, batching) that turned out to be exactly what we needed at scale. A thin abstraction is fine until the layer you abstracted away is the layer where your production problems live.
2. One topic per event type is rarely the right granularity. Modeling each event type as its own topic looks clean in a domain diagram, but it pushes partition counts past what brokers can handle. Group related events by domain and filter on the consumer side instead.
3. Connection management on async messaging clients is where scale problems hide. Throughput numbers get all the attention, but the failures that took us down were about connections: imbalance across brokers, stale DNS after restarts, and hard caps on connection counts. Treat connection behavior as a first-class concern when you pick and configure a client.

What’s Next

Even after the major architectural improvements and performance wins, some areas still need attention. These aren’t failures; they’re the natural next steps in maturing the platform.

1. Idempotency on consumers. Kafka can replay messages during consumer restarts, partition rebalances, offset resets, and retention recovery. To guarantee correct processing in all cases, we need idempotent consumption, ensuring the same event produces the same effect only once even if received multiple times. Our approach: assign a durable idempotency key to every message at publish time (for example, a composite of payment ID and event type). On the consumer side, before processing any message, check a deduplication store (Redis or DB) for the key. If it has already been processed, skip it; if not, process it and record the key. This makes consumer behavior safe under any replay scenario.
2. Atomic writes (post MSK Connector). After removing the MSK Connector, we lost DB and Kafka atomicity. Today, writes to the DB and publishes to Kafka are separate operations, which creates windowed failure cases: a DB write succeeds but the Kafka publish fails, or a Kafka publish succeeds but the DB write rolls back.
3. Retries and DLQs. We need a consistent error-handling framework across consumers. Consumers should decide which errors are retryable, define backoff behavior (fixed, exponential, max attempts), and determine when a message should move to a DLQ instead of being retried indefinitely. A structured retry and DLQ policy will prevent infinite retry loops, isolate poison messages, and make failure handling predictable.

Editor: Parth Sawhney

Five Years of Kafka at Razorpay’s UPI Switch was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Contributor: Saijal Shrivastava

We cut our metrics ingestion by 62% without deleting a single useful dashboard or breaking an alert.

Daily ingestion dropped from about 450 billion samples to about 170 billion. The surprising part was not one large optimization. It was three simple discoveries hiding in plain sight:

1. Scrape intervals can multiply ingestion.
2. High-availability scraping can silently send duplicate metrics when deduplication happens only at the storage layer.
3. High-cardinality metrics are easy to create, expensive to keep, and hard to remove without ownership.

We found these while preparing to move from a self-hosted VictoriaMetrics cluster to a managed monitoring platform. The migration started as a reliability project. The audit turned it into a cost and signal-quality project. We were not just moving monitoring data. We were moving years of accumulated assumptions about what was worth scraping.

The Incident that Forced the Audit

Our self-hosted VictoriaMetrics cluster had served us well for years. Then one storage node hit EBS volume throttling. Traffic shifted to the remaining storage nodes, the extra load pushed another node past its limits, and the storage layer collapsed. We saw the same pattern whenever a storage node became unavailable: traffic redistributed, the remaining nodes overloaded, and the cluster moved back toward failure.

The collection, ingestion, and query components were still running, so the system looked alive from the outside. But the storage layer was down. Metrics stopped being persisted. Dashboards went blank. Alerts stopped evaluating. Recovery took hours. Teams fell back to logs and manual checks to understand production health.

That made the reliability problem obvious. We needed a more stable monitoring architecture. We evaluated several options: self-hosted designs with disaster recovery built in, and fully managed alternatives. But when we priced the managed options, one number changed the project: at roughly 400 billion daily samples, the estimates ran into hundreds of thousands of dollars per month. At that scale, every duplicate scrape, every unnecessary 30-second interval, and every unbounded label had a direct cost. So before migrating, we audited what we were actually shipping.

Discovery 1: Scrape Interval is a Multiplier

The easiest problem to miss was scrape interval.

Our infrastructure metrics were scraped every 30 seconds. Application metrics were scraped every 60 seconds. That meant infra agents produced twice as many samples per metric before we even looked at metric count or label cardinality.

A 30-second scrape interval sounds harmless when applied to one target. At fleet scale, it becomes a multiplier across every node, pod, container, and exported metric. If the data is not used at that resolution, the extra samples add cost without improving monitoring.

We audited alert rules and dashboards to check whether anything depended on sub-minute resolution. Nothing did. Capacity planning, node health alerting, and resource utilization were all fine at one-minute resolution.

So we standardized infra scraping to 60 seconds.

This one change halved infra metric ingestion -

• No dashboards went dark.
• No alerts broke.

The data became cheaper without becoming less useful.The lesson was simple: scrape interval is not a default. It is a cost and fidelity decision.

Discovery 2: High-availability Scraping can Hide Duplicate Ingestion

The biggest discovery was duplicate ingestion.

Our application metrics used high-availability vmagent pairs: two replicas per business unit, both scraping the same targets. This is a common pattern. It protects monitoring from agent failure, and with self-hosted VictoriaMetrics it appears to work cleanly because VictoriaMetrics deduplicates identical samples at the storage layer.

That storage-layer detail matters.

Deduplication at storage means the duplicates are hidden from queries. Dashboards look correct. Alerts behave correctly. Nobody looking at Grafana sees the duplication.

But the duplicate samples still travel through the monitoring pipeline. Both vmagent replicas scrape the same targets. Both push the same samples. The system spends compute, network, and ingestion capacity on both copies.

In other words, storage deduplication protected correctness, not cost.

At our scale, that distinction mattered. Hundreds of billions of application samples per day were being scraped twice and pushed twice.

We could have switched scraping tools, but that would have meant changing service discovery, relabeling pipelines, and scrape configuration across teams. That would have turned a monitoring migration into a large coordination project.

We wanted three things at once:

• keep vmagent,
• preserve high availability,
• ensure only one replica scraped at a time.

VictoriaMetrics did not provide agent-level active-standby scraping, so we built it: one agent scrapes while the other waits.

Making vmagent active-standby

We added a leader-election sidecar to each vmagent pod. Both pods stay running, but only the leader scrapes. The follower stays ready to take over.

The design has three parts:

vmagent reads its scrape configuration through a symlink. The sidecar controls where that symlink points.

When the pod is leader, the symlink points to the real scrape config. When the pod is follower, it points to the dummy config. On leadership changes, the sidecar updates the symlink and sends SIGHUP to vmagent, which reloads config without a container restart.

The dangerous failure mode is split brain: both pods believing they are leader and scraping the same targets. That would recreate the duplication we were trying to remove.

So the system is biased toward stopping scrapes rather than risking duplicates:

• If the sidecar cannot reach the Kubernetes API, it assumes it is not leader and switches to the dummy config.
• Two failed lease renewals trigger immediate self-demotion.
• A lease expires after 15 seconds without renewal.
• One failed renewal is tolerated as a grace period, but not two.

In graceful shutdown tests, failover took about seven seconds. In hard-crash tests, it took about 22 seconds. In network partition tests, the old leader demoted itself before the follower took over, preventing duplicate scrapes. The sidecar has now run in production for more than six months with no split-brain incidents.

Discovery 3: High-cardinality Metrics Need an Ownership Model

The third problem was not duplication. It was noise.

We analyzed the top 500 highest-cardinality metrics against actual usage in alerts and dashboards over 90 days. We found 167 metrics that were not used anywhere: not in an alert, not on a dashboard, and not in observed queries.

They were still being scraped, shipped, stored, and paid for.

High-cardinality metrics are especially dangerous because they often look reasonable at creation time. A label such as url, uri, user_id, merchant_id, or raw request path can multiply one metric into millions of time series. The exporter still works. The dashboard may still load. But the monitoring system now carries a large amount of data that few people understand and even fewer people use.

Without an effective strategy, high-cardinality metrics become permanent. Teams are afraid to remove them because someone might depend on them. Platform teams are afraid to drop them because ownership is unclear. So the safest local decision is to keep everything, and the global result is an expensive monitoring system full of unused series.

We dropped unused metrics at the vmagent layer through relabeling rules. That gave us one reversible control point instead of waiting for every application team to release code, and removed about 50 billion samples per day.

A high-cardinality metric is not automatically bad, but it must justify its cost. If nobody uses it, owns it, or knows why it exists, it should not be in the default monitoring path.

The Result Post Migration

The most valuable outcome was not just the lower bill. The monitoring system became easier to reason about.

We knew which metrics were duplicated. We knew which scrape intervals were intentional. We knew which high-cardinality metrics were justified. And we had a cleaner path for future migrations because the proxy absorbed long-tail consumers.

What We Would Tell Other Teams

Audit your systems regularly. A migration makes cost visible, but the waste usually existed long before the migration started.

Treat scrape interval as a multiplier. If a metric does not need 30-second resolution, every extra scrape is cost without signal.

Do not assume high-availability scraping is free. Storage-level deduplication may make dashboards correct, but it does not remove the compute, network, or ingestion cost of duplicate samples.

Give high-cardinality metrics an ownership model. If a metric creates millions of series, someone should know why it exists, where it is used, and when it can be removed.

Optimize at the edge when possible. Dropping unused metrics and preventing duplicate scrapes at vmagent reduced ingestion before data entered the rest of the monitoring pipeline.

We started with a reliability incident. We ended with a monitoring system that was more stable, easier to operate, and 62% cheaper to run. The biggest win was not moving data to a new platform. It was learning which data deserved to move at all.

Editor: Parth Sawhney

How Razorpay Cut Its Metrics Bill by 62% Without Losing a Dashboard was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Contributors: Guptaanuj

CI/CD pipelines have always had a money-vs-stability problem. Run on-demand AWS instances and your build infrastructure is rock solid, expensive, and predictable. Run on spot instances and your costs drop 70–90%, but AWS can pull the rug with 2 minutes of warning.

For most teams, this is a false choice. Either pay full price for reliability, or save money and accept that builds will fail in ways nobody can debug.

At Razorpay, we stopped accepting that trade-off. We built a self-healing infrastructure layer for GitHub Actions on Kubernetes that runs 80% of our CI workloads on spot instances while maintaining a 99.2% job success rate. When AWS terminates a spot node mid-build, our system detects it, retries the job, cleans up the orphaned pods, and the developer never knows.

This is the story of what we built, why polling wasn’t an option, and the war stories that taught us how to do retries without burning everything down.

The Problem With Spot Instances On CI

Spot instances are AWS capacity that nobody else wants right now. The pricing is brutal compared to on-demand: a c5.2xlarge that costs around $0.34/hour on-demand drops to roughly $0.08/hour on spot. For workloads like CI/CD, where jobs are short-lived and parallelizable, the math is obvious.

The catch is in the contract. AWS reserves the right to take spot capacity back at any moment, with a 2-minute warning. That works for some workloads. For others, it’s catastrophic.

GitHub Actions runners on Kubernetes is the hard case. A typical CI job goes like this: GitHub assigns the job to a runner. The runner is a pod on a Kubernetes cluster running on an AWS Spot instance. The job downloads dependencies, runs tests, builds artifacts. The runner reports back to GitHub.

Now insert a spot termination at minute 4 of a 7-minute build. What happens?

• The runner pod dies mid-job.
• GitHub never gets a completion signal; the job hangs until timeout, then marks as “failed”.
• A new runner pod gets scheduled (assuming spot capacity is available).
• The orphaned pod from the killed node sits in “Failed” state forever, consuming cluster slots.
• The developer sees a red X with no useful explanation.

Multiply that across 10,000+ GitHub Actions jobs per day across hundreds of repositories. The math turns ugly fast. Stale pods accumulate. Failed jobs require manual retries. Engineers stop trusting CI signals. Some teams give up and migrate back to on-demand, paying 4–5x more for a problem that was solvable.

That was the state we were in. Spot instances saved real money but extracted a real tax in operational toil and developer trust.

What We Built

The system is four components working as a pipeline, plus one that runs on the side. Each one has a single responsibility and is deliberately ignorant of the others.

spot-node-metric is the early warning system. A DaemonSet that runs on every node, tracking AWS Spot interruption notices in real-time. When a node receives the 2-minute warning, this metric fires, which lets the scheduler stop assigning new jobs to nodes that are about to disappear.

spot-loss-checker is the classifier. When a job fails, this service decides whether it was a spot termination or a real failure. The detection is surprisingly simple: it scans the runner pod logs for the pattern √ Connected to GitHub. If that line is present, the runner had successfully registered before dying; the failure was a spot termination. If it’s absent, the runner never connects, which usually means a real configuration problem. This pattern catches 95%+ of actual terminations and tolerates transient network blips without false-flagging them.

rerun-failed-jobs is the recovery engine. When spot-loss-checker marks a job as retriable, this service calls the GitHub API to rerun it. The retry tracks an idempotency key so the same job can’t be retried twice if duplicate webhooks fire. Exponential backoff prevents the system from hammering the GitHub API during large spot disruption events.

rerun-failed-jobs is the recovery engine. When spot-loss-checker marks a job as retriable, this service calls the GitHub API to rerun it. The retry tracks an idempotency key so the same job can’t be retried twice if duplicate webhooks fire. Exponential backoff prevents the system from hammering the GitHub API during large spot disruption events.

k8s-runner-cleanup is the housekeeper. Every 15 minutes, it scans for stale runner pods in the actions-runner-system namespace. It checks each pod’s logs for the same connection pattern. Pods that never established connection get deleted. This is what prevents the orphaned-pod accumulation problem.

github-action-monitoring is the entry point. A Flask service that receives webhooks from GitHub every time a job changes state. It extracts metadata: workflow name, repository, runner ID, queue time, run time, conclusion. It exposes Prometheus metrics with histogram buckets so we can compute P50/P90/P95/P99 percentiles for every dimension we care about.

The pipeline is event-driven end to end. A job fails, a webhook fires, the classifier runs, the retry fires, all within seconds. No polling, no batching, no waiting for the next cron tick.

The Four Design Decisions That Shaped This

A few calls deserve a closer look because each one had a non-obvious second-best alternative we deliberately rejected.

Webhooks over polling

GitHub provides two ways to know about job state: poll their API on an interval, or receive webhooks when state changes. We chose webhooks. The trade-off looks small until you sit with it.

Polling is simpler operationally. It’s also a delay multiplier. A 30-second poll interval means the average detection latency for a failure is 15 seconds. Multiplied across 10,000 jobs/day and several stages of recovery (detect → classify → retry → restart), polling-based detection adds up to several minutes of recovery time per job. Webhooks fire within milliseconds of the state change.

The cost was operational complexity. Webhooks need a publicly reachable endpoint with proper auth, retry handling for failed deliveries, and idempotency for duplicate fires. None of these are hard problems individually. All of them are skipped in a polling implementation. We paid the complexity tax to buy real-time detection.

Log pattern matching for spot detection

The cleaner architectural answer for spot detection would be to receive AWS’s interruption notice events, correlate them with running jobs, and mark those jobs as spot-killed when the corresponding nodes disappear. We considered building that. We didn’t.

The simpler approach was scanning runner logs for the pattern √ Connected to GitHub. Present means the runner registered successfully and something killed it. Absent means the runner had a real problem.

This is brittle in a textbook sense. If GitHub changes the connection success log format, our detector breaks. We’ve already had that happen once during a GitHub Actions runner update. The cleanup pipeline silently stopped working for nearly a day before the metrics showed the divergence.

The trade-off was deliberate: spend two days writing a brittle but immediately useful detector, or spend two months building correlated event handling that handles every edge case. We took the short path knowing we’d pay maintenance costs later. The detector has been worth it.

Detecting spot kills from runner logs

When a spot node disappears, we need to tell two cases apart: a runner reclaimed by AWS versus a runner that hit a real failure. The clean architectural answer is to consume AWS’s interruption-notice events, correlate them with running jobs, and mark the jobs whose nodes vanish as spot-killed. We scoped that and chose to defer it.

Instead, the detector scans runner logs for the registration success line, √ Connected to GitHub. If a runner logged it and then died, a spot reclaim took it. If the line never appeared, the runner had a real problem. Two days of work against an estimated two months for full event correlation, for a signal we needed immediately.

The dependency this creates is narrow and explicit: one external log string. It sits off the job-execution path. If GitHub changes the format, jobs keep running and rescheduling normally; what degrades is the cleanup-and-attribution signal, not reliability. That bounded blast radius is what made the shortcut defensible.

We found the boundary the hard way. A GitHub Actions runner update changed the log format once, and the cleanup pipeline stopped attributing kills correctly. Metric divergence surfaced it, and we now alert on that divergence directly, so the detection window is minutes rather than the better part of a day. Full event correlation remains the planned evolution; until the maintenance cost of the log-scan outweighs its simplicity, the simple version keeps earning its place.

Histograms over raw metric storage

Early in the build, we stored every job’s queue time and run time as individual Prometheus samples. The cardinality exploded the moment we scaled past a few hundred jobs per minute. Memory pressure caused OOMKills in the monitoring service. The thing built to monitor reliability was the least reliable component in the stack.

The fix was switching to histogram buckets. Instead of storing every job’s exact queue time, we count how many jobs fell into each bucket: 0–100ms, 100–500ms, 500ms-1s, 1–5s, and so on. Server-side percentile calculation gives us P50/P90/P95/P99 from the histograms. Cardinality drops from “number of jobs” to “number of buckets,” which is constant.

The cost is precision. A job that took exactly 437ms gets bucketed as “100–500ms” rather than recorded exactly. For percentile analysis, this doesn’t matter. For debugging a specific failure, we use logs, not metrics. The split is the right one.

Idempotency keys on every retry

The first retry storm we hit taught us something. During a large spot disruption that killed several hundred jobs at once, the system fired retry webhooks for each one. Some webhooks hit our retry service twice because of GitHub’s at-least-once delivery semantics. The result: jobs ran twice, sometimes three times. Test suites that mutate shared state (database fixtures, integration test environments) corrupted themselves.

The fix was idempotency keys. Every retry attempt generates a deterministic key from the job ID and the original failure timestamp. The retry service tracks recent keys and refuses to fire a retry it’s already attempted. Duplicate webhooks get logged and dropped.

This sounds obvious in retrospect. Most production systems with retries need it. The lesson worth carrying: idempotency isn’t a defense against bugs in your code; it’s a defense against the at-least-once delivery semantics of every event system you’ll ever build on.

The War Stories

A few things broke in ways that taught us something.

The retry storm of the first big spot event. A large AWS Spot capacity reclamation killed approximately 200 jobs simultaneously. Our retry service got 200 webhook deliveries in a 30-second window. It fired 200 GitHub API calls to trigger reruns. GitHub responded with rate limit headers, but the retry service didn’t honor them, so it kept retrying. The rate limit got worse. Some jobs were retried 4–5 times before the system stabilized.

The fix was exponential backoff with jitter on retry attempts, plus respecting GitHub’s rate limit headers. After this, the system gracefully degraded under load instead of amplifying the disruption.

The double-retry bug from racing webhooks. Two webhook deliveries for the same failed job arrived within 200ms of each other (GitHub’s at-least-once semantics). Both made it past the deduplication check because the dedup window was 100ms. The job ran twice. The integration tests it ran had ordering dependencies; they corrupted shared state and failed in confusing ways for the next 6 hours.

The fix was the idempotency keys above, plus widening the dedup window to 30 seconds and using a proper distributed lock instead of an in-memory check.

The silent cleanup failure. GitHub updated the runner binary, which changed the connection success log line. Our log pattern matcher stopped recognizing successful connections, which meant k8s-runner-cleanup started flagging every pod as stale. Within an hour, it had deleted every active runner pod in the cluster. CI ground to a halt. Recovery took ~45 minutes once we figured out what was happening.

The fix was adding a sanity check: if cleanup is about to delete more than 20% of active pods in a single run, abort and alert. Slow degradation gets caught. Catastrophic deletion gets prevented.

What It Bought Us

Six weeks of engineering effort produced an infrastructure layer that has been quietly running ever since. The numbers tell the practical story.

The cost story is the headline: 60% reduction in CI compute spend, with reliability that’s better than what we had before. The toil story is the more important one: zero manual retries required across 10,000+ jobs per day. The on-call alerts for CI failures have effectively disappeared. The system is in production org-wide.

The system is in production for five teams handling infrastructure, platform, and payments workloads.

What’s Next

The current detection logic treats all failures the same: if a runner disconnected unexpectedly, retry it. That’s correct most of the time, but it retries permanent failures alongside spot terminations, which wastes API calls and pollutes the metrics. We’re working on classifying failures as transient (network blip, spot termination, resource exhaustion) vs. permanent (bad test, missing dependency, syntax error). Transient failures retry; permanent failures route to the responsible team.

Cost attribution is the next obvious layer. The system knows which teams’ jobs ran on spot instances and which were rescued from spot terminations. Exposing that as per-team cost savings makes the platform’s value visible to leadership. Right now the savings are real but invisible to the people whose budgets they affect.

Multi-cluster support is the long-term direction. Today the system runs in a single Kubernetes cluster. As Razorpay’s CI footprint grows, a centralized control plane that spans clusters would simplify operations and improve resource utilization across the org.

What We’d Say To Other Teams Building This

Three lessons that aren’t specific to GitHub Actions on Kubernetes.

Webhooks beat polling for anything event-driven, even though webhooks cost more to build. The latency difference compounds across stages. A pipeline that’s polling at every step accumulates seconds of delay per job. A webhook-driven pipeline reacts instantly. Pay the auth and idempotency tax upfront; reap the recovery time benefits forever.

Brittle detectors that work today beat elegant detectors that ship in six months. Our log-pattern matcher will break when GitHub changes their log format again, and we’ll fix it again. That maintenance cost is much smaller than the cost of waiting for the perfect event-correlation system. Ship the version that works now and budget for the inevitable maintenance.

Idempotency is not optional in production retry logic. Every event system you’ll ever build on has at-least-once delivery semantics. Every webhook fires more than once eventually. Every retry path will be invoked twice in some sequence of events. Build for that on day one. Don’t wait for the second incident to teach you.

The deeper takeaway is that spot instances aren’t risky infrastructure. They’re predictable infrastructure with one specific failure mode. Once you’ve built a system that handles that failure mode automatically, the cost savings are nearly free. The trade-off between reliability and cost wasn’t fundamental. It was just under-engineered.

Editor: Parth Sawhney

CI Doesn’t Need On-Demand: Moving Our Build Pipelines to Spot Instances was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Contributors: Utkarsh Koppikar Rohan

Background

Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries.

The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables — wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month).

The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them.

As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey.

The Full Refresh Pipeline: Our Baseline and the Pain

The original full-refresh pipeline was straightforward.

1. Schedule: Airflow schedules Spark jobs on EMR daily during off-peak hours, passing a Fact Config from S3—a configuration file listing the tables to join, columns to select, join conditions, and the output format.
2. Full Refresh: The job then reads all source tables from the data lake, performs a full Spark join, and overwrites the entire denormalised table back to S3.
3. Notify: Once complete, it publishes the max(primary_table_updated_at) timestamp to the Reporting Service, which uses it to split queries: the Fact covers the historical window, and TiDB — our warm store, a distributed MySQL-compatible database holding recent transactional data; fills the freshness gap for data not yet materialised into the Fact.

This approach created three classes of problems:

1. Operational: Pipelines ran on spot nodes for many hours, making spot loss inevitable. Since the business logic required the full dataset at once, there was no safe checkpoint — a single failure meant restarting from scratch. We eventually chained smaller facts together to create checkpoints.
2. Scaling: Our data has back-dated references — a payment today can reference an order from five years ago, a refund can reference a year-old payment. This forces unbounded joins and full table scans on every secondary table with no safe time window to apply. The Payments fact grew to join over 30 tables and ran for 8–10 hours. Facts initially contained data from 2018, and to manage cost, we reduced fact retention to one year as a stopgap, allowing some reports to fall back to warm stores for older data — but this deepened our TiDB dependency and blocked retention initiatives on the warm store.
3. Cost: Pipelines ran for 70+ hours per day across large EMR clusters. We moved most facts to alternate-day schedules, creating up to 48-hour freshness lags.

Previous Attempts and Challenges: The Path to Solution

We attacked this problem from both short-term and long-term perspectives. Fact decomposition, revised data retention, alternative-source fallbacks, and alternate-day scheduling served as short-term solutions to maintain stability without affecting the user experience.

To solve these problems in the long term, we built a streaming pipeline that subscribes to Kafka CDC topics, models the Fact config as a dependency graph, and performs TiDB lookups to produce denormalised rows in near real time. (Details: Real-time Denormalised Data Streaming Platform)

The solution worked for a year but eventually failed for three primary reasons:

1. Highly mutable data: 10–20 change streams joined at 10M+ events per 30-min window drove costs unsustainably high.
2. All data in warm store: Back-dated references forced us to keep all historical data in TiDB, blocking any retention initiative.
3. Write amplification: High-cardinality joins amplified upsert volumes. Since joins were slow on the lake, we had no option but to force all join tables into precomputed facts.

The Pivot: Incremental Facts

The core insight: treat Facts as incrementally maintainable graphs. Instead of regenerating the full table, process only the change events for each entity using the dependency graph to know which related rows to fetch, and a secondary index to know where on the lake to find them.

Consider a real-world payment creation flow, you first create an Order for the product you want to buy, and a Payment is made on that order using a Card by applying a Discount allowed by the ongoing Offer. This relationship can be visualised as a graph as shown in the image and this became one of the core insights for the approach.

Before we into the low level details, we need to understand the following constraints:

1. Data Partitioning: Our data lake on S3 contains source tables in open-source Delta format and partitioned on `created_date`.
2. Data Indexing: We use Z-ordering on the primary key and merchant_id columns. Since these columns have high cardinality, this setup is inefficient for lookup queries.

Three design questions had to be resolved:

1. How do you fetch all updates for an entity on T-1 day?

We built a silver-layer pipeline that creates flattened, deduplicated tables partitioned by updated_date, retaining only the latest change per primary key per day.

2. How do you know which related rows to fetch for a change event?

We modelled the Fact config as a traversable in-memory dependency graph, with edges derived from join conditions. Consider a fact that joins five tables:

SELECT P.*, O.*, C.*, D.*, O.* 
FROM payments P 
LEFT JOIN orders O ON O.id = P.order_id    -- edge: Payments → Orders 
LEFT JOIN cards  C ON C.id = P.card_id     -- edge: Payments → Cards 
LEFT JOIN discounts D on D.payment_id = P.id -- edge: Payments → Discounts 
LEFT JOIN offers O on O.id = D.offer_id  -- edge: Discounts → Offers

This produces the graph as shown above. Each edge carries the join predicate as metadata.

The graph tells us exactly what to fetch for any change event:

• Update on Payments (P1): follow edges outward — look up Orders where orders.id = P1.order_id, and Cards where cards.id = P1.card_id. Combine to form a complete denormalised row.
• Update on Orders (O1): back-traverse to the root — look up Payments where payments.order_id = O1.id to find the primary keys affected. Cards are not in the path from Orders to the root, so it is skipped.

3. How do you look up data efficiently on the lake?

We built a Secondary Index per entity , a small table holding join columns and created_date partition values. For example, a payment index table would include id, card_id, order_id, and created_date.

Because this table has a small sub-set of columns that change infrequently,it is significantly smaller and easier to maintain than the original source table.

The index narrows reads to a handful of partitions, eliminating full table scans. An independent batch pipeline upserts the Secondary Index from the silver layer. This is the core component that removed our dependency on TiDB for historical lookups.

Incremental Processing using Graph Traversals

When a warehouse job is triggered, it parses the Fact configuration to create a dependency graph in memory. It processes updates for each node sequentially, starting from the primary node. Processing logic differs between the primary table and secondary tables. We can therefore broadly classify processing into two phases: the Primary flow and the Secondary flow.

The Primary Flow:

In this flow, we process only the primary table in the graph (payments, in the example above).

For example:

1. Fetch primary table updates from the last checkpoint.
2. Join updates with each secondary index to get created_date partition values; use those to read only the relevant partitions of each source table.
3. Stitch all data into denormalised rows and upsert into the target fact, scoping the merge to affected partitions only.

The Secondary Flow

In this flow, we iterate through all secondary tables in the graph using a level-order traversal and process them sequentially.

1. Fetch updates for the secondary table from the last checkpoint. (e.g offers)
2. Back-traverse the graph: Join with the indexes of all ancestor nodes up to the root (e.g., discounts_index, payments_index for Offers) to enrich each update row with the primary key and partition values needed for the target merge. We don’t need a full source table read, since the required ancestor data is already in their indexes.
3. Merge into Fact: Upsert the partially denormalised rows to the target fact, scoped to affected partitions.

Handling High Cardinality and Dimension Tables

High-cardinality dimension tables are expensive to denormalise due to write amplification: a single update can fan out to thousands of fact-row rewrites.

For example, consider a "Payment Links" table in which a single link can accept thousands of payments. If Payment Links is a secondary table and a column is updated, thousands of payments in the Fact table must be updated to reflect that single change. Hence, not every join belongs in a precomputed fact.

However, joins on our lake are expensive too because our domain doesn’t support time-bounded joins, which requires a full table scan on secondary tables. We wanted to push down a common predicate to reduce scans on the secondary tables.

For example, if you are joining payment_links and orders:

SELECT *
FROM payment_linksLEFT 
JOIN orders on orders.id = payment_links.order_id
WHERE payment_links.merchant_id = m1 and payment_link.created_at between t1 and t2

If we run this query on Trino, it will push down filters (merchant_id, created_at) for payment_links, but will result in a full table scan for orders.

Report generation is usually scoped by merchant_id, which is present in most tables. We decided to use it as a common predicate. Since merchant_id has high cardinality, we cannot partition data on it; instead, we use bucketing on merchant_id with a fixed number of buckets, similar to sharding. We chose Iceberg (V2) as the table format for its native bucketing support.

Adding merchant_id to both sides of the join condition lets Trino push predicate on both the sides, scanning only the matching bucket:

LEFT JOIN orders on orders.merchant_id = payment_links.merchant_id 
AND orders.id = payment_links.order_id
WHERE payment_links.merchant_id = m1 and payment_link.created_at between t1 and t2

This unlocked joins in the data lake. We addressed the remaining gaps by building an Iceberg replication pipeline and tuning non-Fact tables with sort orders and `merchant_id` bucketing. We also used hidden month/day partitioning to reduce merge scans. Finally, we added a query builder in Reporting Service to route joins intelligently. Some joins are served from the precomputed Fact table, while others run between Fact and Iceberg tables at query time.

Production Rollout

Rolling out to production was more challenging than anticipated. We needed to ensure that no inaccurate data reached our merchants.

We followed three phases to ensure accuracy before decommissioning the legacy flow:

1. Data reconciliation: Compared incremental and full-refresh facts record-by-record, resolving checkpointing, out-of-order events, and index lag issues until we hit 100% accuracy.
2. Shadow testing: Ran A/B experiments generating shadow reports from incremental facts in parallel with live reports. Automated nightly comparisons flagged mismatches; we iterated until both were identical.
3. Gradual live rollout: Progressively shifted live traffic to incremental facts with a fallback to the legacy flow, addressing edge cases without merchant impact.

Moving to a three-pipeline architecture from a single full refresh pipeline naturally raises the chance of a hiccup here and there, so we’ve made the system self-healing. Each pipeline now publishes its own checkpoints, meaning downstream pipelines can see exactly where their upstream counterparts are at and intelligently decide how much data to process if they’re lagging. If we do hit a delay, we automatically fall back to sources like TiDB to keep data fresh in reports. We also have Data Quality checks watching the whole flow; when they spot a mismatch, they flag it on our slack channels, narrowing down the debug scope.

Impact

We migrated the five largest facts to the incremental strategy and the impact numbers are shown below:

The ~90% reduction in compute time translated to proportional cost savings. Restoring full historical coverage in Facts allowed us to reduce TiDB retention significantly. Runtime joins enabled decomposing large monolithic facts into smaller, domain-specific ones — reducing blast radius and maintenance cost.

Lessons Learned

1. Streaming isn’t always the answer: State-heavy joins at high event volumes eventually become cost-prohibitive. Batch + incremental can win.

2. The lake should be queryable, not just archival: Secondary indexes on the lake eliminated warm-store dependency for historical lookups. Iceberg is adding native support for secondary indexes in upcoming releases.

3. Not every dimension belongs in a materialised fact: Runtime joins with bucketed Iceberg tables are cheaper than write amplification at scale.

4. Facts are graphs, not tables: Modelling fact configs as traversable dependency graphs is what unlocks incremental maintenance.

Future Roadmap

We’ve seen incredible operational and cost improvements from migrating our first five Facts, and it’s clear that the Incremental Strategy is the right path forward. We’re excited to transition the rest of our facts onto this new approach. Alongside this, we’re also diving into these key initiatives to push things even further:

1. Real-time secondary index: We are exploring Apache Flink to merge the index pipeline into CDC ingestion. This approach achieves near-real-time freshness and reduces latency for lookups and ad-hoc queries.
2. Extend support to more Sinks: Applying incremental processing to unlock savings and near-real-time freshness for warehouse tables in other destinations like ClickHouse, TiFlash, etc.

Editor: Parth Sawhney

How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Contributors: Siddharth Tripathi

Today, more than 220 Razorpay employees each have their own always-on AI agent. On a typical day, about 84 of them are actively working.

Every one runs in its own isolated Kubernetes namespace, with its own encrypted storage, its own cloud identity, and its own network policy. It learns new skills as its owner works, and it keeps running long after they close their laptop: one employee’s agent has already logged more than 15,000 sessions, 90% of them while its owner was asleep or away.

Provisioning a new one takes under two minutes.

Running the agents was never the hard part. Running them safely was: 220 of them, each with shell access and live credentials, without any single agent becoming a path into another employee’s data or out to the open internet. The answer came down to one design choice, and everything in this post is a consequence of it:

Isolation is a property of the infrastructure, not the application.

Hermes itself is an open-source agent by Nous Research; what we built is the platform that runs it safely & isolated, for the whole company.

Proof It’s Real: One Instance, Eight Weeks In

Before any of the architecture, here’s the proof that people actually use this. The following is one real instance from our cluster, over its first eight weeks (numbers pulled live, the person anonymised).

In eight weeks, this one user’s Hermes ran 15,039 sessions. Only 391 of those were the person sitting down to chat with it; the other 13,570 were autonomous runs the agent kicked off on its own schedule while its owner was asleep or in meetings. That ratio is the whole idea in one statistic: the assistant does most of its work when you’re not there.

What is it doing in those runs? They’d wired up 21 always-on scheduled jobs that turn Hermes into a personal intelligence service:

• Ingest. Every hour it pulls from 23 Slack channels (plus DMs), 6 GitHub repos, and Google Workspace (calendar, Gmail, Drive, Docs); on a daily cadence it sweeps 62 Twitter/X handles, market RSS feeds, and a set of Substacks.
• Synthesise. A pipeline compiles all of that raw material into a personal wiki (pages for people, projects, decisions, threads, and concepts), then extracts an entity graph and builds a vector index (Titan-v2 embeddings) so the agent can answer questions across everything it has read.
• Deliver. It posts a 07:15 morning digest and a 19:00 end-of-day digest to a private Slack channel, follows the morning digest with a synthesised news thread, and nudges a running task tracker every two hours.

On top of the automation, the instance has taught itself 98 skills across 25 categories, a “skill” being a reusable, written procedure the agent can follow (Part 1 shows how they’re made): from github-pr-workflow, systematic-debugging, and test-driven-development on the engineering side to self-authored, Razorpay-specific ones like an “AI-playbook gap-finder” and a “wiki brain.” None of these were written by a platform admin; the agent distilled them from repeated work, and none of them leak out of this one person’s sandbox.

That’s a single tenant, and an outlier in ambition, not in kind. The same primitives (a persistent pod, scheduled runs, private skills, audited egress) sit under all 220 instances, waiting for each person to lean on them as hard.

Part 1: Why We Built This, and What You Get

The Problem

We wanted to give everyone at Razorpay their own “second brain”: a personal, always-on assistant that holds their working context and helps them get things done. Handing everyone a coding agent to run on their laptop was the obvious starting point, but for our use case it fell short on three counts.

First, a laptop is only awake when its owner is. It sleeps on the commute and stays shut over the weekend; the VPN drops when the lid closes; cloud credentials expire overnight; and nothing the agent does on it leaves a trail a security team can see. We wanted an assistant that keeps working when the person doesn’t, somewhere always-on and observable, which rules the laptop out on both counts.

Second, we wanted to be multi-model. Not every task deserves the most expensive model. We wanted to route routine work to a cheaper or open-source model and reserve a stronger one for when it’s worth it, rather than being pinned to a single model or provider.

Third, we wanted skills that learn: an assistant that gets better at your job over time, not a generic one.

And because we’re a fintech, none of this could come at the cost of safety. A tool that holds each person’s working context must not leak one employee’s data into another’s, be reachable from the open internet, or become a path for exfiltration or a poisoned dependency.

A shared, multi-tenant application would put all of that trust in application-layer checks getting it right every single time. That wasn’t a bet we were willing to make. So we did something different: we made isolation a property of the infrastructure, not the app.

How We Solved It

Our implementation of Hermes resides within an always-on Kubernetes cluster, where every employee is provisioned their own personal container, isolation becomes a fundamental property of the architecture rather than a fragile policy. This ensures that every assistant persists in its work, operating autonomously whether the owner is at their desk or offline.

Access is mediated by a centralized gateway where users authenticate via Google. Upon signing in, they are programmatically routed to their specific, isolated environment. We have effectively pushed isolation to the container boundary and identity verification to the SSO layer, ensuring that ownership of each “brain” is enforced at the network level. Furthermore, the environment is hardened through total network isolation, remaining accessible only via the internal corporate network and shielded from the open internet.

The platform is deliberately agent-runtime-agnostic. The same identity, isolation, and auditing that host Hermes today could host a different agent, or any future tool that speaks the same protocols, tomorrow, each as just another tenant. A security team can stand up a new runtime as a template, a namespace, and a cloud-identity role, without rebuilding the platform.

What You Actually Get

From a person’s side it’s simple: open a URL, sign in with Google, and you’re in your own assistant. Three things make it more than a chat box.

It’s always on. Because your assistant lives in the cluster, not on your laptop, it keeps its context and keeps working when you’re offline, which is what makes the next part possible.

It’s multi-model. Routine work can go to a cheaper or open-source model; the hard problems can go to a stronger one. You’re not locked to one model or one vendor, and the assistant can switch per task.

It learns your work. This is the headline, so it’s worth making concrete.

A “skill” is a reusable, written procedure the assistant can follow. Normally a human writes them. Hermes flips that: because your assistant is persistent, it watches how you get a recurring task done and turns that into a skill scoped to you.

Before. You manually guide your assistant through a recurring report — pulling data, comparing stats, and formatting the output — each week.

After. Hermes recognizes the pattern, codifying it into a reusable “weekly report” skill. It automates the task, refining its performance with each subsequent run.

Skill creation is autonomous by default: the agent notices a non-trivial workflow and saves it for reuse without being asked. The “weekly report” above is one shape it takes; others we’ve seen in the wild include an agent that learned to triage its owner’s on-call alerts, one that drafts release notes from merged pull requests in a team’s house style, and one that maintains a running wiki of everything its owner reads. Because these are procedures the agent writes for itself, the platform can optionally put every skill write behind a human approval gate, the same approve/deny flow we use for risky commands, so in a locked-down deployment nothing the agent teaches itself lands unreviewed.

Two earlier choices pay off here. Because skills live inside your personal container, what the assistant learns about your job never leaks into a colleague’s; the same isolation that protects your data keeps your skills yours. And because the cluster is always on, the assistant can do this distilling in the background, so you come back to a sharper assistant rather than a cold one.

The Real Point: Safe Path has to be the Easy Path

There’s a strategic idea underneath all of this. When security offers no sanctioned alternative, every “can I install X?” request either gets denied (driving the tool underground) or approved on faith. Neither scales.

Our answer was to make the sanctioned path genuinely faster and easier than running an agent on a laptop: one command provisions a fully isolated, audited, always-on assistant in under two minutes. Once the safe path is the easy path, shadow AI stops being a policy problem you police and becomes a default that people simply choose. That, more than any single control below, is the thing we’d want another team to take away.

Part 2: How It Works

Isolation is Infrastructure, Not Application

That one sentence is the spine of the whole design, so it’s worth being precise about what it means.

A shared, multi-tenant application would put every safety guarantee in application code: the app would be responsible for checking, on every single request, that user A can’t see user B’s data. Get one check wrong, once, and the boundary is gone. For a tool with shell access and live cloud credentials, that is too much to trust to code.

So we don’t. Each person gets their own pod, their own namespace, their own encrypted volume, and their own cloud identity, and the boundary between them is enforced by Kubernetes and the network, not by the application. A compromised agent doesn’t reach another employee because there is no application code path from one tenant to another to get wrong; the isolation is structural.

Everything below is a consequence of taking that seriously. We organised it as four security planes.

The Four Security Planes

Each plane is independently defensible; together they’re defense in depth.

Plane 1. Identity: Only Ever Your Own Agent

Threat: a user must only ever reach their own agent, and nobody should be able to hand-edit a request into someone else’s.

Design: one gateway, Google login, and a rule that maps the signed-in identity to exactly one pod and refuses everything else.

Result: reaching another person’s agent would mean forging a signed session, not just changing a URL.

There is one way in: a gateway that every request passes through. You sign in once with Google, and from then on the gateway does two things on every request: it confirms who you are, and it checks that the agent you are asking for is yours. If the identity in your session does not match the agent in the URL, the request is refused. That check runs in the gateway on every request, against a session whose integrity is cryptographic (not a per-app config rule someone can fat-finger), which closes off the confused-deputy bugs common in shared AI platforms.

The gateway itself is locked down: no shell, non-root, read-only filesystem, minimal privileges, and spread across availability zones so losing a node never takes it offline.

Under the hood, the gateway is a small Go proxy. Login is Google Workspace OIDC with PKCE (which binds the login handshake to the client that started it); on callback it verifies the token’s signature, issuer, hosted-domain and verified-email claims, canonicalises the email to a username, and mints an AES-256-GCM session cookie with authenticated metadata that prevents replay or downgrade. Every later request checks that the session’s user matches the requested subdomain before proxying.

Plane 2. Isolation: One Pod Per Person

Threat: a compromised agent shouldn’t be able to touch another employee.

Design: one pod, one namespace, one cloud identity, one encrypted volume per person. Nothing shared.

Result: the maximum blast radius is a single employee, and you can delete it with one command.

Each person gets a dedicated Kubernetes namespace with their own pod, their own encrypted volume, and their own cloud identity. No two users share a process, a filesystem, a model server, or a credential cache. If an agent is compromised (a malicious image, or a successful prompt-injection), the damage is bounded to that one namespace, and tearing it down is a single kubectl delete namespace. This is the same property that makes it safe to try untrusted agent runtimes at all.

Inside the pod, Hermes runs as an init container plus three main containers:

• Gateway: the AI engine (inference, tool execution, conversation state).
• Dashboard: the web chat UI, talking to the gateway over localhost.
• Socat loopback relay: a small sidecar (the “golden hacks” section explains why).
• Init: seed-config: seeds the config on first boot and preserves the user’s customisations after. Cattle, not pets.

Under the hood. Cloud credentials come from IRSA (IAM Roles for Service Accounts): the pod is handed short-lived, auto-rotated AWS credentials tied to its Kubernetes identity, so there are no long-lived API keys sitting in any pod.

Plane 3. Network: The Only Way Out is The Audited One

Threat: a compromised agent shouldn’t be able to phone home, exfiltrate, or reach another tenant.

Design: kernel-level network policy that allows egress only to the audit proxy and DNS, and denies traffic between tenants by default.

Result: the sole route to the internet is the audited path, and lateral movement is dropped in the kernel.

Each tenant’s namespace gets a network policy that permits exactly two destinations: the in-cluster audit proxy, and DNS. Everything else is dropped. Between tenants, the default is deny, so a compromised agent can’t scan or reach its neighbours. Because the enforcement happens in the Linux kernel rather than in the application, an agent can’t opt out of it by being clever. New policies roll out in observe-then-enforce mode, logging would-be violations before they actually drop, so we never break legitimate traffic on day one.

Under the hood. Networking runs on Cilium in eBPF mode, which replaces kube-proxy and matches traffic against Kubernetes identities (namespace, labels, service account) instead of IPs that churn on every restart, so policies don’t silently break when pods reschedule. Drops happen in-kernel and are observable via Hubble flow logs.

Plane 4. Observability: Every Request Screened and Attributed

Threat: if an agent does try to misbehave, we have to see it and stop it in real time.

Design: route every outbound call through one proxy that screens it against a live LLM policy and records it per user.

Result: ~15M calls a month screened, ~219k blocked as they happen, every one tied to a named person.

Every call out of every agent goes through CrabTrap, an open-source TLS-intercepting egress proxy (from Brex); the network plane guarantees pods can’t bypass it. For each call, CrabTrap does three things: it attributes the call to a specific person via a per-user token, records it to an encrypted, access-controlled audit store, and screens it against a live LLM-based policy that decides, in real time, whether the request is allowed to leave. The audit trail and the enforcement run on the same inline path, so we get a complete record and real-time blocking at once. Because the policy is data, not code, we can tighten it per tenant without redeploying a single agent pod.

The scale is what makes it meaningful. Over a recent 30-day window:

Every one of those calls is tied to a named person, not an IP, which is exactly what you need when the thing making the call is an autonomous agent.

Under the hood. “TLS-intercepting” means CrabTrap re-signs upstream certificates with its own certificate authority so it can see inside HTTPS. That only works if every TLS client in the pod trusts that CA, which turned out to be one of the four “golden hacks” below. Repeat identical checks are cached (hundreds of thousands of decisions reused), so screening adds negligible latency.

Multi-model Routing

Not every task deserves the most expensive model. Inference is routed through Razorpay’s internal LLM Gateway, a single endpoint that fronts a roster of models across providers (Claude, GPT, Qwen, Kimi, GLM, DeepSeek, and more) behind one scoped key per agent.

Routing is per task, and mostly the agent’s own call. It runs on a cheaper, often open-source model by default, and reaches for a stronger one when a task actually needs the reasoning: a long refactor, a gnarly debug, a piece of analysis where a weak model would quietly get it wrong. Because the Gateway exposes the whole catalogue, the agent can discover what’s available and switch mid-session without any config change or redeploy, and a model that’s added centrally is instantly available to all 220 agents. AWS Bedrock (keyless, via IRSA) sits alongside as a first-class path for teams that want to stay on Anthropic models specifically.

The point of the indirection is that the choice of model is a routing decision, not a deployment one, and it’s a decision we can see: whichever model a call targets, it still flows through the audit proxy in Plane 4, attributed to a person.

The Four Golden Hacks

These four bugs each took us hours to debug. Getting a hardened, off-the-shelf agent image to actually run inside a locked-down Kubernetes environment (with short-lived cloud credentials, TLS inspection, and network-isolated routing) turned up four non-obvious, undocumented workarounds, each a real production failure mode with an error that pointed nowhere near the cause. They are the most reusable thing we learned.

• Cloud-credential token permissions via a supplementary group. The pod-identity webhook mounts the IRSA credential token owned by root, group-readable only. The agent’s privilege-drop step resets its supplementary groups, dropping the one that could read the token, so the runtime user can’t read its own credentials, and every model call fails with “permission denied.” The fix is to add the runtime user to that group before the privilege drop:

# let the unprivileged runtime user read its own IRSA credential token
usermod -aG "$(stat -c '%g' "$AWS_WEB_IDENTITY_TOKEN_FILE")" hermes

• WebSocket localhost check via a socat sidecar. The dashboard hardcodes a rule that WebSocket connections must originate from loopback (127.0.0.1/localhost), with no config override. In Kubernetes, service-routed traffic arrives from the pod’s cluster IP and gets rejected. A tiny socat sidecar transparently relays external connections through loopback, so every connection looks loopback-originated to the dashboard and the chain completes.
• npm workspace repair. The upstream image ships an incomplete npm workspace (one package has its manifest but not its materialised files) and the lockfile check skips reinstalling it, so the UI crashes with a missing-module error. The dashboard wrapper detects the missing file and runs a targeted npm install (preserving proxy settings) at boot.
• Multi-library CA trust. A TLS-intercepting proxy re-signs every upstream certificate with its own certificate authority, and the pod has four different TLS clients that each trust a different bundle. You have to point all four at the proxy’s CA, or one library rejects the re-signed cert with an error that surfaces in a completely different component than the missing setting:

export AWS_CA_BUNDLE=/etc/crabtrap/ca.pem        # boto3 (AWS SDK)
export REQUESTS_CA_BUNDLE=/etc/crabtrap/ca.pem   # Python requests
export SSL_CERT_FILE=/etc/crabtrap/ca.pem        # httpx / the model SDKs
export NODE_EXTRA_CA_CERTS=/etc/crabtrap/ca.pem  # the Node UI

Supply chain: Know Exactly What’s Running

Agent plugins and tool integrations are often unsigned, unvetted third-party code running with developer-level trust, so we treat the images themselves as part of the attack surface. Every image reference in the platform is pinned by sha256 digest, stored in a private, tag-immutable registry with scan-on-push, and built on distroless bases. That makes each pod’s spec content-addressable: the same manifest yields the same bytes on every node, every time.

We learned this the hard way. One week, newly created pods began failing to boot with agent init failed: anthropic required, while every existing pod (same manifest, same image: line) kept running fine. After a half-day of ruling out our own platform, the truth landed upstream: the :latest tag had been rebuilt with several Python dependencies silently dropped, so pods that pulled before the rebuild had the good bytes cached on their node and pods that pulled after got the broken ones. Two pods, one image: line, genuinely different code, and then it happened a second time. Every image is now pinned to an immutable sha256 digest, so “the same manifest” really does mean “the same bytes,” on every node, forever.

Onboarding in Under Two Minutes

Adding a person is one command:

bin/onboard-user.sh firstname.lastname@razorpay.com

That script canonicalises the email to a username, provisions a scoped model-gateway key, renders the Kubernetes manifests from templates, applies them in dependency order (namespace → service account / cloud identity → volume → config → deployment → service), wires up the per-user audit-proxy channel, waits for the pod to come up, and prints the URL. No cloud console, no identity-provider clicks, no manual IAM. Under two minutes, start to finish. Offboarding is just as clean: snapshot the volume for preservation, then delete the namespace.

Conclusion

AI agents are quickly becoming the highest-privilege software inside a company: they read source, run commands, hold live credentials, and increasingly act on their own. The question is no longer whether employees will use them. It’s whether they’ll use them inside infrastructure built for that privilege, or on a laptop where nobody can see.

For us, the answer came down to one principle, and everything in this post is a consequence of it: isolation is a property of the infrastructure, not the application. One pod, one namespace, one identity, one encrypted volume per person; every outbound call screened and audited; skills each person owns privately. Take that seriously and something useful happens: the secure path also becomes the easy path, genuinely faster than running an agent on a laptop. That is the only version of security that actually gets adopted.

Two hundred and twenty of our colleagues are already on it, most of the work happening while they sleep.

Editor: Parth Sawhney

Running Hermes at Razorpay: a Network-isolated, Self-improving “Second Brain” for every Employee was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

A consent-native CDP that serves audience segments across 500M+ user profiles in under 30ms, with PII isolated to the source systems.

The Problem We Were Solving

A customer opens her favourite online shopping app, adds a few items to her cart, and pays ₹1,200 via UPI, powered invisibly by Razorpay. A week later she returns and pays using a saved Visa card from her laptop. Later that month, she places a larger ₹8,500 order through net banking from work.

Three transactions. Three different payment instruments. Three different devices. To the merchant’s engineering team, and to Razorpay’s data systems, these could look like three completely different people, unless you’ve done the hard work of figuring out they’re all the same customer.

Now suppose this is a D2C fashion brand approaching their Diwali sale. The merchant’s growth team has a clear plan: “Identify customers who have transacted at least once in the last 30 days, have spent more than ₹5,000 cumulatively this quarter, and haven’t enrolled in our loyalty programme. Send them an early-access nudge with a personalized discount 48 hours before the public sale opens.”

A year ago, answering that question at Razorpay meant filing a cross-team data request, waiting for an analyst to write a custom Spark job, and getting an answer in 2–3 days. By the time the merchant had the segment, the Diwali sale was already live. The early-access window had closed. The campaign got sent to a broader, less-targeted audience, wasting spend on customers who would have bought anyway and leaving cold customers untouched.

Now multiply that pain by millions of merchants. Razorpay powers payments and growth for over 12 million merchants. From D2C fashion brands and SaaS startups to subscription platforms, ed-tech companies, and the 2 million+ local merchants accepting QR payments every day. Together, they process billions of transactions. Every one of those merchants is, in their own way, trying to grow. Running a sale, recovering an abandoned cart, nudging a churning customer, and identifying their next 1,000 high-value buyers.

We knew this was a fundamental capability gap that needed structural solving. That’s what drove us to build the Customer Data Platform (CDP), an in-house platform that sits at the heart of Razorpay’s data-driven product decisions.

DPDPA also reshaped what the platform had to be. India’s Digital Personal Data Protection Act introduced strict requirements around consent-scoped data processing, purpose-specific access, and the ability to honor user consent decisions at the data layer rather than the application layer. The CDP was built with these requirements baked into the architecture rather than bolted on afterwards.

The Architecture

Before diving in, it’s worth mentioning why a CDP at this scale is difficult. Teams that have worked on similar systems will recognize the challenge immediately. It has to balance data freshness, low latency, consent enforcement, cost, and correctness simultaneously. Few of these dimensions trade off against each other.

Razorpay’s EDW holds billions of derived attribute rows across 500M+ user profiles, refreshed daily from payment transaction data flowing in at tens of thousands per second. These attributes are stored in S3 with server-side encryption using client-managed KMS keys, which keeps access strict and explicitly enforced. A single segment definition touches multiple attribute tables, each with hundreds of millions of rows. Naive joins blow up Spark cluster memory in seconds.

Real-time membership checks complete in under 30ms, including network hops between services, and the current system sustains this at more than 1,500 RPS. Cost is another major constraint. At 500M+ profiles and hundreds of active segments, even small inefficiencies in compute or storage compound into significant monthly spend.

Correctness is equally important. Segments directly power campaigns that reach real merchants and customers. Even a small mismatch in segment membership can result in the wrong campaign being sent to users, impacting merchant trust and creating compliance risks.

Stage 1: The Segmentation DAG, from Rules to Membership Lists

What a Segment Definition Looks Like

A segment is a named, rule-based collection of user identifiers. Rules are expressed as JSONB configs of AND/OR/NOT conditions over derived attributes:

{
  "name": "seg_loyalty_nudge_candidates",
  "purpose": "internal_crm",
  "refresh_config": {
    "frequency": "daily",
    "ttl_days": 30
  },
  "rules": {
    "operator": "AND",
    "conditions": [
      {
        "attribute": "avg_order_value",
        "op": "eq",
        "value": 5000
      },
      {
        "attribute": "loyalty_enrolled",
        "op": "eq",
        "value": false
      }
    ]
  }
}

How the DAG Runs

A segment definition looks simple: a few conditions, a refresh cadence, and an output purpose. But executing it at scale is expensive. Every run may involve scanning hundreds of millions of profiles, joining across large attribute tables, applying consent filters, and generating outputs for multiple downstream serving systems.

The Airflow DAG runs daily and processes two categories of work:

• New segment requests from segment_requests
• Existing segments scheduled for refresh based on refresh_config.frequency

For each segment, the DAG first determines the minimum set of attribute tables required for execution. The EDW holds hundreds of derived attributes, many backed by large tables with hundreds of millions of rows. Loading all attributes for every segment run would unnecessarily increase Spark memory usage and shuffle overhead. To avoid this, the DAG parses the JSONB rule tree, extracts only the referenced attributes, and builds a targeted load plan for execution.

The Spark job evaluates the segment conditions and produces the final membership list of users matching the rules. The computed output is then written to S3 in a format based on the downstream serving requirement, and is encrypted at rest using a KMS key.

After successful generation, metadata such as segment status, output path, size, and refresh timestamps are updated in Postgres. The pipeline then emits segment.created or segment.updated events through SQS for downstream consumers.

Segment Reuse and Deduplication

Segment reuse was one of the most consequential decisions in the platform. It saved both compute cost and storage. Different teams often request the same segment with minor formatting differences in configuration. Without reuse, each request would independently trigger Spark computation, generate separate S3 outputs, and provision separate serving infrastructure.

To avoid this, the platform computes a deterministic hash for every segment definition. Before scheduling execution, the system checks whether an active segment with the same hash already exists. If a match is found, the computed dataset is reused instead of recomputing the segment.

The difficult part was canonicalisation. Semantically identical rules must generate the same hash even if their JSON structure differs. For example, A AND B and B AND A should resolve to the same segment identity. To achieve this, the pipeline normalises operators, sorts conditions deterministically, flattens nested logical groups, and standardises JSON ordering before hashing.

Data reuse and serving reuse are handled independently. If a matching segment already exists in S3 but a new consumer requires DynamoDB serving, the system creates a new serving configuration and loads the existing dataset into DynamoDB without rerunning the Spark computation. This separation allows the platform to minimise compute cost while still supporting different serving patterns for different consumers.

Stage 2: Segment Ingestion, Why We Split the Pipeline in Two

Every computed segment follows the same initial path: once the DAG finishes processing, a completion event is published to SQS. From there, the ingestion flow diverges based on the serving mode required by the segment.

S3-Backed Segments

For batch-only use cases, the ingestion flow is intentionally lightweight. The Segment Worker validates the event, updates the segment state in the database, and emits a segment.active event to Kafka for downstream consumption. Since these segments are served directly from S3 and do not require low-latency infrastructure, the processing completes within seconds without additional orchestration or long-running workflows.

DynamoDB-Backed Segments

Segments that require real-time membership checks follow a separate ingestion path. In this case, the Segment Worker starts a Temporal workflow keyed by segment_id.

The operational requirements here are very different from S3-backed segments. A DynamoDB import may involve loading millions of records from S3 into a newly provisioned table. Large imports can take several minutes and are vulnerable to transient failures such as worker crashes, S3 throttling, or DynamoDB capacity spikes. To handle this reliably, the workflow is broken into independently retryable activities:

1. Event validation. Verifies metadata such as segment ID, S3 path, record count, and checksums before ingestion begins.
2. DynamoDB import. Reads the computed membership dataset from S3 and writes it into a new versioned DynamoDB table.
3. Version promotion. Promotes the new table version only after import validation succeeds.
4. Lineage logging. Stores execution metadata, retries, timestamps, and audit information for operational visibility.

Zero-Downtime Refreshes Via Table Versioning

Each refresh creates a new DynamoDB table version:

• vN is the current live version
• vN-1 is the previous stable version

Older versions are cleaned up asynchronously. The active serving pointer is switched only after the new table is fully imported and validated. If an import fails midway, the existing live table continues serving traffic without interruption. This versioned approach avoids partial refresh visibility and ensures consumers always read from a stable dataset during segment refreshes.

Temporal also provides automatic retries with exponential backoff for transient infrastructure failures. If a worker crashes during execution, the workflow resumes from the last completed activity instead of restarting the full import.

Stage 3: Segment serving

At first glance, segment serving looks simple: store membership data in DynamoDB and query it during runtime. In practice, this layer has strict latency, isolation, and privacy requirements. Real-time membership checks are part of critical request paths such as ad serving and personalisation flows. Multiple segment lookups may happen within a single request, so the serving layer is designed to operate within a tight p99 latency budget of sub 30ms. [VERIFY: sub-30ms p99 claim]

One Table Per Segment

Each segment is stored in its own DynamoDB table: segment_<segment_id>_<unique_version_hash>.

This design isolates traffic and operational behaviour across segments:

• High traffic on one segment does not impact others.
• Capacity can be provisioned independently per segment.
• TTL, refresh lifecycle, and cleanup remain local to the segment.
• Failures during ingestion or refresh stay isolated to a single segment.

More tables means more operational overhead. The isolation was worth it.

Privacy-Preserving Membership Checks

The partition key for every table is the SHA-256 hash of the user's phone number. During lookup, the caller sends a pre-hashed identifier and receives a boolean response indicating whether the user belongs to the segment. The serving layer never receives or stores raw phone numbers. PII does not leave the originating system and is never written into DynamoDB.

This effectively turns the serving layer into a membership lookup system that answers: "Is user X part of segment Y?" without needing access to the actual identity of the user. This was a deliberate architectural decision to reduce the compliance and security surface area of the system.

Multiple Serving Paths From the Same Segment

A single computed segment can support multiple downstream serving modes simultaneously.

For example:

• A campaign system may use DynamoDB for low-latency eligibility checks.
• A partner integration may consume the same segment as a batch export from S3.

The platform tracks these independently using the segment_servings table in Postgres. This separates segment computation from segment delivery and allows different consumers to use the same underlying dataset without triggering duplicate computation.

The Estimation Engine: Fast Audience Sizing With Theta Sketches

One of the biggest usability challenges in the platform was segment estimation. Before the estimation engine existed, the only way to know the size of a segment was to run the full computation pipeline. That meant executing Spark jobs on the EDW, generating outputs, and waiting for the final count. Even a small change in segment conditions required rerunning the workflow, which slowed down experimentation significantly.

The goal of the estimation engine was simple: give teams a fast and reasonably accurate estimate of segment size before triggering the actual computation pipeline.

Why We Chose Theta Sketches

Approximate cardinality estimation is a well-known problem, and HyperLogLog (HLL) is often the default choice because of its low memory footprint. However, our segment system required more than simple cardinality estimation.

Most segment definitions involve combinations of AND, OR, and NOT. For example: (state = 'Maharashtra' OR state = 'Karnataka') AND NOT (loyalty_enrolled = true).

This requires unions, intersections, and set differences across user groups. While HLL handles unions efficiently, intersections and differences rely on inclusion-exclusion approximations, where errors compound quickly for nested conditions.

Theta Sketches support these set operations directly with predictable error bounds, which made them a better fit for segment estimation.

Building the Sketch Store

During attribute generation, the platform creates a Theta Sketch for every attribute-value pair in the catalogue using the Apache Datasketches library.

Examples:

• state = 'Maharashtra'
• loyalty_enrolled = false
• payment_method = 'UPI'

Each sketch contains hashed user identifiers matching that condition. These sketches are stored alongside the attribute data and refreshed periodically through batch recomputation jobs. This shifts the expensive work to an offline preprocessing step so that estimation queries do not require scanning large attribute tables at runtime.

Query-Time Estimation

When a team creates or edits a segment, the estimation engine parses the segment rules into a logical tree. Instead of loading raw attribute data, the engine fetches the required Theta Sketches and evaluates the tree using set operations.

The final sketch returns an estimated cardinality for the segment. Since the computation happens entirely on compact probabilistic summaries, estimates are returned within seconds without triggering Spark jobs or EDW scans.

For larger segments, the observed estimation error stays within a small range and is accurate enough for audience planning and campaign sizing decisions. For a Theta Sketch with k = 4096, the expected relative standard error is 1/√k ≈ 1.56%. In practice, for segments with more than a few thousand members, estimates land within ±3% of the true count 95% of the time. For very small segments (fewer than k distinct elements), the sketch holds exact members and returns a precise count.

Handling Drift

Theta Sketches are periodically rebuilt from the latest attribute datasets. As user attributes evolve over time, incremental updates alone can introduce drift between the sketches and the actual population. To maintain estimation accuracy, the platform runs scheduled rebuild jobs that regenerate sketches from the latest attribute store snapshots.

Impact on Segment Creation

The estimation engine changed the segment creation workflow from a long-running batch process into an interactive experience. Teams can now modify segment conditions, preview audience sizes in near real-time, and refine targeting before triggering the actual computation pipeline.

This reduced unnecessary Spark executions while making segment iteration significantly faster for internal teams.

What We Would Say to Other Teams Building This

A few lessons that aren't specific to customer data platforms.

Compute reuse beats compute optimization. A 30% faster Spark job is a one-time win. Detecting that the job doesn't need to run at all is a permanent one. Segment reuse through deterministic hashing was the single biggest cost lever in the platform. Build canonicalisation carefully and the savings compound forever.

Hash before you store. Privacy-preserving membership checks aren't just a compliance feature; they're an architectural simplification. The serving layer has nothing sensitive to protect because it never had the sensitive data in the first place. The cost is one extra hash on the caller side. The benefit is removing the entire serving layer from the PII blast radius.

Approximate is usually enough. The estimation engine using Theta Sketches turned a multi-hour iteration loop into a sub-second one. Most audience planning decisions don't need exact counts; they need confidence intervals. If your platform has a slow-feedback loop blocking experimentation, ask whether the consumers actually need exact answers.

Closing Thoughts

The platform continues to evolve as we expand attribute coverage, improve refresh frequency, and move toward near real-time membership updates. We are also working toward making segment creation accessible to non-technical teams through self-service tooling.

Many of the architectural decisions in the platform were driven by scale, latency, and compliance constraints. The same patterns (approximate estimation using sketches, durable ingestion workflows, and privacy-preserving serving systems) are applicable beyond customer segmentation and can be useful in any large-scale audience or membership platform.

As the system grows, the challenge is no longer just computing segments efficiently, but building infrastructure that remains reliable, flexible, and operationally manageable as new use cases emerge.

Editor: Parth Sawhney

Turning Scattered Data Into Queryable Segments at Scale: How Razorpay Built Its Customer Data… was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Co-authors: Mahlaqahaque Mh, Keertiv, Hari Prasad Pujari

How we taught AI to read code like a senior security engineer

Every day, Razorpay engineers ship thousands of lines of code. Every line births new security findings. SAST scanners flag suspicious patterns. Dependency checkers find vulnerable libraries. Secret detection tools catch hardcoded credentials.

The alerts pile up. Hundreds become thousands. The backlog becomes noise.

We hit a breaking point. Developers faced security ticket counts climbing into the thousands, with most of them turning out to be false positives. The classic “alert that cried wolf” scenario played out daily. When everything is marked critical, nothing is. Developers stopped trusting security findings altogether.

Security engineers weren’t having a better time. Validating issues manually while fielding constant ad-hoc requests from frustrated developers. Trying to stop a waterfall with a teaspoon. The human bottleneck became the limiting factor in our security posture.

The core problem was simple. Traditional static analysis tools excel at finding patterns that might be vulnerabilities. They lack context. They can’t distinguish between a properly sanitized SQL query and a vulnerable one. Between a test API key and a production secret. Between a dangerous data flow and one protected by business logic.

For every 10 alerts, 7–8 were false positives. Manual triage became the bottleneck. Security couldn’t scale with engineering velocity.

That’s when we built what we call the Autonomous Security Special Ops system. An AI-powered engine that handles the heavy lifting so humans can focus on what actually matters.

The Three-layer Intelligence System

Rather than throwing more human hours at the problem, we built an AI architecture operating in three layers.

L1: Context-Aware AI Triage (Live). Our intelligent first responder. Powered by 29 specialized sub-skills , it reads code context like a senior security engineer. When a SAST finding lands, L1 fetches the issue via Semgrep APIs, pulls source code from the repository through a GitHub fine-grained token, and analyzes the full context — not just the flagged line but the surrounding code, data flows, and sanitization logic. Then it validates: is this a genuine vulnerability or a false positive? Valid issues route to L2. False positives feed into rule tuning. Current accuracy: 75–80%.

L2: Autonomous Remediation Bot (Live). When L1 confirms a valid finding, [VERIFY: Vyom / Slash] doesn’t just flag it. It invokes specialized security plugins for SAST, Software Composition Analysis, and hardcoded secrets. It generates secure code fixes. It creates pull requests with remediation. It logs everything for audit trails and pushes metrics to our dashboards. The bot operates continuously, processing new issues in real-time as they’re created and systematically working through the existing backlog by criticality.

L3: Scanner Auto-Tuning (Coming Soon). This layer will extract patterns from false positives identified by L1, fine-tune Semgrep rules in real-time, update our rule registry automatically, and create a continuous feedback loop. L3 deploys once L1 accuracy reaches 90%+. A self-improving security system that gets better at distinguishing signal from noise with minimal human intervention.

Human vs. Bot: The Math

The economics tell the story.

Roughly 960x speed-up. The throughput number is the headline, but consistency matters as much in practice. Manual review at 800 findings deep on a Friday afternoon is not the same review as finding #1 on Monday morning. The AI applies the same rigorous analysis to finding 23,000 as it did to the first one.

How L1 Actually Works (and how to build your own)

A few design decisions made the difference between demo and production.

Progressive disclosure for skill loading. We only load relevant sub-skills (2–4 per finding) instead of all 29. This reduced tokens per analysis from roughly 50K to 2.5K, which directly cuts cost and reduces hallucinations from context overload. More context isn’t always better. The right context, scoped tightly, almost always is.

A skill router on metadata. Finding metadata (rule ID, file type, framework) routes to the right skills. Multiple skills then run in parallel, with results aggregated by weighted consensus. A SQL injection finding in a Go HTTP handler loads different skills than the same rule firing in a Python data pipeline.

Grounded verdicts only. The AI must pull actual source code (50–100 lines of context) before any verdict. No source code, no verdict. This prevents speculation, the failure mode where the model produces a confident-sounding analysis based on its priors rather than the actual code in front of it.

Confidence-gated automation. Only verdicts above 85% confidence trigger auto-remediation. Lower-confidence findings go to human review. This isn’t a limit on the AI; it’s a contract with the security team about what gets to ship without their eyes on it.

Specialization over generalization. We started with a general-purpose code analyzer. It hit ~60% accuracy. Mediocre at everything. When we broke it into 29 specialized sub-skills (each tuned for specific vulnerability classes, frameworks, and code patterns), accuracy jumped to 75–80%. Security knowledge needs to be deep, not just broad.

For anyone building similar systems: the model isn’t the hard part. The skill architecture is.

How the Pipeline Holds Together: Harness

Production AI workflows have the same requirements as any other production workflow. Idempotency. Retries. Observability. Secret management. Auditability of every decision.

Harness is our orchestration backbone for this system. It gives us:

• Pipeline-as-code: Every triage run versioned, auditable, reproducible
• Native cron + webhook triggers: L1 runs every 6 hours; L2 reacts to new issues in real-time
• Conditional execution: Routing between L1 verdicts, L2 remediation, L3 tuning
• Built-in observability + RBAC: Critical for security workloads where every AI decision must be traceable
• Human evaluation in the loop: All triaged results produced by the skills, including Semgrep rule metrics and per-rule issue counts, are reviewed by a human before any action is taken

Without this infrastructure layer, an AI agent making security decisions in production is a liability. The model can be world-class and the system can still fail in ways nobody can debug.

The stage-by-stage evolution

The current system didn’t emerge fully formed. It evolved through four distinct stages, each unlocking a different operating model.

*Manual TP rate reflects what humans confirmed valid, but with significant fatigue-driven inconsistency.

The improvements aren’t just incremental. Each stage changed what the security team could do. Manual review forced sampling. Rule-based filters reduced noise but couldn’t read context. The AI agent could read context but couldn’t yet learn from disagreements. The feedback loop closes that gap.

How the System Gets Smarter

This is where it goes from useful to compounding.

The mechanism is straightforward but the details matter:

1. The AI agent triages a finding and posts its verdict directly on the DevRev ticket.
2. If a security engineer disagrees, they add a comment explaining why. “This is a real issue because input flows to eval() after a sanitization bypass.” Or: “This is FP because the framework auto-escapes here.”
3. The AI agent reads comments on the ticket on the next run.
4. The reasoning gets converted into either an update to an existing skill or a brand-new skill.
5. The next run applies the new knowledge across all future findings of that type.

The critical safety detail: comment access is restricted. The AI only reads comments from a trusted allowlist of security engineers and approved reviewers. Comments from other users, developers, PMs, anyone outside the loop are ignored entirely. This prevents skill drift. A developer can’t trick the AI into marking real vulnerabilities as false positives by leaving misleading comments on a ticket they want closed.

The result is that tribal knowledge gets captured once and applied at scale. A senior security engineer’s reasoning about a tricky framework-specific edge case becomes a skill that every future finding benefits from. Month-over-month, accuracy climbs. Quarter-over-quarter, FP rates drop.

Most AI security tools are static. They ship with a fixed set of behaviors and stay that way until the next vendor release. Ours improves every week without retraining the underlying model. The feedback loop is the difference between a tool that helps once and a system that compounds.

What We Learned

A few lessons that aren’t specific to security.

Context is everything, and less is often more. Feeding more context doesn’t help. Feeding the right context does. Progressive disclosure was the single highest-leverage decision we made.

Specialization beats generalization. A general security analyzer is mediocre at everything. 29 specialists are excellent at their specific domain. Anyone building agentic systems on production data should resist the urge to build a single “do everything” agent. The performance ceiling is much lower than you’d expect.

Production infrastructure matters as much as the model. Without Harness’s orchestration, observability, and audit trails, an AI agent making decisions in production is a liability. The model can be excellent and the system can still fail in ways nobody can debug or remediate.

Feedback loops are the compounding asset. A one-time accuracy improvement is a fix. A learning system is an asset. The difference is the feedback loop architecture, not the model.

Instead of spending hours triaging obvious false positives, our security team now focuses on reviewing edge cases the AI flags as uncertain, updating skills as new vulnerability patterns emerge, making strategic security architecture decisions, and responding to actual incidents.

Open Questions

How much context is enough? We keep context tight today. But as models get better and we push accuracy higher, that number will change. We’re still figuring out the right balance.

On the feedback loop: The system learns from security engineer comments, but how do you prevent skill drift over time as edge cases accumulate? Quality control on the skills themselves is still evolving.

On accuracy: How do we get from 75–80% to 95%+? The path isn’t fully clear yet; more skills, better training data, or both (we’re actively working on both).

Cross-repo and transitive context remains an open problem. The AI currently operates within a single repository at a time, which means a vulnerability in a shared SDK can silently propagate to dozens of downstream services, this remains an active area of work.

Beyond SAST: The Expanding Scope

We’re not stopping at static analysis.

The same architecture that conquered SAST is now being trained on Software Composition Analysis. Identifying vulnerable dependencies. Understanding transitive risk. Prioritizing which CVEs actually matter in our specific usage context rather than treating every CVSS score as gospel.

And on secrets detection. Moving beyond regex patterns to understand what actually constitutes a dangerous secret versus a test API key or a public configuration token. The same skill architecture, retrained for a different vulnerability class.

The platform is extensible by design. Each new security domain gets its own specialized sub-skills. The three-layer pattern (triage, remediation, auto-tuning) applies universally. The feedback loop mechanism works the same way regardless of vulnerability class.

The Bigger Shift

Traditional security models assumed small, controlled codebases with limited changes. Security reviews were gates. Necessary friction points that slowed things down.

That model broke when engineering teams grew to hundreds of developers shipping multiple times daily.

The new model is autonomous, continuous, and learning. Security isn’t a gate. It’s an intelligent system running in parallel with development. Automatically identifying real risks. Fixing what can be fixed. Tuning tools to reduce noise. Escalating only what needs human judgment.

A one-person security team can now manage what used to take five. Not because the work got smaller, but because the work got smarter.

AI doesn’t replace security engineers. It makes them exponentially more effective by handling the mechanical parts that don’t require human creativity, then learning from their judgment every time it gets one wrong.

The Autonomous Security Special Ops system is live at Razorpay. L1 triage runs every 6 hours. L2 remediation processes new issues in real-time. L3 auto-tuning deploys once L1 accuracy reaches 90%+. The feedback loop runs continuously, with every disagreement between a security engineer and the AI becoming a skill update for the next cycle.

Next month will be better than this month. Next quarter will be better than this quarter. That’s the power of systems that learn rather than systems that just execute.

Editor: Parth Sawhney

From 750 Hours to 2 Hours: AI-Powered Security Triage at Razorpay was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our on-call engineers were spending 30 minutes investigating every production alert. Here’s what happened when we automated it.

At 3 AM, alerts don’t care about your sleep schedule.

When our payment infrastructure threw an error last month, our on-call engineer spent 32 minutes jumping between six different monitoring systems before understanding what was broken. One tool for metrics. Another for logs. Third tool for pod health. And multiple more for infrastructure, deployment history and database health.

By the time they identified the root cause (a bad deployment), payment failures had already impacted customers for nearly 40 minutes.

This wasn’t their fault. They followed our runbook perfectly. The problem was that no single system could tell them “here’s what’s wrong and why.” They had to manually connect dots across disconnected observability tools.

That’s when we asked ourselves: what if AI could do this investigation for us?

The Metric Nobody Optimizes For

The SRE world talks endlessly about Mean Time to Detect (how fast you catch problems) and Mean Time to Resolve (how fast you fix them).

But there’s a critical phase hiding between them: Mean Time to Investigate.

MTTI is the gap from “we know it’s broken” to “we know what to fix.” At Razorpay, this phase was consuming 20–40 minutes per incident. With 15–20 incidents weekly, that’s 6–8 hours of engineering time spent doing repetitive investigative work.

Worse, the quality was inconsistent. Senior engineers knew exactly which systems to check for payment alerts. Junior engineers sometimes checked irrelevant dashboards or missed critical correlations. The investigation depended entirely on who was on-call that night.

What We Built (And Why It Works)

Razorpay Oncall Agent is a multi-agent AI system that automates incident investigation. The architecture is built on LangGraph, a framework for creating stateful workflows with conditional logic, and uses LLM as the reasoning engine.

Here’s how the components work together. A Supervisor Agent acts as the incident commander, receiving alerts from an alerting tool and creating an investigation strategy. It queries our RAG (Retrieval-Augmented Generation) systems for context: one RAG stores application architecture and dependencies, another stores alert-specific diagnostic runbooks. This contextual grounding ensures investigations aren’t generic but tailored to our specific services and known failure patterns.

The Supervisor then dispatches tasks to Specialist Agents that operate in parallel. The Kubernetes Agent checks pod health and recent deployments. The Observability Agent analyzes error logs. The PromQL Agent queries performance metrics. The AWS Agent validates infrastructure health. Each agent is domain-focused, executing specific checks within 5–8 seconds.

As agents complete their investigations, findings get stored in Memory as structured evidence: what was checked, what was found, confidence levels, and supporting data. The Supervisor correlates all evidence, builds an Incident Evidence Timeline ordering events by timestamp, scores multiple hypotheses based on temporal correlation and evidence strength, and selects the most likely root cause. The entire process completes in under 90 seconds, posting a structured analysis to Slack with citations, confidence scores, and recommended actions.

The system learns continuously. Every investigated case gets stored back into the RAG systems, enriching future investigations with institutional knowledge. This compounding effect means the system gets more accurate over time as it encounters more incident patterns.

The on-call engineer still gets the 3 AM alert. But now it comes with a complete investigation already done.

The Business Impact

Three months into shadow mode, the results are compelling:

80% MTTI reduction. Investigations that took 30 minutes now complete in 90 seconds. That’s 25 minutes saved per incident.

50–60% MTTR improvement. Faster investigation means faster resolution. Less customer impact, less revenue at risk.

Consistency across the team. Junior engineers now receive the same quality analysis as senior engineers. Knowledge doesn’t walk out the door when someone leaves.

Reduced on-call stress. Engineers report feeling more confident taking alerts because they know the AI will do the initial investigation.

The time savings compound. Six to eight hours of engineering time saved weekly. That’s time our SRE team now spends on prevention: improving monitoring, building resilience, reducing the incidents that fire in the first place.

The Leadership Lesson

Here’s what building Oncall Agent taught us about AI in operations.

Specialize, don’t generalize. We tried building one AI that understood everything. It failed. Breaking the problem into specialist agents that collaborate worked beautifully. This mirrors how expert teams actually operate.

Start with the painful, repetitive work. We didn’t start by automating complex edge cases. We automated the investigation pattern engineers repeat dozens of times weekly. The 80/20 rule applies to automation too.

Build for continuous improvement. Every incident Oncall Agent investigates feeds back into its knowledge base. The system gets smarter with every case it processes. That compounding effect is what makes AI systems genuinely valuable over time.

The Question I Keep Asking

Is this the future of incident response?

I think so. Not because AI will replace SRE teams (it won’t), but because the complexity of modern distributed systems is growing faster than our ability to hire and train people who can understand them.

We need systems that can do the repetitive investigative work so humans can focus on the genuinely complex problems that require creativity, judgment, and experience.

That’s not replacing engineers. That’s letting them be engineers.

Razorpay Oncall Agent is currently in shadow mode at Razorpay, providing 90-second root cause analyses for production incidents. We’re targeting 80%+ accuracy and plan to roll out to production after validating across hundreds of incident cases.

Leading engineering teams at scale? I’d be curious to hear how you’re thinking about AI in operations. What metrics matter most to you? Where are you seeing the biggest time sinks?

Drop your thoughts in the comments.

Editor: Parth Sawhney

Razorpay Oncall Agent: From 30-Minute Investigations to 90-Second AI Analysis was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve ever tried to use a gift card for an online purchase, you know this frustration intimately.

The gift card covers most of it, but not quite all. The checkout forces you to choose: use the gift card and abandon some items, or ignore the gift card and pay the full amount another way. Either choice feels wrong.

This isn’t a technical limitation of payment processing. It’s an architectural one. Most payment systems treat each method as an isolated, complete transaction. You pay with a card OR UPI OR a gift card. The concept of composing multiple methods to fulfill a single order simply doesn’t exist in traditional payment gateway architectures.

At Razorpay, this limitation was costing merchants real money. Gift card redemption rates suffered because customers abandoned partial-value cards. Average order values stayed lower because customers couldn’t combine store credit with additional payment. The business case for solving this was clear.

That’s why we built Linked Payments, a system that treats payment methods as composable building blocks. Customers can now use a gift card for $50, then cover the remaining $25 via card, UPI, or any other method. The system handles authorization sequencing, failure recovery, and settlement splitting automatically.

The Complexity Hidden in “Just Combine Them”

The challenge sounds simple until you consider what payment systems actually do.

Traditional payment flows are beautifully simple. Customer initiates payment. System authorizes the full amount from one method. If authorization succeeds, capture the funds. Settle to the merchant. Either the payment worked or it didn’t. One authorization, one capture, one settlement.

Linked payments shatter this simplicity.

Now you have multiple authorizations for a single order. Sequential dependencies where the second payment only happens if the first succeeds. Partial failure scenarios where you need to handle one method succeeding while another fails. Split settlements where money comes from different sources.

Here’s what makes this architecturally challenging at Razorpay: each payment method has its own microservice and its own database. Gift cards are managed by one service, UPI by another, cards by yet another. In a monolithic architecture, coordinating sequential payments would be straightforward with database transactions. But in a microservices architecture, maintaining consistency across independent services with separate databases requires explicit orchestration, distributed state management, and careful handling of partial failures.

The critical design decision was this: we don’t capture any payments until all authorizations in the chain succeed. This two-phase approach (authorize all, then capture all) elegantly handles failure scenarios. If the gift card authorizes but UPI fails, we only have authorization holds, not actual captures. The system simply reverses the authorizations, and critically, these reversals don’t incur charges to merchants because no money was captured. The customer can retry with a different combination.

The Architecture That Makes It Invisible

Despite internal complexity, we designed LinkedPayments to be completely transparent to merchants. Here’s one of the key architectural advantages: merchants don’t need to create special orders for linked payments.

This matters because you never know upfront whether a customer will combine payment methods or use just one. Forcing merchants to anticipate this during order creation would be impractical.

Instead, merchants create orders exactly like any normal payment. Just amount and currency. That’s it.

The magic happens dynamically at checkout. When customers reach the payment page and select “Use Gift Card + Another Method,” the LinkedPayments flow activates automatically. The system handles the entire chain without merchants having anticipated this during order creation. Everything updates dynamically based on the customer’s actual payment preference.

Webhook events follow a similar principle of simplicity. Rather than sending merchants a confusing stream of partial updates, we withhold authorization webhooks until all payments in the chain are authorized. This prevents merchants from receiving premature notifications suggesting the order is ready when subsequent payments could still fail. Only after all authorizations succeed do merchants receive notifications. This design choice simplifies merchant webhook handling dramatically.

What Happens When Things Go Wrong

Payment systems are distributed. Failures happen. LinkedPayments needed to handle these gracefully.

The two-phase authorization pattern handles most failures elegantly. If any authorization in the chain fails, no money has been captured yet. The system reverses authorization holds and the customer retries. No charges, no complicated refund logic.

For refunds on successful orders, the system provides merchant flexibility. Merchants can choose to refund from the gift card, from UPI, or split the refund across both methods based on their business policies or customer preferences. The system calculates the distribution, triggers appropriate reversals, and updates settlement records.

Idempotency ensures network issues don’t cause double-processing. Every authorization, capture, and reversal uses idempotency keys. Retries naturally use the same key, making operations safely repeatable.

The state machine provides reliability. Each payment progresses through well-defined states (Created, authorized, captured, refunded) with explicit transitions. When something gets stuck, you can see exactly where in the flow it stopped and what the next valid transitions are.

The Part That Just Works

Here’s one of the most elegant design choices: reconciliation and settlement work out-of-the-box without requiring any changes.

The system was architected so each linked payment behaves like an independent payment from the settlement perspective. When $50 comes from a gift card and $25 from UPI, existing settlement infrastructure treats them as two separate transactions associated with the same order. The gift card portion settles according to existing gift card rules. The UPI portion settles through standard UPI flows.

Merchants receive settlement reports that automatically break down amounts by payment method using the same reporting infrastructure they already use. No new report formats, no special reconciliation processes, no integration work.

This design choice meant we could ship LinkedPayments without merchants updating their financial operations, accounting integrations, or reconciliation workflows.

The Real-World Impact

The business impact of LinkedPayments is currently focused on gift card programs, where we’ve seen measurable improvements.

Gift card redemption rates increased. Customers who previously abandoned partial-value cards now use them confidently, knowing they can combine with other methods. This drives loyalty and repeat purchases.

Average order values grew for merchants offering gift cards. When customers can apply gift card balances as partial payment, they’re more willing to make larger purchases. The psychology of “I’m already getting $50 off” encourages adding more to the cart.

Payment success rates improved through flexibility. If a card payment fails, customers can try splitting it with a gift card and smaller card payment. This converts failed checkouts into successful orders.

However, LinkedPayments was designed to support any combination of payment methods. The same infrastructure enabling gift card + UPI can support store credit + card, wallet + netbanking, corporate credit + personal card, or multiple cards for high-value purchases. These use cases represent future expansion opportunities as we validate the architecture with current gift card adoption.

The Composability Lesson

LinkedPayments demonstrates a broader principle about platform design: composability multiplies capability.

When you design payment methods as modular, chainable units rather than isolated silos, you enable use cases you didn’t originally anticipate. We’ve already seen merchants exploring scenarios we never explicitly designed for: corporate expense reimbursements, subscription payments with account credit, group purchases with multiple contributors.

This composability emerges naturally from the architecture. Because the orchestration logic treats payment methods uniformly and the state machine handles arbitrary sequencing, any combination of supported methods “just works” without special-case implementation.

The lesson applies beyond payments. When building platform capabilities, investing in composability early pays dividends. Each new composable unit doesn’t just enable one new use case; it enables N new combinations with existing units. The value grows combinatorially rather than linearly.

When Architecture Decisions Compound

Building LinkedPayments taught us that the right architectural choices create long-term leverage.

Event-driven coordination through Kafka and order_meta allowed sequential payment processing while maintaining loose coupling between microservices.

Two-phase authorization (authorize all, then capture all) eliminated an entire class of complex failure scenarios.

Out-of-box settlement meant zero merchant integration work, dramatically lowering adoption barriers.

These weren’t just technical decisions; they were product decisions about how to build systems that merchants could trust and adopt easily. The technical elegance enabled the business outcome.

LinkedPayments is currently powering gift card combinations at Razorpay, with architecture designed to support any payment method pairing. The system continues to evolve as we explore additional use cases and gather merchant feedback on the composability model.

Building payment infrastructure or fintech platforms? The balance between system complexity and user simplicity is always interesting. What approaches have worked for your architecture?

Editor: Parth Sawhney

The Checkout Frustration Razorpay Fixed: Combining Payment Methods was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

You’re running a loyalty program. Customers are redeeming points for gift cards, ongoing offers, memberships, or experiences. Then your provider’s API times out.

You refresh. Try again. Out of stock for the denomination your customer wanted. You check another provider, but you’re not integrated with them. Your customer service team starts getting calls. The loyalty program meant to drive engagement is now creating frustration.

This scenario plays out repeatedly across fintech platforms, and it reveals a fundamental architectural problem. Most gift card platforms are tightly coupled to a single provider. When that provider fails, runs out of stock, or becomes unavailable, your entire rewards program stops. There’s no fallback, no intelligent routing, no abstraction layer handling the complexity.

At Razorpay, we power rewards programs for banks running credit card loyalty, wallet apps offering cashback, corporate gifting platforms, and incentivized advertising. A single-provider architecture couldn’t meet the diverse requirements: instant vouchers for loyalty programs (under 3 seconds), bulk fulfillment for corporate gifting, and high-volume automated distribution for ad rewards.

That’s why we built Rewards Marketplace, treating rewards distribution as an e-commerce problem rather than a simple API integration. Just like Amazon abstracts multiple sellers behind a single product listing, we abstract multiple providers behind a single reward offering. The result: 60% of orders fulfilled in under 1 seconds, zero visible stock-outs, 99.95% success rate, and seamless multi-provider routing.

The Multi-Provider Challenge

Building a rewards platform sounds straightforward until you encounter the real-world complexities.

Provider reliability varies dramatically. One provider offers 99.5% uptime but charges premium rates. Another is cheaper but fails 3% of the time during peak hours. A third has a limited catalog but excellent API performance. If you’re locked to one provider, their limitations become your ceiling.

Customer expectations don’t match provider capabilities. Loyalty programs need instant delivery (under 3 seconds). Corporate gifting can tolerate 3–5 minute fulfillment for bulk orders. These different SLA requirements mean you need both synchronous and asynchronous fulfillment paths based on order characteristics and provider capabilities.

Inventory visibility is opaque. Providers rarely expose real-time stock levels. Customers discover out-of-stock items only during redemption. Worse, providers frequently disable denominations without notice. The ₹500 Amazon card that was available yesterday might be discontinued today, and you won’t know until an order fails.

Overselling is a real risk. When multiple customers request vouchers simultaneously and you’re managing pre-purchased inventory, you need atomic claiming to prevent double allocation. Without proper concurrency control, you risk selling the same voucher twice.

Reconciliation gets complex when you’re buying from multiple providers at different rates and settling on different timelines. Provider A settles at T+3, Provider B at T+7. Tracking purchases, sales, and settlements without leaking provider-specific logic to client systems requires careful abstraction.

These aren’t edge cases. They’re the fundamental challenges of multi-provider distribution at scale.

The E-Commerce Insight

The breakthrough came from recognizing that rewards distribution mirrors e-commerce marketplace patterns.

Amazon doesn’t expose which seller fulfilled your order. You search for “wireless mouse,” see a product listing, place an order. Behind the scenes, Amazon routes to the optimal seller based on price, availability, shipping speed, and reliability. If one seller is out of stock, another fulfills it seamlessly.

We applied the same pattern to rewards. Customers see “Amazon Gift Card ₹500.” Behind the scenes, we route to the optimal provider based on inventory availability, API performance, cost, and current provider health. If one provider fails, another handles it automatically.

This abstraction required building three distinct layers, each solving specific problems.

Layer 1: The Catalog (Product Discovery)

The catalog provides a provider-agnostic view of available rewards organized hierarchically: Brand (Amazon, Flipkart, Netflix) contains Rewards (Gift Card, Subscription), which have Variants (₹500 vs ₹1000, duration, online vs offline).

Provider-agnostic availability is the key feature. The catalog shows “Available” as long as ANY fulfillment source can deliver, whether that’s pre-purchased inventory or any connected provider. Users never see provider-specific failures like “Qwikcilver out of stock.” The system internally routes to alternatives.

Real-time inventory synchronization runs every 6 hours, fetching catalog and stock status from all providers. When a provider disables the ₹500 denomination, the sync job detects this and marks the variant unavailable only if no other provider offers it. This keeps the catalog fresh without manual intervention.

Extensibility means new brands or reward types get added without code changes. When a provider adds “Zomato” to their catalog, the next sync cycle automatically creates the brand hierarchy and makes it available for ordering immediately.

Layer 2: The Marketplace (Order Orchestration)

The marketplace layer orchestrates the complete order lifecycle while handling the messy reality of partial failures and multi-entity coordination.

Order and item state machines track granular status. Orders have states (INITIATED, PAYMENT_PROCESSING, PROCESSING, SUCCESS, PARTIAL_SUCCESS, FAILED). Individual order items have their own states (INITIATED, PROCESSING, SUCCESS, FAILED, REFUND_INITIATED, REFUNDED). This separation enables delivering what worked while refunding what failed.

Here’s why PARTIAL_SUCCESS matters. In a multi-provider world, failures are independent. One provider might succeed while another times out. A customer orders 5 gift cards. Three succeed, two fail. With PARTIAL_SUCCESS, we deliver the three immediately and automatically refund the two that failed. The customer gets immediate value rather than waiting for the entire order to retry.

Payment strategy orchestration handles different payment flows transparently. Customers can pay via Razorpay Payment Gateway (card, UPI, netbanking), loyalty points through wallet integration, or receive free rewards through direct allocation. The marketplace abstracts these differences, applying the appropriate payment strategy based on order context.

Reconciliation and financial tracking happen through a provider-agnostic ledger tracking wallet debits, credits, manual transactions, and direct allocations. This enables reporting and auditing across multiple providers without exposing provider-specific settlement logic to client systems.

Layer 3: Fulfillment and Procurement

SKU availability sync operates on two layers to keep catalog state accurate at all times. The first is a daily reconciliation job that checks each provider for newly added SKUs and disabled denominations, ensuring the catalog reflects the latest provider catalog state. The second is an event-driven layer, providers that support webhooks push denomination change events (stockouts, disables, re-enables) in real time, and the system processes these immediately to update catalog availability. For providers that do not support webhooks, we compensate with periodic polling to ensure the catalog stays current. Together, these layers ensure users are never shown an incorrect denomination state, catching mid-window changes that a daily sync alone would miss.

Inventory-first routing via micro-procurement checks pre-purchased vouchers before calling external providers. Rather than maintaining large static inventory pools, we use micro-procurement, procuring vouchers in small batches aligned to actual demand signals. This approach delivers two key advantages: first, if a provider’s real-time API is down, we can still serve orders from the micro-procured buffer; second, latency stays low since fulfillment hits local inventory (~100ms) rather than waiting on an external provider call. The limitation is finite stock requiring a replenishment strategy, though capital requirements are lower than bulk pre-procurement.

Procurement layer handles on-demand fulfillment when inventory is insufficient. For small orders (under configured threshold), it calls providers synchronously, returning vouchers in 2–8 seconds. For bulk orders, it queues async procurement jobs that complete in 60–120 seconds. The system calls providers to get the reward.

Concurrency control uses pessimistic locking during inventory claiming. When multiple customers request vouchers simultaneously, SELECT FOR UPDATE locks relevant rows until the transaction commits. This prevents overselling, with lock duration typically under 50ms for standard claims.

The Results That Matter

The architectural approach delivered measurable improvements across customer experience, merchant operations, and platform reliability.

60% of orders fulfilled in under 1 second through the inventory fast path. Customers receive instant gratification for loyalty redemptions rather than waiting for provider APIs.

40% increase in loyalty program enrollment for merchants using the platform. Reliability builds trust. When customers know redemptions work consistently, they engage more with loyalty programs.

80% reduction in operational toil through auto-replenishment and sync jobs. Teams no longer manually monitor inventory or coordinate provider integrations.

100% prevention of overselling through atomic inventory locking

Zero visible stock-outs for customers. Catalog shows “Available” as long as any provider has inventory, eliminating the “out of stock” experience common in single-provider systems.

The platform now handles diverse use cases on unified infrastructure: instant loyalty redemptions, bulk corporate gifting, and automated ad reward distribution, all using the same abstraction layers.

The E-Commerce Pattern Beyond Rewards

The interesting insight is how broadly this pattern applies.

Any domain with multiple providers or suppliers, variable availability and pricing, different SLA requirements across use cases, and need for reliability through redundancy can benefit from marketplace patterns: catalog abstraction, intelligent routing, inventory optimization, and graceful degradation.

We’ve seen similar patterns work for payment gateway selection (routing transactions across multiple payment providers), shipping provider coordination (choosing carriers based on cost and delivery speed), and even cloud infrastructure (multi-cloud strategies with provider abstraction).

The key architectural principles transfer: separate product discovery from fulfillment, abstract provider differences behind unified interfaces, handle partial failures gracefully, optimize through smart routing and inventory, and design for provider additions without client changes.

When Abstraction Creates Resilience

Building Rewards Marketplace taught us that the right abstractions don’t just hide complexity; they enable capabilities impossible with direct integration.

Adding new providers requires zero client-side changes. The catalog, marketplace, and fulfillment layers abstract provider differences. Clients continue using the same APIs regardless of backend provider changes.

Different use cases share infrastructure. The same platform serves instant loyalty redemptions and bulk corporate gifting through smart routing and dual fulfillment paths (sync/async).

Partial success handling delivers value even when some providers fail. Customers get what’s available immediately rather than waiting for retry of the entire order.

The marketplace pattern transformed rewards distribution from fragile single-provider integration to resilient multi-provider platform. That resilience unlocks business capabilities that weren’t possible before: ambitious SLA commitments, diverse use case support, and reliable scale.

Rewards Marketplace powers gift cards, vouchers, and incentive distribution at Razorpay across loyalty programs, corporate gifting, and advertising platforms. The architecture continues to evolve as we add providers and expand reward types supported.

Building marketplace platforms or multi-provider integrations? The abstraction patterns and resilience strategies are universal. What challenges are you solving in your architecture?

Editor: Parth Sawhney

Building a Multi-Provider Rewards Platform: An E-Commerce Approach to Rewards at Scale was originally published in Razorpay Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.