Wasteman’s note: I will refer to coding agents interchangeably with “codex” in this post, as it’s the only coding agent I use today.

A harness is not enough

When I first started using agents I was skeptical of the quality of the code it would output, so I was pretty meticulous about reviewing code line by line, and slowly adding more rules to make sure codex doesn’t repeat mistakes. The problem I was solving here was, “can I get the agent to write correct code”, facilitated by this harness of rules, and good tests. Overtime the harness I added started to be really effective, and I could reliably get codex to one shot tasks.

I started to trust codex enough that I would barely review the code and perform very light UX validations before pushing code through. At some point however, the quality of codex’s output started to degrade when I wanted to make changes, especially if it was something fundamental about my app. Since I ignored the code for a sufficient amount of time, I was not well equipped to debug through all the slop that had accumulated.

We all have that experience with a manager who knows nothing about your system and is always asking you stupid questions about why things take so long, and asking “shouldn’t that task be simple?”. And I realized I had become that manager, except my employees were a bunch of codex agents. I didn’t understand the code or system enough to ask the right questions and guide codex to write the code the way I wanted it to.

Wasteman’s note: Candidly I also became this type of manager during my short stint in management, so I’m not surprised it happened again with non human agents. Probably more evidence of why I should stay an individual contributor.

The harness I built solved the narrow problem of, can it write “correct” code exactly as specified in my unit tests and rules. But I am left with two new problems

1. Does codex have enough context to infer what the definition of correctness is given a new task?

2. Does it have context on “why” we made decisions in the past, and does it have any intuition for what is the “right” thing to do for our domain?

Context is more than code, and its more than documentation

One of the ways I tried to solve this context problem, is by having in repo documentation about the “why” we did things with the hope that codex would read through this and avoid falling down common pitfalls. It did improve performance for a time, but no matter how rigorous I was in adding decision logs and documentation in code, I eventually ran into the same problems again.

The two fundamental problems with this approach I observed are

1. The more context documentation you provide, the more likely codex will forget it when it compacts. So even if the information is there, codex may not follow it because it forgot.

2. Human language is fundamentally a lossy form of communication

I think (1) can be solved with smarter models, but I think (2) is actually a fundamental problem when working with models that can’t continually learn (I am referring to continual learning as the actual weights of the model changing not just a KV cache in memory). When we speak and write, we aren’t just regurgitating saved thoughts in our brain. Language is just the interface into our minds where a much deeper representation of our knowledge exists. But there is so much knowledge in our brains we either don’t know how to express, or there are no words to describe it.

From Peter Naur’s essay Programming as Theory Building

A main claim of the Theory Building View of programming is that an essential part of any program, the theory of it, is something that could not conceivably be expressed, but is inextricably bound to human beings

And I think this is what I am getting at, when I say that context is more than code and documentation. Humans have the ability to represent knowledge in a deep representation that cannot be mapped 1:1 purely in human language. Peter Naur would say humans can “build a theory” of the system, which you cannot simply codify in markdown files. Until AI labs give us a model that can continually learn, this will be be a problem.

Wasteman’s note: I don’t agree with Naur’s statement that this is inextricably bound to human beings, but I think he is correct that you need an agent that has a deeper understanding of the system than the code and documents can provide.

Nothing replaces understanding the code

It’s easy to get enamored with using coding agents in an imperative way, “Build me X” without any strong opinions on how codex should build it. But at some point the code will become complex enough that codex cannot efficiently solve the problem on it’s own with its limited context window. This is where your understanding of the system is essential to getting the most out of your agent.

The better you understand the code, the better questions you can ask and the better intuition you have on whether codex is solving the problem the right way or not. This is precisely why so many of us have observed that senior engineers are the ones that have benefited the most from AI. They know enough to ask the right questions, and have built years of intuition that they use to guide the agent.

Humans are the source of context

In my view, the missing piece of the loop to get a truly autonomous coding agent is an agent that can continually learn and share context forward to new subagents completing subtasks. With today’s models, we have to be this agent. We have deep representations of the systems we have built, compacted in a much more intelligent way than a KV cache of weights of tokens. So we have to be the ones injecting our opinions and “taste” about what the right problems to solve are, and the right ways to solve them are.

In this post, I’ll share lessons from my years working on financial systems at big tech companies. Our focus will be building an accounting system, but the principles apply to more general financial systems as well.

This post will go over the following

1. Basic financial definitions relevant to the post

2. High level goals of an accounting system

3. Engineering principles to achieve those goals

4. Best Practices

Definitions

• General Ledger (GL): The primary accounting record of the company, summarizing all financial transactions over a specific time period. You can think of this as an aggregation of it's corresponding sub-ledgers.

• Sub-ledger: Contains detailed information about all individual transactions related to a specific GL. Records in the sub ledger will have much more granular data then the general ledger, like who the specific customer is, specific line items in an order, etc. The difference in data between the sub-ledger and GL will depend on the type of business and volume of data you are working with. Some small businesses can get away with not having any sub-ledgers at all, but it is doubtful that they would ever need custom software to manage something that is so low in scale.

• Financial Record: This refers to the general ledger and sub-ledgers.

• Material: Materiality refers to whether misstatement of information in your financial statements would impact a reasonable stakeholder’s decision making statements. Note that this definition is somewhat ambiguous by design, as different businesses have different materiality thresholds. For example what might be material for a business making $250,000 of revenue per year, will not be material for a business making $1 billion in revenue. From a design perspective, the main value of this concept is to classify different categories of financial data.

High Level Data Flow

Goals

The three main goals of your accounting system are to be (1) Accurate, (2) Auditable and (3) Timely.

Accurate

The financial record needs to reflect the known state of the business. This statement is a little broad and up to some interpretation so I will give some real examples.

If we sell a 10 units of a product that costs $9.99, the corresponding financial records must add up to $99.90. This seems obvious but when you are aggregating thousands (in a lot of cases millions) of transactions, simple summation or rounding errors between systems can cause material inaccuracies.

Wasteman’s Note: People say naming is the hardest problem in computer science, I would say a close second is addition. After working on large scale financial systems for the past few years, I can’t remember how many times the smallest bugs caused large discrepancies in our data. Also don’t get me started on summations over floats. I learned the hard way why you should always use integers.

The financial record also needs to be complete. More specifically, both the sub-ledger and the general ledger are a complete representation of all business activities that occurred at a specific time. If there is an event that occurred but is not in the financial record, than the system is not complete. Note, that this doesn’t imply eventual consistency not acceptable. You just need to know when your data will become complete, to notify stakeholders that data has settled.

Wasteman’s Note: Another surprisingly really hard problem is guaranteeing completeness. As your system scales, data hops between many systems and at each hop data can easily be mutated or dropped by accident.

Auditable

Very related to accuracy, your financial record must be easily auditable so that stakeholders can detect errors and accurately measure performance of your business. And even if you don’t care, the IRS definitely does.

Timely

This one depends entirely on your business and it's specific needs. Small businesses can get away with just dumping all numbers near the end of the month, just in time to close the books. Larger businesses generally want to avoid this, and have a near real time system. This allows them to monitor financials within the month, make decisions based on financial data faster, and reduce the rush to close the month/quarter in the first few days of the month.

But whatever that need is, our accounting system should meet the needs of your business, and whatever timely means to them.

Wasteman’s note: People tend to get lost in conversations about batch vs streaming systems with respect to timeliness a lot. My take is that this isn’t an important distinction to make for most systems. If you care about super low latency cases within seconds to minutes, then this matters. But you would be surprised at how often I hear people arguing about which to do, when the consumer doesn’t need to see updates more than a couple times a day. Just because they asked for it doesn’t mean they need it.

Engineering Principles

The three main engineering principles your accounting system should abide by are

1. Immutability and Durability of data

2. Data should be represented at the smallest grain

3. Code should be Idempotent

Immutable and Durable

This allows for auditability, which helps debugging and in turn accuracy. When data is immutable, you have a record of what the state of the system was at any given time. This makes it really easy to recompute the world from previous states, because no state is every lost.

Building on, once data is stated in the financial record it cannot be deleted. Any corrections to the system must be represented as a new financial transaction. For example let’s say your system had a bug and accidentally reported that a service was sold for $1000, when it should have been $900. To correct this mistake, you should first reverse the accounting entries corresponding to the mistake, and restate the accounting entry for the correct amount.

It will look something like this:

So you can see that in the financial record, there is evidence that the balance of Accounts Receivable (AR) and Revenue was $1000 at some point, but was corrected later. Even though that balance was incorrect, we want an audit trail of what the balance was at any given moment.

Data recorded at the smallest grain

Similar to the above principle, this is also critical for enabling a clear audit trail. Even though financial reports and the general ledger are aggregated, they are computed from more granular events. When the data doesn't make sense, you need the most granular data to debug what might have been the issue.

Saving data at the lowest granularity also makes it really easy to correct data that is derived from that dataset. If a single immutable dataset is the core source of truth for all views of that data, to correct the view all you need to do is rerun the pipeline that creates that view after fixing your data.

Similarly when accountants are preparing to close the books, they reconcile account balances with all the transactions that occurred to validate that the books are accurate. When a discrepancy is discovered, you can dig into the exact transaction that might be causing the issue.

Idempotency

Every financial event can only be processed once, duplicates in the financial record will cause obvious inaccuracies. For that reason, all code that produces financial records should be idempotent.

Best Practices

Over the years, I have run into quite a few gotchas that have caused me a lot of pain. Below are best practices I recommend, to avoid the many pitfalls I have personally faced.

Prefer integers to represent financial amounts. Makes arithmetic much easier. Certain decimal representations are okay, avoid floats at all cost.

Granularity of your financial amounts should support currency conversions with minimal loss of precision. If you are only working with dollars, representing values in cents might be sufficient. If you are a global business, prefer micros or a decimal like DECIMAL(19, 4)(larger than 4 decimal places may also be used if necessary). The decimal choice is quite popular among financial systems, but micros has been the standard for ads financial systems. This limits loss of precision when converting between currencies.

Wasteman’s note: Micros of a currency = base currency unit * 1,000,000. E.g $1.23 = 1,230,000 micros. I first came across this when working with Google’s metrics API.

Use consistent rounding methodologies. At scale the way you round can create material differences between expected amounts. For example one rounding methodology is to round all values 5 and up to the next significant digit, and 4 and below rounds down. Another valid way is to always round up. All that matters is you are consistent across the board. When you are dealing with millions of transactions, being off by 1 cent per transaction can lead to material differences. (10 million transactions off by 1 cent, leads to a difference of $100k). This may not be material to your business at this scale, but it’s material enough for the government to come after you for underpaying taxes.

Wasteman’s note: If you are a global business there can be a lot of gotchas with rounding and currency conversions. I would go as far as saying you should make a centralized library/service to handle both rounding and currency conversions. Different governments respect different rounding rules when calculating taxes, so having all these nuances abstracted into a single library/service will reduce complexity.

Delay currency conversion until the end of calculations. Preemptively converting currencies can cause loss of precision. Delay currency conversions until after aggregations occur in their local currency.

Use integer representations of time. This one is a little controversial but I stand by it. There are so many libraries in different technologies that parse timestamps into objects, and they all do them differently. Avoid this headache and just use integers. Unix timestamp, or even integer based UTC datetimes work perfectly fine. The less data conversions that occur between systems, the better. (Read about Etsy’s own problems with timestamp types here)

Wasteman’s note: I haven’t even talked about daylight savings related bugs. Using an incrementing integer can help you avoid this altogether. If you really insist on using datetimestamps, please at least use UTC. You would be surprised at how many very large businesses use non UTC timestamps.

Thanks for reading this post, I am sure I made a controversial statement somewhere but please feel free to comment and start a discussion. I am very open to learning and hearing other people’s thoughts. And if you enjoyed this post, consider supporting me by subscribing below!

And for further reading, here are some really good blog posts on accounting tailored towards software engineers.

• Accounting for Software Engineers

• Accounting for Developers Part 1

• Accounting for Computer Scientists

A really simple way of monitoring your data pipelines is by using plain SQL queries, integrated with an alerting system. I started using this method years ago when I began working on stream/batch processing systems, and over time found it to be a surprisingly robust way of monitoring data.

In this blog post, I will share some basic SQL alerting patterns that I have found useful, and hopefully, you all find useful as well.

Data Staleness

Some datasets are expected to be up to date for some period, and you will usually document this in an SLA for downstream consumers. For example, I worked on an advertising metrics system for many years that had an SLA of 24 hours for fresh data, and we needed to notify consumers if there were any delays.

A really simple way of alerting when we approach our SLA is to run a SQL query like this

SELECT
    DATEDIFF(HOUR, MAX(event_date_time_utc), CURRENT_TIMESTAMP()) < 12
FROM
    source_table;

It returns a boolean on whether there is data in the table for events that occurred within the last 18 hours. If this query returns False, we will alert. Note that even though our SLA was 24 hours, we alerted our internal team well before then to give us time to push any fixes if need be. If we were certain we would not meet our SLA we would notify our consumers. It was standard practice for us to have these types of alerts on all of our consumer-facing datasets, all with different thresholds depending on the SLA.

Data Duplicates

This is a common alert, especially when working with OLAP data stores that do not enforce primary keys. Redshift, Snowflake, Iceberg and Deltalake are all various databases and table storage formats I have used that don’t have any primitives to enforce the uniqueness of data. For this reason, we needed some way of detecting duplicates early, and a really easy way to do this was with a basic SQL query. Below I will show three different methods I have used depending on the use case.

Example 1: Existence of duplicates

SELECT
    COUNT(DISTINCT event_id) AS distinct_cnt,
    COUNT(*) AS cnt
FROM
    events
HAVING
    ABS(cnt - distinct_cnt) > 0

This query just counts the total records in the table and compares it with the distinct count of the unique key on the table.

Example 2: Return the duplicate records

If you would also like to know which records are duplicates the following query will return all event_ids that have duplicates.

SELECT
    event_id,
    COUNT(*) AS event_count
FROM
    events
GROUP BY
    event_id
HAVING
    event_count > 1

Note that if the unique key for your table is not a single column, you can simply group by your entire composite key instead. (The same method can be used for example 1)

SELECT
    key1,
    key2,
    ...
    keyn,
    COUNT(*) AS event_count
FROM
    events
GROUP BY
    key1, key2, ..., keyn
HAVING
    event_count > 1

Example 3: Approximation for large datasets

For very large datasets the above strategy may take a long time to compute. An alternative is to use APPROX_COUNT_DISTINCT as an approximation and alert when the difference between count, and APPROX_COUNT_DISTINCT is greater than the error threshold of your APPROX_COUNT_DISTINCT implementation. In this example, we are assuming the error threshold is 2.5% for this implementation. APPROX_COUNT_DISTINCT also takes an optional parameter that can reduce the error rate but also affects the performance of the query. Depending on the SQL engine you use, this function may be implemented differently with a different name. But all these engines use the HyperLogLog algorithm under the hood.

WITH counts AS (
    SELECT
        APPROX_COUNT_DISTINCT(line_item_id) AS distinct_count,
        COUNT(*) AS total_count
    FROM
        revinfra.ad_rev_rec_event
)
SELECT
    distinct_count,
    total_count
FROM
    counts
WHERE
    ABS(total_count - distinct_count) > (total_count * 0.025);

Gaps in Continuous Data

Imagine you have some data stream where you expect events to occur fairly continuously, for example, ad impressions on some large platform like Google. With the volume of searches a day, there should never be a 10-minute time period where there aren’t ad impressions recorded.

WITH event_gaps AS (
    SELECT
        event_date_time_utc,
        LEAD(event_date_time_utc) OVER (ORDER BY event_date_time_utc) AS next_event_time,
        DATEDIFF(MINUTE, event_date_time_utc, LEAD(event_date_time_utc) OVER (ORDER BY event_date_time_utc)) AS gap_minutes
    FROM
        ad_impressions
)
SELECT
    event_date_time_utc AS start_time,
    next_event_time AS end_time
FROM
    event_gaps
WHERE
    gap_minutes > 10;

The query uses a CTE that creates a mapping between every event, and the timestamp of the following event using the LEAD window function. It saves the difference for each row as gap_minutes. The main query then checks that the gap_minutes is never greater than 10 minutes.

In my experience, I have found this type of alert really useful for detecting missing batches of data in an upstream service. For example, the upstream data we consumed for a specific application originated from Kafka events that were ETLed to S3 for us to consume. There was a single failed write from Kafka to S3 for a small period, and this alert caught the issue and we were able to recover these events later. In theory, this failure should have been caught by the upstream team, but 🤷, all that matters is that we caught it.

I have also found this alert to be useful in catching bugs I deployed in my own application. There was a time when I deployed a code change that was passing along the wrong timestamp, making it seem like data was missing at specific hours. Instead of passing along the event_date_timestamp (that represented when the event occurred), I passed along the job_run timestamp which was the same for every event processed in the same run 🤦. Luckily, this alert caught the issue and we fixed our data before we affected downstream consumers.

Data Anomalies

If you have a dataset that is relatively consistent over time, it may be useful to be alerted of any anomalies. This class of alerts is the trickiest to get right because the utility of your alert depends on the thresholds you set. A threshold that is too strict will cause you to miss real issues, while a threshold that is too lenient will cause the alert to be noisy.

This post doesn’t go into the process of figuring out those thresholds, but below I show you some sample queries that you can use to alert you of anomalies. For the examples below assume I have a table revenue_events with columns event_date_time_utc and revenue.

Note: The following queries assume that your dataset follows a normal distribution.

Example: Abnormally High/Low Values

WITH stats AS (
    SELECT
        AVG(revenue) AS avg_amount,
        STDDEV(revenue) AS std_amount
    FROM
        revenue_events
)
SELECT
    event_date_time_utc,
    revenue
FROM
    revenue_events
JOIN
    stats
WHERE
    revenue > (avg_amount + (3 * std_amount))
    OR revenue < (avg_amount - (3 * std_amount));

The CTE checks for the average revenue in the table, and the standard deviation using the AVG and STDDEV functions. The main query then searches for records that are greater than 3 standard deviations from the average.

Example: Value Outside Percentile Range

WITH percentile_bounds AS (
    SELECT
        PERCENTILE_APPROX(revenue, 0.05) OVER () AS lower_bound,
        PERCENTILE_APPROX(revenue, 0.95) OVER () AS upper_bound
    FROM
        revenue_events
)
SELECT
    event_date_time_utc,
    revenue
FROM
    revenue_events
JOIN
    percentile_bounds
WHERE
    revenue NOT BETWEEN lower_bound AND upper_bound;

This query searches for records in the bottom and top 5 percentiles of the total distribution and on its own may not be a particularly useful alert.

In practice, this query should likely be matched with time bounds to make sure you are only alerting for responses that should be looked into. For example, if you are a seasonal business, you might want to alert that you got a revenue event in the bottom fifth percentile during a historically high volume time.

Example: Sudden Spikes or Drops

WITH revenue_diff AS (
    SELECT
        event_date_time_utc,
        revenue,
        LAG(revenue) OVER (ORDER BY event_date_time_utc) AS prev_revenue,
        revenue - LAG(revenue) OVER (ORDER BY event_date_time_utc) AS diff
    FROM
        revenue_events
)
SELECT
    event_date_time_utc,
    revenue,
    prev_revenue,
    diff
FROM
    revenue_diff
WHERE
    ABS(diff) > (
        SELECT
            AVG(ABS(diff)) + 3 * STDDEV(ABS(diff))
        FROM
            revenue_diff
    );

The CTE first calculates the difference between each event’s revenue and the event directly preceding it. The main query then calculates if any of the diffs are outside 3 standard deviations of the average diff.

I don’t actually know the inner workings of Substack’s system, but I can infer that this is a classic replica lag: read your writes consistency problem.

I actually wrote a post about this a couple of years ago, where I described the problem and how to fix it in a Rails application. If you are interested, feel free to check it out. But back to it, what is read-your-writes consistency? And how does this cause the weird UX we are seeing with Substack today?

Read your writes consistency is a guarantee that after any write you make to a database, subsequent reads of that data should reflect the changes from any previous writes. In Substack’s case, if I comment on a post I should see this change reflected immediately.

In a very simple single-instance database architecture, you probably won’t face this problem. Almost every modern database today supports some form of database transactions, with pretty good isolation levels. This problem tends to appear when we rearchitect our system to scale to support read load. Substack obviously is a read-heavy system, many more people read posts on substack than they actually write them.

The most common pattern to scale for a read-heavy system is to provision read replicas of your main database instance. Many companies tend to start with a basic relational database like MySQL or Postgres (Substack uses Postgres to my understanding), which only supports writes to a single leader instance by default. As read load increases, the first solution people go towards is provisioning read replicas of the main leader. You can then update your application to send writes to the leader instance, and all reads to replicas.

Another common pattern is to use some sort of caching, where reads are routed to a cache before reading from the database. The same principle applies here, you have more machines that your app can read from instead of reading from the source database machine directly.

So where does the problem occur? The problem occurs when there is replication lag; the time it takes for changes to propagate from the leader instance to all the read replicas. When you comment on a substack post, it submits a write to the leader instance. When your page refreshes it reads from a read replica that may not have the data from your comment yet. This is why we see this confusing UX, and it usually self-resolves when we refresh our page a few times.

How do we fix it?

It will really depend on Substack’s architecture but a few solutions come to mind. The easiest solution is to force all reads that occur within a time period of a write to read from your leader instance. If your app by default reads from a read replica or some cache, force the app to read from the leader database right after a write. If your system requires a much larger scale of reads, this strategy might not be feasible either though.

Another way is to rearchitect your application to use a horizontally scalable database. Instead of having all reads route to the single leader instance after a write, if multiple instances support writes you can route them to the instance that stores that specific piece of data. This way instead of all reads being forced onto a single instance, they are distributed across multiple instances because data is written on multiple machines. Obviously, this poses its own challenges, and you probably want a replication strategy here as well. But being able to horizontally scale does give you room to serve many more concurrent users. This is mostly a solved problem, most horizontally scalable databases implement a quorum-based approach to make sure you are returned the latest write, preventing this problem altogether.

If the problem occurs from caching, we will need to inspect the semantics of the cache. Is it a write-through cache or a read-through cache? What are we saving in the cache? If we cache comments on a post already when a new comment comes in does it add to the existing entry or create a new entry? All of these questions are relevant to architect the correct solution.

Obviously, I have no insight into Substack’s actual architecture so who knows if the above solutions will work. But the same principle applies no matter their architecture, you want to read from the machine that has the changes you just wrote.

Subscribe now