Tell that to Synapse customers. Many millions of dollars are missing.
Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.
FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.
Synapse says that it was actually the Bank (Evolve) that made the accounting mistakes, including missing transactions, debits that weren't reported, sending in flight transaction to Mercury while debiting Synapse incorrectly etc.
> In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts.
When the banks do this it's called "fractional reserve banking", and they sell it as a good thing. :)
I’m constantly amazed by how much the crypto community thinks they understand fractional reserve banking while getting it so completely wrong.
In fractional reserve banking, money that is loaned out is accounted for as liabilities. These liabilities subtract from the overall balance stored (reserved) at the bank. The bank is not printing money new money, no matter how many times this idea gets repeated by people who are, ironically, pumping crypto coins that were printed out of thin air.
I think it’s incredible that cryptocurrencies were literally manifested out of bits, but the same people try to criticize banks for doing this same thing (which they don’t).
At the point I make a loan, 2 things happen on my balance sheet: I have a new liability to you (the increased balance in your account), and I have a new asset (the loan that you’re expected to pay back). They cancel each other out and it therefore seems as if I’m creating money out of thin air.
However, the moment you actually use that money (eg to buy something), the money leaves the bank (unless the other account is also at this bank, but let’s keep it simple). Liabilities on the balance sheet shrink, so assets need to follow. That needs to come from reserves because the loan asset keeps its original value.
The reserve comes from the bank, not from you. Added layer here: Banks can borrow money from each other or central banks if their cash reserves runs low.
Finally: it tends to be the case that the limit on lending is not the reserves, but on the capital constraints. Banks need to retain capital for each loan they make. This is weighed against the risk of these loans. For example: you could lend a lot more in mortgages than in business loans without collateral. Ask your favorite LLM to explain RWAs and Basel III for more.
That is exactly what happens. Reserve ratio used to be 10%, same as your example. The reserve ratio is currently zero, lowered in 2020 during pandemics. But banks still can't lend out more than deposits.
Pardon me an old story... I never built a double entry accounting system but decades ago I did build a billing system for a internet/telcom startup that grew to a modest 8 figures revenue.
By accident and not knowing any better as a young dev, I ended up building the billing logic from day one, and for better and worse building it in two places in the system (on a consumer-facing billing webpage, and on a separate backed process that generated invoices and charged credit cards.)
It turned out to be remarkably hard to keep them in sync. We were constantly iterating trying to get traction as we burned down our capital, releasing new products and services, new ways of discounting and pricing (per use, per month, first X free, etc), features like masterpayer/subaccounts for corporate accounts, user-assignable cost centers, tax allocation to those cost centers with penny allocation, etc such that new wrinkles and corner cases would keep popping up causing the numbers on my two screens/methods not to match.
Being personally responsible for the billing, I would go over all the invoices by hand for a couple days each month to insure they matched before we charged the credit cards and mailed out printed invoices as a final check to prevent mistakes. There was always/often some new problem I'd find affecting one or a small handful of customers which I would then fix the code before we billed. I never felt good letting go and not doublechecking everything by hand.
I thought about refactoring the billing logic to occur in one place to eliminate these mismatches and my manual crosschecking, but after a lot of thought I realized I wasn't comfortable with a single codebase and liked having two codebases as it helped me catch my own errors. I then just made it easier and easier to run automate and crosschecks between the two. The billing code was a little too gnarly to be proud of, but I was very proud of the outcome in how accurate our billing was, the lack of complaints and many near misses we avoided for many years. I do feel twinges of guilt for the complexity I left my successors but I still don't really regret it.
After that experience, the motivation for double entry bookkeeping has always made a lot of sense to me. I had sort of reinvented it in my own hacky way with double logic billing code to prevent my mistakes from causing problems for my customers...
No tests either? If you lose track of enough money every transaction that you can make an example of 'Every $5 purchased resulted in $4.98 in the transaction log' I think your problem is far, far bigger than not having double entry bookkeeping.
Who builds a financial system like that an considers it normal? The compensation is one thing, but you'd flee a service like that with all possible haste.
These guys. They said it themselves. “We could have built it right, but we didn’t.” They chose not to. It was not an accident. They made jokes about “dancing cents”. They did these things because there would never be meaningful consequences for doing them, and they knew it. They move fast, they broke things - money things - and they laughed about it. And now they’re lecturing people as if having willfully made these decisions gives them both moral and technical authority. This is magnificently pompous, startup VC culture nonsense.
Yeah that was my thought as well. Ledgers have tons of benefits but they’re not going to fix your dancing cents problem. You’re going to have bad number on the ledger.
Sure, maybe that points you to the bugs, but so would writing basic tests.
A ledger where you insist that every entry touches exactly two accounts, in a business where transactions regularly involve various types of fees, could easily misplace or misattribute a few cents here and there.
This type of business can also have fun dealing with accruals. One can easily do a bunch of transactions, have them settle, and then get an invoice for the associated fees at variable time in the future.
> where you insist that every entry touches exactly two accounts
A ledger is where every transaction balances to 0. It can involve multiple accounts, but the sum of all transfers between all accounts in a single transaction must sum to 0. This is the property of double entry that actually matters.
If told people that I was using a database system where one and 100th of the data was missing after every 10 transactions would you seriously take my advice as an engineer blog post like these that seem to be calmly focused toward an advertisement of a person’s or a group’s promotion and that it’s just introducing concepts.
Does anyone have a good explanation for why a ledger database should care about credits and debits and normal balances? This has always seemed more natural to me as a front-end / presentation layer concept. Your ledger entries always sum to zero for each transaction, your income account has a negative balance, and you display it in a sensible manner.
I’m also surprised that this whole article starts by discussing stock trading but has no mention of how to represent stock trades. I assume they are “Sagas” consisting of money moving from the customer to the clearinghouse (or prime broker or PFOF provider or whatever) and shares moving from that provider to the account at which the shares are held. And maybe other associated entries representing fees? It seems to me that this is multi-entry accounting, which is quite common, and that entries don’t actually come in pairs as the article would like us to think.
> This has always seemed more natural to me as a front-end / presentation layer concept.
Consistence is a property of the backend, if that is wrong there is not hope for later
> Your ledger entries always sum to zero for each transaction, your income account has a negative balance, and you display it in a sensible manner.
'sensible manner' is the problem here. The data/money will be diverge with time, and without proper storage of the data it will by impossible to figure out.
The problem here is NOT store 'a transaction'. That with a RDBMs works. Is to store the FLOW of MANY transactions and the divergent ways things works.
Like, your bank is telling you has $100 and your system $120. And your system sum right, but the bank rules.
Or when you do a return and cents are lost in the interchanges and chargebacks.
Or, just wrong data entry, sync, import/export, etc.
---
The way to see this is that `double entry` is a variation of `inmutable data that don't mutate and always track the flow of it' that is golden for business apps.
It is best explained by common scenarios an Italian merchant in the Middle Ages experienced. The basic concept is Assets==Liability (plus Equity). Where positive Assets are entered on the left hand side (debit). And positive Liabilities are entered on the right hand side (credit). In accounting, debit and credit just means left and right.
1. Merchant takes out a loan for $5,000 and receives $5,000 in cash.
• Assets (Cash) increase by $5,000 (Debit).
• Liabilities (Loan Payable) increase by $5,000 (Credit).
• Equity remains unchanged.
2. Merchant buys inventory for $1,000 cash.
• Assets (Cash) decrease by $1,000 (Credit).
• Assets (Inventory) increase by $1,000 (Debit).
• Total assets remain unchanged, and liabilities and equity are unaffected.
3. Merchant sells all inventory for $1,500 cash.
• Assets (Cash) increase by $1,500 (Debit).
• Assets (Inventory) decrease by $1,000 (Credit) (recording cost of goods sold).
• Equity (Retained Earnings) increases by $500 (Credit), representing the profit ($1,500 sales - $1,000 cost).
4. Customer1 deposits $500 in cash for future delivery of goods.
• Assets (Cash) increase by $500 (Debit).
• Liabilities (Unearned Revenue) increase by $500 (Credit).
• Equity remains unchanged.
5. Customer1 transfers half of the future delivery of goods to Customer2.
• No changes to assets, liabilities, or equity occur at this point. The merchant’s obligation to deliver goods (reflected as Unearned Revenue) is still $500 but now split between two customers (Customer1 and Customer2). Internal tracking of this obligation may be updated, but the total financial liability remains the same.
I understand this. But we’re talking about computers, not Italian merchants. Italian merchants had actual pieces of paper. Computers have tables and views and frontends that are separate from the tables.
Any self-respecting accounting system should be able to produce a balance sheet that matches the conventions you’re describing. I don’t think it follows that the actual numbers in the database that get summed to produce the total liabilities should be positive.
> In double-entry, a transaction is the tuple (amount, credit_account, debit_account).
Every “double entry” accounting package I’ve ever used can easily handle transactions that are awkward in this schema and transactions that don’t fit at all.
Moving $1 from one current account to another? I guess you declare that the $1 needs to be a positive amount, but your two accounts have the same normal balance, and calling one a “debit account” is a bit awkward.
Adding an accounts payable entry that is split between two expense accounts? Not so easy.
I've never understood that either. As a computer guy it always struck me as a redundancy -- the kind of redundancy where one thing will always eventually be wrong.
I assumed it has to do with the fact that it was invented for bookkeeping by hand. Accountants assure me that it's even more important with computers, but I've never managed to figure out how that works
I've been doing this so long that I've finally realized that if someone can't explain why something is the way it is, that means it's wrong, or at least some arbitrary choice among several equally good alternatives.
Being able to explain it is something different then you being able to understand it.
And double entry bookkeeping should be both easy to explain (there are countless articles for it, precisely because it is a pretty easy concept) And easy to understand if you have ever tried to keep a ledger of transactions around and wanted to audit it for errors.
I always get hung up on the different kinds of accounts and their respective definitions of "credit" and "debit". It isn't that much to memorize but it's very counter to the way I understood those terms and it keeps throwing me off.
> As a computer guy it always struck me as a redundancy -- the kind of redundancy where one thing will always eventually be wrong.
That's the purpose. If you have a system with no redundancy, it's equally true that something will always eventually be wrong. But in that case, you'll have no way of knowing what's wrong.
With the redundancy, you can detect problems and often determine what happened.
is there no individual accountability regime in the US?
in the UK, as an engineer, if I'd built this I would expect the regulator to come after me personally for not ensuring the system had adequate controls to protect clients money/investments
>is there no individual accountability regime in the US?
Here in the US, programmers like to call themselves Engineers, forcing everyone else to use the term "Professional Engineer" or "Licensed Engineer" or some other modifier before their title. I hate it, I wish they would stop, but it's not going to happen.
Software here is a wild, wild, West. The motto most live by is "move fast and break things"... even when those things are people's lives.
For certain regulated professions there is, if a building falls down due to a bad design the professional engineer (PE) that signed and sealed the plans can be held personally liable.
Has that ever happened? It's incredibly hard to prosecute directors in the UK for obvious malfeasance. I have never heard of a software engineer being sanctioned for crap code.
The secret is to have everyone in on it. Everyone is guilty but nobody is quite culpable enough to punish.
The low level guys were just doing their jobs, and each individual transaction was mostly legal. A few weren't but it's hard to sort out which ones. Maybe the management should be responsible for ensuring that nobody ever did anything illegal, but they can't really supervise everything all the time, can they?
Poof. Guilt is just a rounding error that all rounds down to zero. The government passes some new regulations to prevent that particular scenario from happening again, and the same people set about finding a new scam.
I don't see how a rank and file programmer would ever be personally responsible for their code. You can blame management for forcing untested or known flawed logic, but not some shmoe that pushes an "off by 1" bug while working weekends and late nights with no testing and hard deadlines.
Deliberately implementing a financial system that ignores established (and probably legally required) accounting practices? That's kind of like a structural engineer willfully disregarding the building code because that's what management asked for.
In North America, “engineer” doesn’t necessarily mean a software engineer with a professional certification. Software developers have taken to calling themselves engineers. Whether engineering professional bodies should start going after people for this or not is a different topic.
But it’s entirely possible for someone who calls themselves an engineer to not actually be a certified engineer. So the activity wouldn’t be regulated because the person isn’t part of a professional body that regulates members.
In that case, lack of competence would be a civil issue unless it resulted in something criminal.
"Professional Engineer" is a protected title that requires licensing to be used for a discipline. That licensing process does not exist for software in the US right now.
There's no redundancy. Imagine you have 2 buckets, you reach into bucket A and grab some cash and put it in bucket B. $x dollars leaving bucket A is entry part 1, $x dollars entering bucket B is entry part 2. That's all it is. I honestly don't understand what single entry is by comparison - it sounds like losing an essential attribute of the transaction.
Feeling vindicated for the double entry transaction system we built at clearvoice.com for our two-sided marketplace, leveraging the fantastic DoubleEntry Ruby Gem from Envato.
It took me a while to realise what double-entry bookkeeping actually was, then one day it hit me: it's literally just the flow of money through a system. Understand that, and you understand the cornerstone of accountancy.
Im sure there is a point in the article but ive never seen a dancing cent nor can i imagine one. My numbers are all strings "5" is "5" forever. If one somehow ends up storing it as 4.99 why would the other entry be correct?
Your second sentence tells us your first sentence was a lie. You can clearly imagine one which is why you specified which data type you use for money. You know a floating point issue is an issue.
Now let's say your price is "0.00023" per unit and someone uses "213.34" units. Can you imagine it now?
There was no question, that I was answering. Simple someone claiming they have no idea how it would even be possible for a $5 end up being $4.98, while literally stating they know about floating point issues.
I mean i want to understand not that the problem doesnt exist.
If the price is "0.00023" per unit and someone uses "213.34" units I feed those strings into a multiplication function that returns the correct string 100% of the time.
That much i understand. I dont get how that category of problems is addressed by the solution described.
What i also understand is that you inevtably get to deal with accountants or other finance people. Working in a format they understand is not optional. They will have you transform whatever you have into that anyway.
> If the price is "0.00023" per unit and someone uses "213.34" units I feed those strings into a multiplication function that returns the correct string 100% of the time.
But you're not coming up with a valid monetary amount.
This is more a problem with ill-specified contracts though. It was a constant source of annoyance when I was sole-trading for a bit, because what happened was something like this:
I'd be quoted a day-rate. That was what I was actually going to get paid, one day. But then I'd be told to bill it as an hourly rate. And then actually to bill it as 7.5 hours.
But I wasn't told what the hourly was - the hourly was whatever my day rate was, divided by 7.5. So this led to the problem that it produced an irrational number as a result.
Technically this should've been fine...except no one I dealt with knew or cared about this concept - they all used Excel. So if I rounded the irrational to nearest upper cent (since that's the smallest unit which could be paid) they complained it didn't add up. If I added a "correction item" to track summing up partial cents, they complained it wasn't part of the hourly.
In the end I just send python decimal.Decimal to maximum precision, flowed through invoices with like 8 digits of precision on the hourly rate, and this seemed to make Excel happy enough. Of course it was completely useless for tracking purposes - i.e. no one would ever be able to pay or unpay 0.666666666667 cents.
Because what's not in employment contracts that really should be? Any discussion on how numbers are to be rounded in the event of uneven division. You just get to sort of guess what accounting may or may not be doing. In my case of course it didn't matter - no one was ever going to hold me to anything other the day rate, just for some reason they wanted to input tiny fractions of a cent which they actually couldn't track.
And it's not an idle problem either: i.e. in the case of rounding when it comes to wages, should it be against the employee? It's fractions of a cent in practice, but we're not going to define it at all?
Irrational is the wrong word, it was a .3 or .6 repeater or something similar. Same effect: pile in digits so excel would round it off correctly back to the original rate I was quoted.
The example of the price and units is actually a real-world example. If you look at how much you pay for electricity you'll see you're paying something like 0.321 per KwH and you're not billed in full units.
Your issue is just people being lazy and forcing a day rate into an hourly employment system.
Losing cents is because somebody didn't use Decimal for currency. And that's just flat out malfeasance--possibly even regulatory malfeasance.
Double entry is irrelevant, here.
I love the "use crypto" to fix the problem suggestion. LOL!
It's kind of poignant since crypto has given me a fantastic club to beat executives over the head with whenever they want to do stupid handling of money. Since crypto has so many decimal places, you break dumbass floating point handling of money immediately and irretrievably for way more than just "a couple cents". Your CFO starts jumping up and down really excitedly about handling money properly when he has to worry about a rounding error forcing him to compensate a Bitcoin price jump/crash.
The blockchain industry is basically founded on incompetence. If you go back to early Silk Road and Mtgox you can see some shocking things. Like the founders were posting questions on Stackoverflow about database basics. They then ended up building systems with race conditions (withdrawal code for both systems.)
See, if you're an engineer (from a web background) you might think that using a regular backend language with a relational DB would be fine. But this service is typically optimized for concurrency. That might sound amazing. But what you really want is a simple queue. This ensures everything is applied in order and makes it impossible to accidentally have transactions execute on stale state. Plus, a queue can be executed amazingly fast -- we're talking Nasdaq scales. If you use a standard DB you'll end up doing many horrible broken hacks to simulate what would be trivial engineering had you used the right design from the start.
You've got other things to worry about, too. Financial code needs to use integer math for everything. Floating point and 'decimals' lead to unexpected results. The biggest complexity with running large-scale blockchain services is having to protect 'hot wallets.' Whenever an exchange or payment system is hacked the target is always the funds that sit on the server. When the industry began there were still many ways to protect the security of these hot wallets. The trouble is: it required effort to implement and these protocols weren't widely known. So you would get drive-by exchanges that handled millions of dollars with private keys sitting on servers ready to be stolen...
Today there are many improvements for security. New cryptographic constructs like threshold ECDSA, hardware wallets, hardware key management (like enclaves), smart contract systems (decentralized secret sharing... and even multi-sig can go a long way), and designs that are made to be decentralized that remove the need for a centralized deposit system (still needs some level of centralization when trading to a stable coin but its better than nothing.)
I would say the era where people 'build' ledgers though is kind of over. To me it appears that we're organizing around a super-node structure where all the large 'apps' handle their own changes off-chain (or alternatively based on regular trust.) The bottom layer will still support payments but it will be used less often. With more transaction activity happening on layers above. I think its still important to scale the chain and make it as secure as possible. Bitcoin has a unique focus here on security above everything else. Making it ideal for long-term hedges. For every day stuff I can't think of any chains that have more credible R & D than Ethereum. They seem to even have been doing research on the P2P layer now which traditionally no one has cared about.
OK but apparently they did get to make the startup mistakes? They built the quick thing that worked well enough, got some customer traction, and then when they had bugs they were able to rework it and continue.
Frankly I'm not even convinced that double-entry is the sole right answer in this space. There are things you need to be able to represent, but the original reasons for doing double-entry (making errors when subtracting figures) no longer apply.
(I've worked at investment banks and fintech startups)
Hmm, can you elaborate on the original reasons for double entry and them not applying any.kre? I'm not in that space and double entry always seemed an extremely weird, arbitrary requirement / paradigm. Thanks!
The main point of double-entry account keeping is the notion that money never vanishes, it's always tracked as going somewhere.
I think this tends to get misrepresented by people trying to literally keep two entries like we were working with pen and paper book-keeping though.
Because if I have a simple list of transactions, I can easily use a SQL query to recreate a view of double-entry book-keeping. It's perfectly fine to just record the amount of money, and two entity names in a table.
Coz the whole point of the system is that you can explain where the money went at any given time, referenced against something which should be independently auditable: i.e. a vendor receipt which in turn says "there's a physical item of X we gave you".
The "double entry" system's actual merit is when there's multiple people keeping the books. i.e. if you're in a business and your department is spending money on goods, then when you're doing that those transactions should be getting reflected in the shipping departments books (owned by a different person) and maybe your accounting department or the like. The point is that there have to be actual independent people involved since it makes fraud and mistakes hard - if you say you bought a bunch of stuff, but no one else in the business received it or the money to pay for it, then hey, you now need to explain where the money actually is (i.e. not in your pocket).
Tell that to Synapse customers. Many millions of dollars are missing.
Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.
FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.
See https://www.forbes.com/sites/zennonkapron/2024/11/08/what-th...
Synapse says that it was actually the Bank (Evolve) that made the accounting mistakes, including missing transactions, debits that weren't reported, sending in flight transaction to Mercury while debiting Synapse incorrectly etc.
https://lex.substack.com/p/podcast-what-really-happened-at-s...
At a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air.
Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.
This led to an argument with my boss, who assumed things magically worked.
Days later, we received an email announcing an audit one one of these accounting discrepancies.
JPMC proposed using crypto, internally, to consistently manage cash flow.
Not sure if it went anywhere.
It's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.
> In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts.
When the banks do this it's called "fractional reserve banking", and they sell it as a good thing. :)
I’m constantly amazed by how much the crypto community thinks they understand fractional reserve banking while getting it so completely wrong.
In fractional reserve banking, money that is loaned out is accounted for as liabilities. These liabilities subtract from the overall balance stored (reserved) at the bank. The bank is not printing money new money, no matter how many times this idea gets repeated by people who are, ironically, pumping crypto coins that were printed out of thin air.
I think it’s incredible that cryptocurrencies were literally manifested out of bits, but the same people try to criticize banks for doing this same thing (which they don’t).
Clarifying question:
So for every $1 deposited, I can lend $0.90 but must hold $0.10 as my reserve?
It’s a bit more complicated than that.
At the point I make a loan, 2 things happen on my balance sheet: I have a new liability to you (the increased balance in your account), and I have a new asset (the loan that you’re expected to pay back). They cancel each other out and it therefore seems as if I’m creating money out of thin air.
However, the moment you actually use that money (eg to buy something), the money leaves the bank (unless the other account is also at this bank, but let’s keep it simple). Liabilities on the balance sheet shrink, so assets need to follow. That needs to come from reserves because the loan asset keeps its original value.
The reserve comes from the bank, not from you. Added layer here: Banks can borrow money from each other or central banks if their cash reserves runs low.
Finally: it tends to be the case that the limit on lending is not the reserves, but on the capital constraints. Banks need to retain capital for each loan they make. This is weighed against the risk of these loans. For example: you could lend a lot more in mortgages than in business loans without collateral. Ask your favorite LLM to explain RWAs and Basel III for more.
That is exactly what happens. Reserve ratio used to be 10%, same as your example. The reserve ratio is currently zero, lowered in 2020 during pandemics. But banks still can't lend out more than deposits.
There's more to it than that; balances are exceeded by the sum of "assets held by the bank" and "assets owed to the bank".
Pardon me an old story... I never built a double entry accounting system but decades ago I did build a billing system for a internet/telcom startup that grew to a modest 8 figures revenue.
By accident and not knowing any better as a young dev, I ended up building the billing logic from day one, and for better and worse building it in two places in the system (on a consumer-facing billing webpage, and on a separate backed process that generated invoices and charged credit cards.)
It turned out to be remarkably hard to keep them in sync. We were constantly iterating trying to get traction as we burned down our capital, releasing new products and services, new ways of discounting and pricing (per use, per month, first X free, etc), features like masterpayer/subaccounts for corporate accounts, user-assignable cost centers, tax allocation to those cost centers with penny allocation, etc such that new wrinkles and corner cases would keep popping up causing the numbers on my two screens/methods not to match.
Being personally responsible for the billing, I would go over all the invoices by hand for a couple days each month to insure they matched before we charged the credit cards and mailed out printed invoices as a final check to prevent mistakes. There was always/often some new problem I'd find affecting one or a small handful of customers which I would then fix the code before we billed. I never felt good letting go and not doublechecking everything by hand.
I thought about refactoring the billing logic to occur in one place to eliminate these mismatches and my manual crosschecking, but after a lot of thought I realized I wasn't comfortable with a single codebase and liked having two codebases as it helped me catch my own errors. I then just made it easier and easier to run automate and crosschecks between the two. The billing code was a little too gnarly to be proud of, but I was very proud of the outcome in how accurate our billing was, the lack of complaints and many near misses we avoided for many years. I do feel twinges of guilt for the complexity I left my successors but I still don't really regret it.
After that experience, the motivation for double entry bookkeeping has always made a lot of sense to me. I had sort of reinvented it in my own hacky way with double logic billing code to prevent my mistakes from causing problems for my customers...
No tests either? If you lose track of enough money every transaction that you can make an example of 'Every $5 purchased resulted in $4.98 in the transaction log' I think your problem is far, far bigger than not having double entry bookkeeping.
Who builds a financial system like that an considers it normal? The compensation is one thing, but you'd flee a service like that with all possible haste.
These guys. They said it themselves. “We could have built it right, but we didn’t.” They chose not to. It was not an accident. They made jokes about “dancing cents”. They did these things because there would never be meaningful consequences for doing them, and they knew it. They move fast, they broke things - money things - and they laughed about it. And now they’re lecturing people as if having willfully made these decisions gives them both moral and technical authority. This is magnificently pompous, startup VC culture nonsense.
Sounds like they refunded anything that went wrong so it's not really as bad as you make it sound.
critically, only when customers reached out. which means tons of people that weren't eagle eyed got defrauded.
Refunds limit damages in a lawsuit, but don’t prevent legal issues.
Especially important when explicitly saying you’ve done these things.
Yeah that was my thought as well. Ledgers have tons of benefits but they’re not going to fix your dancing cents problem. You’re going to have bad number on the ledger.
Sure, maybe that points you to the bugs, but so would writing basic tests.
A ledger where you insist that every entry touches exactly two accounts, in a business where transactions regularly involve various types of fees, could easily misplace or misattribute a few cents here and there.
This type of business can also have fun dealing with accruals. One can easily do a bunch of transactions, have them settle, and then get an invoice for the associated fees at variable time in the future.
> where you insist that every entry touches exactly two accounts
A ledger is where every transaction balances to 0. It can involve multiple accounts, but the sum of all transfers between all accounts in a single transaction must sum to 0. This is the property of double entry that actually matters.
If told people that I was using a database system where one and 100th of the data was missing after every 10 transactions would you seriously take my advice as an engineer blog post like these that seem to be calmly focused toward an advertisement of a person’s or a group’s promotion and that it’s just introducing concepts.
Does anyone have a good explanation for why a ledger database should care about credits and debits and normal balances? This has always seemed more natural to me as a front-end / presentation layer concept. Your ledger entries always sum to zero for each transaction, your income account has a negative balance, and you display it in a sensible manner.
I’m also surprised that this whole article starts by discussing stock trading but has no mention of how to represent stock trades. I assume they are “Sagas” consisting of money moving from the customer to the clearinghouse (or prime broker or PFOF provider or whatever) and shares moving from that provider to the account at which the shares are held. And maybe other associated entries representing fees? It seems to me that this is multi-entry accounting, which is quite common, and that entries don’t actually come in pairs as the article would like us to think.
> This has always seemed more natural to me as a front-end / presentation layer concept.
Consistence is a property of the backend, if that is wrong there is not hope for later
> Your ledger entries always sum to zero for each transaction, your income account has a negative balance, and you display it in a sensible manner.
'sensible manner' is the problem here. The data/money will be diverge with time, and without proper storage of the data it will by impossible to figure out.
The problem here is NOT store 'a transaction'. That with a RDBMs works. Is to store the FLOW of MANY transactions and the divergent ways things works.
Like, your bank is telling you has $100 and your system $120. And your system sum right, but the bank rules.
Or when you do a return and cents are lost in the interchanges and chargebacks.
Or, just wrong data entry, sync, import/export, etc.
---
The way to see this is that `double entry` is a variation of `inmutable data that don't mutate and always track the flow of it' that is golden for business apps.
It is best explained by common scenarios an Italian merchant in the Middle Ages experienced. The basic concept is Assets==Liability (plus Equity). Where positive Assets are entered on the left hand side (debit). And positive Liabilities are entered on the right hand side (credit). In accounting, debit and credit just means left and right.
1. Merchant takes out a loan for $5,000 and receives $5,000 in cash. • Assets (Cash) increase by $5,000 (Debit). • Liabilities (Loan Payable) increase by $5,000 (Credit). • Equity remains unchanged.
2. Merchant buys inventory for $1,000 cash. • Assets (Cash) decrease by $1,000 (Credit). • Assets (Inventory) increase by $1,000 (Debit). • Total assets remain unchanged, and liabilities and equity are unaffected.
3. Merchant sells all inventory for $1,500 cash. • Assets (Cash) increase by $1,500 (Debit). • Assets (Inventory) decrease by $1,000 (Credit) (recording cost of goods sold). • Equity (Retained Earnings) increases by $500 (Credit), representing the profit ($1,500 sales - $1,000 cost).
4. Customer1 deposits $500 in cash for future delivery of goods. • Assets (Cash) increase by $500 (Debit). • Liabilities (Unearned Revenue) increase by $500 (Credit). • Equity remains unchanged.
5. Customer1 transfers half of the future delivery of goods to Customer2. • No changes to assets, liabilities, or equity occur at this point. The merchant’s obligation to deliver goods (reflected as Unearned Revenue) is still $500 but now split between two customers (Customer1 and Customer2). Internal tracking of this obligation may be updated, but the total financial liability remains the same.
I understand this. But we’re talking about computers, not Italian merchants. Italian merchants had actual pieces of paper. Computers have tables and views and frontends that are separate from the tables.
Any self-respecting accounting system should be able to produce a balance sheet that matches the conventions you’re describing. I don’t think it follows that the actual numbers in the database that get summed to produce the total liabilities should be positive.
In double-entry, a transaction is the tuple (amount, credit_account, debit_account).
In singly-entry, it is the tuple (amount, account).
> In double-entry, a transaction is the tuple (amount, credit_account, debit_account).
Every “double entry” accounting package I’ve ever used can easily handle transactions that are awkward in this schema and transactions that don’t fit at all.
Moving $1 from one current account to another? I guess you declare that the $1 needs to be a positive amount, but your two accounts have the same normal balance, and calling one a “debit account” is a bit awkward.
Adding an accounts payable entry that is split between two expense accounts? Not so easy.
I've never understood that either. As a computer guy it always struck me as a redundancy -- the kind of redundancy where one thing will always eventually be wrong.
I assumed it has to do with the fact that it was invented for bookkeeping by hand. Accountants assure me that it's even more important with computers, but I've never managed to figure out how that works
I've been doing this so long that I've finally realized that if someone can't explain why something is the way it is, that means it's wrong, or at least some arbitrary choice among several equally good alternatives.
https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
Being able to explain it is something different then you being able to understand it.
And double entry bookkeeping should be both easy to explain (there are countless articles for it, precisely because it is a pretty easy concept) And easy to understand if you have ever tried to keep a ledger of transactions around and wanted to audit it for errors.
I always get hung up on the different kinds of accounts and their respective definitions of "credit" and "debit". It isn't that much to memorize but it's very counter to the way I understood those terms and it keeps throwing me off.
> As a computer guy it always struck me as a redundancy -- the kind of redundancy where one thing will always eventually be wrong.
That's the purpose. If you have a system with no redundancy, it's equally true that something will always eventually be wrong. But in that case, you'll have no way of knowing what's wrong.
With the redundancy, you can detect problems and often determine what happened.
Neither have I. It always seems like a massive cargo culting. Human accountants are liable to make very different kinds of mistakes than computers.
It's a suicide to build a finance system or similar without double entry ledger.
Worse, I've worked at where the transaction reference id from vendor is not recorded, it's lost so the past data cannot be reconciled!
is there no individual accountability regime in the US?
in the UK, as an engineer, if I'd built this I would expect the regulator to come after me personally for not ensuring the system had adequate controls to protect clients money/investments
with a potentially unlimited fine + prison time
>is there no individual accountability regime in the US?
Here in the US, programmers like to call themselves Engineers, forcing everyone else to use the term "Professional Engineer" or "Licensed Engineer" or some other modifier before their title. I hate it, I wish they would stop, but it's not going to happen.
Software here is a wild, wild, West. The motto most live by is "move fast and break things"... even when those things are people's lives.
The PE thing is more than 100 years old in the US. By 1947 every state had a PE licensure program. It has nothing to do with programmers.
In fact it was lobbied in order to disinclude software engineering from it's purview in most states.
Not for software that I'm aware of.
For certain regulated professions there is, if a building falls down due to a bad design the professional engineer (PE) that signed and sealed the plans can be held personally liable.
Has that ever happened? It's incredibly hard to prosecute directors in the UK for obvious malfeasance. I have never heard of a software engineer being sanctioned for crap code.
for engineers[1] it is relatively new, having only been introduced in the last couple of years
[1]: technically those performing a Certified Function
It would be nearly impossible to prosecute for just bad code. It would require more and is limited very small scope of people.
I leave you with the 2008 financial crisis as exhibit A on exactly how nobody gets in trouble for lack of financial accountability
The secret is to have everyone in on it. Everyone is guilty but nobody is quite culpable enough to punish.
The low level guys were just doing their jobs, and each individual transaction was mostly legal. A few weren't but it's hard to sort out which ones. Maybe the management should be responsible for ensuring that nobody ever did anything illegal, but they can't really supervise everything all the time, can they?
Poof. Guilt is just a rounding error that all rounds down to zero. The government passes some new regulations to prevent that particular scenario from happening again, and the same people set about finding a new scam.
I don't see how a rank and file programmer would ever be personally responsible for their code. You can blame management for forcing untested or known flawed logic, but not some shmoe that pushes an "off by 1" bug while working weekends and late nights with no testing and hard deadlines.
Pushing a bug, yeah that happens.
Deliberately implementing a financial system that ignores established (and probably legally required) accounting practices? That's kind of like a structural engineer willfully disregarding the building code because that's what management asked for.
Depends if the particular activity is regulated or not.
a stock trading platform, as described in the article?
Could well be the entity actually selling the services.
In North America, “engineer” doesn’t necessarily mean a software engineer with a professional certification. Software developers have taken to calling themselves engineers. Whether engineering professional bodies should start going after people for this or not is a different topic.
But it’s entirely possible for someone who calls themselves an engineer to not actually be a certified engineer. So the activity wouldn’t be regulated because the person isn’t part of a professional body that regulates members.
In that case, lack of competence would be a civil issue unless it resulted in something criminal.
There isn't even a way to get certified as a professional engineer for software in the US.
it's what you're doing (your "function") that's regulated
not your job title, or piece of paper that you have that says you're X, Y or Z
"Professional Engineer" is a protected title that requires licensing to be used for a discipline. That licensing process does not exist for software in the US right now.
I'm not sold on double entry here.
If a new transaction enters the system now, I could follow the advice and record it as two sources of truth. Or I could just record it once.
If I could turn a single transaction into two ledger entries today, I could do the same later if I needed to.
There's no redundancy. Imagine you have 2 buckets, you reach into bucket A and grab some cash and put it in bucket B. $x dollars leaving bucket A is entry part 1, $x dollars entering bucket B is entry part 2. That's all it is. I honestly don't understand what single entry is by comparison - it sounds like losing an essential attribute of the transaction.
Accounting systems are super hard when you do them wrong and kind of trivial when you do it right.
There is no in-between.
Martin fowler wrote quite a bit on the subject and it's a good match for event-sourcing.
> It’s just that…it goes without saying that fintech companies should know better
haha, if only.
move fast, break everything if it gives profit
Feeling vindicated for the double entry transaction system we built at clearvoice.com for our two-sided marketplace, leveraging the fantastic DoubleEntry Ruby Gem from Envato.
Btw for those wondering what to do instead is to simply use https://tigerbeetle.com/
It took me a while to realise what double-entry bookkeeping actually was, then one day it hit me: it's literally just the flow of money through a system. Understand that, and you understand the cornerstone of accountancy.
Im sure there is a point in the article but ive never seen a dancing cent nor can i imagine one. My numbers are all strings "5" is "5" forever. If one somehow ends up storing it as 4.99 why would the other entry be correct?
Your second sentence tells us your first sentence was a lie. You can clearly imagine one which is why you specified which data type you use for money. You know a floating point issue is an issue.
Now let's say your price is "0.00023" per unit and someone uses "213.34" units. Can you imagine it now?
Question was about double entry. Does recording it twice solve floating point issues?
There was no question, that I was answering. Simple someone claiming they have no idea how it would even be possible for a $5 end up being $4.98, while literally stating they know about floating point issues.
I mean i want to understand not that the problem doesnt exist.
If the price is "0.00023" per unit and someone uses "213.34" units I feed those strings into a multiplication function that returns the correct string 100% of the time.
That much i understand. I dont get how that category of problems is addressed by the solution described.
What i also understand is that you inevtably get to deal with accountants or other finance people. Working in a format they understand is not optional. They will have you transform whatever you have into that anyway.
I learn not to wonder why but maybe i should.
> If the price is "0.00023" per unit and someone uses "213.34" units I feed those strings into a multiplication function that returns the correct string 100% of the time.
But you're not coming up with a valid monetary amount.
This is more a problem with ill-specified contracts though. It was a constant source of annoyance when I was sole-trading for a bit, because what happened was something like this:
I'd be quoted a day-rate. That was what I was actually going to get paid, one day. But then I'd be told to bill it as an hourly rate. And then actually to bill it as 7.5 hours.
But I wasn't told what the hourly was - the hourly was whatever my day rate was, divided by 7.5. So this led to the problem that it produced an irrational number as a result.
Technically this should've been fine...except no one I dealt with knew or cared about this concept - they all used Excel. So if I rounded the irrational to nearest upper cent (since that's the smallest unit which could be paid) they complained it didn't add up. If I added a "correction item" to track summing up partial cents, they complained it wasn't part of the hourly.
In the end I just send python decimal.Decimal to maximum precision, flowed through invoices with like 8 digits of precision on the hourly rate, and this seemed to make Excel happy enough. Of course it was completely useless for tracking purposes - i.e. no one would ever be able to pay or unpay 0.666666666667 cents.
Because what's not in employment contracts that really should be? Any discussion on how numbers are to be rounded in the event of uneven division. You just get to sort of guess what accounting may or may not be doing. In my case of course it didn't matter - no one was ever going to hold me to anything other the day rate, just for some reason they wanted to input tiny fractions of a cent which they actually couldn't track.
And it's not an idle problem either: i.e. in the case of rounding when it comes to wages, should it be against the employee? It's fractions of a cent in practice, but we're not going to define it at all?
> the hourly was whatever my day rate was, divided by 7.5. So this led to the problem that it produced an irrational number as a result.
The only way for this to be true is if your day rate was irrational to begin with.
Irrational is the wrong word, it was a .3 or .6 repeater or something similar. Same effect: pile in digits so excel would round it off correctly back to the original rate I was quoted.
The example of the price and units is actually a real-world example. If you look at how much you pay for electricity you'll see you're paying something like 0.321 per KwH and you're not billed in full units.
Your issue is just people being lazy and forcing a day rate into an hourly employment system.
nit: double entry accounting only goes back to the 13th century, not thousands of years.
Fibonacci, if I remember correctly.
You do not remember correctly: https://en.wikipedia.org/wiki/Double-entry_bookkeeping#Histo...
Losing cents is because somebody didn't use Decimal for currency. And that's just flat out malfeasance--possibly even regulatory malfeasance.
Double entry is irrelevant, here.
I love the "use crypto" to fix the problem suggestion. LOL!
It's kind of poignant since crypto has given me a fantastic club to beat executives over the head with whenever they want to do stupid handling of money. Since crypto has so many decimal places, you break dumbass floating point handling of money immediately and irretrievably for way more than just "a couple cents". Your CFO starts jumping up and down really excitedly about handling money properly when he has to worry about a rounding error forcing him to compensate a Bitcoin price jump/crash.
The blockchain industry is basically founded on incompetence. If you go back to early Silk Road and Mtgox you can see some shocking things. Like the founders were posting questions on Stackoverflow about database basics. They then ended up building systems with race conditions (withdrawal code for both systems.)
See, if you're an engineer (from a web background) you might think that using a regular backend language with a relational DB would be fine. But this service is typically optimized for concurrency. That might sound amazing. But what you really want is a simple queue. This ensures everything is applied in order and makes it impossible to accidentally have transactions execute on stale state. Plus, a queue can be executed amazingly fast -- we're talking Nasdaq scales. If you use a standard DB you'll end up doing many horrible broken hacks to simulate what would be trivial engineering had you used the right design from the start.
You've got other things to worry about, too. Financial code needs to use integer math for everything. Floating point and 'decimals' lead to unexpected results. The biggest complexity with running large-scale blockchain services is having to protect 'hot wallets.' Whenever an exchange or payment system is hacked the target is always the funds that sit on the server. When the industry began there were still many ways to protect the security of these hot wallets. The trouble is: it required effort to implement and these protocols weren't widely known. So you would get drive-by exchanges that handled millions of dollars with private keys sitting on servers ready to be stolen...
Today there are many improvements for security. New cryptographic constructs like threshold ECDSA, hardware wallets, hardware key management (like enclaves), smart contract systems (decentralized secret sharing... and even multi-sig can go a long way), and designs that are made to be decentralized that remove the need for a centralized deposit system (still needs some level of centralization when trading to a stable coin but its better than nothing.)
I would say the era where people 'build' ledgers though is kind of over. To me it appears that we're organizing around a super-node structure where all the large 'apps' handle their own changes off-chain (or alternatively based on regular trust.) The bottom layer will still support payments but it will be used less often. With more transaction activity happening on layers above. I think its still important to scale the chain and make it as secure as possible. Bitcoin has a unique focus here on security above everything else. Making it ideal for long-term hedges. For every day stuff I can't think of any chains that have more credible R & D than Ethereum. They seem to even have been doing research on the P2P layer now which traditionally no one has cared about.
> Financial code needs to use integer math for everything. Floating point and 'decimals' lead to unexpected results.
Decimals are integers. There's no difference between integer math and decimal math, only in the meaning of the bit pattern afterwards.
OK but apparently they did get to make the startup mistakes? They built the quick thing that worked well enough, got some customer traction, and then when they had bugs they were able to rework it and continue.
Frankly I'm not even convinced that double-entry is the sole right answer in this space. There are things you need to be able to represent, but the original reasons for doing double-entry (making errors when subtracting figures) no longer apply.
(I've worked at investment banks and fintech startups)
Hmm, can you elaborate on the original reasons for double entry and them not applying any.kre? I'm not in that space and double entry always seemed an extremely weird, arbitrary requirement / paradigm. Thanks!
The main point of double-entry account keeping is the notion that money never vanishes, it's always tracked as going somewhere.
I think this tends to get misrepresented by people trying to literally keep two entries like we were working with pen and paper book-keeping though.
Because if I have a simple list of transactions, I can easily use a SQL query to recreate a view of double-entry book-keeping. It's perfectly fine to just record the amount of money, and two entity names in a table.
Coz the whole point of the system is that you can explain where the money went at any given time, referenced against something which should be independently auditable: i.e. a vendor receipt which in turn says "there's a physical item of X we gave you".
The "double entry" system's actual merit is when there's multiple people keeping the books. i.e. if you're in a business and your department is spending money on goods, then when you're doing that those transactions should be getting reflected in the shipping departments books (owned by a different person) and maybe your accounting department or the like. The point is that there have to be actual independent people involved since it makes fraud and mistakes hard - if you say you bought a bunch of stuff, but no one else in the business received it or the money to pay for it, then hey, you now need to explain where the money actually is (i.e. not in your pocket).
Cool post, wish it existed 2 years ago when we started building Pave Bank, or 10 years ago when we started building Monzo.
If you're starting a bank or need a ledger these days (and aren't using a core banking provider that has one), then i usually recommend Tiger Beetle.