Myth: Eric Brewer on Why Banks are BASE Not ACID - Availability Is Revenue
Wednesday, May 1, 2013 at 9:00AM 
In NoSQL: Past, Present, Future Eric Brewer has a particularly fine section on explaining the often hard to understand ideas of BASE (Basically Available, Soft State, Eventually Consistent), ACID (Atomicity, Consistency, Isolation, Durability), CAP (Consistency Availability, Partition Tolerance), in terms of a pernicious long standing myth about the sanctity of consistency in banking.
Myth: Money is important, so banks must use transactions to keep money safe and consistent, right?
Reality: Banking transactions are inconsistent, particularly for ATMs. ATMs are designed to have a normal case behaviour and a partition mode behaviour. In partition mode Availability is chosen over Consistency.
Why? 1) Availability correlates with revenue and consistency generally does not. 2) Historically there was never an idea of perfect communication so everything was partitioned.
Your ATM transaction must go through so Availability is more important than consistency. If the ATM is down then you aren’t making money. If you can fudge the consistency and stay up and compensate for other mistakes (which are rare), you'll make more money. That’s the space most enterprises find themselves so BASE is more popular than it used to be.
This is not a new problem for the financial industry. They’ve never had consistency because historically they’ve never had perfect communication. Instead, the financial industry depends on auditing. What accounts for the consistency of bank data is not the consistency of its databases but the fact that everything is written down twice and sorted out later using a permanent and unalterable record that is reconciled later. The idea of financial compensation for errors is an idea built deeply into the financial system.
During the Renaissance, when the modern banking system started to take shape, everything was partitioned. If letters, your data, are transported by horse or over ships, then it's likely you data will have a very low consistency, yet they still had an amazingly rich and successful banking system. Transactions were unnecessary.
ATMs, for example, chose commutative operations like increment and decrement, so the order in which the operations are applied doesn’t matter. They are reorderable and can be made consistent later. If an ATM is disconnected from the network and when the partition eventually heals, the ATM sends sends a list of operations to the bank and the end balance will still be correct. The issue is obviously you might withdraw more money than you have so the end result might be consistent, but negative, which can’t be compensated for by asking for the money back, so instead, the bank will reward you with an overdraft penalty.
The hidden philosophy is that you are trying to bound and manage your risk, yet still have all operations available. In the ATM case this would be a limit on the maximum amount of money you can take out at any one time. It’s not that big of a risk. ATMs are profitable so the occasional loss is just the risk of doing business.
Consistency it turns out is not the Holy Grail. What trumps consistency is:
- Auditing
- Risk Management
- Availability
In a post-internet world where write availability is key the real world looks more like weak consistency + delayed exceptions + compensation rather than a mistake free world of perfect communication and transactions. Just like the old days, but now you have far more options on the ACID <---> CAP spectrum.
Reader Comments (15)
Really interesting post. Of courses have had to implement extensive auditing capabilities to compensate for potential lack of consistency, and when detected, processes to try and rectify
This introduces a level of complexity that banks have decided is acceptable to them, because the gains are signficant. But not every organization, or their apps, could tolerate this - so there are still plenty of places where consistency matters
Hi Todd,
I'm not sure where you're based but the example doesn't really seem to hold in the UK. I have no knowledge of the actual implementation but in an environment of free ATM transactions, availability doesn't seem the primary concern. An ATM out of service is not an uncommon sight and further, exceeding your limits will result in refusal of the machine to give you the requested cash.
Still, interesting points made about commutative operations.
Cheers,
Tim
I wonder if this works because money is fungible--the $1 the bank gets back from me in compensation is not meaningfully different from the $1 the ATM disbursed that it shouldn't have.
It doesn't have the same effect if I serve the wrong dating profile and promise to "make it up later." :)
Tim,
I'm also in the UK, and agree that the ATM system does not at first sight seem as clear within the UK, but I believe it it still true.
I _believe_ that what takes ATMs offline is a disconnection between the ATM and the service it uses for validating cards numbers and quick checks for if a card has been reported as fraudulent. Each ATM provider (actually groups of banks) does seem to run their systems separately, before applying them together at a later date.
I've certainly seen ATMs say that are unable to show my balance, but able to provide withdrawls.
I wouldn't be surprised if partitioning like this is why no transactions are applied on weekends / bank holidays, and yet you're still able to use your card.
Another Tim
The A can (should) still stand for Atomic. Who wants half a transaction masquerading as the entire transaction?
I worked on atm networks for years and yes they all work like this. Most networks work in online mode today. But the protocols support offline transactions. Remember that these networks support credit card transactions too (where tips can be applied after the transaction is authorized). Offline mode can be used to support the disconnected case or to enable scalability, validation is online but financial transactions are batched. Fully disconnected is less likely as pin checking must be online (at least until chip and pin).
Transactions are not atomic from atm to back office, if there is a communication error the transaction can be reversed. Some atm transactions are marked as forced-post, to indicate that money has already been given out these are processed even if they break back office rules, like no overdraft.
Some of these details are remnants of the past for the west, but disconnected operations are still very important in Africa, South America and India where long distance communications can be ropey.
While Atm transactions can be free, credit card transactions involve fees for the bank and revenue for the merchant. There is also reputational damage, when service its not available. During the first year or so of the financial crises there were a lot of rumours about different institutions. These institutions were very nervous that an interruption to service could trigger a run on their bank.
Good job guys keep it up
Good article Informative, interesting and to the point.
Tim,
> availability doesn't seem the primary concern. An ATM out of service is not an uncommon sight
a high rate of unavailbility does not prove that availability is not the highest priority as posited in the OP. It just shows that the network in question is poor at implementing that priority.
> exceeding your limits will result in refusal of the machine to give you the requested cash.
This is likewise not a contra-indication that availability is prioritized over consistency. This is covered in the risk management part of the article.
What is missing from this article is recognition that the actual financial update posted at the bank contains much more than a simple balance update. There can be dozens of tables updated and even more read. All of this work must be atomic. The ATM network has stand-in capability to allow for unreliable communications or host dropouts, at multiple levels, but when the message does get to the bank it ends up being a good old-fashioned ACID update. The ATM and online credit card transaction system with its support for standing in, force-posting, pre-authorizations, partial and incremental authorizations, and so on allows a network to be pretty available simply because the institutions have opted in to an agreement to make each other good. The banking system is not so willing to let a withdrawal be partially posted.
Thanks, nice post
Interesting article, thanks for information.
its really unfortunate that information professionals must be bombarded with such arguments from the web development community.
the classic ATM example is not to be taken literally... its symbolic of an IMPORTANT transaction. one that most can easily understand.
the fact that modern banking institutions have implemented risk management in order to facilitate availability does not change anything about the importance of ACID compliant transactions!
where I bank I can only take out $500 (risk) every 24 hours(time to consistency). lets imagine a new law where banks could no longer introduce such measures? where someone could take out large sums of cash? banks would quickly run from BASE to ACID!
How's that eventual consistency looking today?
it's about distributing bills and moving bits and stuff if anybody cares.
I do really like to read a sequel to this, considering the arguments in the comments about ACID.