Stuff The Internet Says On Scalability For July 11th, 2022

Never fear, HighScalability is here!

Every cell a universe. Most detailed image of a human cell to date. @microscopicture

Other images considered: one byte of RAM in 1946; visual guide on troubleshooting Kubernetes; Cloudflare using lava lamps to generate cryptographic keys; 5MB of data looked like in 1966

My Stuff:

• Love this Stuff? I need your support on Patreon to help keep this stuff going.


• Know anyone who needs to go from zero to cloud? My book teaches them all they need to know about the cloud. Explain the Cloud Like I'm 10. It has 440 mostly 5 star reviews on Amazon. Here's a 100% sincere, heartfelt review:

  

• Do you like Zone 2, Zone 5, interval, REHIT, HIIT, and bodyweight workouts? I made an app for that. Max Workout. I’m not just the programmer, I’m a client. I use it 4 times a week and if you want to retrain your mitochondria, get fit, get healthy, and live longer, I think you’ll like it too.

Number Stuff:

• 9: on-prem servers to run Stack Overflow. 200 sites with 6000 requests/second, 2B views/month, millisecond latency, <10% utilization.


• 3: hours a day of thinking is all it takes to get a Fields Medal.


• .73: our Kardashev Scale rating. We have a long ways to go.


• 20+: shards for Riot Games’ millions of users. 500,000 events per second peak. 8 terabytes of data generated per day. Kafka buffers data on-prem and moved to AWS on the backend.


• 4-bit: microcontroller on plastic. Using 4-bit and 8-bit logic instead of 16-bit or 32-bit helped. As did separating the memory that stores instructions from the memory that stores data. But they also cut down on the number and complexity of the instructions the processor is capable of executing. The team further simplified, by designing the processor so it executes an instruction in a single clock cycle.


• 100 trillion: digits in Google’s calculation of pi in 157 days of calculation time, which is just a little more than a month longer than the 121 days it took for the 31.4 trillion-digit calculation in 2019.


• 40%: increase in performance using Go generics in Google B-Tree implementation.


• 103: bits of advice from Kevin Kelley after 70 years spent balancing on this spinning blue-green globe.


• 1,000,000: concurrent connections is possible, you just have to tune everything correctly.


• 95%: IRShield's effectiveness against passive WiFi eavesdropping attacks.


• $100 million: GDPR fines in the first half of 2022, an increase of 92% over H1 2021.


• 18: months a botnet hid inside the rotting corpse of corporate security.


• 0.2: watts used for computation in the human cerebral cortex, but communication costs are 20-fold computational costs.


• 2/3: subsea fiber optic cable faults caused by fishing vessels and ships dragging anchors.


• 1 TW: solar to be deployed annually by 2030


• $3.4 million: value of unpaid work by reddit moderators.


• $34 million: permanently locked into the AkuDreams contract forever because of a bug. Do you really want software to control the world with no other compensating layers? Really?


• Five Billion: image-text pair dataset from LAION for open-source large-scale machine learning.


• 100+ Mbps: Starlink download speed.


• 20 trillion: events processed per day by Netflix (2021).


• 200 trillion: objects held in S3. That's about 29,000 objects per Earthling. S3 averages over 100 million requests per second and has grown by a factor of 250,000 in less than 16 years.


• 100 terawatt-hours: electricity used by the Bitcoin network in 2021, more than the typical annual energy budget of Finland.


• ~47%: speed increase of PHP 8.1 over PHP 8.0.


• $5000: monthly spend on Linode servers for the Overcast podcasting app.

Quotable Stuff:

• @functi0nZer0: BREAKING: Shock rippled across Solana today as everyone realized that their secret sauce for increasing transactions per second was to just make seconds last longer


• Pinterest: Today, Pinterest’s memcached fleet spans over 5000 EC2 instances across a variety of instance types optimized along compute, memory, and storage dimensions. Collectively, the fleet serves up to ~180 million requests per second and ~220 GB/s of network throughput over a ~460 TB active in-memory and on-disk dataset, partitioned among ~70 distinct clusters.


• Simon Wilson: One of my favourite uses for the GPT-3 AI language model is generating explanations of how code works. It’s shockingly effective at this: its training set clearly include a vast amount of source code.


• @brianleroux: AWS Lambda is many things but it is *not expensive*. Its $1 for 6M invocations. The whole point is stateless on-demand to achieve 100% utilization. Said another way, provisioned server capacity is not 100% utilization. We over provisioned them to allow for spikes. Comparisons where the Lambda is being invoked 24/7 is totally disingenuous. A cluster of servers running 24/7 mostly idle is *not* the same thing. Statelessness is a feature. It enables 100% utilization and, more often than not, that efficiency will translate directly to saving big $$$ not costing more.


• Leslie Lamport: Well, they didn’t have an algorithm, just a bunch of code. Very few programmers think in terms of algorithms. When trying to write a concurrent system, if you just code it without having algorithms, there’s no way that your program is not going to be full of bugs…Yes, the importance of thinking and writing before you code needs to be taught in undergraduate computer science courses and it’s not. And the reason is that there’s no communication between the people who teach programming and the people who teach program verification.


• @jmhodges: GraphQL makes your public API equal to a generic database and -- worse -- a generic graph database. The amount of maintenance work is sky high. Locking the query capabilities down means you're just doing normal APIs but not locking it down means infinite performance work


• DSHR: Moving all storage to the NVMe interface is an interesting trend.


• @adrianco: This is also useful info: "According to Intel, the CPU still represents about 32 percent of the IT equipment power budget, with memory only burning 14 percent, peripheral costs around 20 percent, the motherboard around 10 percent, and disk drives 5 percent."


• @chrismunns: I think one of the biggest lies we tell ourselves in IT/tech is that we need to prepare for "scale" of traffic. Generally speaking 99% of apps never break 1000 rps. Even with that you can likely serve millions of active users in a month. 1k rps comes easily with Lambda, or Fargate, or DIY with EC2/ECS/EKS/etc. You won't stress a load balancer with this much traffic. you likely won't stress a database of any major recent engine either. Internally facing enterprise apps (a surprising amount of the IT footprint for most companies) barely break 10s of RPS. Again why would you even run a server for any of that. As soon as we find a way for people to really understand async patterns, to realize that cached data gets you pretty far for a lot of apps, and that not running a server 24/7 just to HA a 2 rps app can be ok, then we'll see true sustainability of IT


• @aeduhm: I do this for a multi-region setup where I want a single custom domain with latency routing to hit the closest AppSync API. I did some global speed testing, and with my very unscientific approach I was seeing a ~30-50ms latency increase with the APIGW proxy.


• @josh_wills>: OH: "This microservice could have been a SQL query"


• adrian_b: In conclusion, to avoid the huge amount of resources needed by a CPU for guessing about the programs, the solution would be a high-level language compiler able to transparently allocate the data into a hierarchy of heterogeneous memories and schedule transfers between them when needed, like the compilers do now for register allocation, loading and storing. Unfortunately nobody has succeeded to demonstrate a good compiler of this kind.


• @werner: I still get annoyed by every piece of AWS that is not #serverless!


• @benschwarz: This is true, at least in my experience. In the earlier days, for a long while @Calibreapp handled tens of millions of API requests per month on a $7 heroku dyno. You don’t need k8s.


• @guzmonne: Use tags. Don't be afraid to populate them with information. Names, dates, URL of pipelines, project information, etc. And, if what you are creating is disposable mark it so.


• @JoeEmison: Checking back in on the Branch AWS bill; I think I last posted about 2 years ago, when it was still under $1,000/month. In the meantime, Branch has grown ~15X YoY for two years, and so the bill has also grown, to just around $10K/month. As a reminder, Branch is fully serverless, and carries a complete copy of production into every staging/test/etc environment, as well as every developer environment. Every developer has his/her own AWS account (also too with every other environment). Biggest spend is on DynamoDB at $4k/mo. We do buy capacity for some tables in prod, but otherwise go on demand. The majority of our DynamoDB cost is in backups, not in usage. Next is Lambda, at around $2200/mo. We do buy some provisioned capacity for Lambda, but might stop soon (not sure it's doing much for us). If we do that, the bill should be closer to $1700/mo. We have zero devops full time. Our more senior developers do all our devops in PRs. It’s a minimal amount of time.


• @doctorow: In other words, Amazon has converted its "customer-centric" personalization system, which emphasized the products it predicted you would like best, into an auction house, where the products that have paid the most come first. Amazon sellers - 60% of sales and rising - must choose between spending on better products or better placement on Amazon. If they choose better products, you'll never find them. Amazon made $32b last year contributing to the crapification of everything it sold.


• @t3rabytes: For the last 6 months I've been planning a mega infra migration for the @EpicGames Store and this morning we pulled it off without a hitch. That's: - a new domain for the Store (www. -> store.) - a new AWS account - raw EC2 to Kubernetes - additional CDN connectivity. The biggest drivers for this were SEO and single point of failure reduction. Beyond that though, we get some insane eng quality of life improvements -- prod deploy time drops from 45 minutes to 6, ephemeral/branch envs, faster auto-scaling, etc.


• @_Amanda_Walker: As a Xoogler, I did a bunch of the opposite. "We don't need X, we're not Google." X = stuff like "microservice architecture", "hot failover", "ability to handle millions of qps", etc. At one point I said "I could run production on my laptop. Don't worry about scaling yet."


• huksley: made a mistake and accidentally created a serverless function that called itself. In a recursive loop, with a 30s timeout. I thought I fixed it and deployed the code to the dev environment. Imagine the terror when I woke up the next day to see the AWS Billing alert email saying I already owed $1,484! I removed a function and deployed it again in 30 minutes, but it was too late. It has already run for 24 hours, using over 70 million Gb-Second!


• Ably: Round trip latency from any of our 205 global points of presence (PoPs) that receive at least 1% of our traffic: < 65ms for the 99th percentile (measured at the PoP boundary within the Ably access network, which will be closer than a datacenter).


• Plausible: We moved from our PostgreSQL database to ClickHouse to make the app much faster and handle much bigger websites. Moving to ClickHouse was the best technical decision we’ve made, and we’re now able to count more than one billion page views per month while still providing a very fast-loading dashboard.


• Rick Houlihan: Sooner or later when us-east-1 has problems enough "well architected" workloads will be deployed for failovers to brown out other AWS regions and cause cascade failures.


• Elizabeth Kerr: For the first time in 2022, Microsoft Azure surpassed Amazon Web Services (AWS). Azure increased its adoption rate from 73 percent the previous year to 77% this year, 2022, while AWS dropped from 77 percent adoption rate to 76 percent.


• Mike Shatzkin: We have gone from about 5000 bookstores in 1990 to more like 1250 today, and, on average, today’s stores carry a fraction of the number of titles the bigger ones did back then. We have gone from about 500,000 titles available in 1990 to about 20 million titles available today. We have gone from perhaps 15 or 20 publishers with their own distribution organization that were real competitors for most books an agent might sell in 1990 to five (soon to be four) today. We have gone from a world where nobody could effectively deliver a book without a publishing organization in 1990 to one where anybody can today.


• @GergelyOrosz: Years of working in Uber’s payments team changed my view on distributed systems where participants can make money. I don’t believe any such distributed system can be as efficient as a centralised one. A centralised system spends SO much on fraud reduction and customer support. In ~10 years there’s been much talk on building a P2P version of Uber. Yet it never gained momentum. Any such system is doomed to fail as many drivers & riders would abuse the system to maximise short-term profits. Incredible what both parties do to make or save money. What changed my belief is seeing all the things drivers do to make money (using every single arbitrage opportunity - eg forcing riders to cancel & collect the cancel fee, as this is more profitable at eg airports) and also riders (from money laundering to car theft etc).


• glenngillen: as for the service dynamic: the whole "2 pizza team" thing is both a blessing and a curse IMO. Once you launch a service at AWS’s scale you’ve immediately got tens of thousands of users. And AWS takes stability/availability/scalability incredibly seriously. I think it’s more likely that team is now in a permanent operations mode fixing weird edge cases you find at scale and keeping the lights on.


• Daniel Spielman: That’s true with a lot of my research. A lot of the time, the way I’ve done something is not because it was the problem I was setting out to solve. I was setting out to solve something else, and it didn’t work. But I was aware enough of the intellectual landscape to figure out what I could use it for.


• @WillManidis: In April, a team of cyberattackers attempted to breach an undersea cable off the coast of Hawaii. The cable is part of a network that carries over 95% of the world’s data.


• @fchollet: To put the "scale" narrative into perspective... The brain runs on 15 watts, at 8-35 hertz. And while we have ~90B neurons, usually only ~1B are active at any given time. The brain is very slow and does a lot with very little. The brain's energy consumption is similar to that of a very bright LED bulb -- or 1/5th of that of a standard incandescent bulb. It's not exactly a data center.


• Hiroyuki K. M. Tanaka: The cosmic time synchronizer works by synchronizing devices around cosmic ray events detected by those devices. This could bring accurate timing abilities to remote sensing stations, or even underwater, places that other methods cannot serve.


• @hkarthik: The longer an engineer works in a monolith, the more they yearn to decompose it into micro-services. The longer an engineer works across a ton of micro-services, the more they yearn for the days of working in a monolith. The grass is always greener.


• @jeremy_daly: Yesterday @MongoDB announced the GA of Atlas Serverless Instances! The usage based discounts might just be the greatest feature of this! The future of #serverless is now!


• @fredbrancz: Recently heard from a @ParcaDev user: "We had no idea 55% of our CPU time was spent in Istio/Envoy, but Parca made it obvious!" 55%!! More than half their infrastructure cost!


• @GergelyOrosz: Happens every single time: 1. Company realizes: "We pay ~2-3% of our revenue to payment processors! This accounts for more than $1M/year!" 2. "We should cut out the middleman." 3. They hire a team, build their own processing... to realize it's more complex & way more expensive


• @houlihan_rick: RDBMS relies on CPU efficiency to offset the time complexity of indexed joins. Since 2014 not even the Top500 supercomputers have been able to match Moore's law. The trend is clear...TCO of RDBMS is rising as CPU performance is flattening.


• David Rosenthal: Thus we need to ask in whose interest a vast increase in the supply of transactions would be? It pretty clearly isn't in the miners' interest, as we see with the history of attempts to increase the Bitcoin block size. Liimiting the supply of transactions allows the miners to profit handsomely in periods of high demand. At first glance low fees would appear to be in the users' interest, but in practice it isn't. Low fees lead to flooding, and outages like Solana's. In practice, miners control the system, so even if it were in the users' interesst it wouldn't matter. Catch-22.


• Kevin Mitchelle: Evolution is in the design game. But it produces functional designs by exploration and selection, not by raw creation from the void. It’s not just luck that living systems have the functionalities they do. The principles that enable them exist and can be found by evolution


• Small Datum: The impact from fixing a p50 perf bug is easy to predict. Everything gets slightly faster and/or more efficient. But p99 perf bugs are outliers -- they don't make everything slower so fixing them doesn't make everything faster.


• Jordana Cepelewicz: Faced with a threat, the brain has to act fast, its neurons making new connections to learn what might spell the difference between life and death. But in its response, the brain also raises the stakes: As an unsettling recent discovery shows, to express learning and memory genes more quickly, brain cells snap their DNA into pieces at many key points, and then rebuild their fractured genome later.


• @asymco: The App Store has 1.8 million apps, and it is visited by 600 million people every week. The earnings of smaller UK app developers have grown by 84% in the last two years, with the number of smaller developers joining the App Store up by 40%.


• Slack: What caused us to go from a stable serving state to a state of overload? The answer turned out to lie in complex interactions between our application, the Vitess datastores, caching system, and our service discovery system.


• IIAOPSW: This is quantum woo fueled neo cartesian dualist bullocks. Sorry not sorry. For starters, if events in space and any point in time are accessible as "memories", the laws of physics that enable that shouldn't constrain the only application to memory.


• Backblaze: The 6TB Seagate (model: ST6000DX000) continues to defy time with zero failures during Q1 2022 despite an average age of nearly seven years (83.7 months).


• toeknee123: Implementing a simple and effective tagging strategy for visibility and monitoring is one of the most important aspect of cloud cost control. From our experience, this is where most of our own users start.


• myrandomcomment: What is old is new again. IBM mainframes had a concept called a "channel controller". Everything connected to the mainframe basically was a computer itself that offloaded the main system. Every DASD (disk), communication link was its own system.


• SOLAR_FIELDS: defining the logic of checks themselves in YAML is a trap. We went down this DSL route first and it basically just completely falls apart once you want to add moderately complex logic to your check. AirBnB will almost certainly discover this eventually. YAML does work well for the specification of how the check should behave though (eg metadata of the data check). The solution we were eventually able to scale up with was coupling specifications in a human readable but parseable file with code in a single unit known as the check. These could then be grouped according to various pipeline use cases.


• pbalau: There is no tool in the world that will make you good at something you don't know.


• ZeroCool2u: My main gig runs workloads primarily on AWS, but I work with a small company as well that is completely on GCP and I gotta say the difference is night and day in instance allocation and startup time. GCE is so much faster it's infuriating when I've gotta go back to work and sometimes have to wait more than 10 minutes in the worst case for an EC2 instance to finish booting in an EKS cluster that's already running!


• @HNStatus: For the record, HN is a single process on a single server. I can't snap my fingers and magically make it redundant.


• @jmhodges: GraphQL is a trap. GraphQL makes your public API equal to a generic database and -- worse -- a generic graph database. The amount of maintenance work is sky high. Locking the query capabilities down means you're just doing normal APIs but not locking it down means infinite performance work


• @Carnage4Life: Today I learned 99% of cryptocurrency transactions happen on centralized exchanges like Coinbase & Binance which dont write to the blockchain but instead to their own SQL databases. Obvious in hindsight since blockchains are too slow & unscalable to build a product like Coinbase


• @Gankra_: "We still do not have actual code failures that have been traced to this problem; we do now have small sections of contrived code that can result in the problems discussed here. And we do not have any way to convince ourselves that real code is immune from such problems." -Boehm


• @sheeshee: fascinating. you could as of today really build awesome micro stacks with tools like firecracker, sqlite, k3s etc etc - seems that cloudflare is also opensourcing their serverless V8-based thing.. the new startup empire is needing one machine, vasily, one machine only :)


• @castrojo: There are now over 152,000 people on the Kubernetes slack in case you were wondering if cloud native is slowing down.


• @nicktornow: But I don’t know how we (Twitter, society) will manage the moderation problem for "dangerous" information and thoughts and the real damage that can happen to us all through appeals to our baser instincts, as is on full display in the gruesome history of the 20th century, etc.


• @GergelyOrosz: I was a happy Google Apps Engine customer 2010 spending ~$2,000/month on an app that grew to ~1M users. In 2011, Google decided to increase prices ~5-10x changing billing fundamentals, gave 30 days notice and no tooling to prepare. I’m never going back to GCP / GAE again.


• @rakyll: There is a real toil on working on infrastructure that is not visible. You see a region launch from a cloud provider and dismissing it like "yea, whatever" even though the immense scale of work that went into it.


• Ken Shirriff: The Apollo missions to the Moon required complex hardware to communicate between Earth and the spacecraft, sending radio signals over hundreds of thousands of miles. The premodulation processor was a key component of this system, combining voice, scientific data, TV, and telemetry for transmission to Earth.1 It was built from components that were welded together and tightly packed into a 14.5-pound box.


• @houlihan_rick: I have had this #SingleTableDesign/@GraphQL conversation a few times. Document/Item == GraphQL Type. GraphQL Response == A structure of Type instances. Single Table Query == An array of Type instances. GraphQL Resolver == Iterate array, map instances to Response


• @GergelyOrosz: Blockchain, applied in the real world with a case study: To handle the ‘staggering’ amount of data of 500K loads per year (~1,500/day), Walmart Canada contracted an innovative blockchain solution using a private cloud, running 600 VMs to handle thousands of transactions per day.


• @pati_gallardo: To all the ones saying "for a large enough N Big O comes back into play" The problem is that since the mental model doesn’t include caches folks don’t even think to measure the performance differences between their O(log n), O(n log n) or O(n) algorithms.


• Rob Verger: In June of 2020, a pilot flying an F-22 in Alaska reportedly became disoriented, and the aircraft likely would have crashed were it not for the intervention of a software system on the fighter jet.


• @JoeEmison: "Our technical debt is terrible and so we have to rewrite everything" and "We are going to use the same team we have been using to develop the code for years to do the rewrite" are wild things to say back to back but literally 100% of the time that’s what I hear.


• @houlihan_rick: Small reads/writes are not good for S3 when they are happening frequently. It is much cheaper to use DynamoDB for high frequency access to small Items.


• @KevinNaughtonJr: my typical day at Google: 9am - reverse a linked list 11am - count unique ways to climb a staircase with dp 12pm - lunch 3pm - help animal escape NxM matrix efficiently 4pm - invert a binary tree 5pm - commute home using Dijkstra's


• @thdxr: company I advise moved to serverless from EKS - their old bill for just application hosting was ~$4000 a month. new bill is $70…cost savings with lambda can be counter intuitive. doing 60 requests a minute feels like a lot. if the average req takes 20ms to complete, you're only using 1.2s of that minute. 29 minutes a day. with a container, you pay for 24 hours to use it for 29min - 98% over provisioned


• @bernhardsson: This might be controversial but I'm increasingly convinced that the future of infrastructure is to put all the infra inside the app code. What Pulumi & CDK does to infra but one step further and you just put everything together in code.


• @royvanrijn: 💻 Developers: "I hate meetings" "Just let me code, I'll skip this one" "Sigh, not *another* meeting" ☹️ Also developers: "This project has no clear vision." "Management has no idea what's going on."


• @jpetazzo: OH: "we have over 170 microservices, because our principal engineer is very knowledgeable about distributed systems"


• Brent Mitchell: I have been using serverless computing and storage for nearly five years and I'm finally tired of it. I do feel like it has become a cult.


• Neel Chauhan: I’m not the first person to tell you this, and certainly not the last, but Web3 is in fact centralized, just as Web2 was. Web3 is just a worse version of Web2.


• @stewartbrand: Interesting: how much bad news is anecdotal and good news is statistical.


• jedberg: I'm a huge proponent of microservices, having worked on one of the earliest and largest ones in the cloud. And I absolutely think that they provide huge advantages to large companies -- smaller teams, easier releases, independently scaling, separation of concerns, a different security posture that I personally think is easier to secure, and so on…every startup I advise I tell them don't do microservices at the start. Build your monolith with clean hard edges between modules and functions so that it will be easier later, but build a monolith until you get big enough that microservices is actually a win.


• Hugo Rocha: Event-driven services and the ability to route events to specific service instances propose an elegant way to remove concurrency from a solution, by solving concurrency by design, we provide the foundations for a truly horizontally scalable system.


• @colmmacc: Not to give away our super secret strategy but something that people often under appreciate about AWS is that new regions are by far the biggest needle mover for our business. The speed of light, sovereignty, and data locality all matter! Right now is a very interesting time at AWS because we're completing one of our key super-scaling journeys. When I joined, regions were pets, but we've been making them cattle. The numbers are always mind-blowing, but it's not something people cover or see a lot of.


• @penberg: With V8 providing virtualization, deployments on edge effectively become either JavaScript or WebAssembly serverless functions. However, the next frontier in edge computing infrastructure is solving database access, which is still has a latency problem. 7/


• Yasemin Saplakoglu: Contrary to expectation, the synaptic strengths in the pallium remained about the same regardless of whether the fish learned anything. Instead, in the fish that learned, the synapses were pruned from some areas of the pallium — producing an effect "like cutting a bonsai tree," Fraser said — and replanted in others.


• @houlihan_rick: Scale is enabled by efficiency, and efficiency lowers cost. Designing for scale usually means more dollars in your wallet.


• @houlihan_rick: The most important lesson I learned at Amazon is that whenever you don't think you need to worry about designing for scale you are wrong.


• @KarlBode: I'm sorry you can't ride your $2500 exercise bike because the server is down, welcome to the future


• @muratdemirbas A distributed system is a collection of autonomous nodes, communicating over a network to address a problem collectively, with no shared memory, with no common physical clock.


• @Grady_Booch: The read latency of the @aws Elastic File System is 600 microseconds, The read latency of the first commercial hard drive - the @ibm 350 Disk Storage Unit, released in 1956 - was 600 milliseconds.


• @317070: Very happy to share my secret project for the last 3 years: we were busy getting AI to run a nuclear fusion reactor. Spoiler alert: it works! Deep reinforcement learning is pretty great at working with sci-fi things where human intuitions tend to break down.


• Guy Sela: When DynamoDB is 10x Cheaper Than S3. Never assume that S3 would necessarily be cheaper than a NoSQL solution, look at the numbers for your specific use case and you might reach surprising results. Remember to compress data before calculating the size. Consider the simplicity of your solution and potential future use cases as part of your "hidden" cost calculation.


• @ccsdevio: This is a short AWS appreciation thread. For two dollars a month (knock on wood), I have a site running on an EC2, behind a load balancer, served by CloudFront on a CDN with HTTPS, with automatic health checks performed 16 times a minute in 8 regions across the world. I have a cloudwatch alarm enabled that hits pagerduty if those healthchecks start failing, and another that stops the EC2 and hits pagerduty if CPU usage gets too high (


• @mjtech01: Our children's @philips electric toothbrush contains a 32-bit ARM processor, clocked at 48MHz. I've even got the development tools for the Cypress PSoC 4, and they've nicely labelled the debug pads. It's just a toothbrush. Really.


• hermannj314: If the contractor finds asbestos, he's allowed to quote me more to do the work I hired him for. If my mechanic says my car has old parts not in inventory, they may cost more and take longer to replace or maybe they charge more to do the work. When the laws change, your lawyer gets billed to review and update your contracts. As new legal precedents are formed, new legalese is created and updated and contracts are renegotiated. Few things in the world are future proof for every conceivable externality. No one considers these professionals idiots simply because their domain has 'technical debt' too. It is ok to have technical debt, to mention it, to bill to fix it, and to talk about it. And you only look like a fool if you try to cover it up and pretend it isn't there because 'its not the customers's problem'


• Whitney Claven: If I drop my coffee mug and it falls, I’d like to think that’s gravity," says Adhikari. "But, in the same way that temperature is not ‘real’ but describes how a bunch of molecules are vibrating, spacetime might not be a real thing. We see flocks of birds and schools of fish undertake coherent motion in groups, but they are really made up of individual animals. We say that the group behavior is emergent. It may be that something that arises out of the pixelation of spacetime has just been given the name gravity because we don’t yet understand what the guts of spacetime are


• @AdamHisley: Went from K8s to Lambda. The main motivation at the time was devex/dev ownership. K8s seemed to work best as a walled garden "platform." Lambda was simpler for devs to grok and after a few months to pilot + write CDK constructs we had eng teams owning their infra provisioning.


• @reubenbond: Any sufficiently large distributed system is always in a state of partial failure


• @matthewpskelton: "Cloud versus Datacenter" is mostly missing the point. Is your infrastructure programmable via code? Is your infrastructure testable via code? "Cloud" is programmable as standard. That's why cloud is a force multiplier, not just because someone else is running things.


• Steven Lemon: Ultimately, rather than separate our monolith into separate services, we started to break our solution into separate projects within the existing monolith. This division gave us a bit of additional structure and a better indication of where coupling and duplication existed, without the extra weight and challenges of microservices.


• @BorisTane: if your team is building abstraction layers on top of managed cloud services in an effort to remain "vendor-agnostic", you're seriously doing this wrong instead, embrace the services at your disposal and use all their relevant features. don't drive the lowest common denominator


• Geoff Huston: So perhaps an address these days is just an ephemeral transport token that distinguishes one packet flow from another, and really has little more in the way of semantic meaning. In this world what’s holding the Internet together is not IP addresses and routing: it's the DNS and the common name system.


• temuze: Back at my old job in ~2016, we built a cheap homegrown data warehouse via Postgres, SQLite and Lambda. Basically, it worked like this: - All of our data lived in compressed SQLite DBs on S3. - Upon receiving a query, Postgres would use a custom foreign data wrapper we built. - This FDW would forward the query to a web service. - This web service would start one lambda per SQLite file. Each lambda would fetch the file, query it, and return the result to the web service. - This web service would re-issue lambdas as needed and return the results to the FDW. - Postgres (hosted on a memory-optimized EC2 instance) would aggregate. It was straight magic. Separated compute + storage with basically zero cost and better performance than Redshift and Vertica. All of our data was time-series data, so it was extraordinarily easy to partition. Also, it was also considerably cheaper than Athena. On Athena, our queries would cost us ~$5/TB (which hasn't changed today!), so it was easily >$100 for most queries and we were running thousands of queries per hour.


• jiggawatts: This is mere kilobytes of data over the length of the entire movie, and can tolerate hours of latency for most aspects (e.g.: analysis, charge-back, ML, etc...). A lot of the time I look at "modern cloud architectures" and I see a firehose of raw data being spewed out in hideously inefficient formats. Think 1 kilobyte of JSON to represent a metric, often a single number or data point. It's not just NetFlix, this is the default for most of Azure Monitor and its agents as an example that would affect a lot of people.The write amplification is just insanity. Orders of magnitude of overhead, not data. It's as if the entire world forgot that binary formats exist. Google in no small part benefits from remembering this lost art -- they default to gRPC and similar packed binary formats. One reason Kubernetes (written mostly by ex-Googlers) is so fast is because internally it's all compiled code and gRPC. Compared to, say, Azure itself it is ludicrously faster. Literally hundreds of times lower latencies for common operations.


• Doug: So this is actually one of the nice things that I also really like about Concord is that if you take a single process and look at it, it’s not just a state machine. It’s actually a state machine that’s composed of smaller state machines who themselves might be composed of smaller state machines. What that means is that you can kind of draw your boundaries in lots of different places. You can take a single program that is maybe performing too slowly and split it into two smaller ones or you can take a swath of programs that all have similar kind of needs in terms of data and stitch them together, and it lets you play with the performance profiles of them in a really nice way.


• The Victorian Internet: the construction of a global telegraph network was widely expected, by Briggs and Maverick among others, to result in world peace: ‘‘It is impossible that old prejudices and hostilities should longer exist, while such an instrument has been created for the exchange of thought between all the nations of the earth.’’...So much for universal peace and understanding. The telegraph was providing new ways to cheat, steal, lie, and deceive.


• The Victorian Internet: BY THE EARLY 1870S, the Victorian Internet had taken shape: A patchwork of telegraph networks, submarine cables, pneumatic tube systems, and messengers combined to deliver messages within hours over a vast area of the globe. New cables were being laid all over the world. Malta had been linked to Alexandria in 1868, and a direct cable was laid from France to Newfoundland in 1869. Cables reached India, Hong Kong, China, and Japan in 1870; Australia was connected in 1871, and South America in 1874. In 1844, when Morse had started building the network, there were a few dozen miles of wire and sending a message from, say, London to Bombay and back took ten weeks. But within thirty years there were over 650,000 miles of wire, 30,000 miles of submarine cable, and 20,000 towns and villages were on-line—and messages could be telegraphed from London to Bombay and back in as little as four minutes. ‘‘Time itself is telegraphed out of existence,’’


• Cal Newport: The Deep Work Hypothesis: The ability to perform deep work is becoming increasingly rare at exactly the same time it is becoming increasingly valuable in our economy. As a consequence, the few who cultivate this skill, and then make it the core of their working life, will thrive.


• Matt Rickard: Early-stage startups shouldn't run on Kubernetes yet. But eventually, growth-stage and large companies should be running on Kubernetes in some form. Kubernetes Maximalism doesn't mean one-size-fits-all. Infrastructure should progressively grow with your workloads and team. How can you choose the right technology now so that you can maximize growth and minimize pain later when you inevitably outgrow it?


• CharlieDigital: The purpose of a startup isn't to play around with tech: it's to find product-market fit and zero in on the business value. If you think of a startup as a place to play around with tech, you have no place in a startup. You should almost always choose the "dumbest" possible technology used in the "dumbest" possible way so that you can hire anyone and they can step in and be productive when you need to scale. It minimizes ramp up time. It minimizes deployment and operational complexity. It minimizes the ways the system can fail. Literally build the dumbest thing that solves the core business problem that creates value for the user and then figure it out from there.

Useful Stuff:

• Hybrid Cloud is BS? Maybe not if you’re Walmart. Walmart Amps Up Cloud Capabilities, Reducing Reliance on Tech Giants:
  • developed the capability to switch seamlessly between cloud providers and its own servers, saving millions of dollars
  • hybrid cloud allows us to be able to draw the best that the public cloud providers can offer and to be able to combine that with something that is really purpose-built for us,
  • 10,000 edge nodes across a network of servers that sit at Walmart stores and in its distribution facilities, it has also built custom software that allows it to run its back end operations across any cloud system,
  • Walmart said its cloud system has saved the company between 10% and 18% in annual cloud spending and has helped reduce cloud outages
  • They didn't say how the money was saved. And is that worth the CAPEX that must have been invested in the development of such a complex system?
  • @forrestbrazeal: 5 things I've learned the hard way about multi-region / multi-cloud: 1. Building your own control plane, whether cross-region or cross-cloud, involves cost and complexity that you will have to deal with every day - not just when a managed service has downtime…And a final reminder: No matter how many clouds, regions, or zones you use, the ultimate accountability for building systems that are resilient when you need them always rests with you.


• It's relations all the way down. Carlo Rovelli defined relational quantum mechanics (RQM) as meaning something like all the properties of a physical object only exists in relation to another physical object. This resonated with me because after years of deep diving into relational and object modeling, I came to a similar conclusion in the realm of software. When I was teaching a relational database class, the typical way of introducing normalization was it's a way of reducing update anomalies. But when you consider the nature of relations it's deeper than that. The only thing that holds a relation together is identity. All other properties are a product of some relationship. Even a natural property like created_at is a relationship with time. Same with location, and every other property. In practice, that's doesn't matter because in practice relational models are for solving problems, not for plumbing the mysteries of the itness of what something actually is. And if you've ever been in a design meeting conversations often turn more into more of a philosophical inquiry than a practical problem solving session. When people say something is "over designed" this is often the reason. We turn design into a psychonalaytic therapy session with our client. We plumb their depths searching for insight. In the end time runs out, so we slap some properties in a table, add some foreign keys, make some queries, and call it a day, knowing that whatever we did missed the mark. Of course the ground shifts later because the model never matches the territory and epicycles keep being added to get the right result as defined by practicality. Object modeling produces the same insights. In an object model what exists is object identity. All other ISA, HASA, USES decisions are ultimately relational. This is also why no two people or groups will produce the same design for the same problem. They're grabbing different parts of the elephant. This relates to RQM because the essential idea behind RQM is that different observers may give different accurate accounts of the same system. Most of what we call technical debt stems from the fact the nothing in software has a true essence. Nothing we build is built on a true and everlasting nature. Our conceit is that we are building something based on a deeper truth, but when we look at something it's from a prism of relations, none of which are objectively right from a software persepctive. Whole networks of relations shift over time as problems change, people change, technology changes, understanding changes, as everything changes from that heroic point in time a human mind took a stand, created code, and moved on. The antidote is said to be simplicity. But in the same way the Zen and The Art Of Motorcycle Riding rediscovers Taoism through the concept of quality, simplcity is just another relation in the vast possibility space of relations. 


• Triple OG Stack Overflow still marching to the beat of their own servers. Engineering Stack Overflow with Roberta Arcoverde:
  • 1.3B page views / month
  • we have a code base that is 14 years old. We run on prem, we run on our own data center. We haven't gone to the cloud. We also have a monolithic application. So we have not breaking down into services or microservices in. We have a multi-tenant web app.net based, running on a single app, pull across nine web servers only. And that same application single app pool on AIS is handling 6,000 requests per second at the moment
  • But so far our engineering teams working on the Q and a platform, I think we are at around 50 engineers at this point.
  • we could, in theory, be running on a single web server. We wouldn't want to do that, but theoretically it would be possible.
  • we always start from asking the question, what problem are you trying to solve? And in our, the problems that these tools and bandwagons try to solve are not problems that we were facing ourselves. So when you think about things like a monolith, for example, right? Why do you break down a monolithic to microservices or services typically because you want to scale two separate teams.
  • You want to have multiple teams working on the same project without stepping on each other tolls. You want to have fast deploys. For example, fast deploys have never been a problem to us. We put stack overflow in production multiple times a day in four minutes. That's the time it took to build Seco floats abroad. If we had to revert a change, for example, it was always super fast, right? In, in a matter of minutes, we could revert our deployments, even though it is still a monolith, not a bunch of really small applications, but a single big application and invest in efficiency on those things.
  • And that's important for keeping in mind that stack flow was built to scale that way. Right? We are designed for low latency. We are designed to grab requests, run them as acute, a few queries and return as soon as possible so that we can pick up the next one. We cannot handle a lot of memory pressure. So we also designed for low allocations everywhere. We try to avoid creating objects that will have to be collected very often. We try to avoid memory pressure on those nine web servers so that we don't have to stall on garbage collections because a stall and a garbage collect is terrifying for these web servers. So we try to run smooth and with a very low memory footprint, because that's how that scalability model was designed to work on, on the infrastructure that we have.
  • 80% of our traffic is anonymous. People go to the question show page, which is the page that we show when you are Googling something and you get a result that takes you directly to stack overflow the solution to your problem, right
  • we have two different levels of cash on, on the front, right on, on the memory and the web servers. And we also have our SQL server servers. They have 1.5 terabytes of Ram. So a third of the entire database can be very quickly access the in RAM. @alexcwatt~: giving SQL more RAM is better than caching page fragments with Redis.
  • So we removed all cash and that was like three or four years ago. We stopped cashing that page. We stopped caching the content and little did we know it didn't really make any measurable effects on the performance. We were still able to handle requests and send responses in super fast because of how the architecture was built at the time. So currently for example, the average time to render on that page is around 20 milliseconds. So even without cash.
  • we did this regular exercise where we would try to understand how much it would cost to run stack overflow in the cloud. And it was just never worth it, which is why we never did it these days. When we think about the cloud, we are thinking less about the power that it would take and more about latency. The other thing that we optimize for a lot with the current design is low latency. So we have an infrastructure that has single hops between nodes and those hops are connected via 10 gigabytes network cables and make Craver will tell you all about it.
  • Well, pragmatism is what I'm hearing. Just comes down to productivity and pragmatism. Those are the words of the day.
  • We have rolling builds. So we have those nine web servers. They are under an HTA proxy front, and every time that we need to, to run a new, to deploy a new version of the site, we take a server out of rotation, updated there, put it back on rotation.
  • Shopify also runs on monolith. Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity.
  • Dan Luu: Wave is a $1.7B company with 70 engineers whose product is a CRUD app that adds and subtracts numbers. In keeping with this, our architecture is a standard CRUD app architecture, a Python monolith on top of Postgres. Starting with a simple architecture and solving problems in simple ways where possible has allowed us to scale to this size while engineers mostly focus on work that delivers value to users.
• But Airbnb moved from a monolith to a microservices. Results? Mixed. Airbnb’s Microservices Architecture Journey To Quality Engineering.
  • Main philosophy: architecture can't impact business growth.
  • 2008-2017. They used a monolith and monorepo until they reached about $2.6B in revenue, so you have some runway before you need to switch. At first, small teams worked on features with limited dependencies.
  • What problems did they have? Velocity of software changes decreased. Hard to work on features in parallel. Component ownership is confused.
  • 2017-2020. Microservices. The monorepo was split with a dedicated service migration team responsible for component transition. Yes, it helps to have resources.
  • They reached $5B in revenue, but features are a cross-cutting concern, so implementing a feature required changes in multiple services involving multiple teams. Fine-grained services distributes complexity across services, make it hard to collaborate across teams, and even small changes are disruptive.
  • Thrift was replaced by GraphQL as a unified data access layer. Annotations and codegen were used to improve the code. Love me some codegen.
  • 2020- present. Micro + Macroservices.
  • The main changes were unified APIs, central data aggregator, and service block facade APIs. It's not exactly clear what this means at a code and team level.
  • At this point people will say the problem was they weren't doing microservices right. This is the self-fulfilling prophecy problem every philosophy has. If it didn't work, you weren't doing it right. In practice right-sizing services is always a problem because there are rarely natural service boundaries.
  • Keep in mind a monolith should still be service based on the inside, it shouldn't just be a tangle of code. So in theory taking monolith should be just taking those already existing service point APIs and making them remote. Ownership or lack thereof wouldn't have to change. And I think the monorepo is a misdirection. What matters is code organization, the number of repos means nothing.
• Videos from Strange Loop now available.
• Which cloud rules them all?
  • According to the Cockroach Labs 2022 Cloud Report the answer is…nobody. All three cloud providers offer price-competitive options. While there are no winners, they have some good advice on making choices:
  • Thanks to AMD, GCP and Azure instances are outpacing AWS instances. GCP’s t2d instance type (which uses AMD Milan processors) took the top overall spot for both large and small nodes in our OLTP testing.
  • For even relatively small amounts of persistent block storage, the cost of running a particular workload is much more influenced by the cost of the storage than it is the cost of the instance. For persistent workloads, it is extremely important to optimize cost calculations based on this consideration. This means preferring mid-tier storage options (pd-ssd on GCP, gp3 on AWS, premium-disk on Azure) unless your workload requires an either extremely high number of IOPS or very low storage latency.
  • In both the OLTP and CPU benchmarks, we saw a per-vCPU performance advantage to running on smaller instances of a particular instance type, regardless of CPU platform, cloud, or instance type.
  • while you may save a bit of money by choosing instance types with a lower vCPU to RAM ratio, you will likely see more consistent performance from instance types with more available memory, at least for the purposes of running CockroachDB.
  • How do their oligopic pricing policies compare? Cloud Pricing Comparison: AWS vs. Azure vs. Google Cloud Platform in 2022. Of course, pricing is too complex to compare directly, but you know bandwidth will always break the bank.
  • And with Announcing Pub/Sub: Programmable MQTT-based Messaging, Workers, KV, Durable Objects, R2, CloudFlare is making a case to become a simpler AWS for developers. As is Fastly: Fastly + Fanout: why real-time messaging and edge computing are an amazing combination.
  • @NayutSitachitt: 1/ Understanding AWS - A perspective from a manager who has $AMZN as one of the largest positions in the portfolio for over 7 years. In the thread, I will also discuss $MSFT's Azure, $GOOG's GCP. Let's dive in.
  • @mike_julian: A friend asked me recently what I think the big things are that make GCP, Azure, and AWS different from each other. I think the biggest thing is how they view customers. GCP, organizationally, seems to have disdain for customers. What little I've seen from Azure toward customers can best be characterized as "inaction". AWS has the most interesting--and rare!--approach to customer feedback of all.
  • Daniel Compton: The main alignment here is Google adding data transfer charges to match AWS and breaking architectural promises they’ve made to their customers in the process. This is an incredibly short-sighted move and will damage customer trust in Google Cloud for many years.
• Or should you just go bare metal?
  • @martin_casado: Three growth startups ($20m ARR+) in the last week have mentioned to me that part of their operations are on Equinix bare metal with K8. Perhaps a sampling anomaly. But I do hear this more and more.
  • @jrhunt: Riot's and Netflix's talks from re:invent last year highlight some of the key developments in maintaining local infra + cloud infra. Reality is that local infra gives the latency wins. The cost effective performance wins are good on paper but the real win is on latency.
  • @manishrjain: So much money saved. 2 month rental of an AWS server = outright purchase of an equivalent server. Hardware has advanced enough to not break as much, and be swapped easily. And k8s has leveled the playing ground for dealing with fault tolerance. Makes sense to skip public cloud…at Dgraph, we were able to set up the entire "first" rack, full of servers, with storage and redundancy, with k8s -- with 2 SREs working part time over 2 months. Second time would have been much faster. The website, the staging, the QA, and some prod were all running there.
  • @alex_saroyan: One of our customers got 40 racks across 5 locations. Only on egress traffic they save 90% compared to their early days in the public cloud.
  • @subhashchy: Our core platform runs on baremetal servers and a lot of edge locations are powered by equnix metal. The only drawback of BM was very slow speed of delivery and no on demand scaling
  • @chirayupatel: Bunch of alternatives here. We run server clusters on OVH with SSD only storage and containers. An order of magnitude cheaper than AWS + no more VM's, gp1, gp2, IOPs etc.
  • @colemanscott1: Heard several enterprises going multi-cloud are using collo providers as centers of gravity to host operational capabilities of their choosing vs. a tangle of native services across clouds. Combo of cost, capability, and end-run on tech debt. Also a small sample.
  • @GirlwDaMostCake: Executives that think everything needs to run in the cloud, regardless of cost. We run the equivalent of 300 p2 larges steady state every day. Plus we do our model training in OCI and some on prem as well.
• One question about Apple’s AR/VR ambitions is do they have the technical chops to pull off a global network of one billion users using VR? Can they do the backend? Here’s how Meta is planning to handle their cloud gaming infrastructure Under the hood: Meta’s cloud gaming infrastructure:
  • we rely on edge computing, where we bring the cloud gaming infrastructure closer to players by deploying in edges that are in metropolitan areas close to large populations.
  • we partnered with NVIDIA to build a hosting environment on top of NVIDIA Ampere architecture-based GPUs
  • we use Twine, our cluster management system, on top of our edge computing operating system. We build orchestration services to manage the streaming signals and use Twine to coordinate the game servers on edge.
  • We built and used container technologies for both Windows and Android games.
  • we landed on WebRTC with Secure Real-Time Transport Protocol (SRTP) technology as our solution to streaming user inputs and video/audio frames for games.
  • Whenever a player clicked to perform an action in a game (e.g., to make their character jump), we captured the click event and sent it to the server, where the game received the event. The game then rendered a frame that contained the result of that action (i.e., the character jumping). We captured that rendered frame, copied it out, and encoded it using a video encoder. The frame was then packetized so it could fit into a User Datagram Protocol (UDP) packet and sent through the network to the player. Finally, on the player’s side, there was a jitter buffer to help smooth play as the packets came in. We decoded the packets into frames, then rendered the frame for the player. Our current model significantly improves on this and improves latency using GPU encoding.
  • On the networking side, having edge computing sites close to the player also helps reduce video and audio latency — even more significantly than the savings from moving the entire streaming pipeline to GPUs.
  • Over on the player side, we also now use hardware decoding to reduce the decode time.
  • Meanwhile, Microsoft says 10 million users have streamed games on Xbox Cloud Gaming.
  • What is Edge Compute?:
  • loudmax: Stadia's problem was never technical, it was entirely due to bad management. The fact that they got games to play reasonably well at reasonably high resolutions is an impressive achievement, and it set the foundation to build a dominant gaming platform appealing to everyone. Then Stadia's management figured out a way to emphasize all of Stadia's weaknesses and play down its advantages and price it in such a way to appeal to nobody.
• Serverless architecture for optimizing Amazon Connect call-recording archival costs. Long term retention requirements cost a lot in storage. But you don’t probably have to keep data stored in its original high fidelity encoding. Recode and downsample it to save big bucks.
• Information from I/O 2022 is now available.
• Meta on Transparent memory offloading: more memory at a fraction of the cost and power:
  • Transparent memory offloading (TMO) saves 20 percent to 32 percent of memory per server across millions of servers in our data center fleet
  • But alternative technologies such as NVMe-connected solid state drives (SSDs) offer higher capacity than DRAM at a fraction of the cost and power. Transparently offloading colder memory to such cheaper memory technologies via kernel or hypervisor techniques offers a promising approach to curb the appetite for DRAM.
  • LinkedIn is doing something similar. Scaling storage in the datacenter with RoCE: wel discuss how we solved this challenge by providing a flexible, scalable, performant and reliable block storage service using Remote Direct Memory Access (RDMA).
• If it doesn’t go to zero it’s not serverless. Period.
  • Serverless Aurora v2 still isn’t serverless.
  • @jeremy_daly: There are two major problems with Serverless Aurora v2 that fundamentally miss the mark of true #serverless(ness). One is the inability to auto-pause (as Mike mentions), and the other is the missing Data API. These might make it unusable (for now). ☹️ Some thoughts… 🧵
  • But not all agree: Data Dog reports on The State of Serverless. In any case serverless is still growing.
  • Let’s Architect! Serverless architecture on AWS
  • Build a Twitter Leaderboard app with Redis and AWS Lambda (part 1)
  • Dynamic Kubernetes Cluster Scaling at Airbnb: An important part of running Airbnb’s infrastructure is ensuring our cloud spending automatically scales with demand, both up and down. Our traffic fluctuates heavily every day, and our cloud footprint should scale dynamically to support this. To support this scaling, Airbnb utilizes Kubernetes, an open source container orchestration system. We also utilize OneTouch, a service configuration interface built on top of Kubernetes
  • Paul Johnston: Serverless systems should be (but rarely are) thought of as a large number of decoupled but very small systems, that are completely independent of each other, connected by events.
  • @simonw: "Serverless means per-request billing" is my new favorite definition of that term. I'd previously been using "scale to zero" but that always requires extra explanation. "Per-request billing" contains everything you need to know.
• I think IP is the biggest value creator. Erik Bernhardsson: If you rank every major software innovation (OOP, containers, version control, deep learning, consensus algorithms, asymmetric encryption, whatever...) by the value it created, the relational database would come out at the top and it's not even close
• 6 months of effort went into one article and it takes a mere 45 minutes to read. It's the asymmetry of creation that burns people out. Scaling containers on AWS in 2022.
  • Fargate is now faster than EC2. I am seeing a massive migration to ECS on Fargate — it's so much easier!
  • ECS on Fargate improved so much and is the perfect example for why offloading engineering effort to AWS is a good idea
  • App Runner is on the way to becoming a fantastic service
  • Up to a point, EKS on Fargate is faster than EKS on EC2
  • EKS on EC2 scales faster when using karpenter rather than cluster-autoscaler, even in the worst possible scenario
  • EKS on EC2 is a tiny bit faster when using IPv6
  • Lambda with increased limits scales ridiculously fast. With a legitimate workload and increased limits, as we just saw, we are now living in a world where we can instantly get 150 TB of RAM and 90 000 vCPUs for our apps
  • What I am seeing most often with my customers, is the largest part of the worker plane using EKS on EC2 Spot which is good enough for most applications.
  • bleuarff: at work we recently experienced traffic spikes like 35-fold in the span of 30 seconds for a few minutes, then faded regularly for a few hours. Our scaling was too slow to be of any use for such pattern. There are legitimate needs for things that scale as fast as possible.
• It's a databaseapalusa.
  • Redis vs. KeyDB vs. Dragonfly vs. Skytable. 1. Redis: 112,100 / 99,892 2. KeyDB: 288,931 / 282,997. 3. Dragonfly: 408,322 / 392,446. 4. Skytable: 619,992 / 676,091.
  • Dragonfly: a modern in-memory datastore, fully compatible with Redis and Memcached APIs. Dragonfly implements novel algorithms and data structures on top of a multi-threaded, shared-nothing architecture.
  • Even more on Distributed Databases Compared.
  • CloudFlare released D1, their SQL database based on SQLlite.
  • Anna: A Key-Value Store For Any Scale: shows how a key-value storage (KVS) system can be architected to scale across many orders of magnitude. (Spoiler Anna can give you only upto causal consistency, but cannot provide strong consistency at key level, and nothing stronger than read-committed at the multi-key level.)
  • fabian2k: I still don't see how [PlanetScale's] pricing model is usable in a database with a query planner. The user is not in control over the query plan, the database is. This is a recipe for disaster, and I'd never feel comfortable with this. Even with a good understanding of how the database works there is pretty much no way to ensure that the database doesn't do some surprising full table scans.
  • tigerbeetle: a financial accounting database designed for mission critical safety and performance to power the future of financial services.
  • Warp: Lightweight Multi-Key Transactions for Key-Value Stores: a simple yet powerful idea to provide efficient multi-key transactions with ACID semantics on top of a sharded NoSQL data store.
  • PranaDB: a distributed streaming database, designed from the outset to be horizontally scalable.
  • FrostDB: an embeddable columnar database written in Go.
  • FoundationDB: A Distributed Unbundled Transactional Key Value Store (Sigmod 2021)
• This is a deep cut. Graviton 3: First Impressions:
  • The design seems to be very narrowly targeted at maximizing compute density in the cloud. To that end, AWS has chosen very conservative core clocks. At 2.6 GHz, Graviton 3 clocks a mere 100 MHz over its predecessor, Graviton 2…Such a core running at 2.6 GHz on 5 nm should absolutely sip power. That in turn lets AWS pack three of these chips into a single node, increasing compute density. The final result is a chip that lets AWS sell each Graviton 3 core at a lower price, while still delivering a significant performance boost over their previous Graviton 2 chip.
  • James Hamilton also has some thoughts: Graviton3 has the performance needed for modern compute intensive applications such as gaming, video encoding, machine learning, and high-performance computing while also delivering the power-performance and price-performance customers seek for less demanding, general-purpose workloads. Formula 1 ran their Computational Fluid Dynamics workloads on C7g and found them to deliver 40% better performance than C6gn. Honeycomb.io, an early adopter of Graviton, saw a 35% performance improvement and a 30% reduction in latency compared with C6g for their telemetry ingestion workload.
• Good to see language flame wars are still a thing, oddly though this one doesn’t make much of a case. Building a Cloud Database from Scratch: Why We Moved from C++ to Rust:
  • As an early-stage database startup, we entirely deleted our C++ codebase after 7 months of development and rewrote everything from scratch in Rust programming language.
  • Rust guarantees zero-cost abstraction so that we won't have performance capped and doesn't require runtime garbage collection so that we can have the latency spike potentially caused by memory management under our control
  • Rust also has Fragmented async ecosystem; Cumbersome error handling; Insufficient support of AsyncIterator
  • Rust: A Critical Retrospective: This next point dovetails into why Rust is not yet suitable for a fully auditable kernel: the language isn’t finished.
  • Lies we tell ourselves to keep using Golang: Or you can be horrified, as you realize that those complex problems only exist because Go is being used. Those complex problems would not exist in other languages, not even in C
• Small Datum with a great explanation of how RocksDB efficiently manages access to its block cache with mutexes. RocksDB internals: LRU.
• Good advice from Stack Overflow on how to limit damage from aggressive DDOS attacks targeting expensive SQL queries. Stack under attack: what we learned about handling DDoS attacks:
  • Insist that every API call be authenticated. This will help identify malicious users.
  • Minimize the amount of data a single API call can return.
  • Rate-limit all API calls.
  • Filter malicious traffic before it hits your application.
  • Block weird URLs.
  • Block malicious IPs even if legitimate traffic can originate from them.
  • Automate your blocklist.
  • Tar pitting is a great way to slow down botnets and mitigate volume based attacks.
• It turns out when you turn a row oriented OLTP database into a columnar OLAP database, optimizations are possible. AlloyDB for PostgreSQL under the hood: Columnar engine.
  • But that's not the interesting part. Google has long advocated the use of ML over configuration to improve software. And here it is: The embedded machine learning in AlloyDB makes accessing this technology easier than ever. AlloyDB automatically organizes your data between row-based and columnar formats, choosing the right columns and tables based on learning your workload, and converting them to columnar format automatically. The query planner smartly chooses between columnar and row-based execution plans, so transactional performance is maintained. This allows AlloyDB to deliver excellent performance for a wide range of queries, with minimal management overhead.
  • @ahachete: I'm impressed with @googlecloud's AlloyDB, both in terms of tech and pricing. An example of the latter, for 4vCPU + 32GB RAM, hourly, in EU West: * AlloyDB: $0.685 * Aurora Postgres on-demand (db.r6g.xlarge): $0.519 + I/Os * Aurora RI, no upfront: $0.340 + I/Os
  • Machine Learning Operations (MLOps): Overview, Definition, and Architecture
  • Designing Arithmetic Circuits with Deep Reinforcement Learning
• Like Facebook before it with PHP, Shopify is taking their favorite language Ruby, and dragging it into the future. Amazing what you can do with money and talent. Shopify Invests in Research for Ruby at Scale.
• You migh imagine Pinterest has a lot of read traffic and so they would be experts at caching. You would be right. Improving Distributed Caching Performance and Efficiency at Pinterest.
  • Pinterest leverages both synthetic load generation and production shadow traffic to evaluate the impact of tuning and optimizations.
  • While memcached can be arbitrarily horizontally scaled in and out to address a particular cluster’s bottleneck, vertically scaling individual hardware dimensions allows for greater cost efficiency for specific workloads.
  • using extstore to expand storage capacity beyond DRAM into a local NVMe flash disk tier increases per-instance storage efficiency by up to several orders of magnitude, and it reduces the associated cluster cost footprint proportionally.
  • Approximately half of all caching workloads at Pinterest are compute-bound (i.e. purely request throughput-bound). Successful optimizations in compute efficiency translate into the ability to downsize clusters without compromising serving capacity.
  • At Pinterest, most workloads (including the distributed cache fleet) run on dedicated EC2 virtual machines.
  • running memcached under a real-time scheduling policy, SCHED_FIFO, with a high priority — instructing the kernel to, effectively, allow memcached to monopolize the CPU by preempting. This one-line change, after rollout to all compute-bound clusters, drove client-side P99 latency down by anywhere between 10% and 40%, in addition to eliminating spurious spikes in P99 and P999 latency across the board.
  • TCP Fast Open (TFO) is a mechanism for reducing the latency overhead of establishing a TCP connection by optimizing away one RTT in an otherwise costly TCP 3WHS (3-way handshake), while also allowing eager transmission of early data during the handshake itself.
• Does Laravel scale?:
  • When asking that question you might begin by defining what Laravel is: a PHP web application framework with expressive, elegant syntax.
  • @usefathom runs Laravel on infrastructure that can handle over 157 billion requests per month.
  • It starts off saying you don't really need to scale and your benchmarks are crap, so nothing yet.
  • Then the rest is good yet standard advice on how to scale a backend, because, you know, frameworks aren't the bottleneck.
• Scale out vs scale up? Some things that used to be only solved with scale up can now be solved by scaling out. Because sometimes scaling doesn't matter, but sometimes it does. The Unique Architecture behind Amazon Games’ Seamless MMO New World.
  • New World became the fifth highest-played game by concurrent users in the history of the video game distribution service Steam, with more than 900,000 concurrent users in its first weekend.
  • Amazon DynamoDB handles about 800,000 writes every 30 seconds to store game state.
  • While a traditional MMO simulates, redraws, analyzes, and processes user states and actions around five times per second, New World does these 30 times a second. "In the old world, if things needed to scale up, you bought bigger hardware"
  • As a player moves through the game, state information is handed from hub instance to hub instance, spreading the compute load over virtual servers hosted globally in multiple AWS Regions, physical locations that contain clusters of data centers.
  • the Amazon EC2 instances for a single world in Aeternum can simulate more than 7,000 artificial intelligence entities and hundreds of thousands of objects for 2,500 players. Each server set often processes millions of state changes per second,
  • To power the known core game play activities, New World uses AWS Lambda, a serverless, event-driven compute service that lets developers run code for the backend service
• Uber's Better Load Balancing: Real-Time Dynamic Subsetting:
  • Uber infrastructure operates in multiple zones in multiple regions, across both on-prem data centers and public cloud providers. Most of the control plane components are zonal, and each typically consists of several sub-components. At a very high level, Uber’s service mesh architecture is centered around an on-host L7 reverse proxy
  • Subsetting in the context of load balancing means dividing the set of potential backend tasks into overlapping "subsets," so that while all backends receive traffic in aggregate, each proxy performs the load balancing on a limited set of tasks.
  • The basic idea is: if an on-host proxy knows how much QPS a callee service is receiving, it could derive the ratio of load it is contributing to the overall traffic. With this information, an on-host proxy could decide its subsetting size dynamically based on ratio ( i.e., it should expand its subsetting size if it starts to make more requests to a destination service).
  • Dynamic Subsetting system where aperture parameters of the mesh scale automatically and dynamically with the changing traffic. The system has been running in production for nearly 2 years for millions of containers.
  • The primary success we’d like to call out is the reduction in maintenance cost: since the rollout (~18-12 months ago) we have had 0 cases of service owners complaining about subsetting. With 8 larger services manually tuned, they reported 15-30% p99 CPU utilization reduction.
• And they all fall down...Cascading failures in large-scale distributed systems:
  • A cascading failure is a failure that increases in size over time due to a positive feedback loop. The typical behavior is initially triggered by a single node or subsystem failing. This spreads the load across fewer nodes of the remaining system, which in turn increases the likelihood of further system failures resulting in a vicious circle or snowball effect
  • The most common cause is server overload or a consequence of it. When that happens, the drop in system performance often affects other areas of the system.
  • Resources of a server are limited. If the load increases above a certain threshold, the server’s performance metrics, such as latency or error rates, deteriorate. This translates into a higher risk of a crash
  • When resource exhaustion causes a server to crash, traffic spreads to other servers, increasing the likelihood that those will crash as well. A cycle of crashing servers establishes.
  • Now, to address the cascading scenario, various measures can be taken. The first and most intuitive option is to increase resources.
  • Try to avoid health check failures/deaths
  • Restart your servers in case of thread-blocking requests or deadlocks.
  • Drop traffic significantly and then slowly increase the load so that the servers can gradually recover.
  • Switch to a degraded mode by dropping certain types of traffic.
  • Eliminate batch/bad traffic to reduce system load
  • Move from an orchestration pattern to a choreography pattern by applying a publish–subscribe pattern design,

Soft Stuff:

• Litestream: a standalone streaming replication tool for SQLite.


• SplinterDB: a key-value store designed for high performance on fast storage devices.


• Zig (Zig at Uber): a general-purpose programming language and toolchain for maintaining robust, optimal and reusable software.


• Inform: a programming language for creating interactive fiction, using natural language syntax.


• Trino: a fast distributed SQL query engine for big data analytics.


• kic-reference-architectures: This repository has the basics for a common way to deploy and manage modern apps.


• ATL: This is a working space for sketching a tensor language.

Pub Stuff:

• Convergence of multiple synthetic paradigms in a universally programmable chemical synthesis machine: For the last decade people have been telling me the digitization of chemistry was impossible. They said it was too complicated, sensitive and expensive. Our chemical programming language runs on cheap hardware and can make any molecule or material. #chemputation. Here’s an example.


• Automated performance prediction of microservice applications using simulation: Our approach accurately predicts the deployment performance of large-scale microservice applications in various configurations from a single execution trace. This provides valuable insights on the performance of an application prior to its deployment on a real platform.


• Zero-Overhead Metaprogramming: Our evaluation with self-optimizing interpreters shows that unrestricted metaobject protocols can be realized for the first time without runtime overhead, and that this optimization is applicable for just-in-time compilation of interpreters based on meta-tracing as well as partial evaluation.


• Powering a microprocessor by photosynthesis: Here, we describe a bio-photovoltaic energy harvester system using photosynthetic microorganisms on an aluminium anode that can power an Arm Cortex M0+, a microprocessor widely used in Internet of Things applications. The proposed energy harvester has operated the Arm Cortex M0+ for over six months in a domestic environment under ambient light.


• Google SRE Books: Building Secure & Reliable Systems; The Site Reliability Workbook; Site Reliability Engineering


• Succinct Data Structures and Delta Encoding for Modern Databases: . TerminusDB makes a radical departure from historical architectures to address these changes. First, we implement a graph database with a strong schema so as to retain both simplicity and generality of design. Second, we implement this graph using succinct immutable data structures which enable more sparing use of main memory resources.


• Monarch: Google’s Planet-Scale In-Memory Time Series Database: Google’s system for storing time-series metricsTime-series data describes data points that occur over time. Storing and using this type of information is an active area of research and industry development.. Time series metricsMetrics are one of the main components in an observability stack (among tracing, events, and logging). The paper Towards Observability Data Management at Scale has more information on the other components of the stack. are used for alerting, graphing performance, and ad-hoc diagnosis of problems in production.


• A Probabilistic Chemical Programmable Computer (video): We present a hybrid digitally programmable chemical array as a probabilistic computational machine that uses chemical oscillators partitioned in interconnected cells as a computational substrate. This hybrid architecture performs efficient computation by distributing between chemical and digital domains together with error correction.


• Algorithms for Modern Hardware: This is an upcoming high performance computing book titled "Algorithms for Modern Hardware" by Sergey Slotin. Its intended audience is everyone from performance engineers and practical algorithm researchers to undergraduate computer science students who have just finished an advanced algorithms course and want to learn more practical ways to speed up a program than by going from O(n \log n)O(nlogn) to O(n \log \log n)O(nloglogn).


• Decoupled Transactions: Low Tail Latency Online Transactions Atop Jittery Servers (a href=https://www.youtube.com/watch?v=72UZ8DxPa8o">video): We present a thought experiment for a decoupled transactions database system that avoids cascading slowdown when a subset of its servers are sick but not necessarily dead. The goal is to provide low tail latency online transactions atop servers and networks that may sometimes go slow


• Designing Event-Driven Systems: Author Ben Stopford explains how service-based architectures and stream processing tools such as Apache Kafka® can help you build business-critical systems.