advertise
Friday
Sep222017

Stuff The Internet Says On Scalability For September 22nd, 2017

Hey, it's HighScalability time: 

 

Ever feel like howling at the universe? (Greg Rakozy)

 

If you like this sort of Stuff then please support me on Patreon.

 

  • 10 billion: API calls made every second in Google datacenters; $767,758,000,000: collected by Apple on iPhones sold to the end of June; 20: watts of power consumed by human brain, autonomous vehicles peak at 3000 watts; 59%: drop in leads using AMP; 27%: success rate of AIs guessing passwords; 2.8 kilometers: distance devices running on almost zero power can xmit using backscatter; 96: age at which Lotfi Zadeh, inventor of Fuzzy Logic, passed away; 35%: store time series data in a RDBMS; $1.1 billion: Google's spend on self-driving tech;  $5.1 billion: Slack valuation; 15%: bugs reduced by strong typing; ~1 ft: new smartphone GPS accuracy; 

  • Quotable Quotes:
    • Napoleon: [Sir Hudson Lowe] was a man wanting in education and judgment. He was a stupid man, he knew nothing at all of the world, and like all men who knew nothing of the world, he was suspicious and jealous.
    • Rich Werner: Data center operations, to me, is 362 days of boredom. And then you get these hurricanes coming through, and it’s three days of pulling your hair out.
    • @pacoid: @kenneth0stanley #TheAIConf "We're not interested in complexity for its own sake" -- ref. operational closure in second-order cybernetics
    • Animats: Much as I like Rust, I have to agree. When you have to get it done, use Go. When you want to explore advanced experimental programming constructs, use Rust. The Go guys knew when to stop. Arguably they stopped too early, before generics or parameterized types, and "interface[]" is used too much. But that doesn't seem to be a big overhead item. Rust has the complexity of C++ plus additional complexity from the functional community. Plus lots of fancy template cruft.
    • @mims: People who say data is the new oil are wrong. Non-volatile flash memory is the new oil.
    • @cmeik: As former member of a NoSQL startup, "Safety, reliability [as well as pay up front, save later] doesn't sell" sure sounds familiar.
    • @PaulDJohnston: ... simply because we've had 20 years of "servers" and 10 years of "instances" and now "containers"... they are all the same...
    • Venkatraman Ramakrishnan~ Inventions in one discipline can build on—and spur—basic research in many others, often unwittingly. It’s a virtuous cycle, and scientists take joy in exploiting all of it. Scientists are very promiscuous and the good ones are the most promiscuous.
    • @rightfold: Most of programmers learn early to avoid premature optimization. Next step: teach people about premature distributed computing.
    • @indievc: “Not heroine…Not cocaine….But Venture Capital is the drug flowing through the veins of most Silicon Valley startups”
    • @skamille: Editing is a different profession than writing, but code review and programming are both performed by the same people
    • XNormal: Mainframes had the reputation of being very expensive. But this is misleading. In terms of cost per processing task they were much more efficient than mini and microcomputers.
    • @swardley: "Culture eats strategy for breakfast" is code for "I don't know what the heck I'm talking about but this meme sounds smart"
    • James Glanz: Yet another data center, west of Houston, was so well prepared for the storm — with backup generators, bunks and showers — that employees’ displaced family members took up residence and United States marshals used it as a headquarters until the weather passed.
    • @pacoid: Neuroevolution talk @kenneth0stanley #theaiconf -- "exact gradient is not always the best move"; evolution uses fitness fn, not objective fn
    • @GossiTheDog: Holy crap. CCleaner trojan 1st stage payload is on 700k PCs, with these orgs targeted for 2nd stage (successfully) 
    • Scott Aaronson: In the meantime, the survival of the human race might hinge on people’s ability to understand much smaller numbers than 10^122: for example, a billion, a trillion, and other numbers that characterize the exponential growth of our civilization and the limits that we’re now running up against.
    • Timothy Morgan: Compute and networking could hit the Moore’s Law wall at about the same time, and that is precisely what we expect.
    • ralmidani: As I've said before, universal web components are a pipe dream. Developers disagree on even the most trivial things, like the best way to parse a query string. What makes anyone think those disagreements will magically disappear once web components become a standard?
    • Mallory Locklear: [Virus] Jumps between species have driven most major evolutionary innovations in the viruses. Meanwhile, co-divergence has been less common than was assumed and has mostly caused incremental changes.
    • @SwiftOnSecurity: Linux is like if the creator of git wrote an operating system.
    • A troll ate the rest of the quotes. Luckily, if luck it be, a copy was made and you can read all of them by clicking through to the full post.

    Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

    Click to read more ...

Monday
Sep182017

Evolution of data structures in Yandex.Metrica

Yandex.Metrica is the world's second largest web analytics system. Metrica takes in a stream of data representing events that took place on sites or on apps. Our task is to process this data and present it in an analyzable form.


Processing the data in itself is not a problem. The real difficulty lies in trying to determine what form the processed results should be saved in so that they are easy to work with. During the development process, we had to completely change our approach to data storage organization several times. We started with MyISAM tables, then used LSM-trees and eventually came up with column-oriented database, ClickHouse. In this article I'll explain what led us to settle on this last option.

Yandex.Metrica was launched in 2008 and has now been running for more than nine years. Every time we changed our approach to data storage in the past it was because a particular solution proved inefficient: either there was insufficient performance reserve, or the solution was unreliable, or it used too many computational resources, or it just did not allow us to implement what we needed to.

The old Yandex.Metrica for websites has more than 40 "fixed" report types (for example, the visitor geography report), several in-page analytics tools (like click maps), Webvisor (which lets you study individual user actions in great detail), as well as the separate report constructor.

With the new Metrica and Appmetrica, you can customize every report instead of dealing with "fixed" types. You can add new dimensions (for example, in a search term report you can break data down further by landing page), segment and compare (between, let's say, traffic sources for all visitors vs. visitors from San Francisco), change your set of metrics, etc. The new system, therefore, demands a completely different approach to data storage than what we used earlier.

MyISAM

Click to read more ...

Friday
Sep152017

Stuff The Internet Says On Scalability For September 15th, 2017

Hey, it's HighScalability time: 

 

Earth received Cassini’s final signal at 7:55am ET. Let's bid a fond farewell. After a 13-year tour of duty, job well done!

 

If you like this sort of Stuff then please support me on Patreon.

 

  • 12.9 million: DynamoDB requests per second on Prime Day; 4 billion: transistors on Apple's A11 Bionic chip; 4x: extreme weather events since 1970; 51: qubit device; 50%: Messenger.com converted to Reason56.6 million: US cord cutters; 5000: bikes abandoned at Burning Man; 500 million: yearly visitors to Apple stores; 30 min: time to send one HD color image from Mars to Earth; 

  • Quoteable Quotes:
    • @randyshoup: Interesting idea of a *Negative* MTTR by @adrianco: notice something is going to fail and proactively fix it before it breaks!
    • @rob_pike: "The Equifax executives who let my data be stolen will probably suffer fewer consequences than I will for an overdue library book." @nytimes
    • @avantgame: on weaponized social media: "We’re in an information war with Russia. It’s time we started acting like it."
    • Jamie Dimon: It's [Bitcoin] worse than tulip bulbs. It won't end well. Someone is going to get killed
    • @manisha72617183: First they tell you that Scrum is not a magic bullet.Then they spend the rest of the time saying how it’s the best thing since sliced bread🙄
    • yogthos: My team has been using Clojure for 7 years now, and we're very happy with it. It's still a pleasure to work with, and the stability of the language has been really welcome.
    • @GossiTheDog: Another way of looking at Equifax is they did an incredible job of keeping infrastructure that size with that much legacy secure for so long
    • API Evangelist: when it comes to the shear volume, and regular drumbeat of serverless stories Microsoft is keeping pace. After watching several months of sustained storytelling, it looks like they could even pass up Amazon in the near future.
    • amelius: Well, I hear a lot of people complaining that the results on DuckDuckGo are still worse than on Google, even though both search-engines produce results within a second. And these are people that really want to quit using Google for privacy reasons. I never hear people complaining that a search is slow. So I do think that search-quality is where the competition is happening.
    • @SwiftOnSecurity: How you think multinational hypercorps get hacked: NSA 0days on the black market How multinational hypercorps get hacked: admin/admin
    • m-masa: Snapchat to me is sharing your shaky drunken escapades at 3AM with your friends to let them know you made it home and survived the night. Instagram seems more like an endless observation of copy-and-paste, superficial things and people and places. It's evolved more into a (usually inaccurate) portrayal of status than anything else.
    • Dmitri Zimine: When Serverless replaces micro-services, it is not going to be free lunch either. We are paying by introducing more complexity, now for the benefit of massive cost savings.
    • Kris De Decker: In London, a solar panel produces 65 times less energy on a heavy overcast day in December at 10 am than on a sunny day in June at noon
    • nostrademons: The real interesting work in search is in ranking functions, and this is where nobody comes close to Google. Some of this, as other commenters note, is because Google has more data than anyone else. Some of it is just because there've been more man-hours poured into it. IMHO, it's pretty doubtful that an open-source project could attract that sort of focused knowledge-work (trust me; it's pretty laborious) when Google will pay half a mil per year for skilled information-retrieval Ph.Ds.
    • rkangel: Up to now this is all classic Eve - betrayal by people you trust. The postscript is less nice though: gigx in a moment of anger asked in in game chat for real life contact details for TheJudge so that he could 'cut off his hands'. This is obviously not OK and CCP banned gigx permanently. This has the side-effect of putting the final nail in the CO2 coffin.
    • Rick Altherr: At one point I did that calculation and I was seeing one hard drive die every five minutes. 
    • EliE: the fundamental reason why ransomware is so successful, and here to stay, is that people simply don’t backup their data.
    • EliE: no matter how many times the bitcoins are moved, ultimately they must be cashed out at exchange points. So we just need to keep tracing movements until we reach a cash-out wallet.
    • @radjanirad: Just a few hours ago, Cassini received the command to turn off the RADAR instrument - for the last time. :( #cassini
    • @postwait: Most monitoring "innovations" have been mostly aesthetic, but their marketing is deafening and drowns out real innovation. #UphillBattle
    • pab: I have two years experience pair programming, and to quote asthasr, I found it an absolute slog.
    • @matthew_d_green: I have an idea. Let's combine all the hard parts of cryptography with all the asshole parts of the finance industry.
    • Pete Saia: It’s important to understand that it isn’t all or nothing. Serverless is in our future, but it isn’t our exclusive future.
    • Errata Security: The 9,000 devices were split almost evenly between Apple and Android. Almost all of the Apple devices randomized their addresses. About a third of the Android devices randomized. (This assumes Android only randomizes the final 3 bytes of the address, and that Apple randomizes all 6 bytes -- my assumption may be wrong).
    • David Rosenthal: Today's eclipse records would be on the Web, not paper or bone. Will astronomers 3200 or even only 580 years from now be able to use them?
    • Peter Zaitsev: To be competitive with non-open-source cloud deployment options, open source databases need to invest in “ease-of-use.” There is no tolerance for complexity in many development teams as we move to “ops-less” deployment models.
    • Jeremy Hsu: the advantage of the flip-flop qubit comes from inducing an electric dipole—separation of positive and negative charges—by pulling the electron a little bit away from the nucleus of the phosphorus atoms (which are themselves embedded in silicon). That electric dipole enables the spin-based silicon qubits to remain entangled together over longer distances and able to influence one another through quantum physics.
    • Cory Doctorow: All these forms of cheating treat the owner of the device as an enemy of the company that made or sold it, to be thwarted, tricked, or forced into con­ducting their affairs in the best interest of the com­pany’s shareholders. To do this, they run programs and processes that attempt to hide themselves and their nature from their owners, and proxies for their owners (like reviewers and researchers).
    • Jonathan Golden: How do you know, though, when to pull resources away from other growth initiatives to address these edge cases? My rule of thumb was when a problem was occurring at least 50 times a day, it was time to solve it more holistically. At a time when we were growing anywhere from 300%–600% per year — and edge cases were growing at least as fast — that’s when the potential explosion of problems proliferated.
    • A Mind at Play: Well, the good of this command is that if you’re in a loop you can have this command in that loop and every time it goes around the loop it will put a pulse in and you will hear a frequency equal to how long it takes to go around that loop. And then you can put another one in some bigger loop and so on. And so you’ll hear all of this coming on and you’ll hear this “boo boo boo boo boo boo,” and his concept was that you would soon learn to listen to that and know whether when it got hung up in a loop or something else or what it was doing all this time, which he’d never been able to tell before.

    Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

    Click to read more ...

Wednesday
Sep132017

Have you noticed there's a lot more collaboration going on these days? Why?

 

Thanks to zero marginal cost digital production methods, we're seeing content markets—for the first time—develop in conditions free from supply and price constraints.

In the process we've learned something: consumers have an unquenchable thirst for new content; content creators are willing to oblige with an equally prodigious stream of new content; platforms that best control access to the customer are the biggest winners; the reward for content creators varies drastically by medium and platform.

For consumers, life is now a streaming fixed priced buffet of unending variety and diversion.

For producers, the changes have been terrifying. Old modes have crumbled, leaving everyone scrambling to figure out what, if anything, comes next.

To adapt, content creators are learning to exploit capture loops, bundling, and collaboration to extract money from a digital economy that has collectively decided it rarely wants to pay artists directly for their content anymore.

The most highly evolved form of digital content platform strategies can be found in the book market. Why? Because Amazon.

Kindle Unlimited is the Clear Platform Winner

Click to read more ...

Tuesday
Sep122017

Sponsored Post: Close.io, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

  • Close.io is a ~25 person fully remote team that is profitable and building a product our customers love! We’re hiring Senior Backend Developers to join our team. Our backend tech stack currently includes Python (Flask, Gunicorn, TaskTiger), Elasticsearch, MongoDB, Postgres, and Redis running in Docker/Kubernetes on AWS. Learn more and apply here!

  • Advertise your job here! 

Fun and Informative Events

  • October 10 Live Webinar. Fast & Frictionless - The Decision Engine for Seamless Digital Business. Join us for a live webinar on Tuesday, October 10 at 11:00 am Pacific Time featuring guest speakers Michele Goetz, Principal Analyst at Forrester Research, and Matthias Baumhof, VP Worldwide Engineering at ThreatMetrix®. A positive customer experience is required for successful enterprise digital transformation. Digital businesses depend on speed and efficiency to drive operational decisions. Making faster, accurate, and real-time customer trust decisions removes friction and delivers superior business outcomes. In this session, you’ll learn: How risk-based authentication leveraging digital identities is key to empowering customer transactions; How real-time customer trust decisions can reduce fraud and improve customer satisfaction; How a high performance Hybrid Memory Architecture (HMA) database helps continuously evaluate across a multitude of factors to drive decisioning at the lowest operational cost. Register now.

  • Advertise your event here!

Cool Products and Services

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • What engineering and IT leaders need to know about data science. As data science becomes more mature within an organization, you may be pulled into leading, enabling, and collaborating with data science teams. While there are similarities between data science and software engineering, well intentioned engineering leaders may make assumptions about data science that lead to avoidable conflict and unproductive workflows. Read the full guide to data science for Engineering and IT leaders.

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections such as MySQL, Salesforce, S3, and many others, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...

Friday
Sep082017

Stuff The Internet Says On Scalability For September 8th, 2017

Hey, it's HighScalability time: 

 

May you live in interesting times. China games swarming drone attacks. Portable EMP anyone? (Tech in Asia)

 

If you like this sort of Stuff then please support me on Patreon.

 

  • 100GB: entire corpus of articles written at the NY Times; 80GB: data for one human genome; 3%: Linux desktop market share; 3.5M: fake Wells Fargo accounts; $18,000: world’s most expensive vacuum; 2000: Netflix recommender taste groups; 27%: year-over year-growth rate of Python on SO; 4M: Time Warner hacked; 143M: Equifax hacked; $800M: ICO funding in Q2; $257M: Filecoin ICO; 

  • Quotable Quotes:
    • Brendan Gregg: jobs are also migrating from both Solaris and Linux to cloud jobs instead, specifically AWS. The market for OS and kernel development roles is actually shrinking a little. The OS is becoming a forgotten cog in a much larger cloud-based system. The job growth is in distributed systems, cloud SRE, data science, cloud network engineering, traffic and chaos engineering, container scheduling, and other new roles. 
    • @DrQz: The Performance Paradox: The better u do ur job, the more invisible u become. https://goo.gl/1aTRvw  🐵 🙄
    • @kennwhite: $100,000+ spent on thousands of [Facebook] ads, tied to on 470 fake accounts, all linked to a propaganda troll farm with ~600 staff in St. Petersburg.Kenn White added,
    • marssaxman: 10.6.8 was the best Mac OS ever. Since then I've felt increasingly uncomfortable with the heavy-handed, paternalistic direction Apple has been taking their OS; it just doesn't feel like home anymore. I believe in personal computers as tools of personal empowerment; it's my machine, not Apple's. I really resent being told what I can and can't do with it, and I neither need nor want an itunes account.
    • @jemangs: "Amazon spent $16.1 billion on R&D last year, a figure that should strike fear into its competitors" - Recode
    • @xaprb: OH: "I have some really junior staff and they were bitching about having to wait 5 minutes for an EC2 instance. GET OFFA MY LAWN."
    • Nora Jones: Chaos doesn't cause problems, it reveals them. 
    • Littlefinger: chaos is a latter.
    • Ken Stanley: sometimes in order to make discovery possible, you have to stop having an objective
    • GeneticGenesis: Whenever a "config change" (Note: this includes adding or removing targets to a target group, EG Autoscaling) happens on an ALB, the ALB drops all active connections, and re-establishes them at once, at high load, this obviously causes significant load spikes on any underlying service.
    • Stefano Bernardi: Call me old fashioned, but wanting to raise half a billion dollars for a pre-product endeavor is absolutely f*cking insane.
    • revscat: This was my first experience with modern JavaScript frameworks and TypeScript. I wanted to do it right, so worked closely with team members who were more versed in this stuff, and followed the various recommended best practices. By the time all was said and done the PR for this thing had 27 files in it. For a modal. This seems ludicrous to me. 
    • Tony Seba: [on disruption] Technology convergence is when several technologies and business model innovations converge at one point in time to enable functionality at a certain cost.
    • Tony Seba: Business model innovation is every bit as disruptive as technological innovation.
    • Tony Seba: By 2030 95% of all passenger miles are going to be autonomous electronic vehicles. There goes the internal combustion engine industry. There goes the individual ownership of cars. We will have cars as a service just as we have movies as a service. 
    • @Noahpinion: 15 years ago, the internet was an escape from the real world. Now, the real world is an escape from the internet.
    • @jbeda: Hot take: [new AWS LB] similar to but more limited to GCP L3 LB. AWS LB is zonal and looks to do NAT. GCP L3 LB supports anycast across regions and DSR.
    • @GossiTheDog: Tip - if you want in to a bunch of factory networks, don't target the companies - target their ICS suppliers. Find names via case studies.
    • @GossiTheDog: Because vendors usually self managed black box VPN appliances at sites, the actual company doesn't see logs = doesn't know they are owned.
    • @ftrain: Giant company: We are geniuses worth a trillion dollars. Me: I would like to log into two different accounts at once. Company: Holy shit.
    • @GossiTheDog: Equifax's infrastructure is a weird mix of IBM WebSphere, Apache Struts, Java.. it's like stepping back in time a decade.
    • catvalente: “The Internet used to be full of original content & lively debate” is the new “in my day we walked to school in the snow uphill both ways”
    • Lots of quotes make a web page too heavy for the computer to lift, so I saved them for the full page, which is made of unobtanium. Please click through to read the rest...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Sep012017

Stuff The Internet Says On Scalability For September 1st, 2017

Hey, it's HighScalability time: 

Obviously, cloud native is simplicity itself. (Cloud Native Landscape Project)

If you like this sort of Stuff then please support me on Patreon.

 

  • $10: price for IPv4 address (and falling); 10-15%: better IPv6 network performance; 711M: Record Onliner Spambot Dump; 41.4-tesla: strongest resistive magnet; 85%: cell towers offline due to Hurricane Harvey; 1M: Facebook accounts turned off every single day; 700K: Lyft drivers; $160B: Crypto Market Cap; $1.5M: bounty for iPhone jail break; 2.93M: pirated views of Mayweather-McGregor fight; 465k: people need to update pacemaker firmware; 70,065,920: views of Taylor Swift in 2 days; 2 trillion: kafka messages per day at LinkedIn; $35 billion: saved when planes fly themselves; 990: bird species in North America; 108B: number of people who have ever lived; 500M: DuckDuckGo anonymous searches in one month; 

  • Quotable Quotes:
    • @asymco: The iPhone is not only the best selling product of all time but it may be the most used. 60 trillion minutes a year.
    • @Nick_Craver: Stack Overflow Questions (last 30 days): 1,280,911,699 Hits 26.92 ms Avg Render Time 11,040,996,940 SQL Queries 4,489,374,544 Redis Hits
    • @GBrayUT: Rough CDN numbers [for Stack Overflow] from bosun ~2000 hits/sec to sstatic at peak. Comes out to another 885 million hits/week, or ~3.8 billion CDN Hits/30 days
    • @msuriar: Why is loadshedding such a thing at Google? Because in 2009 we had 3 public visible Gmail outages. #SREcon
    • @phoronix: The @AMDRyzen #Threadripper 1950X managing a #Linux kernel build in about 36 seconds, not bad!
    • @cmeik: When I told Barbara Liskov that some applications didn't need serializability she gasped.
    • Dan Lyons: A century ago, factory workers went on strike to demand better conditions. Today, startup "hustlers" celebrate their own exploitation. 
    • @nigewillson: Forget 20+ Billion #IOT devices by 2020, imagine the Disruption from 1 billion drones by 2030 !  http://ab.co/2wnLXfi  #drones @thomasfrey
    • @markcallaghan: Peak write rate might be 20% better in recent MyRocks. FlushWAL might be why
    • nnfy: Threadripper is not just about speed. It is about PCIE lanes. Intel has been taking advantage of its monopoly for decades, artificially limiting PCIE lane count, among other things. This may be the start of a new era in computing, allowing GPGPUs to be used more readily, not to mention specific neural network and deep learning applications.
    • api: Most ICO projects are complete and utter vapor consisting of nothing more than a stock bootstrap web site and a "white paper" that could have been generated by a Markov chain model using crypto and decentralized systems buzzwords. A few have a bit of code online that doesn't work. Of those that have shipped most of them are unusable, and of those that are usable I am only aware of maybe one or two that are at all interesting. Of those none are very compelling and none are things I couldn't do without. There are literally no hits in this space. It's all junk, and most of it is outright scams. I'd be surprised if a single current generation coin ever actually delivers real ROI.
    • @sarahmei: 1. Software is not about rigor. It's about mapping what people want onto what machines can do. It's messy, it's chaotic, it's wonderful.
    • @QuinnyPig: #SREcon "How did you get management support for this?" @ING: "We put them into the on-call rotation." Oh my god it's brilliant.
    • @PaulDJohnston: You can never know all of AWS. So don't try.
    • retox: Only on HN would targeting 100 million devices for a preview be seen as small fry.
    • Ann Mutschler: the retina’s 1 million or so ganglion cells are composed of about 20 distinct types. Each plays a slightly different role in transmitting the perception of shape, color, depth, motion and other visual features to the brain.
    • Stack Overflow: Python and R are associated with a country’s income. Python is visited about twice as often in high-income countries as in the rest of the world, and R about three times as much.
    • Itai Gurari: Human: What is the purpose of existence? Machine: to find out what happens when we get to the planet earth
    • @CodeWisdom: "A language that doesn’t affect the way you think about programming is not worth knowing." - Alan Perlis
    • Team Shellphish: Kubernetes (kubernetes.io) - The distribution of docker containers across our cluster, and the load-balancing and failover of resources, was handled by kubernetes. In our final setup, the Mechanical Phish was so resilient that it could probably continue to function in some form even if the rack was hit with a shotgun blast.
    • Jeff Dorsch: What is clear, given the focus of presentations, is that the bleeding edge of computing has shifted significantly in the past 12 months. While performance and power are still critical elements, the primary concern is no longer how to cram more transistors on a piece of silicon or how to put multiple chips together. It is now much more geared toward new architectures that can bridge the digital and physical worlds.
    • Karen Stabiner: “At least once a week, I get something about a new start-up with a new gadget that’s going to make our lives so much better,” said Sisha Ortuzar, Mr. Colicchio’s partner at ‘Wichcraft. “They’re distractions. We just want to get back to making sandwiches.”
    • Lila MacLellan: And everything changed. Pre-Keith, Dwyer explains, “it was very clear no one took us seriously and everybody thought we were just idiots.” When “Keith” contacted collaborators, Gazin says, “they’d be like ‘Okay, bro, yeah, let’s brainstorm!’”
    • userbinator: Looks like those Seagates are still disappointingly failure-prone, while HGST remains the most reliable.
    • ISTC: network bandwidth already available today can be on par with the main memory bandwidth.
    • Michael P Frank: This idea of reversible computing goes to the very heart of thermo­dynamics and information theory, and indeed it is the only possible way within the laws of physics that we might be able to keep improving the cost and energy efficiency of general-purpose computing far into the future.
    • jcampbell1: Grammarly absolutely crushed it with Adwords on the content network. There was so much cheap inventory they were able to buy such as on dictionary sites.
    • There is a bit of a Y Combinator attitude against buying advertising to build a startup. If it ain't viral, then the product isn't good enough. 
    • jedberg: [Amazon] absolutely will not look at your data under any circumstance, even if you specifically ask them to. Multiple times I was trying to troubleshoot an issue and asked them to "just log into the database" and they said they have very strict policies against that. There is no way they have any of Target's data.
    • @benthompson: The fundamental failing in discipline after discipline is the overvaluing of what can be measured, and the dismissal of what can not.
    • dmitrygr: I read the paper. Still not sure how this [Mercury Protocol: Communication Platform Built on the Ethereum Blockchain] solves anything besides transferring wealth from the gullible to the authors:
    • @Nick_Craver: Code review just now: a errant boolean check in our Master.cshtml can cost upwards of 100 million hits to redis a day. Crazy to think about.
    • moxious: First, the idea of running a distributed ledger to keep track of burger points. This is the tech equivalent of using a bazooka to address an ant problem.
      • mcgarnagle: Say what you will, at least the whoppercoins are backed by something that has survived the test of time, the whopper.
    • contingencies: For financial transaction services I can recommend sharding first by customer, then by ledger. As a result, instead of enforcing double-entry book-keeping standards within a single database, do it at an application-specific middleware server layer to enforce only the guarantees you need.
    • @stevesi: AR by itself isn't a platform but it is precisely the kind of platform feature that makes cross-platform impossible.
    • Camillo Bruni: Changing the property or element type typically causes V8 to create a different HiddenClass which can lead to type pollution which prevents V8 from generating optimal code
    • redwood: I lived in a country where everyone has a power generator in their building. Let's just say the quality of life was significantly lower. This cloud shift is like an unstoppable tidal wave. I'm always surprised when I hear people with your argument [that the cloud is not a utility]
    • Tim Harford: It’s not simply that Blade Runner fumbled its futurism by failing to anticipate the smartphone. That’s a forgivable slip, and Blade Runner is hardly the only film to make it. It’s that, when asked to think about how new inventions might shape the future, our imaginations tend to leap to technologies that are sophisticated beyond comprehension.
    • @krishnan: For Google, Kubernetes is just an enterprise trojan horse to onboard customers to their cloud 
    • @roboYeti: Started playing with KSQL. Joining tables with streams from my Kafka topics -no problem. @jaykreps it's awesome piece of Engineering. Thanks
    • Tom Griffiths: This is a moment where we are starting to recognize that we're going to need to interact with systems that, at least in restricted domains, are going to be smarter than us. Thinking about how to design those interfaces between humans and machines in ways that make it possible for us to interact with those systems in a way that allows us to function effectively is an important research challenge, and a significant social challenge.
    • mrmrcoleman: At a previous company we had a 'farm' of test phones that we used for testing new versions of software. A large number of those test were 'stress testing' which would probably be analogous to the sort of load you would have it they were to be treated as a 'datacenter'. Based on that I'm pretty sure that the biggest issue with managing the cluster would be the extremely high failure rate. That's not to say that this won't work because I haven't done the maths, but it kid a unique challenge.
    • Michael Mullany: What I take away from my analysis of these Hype Cycles is not just how difficult it is to make predictions, and how much wasted effort goes into technologies that doesn't tend to work, but also how exciting and wondrous is the progress that we've made in technology. The labor of the last two decades has given rise to an Age of Wonders

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Aug292017

Sponsored Post: Close.io, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

  • Close.io is a ~25 person fully remote team that is profitable and building a product our customers love! We’re hiring Senior Backend Developers to join our team. Our backend tech stack currently includes Python (Flask, Gunicorn, TaskTiger), Elasticsearch, MongoDB, Postgres, and Redis running in Docker/Kubernetes on AWS. Learn more and apply here!

  • Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Grokking the System Design Interview is an interactive course that explains the Large Scale Distributed System Design problems in detail. System design questions have become a standard part of the software engineering hiring process. This course is created by hiring managers who’ve been working at Google, Facebook, Microsoft, and Amazon, and is a complete guide to master the System Design Interviews.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • What engineering and IT leaders need to know about data science. As data science becomes more mature within an organization, you may be pulled into leading, enabling, and collaborating with data science teams. While there are similarities between data science and software engineering, well intentioned engineering leaders may make assumptions about data science that lead to avoidable conflict and unproductive workflows. Read the full guide to data science for Engineering and IT leaders.

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections and transformations, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...

Friday
Aug252017

Stuff The Internet Says On Scalability For August 25th, 2017

Hey, it's HighScalability time

 

View of the total solar eclipse from a hill top near Madras Oregon, August 21, 2017. As totality approaches, dragons gorge on sun flesh; darkness cleaves the day; a chill chases away the heat; all becomes still. Contact made! Diamonds glitter; beads sparkle; shadow band snakes slither across pale dust; moon shadow races across the valley, devouring all in wonder. Inside a circle of standing stones, obsidian knives slash and stab. Sacrifices offered, dragons take flight. In awe we behold the returning of the light.

 

If you like this sort of Stuff then please support me on Patreon.

 

  • ~5: ethereum transactions per second; 29+M: Snapchat news viewers; 100K: largest Mastodon instance; 2xAlibaba's cloud base growth; 1B: trees planted by a province in Pakistan; 90.07%: automated decoding of honey bee waggle dances; $86.4B: Worldwide Information Security Spending; 1200: db migrations from Mysql to Postgres; $7B: Netflix content spend (most not original); 13%: increased productivity by making vacation mandatory; 75%: US teens use iPhones; 30,000x: energy use for Bitcoin transaction compared to Visa; ~1 trillion: observations processed for Gaia mission; 50%: video North American internet traffic; $300 million: cost of cyberattack on world’s biggest container shipping company; 320 million: Freely Downloadable Pwned Passwords; 1700 B.C: world’s oldest trigonometric table;

  • Quotable Quotes:
    • @matthew_d_green: I miss the days when Bitcoin was a cool technical innovation and not a weird religious movement.
    • Ruth Williams: Their new digital-to-biological converter (DBC) can, upon receipt of a DNA sequence, prepare appropriate oligos, carry out DNA synthesis, and then, as required, convert that DNA into a vaccine, or indeed into any RNA molecule or protein.
    • @trashcanlife: Hello, this is container 100406100098090 in Buffalo, United States. I am 38% full.
    • Zhang & Stutsman: Developing new systems and applications on RAMCloud, we have repeatedly run into the need to push computation into storage servers.
    • @kevinmontrose: Tomorrow the Sun will undergo routine maintenance in US region. Will be unavailable for select customers, others will have degraded service.
    • @bryanrbeal: We're officially in an era where every piece of HARDWARE you buy, is actually a service. There is no hardware any more.
    • @jessfraz: Literally throwing away two trash bags of container startups tee-shirts, sorry I just... there's too many
    • @postwait: "Scale" I don't think that word means what you think it means. Hint: it doesn't mean your arbitrary concept of "big."
    • Rod Squad: My friend, SocialBlade founder, Jason Urgo advised my 10-year-old son on how to start programming. Jason told us how he started dabbling with scripts and programs in kindergarten. He told me about the first game he programmed. He also listed some of the first applications he built. And he explained how he taught himself PHP to build the YouTube data compiler.
    • Kim Beaudin: Why do Java developers wear glasses? Because they can’t C#.
    • Christine Hall: Investing in private data centers isn’t as much of a priority for IT organizations as it was just several years back. That’s a takeaway from IT researcher Computer Economics’ annual IT Spending and Staffing Benchmarks report...According to the report, data centers now have the lowest priority for new spending among a list of five categories. Top priority is given to the development of business applications, a category in which 54 percent of respondents plan increased spending. However, only 9 percent have plans to increase data center spending, which the study attributes to increasing reliance on cloud infrastructure, cloud storage, and SaaS
    • morning paper: The core idea of a CGN is to gather all the information needed for a page load in a place that has a short RTT time, and then transfer it to the client in (ideally) one round trip. At a cost of about $1 per user, the authors show that it can reduce the median page load time across 100 popular web sites by up to 53%.
    • Nick Harley: It’s easy to shrug off problems with a ‘move fast and break things’ mentality. But we build software for our users, and sometimes forget they are real people.
    • alexkcd: Proof of work systems are, at the core, a race towards ever greater energy consumption. They're an environmental disaster waiting to happen. Surprised how little attention this gets. I would argue that the benefit of decentralization is not worth the price.
    • @EricNewcomer: Uber generates $1.75 billion in revenue on a $645 million loss
    • Preethi Kasireddy: In order to scale, the blockchain protocol must figure out a mechanism to limit the number of participating nodes needed to validate each transaction, without losing the network’s trust that each transaction is valid. 
    • HowDoIMathThough: A slide I personally find really interesting from anandtech's hot chips coverage - Intel has packaging technology that should allow multiple dies to be combined with extremely fast links extremely cheaply
    • Tim Bray: It may sound hack­neyed in 2017, but: Me, I be­lieve in pro­gress. I be­lieve in build­ing un­der­stand­ing cu­mu­la­tive­ly and striv­ing al­ways for Truth. Un­for­tu­nate­ly, there are places in the world, some quite near­by, where the en­e­mies of progress are strong. As Joel Mokyr teach­es, progress is not pre­des­tined to win; we have to fight for it and nev­er stop, or we can lose it; it’s hap­pened.
    • @rawkode: So fed up with watching micro-service talks where they say "More services == good" and don't even mention operational concerns or intg tests
    • @pcalcado: A little known fact is that approximately 47% of CPU usage across a typical Kubernetes cluster is invested translating between JSON and YAML
    • two2two: I asked my 16 year old nephew 6 months ago how he accesses the news. His answer: Snapchat. I followed that with anywhere else? His response was nope.
    • lima: Red Hat's OpenShift makes it [deploying applications?] a lot easier by providing all of the infrastructure around it (docker registry, docker build from Git, Ansible integration and so on). Best docs of all open source projects I've seen.
    • drdaeman: I'm really wary about using larger black boxes for critical parts. Just Linux kernel and Docker can bring enough headache, and K8s on top of this looks terrifying. Simplicity has value. GitHub can afford to deal with a lot of complexity, but a tiny startup probably can't. Or am I just unnecessarily scaring myself?
    • Stefan Majewsky: Across terminals, median latencies ranged between 5 and 45 milliseconds, with the 99.9th percentile going as high as 110 ms for some terminals. Now I can see that more than 100 milliseconds is going to be noticeable, but I was certainly left wondering: Can I really perceive a difference between 5 ms latency and 45 ms latency? Turns out that I can.
    • michaelt: Our current design [for van routing software] isn't well suited to adaption to a GPU, because it branches a lot and the memory accesses aren't strided evenly. So we couldn't just plug our current code into a java-to-cuda compiler; we'd need to change the design.
    • @GabeAul: It's official! We did the last migration this weekend, so all new Windows development is on Git! Congrats to the team who worked the w/end!
    • Iddo Bentov: What we’re seeing today is just a harbinger of problems to come should decentralized exchanges sweep over the cryptocurrency landscape. But since the problems that we’ve identified are exacerbated when higher value trades take place, we conjecture that such problems will ultimately limit the popularity of decentralized exchanges.
    • Steve Goldfeder: trackers can link real-world identities to Bitcoin addresses. To be clear, all of this leaked data is sitting in the logs of dozens of tracking companies, and the linkages can be done retroactively using past purchase data.
    • @jasongorman: Go read somebody else's code, *then* write more unit tests to catch any bugs you find. Code review doesn't scale.
    • David Rosenthal: Unless decentralized technologies specifically address the issue of how to avoid increasing returns to scale they will not, of themselves, fix this economic problem. Their increasing returns to scale will drive layering centralized businesses on top of decentralized infrastructure, replicating the problem we face now, just on different infrastructure.
    • @kevin2kelly: Bill Joy: I decided to spend my time trying to create the things we need as opposed to preventing what threatens us.
    • Ethan Zuckerman: decentralization is important because it allows a community to run under its own rules.
    • Tim Harford: to take advantage of electricity, factory owners had to think in a very different way. They could, of course, use an electric motor in the same way as they used steam engines. It would slot right into their old systems...you couldn't get these results simply by ripping out the steam engine and replacing it with an electric motor. You needed to change everything: the architecture and the production process. And because workers had more autonomy and flexibility, you even had to change the way they were recruited, trained and paid. Factory owners hesitated, for understandable reasons.
    • Dan Luu: We’ve looked at a variety of classic branch predictors and very briefly discussed a couple of newer predictors. Some of the classic predictors we discussed are still used in CPUs today, and if this were an hour long talk instead of a half-hour long talk, we could have discussed state-of-the-art predictors. I think that a lot of people have an idea that CPUs are mysterious and hard to understand, but I think that CPUs are actually easier to understand than software. I might be biased because I used to work on CPUs, but I think that this is not a result of my bias but something fundamental.
    • creshal: A current-gen 35W laptop CPU will be some 10 times faster[2] as a RasPi, have much faster storage available (SATA3 or NVMe versus… USB2), much faster I/O (GBit LAN and GBit Wifi versus… USB2), and a lot of other benefits. (Like an integrated screen and battery and keyboard and …) It also won't need external hardware to communicate with other cluster members – that 10-port ethernet switch will need power, too. One RasPi is relatively energy efficient; RasPi clusters… not so much.
    • howinator: we moved to k8s because we have quite a few low-usage services. Before k8s, each one of those services was getting its own EC2 instance. After k8s, we just have one set of machines which all the services use. If one service is getting more traffic, the resources for that service scale up, but we maintain a low baseline resource usage. In short, it's resulted in a measurable drop in our EC2 usage.
    • medius: If you are migrating to AWS RDS, I recommend AWS Data Migration service. I migrated my live database (~50GB) from Mysql to Postgres (both RDS) with zero downtime. I used AWS Schema Conversion Tool for initial PG schema. I customized the generated schema for my specific needs.
    • Jon Claerbout: interactive programs are slavery unless they include the ability to arrive in any previous state by means of a script
    • @bascule: ~5 transactions/second  @VitalikButerinCongrats to ethereum community for 5 days of record-high transaction usage! (410061 ... 443356) https://etherscan.io/chart/tx 
    • Sujith Ravi: Delegating the computation-intensive operations from device to the cloud is not a feasible strategy in many real-world scenarios due to connectivity issues (like when data cannot be sent to the server) or privacy reasons. In scenarios, one solution is to take an existing trained neural network model and then apply compression techniques like quantization to reduce model size. The trainer model can be deployed anywhere a standard neural network is used. The simpler projection network model weights along with transform functions are extracted to create a lightweight model that is pushed to device. This model is used directly on-device at inference time.
    • rothbardrand: BCH [Bitcoin Cash] is a hastily written hack job by a third rate team (I talked to some of them on twitter, they really don't understand a lot of what they are doing)... with a drastic difficulty retargeting algorithm. A bit of a pump combined with hash power manipulations lead to this. This is all show to try and prop up the coin. Both the pump and the "profitability" of mining it. %98 of the blocks of this coin are mined by an unknown entity-- in other words, it's not decentralized. It's trivial for that entity to manipulate the difficulty retargeting mechanism in his favor. Stay away. This is not "bitcoin" in any sense.
    • John Allspaw: It’s only when there isn’t universal agreement about a decision (or even if a decision is necessary) that the how, who, and when a decision gets made becomes important to know. The idea of an architecture review is to expose the problem space and proposed departure ideas to dialogue in a broad enough way that confusion about them can be reduced as much as possible. Less confusion about the topic(s) can help reduce uncertainty and/or anxiety about a solution.
    • HBR: Even though the resilient superhero is usually perceived as better, there is a hidden dark side to it: it comes with the exact same traits that inhibit self-awareness and, in turn, the ability to maintain a realistic self-concept, which is pivotal for developing one’s career potential and leadership talent. 
    • Charles Allen: When a local disk fails, the solution is to kill that instance and let the HA built into your application recover on a new VM. When network disk fails or has a multi-instance brownout, you’re just stuck and have to failover to another failure domain, which is usually in another availability zone or in some cases another region! We know this because this kind of failure has caused production outages for us before in AWS. This trend towards network attached storage is one of the scariest industry trends for big data in the cloud where there will probably be more growing pains before it is resolved.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Aug182017

Stuff The Internet Says On Scalability For August 18th, 2017

    Sorry about missing last week, but my birthday won out over working: 

     

    Ouch! @john_overholt: My actual life is now a science exhibit about the primitive conditions of the past.

    If you like this sort of Stuff then please support me on Patreon.

     

    • 1PB: SSD in 1U chassis; 90%: savings using EC2 Spot for containers; 16: forms of inertia; $2.1B: Alibaba’s profit; 22.6B: app downloads in Q2; 25%: Google generated internet traffic; 20 by 20 micrometers: quantum random number generators; 16: lectures on Convolutional Neural Networks for Visual Recognition; 25,000: digitized gramophone records; 280%: increase in IoT attacks; 6.5%: world's GDP goes to subsidizing fossil fuel; 832 TB: ZFS on Linux;  $250,000: weekly take from breaking slot machines; 30: galatic message exchanges using artificial megastructures in 100,000 years; 

    • Quotable Quotes:
      • @chris__martin: ALIENS: we bring you a gift of reliable computing technol--
        HUMANS: oh no we have that already but JS is easier to hire for
      • @rakyll: "You woman, you like intern." I interned on F-16's flight computer. Even my internship was 100x more legit than any job you will have.
      • @CodeWisdom: "Debugging is like being the detective in a crime movie where you are also the murderer." - Filipe Fortes
      • William Gibson: what I find far more ominous is how seldom, today, we see the phrase “the 22nd century.” Almost never. Compare this with the frequency with which the 21st century was evoked in popular culture during, say, the 1920s.
      • Arador: Amazon EC2, Microsoft Azure and Google Gloud Platform are all seriously screwing their customers over when it comes to bandwidth charges. Every one of the big three has massive buying power yet between them their average bandwidth price is 3.4x higher than colocation facilities.
      • @mattklein123: Good thread: my view: 1) most infra startups will fail. It's an awful business to be in (sorry all, much ❤️).
      • Jean-Louis Gassée: With Services, Apple enjoys the benefits of a virtuous circle: Hardware sales create Services revenue opportunities; Services makes hardware more attractive and “stickier”. Like Apple Stores, Services are part of the ecosystem. Such is the satisfying simplicity and robustness of Apple’s business model.
      • cardine: The price difference between Hetzner and AWS is large enough that it could pay for 4x as much computational power (as much redundancy as you'd ever need), three full time system admins (not that you'd ever need them), and our office lease... with plenty of money left over!
      • Brujo Benavides: Communication is Key: Have you ever watched a movie or a soap opera and thought “If you would’ve just told her that, we would’ve avoided 3 entire episodes, you moron!”. Happens to me all the time. At Inaka we learned that the hard way.
      • @f3ew: Doesn't matter how many layers of stateless services you have in the middle, the interesting ends have state.
      • brianwawok: My cloud cost is less than 5% of my bussiness costs using GCE. Would be foolish to move it to lower my costs 2%.
      • @f3ew: Stateless services are as relevant as routers. Pattern match, compute, push to next layer.
      • Horace Dediu~ when you outsource you're taking knowledge out of your company, which ends up gutting it in terms of the value that is added 
      • Jason Calacanis: Google was the twelfth search engine. Facebook was the tenth social network. iPad was the twentieth tablet. It’s not who gets there first. It’s who gets there first when the market’s ready.
      • puzzle: The B4 paper states multiple times that Google runs links at almost 100% saturation, versus the standard 30-40%. That's accomplished through the use of SDN technology and, even before that, through strict application of QoS.
      • @stu: Serverless has in many ways eclipsed the containers discussion for the hot buzz in the industry
      • @mjpt777: GC is a wonderful thing but I cannot help but feel it leaves the typical developer even less prepared for distributed resource management.
      • joaodlf: Spark works and does a good job, it has many features that I can see us use in the future too. With that said, it's yet another piece of tech that bloats our stack. I would love to reduce our tech debt: We are much more familiar with relational databases like MySQL and Postgres, but we fear they won't answer the analytics problems we have, hence Cassandra and Spark. We use these technologies out of necessity, not love for them.
      • tobyjsullivan: No, dear author. Setting up the AWS billing alarm was the smartest thing you ever did. It probably saved you tens of thousands of dollars (or at least the headache associated with fighting Amazon over the bill). Developers make mistakes. It's part of the job. It's not unusual or bad in any way. A bad developer is one who denies that fact and fails to prepare for it. A great developer is one like the author.
      • Geoff Wozniak: Regardless of whether I find that stored procedures aren't actually that evil or whether I keep using templated SQL, I do know one thing: I won't fall into the "ORMs make it easy" trap.
      • @BenedictEvans: Part of what distinguishes today’s big tech companies is a continual push against complacency. They saw the last 20 years and read the books
      • John Patrick Pullen: In the upcoming fall issue of Porter magazine, the 21-yer-old X-Men: Apocalypse star said, "I auditioned for a project and it was between me and another girl who is a far better actress than I am, far better, but I had the followers, so I got the job," according to The Telegraph. "It’s not right, but it is part of the movie industry now."
      • Rachel Adler: Naturally, faster prints drove up demand for paper, and soon traditional methods of paper production couldn’t keep up. The paper machine, invented in France in 1799 at the Didot family’s paper mill, could make 40 times as much paper per day as the traditional method, which involved pounding rags into pulp by hand using a mortar and pestle.
      • pawelkomarnicki: As a person that can get the product from scratch to production and scale it, I can say I'm a full-stack developer. Can I feel mythical now?
      • Risto: Before integrating any payment flow make sure you understand the whole flow and the different payment states trialing -> active -> unpaid -> cancelled. For Braintree there is a flow chart. For Stripe there is one too. Both payment providers have REST API’s so make sure to play through the payment flows before starting actual coding.
      • Seyi Fabode: I have 3 neighbors in close proximity who also have solar panels on their roofs. And a couple of other neighbors with electric cars. What says we can’t start our own mini-grid system between ourselves?
      • pixl97: Muscles/limbs are only 'vastly' more efficient if you consider they have large numbers of nano scale support systems constantly rebuilding them. Since we don't have nanobots, gears will be better for machines. Also, nature didn't naturally develop a axle.
      • Brave New Greek: I sympathize with the Go team’s desire to keep the overall surface area of the language small and the complexity low, but I have a hard time reconciling this with the existing built-in generics and continued use of interface{} in the standard library.
      • @jeffhollan: Agree to a point. But where does PaaS become “serverless”? Feel should be ‘infinite’ scale of dynamic allocation of resources + micro bill
      • @kcimc: common tempos in 1M songs, 1959-2011: 120 bpm takes over in the late 80s, and bpms at multiples of 10 emerge in the mid 90s
      • How to Map the Circuits That Define Us: If neural circuits can teach one lesson, it is that no network is too small to yield surprises — or to frustrate attempts at comprehension.
      • @orskov: In Q2, The Wall Street Journal had 1,270,000 daily digital-only subscribers, a 34% increase compared to last year
      • Thrust Zone: A panel including tech billionaire Elon Musk is discussing the fact that technology has progressed so much that it may soon destroy us and they have to pass microphones to talk.
      • @damonedwards: When we are all running containers in public clouds, I’m really going to miss datacenter folks one-upping each other on hardware specs.
      • @BenedictEvans: 186 page telecoms report from 1994. 5 pages on ‘videophones’: no mention of internet. 10 pages saying web will lose to VR. Nothing on mobile
      • Thomas Metzinger: The superintelligence concludes that non-existence is in the own best interest of all future self-conscious beings on this planet. Empirically, it knows that naturally evolved biological creatures are unable to realize this fact because of their firmly anchored existence bias. The superintelligence decides to act benevolently.
      • Jeremy Eder: As with all public cloud, you can do whatever you want…for a price.  BurstBalance is the creation of folks who want you to get hooked on great performance (gp2 can run at 3000+ IOPS), but then when you start doing something more than dev/test and run into these weird issues, you’re already hooked and you have no choice but to pay more for a service that is actually usable.
      • Katz and Fan: After all, the important thing for anyone looking to launder money through a casino isn’t to win. It’s to exchange millions of dollars for chips you can swap for cool, untraceable cash at the end of the night.
      • Caitie McCaffrey: Verification in industry generally consists of unit tests, monitoring, and canaries. While this provides some confidence in the system's correctness, it is not sufficient. More exhaustive unit and integration tests should be written. Tools such as random model checkers should be used to test a large subset of the state space. In addition, forcing a system to fail via fault injection should be more widely used. Even simple tests such as running kill −9 on a primary node have found catastrophic bugs.
      • Zupa: FPGAs give you most of the benefits of special-purpose processors, for a fraction of the cost. They are about 10x slower, but that means an FPGA based bitcoin miner is still 100k times faster than a processor based one
      • menge101work: I'm not sure if the implication is that our CPUs will have gate arrays on chip with the generic CPU, that is an interesting idea. But if they are not on chip, the gate array will never be doing anything in a few clock cycles. It'll be more akin to going out to memory, the latency between a real memory load and an L1 or L2 cache hit is huge. (reference)
        Not to say that being able to do complex work on dedicated hardware won't still be fast, but the difference between on-die and off-die is a huge difference in how big of a change this could be.
      • Animats: This article [Why Many Smart Contract Use Cases Are Simply Impossible] outlines the basic problem. If you want smart contracts that do anything off chain, there have to be connections to trusted services that provide information and take actions. If you have trusted services available, you may not need a blockchain.The article points out that you can't construct an ordinary loan on chain, because you have no way to enforce paying it back short of tying up the loaned funds tor the duration of the loan. Useful credit fundamentally requires some way of making debtors pay up later. It's possible to construct various speculative financial products entirely on chain, and that's been done, but it's mostly useful for gambling, broadly defined.
      • curun1r: Your characterization of startup cloud costs is laughably outdated. With credits for startups and the ability to go serverless, I've known startups that didn't pay a dime for hosting their entire first year despite reaching the threshold of hundreds of customers and over $1m ARR. One of my friends actually started doing some ML stuff on AWS because he wanted to use his remaining credits before they expired and his production and staging workloads weren't going to get him there. I'd say it makes no sense to buy your 32gb, 16-core single point of fail, waste $40/mo and half a day setting it up and then have to keep it running yourself when you can easily spin up an API in API Gateway/Lambda that dumps data into dynamo/simpledb and front it with a static site in S3. That setup scales well enough to be mentioned on HN without getting hugged to death and is kept running by someone else. And if it is, literally, free for the first year, how is that not a no brainer?
      • Neil Irwin: In this way of thinking about productivity, inventors and business innovators are always cooking up better ways to do things, but it takes a labor shortage and high wages to coax firms to deploy the investment it takes to actually put those innovations into widespread use. In other words, instead of worrying so much about robots taking away jobs, maybe we should worry more about wages being too low for the robots to even get a chance.

    Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

    Click to read more ...