How to Improve MySQL AWS Performance 2X Over Amazon RDS at The Same Cost

AWS is the #1 cloud provider for open-source database hosting, and the go-to cloud for MySQL deployments. As organizations continue to migrate to the cloud, it’s important to get in front of performance issues, such as high latency, low throughput, and replication lag with higher distances between your users and cloud infrastructure. While many AWS users default to their managed database solution, Amazon RDS, there are alternatives available that can improve your MySQL performance on AWS through advanced customization options and unlimited EC2 instance type support. ScaleGrid offers a compelling alternative to hosting MySQL on AWS that offers better performance, more control, and no cloud vendor lock-in and the same price as Amazon RDS. In this post, we compare the performance of MySQL Amazon RDS vs. MySQL Hosting at ScaleGrid on AWS High Performance instances.

TLDR

Click to read more ...

2019 PostgreSQL Trends Report: Private vs. Public Cloud, Migrations, Database Combinations & Top Reasons Used

PostgreSQL is an open source object-relational database system that has soared in popularity over the past 30 years from its active, loyal, and growing community. For the 2nd year in a row, PostgreSQL has kept the title of #1 fastest growing database in the world according to the DBMS of the Year report by the experts at DB-Engines. So what makes PostgreSQL so special, and how is it being used today? We found the answers at the Postgres Conference in March where we surveyed PostgreSQL users, contributors, and SQL and NoSQL database administrators alike. In this free PostgreSQL Trends Report, we break down PostgreSQL hosting use across public cloud vs. private cloud vs. hybrid cloud, most popular cloud providers, migration trends, database combinations with Postgres, and why PostgreSQL is preferred over popular RDBMS alternatives.

Private Cloud vs. Public Cloud vs. Hybrid Cloud

Click to read more ...

How is software developed at Amazon?

How is software developed at Amazon? Get a couple of prime pizzas delivered and watch this excellent interview with Ken Exner, GM of AWS Developer Tools. It's notable Ken is from the tools group because progress in an industry is almost always made possible by the development of better tools.

The key themes from the talk: decomposition, automation, and organize around the customer.

The key idea:

Scaling is by mitosis. Teams split apart into smaller teams that completely own a service. EC2 started as one two pizza team. 

This quote nicely embodies all three of the themes and is the key reason AWS keeps on winning the public cloud. Bottom up, Amazon adaptively grows their entire organization in response to customer inputs. 

If you want an example of how a complex AWS feature was developed from customer input then take a listen to Heavy Networking 433: An Insider’s Guide To AWS Transit Gateways. The AWS Transit Gateway was developed because customers asked for it...and AWS listened.

AWS is eating the world because customers keep on asking for a bigger menu.

And here's a short gloss of the talk...

Click to read more ...

10 Things You Should Know About AWS

Authored by Chris Fregly:  Former Netflix Streaming Platform Engineer, AWS Certified Solution Architect and Purveyor of fluxcapacitor.com.

Ahead of the upcoming 2nd annual re:Invent conference, inspired by Simone Brunozzi’s recent presentation at an AWS Meetup in San Francisco, and collected from a few of my recent Fluxcapacitor.com consulting engagements, I’ve compiled a list of 10 useful time and clock-tick saving tips about AWS.

1) Query AWS resource metadata

 

Can’t remember the EBS-Optimized IO throughput of your c1.xlarge cluster?  How about the size limit of an S3 object on a single PUT?  awsnow.info is the answer to all of your AWS-resource metadata questions.  Interested in integrating awsnow.info with your application?  You’re in luck.  There’s now a REST API, as well!

Note:  These are default soft limits and will vary by account.

2) Tame your S3 buckets

 

Delete an entire S3 bucket with a single CLI command:  

aws s3 rb s3://<bucket-name> --force

Recursively copy a local directory to S3:

aws s3 cp <local-dir-name> s3://<bucket-name> --region <region-name> --recursive

3) Understand AWS cross-region dependencies

Click to read more ...

Save up to 30% by Selecting Better Performing Amazon Instances

If you like the idea of exploiting market inconsistencies to lower your costs then you will love this paper and video from the Hot Cloud '12 conference: Exploiting Hardware Heterogeneity within the Same Instance Type of Amazon EC2.

The conclusion is interesting and is a source of good guidance:

• Amazon EC2 uses diversified hardware to host the same type of instance.  
• The hardware diversity results in performance variation.
• In general, the variation between the fast instances and slow  instances can reach 40%. In some applications, the variation can even approach up to 60%.  
• By selecting fast instances within the same instance type,  Amazon EC2 users can acquire up to 30% of cost saving, if the fast instances have a relatively low probability.

The abstract:

Click to read more ...

Strategy: Kill Off Multi-tenant Instances with High CPU Stolen Time

Are all instances created equal? Perhaps because under multi-tenancy multiple virtual machines run on the same physical host, not all applications will run equally well on every instance. In that case it makes sense to measure and move to a better performing instance. 

That's the interesting idea from @botchagalupe:

Imagine something like a "performance monkey" where an infrastructure is so bound that it can kill lower performing instances automatically.

@adrianco says Netflix has throught of doing the same: 

We've looked at killing off multi-tenant instances that have high CPU stolen time...

Click to read more ...

Scale Indefinitely on S3 With These Secrets of the S3 Masters

In a great article, Amazon S3 Performance Tips & Tricks, Doug Grismore, Director of Storage Operations for AWS, has outed the secret arcana normally reserved for Premium Developer Support customers on how to really use S3:

• Size matters. Workloads with less than 50-100 total requests per second don't require any special effort. Customers that routinely perform thousands of requests per second need a plan.

Click to read more ...

Strategy: Guaranteed Availability Requires Reserving Instances in Specific Zones

When EC2 first started the mental model was of a magic Pez dispenser supplying an infinite stream of instances in any desired flavor. If you needed an instance, because of a either a failure or traffic spike, it would be there. As amazing as EC2 is, this model turned out to be optimistic.  

From a thread on the Amazon discussion forum we learn any dispenser has limits:

As Availability Zones grow over time, our ability to continue to expand them can become constrained. In these scenarios, we will prevent customers from launching in the constrained zone if they do not yet have existing resources in that zone. We also might remove the constrained zone entirely from the list of options for new customers. This means that occasionally, different customers will see a different number of Availability Zones in a particular Region. Both approaches aim to help customers avoid accidentally starting to build up their infrastructure in an Availability Zone where they might have less ability to expand.

The solution: if you need guaranteed resources in different zones, purchase Reserved Instances. This will assure capacity when needed. There's no way to know if the instance types you are interested in are available in an availability zone, so reserving instances is the only solution. 

Click to read more ...

Should any cloud be considered one availability zone? The Amazon experience says yes.

Amazon has a very will written account of their 8/8/2011 downtime: Summary of the Amazon EC2, Amazon EBS, and Amazon RDS Service Event in the EU West Region. Power failed, backup generators failed to kick in, there weren't enough resources for EBS volumes to recover, API servers where overwhelmed, a DNS failure caused failovers to alternate availability zones to fail, a double fault occurred as the power event interrupted the repair of a different bug. All kind of typical stuff that just seems to happen.

Considering the previous outage, the big question for programmers is: what does this mean? What does it mean for how systems should be structured? Have we learned something that can't be unlearned?

Click to read more ...

The Updated Big List of Articles on the Amazon Outage

Since The Big List Of Articles On The Amazon Outage was published we've a had few updates that people might not have seen. Amazon of course released their Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region. Netlix shared their Lessons Learned from the AWS Outage as did Heroku (How Heroku Survived the Amazon Outage), Smug Mug (How SmugMug survived the Amazonpocalypse), and SimpleGeo (How SimpleGeo Stayed Up During the AWS Downtime). 

The curious thing from my perspective is the general lack of response to Amazon's explanation. I expected more discussion. There's been almost none that I've seen. My guess is very few people understand what Amazon was talking about enough to comment whereas almost everyone feels qualified to talk about the event itself.

Lesson for crisis handlers: deep dive post-mortems that are timely, long, honestish, and highly technical are the most effective means of staunching the downward spiral of media attention. 

Amazon's Explanation of What Happened

Click to read more ...