2019 PostgreSQL Trends Report: Private vs. Public Cloud, Migrations, Database Combinations & Top Reasons Used

PostgreSQL is an open source object-relational database system that has soared in popularity over the past 30 years from its active, loyal, and growing community. For the 2nd year in a row, PostgreSQL has kept the title of #1 fastest growing database in the world according to the DBMS of the Year report by the experts at DB-Engines. So what makes PostgreSQL so special, and how is it being used today? We found the answers at the Postgres Conference in March where we surveyed PostgreSQL users, contributors, and SQL and NoSQL database administrators alike. In this free PostgreSQL Trends Report, we break down PostgreSQL hosting use across public cloud vs. private cloud vs. hybrid cloud, most popular cloud providers, migration trends, database combinations with Postgres, and why PostgreSQL is preferred over popular RDBMS alternatives.

Private Cloud vs. Public Cloud vs. Hybrid Cloud

Click to read more ...

Intro to Redis Cluster Sharding – Advantages, Limitations, Deploying & Client Connections

Redis Cluster is the native sharding implementation available within Redis that allows you to automatically distribute your data across multiple nodes without having to rely on external tools and utilities. At ScaleGrid, we recently added support for Redis Clusters on our platform through our fully managed Redis hosting plans. In this post, we’re going to introduce you to the advanced Redis Cluster sharding opportunities, discuss its advantages and limitations, when you should deploy, and how to connect to your Redis Cluster.

Sharding with Redis Cluster

Click to read more ...

SharePoint VPS solution

Microsoft SharePoint is an ideal solution for companies who have multiple offices and staff members who are on the move. Using SharePoint, documents and other materials can be easily shared with both colleagues and managers. Other features include advanced document management, which allows users to virtually check out a document, modify it or just read it, then check in the document again. This allows managers/company owners to see exactly when their staff members are working and just what they are doing. When combined with a highly customizable workflow management system and group calendars, SharePoint can improve the way in which your company functions and operates.

However, many organizations are observed to be failing with SharePoint implementation. So with this article, we are trying to make it simpler for organizations in-house IT administrators to help implement SharePoint over a virtual server environment.

Here we are going to see following key points:

Click to read more ...

AIDA: Badoo’s journey into Continuous Integration

It’s hardly news to anyone that product development and testing involve a lot of boring routine work, which can lead to human error. To avoid complications stemming from this, we use AIDA.

AIDA (Automated Interactive Deploy Assistant) is a utility that automatically performs many of the processes in Git, TeamCity and JIRA.

In this post, we focus on how through using AIDA we were able to automate multiple workflows and create a scheme of continuous integration.

We’ll start by looking at the version control system (VCS) we use here at Badoo, specifically how Git is used to automate creation of release branches, and their subsequent merging. Then we’ll discuss AIDA’s major contribution to both JIRA integration and TeamCity.

Git flow

Click to read more ...

Cloud Deployment: It’s All About Cloud Automation

Many organizations are facing the challenge of migrating their IT to the cloud. But not many know how to actually approach this undertaking. In my previous post I took a hands-on example of SpringSource’s PetClinic reference application with a Tomcat web server front-end and a Cassandra NoSQL database backend and showed how to onboard it to the cloud in a manageable fashion. But many think this methodology is only good for modern applications that were built with some dynamic/cloud orientation in mind. For example, how different would the cloud on-boarding process be if I modify my PetClinic example to use a MySQL relational database instead of the modern Cassandra NoSQL clustered database? In this blog post I intend to show that cloud on-boarding of brownfield applications doesn’t have to be a huge monolithic migration project with high risk. Cloud on-boarding can take the pragmatic approach and can be performed in a gradual process that both mitigates the risk and enables you to enjoy the immediate benefits of automation and easier management of your application’s operational lifecycle even before moving to the cloud...

Click to read more ...

Code deployment tools

G'day, I'm building an application to manage WordPress PHP code on many servers. Our application will push down code updates to each server, as well as performing backups and testing. I'm considering different methods of pushing updated code onto the individual servers. I'm considering something like Capistrano (I've no experience in Ruby though). I've also considered using subversion and then remotely calling svn commands via SSH. Are there any other tools specifically for this purpose? The servers will have persistent data (the WordPress databases) so I don't want to re-image them every update. Plus, they will each have a different set of plugins / themes, so building many images would be too complex. If there are any papers on code deployment, or other recommended reading, please point the links my way. Likewise, if anyone has any suggestions, or would like more details, just let me know. Cheers - Callum.

Click to read more ...

Paper: On Designing and Deploying Internet-Scale Services

Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. I know people crave nitty-gritty details, but this isn't a how to configure a web server article. It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. The author and his team of contributors obviously have a lot of in the trenches experience. Many non-obvious topics are covered. And there's a lot to learn from.

The paper has too many details to cover here, but the big sections are:

Recommendations

Automatic Management and Provisioning

Dependency Management

Release Cycle and Testing

Operations and Capacity Planning

Graceful Degradation and Admission Control

Customer Self-Provisioning and Self-Help

Customer and Press Communication Plan

In the recommendations we see some of our old favorites:

Expect failure and design for failure.

Implement redundancy and fault recovery.

Depend upon a commodity hardware slice.

Keep things simple and robust.

Automate everything.

Personally, I'm still trying to figure out how to make something simple.

Next are some good thoughts on how to design operations friendly software:

Quick service health check. This is the services version of a build verification test.

Develop in the full environment.

Zero trust of underlying components.

Do not build the same functionality in multiple components.

One pod or cluster should not affect another pod or cluster.

Allow (rare) emergency human intervention.

Enforce admission control at all levels.

Partition services.

Understand the network design.

Analyze throughput and latency.

Treat operations utilities as part of the service.

Understand access patterns.

Version everything.

Keep the unit/functional tests from the last release.

Avoid single points of failure.

Support single-version software. Have all your customers run the same version.

Implement multi-tenancy. Apparently a lot of software requires cloning hardware installations to support multiple customers. Don't do that. Have your software work for multiple customers all on the same hardware.

And the paper continues along the same lines in each section. Good detailed advice on lots of different topics.

You'll undoubtedly agree with some of the advice and disagree with some. Greg wants faster release cycles, thinks having server affinity for some things is OK, and thinks the advice on allowing humans to throttle load won't work in a crisis. Perfectly valid points, but what's fun is to consider them. Some companies, for example, have a dead-man's switch that must be thrown before one master can failover to another in a multi-datacenter situation. Is that wrong or right? Only the shadow knows.

The advice to "document all conceivable component failures and modes and combinations" sounds good but is truly difficult to do in practice. I went through this process once on a telco project and it took months just to cover all the failure scenarios on a few cards. But the spirit is right I think.

My favorite part of the whole paper is:
We have long believed that 80% of operations issues originate in design and development, so this section
on overall service design is the largest and most important. When systems fail, there is a natural tendency
to look first to operations since that is where the problem actually took place. Most operations issues,
however, either have their genesis in design and development are best solved there.

Understand this and I think much of the rest follows naturally.

Product: FAI - Fully Automatic Installation

From their website: FAI is an automated installation tool to install or deploy Debian GNU/Linux and other distributions on a bunch of different hosts or a Cluster. It's more flexible than other tools like kickstart for Red Hat, autoyast and alice for SuSE or Jumpstart for SUN Solaris. FAI can also be used for configuration management of a running system. You can take one or more virgin PCs, turn on the power and after a few minutes Linux is installed, configured and running on all your machines, without any interaction necessary. FAI it's a scalable method for installing and updating all your computers unattended with little effort involved. It's a centralized management system for your Linux deployment. FAI's target group are system administrators who have to install Linux onto one or even hundreds of computers. It's not only a tool for doing a Cluster installation but a general purpose installation tool. It can be used for installing a Beowulf cluster, a rendering farm, a web server farm, or a linux laboratory or a classroom. Even installing a HPC cluster or a GRID and fabric management can be realized by FAI. Large-scale linux networks with different hardware and different installation requirements are easy to establish using FAI and its class concept. Remote OS installations, Linux rollout, mass unattended installation and automated server provisioning are other topics for FAI. The city of Munich is using the combination of GOsa and FAI for their Limux project. Features: * Boot methods: network boot (PXE), CD-ROM, USB stick, floppy disk * Installs Debian, Ubuntu, SuSe, CentOS, Mandriva, Solaris, ... * Centralized installation and configuration management * Installs XEN domains and Vserver

Related Articles

FAI wiki

FAI the fully automated installation framework for linux from Debian Administration

Fully Automatic Installation (FAI) Video Interviewby http://www.perspektive89.com/

Rolling Out Unattended Debian Installations by Carla Schroder from LinuxPlanet

A talk on fai and debian

Click to read more ...

Product: Capistrano - Automate Remote Tasks Via SSH

Update: Deployment with Capistrano  by Charles Max Wood.  Nice simple step-by-step for using Capistrano for deployment.

From their website:
Simply put, Capistrano is a tool for automating tasks on one or more remote servers. It executes commands in parallel on all targeted machines, and provides a mechanism for rolling back changes across multiple machines. It is ideal for anyone doing any kind of system administration, either professionally or incidentally.

* Great for automating tasks via SSH on remote servers, like software installation, application deployment, configuration management, ad hoc server monitoring, and more.
* Ideal for system administrators, whether professional or incidental.
* Easy to customize. Its configuration files use the Ruby programming language syntax, but you don't need to know Ruby to do most things with Capistrano.
* Easy to extend. Capistrano is written in the Ruby programming language, and may be extended easily by writing additional Ruby modules.

One of the original use cases for Capistrano was for deploying web applications. (This is still by far its most popular use case.) In order to make deploying these applications reliable, Capistrano needed to ensure that if something went wrong during the deployment, changes made to that point on the other servers could be rolled back, leaving each server in its original state.

If you ever need similar functionality in your own recipes, you can introduce a transaction:

task :deploy do
transaction do
update_code
symlink
end
end

task :update_code do
on_rollback { run "rm -rf #{release_path}" }
source.checkout(release_path)
end

task :symlink do
on_rollback { run "rm #{current_path}; ln -s #{previous_release} #{current_path}" }
run "rm #{current_path}; ln -s #{release_path} #{current_path}"
end

The first task, “deploy” wraps a transaction around its invocations of “update_code” and “symlink”. If an error happens within that transaction, the “on_rollback” handlers that have been declared are all executed, in reverse order.

This does mean that transactions aren’t magical. They don’t really automatically track and revert your changes. You need to do that yourself, and register on_rollback handlers appropriately, that take the necessary steps to undo the changes that the task has made. Still, even as lo-fi as Capistrano transactions are, they can be quite powerful when used properly.


From the Ruby on Rail manual:

Ultimately, Capistrano is a utility that can execute commands in parallel on multiple servers. It allows you to define tasks, which can include commands that are executed on the servers. You can also define roles for your servers, and then specify that certain tasks apply only to certain roles.

Capistrano is very configurable. The default configuration includes a set of basic tasks applicable to web deployment. (More on these tasks will be said later.)

Capistrano can do just about anything you can write shell script for. You just run those snippets of shell script on remote servers, possibly interacting with them based on their output. You can also upload files, and Capistrano includes some basic templating to allow you to dynamically create and deploy things like maintenance screens, configuration files, shell scripts, and more.

Related Articles

Friends for Sale uses Capistrano for deployment.