Product: rPath - Creating and Managing Virtual Appliances

Update: GIGAOM on rPath Burns EC2 Appliances in a Web Portal. rBuilder adds a portal that lets users turn software into virtual appliances. rPath demoed their virtual appliance management system at Monday's AWS Meetup. What they do is help you build a generic virtual machine image deployable on Amazon, VMWare, Xen and other targets. The idea is to build your software application independent of the underlying operating system and deploy it in your own or someone else's datacenter without worrying about all the details. To put their service in context think of rPath as how you build, deploy, and upgrade images and someone like Right Scale has how you can run and managed a cluster of deployed images. To build a Virtual Appliance you pull together all your packages through their web interface or through a Python based "recipe" system, select a VM target, and "cook" it all into a VM image you can immediately deploy and run. To make this magic happen they use the Conary package manager system and they have their own RedHat compatible OS. One of their major features is a very fine grained package management systems which allows them to perform minimal inplace upgrades of deployed images. The downside is you must use their packaging system and their OS for this to work. Any code you want to install must be installable using their packaging system. There's a free community version available on their website for Open Sourcers.. They make their money from people buying a Virtual Appliance of their build and packaging system and deploying it internally. So you can integrate their Virtual Appliance system as part of your build and deployment infrastructure. As part of your nightly build create appliances and have them automatically deployed to your test jigs. Once testing is complete you can deploy into your datacenter. Their smart upgrade features are very nice for a datacenter. Usually package management during upgrades is a complete nightmare. For cloud deployment I think this feature is less useful as I would simply create a new image, fire up a new instance using the new image, and bring down my old images without the cost of a software upgrade. Of course you still have to worry about protocol and data compatibilities. rPath's Virtual Appliance is kind of a hard idea to really understand because it still ahead of curve of what most people are doing. But I think as we move into a world of multiple clouds we must seed with our images, a layer above the clouds is necessary to manage the whole process. rPath is saying we've already built that layer so you don't have to.

Click to read more ...

Product: Collectl - Performance Data Collector

From their website: There are a number of times in which you find yourself needing performance data. These can include benchmarking, monitoring a system's general heath or trying to determine what your system was doing at some time in the past. Sometimes you just want to know what the system is doing right now. Depending on what you're doing, you often end up using different tools, each designed to for that specific situation. Features include:

You are be able to run with non-integral sampling intervals.

Collectl uses very little CPU. In fact it has been measured to use <0.1% when run as a daemon using the default sampling interval of 60 seconds for process and slab data and 10 seconds for everything else.

Brief, verbose, and plot formats are supported.

You can report aggregated performance numbers on many devices such as CPUs, Disks, interconnects such as Infiniband or Quadrics, Networks or even Lustre file systems.

Collectl will align its sampling on integral second boundaries.

Supports process and slab monitoring.

New to the 2.4.0 release is the monitoring of process i/o statistics. Unlike most monitoring tools that either focus on a small set of statistics, format their output in only one way, run either interactively or as a daemon but not both, collectl tries to do it all. You can choose to monitor any of a broad set of subsystems which currently include cpu, disk, inodes, infiniband, lustre, memory, network, nfs, processes, quadrics, slabs, sockets and tcp. The following is an example of simply running the collectl command with no arguments and using its default settings. Below we see what the cpu, network and disk were doing while writing a large file: #<--------CPU--------><-----------Disks-----------><-----------Network----------> #cpu sys inter ctxsw KBRead Reads KBWrit Writes netKBi pkt-in netKBo pkt-out 37 37 382 188 0 0 27144 254 45 68 3 21 25 25 366 180 20 4 31280 296 0 1 0 0 25 25 368 183 0 0 31720 275 2 20 0 1 Output can also be saved in a rolling set of logs for later playback or displayed interactively in a variety of formats. If all that isn't enough there are additional mechanisms for supplying data to external tools via a socket interface or by generating its output as s-expressions, a format of choice for some tools such as supermon. You can even create files in space-separated formats for plotting with external packages like the one below which was done with gnuplot using 1 second samples.

Click to read more ...

Product: ISPMan Centralized ISP Management System

From FRESH Ports and their website: ISPman is an ISP management software written in perl, using an LDAP backend to manage virtual hosts for an ISP. It can be used to manage, DNS, virtual hosts for apache config, postfix configuration, cyrus mail boxes, proftpd etc. ISPMan was written as a management tool for the network at 4unet where between 30 to 50 domains are hosted and the number is crazily growing. Managing these domains and their users was a little time consuming, and needed an Administrator who knows linux and these daemons fluently. Now the help-desk can easily manage the domains and users. LDAP data can be easily replicated site wide, and mail box server can be scaled from 1 to n as required. An LDAP entry called maildrop tells the SMTP server (postfix) where to deliver the mail. The SMTP servers can be loadbalanced with one of many load balancing techniques. The program is written with scalability and High availability in mind. This may not be the right software for you if you want to run a small ISP on a single box or if you want to use this software as an LDAP editor or a DNS management software by itself. ISPMan is written mostly in Perl and is based on four major components. All these components are based on open standards and are easily customizable.

LDAP-directory works as a central registry of information about users, hosts, dns, processes etc. All information related to resources is kept in this directory. The LDAP directory can be replicated to multiple machines to balance the load.

Ispman-webinterface is an intuitive Iinterface to manage informations about your ISP infrastructure. This interface allows you to edit your LDAP registry to change different informations about your resources such as adding a new domain, deleting a user etc. The interface can run on http or https and is only available after successful authentification as an ISPMan admin. Access control to this interface can also be limited to designated IP addresses either via Apache access control functions or via ISPMan ACL.

Ispman-agent is a component of ISPMan that runs on hosts taking part in the ISP, these agents read the LDAP directory for processes assigned to them and take appropriate actions Example : create directory for new domains, create mailbox for users, etc. These agents are a very important part of the system and are should be run continuously. The agents are run via a fault taulerant services manager called « daemontools » that makes sure that the agents recovers immediately in case of any failure.

ISPman-customer-control-panel is an interface targeted towards customers (domain owners). Using this interface the domain owners can manage their own dns, webserver settings, users, mailing lists, access control etc.

Click to read more ...

Product: Hyperic

From Wikipedia: Hyperic HQ is a popular open source IT Operations computer system and network monitoring application software. It auto-discovers all system resources and their metrics, including hardware, operating systems, virtualization, databases, middleware, applications, and services. It watches hosts and services that you specify, alerting you when things go bad and again when they get better. It also provides historical charting and event correlation for faster problem identification. The Hyperic HQ server is a distributed J2EE application that runs on top of the open source JBoss Application Server. It is written in Java and portable C code and runs on Linux, Windows, Solaris, HP-UX and Mac OS X. Hyperic HQ Portal is a Java and AJAX User Interface that includes: * Inventory/Application Model & host hierarchy * Monitoring of network services (SMTP, POP3, HTTP, NNTP, ICMP, SNMP) * Monitoring of host resources (processor load, disk usage, system logs) * Remote monitoring supported through SSH or SSL encrypted tunnels. * Continuous Auto-Discovery of system resources including hardware, software and services * Track log & configuration data * Remote resource control for corrective actions such as starting and stopping services, vacuum database table, or snapshotting a VM * Ability to define event handlers to be run during service or host events for proactive problem resolution * Problem Resource Identification & Root Cause Analysis * Event Correlation * Alerting when service or host problems occur or get resolved via email, pager, Text messaging, RSS * Security/Access Control * Simple plug-in design that allows users to easily develop their own service checks depending on needs, by using the tools of choice (XML, J2EE, Bash, C++, Perl, Ruby, Python, PHP, C#, etc.) I met Javier Soltero, the CEO of Hyperic at the Velocity Web Performance and Operations dinner. I hit him with my best stuff and he didn't flinch a bit. Javier showed a deep understanding of the issues, a real passion for his product and the space, and the knowing good humor of someone who has been through a few wars and learned a little something along the way. I don' know if that translates to an excellent product, but it would at least make me take a look.

Click to read more ...

Product: collectd

From http://directory.fsf.org/project/collectd/ : 'collectd' is a small daemon which collects system information every 10 seconds and writes the results in an RRD-file. The statistics gathered include: CPU and memory usage, system load, network latency (ping), network interface traffic, and system temperatures (using lm-sensors), and disk usage. 'collectd' is not a script; it is written in C for performance and portability. It stays in the memory so there is no need to start up a heavy interpreter every time new values should be logged. From the collectd website: Collectd gathers information about the system it is running on and stores this information. The information can then be used to do find current performance bottlenecks (i. e. performance analysis) and predict future system load (i. e. capacity planning). Or if you just want pretty graphs of your private server and are fed up with some homegrown solution you're at the right place, too ;). While collectd can do a lot for you and your administrative needs, there are limits to what it does: * It does not generate graphs. It can write to RRD-files, but it cannot generate graphs from these files. There's a tiny sample script included in contrib/, though. Also you can have a look at drraw for a generic solution to generate graphs from RRD-files. * It does not do monitoring. The data is collected and stored, but not interpreted and acted upon. There's a plugin for Nagios, so it can use the values collected by collectd, though. It's reportedly a reliable product that doesn't cause a lot load on your system. This enables you to collect data at a faster rate so you can detect problems earlier.

Click to read more ...

Report from OpenSocial Meetup at Google

Update: Facebook pulls a Microsoft and embraces and extends by opening their platform to other social sites like Bebo. Very smart and unexpected. More info at Facebook to let other sites access platform code. This month's regular Facebook Meetup was held at Google and the topic of the day was OpenSocial. For those of you with real lives, OpenSocial "provides a common set of APIs for social applications across multiple websites." Over 200 excited people, hoping to do very exciting things, and dreaming of making an exciting pile of money, watched an OpenSocial presentation put on by a couple of appropriately knowledgeable evangelists. I could feel my social graph being more successfully monetized with each passing minute. Normally the meetings are much smaller, but Google puts on a very nice spread, so I think people may have showed up to dine :-) Or they could have showed up to learn why and how they should code to the new uber social API. By the looks of the full plates and the sounds of energetic chatter, it was likely a bit of both. The crowd seemed skeptical, yet interested. The Facebook world is somewhat self satisfied and that comfy world was being disturbed. It might get ugly I thought, but unfortunately it stayed quite civil and informative. With my bread I had hoped for a bit of circus. My take on OpenSocial: code social application once, run anywhere. Code your social app using Google's gadget model and the social API and it will run on any conforming social network container. It's kind of like a concurrency model based on mobile threads instead of the more traditional message passing model. So your friend's profile app will work just as will on Ning as Orkut. Interestingly, there's a layer of indirection the social network container has to locally interpret what things like friends are. So your friends in SalesForce could mean people you've email once and friends in Ning could mean people you've marked as friends. There's a fairly minimal API of verbs and nouns at this point, but that will undoubtedly grow. They are taking a "do the simplest thing" approach. Or they could have simply needed to get something out to compete with Facebook. Important features like a security model, authentication model, sharing model, and advanced data types are TBD. Lots of tricky things still have to be specified. How do you establish identity across services, who can share what information, how do apps deal with different terms of services, and how they deal with different social network models? OpenSocial is a group of companies so you hear a lot of things like "we'll have to meet and decide that. Joe has a lot of good ideas on how that might be done." The same sort of stuff you hear with all the complex Java standards that everyone hates. Maybe some group will Spring into action and fix some of the problems that develop. What I don't quite understand is how social networks will distinguish themselves from each other with a common API? Using the standard your app will run anywhere so why should I choose a particular social graph provider? So services will have to differentiate by adding nonstandard features which leads to a horrible complex mess of a system. They were already talking about using reflection so you could discover what capabilities a container had. Oh boy. Sounds like a hard road for developers. From a scalability POV you must still host your own applications. So that's no different from Facebook. If you get a million users overnight you have to figure out how to make them scale. On the bright side there was a properties like data store you could use to store data in. The amount of data, types, query model, transaction model, locking model, SLAs, etc seemed open, but not managing state is a big win. From a scalable development POV, I can't help but think the drive towards differentiation will require special coding for each target container and you'll have to pick just a few containers to develop for (think browser wars times 100), but we'll see.

Click to read more ...

Product: Tugela Cache

Tugela Cache is a cache system like memecached, but instead of storing data just in RAM, it stores data in the file system using a b-tree. You trade latency in order to have a very large cache. It's useful for sites that have caching requirements that exceed their available memory. It uses the same wire protocol as memcached so it can be dropped in without a hassle. From the website: As large MediaWiki deployments may gain performance using Memcached, at some level cost of RAM to store all objects becomes too high. In order to balance resource usage and make more use of our Apache server disks, Tugela, the distributed cached on-disk hash database, has arrived. Tugela Cache is derived from Memcached. Much of the code remains the same, but notably, these changes: * Internal slab allocator replaced by BerkeleyDB B-Tree database. * Expiry policy management moved to external program tugela-expire * Much statistics code made obsolete. An interesting point brought up in the comments is using memcached with a larger cache size than physical RAM and then let the OS swap versus using a b-tree to access data on disk. Nginx seems to use the "let the OS swap" approach to good effect. It would be interesting to see which approach works better. For an idea of how an in-process cache and a disk based cache hierarchy can work together take a look at Kevin Burton's IDEA: Hierarchy of caches for high performance AND high capacity memcached. There's also an interesting variation called Memcachedb which is said to be "a better and simplified Tugela." It's more of a persistence mechanism than a cache. It enables transactions, replication, and there's no expiration.

Click to read more ...

Database-Clustering: a8cjdbc - update: version 1.3

The new version of a8cjdbc finished some limitations. Now Clobs and Blobs are supported, and some fixes using binary data. The version was also fully tested with Postgres and mySQL. Since Version 1.3 there is also a free trail version for download available. Check it out and test yourself... Take a look at: http://www.activ8.at/homepage/en/a8cjdbc.php I've downloaded the latest version and setup a environment with one virtual database and two database backends. I tried to make a "non real life szenario": The first backend was a Postgres node, the second was a mySQL node. Everything works fine - failover - recoverylog, etc... with to different backend database types. So check out the trial version and test yourself the clustered driver and give me some results about your experience with a8cjdbc. As I only tested mySQL and Postgres (and the non real life szenario with two different backend types) - maybe someone else have experiences with out databases? greetings Wolfgang

Click to read more ...

a8cjdbc - update verision 1.3

The new version of a8cjdbc finished some limitations. Now Clobs and Blobs are supported, and some fixes using binary data. The version was also fully tested with Postgres and mySQL. Since Version 1.3 there is also a free trail version for download available. Check it out and test yourself... Take a look at: http://www.activ8.at/homepage/en/a8cjdbc.php I've downloaded the latest version and setup a environment with one virtual database and two database backends. I tried to make a "non real life szenario": The first backend was a Postgres node, the second was a mySQL node. Everything works fine - failover - recoverylog, etc... with to different backend database types. So check out the trial version and test yourself the clustered driver and give me some results about your experience with a8cjdbc. As I only tested mySQL and Postgres (and the non real life szenario with two different backend types) - maybe someone else have experiences with out databases? greetings Wolfgang

Click to read more ...

Product: SmartFrog a Distributed Configuration and Deployment Framework

From Wikipedia: SmartFrog is an open-source software framework, written in Java, that manages the configuration, deployment and coordination of a software system broken into components. These components may be distributed across several network hosts. The configuration of components is described using a domain-specific language, whose syntax resembles that of Java. It is a prototype-based object-oriented language, and may thus be compared to Self. The framework is used internally in a variety of HP products. Also, it is being used by HP Labs partners like CERN.

Related Articles

Distributed Testing with SmartFrog

Puppet the Automated Administration System

Click to read more ...