Update 2: 3tera has added Dynamic Appliances, which are "packaged data center operations like backup, migration or SLAs that users can add to their applications to provide functionality."
Update: in an effort to help cross the chasm of how start building a website using their grid OS, 3tera is offering their Assured Success Plan. The idea is to provide training, consulting, and support so you can get started with some confidence you'll end up succeeding.
If you are starting or extending a website you have a problem: what technologies should you use?
Now there are more answers to that question than ever. One new and refreshingly innovative answer is 3tera's grid OS. In this podcast interview with Bert Armijo from 3tera, we'll learn how 3tera wants to change how you build websites.
How? By transforming the physical into the virtual and then allowing the virtual to be manipulated as if it were real. Could I possibly be more abstract? Not really. But when I think of what they are doing that's the mental model I see whirling around in my mind. Don't worry, I promise we'll drill down to how it can help you in the real world. Let's see how.
I think of 3tera's product as like staying at a nice hotel. At home you are in charge. If something needs doing you must do it. If something breaks you must fix it. But at a nice hotel everything just happens for you. Your room is cleaned, beds are made, outrageously expensive candy bars are replaced in the mini-bar, food arrives when you order it and plates disappear when you are done, and the courtesy mint is placed just so on your pillow. You are free to simply enjoy your stay. All the other details of living just happen.
That's the same sort of experience 3tera is trying to provide for your website. You can concentrate on your application and 3tera, through their GUI on the front-end and their AppLogic grid operating system on the back-end, worries about all the housekeeping.
I think Bert summed up their goal wonderfully when said their aim is to:
Get peoples hands off physical boxes and to give them a way to define complex infrastructures in a reusable way that they can then instantiate, trade, sell, or replicate, backup up and manage as individual units. This is what AppLogic that does incredibly way.
What they are doing is taking hard physical resources like CPU and storage and decoupling them from their physical sources so you can just order and use them on demand without worrying how its done under the covers. This is trend that has been happening for a while, but their grid OS takes that process to the next level.
Your physical co-lo cage is now a private virtual data center. Physical boxes, once lovingly spec'ed, bought, and installed are now allocated on demand from a phalanx of preconfigured and separately maintained servers. Physical storage, once lovingly pieced together from disks, controllers, and networks is now allocated from a vast unending sea of virtual storage. Physical load balancers are now programs you can create.
What this means for you is you can take a website architecture you've draw up on your white board and simply and quickly create it in a data center. Its all configurable from a GUI. You can bring on 10 new web servers with a simple drag and drop operation. It's basically your white board diagram come to life, only you get to skip all the nasty implementation bits. In the virtual world the nasty non application related implementation bits are someone else's problem.
3tera's value proposition pretty easy to understand:
Simplify the data center. You no longer need to locate, outfit, staff, maintain,
and support a co-lo space.
Simplify operations. A few people can manage a lot machines.
Simplify disaster recovery. Failover is complicated and often doesn't work as planned.
With AppLogic your redundant data center is always the same because the virtual data center
is copied a unit. You can pick it up and move it anywhere you want.
Simplify the cost model for growth. If you grow how are you going to fund your
hardware? Growing on a grid is more agile, incremental, and requires less upfront
investment.
Simplify your architecture. The grid OS provides a powerful implementation
model of how you should structure, grow, and maintain your system. You don't need to
code it from scratch or think it up yourself.
In short: customers don't care about your servers. Hardware and the data center do not
add value. You core competency is in your application and running your business, not playing with servers.
Well, that's it for the overview. Please listen to this podcast for all the nitty-gritty details.
Download audio file (1:16 minutes, mp3).
Podcast Notes
I know what you are probably saying. You are saying: "But Todd, the podcast is over an hour long, couldn't you have please made it longer? I have nothing else to do today and I need to waste more time!" What can I say, Bert was very knowledgeable and helpful, and this is a new model for building scalable websites so I was trying to figure out how I could physically make a website using their product. That takes a lot of questions. I am happy with the result though. I think I have a good picture of how their system works and I think its well worth investigating if you are in the market for creating or expanding a website.
Here are some notes taken from the podcast.
They started 3 years ago. At that time nobody could understand what they were trying to build. They have just now been able to build the higher level features, like Smart Appliances, that they wanted to build originally. They've been concentrating on making all the plumbing work.
The AppLogic grid operating system allows you to take hard infrastructure servers, load balancers, firewalls, VPNs, all these boxes you need to make a website and it allows you deploy these in a virtual data center
A virtual data center (VDC) is like a cage you would buy from a co-location service except you operate
and manage it through a browser. You can be anywhere in the world and you can use hosting services anywhere in the world.
An entry level package ranges from $500 to a few thousand a month. The starting point is 4 - 32 CPUs, some amount of storage and some amount bandwidth. You add resources as you need to. Overage charges are passed through to you from the data center provider. They don't mark it up.
They don't own any servers. They contract with hosting providers data centers, like Softlayer and Layeredtech, for a uniform set of resources.
They offer templates for a scalable virtualized LAMP infrastructure as a starting point for building your own applications.
Their GUI shows you the architecture. You don't have to think of physical boxes.
There's a controller for the VDC through which you can provision your system.
You can still login to any physical or logical service. You have root access. You can install anything and manage the system, but you don't have to worry about where it physically resides.
To create an application:
- You use the controller to provision a LAMP cluster.
- Then you log into Apache server and configure it how you wish.
- Then restart and it begins to serve.
- Say you want 10 front-end web servers.
- The load balancer is a virtual load balancer you program.
- You use virtual NAS.
- Upload code to the NAS.
- Then have all apache servers run off the NAS. So you don't have to log into all and upload code.
Shared storage is part of the virtual data center by definition. You can create as many volumes as you wish. All are mirrored for high availability. If a virtual server goes down AppLogic will simply restart on another available resource in the data center.
Partners build the grid backbone to which nodes and other resources are attached. AppLogic runs that grid backbone. When you sign up you provision the virtual data center the nodes on the backbone are assigned to your VDC. A controller allows you to provision your VDC. Anything you can do in co-lo cage you can do, but there's nothing physical. AppLogic carries out your commands on the grid.
They provide standards for the hosting service. A variety of machine classifications available. Have customers with 50TB of storage. The largest number of CPUs in a single VDC is over 450.
To see if the VDC meets your requirements you run a test on the VDC. Once you have resources in your VDC they are not shared with anyone else so you can be confident the performance will be as tested. It's not a VPS. Their customers run production systems. They are all running a business of some sort.
Pricing is designed to be attractive for startups, but not artificially low to over-subscribe.
Currently there's no data center API. It's scriptable from the CLI. Smart Appliances can package up a data center operations into a drag and drop package. You can drag them into any application. Their first Smart Appliance is "follow me" which can move your application to a data center that is close to you. If you are in Asia you can move your data center to Asia. So your data center can follow you around. No coding is needed on your part. Just drag it into your VDC.
With AppLogic instead of managing a bunch of different things you manage your application. You do it once. AppLogic maintains the infrastructure for you.
In an upgrade of 10 Apache servers you don't upgrade standing infrastructure. You take a copy of your application and upgrade the copy.
Let's say you have an Apache server you want to patch. You create one prototype, which they call a class volume. Then when application restarts all the new changes will be picked up everywhere.
The power of what it means to be virtual can be seen in their rollback model. You don't upgrade in-place. You upgrade a copy. Because everything is virtual its easy to make copies of your entire data center. So you can copy your data center, keep the original running, and switch to the upgraded version. If the upgrade version doesn't work you can rollback to the original version of your VDC. This would be almost impossible using traditional methods. An application is the full state of the application with all its data. So you are operating a full complete copy of the application with all of its data. You can rollback to a complete running instance of the application. You just restart the old version.
For upgrades that require transformations, like database upgrades, you can write a script to run a database transformation.
They don't over automate. They don't only want to have their way of doing things.
The model an application has having two parts: the appliance and the content. For a web server this means:
- Web Server
- Content that it's serving.
You first create a prototype of what you want your system to look like. This becomes a class from which you later can create instances. There are templates, like the Linux appliance, to build from. Through their on-line system you configure your system, install packages, etc. When it works the way you want you can drag into your catalog as a template for building new instances. You can create hundreds of copies if you choose.
Content would be served off a mount location from inside the VDC.
You can upgrade the catalog element and restart the appliance and it will automatically upgrade for you. Not transactional. It's only an individual basis.
You can pin machines. You can get the environment to make machine specific configurations. You can put appliances into standby so you can quickly add additional resources on demand.
Their load balancer is Pound. No spam detection, but it is session aware. You can use others if you want.
They specialize in the code that runs the grid. They aren't specialists in load balancers and routers, etc.
In the VDC you can share infrastructure. You can email each other a clustered database, for example. You can save and package up an integration effort as an assembly. Save it. Sell it. Share it.
You can create an active-active redundancy scheme and pay for only resources you need because you can bring on resources like the front-end when you need them.
Many companies periodically make a local copy of their VDC and move it to their disaster center.
- Remember, with a VDC it's easy to pick up your whole data center and move it somewhere else. The catalog doesn't have to be copied each time. Just the data for applications can be copied over. Not so bad with a fast backbone.
- Disaster recovery can be triggered by a 3rd party or scripts.
- This model is sufficient for companies that can accept some down time.
- If no data loss can be tolerated you need replicate in an acive-active architecture.
Some companies maintain fungible data centers. They constantly copy their data center over to backup locations. If an app goes down they can fire up a replacement.
With AppLogic you can create a stub that can start an application on demand if it's not already running. This allows you to share resources. You can shut it down at night and save those resources for other applications.
Here's how they would handle TechCrunch:
- Let's say you have an 8 grid data center. Let's say your normal load
takes 20-30% of that.
- First thing you'll do is use more resources from within the grid.
- Then reconfigure appliances with more resources and restart them.
- Then call your provider to add more resources.
- Softlayer, for example, has a 500-1000 server inventory. So you can add servers to your grid within an hour a two. Currently this process requires human intervention.
Finding good OPs people is difficult. So with the VDC you can automate most of it and you don't need a big OPs team.
In you VDC your data center configuration is in the meta data, so its not kept as tacit knowledge. One or two people can run a thousand servers because you aren't really running servers, you are running applications.
Monitoring
- AppLogic in control of all resources. You can build dashboards right off the bat.
- You van plug your monitored variables into their monitoring system.
- The data are available over the web.
- Widgets are available for the display of live stats.
Different Way of Thinking about Your System
- Typically you put the database on fastest server. Instead, they recommend allocating high end machines to everything so your database can run anywhere. A different way of thinking about your system.
- Same with SAN. You don't need a SAN with the storage in the VDC. You are locking yourself into certain ways of thinking that don't apply in the VDC. Concept of using a SAN is just another lock-in.
Some Observations and Conclusions
I think the grid/virtualization approach, in one form or another, is the wave of the future. It simply makes it easier for companies to scale applications. And as applications themselves are structured to run natively on a grid, it will become even easier.
Reaching the full potential of the virtual data center depends on having a more granular billing strategy and more fine grained control over resource management. For example, if I have 6 CPU grid and I want to upgrade. I don't want to pay for a 12 CPU data center just so I can upgrade a copy. I don't need 12 normally, I just transiently need 12. So during my upgrade I want my script to trigger allocating a copy of my VDC, do the upgrade, switch to it, and then decommission the old VDC. So I want for a time to have 6 extra servers for the time it takes to upgrade. Then the old VDC should go away. And I should only be billed for the resources I am using while I am using them. This would also give a more satisfactory solution to the TechCrunch scenario.
You need to architect your system to take advantage of the grid. To me this means a shared nothing architecture that can be grown horizontally by adding more machines on demand. Applications should read their configuration off shared storage so the configuration doesn't need to be configured on each machine and you can bring up new machines based on a template. If you need to scale a new machine should come up and automatically start handling load. Queuing architectures, for example, have this attribute.
They need a data center API so you can treat the data center like an object. This would allow you to orchestrate various data centers around the world as a single coopering unit.
Operations within a grid would benefit from standardization. I know this enters the application realm, but operations like upgrade and failover are common and hard. So it would be useful of common processes could be developed and easily deployed.
They need turnkey options for those new to the game. As it stands the path from signing up to their service and deploying a web service is little scary. They are very honest in saying they do only one part of the overall picture. But many people need a painting, not a brush and paint. It would be helpful to have out of the box plans for solving the most common problems people face.
I would like to thank Bert again for taking this time for the interview! May the grid be with you, always.
Related Sites and Articles
http://www.3tera.com/
On-Demand Infinitely Scalable Database Seed the Amazon EC2 Cloud
Saturday, October 20, 2007 at 8:27AM 
Reader Comments (17)
Ok, so 3tera create virtual appliances from individual applications, or bundles of applications, is that right? For example, you create an appliance with an AMP stack. The appliance fires up, handles the request, shuts down. Is that how it works?
I might have to listen to the 1hr podcast, I'm not sure I can get my head round this from the text...
Yes to the first question. Though the scope is your entire data center (appliances, data, load balancer, etc). And the appliances don't shut down after each request. Once you provision an appliance it stays up until you deprovision it.
Here you can listen for the podcast via a flash-based player: http://boomp3.com/m/9c8bb4eceee3
Ok, I think I get it. So you build your "virtual appliance" with your entire application, web server, code, etc. Then you "provision" the appliance once to start with. Load increases, you "provision" a second appliance, and so on. Is that about it?
One question, are your "virtual appliances" able to run on anything else? Or only on top of 3tera's grid? No use building all this infrastructure if it ties you in to one company.
That's it. They use class/instance terminology. The template appliance is a class and provisioning a new appliance is creating an instance. Of course your application must be structured to automatically scale with addition of new nodes, which can be tricky.
I think the point of using them would be to use their grid because the unit they are talking about is the entire data center, which includes storage and other resources, not just a VM.
I was initial quite excited about 3Tera. But, last time I checked there were still issues supporting MySQL above version 4.1 in their setup. This was a no go for me. Now, I'm not saying that you can't create your own MySQL 5 instances because you can do that I understand. But, I recall that there were some support issues with custom instances. If anyone knows different or this has changed then please correct me.
I find provisioning on 3Tera to be a significant barrier to entry when I can just be up and running in no time on EC2 for so little effort and with so much flexibility. Also, I'd much rather see a truer more granular utility pricing model where I might pay a reasonable setup fee to get going if really necessary and incremental pricing for only what I use over time.
Having said all that though, I think it's all about CTO and keeping up technically with the software. I love the work these guys are doing and I'm always keeping an eye out to see if I can find the right client to try on 3Tera or if 3Tera makes it easier to provision their services on-demand.
Nice podcast, though I have to say I was a bit disappointed in the way the interview went.
Todd, you seemed pretty skeptical of the AppLogic pitch, and I think you approached it the wrong way by not giving Burt the proper chance to respond. You talked over him several times, which was exacerbated by the fact that the levels weren't equal amongst you two. It also seemed that instead of grilling him on the fuzzy portions of him claims you just dismissed them. You had a good opportunity to shed some light on the details (which I seem to be unable to find on the net) and didn't capitalize. It gave the whole interview this sort of "I'm so much smarter than you" condescending feel.
Maybe you could get in touch with one of their engineers and do another podcast and get deep into the details.
> Nice podcast, though I have to say I was a bit disappointed in the way the interview went
It was my first podcast. The good news is I have vast expanses of room for improvement :-) You're right about the levels. I didn't have the recording on two channel. Eventually I just gave up trying to equalize the sound. And Skype latency does make it easy to speak over each other, some bits of the conversation aren't even in the recording, but I did learn to just let people talk rather than try and converse normally. I also learned that normal conversational feedback just sounds dumb, so I won't do that again either.
> you seemed pretty skeptical of the AppLogic pitch
Not at all. My goal was to stitch together in my own mind how I would build a website using their infrastructure, so that's how the conversation went. I have an excellent mental model of how to do that using traditional tech, but using their new tech, I had no idea. Bridging that gap takes some meandering. Being skeptical was not my intent, but getting to the point I was confident I could build a website and it would work, was. And where I saw problems I talked about them. Where I saw good things I talked about them. I think that's pretty reasonable.
> You had a good opportunity to shed some light on the details
What details did you want brought into sharper focus? There's still time.
> I'm so much smarter than you
That's unfortunate as anyone listening to Bert's grasp of the issues would easily conclude the opposite. I will try to improve.
It is similar to creating/installing your application on top of VMWare virtual machine and be able to move your entire application and your data from one workstation to another. All you need is a VMware Player to run your virtual machine in any workstation.
The only difficulty is moving your database. Some of the databases are aware of the hardware and performs raw writes to the hard drives. I don’t even know how they can handle cluster database.
I think 3tera is working on great product. Are they hiring?
> Being skeptical was not my intent, but getting to the point I was confident I could build a website and it would work, was. And where I saw problems I talked about them. Where I saw good things I talked about them. I think that's pretty reasonable.
I meant the "being skeptical" part as a compliment :) Reading through their website It seems like they want to refrain from going into the details. They pitch their "Grid School" service in several places. I was hoping your stance could break down some of those walls for us normal Joes.
> What details did you want brought into sharper focus? There's still time.
1. You started to get into how their terminal concept translates into application level configuration. How does my application pick up that I've moved the db connection from one node to another? Do I have reboot the entire application to pick up the change, or just the node?
2. Speaking of the Controller in general, Burt talked repeatedly about APIs breeding lock-in. How does he figure the controller does not? If my applications have to be changed to interface with the configurations created by the controller, what's the difference? Likewise, how does the custom monitoring bits avoid the same issues. In general the arguments against an API seemed very dubious.
3. When pressed on the rollback issue, Burt mentioned that you just rollback to the old version of the NFS node to revert your data transformations. I'm curious about how this is implemented: Is this using snapshots, or a full copy? Do I create a backup node on my application and copy the data over to achieve this? If my copy takes ten minutes to create, does it take another ten minutes to roll back?
4. It seems like most of the nodes are off the shelf code: Pound for LB, Apache, Mysql. How have you built the NAS piece of the system? Is it using off the shelf code too, or is it something built in house?
5. One issue that your grid is a fixed size was brought up. You pay for 10 CPUS, and have to ask for more which takes X amount of time to provision. What happens when one of the grid machines goes down? Do I have to reboot my application to reallocate resources or does the nodes running on the physical box dynamically spin up on the other machines in my VDC?
> That's unfortunate as anyone listening to Bert's grasp of the issues would easily conclude the opposite. I will try to improve.
I listened to the podcast again with your comments in context, seeing as my post-first-listen comments were a little harsh. I think the issues mostly were the levels. I missed several things Burt said because of the loudness of your voice. I'm thinking now it is probably just perceptual on my part. Hope I didn't offend :).
> Hope I didn't offend :)
Not at all. I can't improve without constructive criticism. And I've forwarded your questions on to Bert. Hopefully he'll get a chance to reply soon.
Todd,
Thanks for forwarding my questions. Just as a little tidbit for others reading this, I stumbled upon the documentation site for AppLogic, which has some great info on the product:
http://doc.3tera.net/
Their website is an absolute beast to find information on. I stumbled onto that link in the release notes for AppLogic 2.0, and cannot find it anywhere else on their site.
Scott, here are Bert's replies to your questions:
1. Most changes made to an appliance in the application definition can be picked up by the appliance simply by restarting the appliance.
2. To avoid lock-in, all of the mechanisms we use to connect appliances and give them access to storage are implemented in a virtualization layer. Simply by appropriately configuring software packages when they're installed in an appliance, AppLogic can control their behavior. Therefore, anyone wanting to move back to physical hardware can simply reinstall their software.
3. In AppLogic, backups are a full executable instance of the application including all data volumes, so roll back involves simply starting the backup.
4. The NAS appliance is open source code, but I'll have to check on what was used.
5. We made a conscious decision in designing AppLogic that each customer's resources must be exclusively theirs in order to ensure predictable performance for production applications. However, this doesn't mean grids are fixed in size. In fact nodes can be added or removed at any time and properly configured applications can take advantage of added resources without any downtime. As for node failures, AppLogic simply restarts affected components on other resources and reconnects them into the instance of the application they came from.
I'm sorry I couldn't get back here sooner, but travel has kept me quite busy over the past week.
You're all raising great questions. I tried to answer some of them earlier today in an email to Todd which I see he's posted. Thanks Todd!
In reading through your comments I see a few more questions I can address.
>>> " ... issues supporting MySQL above version 4.1."
The catalog we supply is a starting point. Kind of like the rectangular bricks in a Lego set. Most users build their own appliances for critical components like databases and web servers. So, while you're correct that the version we supplied was out of date, there aren't any issues I'm aware of with more recent releases of MySQL. And, as we move forward we'll put some extra effort into keeping up to date.
>>> "I don’t even know how they can handle cluster database."
We don't supply a clustered database as part of the standard catalog yet, but we have users who've run clusters successfully. We've gotten a lot of requests for this so right now we're investigating a few contractors with experience clustering databases to add one to the standard catalog.
>>> "I find provisioning on 3Tera to be a significant barrier to entry when I can just be up and running in no time on EC2 for so little effort and with so much flexibility."
I presume you're referring to the prepackaged AMI's, because installing software takes significantly less time on AppLogic and can be done on the system instead of requiring you to build a server locally and then uploading the image. That said, you're correct that we haven't provided a way to share prepackaged applications or appliances yet, and we're working on that now. Building such a repository is a bit more of a challenge for AppLogic than EC2 because we have services running in numerous data centers on different continents and it's a LOT of data to move around. We'll get it done though!
Thanks again to everyone for comments!
Okay, you'll have to forgive me for my first "haha" post yet, but haha: "How? By transforming the physical into the virtual and then allowing the virtual to be manipulated as if it were real."
What?
I mean.. really. :)
--
Dustin Puryear
Author, "Best Practices for Managing Linux and UNIX Servers"
http://www.puryear-it.com/pubs/linux-unix-best-practices
Well, that's how I see it unfolding. Physical resources like CPUs, load balancing, and storage are virtualized behind a grid OS layer. That grid OS layer allows you to treat the virtualized resources like they are real, independent of the underlying physical incarnation. That's what a virtual data center means. You can just pick up a data center and move it. A physical representation can't do that. It helps me think about what's going on and what's different about it.
Dustin,
When you build out the infrastructure for an application today you start with a plan, a definition of what you want the system administrators to build. They then spend a great deal of time loading software, building racks and plugging cables to make that plan a physical system.
AppLogic gives you a way to define that plan in a machine implementable fashion. When you hit run AppLogic then maps the plan for your application onto the grid. The result is that you get the same infrastructure as if it had been built out by hand. Except, of course, that AppLogic can now maintain the infrastructure for you.
As a Unix user you should recognize this - it's what we always did with shell scripts. Launch processes, connect them, redirect outputs and inputs, and control life cycle. We simply do it at the level of a data center.