http://news.oreilly.com/2008/08/luke-kanies-wants-to-modernize.html

Errors aside in the transcript, it’s great article.  Administration is no longer a task, but a process.  If you are still doing administration via SSH, it’s time to look at Puppet.

package { "openssh-server":
      ensure  => latest,
      notify  => Service["sshd"],
}
file { "sshd_config":
       name     => "/etc/ssh/sshd_config",
       checksum => md5,
       ensure   => present,
       owner    => 'root',
       group    => 'root',
       mode     => '0600',
       require  => Package["openssh-server"],
       notify   => Service["sshd"],
}
service { "sshd":
       name       => "sshd",
       ensure     => running,
       enable     => true,
       hasrestart => true,
       hasstatus  => true,
       require => Package["openssh-server"],
}

This 22 line recipe does all of the following:

1. Installs the openssh-server RPM via ‘yum’.
2. Automatically upgrades the openssh-server RPM, if a newer version is available.
3. Makes sure the sshd_config configuration file exists and has the proper permissions.
4. Ensure the sshd server starts at boot time.
5. Ensure the sshd server is currently running.
6. If either the RPM is upgraded OR the sshd_config file changes restart the sshd service.
7. If during any time puppet runs again and the server doesn’t match the recipe it will change it back to this state.
8. Perform this task on every server you specify.

While the above recipe hasn’t been tested on other Unix platforms, only minor changes would be required.  Previously to do this you needed to create custom shell scripts, use Cfengine, purchase an expensive software automation tool, or manually perform this on each server installation.  Most options are hacks and not as graceful as Puppet. Package managers, while moved Unix administration into the 21st century (instead of the medieval times of compiling software), still have some warts.  Specifically package managers lack:

1. a good updating procedure.  Installations are well covered.
2. passing your own configuration files specific to your needs/wants.
3. performing the tasks in a specific order, or making sure specific actions occur before an application is installed
4. a service is running and will run at boot time

Things like ‘yum’ on CentOS/RHEL addressed #1 and #3 somewhat, but didn’t address configuration files, and the state of the service.  Before Puppet, it required creating custom RPMs.  With custom RPMs, the issue then became when updates occurred from the distro provider.

Puppet makes system administration a programming task, rather then manual labor process. It’s still very common to see administrators use a SSH prompt to manage each server. Manually performing administration is a time consuming and error prone process.  Puppet allows us a transfer of our best practices, apply our administration experience to the server’s configuration, and allows us to make network-wide installations with ease.

I agree with the notion; “Operations: The New Secret Sauce” (article #1 , article #2).  Puppet makes deployment of new VPS instances quick and exact.   System administrators are skilled individuals, great with keeping operations running smoothly, but usually not good with automation.  Puppet allows to transfer an administrator’s knowledge into a repeatable process.  Automating system administration is the next advancement with Web 2.0, Cloud Computing, SaaS, or whatever the latest trend may be called.  When using our HostCube service, Puppet puts the power of a large operations center like Google in the hands of much smaller companies, Cloud computing, while may address the quick provisioning of hardware, it doesn’t address operations. The ability to automate the install, configure, patch, monitor and backup are important aspects and HostCube does for you automatically and seamlessly.  The bigger and much more complex problem is system administration, not hardware provisioning.

Puppet allows us to Malkovich a setup, over and over and over again.

• EC2 pay as your go service, while cheap for low CPU/bandwidth usage, can get very costly compared to our fixed plans. With HostCube there are no surprises at the end of the month. Most of today’s Internet applications are CPU bound and EC2 can get very costly in this regard.
• No hardware based load balancer, important for scaling or automatic fail over.
• No persistent storage unless you use their S3 service. Shutdown or the node dies; your data and configuration settings are gone. Using S3 service your disk I/O traffic then becomes network bound and of course is an additional fee.
• When compared to our managed VPSes, you must perform all of the system administration yourself. This IMHO is the biggest added value when comparing services. Our service already includes backups, monitoring, administration, patch management, security, and a control panel that makes many administration tasks a simple click. With EC2 you must be much more involved with the system administration.
• Technical support with EC2 is an additional option. What’s not clear is how much support you really get. Can they assist and give recommendations on how to scale your site?
• Odd instance sizes (1.7 GB of memory, 350GB of storage?) what is that? Computing is based upon the multiples of 2.
• 32-bit by default. We, by default, use 64-bit which is 10-15% faster than the 32 bit version.  It appears you have to use the their Extra Large instance to get 64-bit.  This really becomes noticeable when you use more than 2GB of ram.  Not sure with Amazon but all of our nodes are using 64 bit based Xen.  The hypervisor is really where the performance matters.

In brief, EC2 is really geared towards batch based processing and processing services in low volume. If you need public service access (i.e. web service) and hassle-free administration HostCube is a perfect fit and a much better value.

While the HostCube service has some shortcomings, we are adding services like shared storage, quick (under 15 min.) provisioning and an API in the near future.

IMHO they completely missed the point and some of the blog commentators caught this. Regardless if their service is open, what they failed to mention, you are the system administrator. With that you have the responsibility of installing software, proactive monitoring, patch management, security, hardening and backups. This is fine if you are a full time system administrator; bad if you are a developer.

Cloud computing is about abstracting the technical details of your SaaS (Software as a Service) or PaaS (Platform as a Service). My favorite statement, “It just works!” applies here. What they are calling “open” is really a myth. Regardless if you have root access or not you still are locked into a specific hosting provider, OS, and the software applications you choose. Anyone that has switched dedicated server providers can attest this isn’t a small task. In addition, with the large amounts of SSH, FTP, IMAP and POP3 attacks we see its obvious proper system administration on a large scale isn’t already happening. What makes them think giving root access will make these other issues better?

Developers, for the most part, care about their development environment. In some cases yes, OS flavor does matter, but in most cases it does not. Developers typically want an environment that works and don’t have to worry about how to install and configure software packages. What’s important then? The development language and the tools that aid in the development. Giving root access to each developer (each on their own mind you) install to and configure a software package wastes time. Wading through docs, wikis, forums, and other online info trying to get a package configured, can be time consuming and frustrating experience. In some cases this requires a lot of technical skill. There has to be a better way. The better way is to offer pre-built configurations of services, programming languages and applications. This is what we offer with our HostCube service. Why reinvent the wheel each time you need a LAMP stack installed? Tools like Puppet automate this process and make it consistent.

Let me sidetrack for a minute and discuss the differences between system administrators and programmers. I’ve worked on both sides of the fence and from my experience, most developers make poor system administrators, as do many system administrators (sysadmins) make poor programmers. The mindsets are completely different. Developers care about how quickly they can develop their code and bring it into production. Sysadmins care about the stability, reliability and security of the service they are responsible for. As you can see, these two mindsets are always at odds with each other.

To solve this conflict, I believe in the traditional three tier development methodology. They are:

• Development – code that’s in flux and to “try out” new things
• Staging – some state of code that is stable and in testing before production
• Production – live code that’s being used by users, customers, vendors, etc.

This tiered environment allows for the best of both worlds and is recommended when a customer wants root access to a production server. Unfortunately I have seen many developers perform all of these tiers on their production system! In all of but the smallest of projects, this can lead disaster on many levels.

Creating a VPS for development is where I personally believe developers should/could have root. Let them play in their sandbox, break things and test out new code. Staging (which should mirror production configuration) and production should be managed by system administrators. In my opinion, developers, at least with production, should not have root access. At HostCube the value added is we perform the software installs, proactively monitor, patch management, security, hardening and backups. That’s what we’re experts in and have invested many years developing tools to automate this process. We also do realize unmanaged VPSes serve a valuable niche.