Image Building Tip: Make Clean Without Repeating Yourself

By Don Marti

What do we have to do in make clean? Delete all the files that can be regenerated that we don’t need to keep around.

What’s listed in .gitignore? All the files that can be regenerated that we don’t need to keep around.

Hey, wait a minute. It’s not a good idea to repeat yourself, especially for me, since I have to “start a project” often for demo code. So here’s a way to keep track of all those extre files in one place, with a few lines in Makefile and one handy git command.

OSv on the QEMU Advent Calendar

OSv is on the QEMU Advent Calendar with a ready-to-download VM image.

day 11

Not just a demo, it’s a fully functional Redis data store with a web management console that you can try with a real Redis application. There are more fun projects on the QEMU Advent Calendar site, too.

Making Spam Vanish…in the CLOUD

By Don Marti

We knew that spammers control a lot of bandwidth, because botnets. So, for a long time, people thought that fighting spam by just accepting and discarding huge amounts of it would be counterproductive. The spammers get bandwidth for free, and servers are expensive.

There have been some interesting projects to try to waste the spammer’s bandwidth by fooling the sender into wasting precious bandwidth and CPU by forcing it to retransmit packets, reorder, and retransmit them.

For example, see Spam Cannibal. Other spam tarpits focused on a different part of the spam problem: trapping the address-harvesting part of the spam operation. Address generators would build a dynamic site which has an infinite hierarchy of random pages, each containing a few bogus email addresses and links to more of these fake pages. An example is Sugarplum by Devin Carraway.

Sugarplum created pages such as http://www.devin.com/peaches/lemons and this directory of people who want to buy herbal weight loss products (psst. humans. Not really, just more spamtrap addresses. Don’t tell the address-harvesting bots.)

But in most cases, for a long time, our only hope for fighting spam was to batten down the hatches. Assume that the spammers control more computing power and bandwidth than we do, and just do the best we can to filter it.

vortex image

Is that the best we can do?

Containers, Containers, Containers! More Options for the Cloud

By Dor Laor

Everybody who uses containers has probably heard about Rocket by now. What does this fork re-implementation mean to the Docker community? Let’s dive in.

The Docker technology has disrupted the cloud and datacenter field for the past year. The speed of penetration was amazing and unmatched in the industry. I can’t think of a similar case for such fast-paced adoption of the concept. Neither AWS, OpenStack, nor virtualization were adopted that fast. What made it that attractive for our community to adopt?

Container technology was there for a lot of years. Even before containers, there has always been similar functionality, like Unix’s chroot. However, Docker really changed the game with straightforward simplicity and usability. Docker contributed the following:

  1. A basic one-liner command line: docker run. It masked out all of the LXC complexities and even downloaded the image if needed.

  2. Straightforward image structure: It’s as simple as a directory.

  3. A public image repository, Docker Hub, where you can stash your images for future download, and employ an application-market concept.

Docker had become the hottest thing in tech, sometimes way too hot for reality (otherwise, why would one deploy Docker in a VM, incurring two tiers of management pain?). Beyond Google and AWS, even mighty Microsoft wanted a piece of the pie. We couldn’t believe our eyes, what an amazing meritocracy (no sarcasm). All the vendors of the world unite around one simple format.

Yesterday CoreOS released a fork-like bomb in the form of Rocket. The reason is the very thing that makes Docker attractive, its simplicity, is also a disadvantage since it’s too complex to harness for security, SLA and diverse workloads. Apparently CoreOS didn’t manage to get the Docker team to change the mainline code for it. I guess it’s more than a NIH thing (Not Invented Here) but some limitation of its structure.

This container diversity was expected although it saddens me to see it become an ugly fight:

Twitter thread

We believe that CoreOS has the upper hand when it comes to technology. Managing containers is straightforward, but these folks have the vehicle for deploying them. In addition they wrote cool distributed tools such as etcd and fleet. So while Docker controls the hub and the format, CoreOS leverages it to gain actual users (PR aside…)

There is nothing to fear about Docker momentum, but it has finally met a reality factor and such competition is positive for innovation and speed as well. With all the fuss around Docker, lots of good features like LXC SLA, and buildpacks (the way Heroku and Cloud Foundry handle them) were dropped in the race for Docker compatibility. It was a nice surprise to see that great minds think alike, and just as Rocket uses several phases in container creation, we do the same for OSv. We create an image by spinning up an empty VM (on any given hypervisor) that formats our ZFS filesystem and afterwards listens on ‘nc’ (netcat) to receive a stream of files from the host.

One of the best things about OSv is that we enjoy the best of all worlds. On one hand, we are a library OS, as small as a single container, usually just a kernel, a runtime and an app. An OSv VM can weigh only 20MB, smaller than most containers. On the other hand, we leverage all of the hypervisor features, including live migration, multiple guest kernels on the same host, resource hotplug, isolation, SLA and more.

Let the battle continue as we collect the best of breed features and embrace them as we’ve done with Capstan, with our cloud-init integration, with our RESTful APIs and more.

For the latest cloud and OSv news, subscript to this blog’s feed, or follow @CloudiusSystems on Twitter.

OSv Meetup at EBLUG: Intro Talk and Demo

By Don Marti

OSv is the simple, fast OS platform designed to run one application in the cloud, without the complexity of old-school OSs with their local users, permissions, and sevices. Come see what it’s all about, and enjoy free food and drink from East Bay LUG sponsor, Hurricane Electric.

EBLUG logo

Who: Don Marti, OSv demo monkey

What: Introduction to OSv (with live 9-second build demo!)

Where: East Bay LUG, Fremont, California, USA

When: 7pm Wednesday, November 19, 2014

Why: See the new dedicated cloud OS, without the big travel budget

Please join us for the next EBLUG meeting, co-located with the OSv Meetup group, and get a rapid introduction to OSv, how it works, and where it’s going.

If you’re not already a member of the OSv Meetup group, please join. For general questions on OSv, please join the osv-dev mailing list. You can get general updates by subscribing to this blog’s feed, or folllowing @CloudiusSystems on Twitter.

SSH Tip: Connecting to a Private Network Without Trusting the Bastion Host

By Nadav Har’El

In a typical lab network configuration, one cannot just ssh or scp to the hosts behind the firewall (our company calls them after characters from Norse mythology: Loki, Muninn and friends). Instead, you need to ssh to a bastion host, and from there connect to the internal systems.

ssh user
an ssh user connecting using a bastion host

One can automate these two steps, with a command like

1
ssh -t bastion.example.com ssh loki.lab.example.com

And I guess some of you are already doing that. But this only works for ssh, not scp. One can do scp through an “ssh tunnel”, but this is really ugly to set up every time and to use.

There is also a securty hole in this approach, because anyone who breaks into bastion (which is an external machine and thus exposed to the Internet) can then ssh from there to all the internal machines, or even hijack already-running ssh sessions.

I finally found a much better solution, much easier to use and much more secure.

OSv Early Access Private Beta

Cloudius Systems is introducing an early access version of OSv, the lightweight open-source OS optimized for large cloud workloads, on November 13. Beta program information is now available.

lab mouse

OSv can be used to minimize latency and reduce TCO for many workloads, especially horizontally scaled ones. The software is available as ready-to-run virtual machine images pre-configured to run popular NoSQL data stores such as Redis and Apache Cassandra, as well as in source code form.

“OSv lets you take a project from a developer’s IDE to a flexible, multi-server deployment much faster,” said Dor Laor, CEO of Cloudius Systems. “As customers require more and more throughput at lower and lower operating cost, we enable developers to use the agility of containers along with the proven security and management features known for in virtual machines.”

Cloudius Systems’ early access release features an expanded Web API for management, along with performance improvements that increase efficiency, with an up to a 3.9x throughput gain. Unlike conventional operating systems with their diverse configuration files and complex user and memory management, OSv is designed to provide the minimum OS needed to run a single application at maximum performance in the cloud.

“OSv was designed with virtualization in mind,” said Cloudius Systems CTO, Avi Kivity. “The OS overhead is minimized by a reduction of context switches, data copies, and inefficient constructs such as spinlocks.” OSv virtual machines can be as small as 20MB, far smaller than Docker containers, and boot in less than one second.

Sign up for the OSv beta

You can keep up with the latest OSv news from this blog’s feed, or folllowing @CloudiusSystems on Twitter. Hope to see you at the show.

Photo: Rama for Wikimedia Commons

On Stage Thursday at Amazon re:Invent

If you’re at the Amazon re:Invent show, don’t miss 3rd Annual Startup Launches moderated by Werner Vogels. Cloudius Systems CEO Dor Laor will be there, to talk about….but we can’t tell you that yet.

  • Who: Amazon.com CTO, Dr. Werner Vogels. and five guests

  • What: Learn how innovative new startups use the AWS cloud.

  • Where: Level 2 – Titian 2306

  • When: Thursday, Nov 13, 4:30 PM – 5:15 PM

  • Why: Knowledge, plus special discounts on new products and services.

If you can’t wait, come over to K14 in the exhibit hall for an OSv demo. We’ve got some Raspberry Pi media kits to give away (to especially intense cloud users who need some real hardware to restore balance to their lives.)

demo pod
Unikernels don’t need big heavyweight booths.

You can keep up with the latest OSv news from this blog’s feed, or folllowing @CloudiusSystems on Twitter. Hope to see you at the show.

Raspberry Pi at Amazon Re:invent

Want to see us build a virtual machine for the Amazon cloud at the re:Invent show? Come on over to our booth (K14) and we’ll show you. It’s simple to add a virtual appliance build step to any application build.

Well, that took nine seconds. What else can we do? How about a hardware giveaway? Since your virtual machines are going to take so little of your time, you should have plenty of time to play with a fun hardware kit.

Raspberry Pi media kit

We’ve got some Raspberry Pi media kits to give away. Not just the board, this kit includes

  • Raspberry Pi Model B+ Board

  • 8GB Operating System microSD Card

  • Multicomp Black B+ Case

  • Raspberry Pi Power Supply

  • Wi-Pi Wireless Adapter

  • 3’ HDMi Cable

  • 7’ Ethernet Cable

Just the thing for a home media center, or a starting point for a more ambitious project such as a vintage arcade.

Amazon re:Invent is sold out, but you can sign up for streaming if you’re not able to make it.

You can keep up with the latest OSv news from this blog’s feed, or folllowing @CloudiusSystems on Twitter. Hope to see you at the show.

OSv 0.14 Alpha Includes Management Dashboard

By Tzach Livyatan

We are pleased to announce the OSv 0.14 Alpha Release.

This is the first release to include the dashboard, a browser-based UI for OSv and virtual appliance status.

OSv dashboard

The dashboard includes (among other features):

  • Main tab with Memory, CPU, Disk

  • Thread tab with interactive visualization of thread status and thread usage. You can select and search for particular thread by name.

  • Trace tab, including frequency for each tracepoint. Selection by name and search are available.

  • JVM tab for basic JVM information including memory and GC, when a JVM is installed.

  • Virtual Appliance tab for information relevant to the particular appliance. Tabs for Cassandra and Tomcat are currently available, with more to come.

To build the dashboard, just includes httpserver in the module list. For example:

1
$ make -j 4 image=cassandra,httpserver

Once the appliance is running, the dashboard is available on port 8000 by default. (You can change the port using cloud-init.)

The full 0.14 release announcement is available on the osv-dev mailing list. You can keep up with the latest OSv news from this blog’s feed, or folllowing @CloudiusSystems on Twitter.