Vagrant in the Cloud

My goal for this post is to take the sample app steps from the rails tutorial book, work on them locally, and deploy them in AWS from a script. Then I will share the script. To start with, I have a Vagrantfile that works fine for doing rails development locally, and I have an AWS account. One of the first things I did was set environment variables on my host machine to store my AWS credentials in. Then I can refer to those environment variables in my Vagranfiles, and I don’t need to risk checking an API key into github or something. In the vagrant file, it looks like this:

aws.secret_access_key = ENV['AWS_SECRET_KEY']

And then in my ~/.profile:

export AWS_SECRET_KEY="Wouldn't you like to know"

Then I made a new directory and copied my vagrantfile locally, and began modifying it based on an example vagranfile here.

The first issue I had was some strange security group problems, and I could have wished for clearer error messages. I had an issue very like this one. Note that the error message reported is bascally “group id can’t be null or empty,” but the cause is that the group specified doesn’t belong with the region. One of my main steps for troubleshooting was to create ec2 instances manually through the AWS console, and then look at how AWS was setting values on them by default and copy these values into my config file. This was a big help, since the vagrant-aws plugin is good about telling you that something went wrong, but is not always clear about what. The fix was to make sure that I had a security group that was allowed to create EC2 instances in the region specified in my Vagrantfile.

The next big issue I had is that vagrant ssh wouldn’t work with the boxes I was creating. I would run

vagrant up --provision --provider=aws

And it would always hang out “waiting for SSH to become ready.” My first concern was that I had messed up the security roles in my AWS instance and no one could connect. So I tried this:

$ ec2="path to my ec2 keyfile"
$ instance="public dns for ec2 copied from AWS dashboard"
$ ssh -i $ec2 ubuntu@$instance

and it worked perfectly. So apparently the error is in my Vagrantfile; the workstation I’m on, using the keyfile I provided, has no trouble ssh’ing into that box. I ran this command:

VAGRANT_LOG=info vagrant up --provision --provider=aws 2>&1 | tee log.txt

Here’s what it does: Temporarily set the vagrant log environment variable to have the level “info,” then provision the VM on the AWS provider, but redirect console output, both stderr (2) and stdout (1) into tee, which copies the output stream, so that it gets written to the console and gets written to log.txt.

I noticed right away, given the more verbose logging, that I vagrant is hanging because it keeps retrying to SSH connection and getting a host unreachable error in Ruby’s ssh library. And then I noticed that it’s trying to use a different IP address then the one that I used. It is trying to use the private IP address of the ec2 instance, and I was able to SSH using the public IP address (or public DNS hostname) of the ec2 instance. I tried sshing to the private IP address directly, and I got a better error message: no route to host.

At first I thought that this meant I needed to mess with AWS’s Virtual Private Cloud (VPC) settings, and then I noticed this line in my vagrantfile:

aws.ssh_host_attribute = :private_ip_address

For now, I just changed that to :public_ip_address, and boom, ssh began addressing the server correctly, and I can vagrant up and ssh in just fine.

So I guess this week’s lessons are (aside from “read carefully config files pasted from the internet”):

  1. If AWS gives you a “group ID cannot be null” error, make sure that your persmissions and regions are correct. It really means “the group name in your Vagrantfile file does not correspond to a group that has perission to create an instance of that type in that region.”
  2. vagrant up will hang at waiting for SSH to become available indefinitely if the underlying error is “destination host unreachable”. I would be frustrated with this, but basically, vagrant is telling AWS to spin up an instance, and then having SSH call it in a loop until it answers; Vagrant doesn’t know whether the instance is still booting, or is not reachable for some other reason.

I think getting vagrant up to fire off an SSH session that works is all for this post. Next time, I’ll resume the rails tutorial on AWS, and I’ll try moving my Vagrantfile from Ubuntu to Windows and report some of the difficulties.

Till then, happy learning!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s