Monthly Archives: October 2015

Create a Vagrant Base Box based on Ubuntu

Introduction

In this post, we took the first steps with Vagrant. We have been using an image called ‘precise64’ which could be found at the following location: http://files.vagrantup.com/precise64.box. While this is certainly very nice to get started, I was wondering what it would take to fully create and define your own environment. So that’s what this post is all about.

First off, go ahead and install a normal Ubuntu in Virtual box. I consider this outside the scope of this blog post as I imagine this is fairly easy for most people. Let me give some headstart in case completely new to you:

  • Download Ubuntu ISO
  • Open Virtualbox
  • Create a virtual machine using this ISO (choose a VMDK file)
  • Create a user called ‘vagrant’
  • Install the guest editions
  • Install Openssh server
  • Install the software you would like to see in the resulting Vagrant box file
  • Add vagrant user to sudoers file (see below for more explanation on this one)
  • Install Vagrant public keys to your Ubuntu instal
  • The last two steps might need some more detail so here it comes:

    Add user to the sudoers file

        $ sudo su -
        $ visudo
        # Add the following line to the end of the file.
        vagrant ALL=(ALL) NOPASSWD:ALL
    

    Install Vagrant public keys

    The reason this is needed is to provide the capability to SSH into our virtual machines later on using SSH.

        $ mkdir -p /home/vagrant/.ssh
        $ wget --no-check-certificate https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub -O /home/vagrant/.ssh/authorized_keys
        $ chmod 0700 /home/vagrant/.ssh
        $ chmod 0600 /home/vagrant/.ssh/authorized_keys
        $ chown -R vagrant /home/vagrant/.ssh
    

    If all goes well, you should have a running virtual machine in Virtual Box. As such this has nothing to do really with Vagrant itself. See below screenshots how it will look like.

    Screenshot_VirtualBox

    Ubuntu_Screenshot

    Creating a Vagrant box file

    macbookpro:ubuntu wim$ VBoxManage list vms
    "Ubuntu Development Vagrant Base" {919810de-3097-4845-9bec-45844850e066}
    
    macbookpro:ubuntu wim$ vagrant package --base "Ubuntu Development Vagrant Base"
    ==> Ubuntu Development Vagrant Base: Exporting VM...
    ==> Ubuntu Development Vagrant Base: Compressing package to: ~/Dropbox/Technologies/Vagrant/ubuntu/package.box
    
    macbookpro:ubuntu wim$ vagrant box add "Ubuntu Development Vagrant Base" package.box
    ==> box: Box file was not detected as metadata. Adding it directly...
    ==> box: Adding box 'Ubuntu Development Vagrant Base' (v0) for provider:
        box: Unpacking necessary files from: file:///Users/wauterw/Dropbox/Technologies/Vagrant/ubuntu/package.box
    ==> box: Successfully added box 'Ubuntu Development Vagrant Base' (v0) for 'virtualbox'!
    
    macbookpro:ubuntu wim$ vagrant init "Ubuntu Development Vagrant Base"
    A `Vagrantfile` has been placed in this directory. You are now
    ready to `vagrant up` your first virtual environment! Please read
    the comments in the Vagrantfile as well as documentation on
    `vagrantup.com` for more information on using Vagrant.
    

    After applying the init file, you will see that a Vagrantfile has been created.

    macbookpro:ubuntu wim$ ls -al
    total 6015432
    drwxr-xr-x@ 6 wauterw  staff         204 Nov  3 13:55 .
    drwxr-xr-x@ 9 wauterw  staff         306 Oct 29 15:28 ..
    drwxr-xr-x@ 3 wauterw  staff         102 Oct 29 16:00 .vagrant
    -rw-r--r--@ 1 wauterw  staff        3043 Nov  3 13:55 Vagrantfile
    -rw-r--r--@ 1 wauterw  staff  3079888994 Nov  3 13:32 package.box
    

    Next step is to uncomment a couple of lines in the Vagrantfile.

        
        config.vm.provider "virtualbox" do |v|
           v.gui = true
        end
    

    Now everything should be ready to launch the virtual machine via Vagrant:

    macbookpro:ubuntu wim$ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    ==> default: Importing base box 'Ubuntu Development Vagrant Base'...
    ==> default: Matching MAC address for NAT networking...
    ==> default: Setting the name of the VM: ubuntu_default_1446555586668_78070
    ==> default: Clearing any previously set network interfaces...
    ==> default: Preparing network interfaces based on configuration...
        default: Adapter 1: nat
    ==> default: Forwarding ports...
        default: 22 => 2222 (adapter 1)
    ==> default: Booting VM...
    ==> default: Waiting for machine to boot. This may take a few minutes...
        default: SSH address: 127.0.0.1:2222
        default: SSH username: vagrant
        default: SSH auth method: private key
        default: Warning: Connection timeout. Retrying...
        default:
        default: Vagrant insecure key detected. Vagrant will automatically replace
        default: this with a newly generated keypair for better security.
        default:
        default: Inserting generated public key within guest...
        default: Removing insecure key from the guest if it's present...
        default: Key inserted! Disconnecting and reconnecting using new SSH key...
    ==> default: Machine booted and ready!
    ==> default: Checking for guest additions in VM...
    ==> default: Mounting shared folders...
        default: /vagrant => ~/Dropbox/Technologies/Vagrant/ubuntu
    

    And since we setup also SSH server on the Vagrant base box, we can ssh into the newly created machine

    macbookpro:ubuntu wim$ vagrant ssh
    vagrant@vagrant-VirtualBox:~$
    vagrant@vagrant-VirtualBox:~$

Vagrant: AWS (Launch multiple instances)

Introduction

A while ago, I provided a short intro on deploying a single virtual machine onto AWS (see this post). Since I was wondering how difficult it would be to deploy multiple EC2 instances I gave this a try as well. It turned out to be relatively simple, below are my findings.

Since the Vagrantfile is basically Ruby language and -I admit- I’m not really a Ruby geek, I have been using a JSON file to provide the configuration details of my servers. I then parsed this JSON file in the Vagrantfile.

The JSON file is shown below. Essentially this file provides the necessary details for AWS, such as the keys, the region and availability zone you would want the servers to reside in, but also the instance details including the type and some tags). Really there is nothing more to it, just a rather simple JSON file.

{
    "access_key": "",
    "secret_key": "",
    "region": "eu-west-1",
    "availability_zone": "a",
    "security_groups": "sg-c47cd5a0",
    "keypair_name": "",
    "ssh_username": "ec2-user",
    "ssh_private_key_path": "",
    "subnet_id" : "subnet-05f39e5c",
    "instances": {
        "instance-001": {
	    "ami_id": "ami-69b9941e",
            "instance_type": "t2.small",
	    "tags": {
	       "Name": "Instance-001",
               "Role": "webserver"
            }
        },
        "instance-002": {
            "ami_id": "ami-69b9941e",
            "instance_type": "t2.medium",
            "tags": { 
               "Name": "Instance-002",
               "Role": "webserver"  
            }
	},
        "instance-003": {
            "ami_id": "ami-25158352",
            "instance_type": "t2.small",
            "tags": { 
               "Name": "Instance-003",
               "Role": "database"  
            }        
	}
    }
}

As mentioned before, the Vagrantfile would then be used to parse the JSON file. This happens immediately in the first line. We stored everything in the aws_config variable and use this variable to provide the information to the config2.vm.provider which is needed by Vagrant. The complete Vagrant file can be found below.

aws_config = (JSON.parse(File.read("aws-multiple.json")))

Vagrant.configure("2") do |config|

    config.vm.box = "dummy"
    config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"

    aws_config['instances'].each do |instance|
        instance_key   = instance[0]
        instance_value = instance[1]

        config.vm.define instance_key do |config2|
	    ec2_tags = instance_value['tags']

            config2.vm.provider :aws do |ec2, override|
                ec2.access_key_id = aws_config['access_key']
                ec2.secret_access_key = aws_config['secret_key']
                ec2.keypair_name = aws_config['keypair_name']
                ec2.region = aws_config['region']
                ec2.availability_zone = aws_config['region']+aws_config['availability_zone']
                ec2.subnet_id = aws_config['subnet_id']
                ec2.security_groups = aws_config['security_groups']
                ec2.ami = instance_value['ami_id']
                ec2.instance_type = instance_value['instance_type']
                ec2.tags = {
			'Name' => ec2_tags['Name'],
			'Role' => ec2_tags['Role']
		}
		override.ssh.username = aws_config['ssh_username']
                override.ssh.private_key_path = aws_config['ssh_private_key_path']
            end

        end
    end
end

Once we have the JSON file and the Vagrantfile in our directory, you can run ‘vagrant up’ and if all goes well, you will see multiple machines being launched on AWS. Obviously, as in previous Vagrant related posts, one can also SSH in using the SSH keys as provided in the JSON file.

macbookpro:aws-multiple wim$ sudo vagrant up
Bringing machine 'instance-001' up with 'aws' provider...
Bringing machine 'instance-002' up with 'aws' provider...
Bringing machine 'instance-003' up with 'aws' provider...
==> instance-002: Warning! The AWS provider doesn't support any of the Vagrant
==> instance-002: high-level network configurations (`config.vm.network`). They
==> instance-002: will be silently ignored.
==> instance-002: Warning! You're launching this instance into a VPC without an
==> instance-002: elastic IP. Please verify you're properly connected to a VPN so
==> instance-002: you can access this machine, otherwise Vagrant will not be able
==> instance-002: to SSH into it.
==> instance-002: Launching an instance with the following settings...
==> instance-002:  -- Type: t2.medium
==> instance-003: Warning! The AWS provider doesn't support any of the Vagrant
==> instance-003: high-level network configurations (`config.vm.network`). They
==> instance-003: will be silently ignored.
==> instance-002:  -- AMI: ami-69b9941e
==> instance-003: Warning! You're launching this instance into a VPC without an
==> instance-003: elastic IP. Please verify you're properly connected to a VPN so
==> instance-003: you can access this machine, otherwise Vagrant will not be able
==> instance-003: to SSH into it.
==> instance-003: Launching an instance with the following settings...
==> instance-002:  -- Region: eu-west-1
==> instance-002:  -- Availability Zone: eu-west-1a
==> instance-002:  -- Keypair: Keypair_Amazon_wauterw_Ireland
==> instance-002:  -- Subnet ID: subnet-05f39e5c
==> instance-001: Warning! The AWS provider doesn't support any of the Vagrant
==> instance-001: high-level network configurations (`config.vm.network`). They
==> instance-001: will be silently ignored.
==> instance-003:  -- Type: t2.small
==> instance-001: Warning! You're launching this instance into a VPC without an
==> instance-001: elastic IP. Please verify you're properly connected to a VPN so
==> instance-001: you can access this machine, otherwise Vagrant will not be able
==> instance-001: to SSH into it.
==> instance-002:  -- Security Groups: ["sg-c47cd5a0"]
==> instance-003:  -- AMI: ami-25158352
==> instance-001: Launching an instance with the following settings...
==> instance-002:  -- Block Device Mapping: []
==> instance-002:  -- Terminate On Shutdown: false
==> instance-003:  -- Region: eu-west-1
==> instance-002:  -- Monitoring: false
==> instance-001:  -- Type: t2.small
==> instance-003:  -- Availability Zone: eu-west-1a
==> instance-002:  -- EBS optimized: false
==> instance-003:  -- Keypair: Keypair_Amazon_wauterw_Ireland
==> instance-001:  -- AMI: ami-69b9941e
==> instance-003:  -- Subnet ID: subnet-05f39e5c
==> instance-002:  -- Assigning a public IP address in a VPC: false
==> instance-001:  -- Region: eu-west-1
==> instance-003:  -- Security Groups: ["sg-c47cd5a0"]
==> instance-001:  -- Availability Zone: eu-west-1a
==> instance-003:  -- Block Device Mapping: []
==> instance-003:  -- Terminate On Shutdown: false
==> instance-003:  -- Monitoring: false
==> instance-001:  -- Keypair: Keypair_Amazon_wauterw_Ireland
==> instance-001:  -- Subnet ID: subnet-05f39e5c
==> instance-001:  -- Security Groups: ["sg-c47cd5a0"]
==> instance-001:  -- Block Device Mapping: []
==> instance-003:  -- EBS optimized: false
==> instance-001:  -- Terminate On Shutdown: false
==> instance-003:  -- Assigning a public IP address in a VPC: false
==> instance-001:  -- Monitoring: false
==> instance-001:  -- EBS optimized: false
==> instance-001:  -- Assigning a public IP address in a VPC: false
==> instance-001: Warning! Vagrant might not be able to SSH into the instance.
==> instance-001: Please check your security groups settings.
==> instance-002: Warning! Vagrant might not be able to SSH into the instance.
==> instance-002: Please check your security groups settings.
==> instance-003: Warning! Vagrant might not be able to SSH into the instance.
==> instance-003: Please check your security groups settings.
==> instance-003: Waiting for instance to become "ready"...
==> instance-002: Waiting for instance to become "ready"...
==> instance-001: Waiting for instance to become "ready"...
==> instance-003: Waiting for SSH to become available...
==> instance-002: Waiting for SSH to become available...
==> instance-001: Waiting for SSH to become available...
==> instance-002: Machine is booted and ready for use!
==> instance-003: Machine is booted and ready for use!
==> instance-001: Machine is booted and ready for use!

In case you wouldn’t believe this piece of magic, see the result in below screenshot 🙂

AWS_Vagrant_multipleinstances

Vagrant: AWS (Launch single instance)

Introduction

In this post, we will get Vagrant working together with AWS instead of Virtualbox (as described here) or DigitalOcean (as described here). As in previous posts, we will use a shell script to install Apache on the EC2 instance automatically and run it as a webserver.

Installing AWS plugin

In order for Vagrant to work together with AWS, we need to install a plugin. The plugin can be found here

macbookpro:aws wim$ vagrant plugin install vagrant-aws
Installing the 'vagrant-aws' plugin. This can take a few minutes...
Installed the plugin 'vagrant-aws (0.6.0)'!

Downloading dummy box

macbookpro:aws wim$ vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'dummy' (v0) for provider: 
    box: Downloading: https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
==> box: Successfully added box 'dummy' (v0) for 'aws'!

Creating Vagrantfile

Create a Vagrant file similar to the snippet below. Fill in your specific AWS credentials.

Vagrant.configure("2") do |config|
  config.vm.box = "dummy"

  config.vm.provider :aws do |aws, override|
    aws.access_key_id = "YOUR KEY"
    aws.secret_access_key = "YOUR SECRET KEY"
    aws.keypair_name = "KEYPAIR NAME"
    aws.security_groups = "default"
    aws.ami = "ami-7747d01e"

    override.ssh.username = "ec2-user"
    override.ssh.private_key_path = "PATH TO YOUR PRIVATE KEY ON LOCAL PC"
  end
end 

The access_key_id and secret_access_key can be found on your AWS console dashboard. There are a couple of ways to do this. I have used IAM to create a user and group and used these credentials. Most people by nature are lazy and they will use the root keys. These can be created and found by logging into your dashboard, click on your name (right corner), go to ‘Security Credentials’ and then open the ‘Access keys’ panel.

The keypair can be found in the EC2 dashboard and go to ‘Keypairs’. In case you have not created them, you can create them here and then download them to your local PC. The details can then be added to the Vagrantfile.

The security groups contains the list of all security group name;s you would want to use. To find your security group name, go to the EC2 dashboard and click security groups. Note that you should select a security group that allows SSH from your IP address. While you could set the ‘source’ for the rules to ‘anywhere’ (which means everybody can get access to the TCP or UDP port under discussion), it’s much safer to set it to ‘My IP’ which prevents random people to connect to your instance in case they should obtain the keypair.

The SSH username for the default Ubuntu images would be ‘ec2-user’. Note that if you use a different AMI, also the username could be different.

The AMI file is free to choose. I usually use an Ubuntu server. The easiest way to see what images are available is to launch an EC2 instance using the dashboard and it will present you with a list of available images. Also the AMI id is shown there.

When doing a simple ‘vagrant up’, you will see that an instance gets launched in the AWS dashboard. It will move from pending to ready state.

EC2_Vagrant

You could also see on your local PC’s console:

macbookpro:aws wim$ sudo vagrant up --provide=aws
Bringing machine 'default' up with 'aws' provider...
==> default: Warning! The AWS provider doesn't support any of the Vagrant
==> default: high-level network configurations (`config.vm.network`). They
==> default: will be silently ignored.
==> default: Launching an instance with the following settings...
==> default:  -- Type: m3.medium
==> default:  -- AMI: ami-69b9941e
==> default:  -- Region: eu-west-1
==> default:  -- Keypair: 
==> default:  -- Block Device Mapping: []
==> default:  -- Terminate On Shutdown: false
==> default:  -- Monitoring: false
==> default:  -- EBS optimized: false
==> default:  -- Assigning a public IP address in a VPC: false
==> default: Waiting for instance to become "ready"...
==> default: Waiting for SSH to become available...
==> default: Machine is booted and ready for use!

All set. In order to get access to the server, just do ‘vagrant ssh’ and you will see the something as follows:

Last login: Fri Oct 23 10:45:36 2015 from 173.38.220.41

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/
No packages needed for security; 21 packages available
Run "sudo yum update" to apply all updates.
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[ec2-user@ip-10-11-158-86 ~]$ 

Vagrant: Provisioning with shell script

Introduction

In this post, we described the very basics of Vagrant. In this post, we’ll explore some more advanced concepts. We will boot a virtual machine on top of VirtualBox (we described Digital Ocean in this post in case you would want to try it out using DigitalOcean) and install Apache immediately onto it. Automatically that is, not manually. Let’s go!

Provisioning

Provisioning is generally referring to the installation and configuration of software on your virtual machines in an automated way, e.g.using provisioning tools (shell, Puppet, Chef, Ansible…). For this tutorial, we will use shell script, just get familiar with the concept.

Change your vagrantfile to look like the below snippet. If you are unfamiliar with this, please refer to thispost.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "precise64"
  config.vm.hostname = "wim"
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"
  config.vm.network "forwarded_port", guest: 80, host: 8080
  config.vm.provider "virtualbox" do |vb|
      vb.name ="VM-Test"
  end
  config.vm.provision "shell", path: "provision.sh"
end

As you can see, we are referring to a ‘provisioning.sh’ file. This looks as follows (note: this is tune for Ubuntu as the base image for vagrant is based on Ubuntu)

#!/usr/bin/env bash

echo "Installing Apache ..."
apt-get update >/dev/null 2>&1
apt-get install -y apache2 >/dev/null 2>&1
rm -rf /var/www
ln -fs /vagrant /var/www

We will also put an index.html file in our directory that just echoes the word ‘Index’, or whatever you prefer.

On your CLI, then run the ‘vagrant up’ command

macbook-pro:VagrantBookExample wim$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: VM-Test
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 80 => 8080 (adapter 1)
    default: 22 => 2222 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
 ....
==> default: stdin: is not a tty
==> default: Installing Apache and setting it up...

When you then go to http://localhost:8080, you will see a webpage displaying the word ‘Index’. Don’t believe me…check out for yourself!

index

Vagrant and DigitalOcean (APIv2)

Introduction

In this tutorial we learned to very basics of Vagrant. Essentially, it was showing how to create virtual machines on top of VirtualBox. In today’s article, I wanted to show you how to do this with DigitalOcean. In other words, we will create a recipe to create a virtual machine running on DigitalOcean.

Installing the DigitalOcean plugin

Vagrant does not provide support for DigitalOcean out of the box, but is flexible enough for companies to develop their own plugins. Let’s install the Digitalocean plugin.

macbook-wim:Vagrant wim$ vagrant plugin install vagrant-digitalocean
Installing the 'vagrant-digitalocean' plugin. This can take a few minutes...
Installed the plugin 'vagrant-digitalocean (0.7.7)'!

Once this is done, we need to add the DigitalOcean box (the “image”) to vagrant.

macbook-wim:Vagrant wim$ vagrant box add digital_ocean https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'digital_ocean' (v0) for provider: 
    box: Downloading: https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box
==> box: Successfully added box 'digital_ocean' (v0) for 'digital_ocean'!

Go ahead and create a project folder, perform a ‘vagrant init’ (as lined out in the initial article ).

Before continuing, we need to create an SSH keypaor for authentication with DigitalOcean. Run the following command on your host station:

macbook-wim:digitalocean wim$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/wim/.ssh/id_rsa): 
/Users/wim/.ssh/id_rsa already exists.
....

In order to find your token, login to your DigitalOcean account. Go to ‘API’, ‘Your tokens’ and click ‘Generate new token’ as indicated in the screenshot below:

DO_Token

Important note: Be careful with the token!! Anyone who has it can generate droplets on your DO account.

Your Vagrantfile should look as follows as per the instruction on GitHub for this plugin. Obviously, you are free to choose your image, region, size etc…

# -*- mode: ruby -*-
Vagrant.configure('2') do |config|

  config.vm.provider :digital_ocean do |provider, override|
    override.ssh.private_key_path = '~/.ssh/id_rsa'
    override.vm.box = 'digital_ocean'
    override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box"

    provider.token = ''
    provider.image = 'ubuntu-14-04-x64'
    provider.region = 'ams3'
    provider.size = '512mb'
  end
end

Then launch the virtual machine (DigitalOcean calls them droplets) using the following command:

macbook-wim:digitalocean wim$ vagrant up
Bringing machine 'default' up with 'digital_ocean' provider...
==> default: Using existing SSH key: Vagrant
==> default: Creating a new droplet...
==> default: Assigned IP address: 188.166.74.154
==> default: Rsyncing folder: /Users/wim/Dropbox/Technologies/Vagrant/digitalocean/ => /vagrant...

And here it is….fully up and running.
vagrant_do
Note: I was initially using ams1, but I got a note (when running ‘vagrant up’) that ams1 was not available. Indeed, when checking on the DO web app, ams1 is greyed out.

You should now be able to ssh into the droplet that was just created by using the ‘vagrant ssh’ command.