Monthly Archives: December 2019

Create VM on vSphere and network on ACI (using Terraform)

Introduction

In past couple of posts, we have been experimenting with Terraform in combination with ACI (see here) and vSphere (see here) seperately. In this post, we will combine both and we will first create network constructs on Cisco’s ACI solution and once done, we will create two VM instances on vSphere that are actually using the underlying ACI network constructs.

ACI configuration

A lot has already been discussed in this post here, so I won’t re-iterate all of that here and instead focus on the new items in the Terraform file.

In below snippet, you can see we create a tenant, VRF and BD. The BD will use a subnet of “10.16.100.1/24”. The VMs we sill create in a later phase will hence be using an IP address from that subnet. Furthermore, we will create an Application Profile and two EPGs into that Application Profile. Each EPG will contain a VM. For demonstration purposes, we have also defined a contract between those EPGs. The contracts will allow HTTPS and ICMP traffic (refer to the aci_filter_entry resources). You will see in the ‘aci_application_epg’ resources that we link the contract (aci_contract.Contract_EPG1_EPG2.name) to the application EPGs (resource aci_application_epg).

The full content of the main_aci.tf file is:

provider "aci" {
  username = var.apic_username
  password = var.apic_password
  url      = var.apic_server
  insecure = true
}

resource "aci_tenant" "Tenant_TF" {
  name        = var.aci_tenant
  description = "Tenant created by TF"
}

resource "aci_vrf" "VRF_TF" {
  tenant_dn          = aci_tenant.Tenant_TF.id
  name               = var.aci_vrf
  description        = "VRF created by TF"
  bd_enforced_enable = false
}

resource "aci_bridge_domain" "BD_TF" {
  tenant_dn          = aci_tenant.Tenant_TF.id
  name               = var.aci_bd
  description        = "BD created by TF"
  relation_fv_rs_ctx = aci_vrf.VRF_TF.name
}

resource "aci_subnet" "Subnet_BD" {
  bridge_domain_dn = aci_bridge_domain.BD_TF.id
  ip = var.aci_bd_subnet
}

resource "aci_application_profile" "AppProfile_TF" {
  tenant_dn   = aci_tenant.Tenant_TF.id
  name        = var.aci_app_profile
  description = "App profile created by TF"
}

resource "aci_application_epg" "EPG_TF_1" {
  application_profile_dn = aci_application_profile.AppProfile_TF.id
  name                   = var.aci_epg_1
  description            = "EPG created by TF"
  relation_fv_rs_bd      = aci_bridge_domain.BD_TF.name
  relation_fv_rs_dom_att = [var.vmm_domain]
  relation_fv_rs_cons    = [aci_contract.Contract_EPG1_EPG2.name]
}

resource "aci_application_epg" "EPG_TF_2" {
  application_profile_dn = aci_application_profile.AppProfile_TF.id
  name                   = var.aci_epg_2
  description            = "EPG created by TF"
  relation_fv_rs_bd      = aci_bridge_domain.BD_TF.name
  relation_fv_rs_dom_att = [var.vmm_domain]
  relation_fv_rs_prov    = [aci_contract.Contract_EPG1_EPG2.name]
}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "sleep 5"
  }

  triggers = {
    "epg1" = aci_application_epg.EPG_TF_1.id
    "epg2" = aci_application_epg.EPG_TF_2.id
  }
}

resource "aci_contract" "Contract_EPG1_EPG2" {
  tenant_dn   = aci_tenant.Tenant_TF.id
  name        = "Contract_EPG1_EPG2"
  description = "Contract created by TF"
}

resource "aci_contract_subject" "Contract_Subject" {
  contract_dn                  = aci_contract.Contract_EPG1_EPG2.id
  name                         = var.aci_contract_subject
  relation_vz_rs_subj_filt_att = [aci_filter.Filter_Allow_HTTPS.name, aci_filter.Filter_Allow_ICMP.name]
}

resource "aci_filter" "Filter_Allow_HTTPS" {
  tenant_dn = aci_tenant.Tenant_TF.id
  name      = var.aci_filter_allow_https
}

resource "aci_filter" "Filter_Allow_ICMP" {
  tenant_dn = aci_tenant.Tenant_TF.id
  name      = var.aci_filter_allow_icmp
}

resource "aci_filter_entry" "Filter_Entry_HTTPS" {
  name        = var.aci_filter_entry_https
  filter_dn   = aci_filter.Filter_Allow_HTTPS.id
  ether_t     = "ip"
  prot        = "tcp"
  d_from_port = "https"
  d_to_port   = "https"
  stateful    = "yes"
}

resource "aci_filter_entry" "Filter_Entry_ICMP" {
  name      = var.aci_filter_entry_icmp
  filter_dn = aci_filter.Filter_Allow_ICMP.id
  ether_t   = "ip"
  prot      = "icmp"
  stateful  = "yes"
}

We also have to define the following variables in the variables_aci.tf file.

variable "apic_server" {
  default = "https://10.16.2.1"
}

variable "apic_username" {
  default = "admin"
}

variable "apic_password" {
  default = "***"
}

variable "aci_tenant" {
  default = "Tenant_Terraform_ACI_Vsphere_demo"
}

variable "aci_vrf" {
  default = "VRF_TF"
}

variable "aci_bd" {
  default = "BD_TF"
}

variable "aci_bd_subnet" {
  default = "10.16.100.1/24"
}

variable "aci_app_profile" {
  default = "AppProfile_TF"
}

variable "aci_epg_1" {
  default = "EPG_TF_1"
}

variable "aci_epg_2" {
  default = "EPG_TF_2"
}

variable "aci_contract_subject" {
  default = "Subject_TF"
}

variable "aci_filter_allow_https" {
  default = "allow_https"
}

variable "aci_filter_allow_icmp" {
  default = "allow_icmp"
}

variable "aci_filter_entry_https" {
  default = "https"
}

variable "aci_filter_entry_icmp" {
  default = "icmp"
}

variable "provider_profile_dn" {
  default = "uni/vmmp-VMware"
}

variable "vmm_domain" {
  default = "uni/vmmp-VMware/dom-dvs-demo-dynamic"
}

vSphere configuration

Next, we will focus on the vSphere portion. Again, a lot has been discussed in this post. There are however some neat things to link the ACI network constructs together with the vSphere part. I will focus on these.

You will notice in below snippet that we are using quite some Terraform data sources. Data sources allow data to be fetched from your system to be used elsewhere in the Terraform code. For vSphere, we will read information from our system such as the datacenter, the datastore, the cluster, the template to be used. All this because we will use it elsewhere to tell Terraform where it needs to create these resources.

One thing to point out, and potentially a little more complicated, is the data source for the network. ACI uses the concept of a VMM domain (Virtual Machine Manager). It is a constuct to bind the ACI network constructs to the VMware network constructs. Upon creation, ACI will create a port group on VMWare’s DVS and it will call this portgroup according to the following name format: ||. In the datasource ‘data “vsphere_network”‘, you will see that we are retrieving the network from VMWare using that name format.

For the rest, it’s business as usual, we will just create two instances on vSphere and tell it to use that portgroup ( network_id = data.vsphere_network.vm1_net.id)

provider "vsphere" {
  user                 = var.vsphere_user
  password             = var.vsphere_password
  vsphere_server       = var.vsphere_server
  allow_unverified_ssl = true
}

data "vsphere_datacenter" "dc" {
  name = var.vsphere_datacenter
}

data "vsphere_network" "vm1_net" {
  depends_on = [null_resource.delay]
  name = format(
    "%v|%v|%v",
    aci_tenant.Tenant_TF.name,
    aci_application_profile.AppProfile_TF.name,
    aci_application_epg.EPG_TF_1.name,
  )
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_network" "vm2_net" {
  depends_on = [null_resource.delay]
  name = format(
    "%v|%v|%v",
    aci_tenant.Tenant_TF.name,
    aci_application_profile.AppProfile_TF.name,
    aci_application_epg.EPG_TF_2.name,
  )
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_datastore" "ds" {
  name          = var.vsphere_datastore
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_compute_cluster" "cl" {
  name          = var.vsphere_compute_cluster
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_virtual_machine" "template" {
  name          = var.vsphere_template
  datacenter_id = data.vsphere_datacenter.dc.id
}

resource "vsphere_virtual_machine" "aci_vm1" {
  count            = 1
  name             = var.aci_vm1_name
  resource_pool_id = data.vsphere_compute_cluster.cl.resource_pool_id
  datastore_id     = data.vsphere_datastore.ds.id

  num_cpus  = 2
  memory    = 4096
  guest_id  = data.vsphere_virtual_machine.template.guest_id
  scsi_type = data.vsphere_virtual_machine.template.scsi_type

  disk {
    label            = "disk0"
    size             = data.vsphere_virtual_machine.template.disks[0].size
    thin_provisioned = data.vsphere_virtual_machine.template.disks[0].thin_provisioned
  }

  folder = var.folder

  network_interface {
    network_id   = data.vsphere_network.vm1_net.id
    adapter_type = data.vsphere_virtual_machine.template.network_interface_types[0]
  }

  clone {
    linked_clone  = "false"
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {
      linux_options {
        host_name = var.aci_vm1_name
        domain    = var.domain_name
      }

      network_interface {
        ipv4_address = var.aci_vm1_address
        ipv4_netmask = "24"
      }

      ipv4_gateway    = var.gateway
      dns_server_list = [var.dns_list]
      dns_suffix_list = [var.dns_search]
    }
  }
}

resource "vsphere_virtual_machine" "aci_vm2" {
  count            = 1
  name             = var.aci_vm2_name
  resource_pool_id = data.vsphere_compute_cluster.cl.resource_pool_id
  datastore_id     = data.vsphere_datastore.ds.id

  num_cpus = 2
  memory   = 4096
  guest_id = data.vsphere_virtual_machine.template.guest_id

  scsi_type = data.vsphere_virtual_machine.template.scsi_type

  disk {
    label            = "disk0"
    size             = data.vsphere_virtual_machine.template.disks[0].size
    thin_provisioned = data.vsphere_virtual_machine.template.disks[0].thin_provisioned
  }

  folder = var.folder

  network_interface {
    network_id   = data.vsphere_network.vm2_net.id
    adapter_type = data.vsphere_virtual_machine.template.network_interface_types[0]
  }

  clone {
    linked_clone  = "false"
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {
      linux_options {
        host_name = var.aci_vm2_name
        domain    = var.domain_name
      }

      network_interface {
        ipv4_address = var.aci_vm2_address
        ipv4_netmask = "24"
      }

      ipv4_gateway    = var.gateway
      dns_server_list = [var.dns_list]
      dns_suffix_list = [var.dns_search]
    }
  }
}

In order for everything to work, you will of course need to specify some variables.

variable "vsphere_server" {
  default = "10.x.y.z"
}

variable "vsphere_user" {
  default = "administrator@vsphere.local"
}

variable "vsphere_password" {
  default = "***"
}

variable "vsphere_datacenter" {
  default = "SaS-DC"
}

variable "vsphere_datastore" {
  default = "datastore-UCS-POD1-2"
}

variable "vsphere_compute_cluster" {
  default = "SaS-Cluster"
}

variable "vsphere_template" {
  default = "ubuntu-1604-server-template"
}

variable "folder" {
  default = "wauterw"
}

variable "aci_vm1_name" {
  default = "ACI1-ACI"
}

variable "aci_vm2_name" {
  default = "ACI2-ACI"
}

variable "aci_vm1_address" {
  default = "10.16.100.10"
}

variable "aci_vm2_address" {
  default = "10.16.100.11"
}

variable "gateway" {
  default = "10.16.100.1"
}

variable "dns_list" {
  default = "10.9.15.1"
}

variable "dns_search" {
  default = "cisco.com"
}

variable "domain_name" {
  type    = string
  default = "cisco.com"
}

Deploying ACI and vSphere

Let’s run the ‘terraform init’ command. You will see that we are downloading the required providers.

cisco@wauterw-ubuntu-desktop:~/software/Terraform/ACI-vSphere$ terraform init

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aci: version = "~> 0.1"
* provider.null: version = "~> 2.1"
* provider.vsphere: version = "~> 1.13"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Next, let’s take a look at the plan. You will see that 16 resources will be created (both for VMware as for ACI).

cisco@wauterw-ubuntu-desktop:~/software/Terraform/ACI-vSphere$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_compute_cluster.cl: Refreshing state...
data.vsphere_datastore.ds: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.vsphere_network.vm1_net will be read during apply
  # (config refers to values not yet known)
 <= data "vsphere_network" "vm1_net"  {
      + datacenter_id = "datacenter-2"
      + id            = (known after apply)
      + name          = "Tenant_Terraform_ACI_Vsphere_demo|AppProfile_TF|EPG_TF_1"
      + type          = (known after apply)
    }

  # data.vsphere_network.vm2_net will be read during apply
  # (config refers to values not yet known)
 <= data "vsphere_network" "vm2_net"  {
      + datacenter_id = "datacenter-2"
      + id            = (known after apply)
      + name          = "Tenant_Terraform_ACI_Vsphere_demo|AppProfile_TF|EPG_TF_2"
      + type          = (known after apply)
    }

  # aci_application_epg.EPG_TF_1 will be created
  + resource "aci_application_epg" "EPG_TF_1" {
      + annotation             = (known after apply)
      + application_profile_dn = (known after apply)
      + description            = "EPG created by TF"
      + exception_tag          = (known after apply)
      + flood_on_encap         = (known after apply)
      + fwd_ctrl               = (known after apply)
      + has_mcast_source       = (known after apply)
      + id                     = (known after apply)
      + is_attr_based_e_pg     = (known after apply)
      + match_t                = (known after apply)
      + name                   = "EPG_TF_1"
      + name_alias             = (known after apply)
      + pc_enf_pref            = (known after apply)
      + pref_gr_memb           = (known after apply)
      + prio                   = (known after apply)
      + relation_fv_rs_bd      = "BD_TF"
      + relation_fv_rs_cons    = [
          + "Contract_EPG1_EPG2",
        ]
      + relation_fv_rs_dom_att = [
          + "uni/vmmp-VMware/dom-dvs-demo-dynamic",
        ]
      + shutdown               = (known after apply)
    }

  # aci_application_epg.EPG_TF_2 will be created
  + resource "aci_application_epg" "EPG_TF_2" {
      + annotation             = (known after apply)
      + application_profile_dn = (known after apply)
      + description            = "EPG created by TF"
      + exception_tag          = (known after apply)
      + flood_on_encap         = (known after apply)
      + fwd_ctrl               = (known after apply)
      + has_mcast_source       = (known after apply)
      + id                     = (known after apply)
      + is_attr_based_e_pg     = (known after apply)
      + match_t                = (known after apply)
      + name                   = "EPG_TF_2"
      + name_alias             = (known after apply)
      + pc_enf_pref            = (known after apply)
      + pref_gr_memb           = (known after apply)
      + prio                   = (known after apply)
      + relation_fv_rs_bd      = "BD_TF"
      + relation_fv_rs_dom_att = [
          + "uni/vmmp-VMware/dom-dvs-demo-dynamic",
        ]
      + relation_fv_rs_prov    = [
          + "Contract_EPG1_EPG2",
        ]
      + shutdown               = (known after apply)
    }

  # aci_application_profile.AppProfile_TF will be created
  + resource "aci_application_profile" "AppProfile_TF" {
      + annotation  = (known after apply)
      + description = "App profile created by TF"
      + id          = (known after apply)
      + name        = "AppProfile_TF"
      + name_alias  = (known after apply)
      + prio        = (known after apply)
      + tenant_dn   = (known after apply)
    }

  # aci_bridge_domain.BD_TF will be created
  + resource "aci_bridge_domain" "BD_TF" {
      + annotation                  = (known after apply)
      + arp_flood                   = (known after apply)
      + bridge_domain_type          = (known after apply)
      + description                 = "BD created by TF"
      + ep_clear                    = (known after apply)
      + ep_move_detect_mode         = (known after apply)
      + host_based_routing          = (known after apply)
      + id                          = (known after apply)
      + intersite_bum_traffic_allow = (known after apply)
      + intersite_l2_stretch        = (known after apply)
      + ip_learning                 = (known after apply)
      + ipv6_mcast_allow            = (known after apply)
      + limit_ip_learn_to_subnets   = (known after apply)
      + ll_addr                     = (known after apply)
      + mac                         = (known after apply)
      + mcast_allow                 = (known after apply)
      + multi_dst_pkt_act           = (known after apply)
      + name                        = "BD_TF"
      + name_alias                  = (known after apply)
      + optimize_wan_bandwidth      = (known after apply)
      + relation_fv_rs_ctx          = "VRF_TF"
      + tenant_dn                   = (known after apply)
      + unicast_route               = (known after apply)
      + unk_mac_ucast_act           = (known after apply)
      + unk_mcast_act               = (known after apply)
      + v6unk_mcast_act             = (known after apply)
      + vmac                        = (known after apply)
    }

  # aci_contract.Contract_EPG1_EPG2 will be created
  + resource "aci_contract" "Contract_EPG1_EPG2" {
      + annotation  = (known after apply)
      + description = "Contract created by TF"
      + id          = (known after apply)
      + name        = "Contract_EPG1_EPG2"
      + name_alias  = (known after apply)
      + prio        = (known after apply)
      + scope       = (known after apply)
      + target_dscp = (known after apply)
      + tenant_dn   = (known after apply)
    }

  # aci_contract_subject.Contract_Subject will be created
  + resource "aci_contract_subject" "Contract_Subject" {
      + annotation                   = (known after apply)
      + cons_match_t                 = (known after apply)
      + contract_dn                  = (known after apply)
      + id                           = (known after apply)
      + name                         = "Subject_TF"
      + name_alias                   = (known after apply)
      + prio                         = (known after apply)
      + prov_match_t                 = (known after apply)
      + relation_vz_rs_subj_filt_att = [
          + "allow_https",
          + "allow_icmp",
        ]
      + rev_flt_ports                = (known after apply)
      + target_dscp                  = (known after apply)
    }

  # aci_filter.Filter_Allow_HTTPS will be created
  + resource "aci_filter" "Filter_Allow_HTTPS" {
      + annotation = (known after apply)
      + id         = (known after apply)
      + name       = "allow_https"
      + name_alias = (known after apply)
      + tenant_dn  = (known after apply)
    }

  # aci_filter.Filter_Allow_ICMP will be created
  + resource "aci_filter" "Filter_Allow_ICMP" {
      + annotation = (known after apply)
      + id         = (known after apply)
      + name       = "allow_icmp"
      + name_alias = (known after apply)
      + tenant_dn  = (known after apply)
    }

  # aci_filter_entry.Filter_Entry_HTTPS will be created
  + resource "aci_filter_entry" "Filter_Entry_HTTPS" {
      + annotation    = (known after apply)
      + apply_to_frag = (known after apply)
      + arp_opc       = (known after apply)
      + d_from_port   = "https"
      + d_to_port     = "https"
      + ether_t       = "ip"
      + filter_dn     = (known after apply)
      + icmpv4_t      = (known after apply)
      + icmpv6_t      = (known after apply)
      + id            = (known after apply)
      + match_dscp    = (known after apply)
      + name          = "https"
      + name_alias    = (known after apply)
      + prot          = "tcp"
      + s_from_port   = (known after apply)
      + s_to_port     = (known after apply)
      + stateful      = "yes"
      + tcp_rules     = (known after apply)
    }

  # aci_filter_entry.Filter_Entry_ICMP will be created
  + resource "aci_filter_entry" "Filter_Entry_ICMP" {
      + annotation    = (known after apply)
      + apply_to_frag = (known after apply)
      + arp_opc       = (known after apply)
      + d_from_port   = (known after apply)
      + d_to_port     = (known after apply)
      + ether_t       = "ip"
      + filter_dn     = (known after apply)
      + icmpv4_t      = (known after apply)
      + icmpv6_t      = (known after apply)
      + id            = (known after apply)
      + match_dscp    = (known after apply)
      + name          = "icmp"
      + name_alias    = (known after apply)
      + prot          = "icmp"
      + s_from_port   = (known after apply)
      + s_to_port     = (known after apply)
      + stateful      = "yes"
      + tcp_rules     = (known after apply)
    }

  # aci_subnet.Subnet_BD will be created
  + resource "aci_subnet" "Subnet_BD" {
      + annotation       = (known after apply)
      + bridge_domain_dn = (known after apply)
      + ctrl             = (known after apply)
      + id               = (known after apply)
      + ip               = "10.16.100.1/24"
      + name_alias       = (known after apply)
      + preferred        = (known after apply)
      + scope            = (known after apply)
      + virtual          = (known after apply)
    }

  # aci_tenant.Tenant_TF will be created
  + resource "aci_tenant" "Tenant_TF" {
      + annotation  = (known after apply)
      + description = "Tenant created by TF"
      + id          = (known after apply)
      + name        = "Tenant_Terraform_ACI_Vsphere_demo"
      + name_alias  = (known after apply)
    }

  # aci_vrf.VRF_TF will be created
  + resource "aci_vrf" "VRF_TF" {
      + annotation             = (known after apply)
      + bd_enforced_enable     = "false"
      + description            = "VRF created by TF"
      + id                     = (known after apply)
      + ip_data_plane_learning = (known after apply)
      + knw_mcast_act          = (known after apply)
      + name                   = "VRF_TF"
      + name_alias             = (known after apply)
      + pc_enf_dir             = (known after apply)
      + pc_enf_pref            = (known after apply)
      + tenant_dn              = (known after apply)
    }

  # null_resource.delay will be created
  + resource "null_resource" "delay" {
      + id       = (known after apply)
      + triggers = (known after apply)
    }

  # vsphere_virtual_machine.aci_vm1[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm1" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "ACI1-ACI"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.100.1"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "ACI1-ACI"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.100.10"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = (known after apply)
        }
    }

  # vsphere_virtual_machine.aci_vm2[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm2" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "ACI2-ACI"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.100.1"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "ACI2-ACI"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.100.11"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = (known after apply)
        }
    }

Plan: 16 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

And finally, let’s apply the configurations


cisco@wauterw-ubuntu-desktop:~/software/Terraform/ACI-vSphere$ terraform apply
data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_datastore.ds: Refreshing state...
data.vsphere_compute_cluster.cl: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.vsphere_network.vm1_net will be read during apply
  # (config refers to values not yet known)
 <= data "vsphere_network" "vm1_net"  {
      + datacenter_id = "datacenter-2"
      + id            = (known after apply)
      + name          = "Tenant_Terraform_ACI_Vsphere_demo|AppProfile_TF|EPG_TF_1"
      + type          = (known after apply)
    }

  # data.vsphere_network.vm2_net will be read during apply
  # (config refers to values not yet known)
 <= data "vsphere_network" "vm2_net"  {
      + datacenter_id = "datacenter-2"
      + id            = (known after apply)
      + name          = "Tenant_Terraform_ACI_Vsphere_demo|AppProfile_TF|EPG_TF_2"
      + type          = (known after apply)
    }

  # aci_application_epg.EPG_TF_1 will be created
  + resource "aci_application_epg" "EPG_TF_1" {
      + annotation             = (known after apply)
      + application_profile_dn = (known after apply)
      + description            = "EPG created by TF"
      + exception_tag          = (known after apply)
      + flood_on_encap         = (known after apply)
      + fwd_ctrl               = (known after apply)
      + has_mcast_source       = (known after apply)
      + id                     = (known after apply)
      + is_attr_based_e_pg     = (known after apply)
      + match_t                = (known after apply)
      + name                   = "EPG_TF_1"
      + name_alias             = (known after apply)
      + pc_enf_pref            = (known after apply)
      + pref_gr_memb           = (known after apply)
      + prio                   = (known after apply)
      + relation_fv_rs_bd      = "BD_TF"
      + relation_fv_rs_cons    = [
          + "Contract_EPG1_EPG2",
        ]
      + relation_fv_rs_dom_att = [
          + "uni/vmmp-VMware/dom-dvs-demo-dynamic",
        ]
      + shutdown               = (known after apply)
    }

  # aci_application_epg.EPG_TF_2 will be created
  + resource "aci_application_epg" "EPG_TF_2" {
      + annotation             = (known after apply)
      + application_profile_dn = (known after apply)
      + description            = "EPG created by TF"
      + exception_tag          = (known after apply)
      + flood_on_encap         = (known after apply)
      + fwd_ctrl               = (known after apply)
      + has_mcast_source       = (known after apply)
      + id                     = (known after apply)
      + is_attr_based_e_pg     = (known after apply)
      + match_t                = (known after apply)
      + name                   = "EPG_TF_2"
      + name_alias             = (known after apply)
      + pc_enf_pref            = (known after apply)
      + pref_gr_memb           = (known after apply)
      + prio                   = (known after apply)
      + relation_fv_rs_bd      = "BD_TF"
      + relation_fv_rs_dom_att = [
          + "uni/vmmp-VMware/dom-dvs-demo-dynamic",
        ]
      + relation_fv_rs_prov    = [
          + "Contract_EPG1_EPG2",
        ]
      + shutdown               = (known after apply)
    }

  # aci_application_profile.AppProfile_TF will be created
  + resource "aci_application_profile" "AppProfile_TF" {
      + annotation  = (known after apply)
      + description = "App profile created by TF"
      + id          = (known after apply)
      + name        = "AppProfile_TF"
      + name_alias  = (known after apply)
      + prio        = (known after apply)
      + tenant_dn   = (known after apply)
    }

  # aci_bridge_domain.BD_TF will be created
  + resource "aci_bridge_domain" "BD_TF" {
      + annotation                  = (known after apply)
      + arp_flood                   = (known after apply)
      + bridge_domain_type          = (known after apply)
      + description                 = "BD created by TF"
      + ep_clear                    = (known after apply)
      + ep_move_detect_mode         = (known after apply)
      + host_based_routing          = (known after apply)
      + id                          = (known after apply)
      + intersite_bum_traffic_allow = (known after apply)
      + intersite_l2_stretch        = (known after apply)
      + ip_learning                 = (known after apply)
      + ipv6_mcast_allow            = (known after apply)
      + limit_ip_learn_to_subnets   = (known after apply)
      + ll_addr                     = (known after apply)
      + mac                         = (known after apply)
      + mcast_allow                 = (known after apply)
      + multi_dst_pkt_act           = (known after apply)
      + name                        = "BD_TF"
      + name_alias                  = (known after apply)
      + optimize_wan_bandwidth      = (known after apply)
      + relation_fv_rs_ctx          = "VRF_TF"
      + tenant_dn                   = (known after apply)
      + unicast_route               = (known after apply)
      + unk_mac_ucast_act           = (known after apply)
      + unk_mcast_act               = (known after apply)
      + v6unk_mcast_act             = (known after apply)
      + vmac                        = (known after apply)
    }

  # aci_contract.Contract_EPG1_EPG2 will be created
  + resource "aci_contract" "Contract_EPG1_EPG2" {
      + annotation  = (known after apply)
      + description = "Contract created by TF"
      + id          = (known after apply)
      + name        = "Contract_EPG1_EPG2"
      + name_alias  = (known after apply)
      + prio        = (known after apply)
      + scope       = (known after apply)
      + target_dscp = (known after apply)
      + tenant_dn   = (known after apply)
    }

  # aci_contract_subject.Contract_Subject will be created
  + resource "aci_contract_subject" "Contract_Subject" {
      + annotation                   = (known after apply)
      + cons_match_t                 = (known after apply)
      + contract_dn                  = (known after apply)
      + id                           = (known after apply)
      + name                         = "Subject_TF"
      + name_alias                   = (known after apply)
      + prio                         = (known after apply)
      + prov_match_t                 = (known after apply)
      + relation_vz_rs_subj_filt_att = [
          + "allow_https",
          + "allow_icmp",
        ]
      + rev_flt_ports                = (known after apply)
      + target_dscp                  = (known after apply)
    }

  # aci_filter.Filter_Allow_HTTPS will be created
  + resource "aci_filter" "Filter_Allow_HTTPS" {
      + annotation = (known after apply)
      + id         = (known after apply)
      + name       = "allow_https"
      + name_alias = (known after apply)
      + tenant_dn  = (known after apply)
    }

  # aci_filter.Filter_Allow_ICMP will be created
  + resource "aci_filter" "Filter_Allow_ICMP" {
      + annotation = (known after apply)
      + id         = (known after apply)
      + name       = "allow_icmp"
      + name_alias = (known after apply)
      + tenant_dn  = (known after apply)
    }

  # aci_filter_entry.Filter_Entry_HTTPS will be created
  + resource "aci_filter_entry" "Filter_Entry_HTTPS" {
      + annotation    = (known after apply)
      + apply_to_frag = (known after apply)
      + arp_opc       = (known after apply)
      + d_from_port   = "https"
      + d_to_port     = "https"
      + ether_t       = "ip"
      + filter_dn     = (known after apply)
      + icmpv4_t      = (known after apply)
      + icmpv6_t      = (known after apply)
      + id            = (known after apply)
      + match_dscp    = (known after apply)
      + name          = "https"
      + name_alias    = (known after apply)
      + prot          = "tcp"
      + s_from_port   = (known after apply)
      + s_to_port     = (known after apply)
      + stateful      = "yes"
      + tcp_rules     = (known after apply)
    }

  # aci_filter_entry.Filter_Entry_ICMP will be created
  + resource "aci_filter_entry" "Filter_Entry_ICMP" {
      + annotation    = (known after apply)
      + apply_to_frag = (known after apply)
      + arp_opc       = (known after apply)
      + d_from_port   = (known after apply)
      + d_to_port     = (known after apply)
      + ether_t       = "ip"
      + filter_dn     = (known after apply)
      + icmpv4_t      = (known after apply)
      + icmpv6_t      = (known after apply)
      + id            = (known after apply)
      + match_dscp    = (known after apply)
      + name          = "icmp"
      + name_alias    = (known after apply)
      + prot          = "icmp"
      + s_from_port   = (known after apply)
      + s_to_port     = (known after apply)
      + stateful      = "yes"
      + tcp_rules     = (known after apply)
    }

  # aci_subnet.Subnet_BD will be created
  + resource "aci_subnet" "Subnet_BD" {
      + annotation       = (known after apply)
      + bridge_domain_dn = (known after apply)
      + ctrl             = (known after apply)
      + id               = (known after apply)
      + ip               = "10.16.100.1/24"
      + name_alias       = (known after apply)
      + preferred        = (known after apply)
      + scope            = (known after apply)
      + virtual          = (known after apply)
    }

  # aci_tenant.Tenant_TF will be created
  + resource "aci_tenant" "Tenant_TF" {
      + annotation  = (known after apply)
      + description = "Tenant created by TF"
      + id          = (known after apply)
      + name        = "Tenant_Terraform_ACI_Vsphere_demo"
      + name_alias  = (known after apply)
    }

  # aci_vrf.VRF_TF will be created
  + resource "aci_vrf" "VRF_TF" {
      + annotation             = (known after apply)
      + bd_enforced_enable     = "false"
      + description            = "VRF created by TF"
      + id                     = (known after apply)
      + ip_data_plane_learning = (known after apply)
      + knw_mcast_act          = (known after apply)
      + name                   = "VRF_TF"
      + name_alias             = (known after apply)
      + pc_enf_dir             = (known after apply)
      + pc_enf_pref            = (known after apply)
      + tenant_dn              = (known after apply)
    }

  # null_resource.delay will be created
  + resource "null_resource" "delay" {
      + id       = (known after apply)
      + triggers = (known after apply)
    }

  # vsphere_virtual_machine.aci_vm1[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm1" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "ACI1-ACI"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.100.1"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "ACI1-ACI"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.100.10"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = (known after apply)
        }
    }

  # vsphere_virtual_machine.aci_vm2[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm2" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "ACI2-ACI"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.100.1"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "ACI2-ACI"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.100.11"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = (known after apply)
        }
    }

Plan: 16 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aci_tenant.Tenant_TF: Creating...
aci_tenant.Tenant_TF: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo]
aci_contract.Contract_EPG1_EPG2: Creating...
aci_filter.Filter_Allow_HTTPS: Creating...
aci_application_profile.AppProfile_TF: Creating...
aci_filter.Filter_Allow_ICMP: Creating...
aci_vrf.VRF_TF: Creating...
aci_contract.Contract_EPG1_EPG2: Creation complete after 2s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/brc-Contract_EPG1_EPG2]
aci_application_profile.AppProfile_TF: Creation complete after 2s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/ap-AppProfile_TF]
aci_filter.Filter_Allow_HTTPS: Creation complete after 2s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/flt-allow_https]
aci_filter_entry.Filter_Entry_HTTPS: Creating...
aci_filter.Filter_Allow_ICMP: Creation complete after 2s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/flt-allow_icmp]
aci_contract_subject.Contract_Subject: Creating...
aci_filter_entry.Filter_Entry_ICMP: Creating...
aci_filter_entry.Filter_Entry_HTTPS: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/flt-allow_https/e-https]
aci_filter_entry.Filter_Entry_ICMP: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/flt-allow_icmp/e-icmp]
aci_contract_subject.Contract_Subject: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/brc-Contract_EPG1_EPG2/subj-Subject_TF]
aci_vrf.VRF_TF: Creation complete after 3s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/ctx-VRF_TF]
aci_bridge_domain.BD_TF: Creating...
aci_bridge_domain.BD_TF: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/BD-BD_TF]
aci_subnet.Subnet_BD: Creating...
aci_application_epg.EPG_TF_1: Creating...
aci_application_epg.EPG_TF_2: Creating...
aci_subnet.Subnet_BD: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/BD-BD_TF/subnet-[10.16.100.1/24]]
aci_application_epg.EPG_TF_2: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/ap-AppProfile_TF/epg-EPG_TF_2]
aci_application_epg.EPG_TF_1: Creation complete after 1s [id=uni/tn-Tenant_Terraform_ACI_Vsphere_demo/ap-AppProfile_TF/epg-EPG_TF_1]
null_resource.delay: Creating...
null_resource.delay: Provisioning with 'local-exec'...
null_resource.delay (local-exec): Executing: ["/bin/sh" "-c" "sleep 5"]
null_resource.delay: Creation complete after 5s [id=2493912136660243493]
data.vsphere_network.vm2_net: Refreshing state...
data.vsphere_network.vm1_net: Refreshing state...
vsphere_virtual_machine.aci_vm2[0]: Creating...
vsphere_virtual_machine.aci_vm1[0]: Creating...
vsphere_virtual_machine.aci_vm2[0]: Still creating... [10s elapsed]
vsphere_virtual_machine.aci_vm1[0]: Still creating... [10s elapsed]
vsphere_virtual_machine.aci_vm2[0]: Still creating... [6m10s elapsed]
vsphere_virtual_machine.aci_vm1[0]: Still creating... [6m10s elapsed]
vsphere_virtual_machine.aci_vm1[0]: Creation complete after 6m15s [id=423cfc62-5ffa-3078-fb9f-53836a9f6a22]
vsphere_virtual_machine.aci_vm2[0]: Still creating... [6m20s elapsed]
vsphere_virtual_machine.aci_vm2[0]: Creation complete after 6m25s [id=423c297e-9f95-6f53-6c49-2f4fc41e3e16]

Apply complete! Resources: 16 added, 0 changed, 0 destroyed.

Eventually, let’s see what this created on ACI and vSphere.

First, you will see that the ACI tenant got created and that it has a VRF, a BD and 2 EPG’s.

Drilling into the Tenant, you will see the two EPGS:

And here you will see the two EPGs with the contracts in between.

On vSphere, you will see our two VM instances:

And on vSphere you will see the following network configuration.

And because we have an ICMP contract between both EPGs, we can also verify if the ping works.

Create VM on vSphere with Terraform

Introduction

In this post and this post, we created respectively some EC2 instances on AWS and some network contructs on Cisco’s ACI solution.

We will apply the same principle but instead of creating some servers on AWS, we will create some servers on vSphere.

Terraform code

We will create a main.tf file which contains the entire configuration. It’s quite a lengthy file but it’s not really complicated to be honest.

First, we retrieve all values from our existing vSphere setup. Reason for doing this is that we will use them somewhat later in the file. For the server creation, we are essentially cloning a template, in my case a Ubuntu 16.04 server template which was created already before. We will also assign a static IP address (and DNS, domain…) to the servers.

provider "vsphere" {
  user                 = var.vsphere_user
  password             = var.vsphere_password
  vsphere_server       = var.vsphere_server
  allow_unverified_ssl = true
}

data "vsphere_datacenter" "dc" {
  name = var.vsphere_datacenter
}

data "vsphere_network" "vm1_net" {
  name          = "VM Network"
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_network" "vm2_net" {
  name          = "VM Network"
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_datastore" "ds" {
  name          = var.vsphere_datastore
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_compute_cluster" "cl" {
  name          = var.vsphere_compute_cluster
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_virtual_machine" "template" {
  name          = var.vsphere_template
  datacenter_id = data.vsphere_datacenter.dc.id
}

resource "vsphere_virtual_machine" "aci_vm1" {
  count            = 1
  name             = var.aci_vm1_name
  resource_pool_id = data.vsphere_compute_cluster.cl.resource_pool_id
  datastore_id     = data.vsphere_datastore.ds.id

  num_cpus  = 2
  memory    = 4096
  guest_id  = data.vsphere_virtual_machine.template.guest_id
  scsi_type = data.vsphere_virtual_machine.template.scsi_type

  disk {
    label            = "disk0"
    size             = data.vsphere_virtual_machine.template.disks[0].size
    thin_provisioned = data.vsphere_virtual_machine.template.disks[0].thin_provisioned
  }

  folder = var.folder

  network_interface {
    network_id   = data.vsphere_network.vm1_net.id
    adapter_type = data.vsphere_virtual_machine.template.network_interface_types[0]
  }

  clone {
    linked_clone  = "false"
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {
      linux_options {
        host_name = var.aci_vm1_name
        domain    = var.domain_name
      }

      network_interface {
        ipv4_address = var.aci_vm1_address
        ipv4_netmask = "24"
      }

      ipv4_gateway    = var.gateway
      dns_server_list = [var.dns_list]
      dns_suffix_list = [var.dns_search]
    }
  }
}

resource "vsphere_virtual_machine" "aci_vm2" {
  count            = 1
  name             = var.aci_vm2_name
  resource_pool_id = data.vsphere_compute_cluster.cl.resource_pool_id
  datastore_id     = data.vsphere_datastore.ds.id

  num_cpus = 2
  memory   = 4096
  guest_id = data.vsphere_virtual_machine.template.guest_id

  scsi_type = data.vsphere_virtual_machine.template.scsi_type

  disk {
    label            = "disk0"
    size             = data.vsphere_virtual_machine.template.disks[0].size
    thin_provisioned = data.vsphere_virtual_machine.template.disks[0].thin_provisioned
  }

  folder = var.folder

  network_interface {
    network_id   = data.vsphere_network.vm2_net.id
    adapter_type = data.vsphere_virtual_machine.template.network_interface_types[0]
  }

  clone {
    linked_clone  = "false"
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {
      linux_options {
        host_name = var.aci_vm2_name
        domain    = var.domain_name
      }

      network_interface {
        ipv4_address = var.aci_vm2_address
        ipv4_netmask = "24"
      }

      ipv4_gateway    = var.gateway
      dns_server_list = [var.dns_list]
      dns_suffix_list = [var.dns_search]
    }
  }
}

The corresponding variables file can be found below.

variable "vsphere_server" {
  default = "10.x.y.z"
}

variable "vsphere_user" {
  default = "administrator@vsphere.local"
}

variable "vsphere_password" {
  default = "****!"
}

variable "vsphere_datacenter" {
  default = "SaS-DC"
}

variable "vsphere_datastore" {
  default = "datastore-UCS-POD1-2"
}

variable "vsphere_compute_cluster" {
  default = "SaS-Cluster"
}

variable "vsphere_template" {
  default = "ubuntu-1604-server-template"
}

variable "folder" {
  default = "wauterw"
}

variable "aci_vm1_name" {
  default = "vSphere1"
}

variable "aci_vm2_name" {
  default = "vSphere2"
}

variable "aci_vm1_address" {
  default = "10.16.2.233"
}

variable "aci_vm2_address" {
  default = "10.16.2.234"
}

variable "gateway" {
  default = "10.16.2.254"
}

variable "dns_list" {
  default = "10.9.15.1"
}

variable "dns_search" {
  default = "cisco.com"
}

variable "domain_name" {
  default = "cisco.com"
}

Deploy infrastructure

As we did with AWS resources (see here), we will follow exactly the same pattern.

First we will perform ‘terraform init’. This will essentially download the vSphere provider from Terraform repo.

cisco@wauterw-ubuntu-desktop:~/software/Terraform/vSphere$ terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "vsphere" (hashicorp/vsphere) 1.13.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.vsphere: version = "~> 1.13"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Then, we will perform a ‘terraform plan’

cisco@wauterw-ubuntu-desktop:~/software/Terraform/vSphere$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_network.vm1_net: Refreshing state...
data.vsphere_network.vm2_net: Refreshing state...
data.vsphere_compute_cluster.cl: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_datastore.ds: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # vsphere_virtual_machine.aci_vm1[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm1" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "vSphere1"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.2.254"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "vSphere1"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.2.233"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = "network-245"
        }
    }

  # vsphere_virtual_machine.aci_vm2[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm2" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "vSphere2"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.2.254"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "vSphere2"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.2.234"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = "network-245"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Next, let’s apply the configuration. You should know the drill by now 🙂


And finally, let's apply the configuration.

cisco@wauterw-ubuntu-desktop:~/software/Terraform/vSphere$ terraform apply
data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_network.vm2_net: Refreshing state...
data.vsphere_compute_cluster.cl: Refreshing state...
data.vsphere_datastore.ds: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_network.vm1_net: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # vsphere_virtual_machine.aci_vm1[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm1" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "vSphere1"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.2.254"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "vSphere1"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.2.233"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = "network-245"
        }
    }

  # vsphere_virtual_machine.aci_vm2[0] will be created
  + resource "vsphere_virtual_machine" "aci_vm2" {
      + boot_retry_delay                        = 10000
      + change_version                          = (known after apply)
      + cpu_limit                               = -1
      + cpu_share_count                         = (known after apply)
      + cpu_share_level                         = "normal"
      + datastore_id                            = "datastore-267"
      + default_ip_address                      = (known after apply)
      + ept_rvi_mode                            = "automatic"
      + firmware                                = "bios"
      + folder                                  = "wauterw"
      + force_power_off                         = true
      + guest_id                                = "ubuntu64Guest"
      + guest_ip_addresses                      = (known after apply)
      + host_system_id                          = (known after apply)
      + hv_mode                                 = "hvAuto"
      + id                                      = (known after apply)
      + imported                                = (known after apply)
      + latency_sensitivity                     = "normal"
      + memory                                  = 4096
      + memory_limit                            = -1
      + memory_share_count                      = (known after apply)
      + memory_share_level                      = "normal"
      + migrate_wait_timeout                    = 30
      + moid                                    = (known after apply)
      + name                                    = "vSphere2"
      + num_cores_per_socket                    = 1
      + num_cpus                                = 2
      + reboot_required                         = (known after apply)
      + resource_pool_id                        = "resgroup-308"
      + run_tools_scripts_after_power_on        = true
      + run_tools_scripts_after_resume          = true
      + run_tools_scripts_before_guest_shutdown = true
      + run_tools_scripts_before_guest_standby  = true
      + scsi_bus_sharing                        = "noSharing"
      + scsi_controller_count                   = 1
      + scsi_type                               = "pvscsi"
      + shutdown_wait_timeout                   = 3
      + swap_placement_policy                   = "inherit"
      + uuid                                    = (known after apply)
      + vapp_transport                          = (known after apply)
      + vmware_tools_status                     = (known after apply)
      + vmx_path                                = (known after apply)
      + wait_for_guest_ip_timeout               = 0
      + wait_for_guest_net_routable             = true
      + wait_for_guest_net_timeout              = 5

      + clone {
          + linked_clone  = false
          + template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af"
          + timeout       = 30

          + customize {
              + dns_server_list = [
                  + "10.9.15.1",
                ]
              + dns_suffix_list = [
                  + "cisco.com",
                ]
              + ipv4_gateway    = "10.16.2.254"
              + timeout         = 10

              + linux_options {
                  + domain       = "cisco.com"
                  + host_name    = "vSphere2"
                  + hw_clock_utc = true
                }

              + network_interface {
                  + ipv4_address = "10.16.2.234"
                  + ipv4_netmask = 24
                }
            }
        }

      + disk {
          + attach           = false
          + datastore_id     = ""
          + device_address   = (known after apply)
          + disk_mode        = "persistent"
          + disk_sharing     = "sharingNone"
          + eagerly_scrub    = false
          + io_limit         = -1
          + io_reservation   = 0
          + io_share_count   = 0
          + io_share_level   = "normal"
          + keep_on_remove   = false
          + key              = 0
          + label            = "disk0"
          + path             = (known after apply)
          + size             = 16
          + thin_provisioned = false
          + unit_number      = 0
          + uuid             = (known after apply)
          + write_through    = false
        }

      + network_interface {
          + adapter_type          = "vmxnet3"
          + bandwidth_limit       = -1
          + bandwidth_reservation = 0
          + bandwidth_share_count = (known after apply)
          + bandwidth_share_level = "normal"
          + device_address        = (known after apply)
          + key                   = (known after apply)
          + mac_address           = (known after apply)
          + network_id            = "network-245"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

vsphere_virtual_machine.aci_vm1[0]: Creating...
vsphere_virtual_machine.aci_vm2[0]: Creating...
vsphere_virtual_machine.aci_vm1[0]: Still creating... [10s elapsed]
vsphere_virtual_machine.aci_vm2[0]: Still creating... [10s elapsed]
vsphere_virtual_machine.aci_vm1[0]: Still creating... [20s elapsed]
...
...
vsphere_virtual_machine.aci_vm2[0]: Still creating... [4m20s elapsed]
vsphere_virtual_machine.aci_vm1[0]: Creation complete after 6m32s [id=423c546b-2c1a-3cf4-0360-e0d300c7684d]
vsphere_virtual_machine.aci_vm2[0]: Creation complete after 6m33s [id=423c54c3-b2ac-d866-2ba7-3b231ddfc804]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Let's verify on vSphere

Destroy configuration

Next, we will destroy the two servers as we don't immediately require them. Note that destroying goes much faster (6s) compared to creating the servers (6min). Reason is obviously the cloning step of the template during the creation.

cisco@wauterw-ubuntu-desktop:~/software/Terraform/vSphere$ terraform destroy
data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_datastore.ds: Refreshing state...
data.vsphere_network.vm1_net: Refreshing state...
data.vsphere_network.vm2_net: Refreshing state...
data.vsphere_compute_cluster.cl: Refreshing state...
vsphere_virtual_machine.aci_vm2[0]: Refreshing state... [id=423c54c3-b2ac-d866-2ba7-3b231ddfc804]
vsphere_virtual_machine.aci_vm1[0]: Refreshing state... [id=423c546b-2c1a-3cf4-0360-e0d300c7684d]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # vsphere_virtual_machine.aci_vm1[0] will be destroyed
  - resource "vsphere_virtual_machine" "aci_vm1" {
      - boot_delay                              = 0 -> null
      - boot_retry_delay                        = 10000 -> null
      - boot_retry_enabled                      = false -> null
      - change_version                          = "2019-11-23T15:28:42.507933Z" -> null
      - cpu_hot_add_enabled                     = false -> null
      - cpu_hot_remove_enabled                  = false -> null
      - cpu_limit                               = -1 -> null
      - cpu_performance_counters_enabled        = false -> null
      - cpu_reservation                         = 0 -> null
      - cpu_share_count                         = 2000 -> null
      - cpu_share_level                         = "normal" -> null
      - custom_attributes                       = {} -> null
      - datastore_id                            = "datastore-267" -> null
      - default_ip_address                      = "10.16.2.233" -> null
      - efi_secure_boot_enabled                 = false -> null
      - enable_disk_uuid                        = false -> null
      - enable_logging                          = false -> null
      - ept_rvi_mode                            = "automatic" -> null
      - extra_config                            = {} -> null
      - firmware                                = "bios" -> null
      - folder                                  = "wauterw" -> null
      - force_power_off                         = true -> null
      - guest_id                                = "ubuntu64Guest" -> null
      - guest_ip_addresses                      = [
          - "10.16.2.233",
          - "fe80::250:56ff:febc:7404",
        ] -> null
      - host_system_id                          = "host-266" -> null
      - hv_mode                                 = "hvAuto" -> null
      - id                                      = "423c546b-2c1a-3cf4-0360-e0d300c7684d" -> null
      - latency_sensitivity                     = "normal" -> null
      - memory                                  = 4096 -> null
      - memory_hot_add_enabled                  = false -> null
      - memory_limit                            = -1 -> null
      - memory_reservation                      = 0 -> null
      - memory_share_count                      = 40960 -> null
      - memory_share_level                      = "normal" -> null
      - migrate_wait_timeout                    = 30 -> null
      - moid                                    = "vm-724" -> null
      - name                                    = "vSphere1" -> null
      - nested_hv_enabled                       = false -> null
      - num_cores_per_socket                    = 1 -> null
      - num_cpus                                = 2 -> null
      - reboot_required                         = false -> null
      - resource_pool_id                        = "resgroup-308" -> null
      - run_tools_scripts_after_power_on        = true -> null
      - run_tools_scripts_after_resume          = true -> null
      - run_tools_scripts_before_guest_reboot   = false -> null
      - run_tools_scripts_before_guest_shutdown = true -> null
      - run_tools_scripts_before_guest_standby  = true -> null
      - scsi_bus_sharing                        = "noSharing" -> null
      - scsi_controller_count                   = 1 -> null
      - scsi_type                               = "pvscsi" -> null
      - shutdown_wait_timeout                   = 3 -> null
      - swap_placement_policy                   = "inherit" -> null
      - sync_time_with_host                     = false -> null
      - tags                                    = [] -> null
      - uuid                                    = "423c546b-2c1a-3cf4-0360-e0d300c7684d" -> null
      - vapp_transport                          = [] -> null
      - vmware_tools_status                     = "guestToolsRunning" -> null
      - vmx_path                                = "vSphere1/vSphere1.vmx" -> null
      - wait_for_guest_ip_timeout               = 0 -> null
      - wait_for_guest_net_routable             = true -> null
      - wait_for_guest_net_timeout              = 5 -> null

      - clone {
          - linked_clone  = false -> null
          - template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af" -> null
          - timeout       = 30 -> null

          - customize {
              - dns_server_list = [
                  - "10.9.15.1",
                ] -> null
              - dns_suffix_list = [
                  - "cisco.com",
                ] -> null
              - ipv4_gateway    = "10.16.2.254" -> null
              - timeout         = 10 -> null

              - linux_options {
                  - domain       = "cisco.com" -> null
                  - host_name    = "vSphere1" -> null
                  - hw_clock_utc = true -> null
                }

              - network_interface {
                  - dns_server_list = [] -> null
                  - ipv4_address    = "10.16.2.233" -> null
                  - ipv4_netmask    = 24 -> null
                  - ipv6_netmask    = 0 -> null
                }
            }
        }

      - disk {
          - attach           = false -> null
          - datastore_id     = "datastore-267" -> null
          - device_address   = "scsi:0:0" -> null
          - disk_mode        = "persistent" -> null
          - disk_sharing     = "sharingNone" -> null
          - eagerly_scrub    = false -> null
          - io_limit         = -1 -> null
          - io_reservation   = 0 -> null
          - io_share_count   = 1000 -> null
          - io_share_level   = "normal" -> null
          - keep_on_remove   = false -> null
          - key              = 2000 -> null
          - label            = "disk0" -> null
          - path             = "vSphere1/vSphere1.vmdk" -> null
          - size             = 16 -> null
          - thin_provisioned = false -> null
          - unit_number      = 0 -> null
          - uuid             = "6000C293-fd0a-4f15-2b60-2473c4dcc197" -> null
          - write_through    = false -> null
        }

      - network_interface {
          - adapter_type          = "vmxnet3" -> null
          - bandwidth_limit       = -1 -> null
          - bandwidth_reservation = 0 -> null
          - bandwidth_share_count = 50 -> null
          - bandwidth_share_level = "normal" -> null
          - device_address        = "pci:0:7" -> null
          - key                   = 4000 -> null
          - mac_address           = "00:50:56:bc:74:04" -> null
          - network_id            = "network-245" -> null
          - use_static_mac        = false -> null
        }
    }

  # vsphere_virtual_machine.aci_vm2[0] will be destroyed
  - resource "vsphere_virtual_machine" "aci_vm2" {
      - boot_delay                              = 0 -> null
      - boot_retry_delay                        = 10000 -> null
      - boot_retry_enabled                      = false -> null
      - change_version                          = "2019-11-23T15:28:45.488548Z" -> null
      - cpu_hot_add_enabled                     = false -> null
      - cpu_hot_remove_enabled                  = false -> null
      - cpu_limit                               = -1 -> null
      - cpu_performance_counters_enabled        = false -> null
      - cpu_reservation                         = 0 -> null
      - cpu_share_count                         = 2000 -> null
      - cpu_share_level                         = "normal" -> null
      - custom_attributes                       = {} -> null
      - datastore_id                            = "datastore-267" -> null
      - default_ip_address                      = "10.16.2.234" -> null
      - efi_secure_boot_enabled                 = false -> null
      - enable_disk_uuid                        = false -> null
      - enable_logging                          = false -> null
      - ept_rvi_mode                            = "automatic" -> null
      - extra_config                            = {} -> null
      - firmware                                = "bios" -> null
      - folder                                  = "wauterw" -> null
      - force_power_off                         = true -> null
      - guest_id                                = "ubuntu64Guest" -> null
      - guest_ip_addresses                      = [
          - "10.16.2.234",
          - "fe80::250:56ff:febc:2b9b",
        ] -> null
      - host_system_id                          = "host-266" -> null
      - hv_mode                                 = "hvAuto" -> null
      - id                                      = "423c54c3-b2ac-d866-2ba7-3b231ddfc804" -> null
      - latency_sensitivity                     = "normal" -> null
      - memory                                  = 4096 -> null
      - memory_hot_add_enabled                  = false -> null
      - memory_limit                            = -1 -> null
      - memory_reservation                      = 0 -> null
      - memory_share_count                      = 40960 -> null
      - memory_share_level                      = "normal" -> null
      - migrate_wait_timeout                    = 30 -> null
      - moid                                    = "vm-723" -> null
      - name                                    = "vSphere2" -> null
      - nested_hv_enabled                       = false -> null
      - num_cores_per_socket                    = 1 -> null
      - num_cpus                                = 2 -> null
      - reboot_required                         = false -> null
      - resource_pool_id                        = "resgroup-308" -> null
      - run_tools_scripts_after_power_on        = true -> null
      - run_tools_scripts_after_resume          = true -> null
      - run_tools_scripts_before_guest_reboot   = false -> null
      - run_tools_scripts_before_guest_shutdown = true -> null
      - run_tools_scripts_before_guest_standby  = true -> null
      - scsi_bus_sharing                        = "noSharing" -> null
      - scsi_controller_count                   = 1 -> null
      - scsi_type                               = "pvscsi" -> null
      - shutdown_wait_timeout                   = 3 -> null
      - swap_placement_policy                   = "inherit" -> null
      - sync_time_with_host                     = false -> null
      - tags                                    = [] -> null
      - uuid                                    = "423c54c3-b2ac-d866-2ba7-3b231ddfc804" -> null
      - vapp_transport                          = [] -> null
      - vmware_tools_status                     = "guestToolsRunning" -> null
      - vmx_path                                = "vSphere2/vSphere2.vmx" -> null
      - wait_for_guest_ip_timeout               = 0 -> null
      - wait_for_guest_net_routable             = true -> null
      - wait_for_guest_net_timeout              = 5 -> null

      - clone {
          - linked_clone  = false -> null
          - template_uuid = "423c759f-8c32-d183-8e8d-8f5ff2c266af" -> null
          - timeout       = 30 -> null

          - customize {
              - dns_server_list = [
                  - "10.9.15.1",
                ] -> null
              - dns_suffix_list = [
                  - "cisco.com",
                ] -> null
              - ipv4_gateway    = "10.16.2.254" -> null
              - timeout         = 10 -> null

              - linux_options {
                  - domain       = "cisco.com" -> null
                  - host_name    = "vSphere2" -> null
                  - hw_clock_utc = true -> null
                }

              - network_interface {
                  - dns_server_list = [] -> null
                  - ipv4_address    = "10.16.2.234" -> null
                  - ipv4_netmask    = 24 -> null
                  - ipv6_netmask    = 0 -> null
                }
            }
        }

      - disk {
          - attach           = false -> null
          - datastore_id     = "datastore-267" -> null
          - device_address   = "scsi:0:0" -> null
          - disk_mode        = "persistent" -> null
          - disk_sharing     = "sharingNone" -> null
          - eagerly_scrub    = false -> null
          - io_limit         = -1 -> null
          - io_reservation   = 0 -> null
          - io_share_count   = 1000 -> null
          - io_share_level   = "normal" -> null
          - keep_on_remove   = false -> null
          - key              = 2000 -> null
          - label            = "disk0" -> null
          - path             = "vSphere2/vSphere2.vmdk" -> null
          - size             = 16 -> null
          - thin_provisioned = false -> null
          - unit_number      = 0 -> null
          - uuid             = "6000C29e-322f-077a-728f-2223206e8016" -> null
          - write_through    = false -> null
        }

      - network_interface {
          - adapter_type          = "vmxnet3" -> null
          - bandwidth_limit       = -1 -> null
          - bandwidth_reservation = 0 -> null
          - bandwidth_share_count = 50 -> null
          - bandwidth_share_level = "normal" -> null
          - device_address        = "pci:0:7" -> null
          - key                   = 4000 -> null
          - mac_address           = "00:50:56:bc:2b:9b" -> null
          - network_id            = "network-245" -> null
          - use_static_mac        = false -> null
        }
    }

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

vsphere_virtual_machine.aci_vm1[0]: Destroying... [id=423c546b-2c1a-3cf4-0360-e0d300c7684d]
vsphere_virtual_machine.aci_vm2[0]: Destroying... [id=423c54c3-b2ac-d866-2ba7-3b231ddfc804]
vsphere_virtual_machine.aci_vm2[0]: Destruction complete after 9s
vsphere_virtual_machine.aci_vm1[0]: Destruction complete after 9s

Destroy complete! Resources: 2 destroyed.

Pretty simple but good to make a blog post about it for future reference.

Ciao!