Skip to content

VM-provisioning#

In this article, we examine the use case of provisioning virtual machines (VM) via i-doit. For this purpose, we utilize a well-maintained IT documentation paired with a few scripts to make this automation a reality.

Problem#

Up until now, documentation and configuration files are strictly distinguished: The organization-wide IT is documented in i-doit, including a virtualization cluster. This cluster consists of multiple virtualization hosts (host systems) and the virtual machines running there (guest systems). The configuration of each VM is executed within the administration environment of the cluster. This includes the creation of a new VM with specific settings to CPU, memory, network, disk space and so on. This process is also called provisioning.

For the daily routine, this means that the process is characterized by a dedicated tool which creates a new VM. Subsequently, the completed work is documented in i-doit:

  1. Start the configuration tool of the virtualization cluster
  2. Create and configure a new VM
  3. Switch to the IT documentation (i-doit)
  4. Create a new VM, configure the respective categories, assign the VM to the cluster

There is no data exchange between the applied tool and i-doit, thus the VM configuration has to be performed twice. Errors cannot be ruled out when transferring the configuration. The redundant maintenance of configuration files is therefore a very thankless task for the admin.

This article is outdated and no longer current

Please note that the procedure described here may already be outdated.

Solution#

We wish to optimize this process by using the configuration files available in i-doit and by automating some of the steps. With this in mind, we change the process:

  1. Open the IT documentation (i-doit)
  2. Create a new VM, configure the respective categories, assign the VM to the cluster
  3. The VM will be created and provisioned automatically in the configuration tool of the virtualization cluster

Four manual steps become two. The third step is carried out automatically in the background. The doubled maintenance of the configuration files by the admin can be omitted. Error sources are eliminated and the admin is happy.

Assumptions#

We will completely go through this use case with an example. This is supposed to show the general procedure and can be applied easily in other environments. For the solution we assume:

  1. The virtualization cluster is based on VMware vSphere in version 5.
  2. i-doit is installed on Debian GNU/Linux 8.5. For this, we use the Eval Appliance. The distribution packages should be updated with apt.
  3. The version of i-doit is 1.7.1 or higher. The i-doit host is accessible via the FQDN i-doit.example.net.
  4. To monitor the automation we use the VMware vSphere client.
  5. A user exists in VMware vSphere who has the permissions to provision VMs. In our example this user is called vmprovision with the password vmprovision.

Configuration#

First of all, we have to carry out some preparations so that both systems are able to communicate with each other.

vSphere SDK#

We require VMware SDK for Perl on the i-doit host so that i-doit can inform the vSphere cluster to provision a new VM. Therefore we download version 6.0.2. For this purpose, you need an account at VMware and you have to accept the VMware End User License Agreement (EULA). We decide to download the package as Tarball (.tar.gz) for 64bit operating systems.

vm-provisioning-sdk

vm-provisioning-sdk

After downloading, we copy the Tarball to the i-doit host and extract it. We will log in as root since the root permissions are necessary for almost all of the following commands:

1
2
3
4
5
6
scp VMware-vSphere-Perl-SDK-6.0.0-3561779.x86_64.tar.gz idoitadm@i-doit.example.net:
ssh idoitadm@i-doit.example.net
sudo -s
tar vxzf VMware-vSphere-Perl-SDK-6.0.0-3561779.x86_64.tar.gz
mv vmware-vsphere-cli-distrib/ /usr/src/
cd /usr/src/

At this point, it is a good idea to bring the system up to date - if you have not already done so. We also install additional packages via apt and cpan:

1
2
3
apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove && apt-get clean
apt-get install git build-essential libssl-dev perl-doc libxml-libxml-perl libxml2-dev uuid-dev libuuid-perl libcrypt-ssleay-perl libapache2-mod-perl2 libsoap-lite-perl
cpan install CPAN ExtUtils::MakeMaker Module::Build Net::FTP LWP Crypt::OpenSSL::RSA Class::MethodMaker Socket6 IO::Socket::INET6 Convert::ASN1 Crypt::X509 UUID::Random Archive::Zip Path::Class Try::Tiny Data::Dump Net::INET6Glue LWP::Protocol::https

Since the SDK does not support Debian GNU/Linux officially, we will need to trick the SDK into thinking it is a different operating system:

1
echo ubuntu > /etc/tmp-release

The SDK needs the environment variables http_proxy and ftp_proxy:

1
2
export http_proxy=
export ftp_proxy=

Now we install the SDK by utilizing the provided installation script:

1
./vmware-install.pl

The script once again needs to have the EULA of VMware confirmed with yes. If additional Perl modules are to be installed, you should also confirm this with yes. Once the installation was completed successfully, the following text appears as part of the output:

1
This installer has successfully installed both vSphere CLI and the vSphere SDK for Perl.

At this point, the installation of the SDK is completed. We followed these guidelines. Thank you very much! And now we continue with the next step.

i-doit API#

Part of the communication between i-doit and VMware takes place via the API of i-doit. The API has to be activated and you need to know the API key. To use the API comfortably we use the reference client for PHP:

1
2
3
4
cd /usr/local/share/
git clone https://bitbucket.org/dstuecken/i-doit-api-clients.git
cd i-doit-api-clients/php/
make initialize

For make initialize the API client queries the URL and the API key of the i-doit installation that is going to be used.

Script#

The following script represents the link connection between i-doit and VMware:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
#!/usr/bin/php
<?php

/**
* Example script to privision a new virtual machine in vSphere with i-doit
*
* This script uses the i-doit API and the VMware Perl SDK. For more details, please See the i-doit
* knowledge base.
*
* Licensed under the MIT License (MIT), <https://opensource.org/licenses/mit-license.php>.
*
* Copyright (c) 2016 synetics GmbH
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction,
* including without limitation the rights to use, copy, modify, merge, publish, distribute,
* sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or
* substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
* DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

use idoit\Api\Client as ApiClient;
use idoit\Api\CMDB\Object as CMDBObject;
use idoit\Api\CMDB\Category;
use idoit\Api\Connection as ApiConnection;

require_once('/usr/local/share/i-doit-api-clients/php/apiclient.php');

/**
* Configuration
*/

$vSphereUsername = 'vmprovision';
$vSpherePassword = 'vmprovision';
$vSphereWebService = 'https://vc01.example.net:443/sdk/webService';
$objCMDBStatus = '15'; // 'to be provisioned'
$objCMDBStatus_ready = '16'; // 'provisioned'
$api_entry_point = 'http://localhost/src/jsonrpc.php';
$api_key = '2p1osbg427';
\idoit\Api\Config::$jsonRpcDebug = false;

function filter_number($p_string)
{
// Check, if we got a positive or negative number.
$l_sign = (substr(trim($p_string), 0, 1) === '-') ? '-' : '';

// First we strip the currency ("GHZ", "Euro", "$", ...) including spaces.
$p_string = preg_replace('/([^,\.\d])*/i', '', $p_string);

// If the number is null
if (is_null($p_string) || $p_string === '') return null;

// Check if someone wrote a string like "1.000.000".
if (substr_count($p_string, '.') > 1)
{
    $p_string = str_replace('.', '', $p_string);
} // if

// Check if someone wrote a string like "1,000,000".
if (substr_count($p_string, ',') > 1)
{
    $p_string = str_replace(',', '', $p_string);
} // if

// If we find a single point or a single comma, we use the last found one as decimal point.
if (strpos($p_string, '.') !== false || strpos($p_string, ',') !== false)
{
    if (strpos($p_string, '.') > strpos($p_string, ','))
    {
    $p_string = str_replace(',', '', $p_string);
    }
    elseif (strpos($p_string, '.') < strpos($p_string, ','))
    {
    $p_string = str_replace('.', '', $p_string);
    $p_string = str_replace(',', '.', $p_string);
    }
    elseif (strpos($p_string, '.') === false && is_int(strpos($p_string, ',')))
    {
    $p_string = str_replace(',', '.', $p_string);
    }
} // if

// Finally check if number is not numeric then return null
if (!is_numeric($p_string)) return null;

// Now we replace commas with dots: "1000,10" to "1000.10" and return the rounded value.
return (float) round(str_replace(',', '.', $l_sign . $p_string), 4);
} // function


function to_kbytes($p_value)
{
    if (is_null($p_value) || !is_numeric(substr($p_value, 0, -2)))
    {
        return null;
    } // if
    $l_return = trim($p_value);
    $l_unit   = strtoupper(substr($p_value, -2));
    switch ($l_unit)
    {
    case 'TB':
            $l_return *= 1024;
        case 'GB':
            $l_return *= 1024;
        case 'MB':
            $l_return *= 1024;
    } // switch
    return $l_return;
} // function

try
{
    if (isset($argv[1]))
    {
        if (is_scalar($argv[1]) && ($params = base64_decode($argv[1])))
        {
            if (($params = json_decode($params, true)))
            {
                if (isset($params['objectID']) && $params['objectID'] > 0)
                {
                    if ($params['categoryConst'] == 'C__CATG__GLOBAL')
                    {
                            if ($params['postData']['C__OBJ__CMDB_STATUS'] == $objCMDBStatus)
                            {

                        \idoit\Api\Config::$jsonRpcDebug = false;
                        $l_apiClient = new ApiClient(
                            new ApiConnection(
                            $api_entry_point, $api_key
                            )
                        );

                        // Get number of CPUs
                        $l_request = new \idoit\Api\Request($l_apiClient,
                            'cmdb.category.read',
                            array(
                            'objID' => $params['objectID'],
                            'category' => 'C__CATG__CPU'
                            )
                        );

                        $l_response = $l_request->send();
                        $cpuCount = count($l_response);

                        // Get memory
                        $l_request = new \idoit\Api\Request($l_apiClient,
                                'cmdb.category.read',
                                array(
                                'objID' => $params['objectID'],
                                'category' => 'C__CATG__MEMORY'
                            )
                            );

                        $all_memory = 0;
                        $l_response = $l_request->send();
                        for ( $x = 0; $x < count($l_response); $x++) {

                            $memory_cleaned = filter_number($l_response[$x]['capacity']['title']);
                            $memory = to_kbytes( $memory_cleaned . $l_response[$x]['unit']['title']);
                            $all_memory = $all_memory + $memory;
                        }
                        $all_memory_mb = $all_memory / 1024;

                        // Get HDD
                        $l_request = new \idoit\Api\Request($l_apiClient,
                                'cmdb.category.read',
                                array(
                                'objID' => $params['objectID'],
                                'category' => 'C__CMDB__SUBCAT__STORAGE__DEVICE'
                            )
                            );

                        $l_response = $l_request->send();
                        $hdd_cleaned = filter_number($l_response[0]['capacity']['title']);
                        $hdd = to_kbytes( $hdd_cleaned . $l_response[0]['unit']['title']);
                        $hdd_mb = $hdd / 1024;

                        // Get virtual host and cluster
                        $l_request = new \idoit\Api\Request($l_apiClient,
                            'cmdb.category.read',
                            array(
                            'objID' => $params['objectID'],
                            'category' => 'C__CATG__VIRTUAL_MACHINE'
                            )
                        );
                        $l_response = $l_request->send();

                        $datacenter = $l_response[0]['hosts']['title'];
                        $host = $l_response[0]['primary']['title'];

                        //Get SAN LUN
                        $l_request = new \idoit\Api\Request($l_apiClient,
                            'cmdb.category.read',
                            array(
                            'objID' => $params['objectID'],
                            'category' => 'C__CATG__VIRTUAL_DEVICE'
                            )
                        );
                        $l_response = $l_request->send();
                        for ( $x = 0; $x < count($l_response); $x++) {
                            if (isset($l_response[$x]['host_ldev_client']['ref_title']) && !empty($l_response[$x]['host_ldev_client']['ref_title']))
                            {
                            $lun = $l_response[$x]['host_ldev_client']['ref_title'];
                            }
                        }

                        //Get Network
                        $l_request = new \idoit\Api\Request($l_apiClient,
                            'cmdb.category.read',
                            array(
                            'objID' => $params['objectID'],
                            'category' => 'C__CATG__VIRTUAL_DEVICE'
                            )
                        );
                        $l_response = $l_request->send();
                        for ( $x = 0; $x < count($l_response); $x++) {
                            if (isset($l_response[$x]['switch_port_group']['vs_port_group_title']) && !empty($l_response[$x]['switch_port_group']['vs_port_group_title']))
                            {
                            $network = $l_response[$x]['switch_port_group']['vs_port_group_title'];
                            }
                        }

                        $dom      = new DomDocument("1.0");
                        $dom->formatOutput = true;
                        $dom->preserveWhiteSpace = false;
                        $VirtualMachines  = $dom->createElement('Virtual-Machines');
                        $VirtualMachine = $dom->createElement('Virtual-Machine');
                        $VirtualMachines->appendChild( $VirtualMachine );
                        $VirtualMachine->appendChild( $dom->createElement('Name', $params['postData']['C__CATG__GLOBAL_TITLE']) );
                        $VirtualMachine->appendChild( $dom->createElement('Host', $host) );
                        $VirtualMachine->appendChild( $dom->createElement('Datacenter', $datacenter) );
                        $VirtualMachine->appendChild( $dom->createElement('Guest-Id', '' ) );
                        $VirtualMachine->appendChild( $dom->createElement('Datastore', $lun ) );
                        $VirtualMachine->appendChild( $dom->createElement('Disksize', $hdd ) );
                        $VirtualMachine->appendChild( $dom->createElement('Memory', $all_memory_mb ) );
                        $VirtualMachine->appendChild( $dom->createElement('Number-of-Processor', $cpuCount) );
                        $VirtualMachine->appendChild( $dom->createElement('Nic-Network', $network) );
                        $VirtualMachine->appendChild( $dom->createElement('Nic-Poweron', '1') );
                        $dom->appendChild( $VirtualMachines );

                        $dom->save("/tmp/i-doit-vm-provision.xml");

                                exec("/usr/src/vmware-vsphere-cli-distrib/apps/vm/vmcreate.pl --url $vSphereWebService --username $vSphereUsername --password $vSpherePassword --filename /tmp/i-doit-vm-provision.xml --schema /usr/src/vmware-vsphere-cli-distrib/apps/schema/vmcreate.xsd");
                        echo 'Provisioned machine ' . $params['postData']['C__CATG__GLOBAL_TITLE'] ;

                        //Set CMDB Status to provisioned
                        $l_request = new \idoit\Api\Request($l_apiClient,
                            'cmdb.category.update',
                            array(
                                'objID' => $params['objectID'],
                                'category' => 'C__CATG__GLOBAL',
                                'data'     => array(
                                'cmdb_status' => $objCMDBStatus_ready,
                                )
                            )
                        );

                        $l_response = $l_request->send();

                    }
                    }
                }
                else throw new Exception('Event Error: Object id not existent.');

            }
            else throw new Exception('Event Error: Script Parameters not json-encoded.');

        }
        else throw new Exception('Event Error: Script Parameters not base64 encoded.');
    }
} catch (Exception $e)
{
    echo $e->getMessage();
    exit;
}

We copy this to the /usr/local/bin/i-doit_provision.php file. We also assign the appropriate permissions so that the Apache web server is able to run the script:

1
chmod 775 /usr/local/bin/i-doit_provision.php

This script includes the configuration to access both i-doit and VMware. It has to be adjusted accordingly:

  • $vSphereUsername: User for vSphere (see above)
  • $vSpherePassword: Password for vSphere (see above)
  • $vSphereWebService: URL to the Web Service of vSphere
  • $objCMDBStatus: ID of the CMDB status to be provisioned (see below)
  • $objCMDBStatus_ready: ID of the CMDB status provisioned (see below)
  • $api_entry_point: URL to the API of i-doit
  • $api_key: Key for the i-doit API

Supplement CMDB Status#

The two CMDB status statements to be provisioned and provisioned do not exist in the i-doit default installation yet but they are crucial for the automation. They need to be supplemented as described in the linked article.

vm-provisioning-cmd

Configure Event#

Now i-doit has to be configured to execute the script if certain changes are performed. For this, we use the event controls and create a new hook.

  • Event: Category: (arbitrary sources) Save
  • Description: VM provisioning
  • Type: SHELL COMMAND
  • Command: /usr/local/bin/i-doit_provision.php
  • Additional parameters: leave this blank
  • Mode: Executive live

vm-provisioning-configure

At this point, the configuration is finished and the automation is "activated".

Handling#

After this extensive one-time configuration we can now take a look at how the automation works.

Document the Virtual Environment in i-doit#

A vSphere cluster (object type Cluster) with some ESX hosts (object type Virtual Host) already exists. The object title of the cluster corresponds to the datacenter in vSphere. The object titles of the ESX hosts are the host names which are also displayed in vSphere.

vm-provisioning-virtual

vm-provisioning

Virtual switches are configured with VLAN IDs (category Virtual switches) for each ESX host.

vm-provisioning

The ESX hosts are bound to a central storage which supplies the VMs with disk space (category Logic devices (Client)).

vm-provisioning

The storage side is construed accordingly. The Logic devices (LDEV Server) category contains the assignment of LUNs to ESX hosts.

vm-provisioning

Create New VM#

The vSphere Cluster and the storage are now ready according to the IT documentation. It is time to document VMs – and provision them automatically.

First you create a new VM (object type Virtual server) and set its CMDB status to planned (since it does not exist yet).

vm-provisioning

Afterwards, the desired system configuration is documented. The number of demanded CPU cores is documented in the CPU category. One entry is generated for each core.

vm-provisioning

The required RAM is documented in the Memory category. Number of RAM modules and the chosen value units are not important. The total capacity is summed up instead.

vm-provisioning

In the Network → Port category the title of the first entry is used to configure a virtual port for the VM.

vm-provisioning

The required disk space is specified in the Direct Attached Storage → Device category.

vm-provisioning

In the Virtual machine category you document in which cluster and on what ESX host the VM is supposed to run.

vm-provisioning

The assignment of storage and network is stored in the Virtual machine → Virtual devices category. The screenshot shows which attributes should be filled in.

vm-provisioning

With this the configuration of the VM is sufficiently documented.

Templates

A template is useful to document virtual machines faster and thereby provision them faster. A template is created for each pre-assembled VM and new objects are generated on basis of this template.

Provision VM Automatically#

Now we come to the final step: the provisioning. We do not have much left to do. In the General category the CMDB status is set to to be provisioned. When the change is saved, the script described above starts to read the configuration files from i-doit and transfer them to vSphere. Saving may take some time as the results need to be there first.

vm-provisioning

Once the provisioning has finished successfully, the CMDB status is automatically changed to provisioned. The VM has been created and runs. Done!

vm-provisioning

The communication between i-doit and vSphere is displayed at Administration → CMDB settings → Events → History (Log).