New open source fling: vRealize Operations Export Tool

Summary

This is an Open Source fling. Find out about the VMware Open Source initiative. The vRealize Operations Export Tool allows users to both extract and export metrics and properties from vRealize Operations. The highly modular design of this tool allows almost any output format to be plugged in. Currently, output formats supported include CSV (comma separated files) and generic SQL (through JDBC). This export tool also supports advanced constructs such as, navigating through multi step relationships between objects as well as aliasing. The design is made to be very flexible and fast. Typical applications of this tool include exporting to data warehouse, reporting tools, and analytics tools.

screenshot

Requirements

The vRealize Operations Export Tool is designed to run on a separate machine (virtual or physical) with the following specifications:

  • 2 CPU cores
  • Memory: >1GB free
  • Storage: >5GB free
  • Any Linux with Kernel 2.6 or higher
  • Windows 7,8,10
  • Windows Server 2008, 2012 or 2016
  • Java: JRE 1.8 or higher
  • vRealize Operations: Tested with vRealize Operations 6.4, 6.5 and 6.6

Instructions

Installation on Linux, Mac or other UNIX-like OS

  1. Download the binaries from here: TODO: Link to binaries
  2. Unzip the files:``` mkdir ~/vrops-export cd ~/vrops-export unzip vrops-export--bin.zip cd vrops-export-/bin
  3. Make the start script runnable``` chmod +x exporttool.sh
  4. Run it!``` ./exporttool.sh -d samples/vmfields.yaml -u admin -p password -H https://my.vrops.host -i

Installation on Windows

  1. Download the binaries from here: TODO: Add link
  2. Unzip the files into a directory of your choice, e.g. c:\vropsexport
  3. Open a command window and cd into the directory you created, e.g.``` cd c:\vropsexport
  4. Cd into the bin directory``` cd bin
  5. Run the tool``` exporttool.bat -d ..\samples\vmfields.yaml -u admin -p password -H https://my.vrops.host

Command syntax

 1usage: exporttool [-d <arg>] [-e <arg>] [-F <arg>] [-H <arg>] [-h] [-i]
 2       [-l <arg>] [-m <arg>] [-n <arg>] [-o <arg>] [-P <arg>] [-p <arg>]
 3       [-q] [-R <arg>] [-s <arg>] [-S] [-t <arg>] [-T <arg>] [--trustpass
 4       <arg>] [-u <arg>] [-v]
 5
 6 -d,--definition <arg>       Path to definition file
 7 -e,--end <arg>              Time period end (date format in definition
 8                             file)
 9 -F,--list-fields <arg>      Print name and keys of all fields to stdout
10 -H,--host <arg>             URL to vRealize Operations Host
11 -h,--help                   Print a short help text
12 -i,--ignore-cert            Trust any cert (DEPRECATED!)
13 -l,--lookback <arg>         Lookback time
14 -m,--max-rows <arg>         Maximum number of rows to fetch from API
15                             (default=unlimited)
16 -n,--namequery <arg>        Name query
17 -o,--output <arg>           Output file
18 -P,--parent <arg>           Parent resource (ResourceKind:resourceName)
19 -p,--password <arg>         Password
20 -q,--quiet                  Quiet mode (no progress counter)
21 -R,--resource-kinds <arg>   List resource kinds
22 -s,--start <arg>            Time period start (date format in definition
23                             file)
24 -S,--streaming              True streaming processing. Faster but less
25                             reliable
26 -t,--threads <arg>          Number of parallel processing threads
27                             (default=10)
28 -T,--truststore <arg>       Truststore filename
29    --trustpass <arg>        Truststore password (default=changeit)
30 -u,--username <arg>         Username
31 -v,--verbose                Print debug and timing information 

Certificate and trust management

As of version 2.1.0, the -i option has been deprecated for security reasons. Instead, the tool will prompt the user when it encounters an untrusted certificate. If the user chooses to trust the certificate, it is stored in a personal truststore and reused next time the tool is executed against that host. By default, the trusted certs are stored in $HOME/.vropsexport/truststore, but the location can be overridden using the -T flag.

Notes

  • Start and end dates will use the date format specified in the definition file. Since dates tend to contain spaces and special characters, you probably want to put dates within double quotes (").
  • If you're unsure of what the metric names are, use the -F option to print the metric names and keys for a specific resource type, e.g. -F VirtualMachine
  • The -l (lookback) parameter is an alternative to the start and end dates. It sets the end date to the current time and goes back as far as you specify. You specify it as a number and a unit, e.g. 24h for 24 hours back. Valid unit are d=days, h=hours, m=minutes, s=seconds.
  • The -P flag restricts the export to objects sharing a specified parent. Parents must be specified as resource kind and resource name, for example HostSystem:esxi-01 if you want to export only VMs on the host named "esxi-01".

Definition file

The details on what fields to export and how to treat them is expressed in the definition file. This file follows the YAML format. Here is an example of a definition file:``` resourceType: VirtualMachine # The resource type we're exporting rollupType: AVG # Rollup type: AVG, MAX, MIN, SUM, LATEST, COUNT rollupMinutes: 5 # Time scale granularity in minutes dateFormat: yyyy-MM-dd HH:mm:ss # Date format. See http://tinyurl.com/pscdf9g fields: # A list of fields

CPU fields

  • alias: cpuDemand # Name of the field in the output metric: cpu|demandPct # Reference to a metric field in vR Ops
  • alias: cpuReady metric: cpu|readyPct
  • alias: cpuCostop metric: cpu|costopPct

Memory fields

  • alias: memDemand metric: mem|object.demand
  • alias: memSwapOut metric: mem|swapoutRate_average
  • alias: memSwapIn metric: mem|swapinRate_average

Storage fields

  • alias: storageDemandKbps metric: storage|demandKBps

Network fields

  • alias: netBytesRx metric: net|bytesRx_average
  • alias: netBytesTx metric: net|bytesTx_average

Host CPU

  • alias: hostCPUDemand metric: $parent:HostSystem.cpu|demandmhz # Reference to a metric in the parent.

Guest OS

  • alias: guestOS prop: config|guestFullName # Reference to a property (as opposed to metric)

Host CPU type

  • alias: hostCPUType prop: $parent:HostSystem.cpu|cpuModel # Reference to a metric in a parent
 1
 2### Special properties in the definition file
 3
 4There are a number of special properties that are always available for use in a properties file for getting things like parent resources and resource names.
 5
 6*   $resdId - Internal ID of the current resource.
 7*   $resName - Resource name of the current resource
 8*   $parent - Reference to a parent resource.
 9
10### Referencing parent resources
11
12The syntax for referencing parent resources is as follows:```
13$parent:<Parent Kind>.<metric or property>
14```For example:```
15$parent:HostSystem.cpu|demandmhz
16```Notice that you can stack several $parent keywords. For example, this gets the total CPU demand of a parent cluster based on a VM: \`\`\` $parent:HostSystem.$parent:ClusterComputeResource.cpu|demandmhz \`\`\`
17
18Exporting to SQL
19----------------
20
21As of version 2.0, there is an option to export the data to a SQL database rather than a CSV file. For this reason, we have added a few new settings to the configuration file.
22
23Configuration file settings
24---------------------------
25
26These are the new configuration file settings:
27
28*   *   outputFormat - Must be set to "sql" for SQL export
29    *   connectionString - A JDBC connection string. The format depends on the database vendor.
30    *   username - An optional database username
31    *   password - An optional database password
32    *   databaseType - Type of database. Currently supports postgres, mysql, mssql and oracle. Additional databases can be supported by using the driver option. Note that "driver" and "databaseType" are mutually exclusive.
33    *   driver - The class name of the JDBC driver. Use this setting, along with the JDBC\_JAR environment variable to export to a database type that's not included in the choices for "databaseType".
34    *   sql - The SQL statement to use for inserting data into the database. See below for a full description.
35
36Specifying the SQL statement
37----------------------------
38
39The data is inserted into the database using a user-specified SQL statement (typically an INSERT statement). Variable substitution is done using the metric or property alias preceded by a colon. For example:```
40 sql = "INSERT INTO metrics(timestamp, resname, resId, cpuDemand, memDemand, hostCpuDemand) VALUES (:timestamp, :resName, :resId, :cpuDemand, :memDemand, :hostCPUDemand) ON CONFLICT DO NOTHING"
41   .
42   .
43   .
44# Resource ID
45  - alias: resId
46    prop: $resId
47# CPU fields
48  - alias: cpuDemand
49    metric: cpu|demandPct
50# Memory fields
51  - alias: memDemand
52    metric: mem|guest_demand
53 # Host CPU
54  - alias: hostCPUDemand
55    metric: $parent:HostSystem.cpu|demandmhz
56```Notice how the names in the VALUES clause map to the aliases in the list of fields. The fields "timestamp" and "resName" are always available by default. If you need the resource id, you can define that as a property field mapped to the internal property $resId as shown above. You typically want to use some kind of "upsert" semantics if your database supports it. In the example above, we're using the "ON CONFLICT DO NOTHING" option available in PostgreSQL.
57
58Configuring JDBC drivers
59------------------------
60
61The pre-built binaries come with JDBC drivers for PostgreSQL and MySQL. Due to licensing issues, you will have to download the JDBC drivers for any other database separately. When using a non-bundled database, you need to specify the location of the JDBC jar file in the JDBC\_JAR environment variable. For example, to use a MS SQL database driver you have previously downloaded, simply type:```
62$ export JDBC_JAR=~/tmp/sqljdbc42.jar
63$ ./exporttool.sh -u demouser -p demopassword -H https://10.140.46.21 -i -d ../samples/sqlexport.yaml 

Known issues

    • Very long time ranges in combination with small interval sizes can cause the server to prematurely close the connection, resulting in NoHttpResponseExceptions to be thrown. If this happens, consider shortening the time range. This seems to happen mostly when exporting over a slow connection.
    • Only one parent resource type is supported. This will be fixed in a future release.