Update to 905

Add MQ userid/password to prometheus monitor

Author: ibmmqmet

README.md

@@ -6,7 +6,8 @@ packages including Prometheus, InfluxDB and OpenTSDB.
 
 A minimum level of MQ V9 is required to build this package.
 The monitoring data published by the queue manager is not available before
-that version.
+that version; the interface also assumes availability of
+MQI structures from that level of MQ.
 
 ## MQI Description
 
@@ -30,14 +31,9 @@ have a copy of MQ installed to build against. It uses cgo to access the MQI C st
 installed in the default location on a Linux platform (/opt/mqm) but you can easily change the
 cgo directives in the source files if necessary.
 
-A minimum level of MQ V9 is required. The monitoring data published
-by the queue manager is not available before that version.
-
-Some Windows capability is also included. One constraint in the cgo package is its support
-for path names containing spaces and special characters, which makes it tricky to
-compile against a copy of MQ installed in the regular location. To build these packages I copied
-<mq install>/tools/c/include and <mq install>/bin64 to be under a temporary directory, shown
-in the CFLAGS and LDFLAGS directives.
+Some Windows capability is also included. This has been tested with Go 1.10
+compiler, which now permits standard Windows paths (eg including spaces)
+so the CGO directives can point at the normal MQ install path.
 
 ## Getting started
 
@@ -57,6 +53,10 @@ but a typical directory for the code is /usr/lib/golang.
 
   export GOPATH=$HOME/gowork
 
+* If using a version of Go from after 2017, you must set environment variables
+to permit some compile/link flags. This is due to a security fix in the compiler.
+  export CGO_LDFLAGS_ALLOW="-Wl,-rpath.*"
+
 * Install the git client
 
 ### Windows
@@ -75,6 +75,7 @@ be used on systems without Cygwin installed
 
   set CC=x86_64-w64-mingw32-gcc.exe
 
+* The CGO_LDFLAGS_ALLOW variable is not needed on Windows
 * Install the git client
 * Make sure the MQ include files and libraries are in a path that does
 not include spaces or other special characters, as discussed above.
@@ -121,83 +122,14 @@ There are also no structure handlers for message headers such as MQRFH2 or MQDLH
 
 ## History
 
-08 Jul 2016
-* Initial release
-
-18 Jul 2016
-* Changed structures so that most applications will not need to use cgo to imbed the MQ C headers
-  * Go programs will now use int32 where C programs use MQLONG
-  * Use of message handles, distribution lists require cgo for now
-* Package ibmmq now includes the numeric #defines as a Go file, cmqc.go, for easier use
-* Removed "src/" prefix from tree in github repo
-* Removed need for buffer length parm on Put/Put1
-* Updated comments
-* Added MQINQ
-* Added MQItoString function for some maps of values to constant names
-
-25 Jul 2016
-* Added functions to handle basic PCF creation and parsing
-* Added a monitor command for exporting MQ V9 queue manager data to Prometheus. See
-the [README](cmd/mq_prometheus/README.md) for more details
-
-04 Aug 2016
-* Added a monitor command for exporting MQ data to InfluxDB. See the [README]
-(cmd/mq_influx/README.md) for more details
-* Restructured the monitoring code to put common material in the mqmetric
-package, called from the Influx and Prometheus monitors.
-
-12 Aug 2016
-* Added a OpenTSDB monitor. See the [README](cmd/mq_opentsdb/README.md) for
-more details.
-* Added a Collectd monitor. See the [README](cmd/mq_coll/README.md) for
-more details.
-* Added MQI MQCNO/MQCSP structures to support client connections and password authentication
-with MQCONNX.
-* Allow client-mode connections from the monitor programs
-* Added Grafana dashboards for the different monitors to show how to query them
-* Changed database password mechanism so that "exec" maintains the PID for MQ services
-
-23 Aug 2016
-* Added a collector for Amazon AWS CloudWatch monitoring. See the [README](cmd/mq_aws/README.md)
-for more details.
-
-17 Oct 2016
-* Added some Windows support. An example batch file is included in the mq_influx directory;
-changes would be needed to the MQSC script to call it. The other monitor programs can be
-supported with similar modifications.
-* Added a "getting started" section to this README.
-
-07 Nov 2016
-* Added a collector that prints metrics in a simple JSON format.
-See the [README](cmd/mq_json/README.md) for more details.
-* Fixed bug where freespace metrics were showing as non-integer bytes, not percentages
-
-14 Dec 2016
-* Minor updates to this README for formatting
-* Removed xxx_CURRENT_LENGTH definitions from cmqc
-
-10 Jan 2017
-* Added support for the MQCD and MQSCO structures to allow programmable client
-connectivity, without requiring a CCDT. See the clientconn sample program
-for an example of using the MQCD.
-* Moved sample programs into subdirectory
-
-15 Feb 2017
-* API BREAKING CHANGE: The MQI verbs have been changed to return a single
-error indicator instead of two separate values. See mqitest.go for
-examples of how MQRC/MQCC codes can now be tested and extracted. This change
-makes the MQI implementation a bit more natural for Go environments.
-
-25 Mar 2017
-* Added the metaPrefix option to the Prometheus monitor. This allows selection of non-default resources such as the MQ Bridge for Salesforce included in MQ 9.0.2.
-
+See [CHANGES](https://github.com/ibm-messaging/mq-golang/CHANGES.md).
 
 ## Health Warning
 
 This package is provided as-is with no guarantees of support or updates. There are also no guarantees of compatibility
 with any future versions of the package; the API is subject to change based on any feedback.
 
-##Issues and Contributions
+## Issues and Contributions
 
 For feedback and issues relating specifically to this package, please use the [GitHub issue tracker](https://github.com/ibm-messaging/mq-golang/issues).
 

cmd/mq_aws/config.go

@@ -33,7 +33,7 @@ type mqCloudWatchConfig struct {
 
 	namespace string
 
-	cc mqmetric.ClientConfig
+	cc mqmetric.ConnectionConfig
 
 	interval  string
 	maxErrors int

cmd/mq_coll/config.go

@@ -32,7 +32,7 @@ type mqTTYConfig struct {
 	hostname            string
 	hostlabel           string // Used in the output string
 
-	cc mqmetric.ClientConfig
+	cc mqmetric.ConnectionConfig
 
 	interval string
 

cmd/mq_influx/config.go

@@ -33,7 +33,7 @@ type mqInfluxConfig struct {
 	monitoredQueues     string
 	monitoredQueuesFile string
 
-	cc mqmetric.ClientConfig
+	cc mqmetric.ConnectionConfig
 
 	databaseName    string
 	databaseAddress string

cmd/mq_json/config.go

@@ -29,7 +29,7 @@ type mqTTYConfig struct {
 	monitoredQueues     string
 	monitoredQueuesFile string
 
-	cc mqmetric.ClientConfig
+	cc mqmetric.ConnectionConfig
 
 	interval string
 

cmd/mq_opentsdb/config.go

@@ -33,7 +33,7 @@ type mqOpenTSDBConfig struct {
 	monitoredQueues     string
 	monitoredQueuesFile string
 
-	cc mqmetric.ClientConfig
+	cc mqmetric.ConnectionConfig
 
 	databaseName    string
 	databaseAddress string

cmd/mq_prometheus/README.md

@@ -21,7 +21,7 @@ create a data source in Grafana called "MQ Prometheus" that points at your
 database server, and then import the JSON file.
 
 There is also a script to start the collector so that it processes
-the statistics generated by the MQ Bridge for Salesforce, included in 
+the statistics generated by the MQ Bridge for Salesforce, included in
 MQ V9.0.2
 
 
@@ -77,6 +77,14 @@ of the monitor program. If you want to change the patterns, or new
 queues are defined that match an existing pattern, the monitor must be
 restarted with a `STOP SERVICE` and `START SERVICE` pair of commands.
 
+### Authentication
+This monitor can be configured to authenticate to the queue manager,
+sending a userid and password.
+
+The userid is configured using the `-ibmmq.userid` flag. The password can
+be set either by using the `-ibmmq.password` flag, or by passing it via stdin.
+That allows it to be piped from an external stash file or some other
+mechanism. Command line flags for controlling passwords are not recommended!
 
 ## Configuring Prometheus
 The Prometheus server has to know how to contact the MQ monitor. The

cmd/mq_prometheus/config.go

@@ -19,8 +19,11 @@ package main
 */
 
 import (
+	"bufio"
 	"flag"
+	"fmt"
 	"mqmetric"
+	"os"
 )
 
 type mqExporterConfig struct {
@@ -29,17 +32,23 @@ type mqExporterConfig struct {
 	monitoredQueues     string
 	monitoredQueuesFile string
 
+	// TODO: enable these
+	monitorChannelStatistics bool
+	statisticsQueueName      string
+
 	metaPrefix string
 
-	cc mqmetric.ClientConfig
+	cc mqmetric.ConnectionConfig
 
 	httpListenPort string
 	httpMetricPath string
 	logLevel       string
+	namespace      string
 }
 
 const (
-	defaultPort = "9157" // reserved in the prometheus wiki
+	defaultPort      = "9157" // reserved in the prometheus wiki
+	defaultNamespace = "ibmmq"
 )
 
 var config mqExporterConfig
@@ -61,15 +70,32 @@ func initConfig() {
 	flag.StringVar(&config.metaPrefix, "metaPrefix", "", "Override path to monitoring resource topic")
 
 	flag.BoolVar(&config.cc.ClientMode, "ibmmq.client", false, "Connect as MQ client")
+	flag.StringVar(&config.cc.UserId, "ibmmq.userid", "", "UserId for MQ connection")
+	// TODO: Also enable a different mechanism to read the password
+	flag.StringVar(&config.cc.Password, "ibmmq.password", "", "Password for MQ connection")
+
+	// TODO: turn these on
+	//flag.BoolVar(&config.monitorChannelStatistics, "ibmmq.monitorChannelStatistics", false, "Whether to collect channel stats")
+	//flag.StringVar(&config.statisticsQueueName, "ibmmq.statisticsQueueName", "SYSTEM.ADMIN.STATISTICS.QUEUE", "Which queue holds channel stats")
+	config.monitorChannelStatistics = false
+	config.statisticsQueueName = ""
 
 	flag.StringVar(&config.httpListenPort, "ibmmq.httpListenPort", defaultPort, "HTTP Listener")
 	flag.StringVar(&config.httpMetricPath, "ibmmq.httpMetricPath", "/metrics", "Path to exporter metrics")
 
 	flag.StringVar(&config.logLevel, "log.level", "error", "Log level - debug, info, error")
+	flag.StringVar(&config.namespace, "namespace", defaultNamespace, "Log level - debug, info, error")
 
 	flag.Parse()
 
 	if config.monitoredQueuesFile != "" {
 		config.monitoredQueues = mqmetric.ReadPatterns(config.monitoredQueuesFile)
 	}
+
+	if config.cc.UserId != "" && config.cc.Password == "" {
+		scanner := bufio.NewScanner(os.Stdin)
+		fmt.Printf("Enter password: \n")
+		scanner.Scan()
+		config.cc.Password = scanner.Text()
+	}
 }

cmd/mq_prometheus/exporter.go

@@ -49,10 +49,6 @@ var (
 	gaugeMap = make(map[string]*prometheus.GaugeVec)
 )
 
-const (
-	namespace = "ibmmq"
-)
-
 /*
 Describe is called by Prometheus on startup of this monitor. It needs to tell
 the caller about all of the available metrics.
@@ -185,7 +181,7 @@ func newMqGaugeVec(elem *mqmetric.MonElement) *prometheus.GaugeVec {
 
 	gaugeVec := prometheus.NewGaugeVec(
 		prometheus.GaugeOpts{
-			Namespace: namespace,
+			Namespace: config.namespace,
 			Name:      prefix + elem.MetricName,
 			Help:      elem.Description,
 		},
@@ -195,3 +191,29 @@ func newMqGaugeVec(elem *mqmetric.MonElement) *prometheus.GaugeVec {
 	log.Infof("Created gauge for %s", elem.MetricName)
 	return gaugeVec
 }
+
+/*
+newMqGaugeVec returns the structure which will contain the
+values and suitable labels. For queues we tag each entry
+with both the queue and qmgr name; for the qmgr-wide entries, we
+only need the single label.
+*/
+
+// TODO: Finish this
+/*
+func newMqGaugeVecChl(elem *ibmmq.Statistic) *prometheus.GaugeVec {
+        prefix := "channel_"
+
+        gaugeVec := prometheus.NewGaugeVec(
+                prometheus.GaugeOpts{
+                        Namespace: config.namespace,
+                        Name:      prefix + elem.MetricName,
+                        Help:      elem.Description,
+                },
+                labels,
+        )
+
+        log.Infof("Created gauge for %s", elem.MetricName)
+        return gaugeVec
+}
+*/

cmd/mq_prometheus/main.go

@@ -47,7 +47,7 @@ func main() {
 	log.Infoln("Starting IBM MQ metrics exporter for Prometheus monitoring")
 
 	// Connect and open standard queues
-	err = mqmetric.InitConnection(config.qMgrName, config.replyQ, &config.cc)
+	err = mqmetric.InitConnectionStats(config.qMgrName, config.replyQ, config.statisticsQueueName, &config.cc)
 	if err == nil {
 		log.Infoln("Connected to queue manager ", config.qMgrName)
 		defer mqmetric.EndConnection()
@@ -71,6 +71,12 @@ func main() {
 		allocateGauges()
 	}
 
+	// TODO: continue with the channel stat collection
+	if err == nil && config.statisticsQueueName != "" {
+		mqmetric.InitChlStatistics()
+		//allocateChlGauges()
+	}
+
 	// Go into main loop for handling requests from Prometheus
 	if err == nil {
 		exporter := newExporter()