Upgrading MQ on Solaris

Upgrade MQ 7.1.0.5 to MQ 7.5.0.6 on Solaris platforms

Download the WebSphere MQ 7.5 software  from IBM Passport  site

Copy the installable file WS_MQ_SOL_ON_SPARC_V7.5_IFR2_IMG.tar.Z to MQ server

Stop those services(Application Servers etc.) which are connecting to  the same MQ Queue managers to be  upgraded .
You may just stop those listener ports or Activation Specs connecting to  the Queue Manager if it is not difficult to figure out.

Stop the Queue Managers

run dspmqver and dspmqinf and identify which directories to backup

Take the backup of /var/mqm and /opt/mqm as per the output of above commands.
You may need root access to backup /opt/mqm and /var/mqm. This backup will not be useful for  the restoration from upgraded version but may be useful
for the root cause analysis of any problem faced after upgradation.


Goto the path where installable is copied .
gunzip WS_MQ_SOL_ON_SPARC_V7.5_IFR2_IMG.tar.Z
tar -xvf WS_MQ_SOL_ON_SPARC_V7.5_IFR2_IMG.tar

run the ./mqlicense.sh -text_only
Type 1 to accept the license

run ./crtmqpkg 75 where 75 is suffix (as per your choice) 


This is the most important step which I figured out after trial and error.
After running pkgadd, I kept receiving error
ERROR: WebSphere MQ with package name "mqm" is already installed at "/opt/mqm"

Solution to this problem is to
copy the admin install default file to some  location.
cp /var/sadm/install/admin/default /tmp
 and edit the basedir=/opt/mqm-75 in the default file

Run
 pkgadd -a /tmp/default mqm-75

Ignore the warning and continue Installation
You do not appear to have  a  /var/mqm  filesystem
mounted.  It  is  recommended  that you create and
mount   separate   /var/mqm    and    /var/mqm/log
filesystems before installation.

Continue installation? [y,n,q] : y


 When you get the message , Package will be installed in "/opt/mqm" .
 Choose n and enter the required installation path

 Package will be installed in "/opt/mqm"
        Is this OK? [y,n,?,q] n
        Enter required install location:       
/opt/mqm-75
    Package will be installed in "/opt/mqm-75"
        Is this OK? [y,n,?,q] y

Choose 1(WebSphere MQ Server)  to install

                 1)  WebSphere MQ Server
                 2)  Man pages
                 3)  Sample programs
                 4)  WebSphere MQ Client libraries (including Java, JMS and Web Services support)
                 5)  IBM (c) Java runtime for Solaris(TM)
                 6)  IBM (c) Global Security Kit for WebSphere MQ
                 7)  WebSphere MQ Managed File Transfer Service
                 8)  WebSphere MQ Managed File Transfer Tools
                 9)  WebSphere MQ Managed File Transfer Agent
                10)  WebSphere MQ Managed File Transfer Logger
                11)  WebSphere MQ Advanced Message Security
                12)  Spanish message catalog
                13)  French message catalog
                14)  German message catalog
                15)  Japanese message catalog
                16)  Italian message catalog
                17)  Brazilian Portuguese message catalog
                18)  Traditional Chinese message catalog
                19)  Simplified Chinese message catalog
                20)  Korean message catalog
                21)  Russian message catalog
                22)  Hungarian message catalog
                23)  Polish message catalog
                24)  Czech message catalog
               
Thereafter , Installation is self explanatory . You have to continue selecting y and pressing enter .

In the end , after installation is completed .
Execute the following statement when running as the 'root' user:

    su mqm -c "/opt/mqm-75/bin/mqconfig"

Or change user id to mqm and run /opt/mqm-75/bin/mqconfig

The 'mqconfig' command validates that the system configuration satisfies the
requirements for WebSphere MQ, and ensures that the settings for the 'mqm'
user ID are suitably configured.


Run dspmqinst to list installation
You will see two Installations
Installation1 as MQ 7.1 and Installation2 as MQ 7.5
Change your primary installation by running the below commands as root
uid=0(root) gid=0(root)
root@UAT-MQ # ./setmqinst -x -n Installation1
'Installation1' (/opt/mqm) has been unset as the Primary Installation.
root@UAT-MQ # /opt/mqm-75/bin/setmqinst -i -n Installation2
92 of 92 tasks have been completed successfuly.
'Installation2' (/opt/mqm-75) set as the Primary Installation.

After setting Installation 2 as primary installation , there will be symbolic links created in /usr/bin .
Verify all the symbolic links in  /usr/bin/ are correctly pointed to mqm-75 not mqm

Run the below command to migrate qmgrs to new WebSphere mq version with  mqm id
/opt/mqm-75/bin/setmqm -m -n Installation2

Run the below command to see the status and association
dspmq -o installation -o status

In the last step , check whether in the previous installation , were there any mqm local environment variables pointing to
/opt/mqm version , if yes then modify it to /opt/mqm-75 .
Mostly, you may just need to modify mqm .profile file to point to updated mq installation path .



Now coming to applying the fix pack 6 

Copy the fix pack file to MQ server
gunzip 7.5.0-WS-MQ-SolarisSparc64-FP0006.tar.Z
tar -xvf 7.5.0-WS-MQ-SolarisSparc64-FP0006.tar

Create package
./crtmqfp mqm-75

Execute pkgadd -a /tmp/default and select 2

root@UAT-MQ # pkgadd -a /tmp/default

The following packages are available:
  1  mqm-75                 IBM WebSphere MQ for Solaris
                            (sparcv9) 7.5.0.5
  2  mqm-75-07-05-00-06     IBM WebSphere MQ for Solaris - U200647
                            (sparcv9) 7.5.0.6

Thereafter , Installation is self explanatory . You have to continue selecting y and pressing enter .

After fixpack upgrade is completed , start the QMGR
strmqm or
strmqm -x if its a multiinstance QMGR

MQ Channel Sequence reset

Mostly you have to set  channel sequence for a receiver channel .
You have to set it to a number which sender channel is expecting .

The numbers are in the queue manager error logs

 Sequence of Commands:
 
1)
STOP CHL(CHLNAME)
Or 
STOP CHL(CHLNAME)  MODE(FORCE)
2)
Not applicable for receiver Channel
 RESOLVE CHANNEL (XXXXXXX) ACTION(BACKOUT)

 3)
 RESET CHANNEL(XXXXXXXX) SEQNUM(NUMBER)

SEQNUM can be set to 1 if you are running the command for sender channel but for receiver channel, set it to the number which you are seeing in your error logs .

Many times , it may happen , you don't have access to the sender channel MQ infrastructure . It is managed by another organization . In that case , you have to set the sequence number for the receiver channel to the number which you see in the queue manager error logs . Otherwise , you could have just set the sequence number for sender channel and restarted channel at receiver side.

4)

START CHANNEL(XXXXXXX)

MQRC_CONNECTION_BROKEN

 Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN')
 There may be some other exceptions underlying the above exception.
 WebSphere will try to clean up the connection pool after the above exception as per the purge policy defined . The preferred purge policy is entire pool.
This error can be due to several causes . One may not be able to think all the reasons  and put it down  .  Your problem
can be specific to your environment . So you need to find out why is it happening ?
I will discuss the basics here one by one .

First thing to discuss is HBINT parameter at your Channel level.
This is the heart beat interval .
For a more technical explanation , refer to the below link
http://www-01.ibm.com/support/knowledgecenter/#!/SSFKSJ_7.0.1/com.ibm.mq.csqzae.doc/ic11720_.htm
Now , let me explain with my example
I have a Queue Connection factory defined in WebSphere which talks to MQ Queue Manager using a server connection channel .
Now , this HBINT will disconnect the channel when there are no messages to arrive after every HBINT interval .
This value should be less than DISCINT( Disconnect Interval)  value for the channel .
DISCINT is the length of time after which a channel closes down, if no message arrives during that period.
Value of zero for DISCINT  means no disconnect.
I have seen HBINT to be from 60 seconds to 300 seconds . The important thing here is that no device (Firewall , load balancer, Gateway etc. ) should terminate your connection before HBINT interval . 

TCP:
KeepAlive=Yes
Second thing is to set the above parameter in qm.ini file of your Queue Manager. This setting would depend on your OS setting .
For Solaris , the parameter is tcp_keepalive_interval
Command to get the current value of tcp_keepalive_interval is below
ndd -get /dev/tcp tcp_keepalive_interval
Command to set the value is
ndd -set /dev/tcp tcp_keepalive_interval 900000
The time is in mili seconds . The above value means 15 minutes.
The parameter takes effect immediately. If the machine is rebooted the parameter is reset to the default value. To make the change permanent, add the 'ndd' command to the /etc/init.d/inetinit script.
Please refer to the below link to know the commands for other operating systems.
http://www-01.ibm.com/support/docview.wss?uid=swg21216834
This parameter  should not be set to  very low as it will add to the network traffic . Something around 15-25 minutes should be okay . The logic for setting this value comes from the fact that if your firewall is terminating the connection. Then you can set this value to be lower than firewall timeout value to keep the connection open between WebSphere Application Server and MQ.
Below is the most popular link for resolving the MQRC_CONNECTION_BROKEN error.
http://www-01.ibm.com/support/docview.wss?uid=swg21226703


You may also consider increasing the timeout for your device , say loadbalancer for high availability  .  If you have setup a multi instance Queue manager and you are using a F5 for an automated fail over . and if F5 is timing out the connection after every 5 minutes , then you have to increase the time out parameter for F5.

The crux of the discussion is that there should be no abrupt termination of connections in your MQ infrastructure. And if there is , then you have to find out the root cause and fix the error. MQ relies on your network infrastructure for the assured delivery of messages.