Image Support #3602

About IBM MQ Depth and performance

Added by Hasan 16 days ago. Updated 3 days ago.

Status:FeedbackDue date:
Priority:High% Done:

0%

Assignee:Hasan
Category:-
Image:WebSphere MQ 8.0.0.4 Your Marketplace Account ID:5495-8803-3553
Operating System:RHEL 7.3 Marketplace:Amazon Web Services
JRE:v1.7 JRE/JDK 64-Bit Customer State:singapore
Instance Type:t2.large Customer Country:singapore

Description

Hi Team,

Currently we are using midvision Image IMB MQ 8.0.0.4 , and we wold like to know about the IBM MQ Performance. Currently we have one of our queues of Depth 1K , and we would like to increase our queue depth from 1k to 500k.

So such a huge queue depth increase( 1k -> 500k ), is there any changes we need to do at queue level ? any other server metrics we need to consider. Kindly review and provide the details.

Regards,
Hasan

AMQERR02.LOG (2 MB) Hasan, 07/29/2020 10:40 pm

qm.ini Magnifier (1.41 KB) Hasan, 07/29/2020 10:40 pm

QM_MQT8.SVRCONN_definition (807 Bytes) Hasan, 07/29/2020 10:42 pm

History

#1 Updated by Mariusz 16 days ago

  • Status changed from New to In Progress
  • Assignee set to Mariusz

#2 Updated by Mariusz 15 days ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Mariusz to Hasan

Hi,
Thank you for using our AMIs from AWS Marketplace.
Our MQ installation comes with the default settings. Performance issues can be related to many factors like incorrect size of Buffer pool, incorrect settings of log buffer etc.
Analyzing your MQ insfrastructure is not a part of the standard support I am afraid but there are many online resources which could possibly help you.
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.mon.doc/q038230_.htm
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.mon.doc/q036142_.htm
https://www.slideshare.net/MarkTaylorIBM/ibm-mq-better-application-performance
http://ibm-messaging.github.io/mqperf/

We also offer consultancy services in case you would like one of our expert to do the analysis of your IBM MQ configuration:
https://www.midvisioncloud.com/support-services/consultancy-services/

Kind regards,
Mariusz Chwalek

#3 Updated by Hasan 13 days ago

Hi Marisuz,

I would like to know what kind of support you are going to provide us. Since we are using Midvision product we don't have any IBM support to look into. That's why we are requesting for this.

As i have updated earlier , we are going to hit High volume of messages for this we conducted Loadtest during this we observed very high CPU Load. Currently we are with m3.large , so we want to scaleup our instance to next higher level instance.

So to scaleup instance , we can increase as other EC2 instance, but i have concern about MQ that there won't be any impact since we are using Midvision Image.

If there are any specific steps to scale up , kindly let us know.

Regards,
Hasan

#4 Updated by Mariusz 13 days ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Hasan to Mariusz

#5 Updated by Hasan 13 days ago

Hasan wrote:

Hi Marisuz,

I would like to know what kind of support you are going to provide us. Since we are using Midvision product we don't have any IBM support to look into. That's why we are requesting for this.

As i have updated earlier , we are going to hit High volume of messages for this we conducted Loadtest during this we observed very high CPU Load. Currently we are with m3.large , so we want to scaleup our instance to next higher level instance.

So to scaleup instance , we can increase as other EC2 instance ? But i have concern about MQ that we are using Midvision Image , with scale up it will impact any OS / MQ level settings ?Please note that we are running our MQ in Multi-instance queue manager with 2 servers for High Availability.

If there are any specific steps to scale up , kindly let us know.

Regards,
Hasan

#6 Updated by Mariusz 13 days ago

  • Assignee changed from Mariusz to Hasan

Hi Hasan,

For our IBM products specifically, MidVision Cloud, via our contract with IBM are also able to raise PMR’s on behalf of our customers for level 3 support requirements where we find a bug or issue that we are unable to resolve ourselves. When you raise a ticket, you can track the progress of the ticket (including the progress of any PMR we raise with IBM on your behalf).

Basically it means that when you use our AMIs with IBM MQ installation you can have a full support from IBM.
To see more take a look at: https://www.midvisioncloud.com/questions/

When it comes to verion 8.0.0.4 - I can see that IBM accounced End of support in 31 of April 2021 (https://www.ibm.com/support/pages/node/711837#EOS1).

What if the third party vendor discontinues support for a product version?

In this situation, MidVision Cloud will continue to provide level 1 and level 2 support until our discontinuation of the product or your cancellation of your subscription, whichever is later. 

However, please be aware that we may no longer be able to raise a support ticket with the vendor or obtain a fix from the third party for these products.

It means that you will still be able to use the AMI from AWS but we will not be able to raise a PMR from that date. So obviously if it is possible I suggest using latest versions (https://aws.amazon.com/marketplace/pp/B07GXX65ZD?qid=1596027430825&sr=0-7&ref_=srh_res_product_title)

When it comes to "know-how" questions regarding performance - note that IBM support also doesn't include analysis of your infrastructure as part of their standard support.
For this kind of cases we can provide consultancy:

What constitutes support versus consultancy?


Generally speaking, where you have an error in your logs or identify a bug or something not working as expected, then we’ll treat it as support and take a look at it and work with you on a resolution. 

Where you are seeking advice on configurations, best practice or asking us to perform configuration, set up, data migration, updates and upgrades, problem or project management or other longer term functions, we’d consider this consultancy.

Finally when it comes to scalling up/scalling out MQ installation. You are not limited in any way to do it. You need to remember that if you scale up using larger instance the price will be higher (we charge per CPU) and if you scale out by creating a cluster - you will be charged for any RUNNING instance you use.

Do not hesitate to ask if there is anything else you would like to know.
Kind regards,
Mariusz Chwalek

#7 Updated by Hasan 13 days ago

Hi Mariusz,

Thanks for the update , now understood about the support.

Can i have reference for pricing details for instances for midvision ?

And also need your help to look into below issue , when our app triggered the load test with concurrent 200 threads , our Queue managed stopped and when i verified from logs found the below error ? could you please suggest what could be the reason and how to remidate this.

----- amqrcsia.c : 1377 -------------------------------------------------------
07/29/20 01:32:28 - Process(6325.108) User(mqm) Program(amqrmppa)
Host(tmqna01) Installation(Installation1)
VRMF QMgr(QM_MQT8)

AMQ9999: Channel 'QM_MQT8.SVRCONN' to host 'ip-192-168-80-69 (192.168.80.69)'
ended abnormally.

EXPLANATION:
The channel program running under process ID 6325 for channel 'QM_MQT8.SVRCONN'
ended abnormally. The host name is 'ip-192-168-80-69 (192.168.80.69)'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 930 --------------------------------------------------------
07/29/20 01:30:42 - Process(4796.157) User(mqm) Program(amqrmppa)
Host(tmqna01) Installation(Installation1)
VRMF QMgr(QM_MQT8)

AMQ9513: Maximum number of channels reached.

EXPLANATION:
The maximum number of channels that can be in use simultaneously has been
reached. The number of permitted channels is a configurable parameter in the
queue manager configuration file.
ACTION:
Wait for some of the operating channels to close. Retry the operation when some
channels are available.

Regards,
Hasan

#8 Updated by Mariusz 13 days ago

Hi Hasan,
I am happy I was able to clarify our support options.
Please take a look at the product page to see prices per instance type https://aws.amazon.com/marketplace/pp/B014HOM7PK?qid=1596035239813&sr=0-20&ref_=srh_res_product_title#pdp-pricing.

We also offer bespoke arrangements for enterprise requirements.
If you are looking for a larger or longer engagement, we can offer bespoke pricing using AWS private offers. Please contact to discuss your particular situation.

Regarding this problem with channels I have created a new support ticket in IBM support portal. I will let you know as soon as we have more information.

Kind regards,
Mariusz

#9 Updated by Mariusz 13 days ago

  • Status changed from In Progress to Feedback

Response from IBM:

Please supply :

1)  definition of channel QM_MQT8.SVRCONN

2) qm.ini

3) complete MQ error logs from both qmgr and client.

Regards, Martin Heywood

IBM MQ Support

#10 Updated by Hasan 13 days ago

Hi Mariusz,

PFA for 2. qm.in and 3. MQ Error log .

1. Channel definition

AMQ8414: Display Channel details.
CHANNEL CHLTYPE
ALTDATE ALTTIME
CERTLABL COMPHDR
COMPMSG DESCR
DISCINT HBINT
KAINT MAXINST
MAXINSTC MAXMSGL
MCAUSER MONCHL
RCVDATA RCVEXIT
SCYDATA SCYEXIT
SENDDATA SENDEXIT
SHARECNV SSLCAUTH
SSLCIPH SSLPEER
TRPTYPE

3. Error at Client side

2020-07-29 01:42:18,632 [Thread-67] INFO services.gets.xbsng.service.XBSSubmissionMQConnector - [XBSSubmissionMQConnector] sendToMQ :: STARTED ::
2020-07-29 01:42:18,632 [Thread-11] ERROR services.gets.xbsng.service.XBSSubmissionMQConnector - [XBSSubmissionMQConnector] sendToCBPQueue :: EXCEPTION ::
2020-07-29 01:42:18,717 [Thread-11] ERROR services.gets.xbsng.service.XBSSubmissionMQConnector - com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QM_MQT8' with connection mode 'Client' and host name '192.168.53.14(1417)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
2020-07-29 01:42:18,717 [Thread-11] DEBUG services.gets.xbsng.service.XBSRespLoaderService - THREAD : Message : - 303

Regards,
Hasan

#11 Updated by Hasan 13 days ago

seems channel definition is not properly aligned , PFA for the same.

BR,
Hasan

#12 Updated by Mariusz 13 days ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Hasan to IBM Support

#13 Updated by Mariusz 12 days ago

  • Status changed from In Progress to Feedback
  • Assignee changed from IBM Support to Hasan

IBM replied:


MQ8 is no longer supported.  MQ8.0.0.4. dates back to 2015.  The last fix pack for MQ 8 was :

Fix Pack 8.0.0.15 for WebSphere MQ v8

https://www.ibm.com/support/pages/node/6220286

I would suggest that you apply fixpack 15 asap and then migrate to MQ 9.2. as soon as practical.

I would suspect that in your test the issue is that the application is not disconnecting cleanly and/or there are some AMQ9209 errors over time for this channel.  These are almost certainly transient network glitches but as you do not have Keepalive = yes in your qm.ini they may well have resulted in stale connections.  A Display conn

https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.ref.adm.doc/q086140_.htm

combined with display channel and display chstatus for QM_MQT8.SVRCONN may well shed more light on this situation for you.

We cannot progress this issue further due to the lack of a valid support contract.  I will close this case 03/08/2020.

Regards, Martin Heywood

IBM MQ Support

Hopefully it will help you solve the problem.

We are also happy to help you in upgrade process. Latest version from v8.0.0.15 will be available soon from AWS Marketplace, but also you have an option to upgrade your existing version by applying a fix pack.
If you are not able to access a fix pack from IBM Fix Central - let us know. We will provide you with the link to the latest version.

Kind regards,
Mariusz Chwalek

#14 Updated by Hasan 8 days ago

Hi Mariusz,

I have verified with our application team , during the load test they are disconneting once it's work done.

And also i confused with your earlier statement "When it comes to verion 8.0.0.4 - I can see that IBM accounced End of support in 31 of April 2021 (https://www.ibm.com/support/pages/node/711837#EOS1) " and IBM Response.

Actually we have observed below error in QM ,but unfortunately there is no update / no progress in this case at all. Kindly suggest how could resolve this issue.

AMQ9513: Maximum number of channels reached.

EXPLANATION:
The maximum number of channels that can be in use simultaneously has been
reached. The number of permitted channels is a configurable parameter in the
queue manager configuration file.

Regards,
Hasan

#15 Updated by Mariusz 8 days ago

  • Assignee changed from Hasan to Mariusz

#16 Updated by Mariusz 8 days ago

  • Assignee changed from Mariusz to Hasan

Hi,

I think there has been some misunderstanding. So 31 of April 2021 date is the end of support of version v8.0 but for the latest versions of 8.0 which is now 8.0.0.15. So basically IBM will keep providing support and help us solve the issue if you are on latest version of v8.0, but they will not do it for version 8.0.0.4 I am afraid.

The problem is that we actually cannot do any progress on that problem as IBM automatically closes the ticket if the version is not supported.
We are happy to help you upgrade your version. We can provide you necessary fix package if you would like to do it on your own or assist you if you would like to use RapidDeploy.

Kind regards,
Mariusz

#17 Updated by Hasan 7 days ago

Hi Mariusz,

For Info , our Application is running on Weblogic 10.3.6 , which i belive MQ 9.2 is not compatible. can you confirm on this please .

And also we would like to know how to increase the channel size which will address our below issue.

AMQ9513: Maximum number of channels reached.

Regards,
Hasan

#18 Updated by Mariusz 7 days ago

Hi,

1. I will check compatibility as get back to you with more information. What we would suggest is to upgrade your version at least to 8.0.0.15 so it will give you more time for major upgrade and be able to receive support from IBM until 31 of April 2021

2. Could you possibly take a look at this case and see if it can help:
https://www.ibm.com/support/pages/finding-number-running-channels

Example on how to change the maximum channels for a queue manager from the default 100 to 300:

Step 1: Modify the qm.ini of the queue manager:
CHANNELS:
   MaxChannels=300

Step 2: Stop and restart the queue manager for the change in the qm.ini to take effect.

Thanks,
Mariusz Chwalek

#19 Updated by Hasan 3 days ago

Hi Mariusz Chwalek,

Thanks for updating the details about channels. we will update and share the results.

Meanwhile when we are running our load test , we are observing high load average on server and message consuming is slow.

Currnelty we are running our instance with m3.large and we are doing loadtest with 500k messages per hour. Could you please confirm what is the recommended EC2 instance to accommodate such load.

Regards,
Hasan

Also available in: Atom PDF