Project

General

Profile

Image Support #3822

JBOSS EAP 7.2 : unable to suspend the server.

Added by Shweta 2 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
High
Assignee:
Due date:
% Done:

0%

Operating System:
Linux
JRE:
Not Applicable
Instance Type:
Not Applicable
Your Marketplace Account ID:
2872-5568-4589
Marketplace:
Amazon Web Services
Customer State:
Sydney
Customer Country:
New South Wales

Description

Hi team,

We tried to suspsend the jboss services on production server using jboss cli command. We have continuous incoming requests over jms and http into the application. One request takes 15 second to complete so we specified timeout as 30 sec for jboss to wait for all active requests to complete.

We had enable graceful transaction using following command : /subsystem=ejb3:write-attribute(name=enable-graceful-txn-shutdown,value=true)

Used below command to suspend the server.
:suspend(timeout=30)

But even after specified timeout , server was in PRE_SUSPEND phase only. We waited for 30 minutes but still server was not SUSPENDED.

[:443 /] :read-attribute(name=suspend-state) {
"outcome" => "success",
"result" => "PRE_SUSPEND"
}

We could see from the logs that server suspension started : 2021-07-09 22:05:52,819 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 30000 ms timeout.

Could you please look into the issue.

We referred this RED HAT URL : https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.1/html/configuration_guide/starting_and_stopping_jboss_eap#graceful_shutdown_suspend

Thanks & Regards,
IPAM2 Application Support


Files

standalone-full - Copy.7z (7.23 KB) standalone-full - Copy.7z Shweta, 07/13/2021 07:25 AM
server.7z (4.5 MB) server.7z Shweta, 07/13/2021 07:45 AM
start-MDB.cli (1.51 KB) start-MDB.cli Shweta, 07/20/2021 06:08 AM
stop-MDB.cli (1.5 KB) stop-MDB.cli Shweta, 07/20/2021 06:08 AM
server log file.zip (571 KB) server log file.zip Shweta, 07/20/2021 06:09 AM
#1

Updated by Shweta 2 months ago

Please find the attached jboss configuration standalon-full.xml and server.log for your reference.

#2

Updated by Mariusz 2 months ago

  • Status changed from New to In Progress
#3

Updated by Mariusz 2 months ago

  • Assignee changed from Mariusz to Red Hat Support
#4

Updated by Mariusz 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Hi,

Thank you for contacting Red Hat.

I am going through the attachments. Meanwhile would you like to try with graceful shutdown CLI command instead of suspend ?

If the results are same, please capture 8-10 thread dumps after executing the CLI command. Refer article [1] to capture thread dumps.

Along with the collected thread dumps, please share the latest server.log.

[1] https://access.redhat.com/solutions/2196211

Regards,
Varsha

#5

Updated by Mariusz 2 months ago

Hi,

In addition you can pause the JMS queue too befause suspending server

Here is the CLI command :

[standalone@localhost:9990 /] /subsystem=messaging-activemq/server=default/jms-queue=testQueue:pause()

or if you have MDB, you can pause that too. Details are in https://access.redhat.com/solutions/428023.

Please try the shutdown and pausing the queue and share results.

Regards,
Varsha

#6

Updated by Shweta 2 months ago

Hi Team,

We have 13 MDBs in our application. As mentioned in the above updates , stop-delivery() command seems to be applicable for single MDBs at a time. We are required to stop delivery of all the MDBs in one go. Is there any option to do it ?

Currently , we have configured delivery-group in standlone-full.xml and making the delivery-group INACTIVE by modifying value of ACTIVE to FALSE seems to be working.

But We are facing issue with suspending the server. Could you please let us know why command is not working ?
Please refer the attached server.log

#7

Updated by Mariusz 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#8

Updated by Mariusz 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Hi,

We are required to stop delivery of all the MDBs in one go. Is there any option to do it ?

- We can use CLI batch script which contains CLI command to stop all MDBs.

Currently , we have configured delivery-group in standlone-full.xml and making the delivery-group INACTIVE by modifying value of ACTIVE to FALSE seems to be working.

- The log shows that the onMessage is invoked after executing suspend command too.

,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:13:07,628 INFO [IPAMRPA] (default-threads - 2) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6B2CF21:150555|QueryIPAddressDetailsMDB|onMessage|Inside the finally block of QueryIPAddressMDB","2021-07-09T22:13:07.628+1000",,,,,,,,,,,,,22,9,13,july,7,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||_____",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1626.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:13:07,628 ERROR [IPAMRPA] (default-threads - 2) ERROR|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6B2CF21:150555|QueryIPAddressDetailsMDB|onMessage|RPA Exception Details: IP Address Does Not exist","2021-07-09T22:13:07.628+1000",,,,,,,,,,,,,22,9,13,july,7,friday,2021,local,,,,"nix-all-logs nix_errors",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,__[]_()_|:----.:|||__:_____",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1626.in.telstra.com.au",,,error,error,24,,0,,,,,,,,

,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:08:24,574 INFO [IPAMRPA] (default-threads - 26) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC639C:150765|QueryIPAddressDetailsMDB|onMessage|Message is picked up from request queue","2021-07-09T22:08:24.574+1000",,,,,,,,,,,,,22,9,8,july,24,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||______",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:08:24,574 INFO [RPAMESSAGE] (default-threads - 26) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC639C:150765|QueryIPAddressDetailsMDB|onMessage| <en:Envelope xmlns:com=""http://com.telstra.ipam.webservice"" xmlns:en=""http://schemas.xmlsoap.org/soap/envelope/""><en:Body><com:queryIPAddressDetails> <com:sourceSystem>OM-BIGPOND</com:sourceSystem> <com:customer>BigPond DSL Static</com:customer> <com:ipAddress>22.1.1.23</com:ipAddress> </com:queryIPAddressDetails></en:Body></en:Envelope>","2021-07-09T22:08:24.574+1000",,,,,,,,,"http://com.telstra.ipam.webservice",,,,22,9,8,july,24,friday,2021,local,,"http://schemas.xmlsoap.org/soap/envelope/","UTF-8","nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||<::=",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,"1.0",,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:08:24,574 INFO [RPAMESSAGE] (default-threads - 26) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC639C:150765|QueryIPAddressDetailsMDB|onMessage|Received a Byte Message","2021-07-09T22:08:24.574+1000",,,,,,,,,,,,,22,9,8,july,24,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||___",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:08:24,574 INFO [RPAMESSAGE] (default-threads - 26) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC639C:150765|QueryIPAddressDetailsMDB|onMessage|Message is picked up from request queue","2021-07-09T22:08:24.574+1000",,,,,,,,,,,,,22,9,8,july,24,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||______",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:08:24,574 DEBUG [IPAMRPA] (default-threads - 26) DEBUG|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC639C:150765|QueryIPAddressDetailsMDB|onMessage|Threshold TimeOut Value (in ms): 1625832519567","2021-07-09T22:08:24.574+1000",,,,,,,,,,,,,22,9,8,july,24,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,__[]_()_|:----.:|||___():",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:08:24,573 INFO [IPAMRPA] (default-threads - 26) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC639C:150765|QueryIPAddressDetailsMDB|onMessage|START :- Invoking onMessage()","2021-07-09T22:08:24.573+1000",,,,,,,,,,,,,22,9,8,july,24,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||_:-__()",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:07:09,477 INFO [IPAMRPA] (default-threads - 48) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6D08E17:150841|QueryIPAddressDetailsMDB|onMessage|END :- Invoking onMessage()","2021-07-09T22:07:09.477+1000",,,,,,,,,,,,,22,9,7,july,9,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||_:-__()",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:07:09,477 INFO [IPAMRPA] (default-threads - 48) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6D08E17:150841|QueryIPAddressDetailsMDB|onMessage|Message is sent to Response queue","2021-07-09T22:07:09.477+1000",,,,,,,,,,,,,22,9,7,july,9,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||_____",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1629.in.telstra.com.au",,,,,24,,0,,,,,,,,

:
:
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:06:38,498 INFO [IPAMRPA] (default-threads - 40) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC07C9:150444|QueryIPAddressDetailsMDB|onMessage|Inside the finally block of QueryIPAddressMDB","2021-07-09T22:06:38.498+1000",,,,,,,,,,,,,22,9,6,july,38,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||_____",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1627.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:06:38,498 ERROR [IPAMRPA] (default-threads - 40) ERROR|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6AC07C9:150444|QueryIPAddressDetailsMDB|onMessage|RPA Exception Details: IP Address Does Not exist","2021-07-09T22:06:38.498+1000",,,,,,,,,,,,,22,9,6,july,38,friday,2021,local,,,,"nix-all-logs nix_errors",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,__[]_()_|:----.:|||__:_____",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1627.in.telstra.com.au",,,error,error,24,,0,,,,,,,,

,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:06:36,461 INFO [IPAMRPA] (default-threads - 37) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6D0732A:150441|QueryIPAddressDetailsMDB|onMessage|Message is picked up from request queue","2021-07-09T22:06:36.461+1000",,,,,,,,,,,,,22,9,6,july,36,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||______",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1627.in.telstra.com.au",,,,,24,,0,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:06:36,461 INFO [RPAMESSAGE] (default-threads - 37) INFO|ID:PRE002-EPIC-EMS-SERVER-7.530B60C374EE6D0732A:150441|QueryIPAddressDetailsMDB|onMessage| <en:Envelope xmlns:com=""http://com.telstra.ipam.webservice"" xmlns:en=""http://schemas.xmlsoap.org/soap/envelope/""><en:Body><com:queryIPAddressDetails> <com:sourceSystem>OM-BIGPOND</com:sourceSystem> <com:customer>BigPond DSL Static</com:customer> <com:ipAddress>22.1.1.23</com:ipAddress> </com:queryIPAddressDetails></en:Body></en:Envelope>","2021-07-09T22:06:36.461+1000",,,,,,,,,"http://com.telstra.ipam.webservice",,,,22,9,6,july,36,friday,2021,local,,"http://schemas.xmlsoap.org/soap/envelope/","UTF-8","nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[]_()_|:----.:|||<::=",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1627.in.telstra.com.au",,,,,24,,0,,,,,,"1.0",,

,,,,,,,,,,,,,,,,,,,,,,,,,,"2021-07-09 22:05:52,819 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 30000 ms timeout.","2021-07-09T22:05:52.819+1000",,,,,,,,,,,,,22,9,5,july,52,friday,2021,local,,,,"nix-all-logs",,,,,,,"ipam2-prod-rpa","pcf-ipam2",,1,,,,,,,,,,"--_::,___[...]_(--_-_)_:______.",,,,,,,"/opt/jboss-eap-7.2/standalone/log/server.log",jboss,"lxapp1626.in.telstra.com.au",,,,,24,,0,,,,,,,,

[logs are in revers format]

Hence it seems the making delivery-group INACTIVE does not stop message consumption by MDB. Please stop them explicitly and then try to suspend server.

Did you try with shutdown command ? If yes, what is the result ?

Regards,
Varsha

#9

Updated by Shweta 2 months ago

Hi ,
Thanks for your update but we still have few doubts regarding suspending the server.

1. Why the jboss server could not be suspended even after waiting for more than specified timeout.
2. Could you please share the CLI command to stop all MDBs in one go -We can use CLI batch script which contains CLI command to stop all MDBs. ?

Currently we could not test shutdown command on production as it would cause outage.

We were stopping the MDBs for the very first time . For that we had to add delivery-group annotation (@DeliveryGroup("rpa-mdb-delivery-group")) in all 13 MDBs at code level . Making delivery-group INACTIVEwould have been reflected only after deploying the modified ear. That's the reason why you could see JMS messages coming into the server as the ear was not deployed at that time.

We wanted to suspend the server first to stop new incoming requests > make the delivery-group inactive in standalone-full.xml > and then deploy the ear so that inbound jms messages wont be consumed as soon as MDBs deployed. We had planned to make the delivery-group Active to start the delivery.

We are looking forward to avoid shutdown as new ear deployment required jboss services up and running so what is the point of stopping the server ?

We followed below steps:
1. Suspend the server ( to stop new incoming requests into the server)
2. Stop delivery to MDB by making delivery-group inactive ( so that when new ear deploys ,it wont be consuming new inbound messages)
3. Deploy new ear
4. Start the delivery to MDBs by making delivery-group active
5. Resume the server

Please share the requested information.

#10

Updated by Mariusz 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#11

Updated by Mariusz 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Hello,

I am not certain without testing whether the workflow below will work without any kind of issue. As far as I know, there is no way to pause all MDBs at once, so it would probably be necessary to stop MDBs for each deployment individually.

I see you are using an external JMS provider (Tibco EMS). Is it possible to suspend dispatch or suspend the queues on the broker end? Then the queues could be resumed after deployment of changes. Would that accomplish what you are looking to achieve?

If that won't work, I can check if the generic resource adapter supports any similar operation to pause dispatch, but I am not sure if it does.

Best Regards,

Duane S. Hawkins
Software Maintenance Engineer

#12

Updated by Shweta 2 months ago

Thanks for your latest update.
We think stop delivery to MDB by making delivery-group is working because when we restarted the jboss services we did not receive jms messages as soon as it deployed but our main concern is why the jboss suspend command is not working on production.

please provide the answers to below questions.

1. Why the jboss server could not be suspended even after waiting for more than specified timeout.
2. Could you please check the server.log and let us know why the server got stuck in PRE_SUSPEND state ?

We could see in the log that server suspend started but it got stuck in PRE_SUSPEND state only for half an hour.
--
2021-07-09 22:05:52,819 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0211: Suspending server with 30000 ms timeout.
--
[:443 /] :read-attribute(name=suspend-state) {
"outcome" => "success",
"result" => "PRE_SUSPEND"
}

Thanks,
IPAM2

#13

Updated by Mariusz 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#14

Updated by Mariusz 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Hi,

Create a CLI batch file like stop-MDB.cli :

  1. Start batching commands
    batch

/deployment=MDBStopDeliveryApplication.jar/subsystem=ejb3/message-driven-bean=TestMDB1:stop-delivery()
/deployment=MDBStopDeliveryApplication.jar/subsystem=ejb3/message-driven-bean=TestMDB2:stop-delivery()
/deployment=MDBStopDeliveryApplication.jar/subsystem=ejb3/message-driven-bean=TestMDB3:stop-delivery()
/deployment=MDBStopDeliveryApplication.jar/subsystem=ejb3/message-driven-bean=TestMDB4:stop-delivery()

#similar way for other MDBS

  1. Run the batch commands
    run-batch

$ EAP_HOME/bin/jboss-cli.sh --connect --file=file_path/stop-MDB.cli

Try this is lower environment and share the results.

Meanwhile I am going through yesterdays comments.

Regards,
Varsha

#15

Updated by Shweta 2 months ago

Hi Team,

We will test CLI batch in lower environment and will let you know the results. Meanwhile could you please response to update #12 . We have not received the answers to requested questions yet.

Thanks,
IPAM2

#16

Updated by Mariusz 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#17

Updated by Shweta 2 months ago

Hi Team,

We have created this CLI batch job and it is running without error but it is not giving us the expected results.

Our requirement is to have MDBs disabled at the time of new ear deployment so that new inbound messages wont be consumed by MDBs as soon as it is deployed. With the approach you mentioned we are not able to get that result unless we set delivery-active of 13 queues to "false" before the deployment.

Please look into the below steps and let us know if we can go ahead with this approach
1. Stop delivery of MDB using cli job
2. Set the delivery-active attribute of each MDB to "false" > this can be achieved by adding delivery-group in standalone-full.xml which can be set to "false" to stop MDBs to consume messages as soon as it is deployed.
3. Deploy the new ear using CLI
4. Start the delivery of MDB using cli job
5. Set the delivery-active back to true

Also , please let us know how can we stop new incoming messages via HTTP ?

Please find the attaches stop-MDB.cli , start-MDB.cli , server.log for your reference.

Thanks,
IPAM2

#18

Updated by Mariusz 2 months ago

Hi,
Your comment and attached files have been shared with the RedHat support team.
Regards,
Mariusz

#19

Updated by Mariusz about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Hi,

Our requirement is to have MDBs disabled at the time of new ear deployment so that new inbound messages wont be consumed by MDBs as soon as it is deployed.

- If your requirement is MDB should not consume messages as soon as deployed or on server restart, then you can use the second option mentioned in article i.e

Use the "<assembly-descriptor>" child tag "<delivery>" to "false" inside the "$MDB_JAR/META-INF/jboss-ejb3.xml"

    <assembly-descriptor>
        <d:delivery>
            <ejb-name>TestMDB</ejb-name>
            <d:active>false</d:active>
        </d:delivery>
    </assembly-descriptor>

The MDB will not consume messages until we start them using CLI.

[standalone@localhost:9999 /] /deployment=MDBStopDeliveryApplication.jar/subsystem=ejb3/message-driven-bean=TestMDB:start-delivery() {"outcome" => "success"}

Also , please let us know how can we stop new incoming messages via HTTP ?

- In EAP 6, we can pause the connector. Need to check if there is any feature present in EAP 7 for undertow.

Are you using any load balancer in front of EAP servers ?

can we have a call tomorrow to discuss more on your requirement ?

Regards,
Varsha

#20

Updated by Shweta about 2 months ago

Hi Team,

We already have mentioned in previous updates that we are not able to suspend the jboss server in production .Could you please comment on that ?

And Reg stop delivery to MDBs , As mentioned in Red Hat solutions , we can configure <delivery> "false" in two different ways:
1. configure it in <assembly-descriptor> inside the "$MDB_JAR/META-INF/jboss-ejb3.xml"
2. Configure it using annotation @DeliveryGroup("rpa-mdb-delivery-group") inside MDB class on application code side and set delivery-group to false in standalone-full.xml

Second approach is working fine for us. We are able to deploy the ear without messages being consumed by MDBs as soon as it deployed.

Could you please provide the answers to below questions. We have requested for answers for several time in previous updates also , please provide below information:

1. Why the jboss server could not be suspended even after waiting for more than specified timeout.
2. Could you please check the server.log and let us know why the server got stuck in PRE_SUSPEND state ?We

Also , please let us know your convenient time so that we will schedule a call to discuss this further.

Thanks,
IPAM2

#21

Updated by Mariusz about 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#22

Updated by Mariusz about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Reply from RedHat:

Hi,

I work in EMEA hours. I can request my colleague to join the call in Sydney hours. 

Please confirm the time from your side. 

Regards,
Varsha

Could you possibly suggest the time which suits you, so Varsha or someone else can join the call?

Thanks,
Mariusz

#23

Updated by Shweta about 2 months ago

Hi ,

We are available anytime in between 2 pm to 11 pm AEST .I will schedule a WebEx meeting as per your availability.

Please let us know.

Thanks,
IPAM2

#24

Updated by Mariusz about 2 months ago

Hi,

Let us know when we can have call. We can discuss on link1.

[1] meet.google.com/myc-ayxu-dmv

Regards,
Chandra Shekhar

#25

Updated by Shweta about 2 months ago

Hi Team,

Can we have a call at 4.30 PM AEST today ?

Thanks,
IPAM2

#26

Updated by Shweta about 2 months ago

Hi Mariusz,

Could you please ask Red Hat Team to schedule meeting at 10.30 PM AEST .

Thanks,
IPAM2

#27

Updated by Mariusz about 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#28

Updated by Mariusz about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Reply from RedHat:

Apologies. I missed your update. 

Let me know if you are available tomorrow.
#29

Updated by Mariusz about 2 months ago

Reply from Red Hat:

Here are the answers of your questions :

1. Why the jboss server could not be suspended even after waiting for more than specified timeout.

- The first shared logs shows the MDB was still processing new messages even after requesting suspend. That might be one of root cuase the server did not suspend.

In second attached log shows, server suspended.

From 22nd July's update my understanding is, you are able to suspend the servers after disabling delivery-group. Please correct me if I am misunderstanding.

2. Could you please check the server.log and let us know why the server got stuck in PRE_SUSPEND state ?

- In 14th July's Comment, I shared my observation. The logs shows MDBs were processing the messages. The onMessage was invoked by MDBs even after suspend was called.

Are you still facing same issue even after stopping MDB delivery ? If yes, please capture 8-10 thread dumps after filling suspend command. Attach them to case adlong with latest logs.

Please confirm tomorrow's call time. My colleague Abhijeet will assist you in AEST hours.

Regards,
Varsha

#30

Updated by Shweta about 2 months ago

Please reschedule a call at 6 pm AEST today so that we can have further discussion.

Regards,
IPAM2

#31

Updated by Mariusz about 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#32

Updated by Mariusz about 2 months ago

Received from RedHat:

Hi,

Please reschedule a call at 6 pm AEST today so that we can have further discussion.

If I am not wrong, 6 PM AEST is 1:30 PM IST. Current time in Sydney is 9:39 pm and in India it is 5:10 PM.

Do you mean 6 PM IST ?

I received update on case at 9:22 PM AEST. Please update the case at least 1 hour before call.

Regards,
Varsha

#33

Updated by Mariusz about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta
#34

Updated by Shweta about 2 months ago

Hi Team,

We are not getting a confirmation from your end whenever we are suggesting time for a meeting hence kindly schedule a call as per your availability and let us know the timing. I already have mentioned several times that we are available anytime in between 2 pm AEST to 11 pm AEST.

Thanks,
IPAM2

#35

Updated by Mariusz about 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#36

Updated by Mariusz about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta

Reply from RedHat:

Hi,

Let's have call tomorrow at 6 PM AEST [1:30 PM IST]. I have transferred the ownership to one of my colleague who works in APAC hours. You can schedule the call in APAC hours too.

Please confirm availability regarding tomorrow's 6 PM AEST call and share the meeting details.

Regards,
Varsha

#37

Updated by Shweta about 2 months ago

Hi Team,

We are okay with the mentioned time. Please join the following meeting at scheduled time : https://infosys.webex.com/meet/malvika_singh

Let us know in case of any issues.

Thanks,
IPAM2

#38

Updated by Mariusz about 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Shweta to Red Hat Support
#39

Updated by Mariusz about 2 months ago

Reply from RedHat:

Hi,

I am waiting for you on call, can you please join.

Thanks
Abhijeet.

#40

Updated by Mariusz about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Red Hat Support to Shweta
#41

Updated by Mariusz about 2 months ago

Message from RedHat:

Hi,

I was waiting for you on call from last 2o minutes but no one has joined the call hence I dropped from the call now. Can you please confirm your availability for call again today or tomorrow.

Thanks
Abhijeet.

#42

Updated by Shweta about 2 months ago

Hi Team,

Sorry we could not connect at scheduled time.

As you mentioned in comment #29 > The first shared logs shows the MDB was still processing new messages even after requesting suspend. That might be one of root cuase the server did not suspend.

But that is what we want to know that why the MDBs were still processing new messages. As mentioned in the Red Hat solutions , server suspend should stop all new incoming requests and should wait for timeout specified in the command to complete all ongoing requests.

Even after time specified in the command , new requests were coming into the server and we could see that server was in PRE_SUSPENDED state.

We know it works in our test environment as there is less traffic of requests but in production server is not getting suspended.

1. Is there any solution to this issue?
2. How should we suspend the server successfully ?
3. If possible could you please explain how server suspend should work ?

Please note that we can not test anything on our production environment without outage change activity. This issue we are getting in production only. We have around 6000 requests coming into the server daily.

Please provide the information of above questions. Please let me know if you want to reschedule the meeting.

Thanks,IPAM2

#43

Updated by Mariusz about 2 months ago

Reply from RedHat

Hello David,

I will try to answer the questions as best as I can from a Messaging perspective, and I have added the JCA and JBoss teams to the ticket for investigation into the shutdown behavior:

Why do you see messages continue to process as the server is shut down?

I can think of a couple of reasons for this:

1. The Messaging / JCA subsystem operates on an executor pattern. An executor consists of a thread pool and a work queue. When the threads in the thread pool are busy, items that cannot be processed immediately (because there are no free threads or the MDB instance pool is full) are added to a work queue. Work in the work queue will continue to process even if message flow stops, until the work queue is empty.

2. Many messaging systems support the concept of prefetch or consumer windows. If I read the Tibco documentation correctly, Tibco EMS does support prefetch and prefetch is automatically enabled in the absence of explicit configuration. Messages that are prefetched are buffered at the client - below the layer of application code (it is in the Tibco client), so they appear to "arrive" at the client when they are popped from the prefetch buffer.

One potential mitigation for the issues above are to make sure your jca thread pool and mdb instance pool, as well as the maxSession MDB attribute are adequately sized to handle the larger message load without queuing work.

You can try adjusting the core threads and work queue sizes in your jca subsystem to see if work processes quicker:


        <subsystem xmlns="urn:jboss:domain:jca:5.0">
            <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
            <bean-validation enabled="true"/>
            <default-workmanager>
                <short-running-threads>
                    <core-threads count="50"/>
                    <queue-length count="50"/>
                    <max-threads count="50"/>
                    <keepalive-time time="10" unit="seconds"/>
                </short-running-threads>
                <long-running-threads>
                    <core-threads count="50"/>
                    <queue-length count="50"/>
                    <max-threads count="50"/>
                    <keepalive-time time="10" unit="seconds"/>
                </long-running-threads>
            </default-workmanager>
            <cached-connection-manager debug="true" error="false"/>
        </subsystem>

Similarly, check that the strict-max-pool values for your ejb subsystem are sized high enough to accommodate the number of MDB instances you expect to support in parallel:


            <pools>
                <bean-instance-pools>
                    <strict-max-pool name="mdb-strict-max-pool" max-pool-size="60" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                    <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                    <strict-max-pool name="mdb-strict-max-pool-rpa" max-pool-size="60" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                </bean-instance-pools>
            </pools>


My first inclination here would be to try bumping up the max-thread-counts in your jca core threads and see if this prevents work backlogs.

Finally, if distributed transactions are involved, the container may be waiting for transactions to timeout or complete as part of the shutdown. I will leave it to the JCA team to confirm this.

Best Regards,

Duane S. Hawkins
Software Maintenance Engineer

[1] https://docs.tibco.com/pub/ems/8.6.0/doc/html/GUID-7687D8CC-02F9-4896-A0DC-80B1DFCA35B0.html

#44

Updated by Mariusz about 2 months ago

Another message from RedHat:

To move forward most effectively, it would be useful to get thread dumps as suggested by Varsha on Monday.

This would give us a sense of what work is continuing, whether something is blocked, etc. From that, we may be able to recommend something more specific to target what might be blocking the suspend.

Regards,
Stephen

#45

Updated by Mariusz about 1 month ago

Message from RedHat:

Hello,

It has been 7 days since we requested additional information on your inquiry. Please review previous case emails and/or case history, and provide the requested information to allow us to continue solving this issue.

Regards,
Red Hat Customer Experience and Engagement

#46

Updated by Mariusz about 1 month ago

  • Status changed from Feedback to Marked for Closure

This ticket has been archived by Red Hat support team:
Hello,

It has been 12 days since our last attempt to reach you regarding this case. As we have not received a response, your case will be archived, but will be reopened if/when you contact us or update the case comments.

We hope your issue is resolved and hope you had a pleasant support experience.

Regards,
Red Hat Customer Experience and Engagement

#47

Updated by Mariusz about 1 month ago

  • Status changed from Marked for Closure to Closed

Also available in: Atom PDF