Support #12147
[SCP ID :##5939##] : API Gateway down
| Status: | Work Completed-End life cycle | Start date: | July 30, 2021 | |
|---|---|---|---|---|
| Priority: | Immediate | Due date: | ||
| Assignee: |  Siti Balqis Othman | % Done: | 100% | |
| Category: | RIB | Spent time: | - | |
| Target version: | - |
Description
Hi,
Kindly attend below request:
Please help to check, API Gateway hit issue when the thread pool reached maximum and its not release even after multiple times restart the service.
Summary
1. MB not receive response from API gateway during login
2. api gateway log:
"responseCode" : "99",
"responseMessage" : "Rest CUA Error",
3. systemout:
[7/29/21 22:21:25:544 SGT] 000000b1 ThreadPool I WSVR0652W: The size of thread pool "WebContainer" has reached 80 percent of its maximum.
[7/29/21 22:23:32:652 SGT] 00000070 ThreadPool I WSVR0652W: The size of thread pool "WebContainer" has reached 100 percent of its maximum.
4. try restart API Gateway - take too long time with no response due to exceed max JVM set (i guess)
5. kill apigateway and start again - CPU/memory for API Gateway hit maximum (authentication 1 & 2 hit same issue when MB change pointing)
6. increase JVM at authentication server2 to be same as authentication server1, repeat #5 and the result are same
7. turn off MB services, repeat #5 few times, test using postman then ok (not a good practice)
In my point of view even we increase the JVM to any number still will hit same issue. Please check what is the root cause, revisit webpshere setting and overall programming logic because this already happen few times.
Please also advise us the best way to release the thread pool when it hit maximum.
History
#1
Updated by Stephanie Sufrapto about 3 years ago
- Assignee changed from Stephanie Sufrapto to Chun Feng Lim
CF.
Kindly help to look at this issue. Thank you
#2
Updated by Chun Feng Lim about 3 years ago
- % Done changed from 0 to 10
#3
Updated by Chun Feng Lim about 3 years ago
- Status changed from New - Begin Life Cycle to Development / Work In Progress
- % Done changed from 10 to 30
#4
Updated by Ngoh Chee Onn over 2 years ago
- Assignee changed from Chun Feng Lim to Siti Balqis Othman
Hi Balqis,
Please update the status here, since this case related to the CUA deadlock.
Thank you
#5
Updated by Siti Balqis Othman over 2 years ago
- File API Performance Test Report v2.0.docx added
- % Done changed from 30 to 90
Hi team,
Finding issue based on log analysis from IBM, Agrobank DBA team and JMeter load test.
1) IBM and DBA team requested to check on the Long DB Query of table IB_CUST_PROFILE.
2) JMeter result show there are error found caused by database deadlock during login with 50 concurrent user login
Solution:
1) Perform tweaking on the select query from table IB_CUST_PROFILE
- Convert to Native Query
- Only get userId and userState based on userAlias. Previously it get the entire table of IB_USER
- Only get userName, userTacPhone, userAccessNo, IdType, and IdNo and userState based on userAlias. Previously it get the entire table of IB_USER
2) Perform tweaking on the update function in CUA, convert hibernate update query to native query
- Update last login date, (TB_AM_USER)
- Update session to store, (TB_AM_SESSION)
- Update Password to store (TB_AM_PASSWORD)
Result:
The JMeter load test result after perform tweaking can be found from the attachment.
Testing URL:
https://172.16.49.2:9081/agro-api-gateway-uat-deadlock/ibLoginServices/getSecurityImagePhrase
https://172.16.49.2:9081/agro-api-gateway-uat-deadlock/ibLoginServices/getLogin
#6
Updated by Siti Balqis Othman over 1 year ago
- Status changed from Development / Work In Progress to Work Completed-End life cycle
- % Done changed from 90 to 100
Hi Zahir,
Could help to close this ticket. The changes has been deployed together with BNM OTP Migration on 11/06/2023 under Migration ID208