Support #4563

Avatar?id=1733&size=50

[SCP ID :##2342##] : Bii - RCAS slow respond issue

Added by Ahmad Hazri about 10 years ago. Updated about 10 years ago.

Status:Closed - End of life cycleStart date:July 21, 2014
Priority:NormalDue date:August 25, 2014
Assignee:-% Done:

0%

Category:-Spent time:-
Target version:-

Description

Please attend this query:

Error captured:


    [6/19/14 9:22:33:363 WIT] 00000011 TimeoutManage I   WTRN0006W: Transaction 00000146B1ECEED100000001000002CA8BB72C496215116DFE295D960409CA31E29E622D00000146B1ECEED100000001000002CA8BB72C496215116DFE295D960409CA31E29E622D00000001 has timed out after 120 seconds.
    [6/19/14 9:22:33:372 WIT] 00000011 TimeoutManage I   WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WebContainer : 1,5,main]. The stack trace of this thread when the timeout occurred was: 
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    oracle.net.ns.Packet.receive(Packet.java:283)
    oracle.net.ns.DataPacket.receive(DataPacket.java:103)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
    oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
    oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1010)
    oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1185)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
    com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteQuery(WSJdbcPreparedStatement.java:1082)
    com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeQuery(WSJdbcPreparedStatement.java:703)
    org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:264)
    org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeQuery(LoggingConnectionDecorator.java:903)
    org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:262)
    org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1599)
    org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:252)
    org.apache.openjpa.jdbc.sql.SelectImpl.executeQuery(SelectImpl.java:509)
    org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:384)
    org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:339)
    org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:420)
    org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:230)
    org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:220)
    org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:206)
    org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94)
    org.apache.openjpa.kernel.QueryImpl.singleResult(QueryImpl.java:1284)
    org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1221)
    org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:990)
    org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:848)
    org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:779)
    org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:525)
    org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:254)
    org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:317)
    net.penril.cas.service.impl.AuthenticationServiceImpl.verifyApplicationOwnership(AuthenticationServiceImpl.java:124)
    net.penril.cas.service.EJSRemote0SLnet_penril_cas_service_Authentic_b2eac087.verifyApplicationOwnership(EJSRemote0SLnet_penril_cas_service_Authentic_b2eac087.java)
    sun.reflect.GeneratedMethodAccessor489.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    com.ibm.rmi.util.ProxyUtil$4.run(ProxyUtil.java:612)
    java.security.AccessController.doPrivileged(Native Method)
    com.ibm.rmi.util.ProxyUtil.invokeWithClassLoaders(ProxyUtil.java:608)
    com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1165)
    $Proxy48.verifyApplicationOwnership(Unknown Source)
    net.penril.cas.service._AuthenticationService_Stub.verifyApplicationOwnership(_AuthenticationService_Stub.java)
    net.penril.cas.ws.impl.BaseServices.verifyApplicationAccess(BaseServices.java:33)
    net.penril.cas.ws.CASServices.wsVerifySimpleLogin(CASServices.java:184)
    sun.reflect.GeneratedMethodAccessor207.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
    org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
    org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
    org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
    org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
    org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
    org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
    org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
    org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
    org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
    com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
    com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:944)
    com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
    com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
    com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
    com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
    com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
    com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
    com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
    com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
    com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
    com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:277)
    com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
    com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
    com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
    com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
    com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
    com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
    com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)

History

#1 Updated by Zahir Abd Latif about 10 years ago

  • Assignee changed from Hartanto Boby to Anonymous

Dear Lee Lee,

There is no update on this case since July 21, please help to follow up with to see we closed this case or update the status.

#2 Updated by Anonymous about 10 years ago

  • Due date set to August 25, 2014
  • Status changed from New - Begin Life Cycle to Closed - End of life cycle

On 25/8/2014 4:14 PM, Parlindungan Lumban Tobing wrote:

Hello support,

Please kindly close this PMR. After collaborating and monitoring the prod server we figured out that the issue is not happen until now.

What we did is change the datasource pool minimum value to 0.

Thanks and really appreciate you help.

At Your Service,

Parlindungan Tobing

PT. Bank International Indonesia Tbk. (BII)

IT-SNS/Mainframe Management

Sentral Senayan 3, 3rd Floor

Jl. Asia Afrika no 8

Gelora Bung Karno, Senayan

Jakarta Pusat 10270, Indonesia

Tel +62 21 29228888 ext 32572

Fax +62 21 29039066

Email:

From: [mailto:]
Sent: Monday, August 25, 2014 2:55 PM
To: Parlindungan Lumban Tobing
Cc: Kamariah Bt Abd Jalal; Lee Tek Keng; Dedy Curyani; Mohd Nabil B Mohd Abd Hadi; Norliza Mohd Nasir; ''; ''; ''; Junid Baharuddin; Toto Prasetio
Subject: PMR 04285,999,749

Hello Parlindungan,

We are following-up with you on the "Connection timed out" issue.

Please respond with a status update.

Thank you for using IBM products and support.

Regards
- Ayron Fears
IBM WebSphere Application Server L2 support

On 5/8/2014 12:28 PM, Parlindungan Lumban Tobing wrote:

Dear Pak Nabil,

I have logged this problem to IBM. PMR number for this problem : 04285,999,749 (severity 2)

May I know who will be in charge (PIC) of this PMR? Because IBM support usually want to know deeply about the circumstances and pattern before and after the problem happen as well as the logs from app and WebSphere and also other related information (in detail).

currently, we are still downloading the logs.

At Your Service,

Parlindungan Tobing

PT. Bank International Indonesia Tbk. (BII)

IT-SNS/Mainframe Management

Sentral Senayan 3, 3rd Floor

Jl. Asia Afrika no 8

Gelora Bung Karno, Senayan

Jakarta Pusat 10270, Indonesia

Tel +62 21 29228888 ext 32572

Fax +62 21 29039066

Email:

From: Mohd Nabil B Mohd Abd Hadi [mailto:]
Sent: Tuesday, August 05, 2014 8:54 AM
To: Norliza Mohd Nasir; Toto Prasetio; Junid Baharuddin; Kamariah Bt Abd Jalal; ''; ''; ''
Cc: Parlindungan Lumban Tobing; Lee Tek Keng
Subject: RE: BII RCMS: Analysis on RCMS Application CAS.log

Boby/Lee Yong,

Can we get the details the timeout that you mention is websphere transaction service timeout or database connection timeout?

We also would need the exact error that was thrown by Websphere if need to logged this to IBM support.

Dear Pak Junid/Pak Parlin,

Have you logged this problem to IBM support?

Thanks & Regards.

From: Norliza Mohd Nasir
Sent: Friday, 1 August, 2014 7:48 PM
To: TOTO PRASETIO; JUNID BAHARUDDIN; Kamariah Bt Abd Jalal; Mohd Nabil B Mohd Abd Hadi; ''; ''; ''
Cc: PARLINDUNGAN LUMBAN TOBING
Subject: Re: BII RCMS: Analysis on RCMS Application CAS.log

Thanks Pak Toto.

Looping in Penril.

From: TOTO PRASETIO
Sent: Friday, August 01, 2014 07:44 PM Malay Peninsula Standard Time
To: Norliza Mohd Nasir; JUNID BAHARUDDIN; Kamariah Bt Abd Jalal; Mohd Nabil B Mohd Abd Hadi
Cc: PARLINDUNGAN LUMBAN TOBING
Subject: Re: BII RCMS: Analysis on RCMS Application CAS.log

Will do.

Copy to Parlin, fya.

Toto Prasetio
e:
m:0819-05059961
??
Ability is what you're capable of doing. Motivation determines what you do. Attitude determines how well you do it. (Lou Holtz)

From: Norliza Mohd Nasir

Sent: Jumat, 1 Agustus 2014 18.19

To: Toto Prasetio; Junid Baharuddin; Kamariah Bt Abd Jalal; Mohd Nabil B Mohd Abd Hadi

Subject: Fw: Fw: BII RCMS: Analysis on RCMS Application CAS.log

Pak Toto / Junid - can you assist to get support as requested by Penril?

From: Lee Yong Tan [mailto:]
Sent: Friday, August 01, 2014 07:13 PM Malay Peninsula Standard Time
To: Boby Hartanto <>
Cc: Maybank Regional CAS Project <>; Chan Wen Tong; Kamariah Bt Abd Jalal; Norliza Mohd Nasir; LeeLee <>; Lee Tek Keng; Mohd Nabil B Mohd Abd Hadi
Subject: Re: Fw: BII RCMS: Analysis on RCMS Application CAS.log

Hi Liza,
Can you help to raise support to IBM about the problem of "idle for around 1 hour and not able to reestablish transaction connection to DB from Websphere JNDI"? At mean time we will analyze the log file once received.

On Aug 1, 2014 5:46 PM, "Boby Hartanto" <> wrote:

Hi Liza,

When RCMS didn't get any response from RCAS, most probably RCAS had been experienced transaction service timeout, thus unable to respond anything to RCMS. We are still investigating the root caused, however when we analysed the logs provided previously, we found a pattern of the timeout. Every timeout happened when RCAS trying to query to database after certain time of idle (around 1 hour). RCAS will not be able to establish the connection to database using JNDI. I have attached our analysis for your reference

On Fri, Aug 1, 2014 at 3:39 PM, LeeLee <> wrote:

Hi Liza

We would appreciated that you could assist us to get the log file: SystemOut.log and SystemErr.log for 20 and 21 July.

Thanks

Best Regards,

Lee Lee

Penril Datability (M) Sdn Bhd

Suite A-07-07, Plaza Mont Kiara,

No 2, Jalan Kiara, Mont Kiara,

50480 Kuala Lumpur.

(HP) 6012-3942092

(Tel) 603-6201-2622

(Fax) 603-6201-7622

On 1/8/2014 2:37 PM, Norliza Mohd Nasir wrote:

Lee Lee / Nabil - can we check on this?

From: Stephen Yong Yuan Chuan [mailto:]
Sent: Thursday, July 31, 2014 11:30 AM Malay Peninsula Standard Time
To: Mohd Nabil B Mohd Abd Hadi; TOTO PRASETIO; JUNID BAHARUDDIN; Dev-Raaj AL Shanmugam; Chan Wen Tong; Kamariah Bt Abd Jalal
Cc: Norliza Mohd Nasir; Yong Ai Ping <>; Leong Kok Wai <>; Fitrio Pakana <>; mbbrcms.support <>; James Ta Chee Hoi <>; Wan Chee Khong <>; Tan Suan Fong <>; Cheong Yon Han
Subject: RE: BII RCMS: Analysis on RCMS Application CAS.log

Hi Nabil and Kam,

After my local Indonesia team headed by Fitrio worked with BII team and MBB HQ Infra team on implementing few areas of improvement below, the condition in BII had vastly improved (refer results below as per 25 July).

Besides the updates above on the improvement that had been achieved, I would like to bring your attention to our findings on the BII PRODUCTION RCMS - RCAS integration.

As per verbal discussion with you on 21 July following up on RCAS team’s (Penril) finding against RCMS application logs findings, understand from you RCAS side have no problem and no findings.

We had worked with BII team and collected RCAS web logs with RCMS application logs.

We had found similar “Connection Reset” problem still happening when RCMS calls RCAS for authentication.

Below are the details for your reference and follow-up: -

From 20-Jul onwards until 25-Jul, 2 incidents of “Connection Reset” exception (which took 5 minutes each) from RCMS’s CAS log files:

Date

Time

Map ID

Remarks

Duration Until Exception Written

20/7/2014

13:13:43.821

3279

<- Suspect no response from CAS

20/7/2014

13:18:47.268

0:05:03.447

21/7/2014

8:55:40.560

9056

<- Suspect no response from CAS

21/7/2014

9:00:41.290

0:05:00.730

After analysing RCAS web server access log files from 20-Jul onwards, we are able to map it to the entries in RCMS’s CAS log files. Both requests reached RCAS web server, this eliminate the possibility of intermediary such as Firewall, which intermittently blocked the requests from reaching the RCAS web server. However, the exception is consistently map to “400 – Bad Request” at RCAS web server end. Need to check from RCAS application server log file to identify the root cause of this exception, if any:

DTB

RCAS

Date

Time

Type

Map ID

RCAS API

Date

Time

Method

URI

Status

Size (bytes)

20/7/2014

13:13:43.821

[printReqTime]

3279

N/A

20/7/2014

13:13:43

POST

/rcasws/services/CASServices

400

898

21/7/2014

8:55:40.560

[printReqTime]

9056

N/A

21/7/2014

8:55:40

POST

/rcasws/services/CASServices

400

899

Attached also the list of “400 – Bad Request” extracted from RCAS web server access log – “access_log_BII_RCAS_Web1.log” and “access_log_BII_RCAS_Web2.log”.

Kindly mobilize your team to check with the RCAS vendor, in what circumstance / condition, “400 – Bad Request” response is produced.

Kindly revert to us your follow-up with Penril asap and any action plan / root cause identified for rectification.

Thanks and best regards,
Stephen Yong.

From: Mohd Nabil B Mohd Abd Hadi [mailto:]
Sent: Sunday, 20 July, 2014 12:32 PM
To: Stephen Yong Yuan Chuan; TOTO PRASETIO; JUNID BAHARUDDIN
Cc: Norliza Mohd Nasir; Yong Ai Ping; Leong Kok Wai; Fitrio Pakana; mbbrcms.support; James Ta Chee Hoi; Wan Chee Khong; Kamariah Bt Abd Jalal
Subject: Re: BII RCMS: Analysis on RCMS Application CAS.log

Looping bii team

From: Stephen Yong Yuan Chuan <>
Date: 2014-07-20 04:07:59 am +0000
To: Mohd Nabil B Mohd Abd Hadi <>
Cc: Yong Ai Ping <>,Leong Kok Wai <>,Norliza Mohd Nasir <>,Kamariah Bt Abd Jalal <>,Wan Chee Khong <>,mbbrcms.support <>,Fitrio Pakana <>,James Ta Chee Hoi <>
Subject: BII RCMS: Analysis on RCMS Application CAS.log

Dear Nabil,

As promised, kindly find attached analysis on the TimeOutException encountered for the lengthy login integration to RCAS for your quick reference.

This information is for your info until our full findings report is ready to be sent to you so that you can work on escalation to Penril.

In the spreadsheet there is 3 worksheets:

imap://:993/fetch%3EUID%3E/INBOX%3E40830?header=quotebody&part=1.1.2&filename=image001.jpg

Please refer to “CAS Exception Mapping” worksheet that shows the maximum duration being breached.

imap://:993/fetch%3EUID%3E/INBOX%3E40830?header=quotebody&part=1.1.3&filename=image002.png

The summary shows you that the analysis is on the logs from 16 Jun to 18 July from COA1 server.

The sample exception shows you the connection reset sample:

faultDetail:

{http://xml.apache.org/axis/}stackTrace:java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(SocketInputStream.java:168)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at net.penril.cas.ws.CASServicesSoapBindingStub.wsVerifySimpleLogin(CASServicesSoapBindingStub.java:1178)

at net.penril.cas.ws.CASServicesProxy.wsVerifySimpleLogin(CASServicesProxy.java:86)

at com.ebworx.dtb.host.cas.CASServiceCore.login(CASServiceCore.java:901)

Thanks and best regards,

Stephen Yong,

Hitachi-eBworx Sdn Bhd.

012-3226018

**This message is auto appended to this email and the information disclosed in this message is only meant for the recipient.

DISCLAIMER:

This message may contain confidential and privileged information for its intended recipient(s) only. If you are not an intended recipient, you are hereby notified that any review, dissemination, and distribution, printing or copying of this message or any part thereof are strictly prohibited. Please delete the entire message and inform the sender of the error. Any opinions, conclusions and other information in this message that is unrelated to official business of Hitachi eBworx are those of the individual sender and shall be understood as neither explicitly given nor endorsed by Hitachi eBworx. Hitachi eBworx is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

(PP) Disclaimer:
This message is intended only for the use of the person to whom it is expressly addressed and may contain information that is confidential and legally privileged. If you are not the intended recipient, you are hereby notified that any use, reliance on, reference to, review, disclosure or copying of the message and the information it contains for any purpose is prohibited. If you have received this message in error, please notify the sender by reply e-mail of the misdelivery and delete all its contents.

Opinions, conclusions and other information in this message that do not relate to the official business of Malayan Banking Berhad shall be understood as neither given nor endorsed by it.

(PP) Disclaimer:
This message is intended only for the use of the person to whom it is expressly addressed and may contain information that is confidential and legally privileged. If you are not the intended recipient, you are hereby notified that any use, reliance on, reference to, review, disclosure or copying of the message and the information it contains for any purpose is prohibited. If you have received this message in error, please notify the sender by reply e-mail of the misdelivery and delete all its contents.

Opinions, conclusions and other information in this message that do not relate to the official business of Malayan Banking Berhad shall be understood as neither given nor endorsed by it.

--
Regards,
Boby Hartanto
Software Engineer
Penril Datability (M) Sdn Bhd

Suite A-07-07, Plaza Mont Kiara,
No 2, Jalan Kiara, Mont Kiara,
50480 Kuala Lumpur.
(HP) +601 4701 7767

Also available in: Atom PDF