Support #11285

[SCP ID :##5580##] : Invalid file format

Added by Zahir Abd Latif almost 4 years ago. Updated over 3 years ago.

Status:Work Completed-End life cycleStart date:October 05, 2020
Priority:NormalDue date:November 06, 2020
Assignee:Zahir Abd Latif% Done:

100%

Category:System EnhancementSpent time:-
Target version:-

Description

Hi,
Kindly attend below request:-

Request for CDB to allow/convert txt file encoding UTF-8 with BOM to UTF-8.

utf8bom.png (51.3 KB) Najmi Pasarudin, October 14, 2020 09:55

BOM.docx (2.26 MB) Erni Suhaireen Zulkifli, November 23, 2020 15:40

Support 11285 SCPID 5580.docx (2.27 MB) Erni Suhaireen Zulkifli, November 23, 2020 18:10

Support 11285 SCPID 5580_special character.docx (1.29 MB) Erni Suhaireen Zulkifli, December 04, 2020 17:35

History

#1 Updated by Nurul Athira Abdul Rahim almost 4 years ago

  • Assignee changed from Nurul Athira Abdul Rahim to Najmi Pasarudin

#2 Updated by Najmi Pasarudin almost 4 years ago

  • File utf8bom.png added
  • Assignee changed from Najmi Pasarudin to Nurul Hasnieza Bt Mohd Zamri
Test scenario:
  1. Create Bulk Payment entry
  2. Save Bulk payment file
  3. Extract the data file
  4. Open with notepad
  5. Save as UTF-8 with BOM. Refer screenshot.
  6. Upload bulk payment using text file UTF-8 with BOM
  7. Will get error

#3 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from New - Begin Life Cycle to Development / Work In Progress

#4 Updated by Norhaidah Md Dasuki almost 4 years ago

  • Category set to System Enhancement

#5 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • % Done changed from 0 to 10

Finding: Convert UTF-8 with BOM to UTF-8, system only allow upload UTF-8 file format.

#6 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Development / Work In Progress to Finished Development
  • % Done changed from 10 to 80

#7 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Finished Development to Internal Testing
  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Nurul Athira Abdul Rahim

Hi Athira,
the issue has been fixed. Kindly retest.

Finding: System not allow upload File format UTF8 with BOM.
Solution: Remove BOM character to allow upload file.

#8 Updated by Nurul Athira Abdul Rahim almost 4 years ago

  • Assignee changed from Nurul Athira Abdul Rahim to Erni Suhaireen Zulkifli

Hi Erni Kindly test..

#9 Updated by Nurul Athira Abdul Rahim almost 4 years ago

  • Status changed from Internal Testing to Development / Work In Progress
  • Assignee changed from Erni Suhaireen Zulkifli to Nurul Hasnieza Bt Mohd Zamri

#10 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • % Done changed from 80 to 60

Pending Linux at batchJob server not ignore Bom unicode.

#11 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Due date set to November 06, 2020

#12 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

Currently, fix in window. Got issue when run command in BatchServer.

#13 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Development / Work In Progress to Finished Development

Validate alphanumeric and space only, remove bom with special character replace with empty string.

#14 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Finished Development to Internal Testing
  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Erni Suhaireen Zulkifli

Hi Erni,
Bom invalid file format has been updated. Kindly retest at internal.

Test scenario: Download file > save txt file as utf8 with BOM format > upload file
modules: File upload and Edit Data Entry.

#15 Updated by Erni Suhaireen Zulkifli almost 4 years ago

  • Status changed from Internal Testing to Development / Work In Progress
  • Assignee changed from Erni Suhaireen Zulkifli to Nurul Hasnieza Bt Mohd Zamri
  • % Done changed from 60 to 90

Nieza,

Please recheck file format WDH_AUTO_CREDIT.
Currently unable to upload file with and without BOM format.

#16 Updated by Erni Suhaireen Zulkifli almost 4 years ago

#17 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Development / Work In Progress to Internal Testing
  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Erni Suhaireen Zulkifli

Erni,
currently in IBAM > File format setup for WDH_AUTO_CREDIT > TRNX_CODE footer is empty. Already set the default value. Kindly retest.

#18 Updated by Erni Suhaireen Zulkifli almost 4 years ago

  • Status changed from Internal Testing to System Integration Test
  • % Done changed from 90 to 100

Tested and passed internally.
Kindly proceed for SIT Deployment for user retest and confirmation.

#19 Updated by Erni Suhaireen Zulkifli almost 4 years ago

Test scenario:
1. Create Bulk Payment entry
2. Save Bulk payment file
3. Extract the data file
4. Open with notepad
5. Save as UTF-8 with BOM.
6. Upload bulk payment using text file UTF-8 with BOM

File format to be test:
1. SGB_AUTO_CREDIT
2. IBG
3. SGB_SKDG
4. WDH_SKDG
5. WDH_AUTO_CREDIT
6. SGB_AUTO_DEBIT
7. WDH_AUTO_DEBIT
8. ENROLLMENT
9. KWSP

#20 Updated by Erni Suhaireen Zulkifli almost 4 years ago

  • % Done changed from 100 to 80

Hasnieza,

Currently UTF -8 BOM format does not able to receive file contain ` (reverse single apostrophes)
Please assist to allow file contain this symbol at first line for all file format type.

#21 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from System Integration Test to Code Review
  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Ngoh Chee Ping

Hi Chee Ping,
kindly review my code. Thank you.

issue has been fixed. Please ensure header/first line not accept special character except (,.'/-).
Contents can accept this (`) special character as it got validation at Edit Data Entry. We only remove the special
character except (,.'/-) at header and first line.

#22 Updated by Ngoh Chee Ping almost 4 years ago

  • Status changed from Code Review to Development / Work In Progress
  • Assignee changed from Ngoh Chee Ping to Nurul Hasnieza Bt Mohd Zamri

Please update below items

1) Create a general method removeUTF8BOM for all place to use, no need to create 1 by 1 same method in difference class there.
2) Auto debit upload did not check header to removeUTF8BOM
3) EPF upload and data entry edit data also did not check header to removeUTF8BOM.

#23 Updated by Ngoh Chee Ping almost 4 years ago

Update:

4) Cronjob there file processing no need to call that method to filter, file processing only for transaction details, no header.
5) B2B Processing did not perform the filter, need to add in B2B processing.

#24 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Development / Work In Progress to Finished Development

#25 Updated by Nurul Hasnieza Bt Mohd Zamri almost 4 years ago

  • Status changed from Finished Development to Internal Testing
  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Erni Suhaireen Zulkifli

Erni,
done deployment. Kindly retest.

#26 Updated by Erni Suhaireen Zulkifli almost 4 years ago

  • Status changed from Internal Testing to System Integration Test
  • Assignee changed from Erni Suhaireen Zulkifli to Nurul Athira Abdul Rahim
  • % Done changed from 80 to 100

Tested and passed.
Proceed for User SIT Test

#27 Updated by Erni Suhaireen Zulkifli almost 4 years ago

Attached here is test result for validation:
1. header/first line accept ` (reverse single apostrophes) and remove it from the file but does not accept special character except (,.'/-)
2.Contents can accept this (`) special character

#28 Updated by Nurul Athira Abdul Rahim almost 4 years ago

  • Status changed from System Integration Test to Pending UAT Deployment
  • Assignee changed from Nurul Athira Abdul Rahim to Nurul Hasnieza Bt Mohd Zamri

Hi Nieza, please deploy this fixes to UAT env

#29 Updated by Nurul Athira Abdul Rahim almost 4 years ago

  • Status changed from Pending UAT Deployment to User Acceptance Test
  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Nurul Athira Abdul Rahim

#30 Updated by Nurul Athira Abdul Rahim almost 4 years ago

  • Status changed from User Acceptance Test to Pending Prod Deployment
  • Assignee changed from Nurul Athira Abdul Rahim to Nurul Hasnieza Bt Mohd Zamri

Tested and passed by Rahmat Bohari on 14/12/20

#31 Updated by Nurul Hasnieza Bt Mohd Zamri over 3 years ago

  • Assignee changed from Nurul Hasnieza Bt Mohd Zamri to Najmi Pasarudin

#32 Updated by Najmi Pasarudin over 3 years ago

  • Status changed from Pending Prod Deployment to Pending Review
  • Assignee changed from Najmi Pasarudin to Nurul Athira Abdul Rahim

#33 Updated by Nurul Athira Abdul Rahim over 3 years ago

  • Status changed from Pending Review to Work Completed-End life cycle
  • Assignee changed from Nurul Athira Abdul Rahim to Zahir Abd Latif

Also available in: Atom PDF