Forums

Covering OroCommerce, OroCRM, OroPlatform topics, including community updates and company announcements.

Forums OroCRM OroCRM – Installation/Technical Issues or Problems Duplicate Values in the Exported CSV

This topic contains 18 replies, has 7 voices, and was last updated by  spyout98 1 week, 3 days ago.

  • Creator
    Topic
  • #27890

    joe7
    Participant

    Hi,

    We are having a strange issue when exporting a very simple custom report. Records are being duplicated when there are over 200 records in the resultset.

    For a very simple report with just the Order Number and the Create Date the grid view online shows the correct data (in this case 262 records). When I export that view to CSV the CSV also contains 262 records but 62 of those records are duplicates. (Note the same happens with Export to XLSX).

    Looking at the SQL generated via the debug the query couldn’t be simpler. (Included below if helps).

    There is an old issue here https://oroinc.com/orocrm/forums/topic/reports-generating-csv-with-duplicated-data which is similar but (a) has been resolved and (b) I am not including address fields in the report and there are no joins so it is different.

    Any thoughts on where to start on this or if you have have come across this previously appreciated.

    Thanks,

    Joe

Viewing 15 replies - 1 through 15 (of 18 total)
  • Author
    Replies
  • #27891
    Michael
    Michael
    Keymaster

    Hi Joe,

    What OroCRM version do you have? And when you say “62 of those records are duplicates” – do you mean 31 records match some other 31 records, or it’s 62 and 62? Also are these duplicate records the last records in the file?

    #27892

    joe7
    Participant

    Hi Michael,

    Thanks for the response.

    The version is OroCRM 2.6.1.

    To clarify on the duplicates which come through on the CSV. Record numbers 1 – 200 are unique and correct however record number 201 is a duplicate of record number 1, 202 is the same as record number 2, 203 / 3, 204 /4 etc. So the records from 201 – 262 are all duplicates of the first 61 records. In the online grid view the records are all unique and correct.

    Let me know if I can provide any other info on it to help.

    Cheers,

    Joseph

    #27893

    Yurii Muratov
    Participant

    Hi Joe.

    Could you please add debug breakepoint in vendor/oro/platform/src/Oro/Bundle/ImportExportBundle/Async/Export/PreExportMessageProcessorAbstract.php, line 108, run consumer command in debug, run the export and send the value id $ids variable ?

    it should have an 2-d array with data [0 => [array_of_ids]].

    array_of_ids is the array of order ids that should be exported and this array should not have duplicates.

    So, the export should be processed in one batch (as the default size of one batch is 5000 records) and process given ids.

    #27894

    joe7
    Participant

    Hi Yurii – thanks for the detailed help. I’m just back now so I will try this and let you know one way or the other what is returned.

    Joseph

    #27895

    joe7
    Participant

    Hi,

    Getting back on this I’ve run a number of tests on it and all with the same result.

    When I generate a report with greater than 200 records with only the fields Order Number, Created At and Step then row 1 is repeated at 201, row 2 repeated at 202 etc.

    Now adding the debug code as advised by Yurii (thanks!) the array of IDs does *not* show any duplicates. See the short excerpt below:

    Moving it on I suppose anyone any ideas why (or where) even though the Array has the unique values it is getting duplicated when it merges in the data.

    Thanks.

    Joseph

    #27896

    isranet
    Participant

    The same issue here (OroCRM v2.6.1)
    Trying to export Magento Customers (>10k), the total amount of rows is correct, but records duplicated every 200 rows.

    #27898
    Andrey Yatsenko
    Andrey Yatsenko
    Moderator
    #27899

    isranet
    Participant

    the latest one I can see in the git is 2.6.10

    • This reply was modified 5 months, 1 week ago by  isranet.
    #27901
    Andrey Yatsenko
    Andrey Yatsenko
    Moderator

    2.6.10 for oroinc/crm, but 2.6.12 for oroinc/crm-application,
    You need to update the whole application with all the packages, not only crm.

    #37404
    oro_newbie
    oro_newbie
    Participant

    Hi,

    I’m using version 3 and I still see this. It’s 5000 records this time. I found this:

    in

    ImportExportBundle/Resource/config/services.yml

    but changing it doesn’t help. How do I debug why it’s repeating?

    Thanks,
    A

    #37405
    oro_newbie
    oro_newbie
    Participant

    Hi,

    Anyone able to reproduce the issue?

    Thanks,
    A

    #37410
    Andrey Yatsenko
    Andrey Yatsenko
    Moderator

    Hi Oro_newbie,

    We tried to reproduce an issue on exporting 6400 and 11400 of products with no luck.

    Can you clarify what entities are you trying to export and what exactly version of application do you use, like crm-application v3.0.0

    #37430
    oro_newbie
    oro_newbie
    Participant

    Hi,

    Figured the issue. It IS the line I mentioned previously. I removed cache this time

    and tried, so now it’s working. It’s mentioned as size_of_batch so I’m guessing it’s not the limit of rows but rather the number of records to pull in 1 batch. Maybe somewhere the offset for the next set of batch isn’t happening?

    Here’s my configuration anyway:

    system configuration

    #37490

    spyout98
    Participant

    We are using 2.6.19 OroCRM application, and the issue that oro_newbie said is occurring here too. The exported CSV repeat records after 5000.

    #37504

    spyout98
    Participant

    Is there any workaround?

Viewing 15 replies - 1 through 15 (of 18 total)

You must be logged in to reply to this topic.

You will be redirected to [title]. Would you like to continue?

Yes No
sso for www.magecore.comsso for oroinc.comsso for oroinc.desso for oroinc.frsso for marketplace.orocommerce.comsso for marketplace.orocrm.com