Issues during step 5 delta migration

Hi there,

But new products of M1 were not migrated to M2, only fews.

How many of those products? What is the type of these products?
Regards,
Mall.
 

Hi there,

On M1 : 9613
On M2 : 9493
On M2 before delta : approx 9400
 There is configurable and simple product

So please try with steps as following:
1. Run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_5` SET `offset` = 0 Where entity_name = 'catalog_product_entity';

2. And then, you run CLI command:

php -f bin/ubdatamigration run --step=5

This will take extra time because it will scan from the first product to the last product in your M1 DB to migrate to M2.
And then, you recheck the issue you mentioned. And tell me how it goes.
Regards,
Mall.

Hi there,

he site is now on production…

What is the site you mentioned? M1 or M2 site?
Regards,
Mall.

Hi Tony,

M2

If you checked ‘KEEP ORIGINAL IDs’ setting in step #5, you shouldn’t add more new products in M2 at this time. 

And you can continue with steps as suggested in my reply #3. But please note that when you run these steps, the product’s information in your M2 will be replaced by the product’s information in the M1.
 
Regards,
Mall.

Hi Tommy,

PHP Error[2]: PDOStatement::execute(): MySQL server has gone away

That was a MySQL server’s issue at that time only. Please continue with data migration in step #5 by running the CLI command:

php -f bin/ubdatamigration run --step=5

And let me how it goes then.

Regards,
Mall.
 

Hi there,

When i go in the ud data migration tab in the back, then select the product import settings, I see the number of Total: 9615 products.
But not in the listing of the magento, it says 9473 products.

Please provide me information of your instance:

• URL and Admin credentials of your M2 instance
• SSH credentials of your M2 instance
• Let me know the path to the M2 folder
• And phpMyadmin URL which allows me to view your M1 and M2 databases.

I will help to check further and get back to you then.
 
PS. Please make sure you mark your reply private or simply switch this ticket to the private mode, then it’s safe to share your site info here. Only you and our technical team can access it. 
 
Regards,
Mall.

Hi Tommy,

I checked the backend of your M2 and saw the issue you mentioned.

•  At present, the total of products in your M1 is 9615: http://prntscr.com/w6ui53
• And our migration tool migrated a total of 9507 products: http://prntscr.com/w6uinm
• But the current total of products in your M2 is 9473: http://prntscr.com/w6ujbt

Then, I checked your M1 DB, however, I couldn’t access your URL from my end: http://prntscr.com/w6ukvr

After that, I checked via SSH connection and didn’t see any error messages related to the data migration in step #5.

So, please try again with steps below at your end:
1. Open the PHP file at: pub/ub-tool/protected/controllers/Step5Controller.php
and find the code line:

$orderBy = "entity_id ASC";

and replace it with: 

$orderBy = "type_id DESC";

2. Run the following SQL query in your M2 DB:

UPDATE `ub_migrate_map_step_5` SET `offset` = 0 Where entity_name = 'catalog_product_entity';

3. And then, you run the CLI command: (please don’t use ‘--mode=update’ at this phase)

php -f bin/ubdatamigration run --step=5

Please note that this will take extra time because it will scan from the first product to the last product in your M1 DB to delta migrate to M2.

Once, recheck the issue you mentioned and let me how it goes.

Regards,
Mall.
 

Hi Tommy, 
Please follow Mall’s workaround mentioned in his reply #19 above first and let us know how it goes. 
Regards,
Ubertheme team

Hi Tommy,

Just launched it for giving it a try again, after reindexing, still 9473 product.

That is strange. 

For mysql access, I just saw I need to whitelist your IP

Here is my public IP: 27.72.104.129. 

Please whitelist that IP, I will help to check further and get back to you then.

Regards,
Mall.

Hi Tommy, 

Hi, you can launch SQL via ssh also.

Yes, we are checking via SSH already. However, to help us debug faster, please consider providing the access to PHPMyadmin.
Your case will need pretty much time to detect the root cause of the issue. That’s why we asked for access to PHPMyadmin.
For other cases, the workaround Mall suggested could help resolve the issue though.
Regards,
Ubertheme team

Hi Tommy, 
 
From the result we checked yesterday (checking your M2 DB > UB Migration log tables via SSH), we saw that:
 
Our module tool already scanned all 9613 products in your M1 DB: http://prntscr.com/w7ojz2
 
However, in the migration log tables, our module tool only migrated & mapped 9507 products: http://prntscr.com/w7om2z
 
So, there were some products not migrated to M2: 9613 -- 9507 = 106 products.
 
Then, we did check via the migration log file at: pub/ub-tool/protected/runtime/ub_data_migration.log. However, there was no error message associated with the migration of the products.
 
Besides, the total of products in your current M2 DB: 9473: http://prntscr.com/w7oont (which is similar to the stats in your M2 backend)
 
That means, 9613 -- 9473 = 140 products were not migrated.
 
So, we wonder if you did remove parts of the products in M2 after being migrated from your M1?
 
And we also saw that you did not check ‘KEEP ORIGINAL IDS’ settings in step #5. So, the total 9473 products in your current M2 might also include products migrated by our module and the products you added manually in M2.
 
So, could you try the steps below?
1. Reset data migration in steps: #8, #7, #6, #5
 
2. And then, after you re-perform data migration in step #5 (make sure you finish this step), you pause the migration process & check if the issue you mentioned is solved?
 
If the issue still persists, we will help to debug it further. If step 5 is completed, you can continue other steps: #6,#7,#8?
 
Regards,
Ubertheme team

Hi Tommy,

Before you follow the steps suggested in my reply #31, please try the following steps first:

1. Clean the cache of our module by running the command:
rm -rf pub/ub-tool/protected/runtime/cache/*

2. Then redo step 2, step 3 suggested in the reply #19

And then, you check the issue again and let me how it goes.
 
Regards,
Mall.

Hi there,

Is it possible to export all product created after 16 december in M1 and reimport in M2, without your plugin ?

No.

 140 products missing represent one full week of work.

Could you provide me SKUs of some products in 140 missing products? 
I will help to check further on the settings of these products in M1 DB and I will get back to you then.
Regards,
Mall.

Hi there,

How can I reset category description ?

You can try with steps as following to delta migration in step #4:
1. Run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_4` SET `offset` = 0 Where entity_name = 'catalog_category_entity';

2. And then, you run CLI command:

php -f bin/ubdatamigration run --step=4

Regards,
Mall.

Hi Tommy,

it just remains the product now

Please give me a bit more time, I will check further into the product’s SKUs you mentioned in the reply #37 and will get back to you then.

Regards,
Mall.

Hi there,

Hi, for example  
R06403
R05856-Marine-M
R06402

I have checked your M1 DB and seen those products: http://prntscr.com/w98tob
In M1: sku -- entity_id
R05856-Marine-M <=> 17368
R06402 <=> 17277
R06403 <=> 17302

And I checked the migration log table named ‘ub_migrate_map_step_5’ in your M2 DB and saw there was one product (entity_id = 17277) migrated to M2. And in the M2 DB, it had the new entity_id = 9491. But it seems that you had edited the SKU of this product in M2 after migration: http://prntscr.com/w9927m

Meanwhile, the SKUs: R05856-Marine-M, R06403 hadn’t been migrated to M2 yet. This is strange.

With all that information, could you try the steps below?

1. Reset data migration in steps: #8, #7, #6, #5
 
2. And then, after you re-perform data migration in step #5 (make sure you finish this step), you pause the migration process & check if the issue you mentioned is solved?
 
If the issue still persists, we will help to debug it further. If step 5 is completed, you can continue other steps: #6,#7,#8?
 
Regards,
Mall.
 
 
 

Hi there,

Could it be that we need to import one time with :
$orderBy = “entity_id DESC”;

then change to 

$orderBy = “type_id DESC”;
?

Yes, please apply that tweak code before you do data migration in step #5 to handle the case of your M1 DB (your M1 DB has some configurable products added after some related simple products)
Regards,
Mall.

Hi there,

Ok so we need to disable the tweak, import then apply the tweak ? 

No, you should take the tweaked code in step #5.
Regards,
Mall.

Hi there,

Is it possible to retrieve it by launching step 3 again,

Let’s try with steps:
1. Run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_3` SET `offset` = 0 Where entity_name = eav_attribute;

2. And then, you run data migration in step #3:

php -f bin/ubdatamigration run --step=3

Regards,
Mall.

Hi Tommy, 
 
We’re glad to hear that. In case you need further assistance, please get back to us here. 
 
Meanwhile, if you plan to implement a mega menu or one step checkout functionality for your M2 project, please take a look at our UB Mega Menu and UB One Step Checkout modules here to see if they’re a good fit. 
 
PS. We’d like to offer you 20%OFF if you choose to move forward with those modules. 
Regards,
Ubertheme team

Hi there,

I still have issues about missing products in the migration, but I think it’s an automated message ! 

We are testing data migration with the M1 DB you shared in our local instance: http://prntscr.com/whm7p6
And I will get back to you after I find the root of the missing product issue you mentioned.
Regards,
Mall.
 

Hi Tommy, 
 
As per my previous reply, we are testing data migration with the M1 DB that you shared: http://prntscr.com/wioo8g

In step #5, we selected 9613 products to migrate as the settings in your live instance: http://prntscr.com/wioqst and we did finish data migration in step #5: http://prntscr.com/wior8g.
 
And in the migration log table ‘ub_migrate_map_step_5’ of our module, we saw that our migration tool scanned a total of 9613 products to migrate: http://prntscr.com/wiot0z. However, there were only 9114 products migrated to M2 DB: http://prntscr.com/wiotp7

So, there were 9613 -- 9114 = 599 products that hadn’t been migrated yet.
And there was no error message in the data migration in step #5 which is strange.
 
We still try to debug further to look for the root cause of that issue.
 
For your information,
Regards,
Ubertheme team

Hi Tommy,

I saw you paused in the settings of step #5: http://prntscr.com/wiplr7
Please try steps below at your end:

1. Open the PHP file at: pub/ub-tool/protected/controllers/Step5Controller.php
and find the code line:

$orderBy = "type_id DESC";

and replace it with: 

$orderBy = "type_id DESC, entity_id ASC";

2. Next, clean the cache of our module by running the command:

rm -rf pub/ub-tool/protected/runtime/cache/*

3. You run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_5` SET `offset` = 0 Where entity_name = 'catalog_product_entity';

4. And then, you run data migration in step #5 using the CLI command:

php -f bin/ubdatamigration run --step=5 --limit=1000

After you finish the data migration in step #5, check the issue you mentioned again and let me know how it goes.

PS. For your information, I did apply the above steps in my testing local instance and the issue was solved. There was a total of 9613 products migrated to M2 DB.

Regards,
Mall.

Hi Tommy,

An additional question, does all the modifications done on the products already imported will be lost ?

Yes, because the steps I suggested in the reply #60 helps to perform delta update to all products in your M1.

Meanwhile, I have checked your instance again and seen that you had run data migration in step #5 once. Thus, I have edited the suggested steps in the reply #60 accordingly.

Regards,
Mall.

Hi there,

PHP Error[2]: Creating default object from empty value

That because you did delete at least one migrated product after the first migration.
Let’s try with steps:
+ Run the CLI command:

php -f bin/ubdatamigration clean --step=5

+ And then, you continue with data migration in step #5 by run CLI command:

php -f bin/ubdatamigration run --step=5

Regards,
Mall.

Hi Tommy,

General error: 1364 Field ‘sku’ doesn’t have a default value.

That occurred because your M1 DB had some products with missing SKU. You can run the following SQL query in your M1 DB to look for these products:

Select * From catalog_product_entity Where sku = '' OR sku IS NULL;

and then, fill in the value for the `sku` field on all products that you found. Please make sure that SKU is unique in accordance with the M2 rule (You can follow the rule: SKU-ENTITY_ID)
And then, you can continue with data migration in step #5 by running the CLI command:

php -f bin/ubdatamigration run --step=5;

Regads,
Mall.
 

Hi Tommy, 
 
We wonder if Mall’s workaround in his reply #67 worked fine for you. If the issue still persists, please get back to us here. 

 
Meanwhile, our team is working on a new plan for 2021, if possible, could you please help us answer a few quick questions via this survey? 
 
Your feedback is crucial for us, so we hope you can share with us your thoughts on those questions. 
 
Sincerely,
Ubertheme team