Issues during step 5 delta migration

Hello, 
I’ve done delta migration.
Now, The delta migration was completed.
But new products of M1 were not migrated to M2, only fews.

M1 product : 9600
M2 product : 9400
I’ve tried so far :
UPDATE `ub_migrate_map_step_5` SET `created_time` = ‘2000-01-01 00:00:00’ Where entity_name = ‘catalog_product_entity’;
 
then
php -f bin/ubdatamigration run --step=5 --mode=update
 
 
Didn’t worked, then 
UPDATE `ub_migrate_map_step_5` SET `offset` = 9400 Where entity_name = ‘catalog_product_entity’;
then
php -f bin/ubdatamigration run --step=5
 
 
Didn’t worked either
 
I needed to do theses extra steps the first time :
November 18, 2020
 

 
Best regards, Tommy
 
 

68 answers

Profile photo of Mall Staff 159450.00 $tone December 17, 2020
Public

Hi there,

But new products of M1 were not migrated to M2, only fews.

How many of those products? What is the type of these products?
Regards,
Mall.
 

#1
Profile photo of Mall Staff 159450.00 $tone December 17, 2020
Public

Hi there,

On M1 : 9613
On M2 : 9493
On M2 before delta : approx 9400
 There is configurable and simple product

So please try with steps as following:
1. Run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_5` SET `offset` = 0 Where entity_name = 'catalog_product_entity';

2. And then, you run CLI command:

php -f bin/ubdatamigration run --step=5

This will take extra time because it will scan from the first product to the last product in your M1 DB to migrate to M2.
And then, you recheck the issue you mentioned. And tell me how it goes.
Regards,
Mall.

#3
Profile photo of Mall Staff 159450.00 $tone December 17, 2020
Public

Hi Tony,

M2

If you checked ‘KEEP ORIGINAL IDs’ setting in step #5, you shouldn’t add more new products in M2 at this time. 

And you can continue with steps as suggested in my reply #3. But please note that when you run these steps, the product’s information in your M2 will be replaced by the product’s information in the M1.
 
Regards,
Mall.

#7
Profile photo of Tommy Bordas 710.00 $tone December 17, 2020
Public

Ok I’ve just seen same for customer.
I’ve tried so far, 
Update ub_migrate_map_step_6 Set created_time = ‘2000-01-01 00:00:00’ Where entity_name = ‘customer_entity’;
php -f bin/ubdatamigration run --step=6 --mode=update;

#9
Profile photo of Tommy Bordas 710.00 $tone December 17, 2020
Public

Also, previous customer that were already imported can’t login, does that happens because of the hash ? Do I need to run customer:hash:upgrade with your tool ?
best regards, Tommy

#10
Profile photo of Tommy Bordas 710.00 $tone December 17, 2020
Public

It says that but there is no new customer
[Processing][run] Step #6 migration completed with 0 Customer Groups; 88 Customers
Step #6 migration completed successfully
Total Data Migrated: 100%

#11
Profile photo of Mall Staff 159450.00 $tone December 18, 2020
Public

Hi Tommy,

PHP Error[2]: PDOStatement::execute(): MySQL server has gone away

That was a MySQL server’s issue at that time only. Please continue with data migration in step #5 by running the CLI command:

php -f bin/ubdatamigration run --step=5

And let me how it goes then.

Regards,
Mall.
 

#14
Profile photo of Tommy Bordas 710.00 $tone December 18, 2020
Public

Hi, 

[Processing][run] Step #5 migration completed with 13 Products;
Step #5 migration completed successfully
Total Data Migrated: 100%
 
The process has been completed but still no more product after indexing.
 
Best regards

#15
Profile photo of Tommy Bordas 710.00 $tone December 18, 2020
Public

When i go in the ud data migration tab in the back, then select the product import settings, I see the number of Total: 9615 products.
But not in the listing of the magento, it says 9473 products.
 
Last product id is 9527 however.
 
Best regards,

#16
Profile photo of Mall Staff 159450.00 $tone December 18, 2020
Public

Hi there,

When i go in the ud data migration tab in the back, then select the product import settings, I see the number of Total: 9615 products.
But not in the listing of the magento, it says 9473 products.

Please provide me information of your instance:

  • URL and Admin credentials of your M2 instance
  • SSH credentials of your M2 instance
  • Let me know the path to the M2 folder
  • And phpMyadmin URL which allows me to view your M1 and M2 databases.

I will help to check further and get back to you then.
 
PS. Please make sure you mark your reply private or simply switch this ticket to the private mode, then it’s safe to share your site info here. Only you and our technical team can access it. 
 
Regards,
Mall.

#17
Profile photo of Mall Staff 159450.00 $tone December 21, 2020
Public

Hi Tommy,

I checked the backend of your M2 and saw the issue you mentioned.

  •  At present, the total of products in your M1 is 9615: http://prntscr.com/w6ui53
  • And our migration tool migrated a total of 9507 products: http://prntscr.com/w6uinm
  • But the current total of products in your M2 is 9473: http://prntscr.com/w6ujbt

Then, I checked your M1 DB, however, I couldn’t access your URL from my end: http://prntscr.com/w6ukvr

After that, I checked via SSH connection and didn’t see any error messages related to the data migration in step #5.

So, please try again with steps below at your end:
1. Open the PHP file at: pub/ub-tool/protected/controllers/Step5Controller.php
and find the code line:

$orderBy = "entity_id ASC";

and replace it with: 

$orderBy = "type_id DESC";

2. Run the following SQL query in your M2 DB:

UPDATE `ub_migrate_map_step_5` SET `offset` = 0 Where entity_name = 'catalog_product_entity';

3. And then, you run the CLI command: (please don’t use ‘--mode=update’ at this phase)

php -f bin/ubdatamigration run --step=5

Please note that this will take extra time because it will scan from the first product to the last product in your M1 DB to delta migrate to M2.

Once, recheck the issue you mentioned and let me how it goes.

Regards,
Mall.
 

#19
Profile photo of ubdev Staff 76860.00 $tone December 21, 2020
Public

Hi Tommy, 
Please follow Mall’s workaround mentioned in his reply #19 above first and let us know how it goes. 
Regards,
Ubertheme team

#22
Profile photo of Mall Staff 159450.00 $tone December 22, 2020
Public

Hi Tommy,

Just launched it for giving it a try again, after reindexing, still 9473 product.

That is strange. 

For mysql access, I just saw I need to whitelist your IP

Here is my public IP: 27.72.104.129. 

Please whitelist that IP, I will help to check further and get back to you then.

Regards,
Mall.

#26
Profile photo of ubdev Staff 76860.00 $tone December 22, 2020
Public

Hi Tommy, 

Hi, you can launch SQL via ssh also.

Yes, we are checking via SSH already. However, to help us debug faster, please consider providing the access to PHPMyadmin.
Your case will need pretty much time to detect the root cause of the issue. That’s why we asked for access to PHPMyadmin.
For other cases, the workaround Mall suggested could help resolve the issue though.
Regards,
Ubertheme team

#29
Profile photo of ubdev Staff 76860.00 $tone December 22, 2020
Public

Hi Tommy, 
 
From the result we checked yesterday (checking your M2 DB > UB Migration log tables via SSH), we saw that:
 
Our module tool already scanned all 9613 products in your M1 DB: http://prntscr.com/w7ojz2
 
However, in the migration log tables, our module tool only migrated & mapped 9507 products: http://prntscr.com/w7om2z
 
So, there were some products not migrated to M2: 9613 -- 9507 = 106 products.
 
Then, we did check via the migration log file at: pub/ub-tool/protected/runtime/ub_data_migration.log. However, there was no error message associated with the migration of the products.
 
Besides, the total of products in your current M2 DB: 9473: http://prntscr.com/w7oont (which is similar to the stats in your M2 backend)
 
That means, 9613 -- 9473 = 140 products were not migrated.
 
So, we wonder if you did remove parts of the products in M2 after being migrated from your M1?
 
And we also saw that you did not check ‘KEEP ORIGINAL IDS’ settings in step #5. So, the total 9473 products in your current M2 might also include products migrated by our module and the products you added manually in M2.
 
So, could you try the steps below?
1. Reset data migration in steps: #8, #7, #6, #5
 
2. And then, after you re-perform data migration in step #5 (make sure you finish this step), you pause the migration process & check if the issue you mentioned is solved?
 
If the issue still persists, we will help to debug it further. If step 5 is completed, you can continue other steps: #6,#7,#8?
 
Regards,
Ubertheme team

#31
Profile photo of Mall Staff 159450.00 $tone December 22, 2020
Public

Hi Tommy,

Before you follow the steps suggested in my reply #31, please try the following steps first:

1. Clean the cache of our module by running the command:
rm -rf pub/ub-tool/protected/runtime/cache/*

2. Then redo step 2, step 3 suggested in the reply #19

And then, you check the issue again and let me how it goes.
 
Regards,
Mall.

#32
Profile photo of Mall Staff 159450.00 $tone December 22, 2020
Public

Hi there,

Is it possible to export all product created after 16 december in M1 and reimport in M2, without your plugin ?

No.

 140 products missing represent one full week of work.

Could you provide me SKUs of some products in 140 missing products? 
I will help to check further on the settings of these products in M1 DB and I will get back to you then.
Regards,
Mall.

#36
Profile photo of Tommy Bordas 710.00 $tone December 23, 2020
Public

Hi, 
 
Additionnaly, I juste see that we have updated category description and product description, but they do not appear after delta step 4. (and step 5)
How can I reset category description ?
 

UPDATE `ub_migrate_map_step_4` SET `offset` = 0 Where entity_name = 'catalog_category_entity';
php -f bin/ubdatamigration run --step=5

Best regards, 
Tommy

#38
Profile photo of Mall Staff 159450.00 $tone December 23, 2020
Public

Hi there,

How can I reset category description ?

You can try with steps as following to delta migration in step #4:
1. Run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_4` SET `offset` = 0 Where entity_name = 'catalog_category_entity';

2. And then, you run CLI command:

php -f bin/ubdatamigration run --step=4

Regards,
Mall.

#39
Profile photo of Mall Staff 159450.00 $tone December 24, 2020
Public

Hi Tommy,

it just remains the product now

Please give me a bit more time, I will check further into the product’s SKUs you mentioned in the reply #37 and will get back to you then.

Regards,
Mall.

#41
Profile photo of Mall Staff 159450.00 $tone December 24, 2020
Public

Hi there,

Hi, for example  
R06403
R05856-Marine-M
R06402

I have checked your M1 DB and seen those products: http://prntscr.com/w98tob
In M1: sku -- entity_id
R05856-Marine-M <=> 17368
R06402 <=> 17277
R06403 <=> 17302

And I checked the migration log table named ‘ub_migrate_map_step_5’ in your M2 DB and saw there was one product (entity_id = 17277) migrated to M2. And in the M2 DB, it had the new entity_id = 9491. But it seems that you had edited the SKU of this product in M2 after migration: http://prntscr.com/w9927m

Meanwhile, the SKUs: R05856-Marine-M, R06403 hadn’t been migrated to M2 yet. This is strange.

With all that information, could you try the steps below?

1. Reset data migration in steps: #8, #7, #6, #5
 
2. And then, after you re-perform data migration in step #5 (make sure you finish this step), you pause the migration process & check if the issue you mentioned is solved?
 
If the issue still persists, we will help to debug it further. If step 5 is completed, you can continue other steps: #6,#7,#8?
 
Regards,
Mall.
 
 
 

#42
Profile photo of Tommy Bordas 710.00 $tone December 28, 2020
Public

Hi,
 
I’m trying on a local clone to test. 
But it can’t be an acceptable solution because it delete all the work done before the delta migration, and the interest of the plugin is limited then. 
 
I’ll keep you informed when it’s finished.
Best regards,
Tommy

#43
Profile photo of Tommy Bordas 710.00 $tone December 28, 2020
Public

Hi, 
Migration on local clone just finished, now we just have 9021 products, so 400 less. R05856-Marine-M, R06403 are not here either. 
 
Could it be that we need to import one time with :

$orderBy = "entity_id DESC";

then change to 

$orderBy = "type_id DESC";

?
 
As I said, it can’t be an option to reset all the steps to us, because we edited CMS pages.
 
Best regards, 
 
Tommy
 
 

#44
Profile photo of Mall Staff 159450.00 $tone December 29, 2020
Public

Hi there,

Could it be that we need to import one time with :
$orderBy = “entity_id DESC”;

then change to 

$orderBy = “type_id DESC”;
?

Yes, please apply that tweak code before you do data migration in step #5 to handle the case of your M1 DB (your M1 DB has some configurable products added after some related simple products)
Regards,
Mall.

#45
Profile photo of Tommy Bordas 710.00 $tone January 4, 2021
Public

Hi, 
 
Ok so we need to disable the tweak, import then apply the tweak ? 
 
But again, reseting step will not be an option, because too much work has been done to CMS page and categories and all..
 
Best regards, Tommy
 

#46
Profile photo of Mall Staff 159450.00 $tone January 4, 2021
Public

Hi there,

Ok so we need to disable the tweak, import then apply the tweak ? 

No, you should take the tweaked code in step #5.
Regards,
Mall.

#47
Profile photo of Mall Staff 159450.00 $tone January 5, 2021
Public

Hi there,

Is it possible to retrieve it by launching step 3 again,

Let’s try with steps:
1. Run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_3` SET `offset` = 0 Where entity_name = eav_attribute;

2. And then, you run data migration in step #3:

php -f bin/ubdatamigration run --step=3

Regards,
Mall.

#52
Profile photo of Tommy Bordas 710.00 $tone January 5, 2021
Public

Ok i’ve found this topic : https://www.ubertheme.com/question/delta-migration-fail-in-step-3/
And after doing the steps written there it says : 
Total Data Migrated: 43%
The attribute seems to be back, but no products has it anymore (like just before the reimport of attribute).
 
If I do the same thing on step 5, it will remove all the things done on the product since migration ? 
 
Best regards 

#54
Profile photo of ubdev Staff 76860.00 $tone January 6, 2021
Public

Hi Tommy, 
 
We’re glad to hear that. In case you need further assistance, please get back to us here. 
 
Meanwhile, if you plan to implement a mega menu or one step checkout functionality for your M2 project, please take a look at our UB Mega Menu and UB One Step Checkout modules here to see if they’re a good fit. 
 
PS. We’d like to offer you 20%OFF if you choose to move forward with those modules. 
Regards,
Ubertheme team

#56
Profile photo of Tommy Bordas 710.00 $tone January 6, 2021
Public

Hi, 
 
Independently of my last message, I still have issues about missing products in the migration, but I think it’s an automated message ! 🙂
 
Best regards, 
Tommy

#57
Profile photo of Mall Staff 159450.00 $tone January 6, 2021
Public

Hi there,

I still have issues about missing products in the migration, but I think it’s an automated message ! 

We are testing data migration with the M1 DB you shared in our local instance: http://prntscr.com/whm7p6
And I will get back to you after I find the root of the missing product issue you mentioned.
Regards,
Mall.
 

#58
Profile photo of ubdev Staff 76860.00 $tone January 8, 2021
Public

Hi Tommy, 
 
As per my previous reply, we are testing data migration with the M1 DB that you shared: http://prntscr.com/wioo8g

In step #5, we selected 9613 products to migrate as the settings in your live instance: http://prntscr.com/wioqst and we did finish data migration in step #5: http://prntscr.com/wior8g.
 
And in the migration log table ‘ub_migrate_map_step_5’ of our module, we saw that our migration tool scanned a total of 9613 products to migrate: http://prntscr.com/wiot0z. However, there were only 9114 products migrated to M2 DB: http://prntscr.com/wiotp7

So, there were 9613 -- 9114 = 599 products that hadn’t been migrated yet.
And there was no error message in the data migration in step #5 which is strange.
 
We still try to debug further to look for the root cause of that issue.
 
For your information,
Regards,
Ubertheme team

#59
Profile photo of Mall Staff 159450.00 $tone January 8, 2021
Public

Hi Tommy,

I saw you paused in the settings of step #5: http://prntscr.com/wiplr7
Please try steps below at your end:

1. Open the PHP file at: pub/ub-tool/protected/controllers/Step5Controller.php
and find the code line:

$orderBy = "type_id DESC";

and replace it with: 

$orderBy = "type_id DESC, entity_id ASC";

2. Next, clean the cache of our module by running the command:

rm -rf pub/ub-tool/protected/runtime/cache/*

3. You run the following SQL in your M2 DB:

UPDATE `ub_migrate_map_step_5` SET `offset` = 0 Where entity_name = 'catalog_product_entity';

4. And then, you run data migration in step #5 using the CLI command:

php -f bin/ubdatamigration run --step=5 --limit=1000

After you finish the data migration in step #5, check the issue you mentioned again and let me know how it goes.

PS. For your information, I did apply the above steps in my testing local instance and the issue was solved. There was a total of 9613 products migrated to M2 DB.

Regards,
Mall.

#60
Profile photo of Tommy Bordas 710.00 $tone January 8, 2021
Public

Hi Mall, 
Ok perfect, thank you for solving the problem ! 🙂
An additional question, does all the modifications done on the products already imported will be lost ?
 
Best regards,
Tommy
 

#61
Profile photo of Mall Staff 159450.00 $tone January 9, 2021
Public

Hi Tommy,

An additional question, does all the modifications done on the products already imported will be lost ?

Yes, because the steps I suggested in the reply #60 helps to perform delta update to all products in your M1.

Meanwhile, I have checked your instance again and seen that you had run data migration in step #5 once. Thus, I have edited the suggested steps in the reply #60 accordingly.

Regards,
Mall.

#62
Profile photo of Mall Staff 159450.00 $tone January 11, 2021
Public

Hi there,

PHP Error[2]: Creating default object from empty value

That because you did delete at least one migrated product after the first migration.
Let’s try with steps:
+ Run the CLI command:

php -f bin/ubdatamigration clean --step=5

+ And then, you continue with data migration in step #5 by run CLI command:

php -f bin/ubdatamigration run --step=5

Regards,
Mall.

#64
Profile photo of Tommy Bordas 710.00 $tone January 11, 2021
Public

Sorry did not see your answer, I’ve launched your #64 commands, and now there is this error : 
 

Status: fail
Message: CDbCommand failed to execute the SQL statement: SQLSTATE[HY000]: General error: 1364 Field 'sku' doesn't have a default value. The SQL statement executed was: INSERT INTO `inventory_source_item` (`quantity`, `status`, `source_code`) VALUES (:yp0, :yp1, :yp2)
#66
Profile photo of Mall Staff 159450.00 $tone January 12, 2021
Public

Hi Tommy,

General error: 1364 Field ‘sku’ doesn’t have a default value.

That occurred because your M1 DB had some products with missing SKU. You can run the following SQL query in your M1 DB to look for these products:

Select * From catalog_product_entity Where sku = '' OR sku IS NULL;

and then, fill in the value for the `sku` field on all products that you found. Please make sure that SKU is unique in accordance with the M2 rule (You can follow the rule: SKU-ENTITY_ID)
And then, you can continue with data migration in step #5 by running the CLI command:

php -f bin/ubdatamigration run --step=5;

Regads,
Mall.
 

#67
Profile photo of ubdev Staff 76860.00 $tone 6 days ago
Public

Hi Tommy, 
 
We wonder if Mall’s workaround in his reply #67 worked fine for you. If the issue still persists, please get back to us here. 

 
Meanwhile, our team is working on a new plan for 2021, if possible, could you please help us answer a few quick questions via this survey
 
Your feedback is crucial for us, so we hope you can share with us your thoughts on those questions. 
 
Sincerely,
Ubertheme team

#68

Please login or Register to Submit Answer

Written By

Comments