Delta Migration has been running forever.

Hello. 
We are using 
UB Data Migration Pro (CE) -- 3.2.4
 
We run “php -f bin/ubdatamigration run --step=5 --mode=update” as Delta migration.
But it took 7days.. forever running.. not finish.
The main issue is First that this migration doesn’t work with delta. it is working from initial data.
The main issue is Second that when the MySQL gone issue, If we run “php -f bin/ubdatamigration run --step=5 --mode=update” again, it start from 1st Product. not Continue.
This is ub_migrate_step table.

This is ub_migrate_map_step_5 table
All item created_time is 2020-10-12 (Old time)

 
Please check this critical issue.
Thank you.

14 answers

Profile photo of Mall Staff 184060.00 $tone November 27, 2020
Public

Hi there,

Please try the steps below:

1. Upgrade to the latest UB Data Migration Pro v3.2.5
2. Then, run the following SQL query in your M2 database:

UPDATE `ub_migrate_map_step_5` SET `created_time` = '2000-01-01 00:00:00' Where entity_name = 'catalog_product_entity';

3. Once done, run the delta migration in step #5 using the CLI command:

php -f bin/ubdatamigration run --step=5 --mode=update

And let me how it goes then.

Regards,
Mall.
 
 

#1
Profile photo of Mall Staff 184060.00 $tone November 27, 2020
Public

Hi there,

delta migration is running.

 
But not effect in table

That is strange. Please re-correct databases settings in the configuration file of our module at path:
pub/ub-tool/protected/config/db.php
Let’s make sure that, the credentials of the param named ‘db’ are current your M2 database.
Regards,
Mall.

#3
Profile photo of Mall Staff 184060.00 $tone November 27, 2020
Public

Hi there,

i am sure it is correct.
and Delta step 2, 3, 4 was successful.
Please check step 5 issue carefully.

So lets provide me information about your instance:
+ Admin credentials and URL to the backend of your M2
+ SSH credentials and lets me know the path to M2 folder
+ phpMyadmin URL which can view your M1 and M2 database
I will help to check further and get back to you then.
Regards,
Mall.

#5
Profile photo of biitsllc123 460.00 $tone November 27, 2020
Public

Hello.
yet, migration is running with step5.
for 5hrs, only this value was changed. 
 

from 0 to 58600
Hope this information would be helpful for you.
thank you
 

#7
Profile photo of Mall Staff 184060.00 $tone November 30, 2020
Public

Hi there,
I am sorry for replying to you this late as our team has just been back to work from our weekend. 

for 5hrs, only this value was changed. 
 

That occurred because you were running step #5 with the ‘update’ mode and it only updated products which had been changed and newly added products in M1.

I saw a total of products in your M1: 227789 products: http://prntscr.com/vsjnbv
and our migration tool had migrated a total of 222354 products: http://prntscr.com/vsjn47

2020/11/26 01:59:48 -[Processing][delta] Step #5 migration completed with 100 Products;

I checked the migration log of our module and saw that the Step #5 was stopped after that.

And I saw the status ‘Migrating’ in step #5: http://prntscr.com/vsjme8
 
So, please continue with the delta migration in step #5 at your end by running the CLI command:

php -f bin/ubdatamigration run --step=5 --mode=update

and after you finish step #5, please reindex the data again and check the issue you mentioned. And let me how it goes then.

Regards,
Mall.
 

#8
Profile photo of biitsllc123 460.00 $tone November 30, 2020
Public

Hello!
I am sorry that you couldn’t understand our issue.
We need delta migration with update mode.

php -f bin/ubdatamigration run --step=5 --mode=update

But it runs processing from the first product (all product-  not working with delta migration mode)
so, it will take 3days to complete all 227789 products.
but sometimes, the MySQL connection was broken.
While processing for 3days, If the MySQL connection was broken, and I run the command again.
php -f bin/ubdatamigration run --step=5 --mode=update
Product migration starts with the first product again.
So, We can’t complete this migration step 5.
 
If you would have any question, please let me know
 
Thank you.
 

#9
Profile photo of Mall Staff 184060.00 $tone November 30, 2020
Public

Hi there,

but sometimes, the MySQL connection was broken.
While processing for 3days, If the MySQL connection was broken, and I run the command again.
php -f bin/ubdatamigration run --step=5 --mode=update
Product migration starts with the first product again.
So, We can’t complete this migration step 5.

I have seen your case. Please try steps below to continue with the delta migration in step #5:

1. Open the PHP file at pub/ub-tool/protected/controllers/Step5Controller.php
and find the code lines:

if ($this->runMode == UBMigrate::RUN_MODE_DELTA) {
$lastMigrationTime = UBMigrate::getLastMigrationTime($this->stepIndex, 'catalog_product_entity');
$condition .= " AND updated_at >= '{$lastMigrationTime}'";
//get all product IDs need to delta
if (!$this->strDeltaProductIds) {
$this->strDeltaProductIds = $this->_getAllDeltaProductIds($condition);
}
}

and replace it with: 

//if ($this->runMode == UBMigrate::RUN_MODE_DELTA) {
$lastMigrationTime = UBMigrate::getLastMigrationTime($this->stepIndex, 'catalog_product_entity');
$condition .= " AND updated_at >= '{$lastMigrationTime}'";
//get all product IDs need to delta
if (!$this->strDeltaProductIds) {
$this->strDeltaProductIds = $this->_getAllDeltaProductIds($condition);
}
//}

2. And then, you continue with the delta migration in step #5 by running the CLI command:

php -f bin/ubdatamigration run --step=5

Note: you don’t need to use the param “--mode=update” at this phase. And you need to wait until you finish step #5.

If your server is broken again, you can continue step #5 with that CLI command (without “--mode=update”). 

Regards,
Mall.

#10
Profile photo of biitsllc123 460.00 $tone November 30, 2020
Public

Hello.
Yes. i did.
But yet, only this field is changed.

 
is there a table that keeps track of what product ID it is on that we could maybe manipulate and insert a record to say the last product ID?
We need delta migration (not from first product)
Currently, Step5 is running.  but will take 1~2days to complete.
Thank you.
 

#11
Profile photo of Mall Staff 184060.00 $tone November 30, 2020
Public

Hi there,

is there a table that keeps track of what product ID it is on that we could maybe manipulate and insert a record to say the last product ID?

You can check the data records in the table ‘catalog_product_entity’ in your M2 database:

catalog_product_entity.updated_at (if some of the migrated products had changed information in your M1 after the delta migration, the value of this field in those products will be in your M2 too)

catalog_product_entity.entity_id (after you finished the delta migration in step #5, you will see the new ids of the newly added products in M1 after the first migration)

We need delta migration (not from first product)

Please note that our migration tool scans from the first product to run delta update for both the migrated products in the first migration.

Currently, Step5 is running.  but will take 1~2days to complete.

If you did the steps suggested in my reply #10, it means the migration process is continuing in step #5. Actually, the migration time depends on the volume of your products in M1 and your server’s performance.

Do you want to run delta update for both the migrated products in the first migration? Or you just want to run delta migration for the newly added products only in M1 after the first migration?
 
Regards,
Mall.
 
 

#12
Profile photo of biitsllc123 460.00 $tone December 1, 2020
Public

Hi there.
Now, The delta migration was completed.
But new products of M1 were not migrated to M2.
There are old products from the first migration.
 

 
M1 has 227790 products total
But M2 only shows 222355.
We already did reindexer.
Please check this issue.
Thank you.
 

#13
Profile photo of Mall Staff 184060.00 $tone December 2, 2020
Public

Hi there,

I have checked your instance and seen the issue you mentioned. Then I closed the tweaked code suggested in my reply #10: http://prntscr.com/vtzyzl (it is not needed at this phase).

Now, to solve the issue, you can try the steps below:

1. Run the following SQL query in your M2 database:

UPDATE `ub_migrate_map_step_5` SET `offset` = 222300 Where entity_name = 'catalog_product_entity';

2. Once done, run delta migration in step #5 using the CLI command:

php -f bin/ubdatamigration run --step=5

Note: you don’t need to use the param “--mode=update” at this phase. And make sure that you wait until you finish step #5.
 
In case your server is broken again, you can simply continue step #5 with that CLI command (without “--mode=update”). 
 
Regards,
Mall.
 
 

#14

Please login or Register to Submit Answer

Written By

Comments