Support ⇒ New installations ⇒ Upgrading 9.0.6.1 to 9.2.1 with huge database ⇒ Community Forums ⇒ CPG Dragonfly™ CMS
Forum IndexNew installations

Upgrading 9.0.6.1 to 9.2.1 with huge database Reply to topic


Hi,

I need to change my server to new versions of os, mysql, php and df.

I read that the direct upgrade from 9.0.6.1 to 9.2.1 should be no problem.
When starting install.php everything looks fine, unfortunately the db-conversion stops at "Executing the queries......." after a while.

Apache tells me the reason: "Timeout waiting for output from CGI script", the database has grown to 750MB (search-index already removed).

Is there any recommendation on how to upgrade with huge databases?

regards
Lutz

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


increase the timeout at apache conf level, set it up to a very high value ... ssh to the server and look for background processes running (use top)... mysql will be at the top when the upgrade runs ... this way you know something is happening, however you should wait, next release was suppose to be ready last sunday and it didnt come out simply because he is trying to fix the pm counter box bug

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
none available


Thanks, that helped! I was only with the php5 and cgi-settings...

I do not have that much time to wait, the server has been hacked and is offline quite a while. I switch to another server and build all new except the data, so an update to the latest release should be easy (hopefully).

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


Ok, we are back online.
I reindexed the search-tables with a script found here, but it seems that the job has not been completed due to the large database (it took lots of hours).
The search results contain not all relevant threads...

Is there any way to reindex without using a php-script?

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


at the beginning of the script try using
set_time_limit(0);

.:: I met php the 03 December 2003 :: Unforgettable day! ::.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
CloudLinux / Apache 2.4 LSAPI / MySQLi 5.6 / PHP 5.6 / DCVS


Hi NanoCaiordo, thanks for the hint and the work on dragonfly!

So whe have

indexing.php: set_time_limit(0);

/etc/apache2/apache.conf: TimeOut 1000000

should "php.ini: max_execution_time = 30" be changed?
Any other value? What about "Maximum session lifetime", memory-limit in php5 and apache2?

Best regards,
Lutz

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


Hi again. Thanks for the help but its not running yet.

I tried set_time_limit(0) and all settings I can imagine which may cause failing of the script. It still times out, sometimes after 20 minutes, sometimes after less. Result is a Internal server error without any hint on the cause in the logs.
CPU and memory consumption look alright, it seems to be somewhere in the php/apache/mysql configuration, the database content or a combination of all.

I think the best way to go on would be to translate the script to be called from commandline. The script itself has been posted by Phoenix here: dragonflycms.org/Forum...c/t=23356/

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


Your topic is about upgrading, Phoenix is about rebuilding wordlist in forums.
Two totally different things.

Try
<?php
ignore_user_abort(1);
set_time_limit(0);
?>

.:: I met php the 03 December 2003 :: Unforgettable day! ::.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
CloudLinux / Apache 2.4 LSAPI / MySQLi 5.6 / PHP 5.6 / DCVS


any time i had to upgrade a really big site db being 5+gb i always make a copy and put it on my server(in house ) and upgrade it then copy the upgraded version to production server. i have full control of my in house server. most shared hosting have there's limited and you will get timed out every time. use sqlyog if they allow remote connect to db to import large db with out time outs.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Multiple Setups


Hi NanoCaiordo,
you are right, topic start was my upgrade to the latest version. But when restoring database dumps I always have duplicate-key errors in the indexing tables. I thought it would be better to keep these steps in (my) upgrade together in one topic but I can also start a new thread for this issue if you wish.

Selectively deleting the errorness entries in the database dump is not an option. There are too much entries in one insert statement, deleting one insert statement leads to gaps in the index. Thats why I want to clear the index-tables and rebuild the search index.

The additional ignore_user_abort(1); does not change the behavior ;(

I have full control of the server, its not shared. But I did not yet find out to get the root cause of the Server Error on my testsite without exposing debug-information to the production sites.

Best regards
Lutz

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


Try increasing memory_limit to something higher.

I don't know where your problem is now, but I suggest you to:

1. upgrade, check everything its ok and get a backup
2. rebuilt your forum tables, not sure why you need to to that, but do this as a totally separate operation.

.:: I met php the 03 December 2003 :: Unforgettable day! ::.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
CloudLinux / Apache 2.4 LSAPI / MySQLi 5.6 / PHP 5.6 / DCVS


Thats exactly that what I did.
After moving backup-data to the new server and extracting the 9.2.1 as a fresh webroot, the upgrade worked like a charm.

Only rebuilding the tables will not work completely ;(

I tried already increasing mem, but it probably i changed not on all needed places.

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1


Hi again, finally I found the settings needed for this long running script.

In the script:
set_time_limit(0);
ignore_user_abort(1);

apache-config:
TimeOut 300000

php-config:
memory_limit = 4096M

fcgid.conf:
BusyTimeout 300000

Thanx for your hints!

Server specs (Server OS / Apache / MySQL / PHP / DragonflyCMS):
Debian/2.2./5.1.49/5.3.3/9.2.1

All times are UTC


Jump to: