I make txt upload to mysql using php.
The number of lines in the document is 100.000+.
We select a file, save the file on the server, open the file for reading, read and write data to the array line by line, pass the array to the function that loads the data to the database (CMS WP).
Approximately 3,797 receive (504 Gateway Time-out).

function post_integration($arr, $array) { global $wpdb; $cat = $array['category']; $manufacturer = $array['manufacturer']; $return_status = $array['return_status']; $markup = $array['markup']; for ( $i = 1, $j=0; $i < count($arr); $i++, $j++ ) { set_time_limit(0); $vendor_code = ( iconv ("CP1251", "UTF-8", $arr[$i][0]) ); $vendor_code = substr($vendor_code, 1); $product_name = ( iconv ("CP1251", "UTF-8", $arr[$i][1]) ); $price = ( iconv ("CP1251", "UTF-8", $arr[$i][2]) ); $rabat_group = ( iconv ("CP1251", "UTF-8", $arr[$i][3]) ); $product_desc = ( iconv ("CP1251", "UTF-8", $arr[$i][4]) ); $mortgage_value = ( iconv ("CP1251", "UTF-8", $arr[$i][5]) ); $netto_weight = ( iconv ("CP1251", "UTF-8", $arr[$i][6]) ); $netto_weight = substr($netto_weight, 0, -1); $price = str_replace(",",".",$price); $this_price = ( ($markup / 100) * $price) + $price; $end_price = round($this_price, 2); $query="SELECT vendor_code.meta_value AS \"vendor_code\", p.ID AS \"ID\" FROM wp_posts p JOIN wp_postmeta vendor_code ON p.ID = vendor_code.post_id AND vendor_code.meta_key = 'vendor_code' WHERE post_status = 'publish' AND post_type = 'post' AND vendor_code.meta_value = $vendor_code"; $rabrt_id_query = "SELECT id FROM `wp_rabat_list` WHERE name_rabat = '$manufacturer'"; $result_data = $wpdb->get_results($query); $rabat_data = $wpdb->get_results($rabrt_id_query); $rabat_id = $rabat_data[0]->id; $result_id = $result_data[0]->ID; $result_vendor = $result_data[0]->vendor_code; if(isset($result_vendor)) { $my_post = array(); $my_post['ID'] = intval($result_id); $my_post['post_title'] = wp_strip_all_tags($product_name); $my_post['post_content'] = $product_desc; $my_post['post_category'] = array($cat); wp_update_post($my_post); update_post_meta($my_post['ID'], 'vendor_code', $vendor_code); update_post_meta($my_post['ID'], 'price', $end_price); update_post_meta($my_post['ID'], 'manufacturer', $manufacturer); update_post_meta($my_post['ID'], 'mortgage_value', $mortgage_value); update_post_meta($my_post['ID'], 'netto_weight', $netto_weight); // update_post_meta($my_post['ID'], 'availability', $article); update_post_meta($my_post['ID'], 'return_goods', $return_status); update_post_meta($my_post['ID'], 'discount_group_id', $rabat_id); update_post_meta($my_post['ID'], 'mark_up', $markup); } else { $post_data = array( 'post_title' => wp_strip_all_tags( $product_name ), 'post_content' => $product_desc, 'post_status' => 'publish', 'post_author' => 1, 'post_category' => array($cat) ); //Вставляем запись в базу данных $post_id = wp_insert_post( $post_data ); add_post_meta($post_id, 'vendor_code', $vendor_code); add_post_meta($post_id, 'price', $end_price); add_post_meta($post_id, 'manufacturer', $manufacturer); add_post_meta($post_id, 'mortgage_value', $mortgage_value); add_post_meta($post_id, 'netto_weight', $netto_weight); // add_post_meta($post_id, 'availability', $article); add_post_meta($post_id, 'return_goods', $return_status); add_post_meta($post_id, 'discount_group_id', $rabat_id); update_post_meta($my_post['ID'], 'mark_up', $markup); } } }
  • If this server is yours, then you need to correct the settings so that it does not drop the connection, if there is a provider, then connect with those support and explain your situation to them. - Paulo Berezini

1 answer 1

Try running this long process in the background: test.php is your script that loads txt into the database

 <?php exec("/path/to/file/test.php &"); ?>

or even from the console, if you can:

 php /path/to/file/test.php

You can also try to increase the time of the script, writing in the beginning

 set_time_limit(сколько секунд)
  • Option with the console does not work, it is necessary that the solution worked with the administrative office. Those. This is a plugin that loads huge txt files with goods in the database. - Vlad
  • The host wrote that it does not allow to work with exec - Vlad
  • You can try to increase the execution time of set_time_limit (3600), but IMHO normal hosting should give the opportunity to work with exec :( - Solid
  • Yes, I have already measured) The customer in the prod will take the VPS. I will now break into files and load everything separately, the solution is not the best, but I need to work) I will do exec for the prod! - Vlad
  • If the database can be managed via phpmyadmin. That can convert txt to sql and try to import. - Solid