Restructuring a bad database with PHP loops or MySQL

Question 1

I have a special case with a PHP project where I am working with a database without a 3rd normal form structure. The database consists of only 1 table with loads of columns. Some data that is supposed to be in a separate table, is clamped in 1 column, separated by a certain sign (in this case, semicolon ";").

There is also other columns where clamped data should be moved to the same separated table as mentioned. This must be confusing, so let me elaborate:

**HugeTable**
id |  Column1      |  Column2     |  Column3

123 | Data1;Data2 Data3;Data4 Data5;Data6

I need to put the data above in a separate table that looks like this:

**NewTable**
id   |  idHugeTable  |  Column1  |  Column2  |  Column3
 1   |  123          |  Data1    |  Data3    |  Data5
 2   |  123          |  Data2    |  Data4    |  Data6

So for each clamped data in the huge table, I need to make a new row in the new table. This process would help me to normalize the database so that it is at least workable. Right now it's a nightmare. This needs to be done either through PHP or MySQL, preferably PHP since looping is easier for one-shot queries per loop through the scripting language.

Edit: Example code of what I have tried in PHP:

$delimiter = ";";
$query = "SELECT * FROM HugeTable";

$result = mysqli_query($connection_var, $query);

while ($row = mysqli_fetch_assoc()){
    $column1_data = explode($delimiter, $row['Column1']);
    $column2_data = explode($delimiter, $row['Column2']);
    $column3_data = explode($delimiter, $row['Column3']);

    foreach ($column1_data as $key => $value){
        //skip if empty value
        if ($value == ""){
            continue;
        }
        else{
            $query_ins = "INSERT INTO NewTable (idHugeTable, Column1, Column2, Column3) VALUES (".$row['id'].", ".$column1_data[$key].", ".$column2_data[$key].", ".$column3_data[$key].");";
            mysqli_query($connection_var, $query_ins);
        }//end if
    }//end foreach
}//end while

mysqli_close($connection_var);

Question 2

No PHP is needed. You can do it with pure MySQL code only.

Create table/insert table

CREATE TABLE HugeTable
    (`Column1` VARCHAR(11), `Column2` VARCHAR(11), `Column3` VARCHAR(11))
;

INSERT INTO HugeTable
    (`Column1`, `Column2`, `Column3`)
VALUES
    ('Data1;Data2', 'Data3;Data4', 'Data5;Data6')
; 

CREATE TABLE NewTable
   (`Column1` VARCHAR(11), `Column2` VARCHAR(11), `Column3` VARCHAR(11))
;

First we need MySQL to generate numbers. This MySQL code generates 1 to 100. So the final query will support up to 100 separated values.

Query

SELECT 
 @row := @row + 1 AS ROW
FROM (
  SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) row1
CROSS JOIN (
  SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) row2
CROSS JOIN (
  SELECT @row := 0 
) init_user_params

Result

Now we can look at a method to separate on the ; delimiter. We can use nested SUBSTRING_INDEX functions for that

Query

SELECT SUBSTRING_INDEX(SUBSTRING_INDEX('Data1;Data2', ';', 1), ';', -1) AS DATA

Result

data    
--------
Data1

You can see only the first word is returned if we want the second word we can use

Query

SELECT SUBSTRING_INDEX(SUBSTRING_INDEX('Data1;Data2', ';', 2), ';', -1) AS DATA

Result

data    
--------
Data2

Now we combine the number generator and the SUBSTRING_INDEX to generate the data

Query

SELECT 
  DISTINCT
   SUBSTRING_INDEX(SUBSTRING_INDEX(Column1, ';', rows.row), ';', -1) Column1
 , SUBSTRING_INDEX(SUBSTRING_INDEX(Column2, ';', rows.row), ';', -1) Column2
 , SUBSTRING_INDEX(SUBSTRING_INDEX(Column3, ';', rows.row), ';', -1) Column3
FROM (
  SELECT 
   @row := @row + 1 AS ROW
  FROM (
    SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
  ) row1
  CROSS JOIN (
    SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
  ) row2  
  CROSS JOIN (
    SELECT @row := 0 
  ) init_user_params
)
 ROWS
CROSS JOIN 
 HugeTable

Result

Column1  Column2  Column3  
-------  -------  ---------
Data1    Data3    Data5    
Data2    Data4    Data6

Query NewTable

INSERT INTO 
  NewTable
SELECT 
  DISTINCT
   SUBSTRING_INDEX(SUBSTRING_INDEX(Column1, ';', rows.row), ';', -1) Column1
 , SUBSTRING_INDEX(SUBSTRING_INDEX(Column2, ';', rows.row), ';', -1) Column2
 , SUBSTRING_INDEX(SUBSTRING_INDEX(Column3, ';', rows.row), ';', -1) Column3
FROM (
  SELECT 
   @row := @row + 1 AS ROW
  FROM (
    SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
  ) row1
  CROSS JOIN (
    SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
  ) row2  
  CROSS JOIN (
    SELECT @row := 0 
  ) init_user_params
)
 ROWS
CROSS JOIN 
 HugeTable

Query

SELECT * FROM NewTable

Result

Column1  Column2  Column3  
-------  -------  ---------
Data1    Data3    Data5    
Data2    Data4    Data6

Question 3

If you want to select the ID of the most recently inserted row in a table with an AUTO_INCREMENT column, you will likey be interested in MySQL's LAST_INSERT_ID function.

Question 4

You need to break this down into two problems:

Generate a set of polygons from each section of your path.
Do a simple point in polygon test on the above polygons.

There should be some php out there to accomplish those two tasks.

Question 5

MySQL has a different timeout than PHP. You could increase it in php.ini on the line mysql.connect_timeout = 14400. Also increase the default_socket_timeout = 14400

Note that if your PHP setting allow you to do an ini_set, you can also do as follows:

ini_set('mysql.connect_timeout', 14400);
ini_set('default_socket_timeout', 14400);

Question 6

The details are implementation dependent but generally speaking, results are buffered. Executing a query against a database will return some result set. If it's sufficiently small all the results may be returned with the initial call or some might be and more results are returned as you iterate over the result object.

Think of the sequence this way:

You open a connection to the database;
There is possibly a second call to select a database or it might be done as part of (1);
That authentication and connection step is (at least) one round trip to the server (ignoring persistent connections);
You execute a query on the client;
That query is sent to the server;
The server has to determine how to execute the query;
If the server has previously executed the query the execution plan may still be in the query cache. If not a new plan must be created;
The server executes the query as given and returns a result to the client;
That result will contain some buffer of rows that is implementation dependent. It might be 100 rows or more or less. All columns are returned for each row;
As you fetch more rows eventually the client will ask the server for more rows. This may be when the client runs out or it may be done preemptively. Again this is implementation dependent.

The idea of all this is to minimize roundtrips to the server without sending back too much unnecessary data, which is why if you ask for a million rows you won't get them all back at once.

LIMIT clauses--or any clause in fact--will modify the result set.

Lastly, (7) is important because SELECT * FROM table WHERE a = 'foo' and SELECT * FROM table WHERE a = 'bar' are two different queries as far as the database optimizer is concerned so an execution plan must be determined for each separately. But a parameterized query (SELECT * FROM table WHERE a = :param) with different parameters is one query and only needs to be planned once (at least until it falls out of the query cache).

Restructuring a bad database with PHP loops or MySQL

Answers

php,mysql,database,loops