Viewed   382 times

Using CURL to get content from website. Getting response in object. How to convert that object in to PHP Simple HTML DOM Parser

function get_data($url) 
{
    $ch = curl_init();
    $timeout = 30;
    curl_setopt($ch,CURLOPT_URL,$url);
    curl_setopt($ch,CURLOPT_RETURNTRANSFER,false);
    curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
    curl_setopt($ch,CURLOPT_POST,false);
    curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0");
    //curl_exec($ch);
    $dom = new simple_html_dom(curl_exec($ch));
    print_r( $dom );
    curl_close($ch);
    return $data;
}
$url = 'http://www.example.com';
$data = get_data($url);

?>

Result


simple_html_dom Object ( [root] => simple_html_dom_node Object ( [nodetype] => 5 [tag] => root [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( [0] => simple_html_dom_node Object ( [nodetype] => 3 [tag] => text [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( ) [parent] => simple_html_dom_node Object *RECURSION* [_] => Array ( [4] => 1 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) ) [parent] => [_] => Array ( [0] => -1 [1] => 2 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) [nodes] => Array ( [0] => simple_html_dom_node Object ( [nodetype] => 5 [tag] => root [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( [0] => simple_html_dom_node Object ( [nodetype] => 3 [tag] => text [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( ) [parent] => simple_html_dom_node Object *RECURSION* [_] => Array ( [4] => 1 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) ) [parent] => [_] => Array ( [0] => -1 [1] => 2 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) [1] => simple_html_dom_node Object ( [nodetype] => 3 [tag] => text [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( ) [parent] => simple_html_dom_node Object ( [nodetype] => 5 [tag] => root [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( [0] => simple_html_dom_node Object *RECURSION* ) [parent] => [_] => Array ( [0] => -1 [1] => 2 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) [_] => Array ( [4] => 1 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) ) [callback] => [lowercase] => 1 [original_size] => 1 [size] => 1 [pos:protected] => 1 [char:protected] => [cursor:protected] => 2 [parent:protected] => simple_html_dom_node Object ( [nodetype] => 5 [tag] => root [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( [0] => simple_html_dom_node Object ( [nodetype] => 3 [tag] => text [attr] => Array ( ) [children] => Array ( ) [nodes] => Array ( ) [parent] => simple_html_dom_node Object *RECURSION* [_] => Array ( [4] => 1 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) ) [parent] => [_] => Array ( [0] => -1 [1] => 2 ) [tag_start] => 0 [dom:private] => simple_html_dom Object *RECURSION* ) [token_blank:protected] => [token_equal:protected] => =/> [token_slash:protected] => /> [token_attr:protected] => > [_charset] => UTF-8 [_target_charset] => UTF-8 [default_br_text:protected] => [default_span_text] => [self_closing_tags:protected] => Array ( [img] => 1 [br] => 1 [input] => 1 [meta] => 1 [link] => 1 [hr] => 1 [base] => 1 [embed] => 1 [spacer] => 1 ) [block_tags:protected] => Array ( [root] => 1 [body] => 1 [form] => 1 [div] => 1 [span] => 1 [table] => 1 ) [optional_closing_tags:protected] => Array ( [tr] => Array ( [tr] => 1 [td] => 1 [th] => 1 ) [th] => Array ( [th] => 1 ) [td] => Array ( [td] => 1 ) [li] => Array ( [li] => 1 ) [dt] => Array ( [dt] => 1 [dd] => 1 ) [dd] => Array ( [dd] => 1 [dt] => 1 ) [dl] => Array ( [dd] => 1 [dt] => 1 ) [p] => Array ( [p] => 1 ) [nobr] => Array ( [nobr] => 1 ) [b] => Array ( [b] => 1 ) [option] => Array ( [option] => 1 ) ) [doc:protected] => 1 [noise:protected] => Array ( ) ) 


 Answers

3

You're not creating the DOM correctly, you must do it like this:

// Create a DOM object
$dom = new simple_html_dom();
// Load HTML from a string
$dom->load(curl_exec($ch))

print_r( $dom );

Check the Manual for more details...

Edit

It seems that is a cURL settings problem, please refer to the documentation to configure it correctly...

This is a function I usualy use to download some pages, feel free to adjust it to your needs:

function dlPage($href) {

    $curl = curl_init();
    curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_setopt($curl, CURLOPT_HEADER, false);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($curl, CURLOPT_URL, $href);
    curl_setopt($curl, CURLOPT_REFERER, $href);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
    curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.125 Safari/533.4");
    $str = curl_exec($curl);
    curl_close($curl);

    // Create a DOM object
    $dom = new simple_html_dom();
    // Load HTML from a string
    $dom->load($str);

    return $dom;
    }

$url = 'http://www.example.com/';
$data = dlPage($url);
print_r($data);
Sunday, August 14, 2022
3

@deceze and @Shakti thanks for your help.

+1 for the article link posted by deceze (Handling Unicode Front to Back in a Web App) and it also worth reading Understanding encoding

After reading your comments, answer and of course those two articles, I finally solved my issue.

I have listed the steps I did so far to solve this issue:

  1. Added header('Content-Type: text/html; charset=utf-8'); on the top of my init.php file,
  2. Changed CHARACTER SET of my database table field which is storing those value to UTF-8,
  3. Set MySQL connection charset to UTF-8 mysql_set_charset('utf8', $connection_link_id);
  4. Used htmlentities() function to convert characters $meta_title = htmlentities(trim($meta_title_raw), ENT_QUOTES, 'UTF-8');

Now the issue seems to be solved, BUT I still have to do following thing to solve this issue in FULL.

  1. Get the encoded charset from the source $source_charset.
  2. Change the encoding of the string into UTF-8 if it is already not in the same encoding. For this the only available PHP function is iconv(). Example: iconv($source_charset, "UTF-8", $meta_title_raw);

For getting $source_charset I probably have to use some tricks or multi checking. Like checking headers and meta tag etc. I found a good answer at Detect encoding

Let me know if there are any improvements or any fault on my steps above.

Sunday, August 7, 2022
 
4

Isn't it easy. Try things first then ask. (:

<?php
include 'simple_html_dom.php';
$html = file_get_html('http://www.weather.gov.sg/lws/zoneInfo.do');

$n = 0;
$table = $html->find('table',3)->find('table',0)->find('table',0)->find('table',0)->find('table',3)->find('table',0);

$i = -3;
$rows = $table->find('tr');
$holder = array();

foreach($rows as $element){
    $i++;
    if($i < 0) continue;

    $holder[$i]['name'] = $element->find('td',0)->plaintext;
    $holder[$i]['zone_or_school'] = $element->find('td',1)->plaintext;
    $holder[$i]['risk'] = $element->find('td',2)->plaintext;
    $holder[$i]['from'] = $element->find('td',3)->plaintext;
    $holder[$i]['till'] = $element->find('td',4)->plaintext;
}

var_dump($holder);
?>

if you want to get a particular data then you can filter it out:

foreach($holder as $key => $val)
{
if($holder[$key]['name']=='Bedoc')
$my_data = $holder[$key];
}

this code isn't debuged cause i am on mobile now. But maybe you have get the idea if not works. Thanks

Friday, December 9, 2022
 
4

Here is some (untested) code that should help you get started using the Steam Web API:

$api_key = '...';
$steam_id = '...';
$game_id = 550;

$json = file_get_contents('http://api.steampowered.com/IPlayerService/GetOwnedGames/v0001/?key=' . $api_key . '&steamid=' . $steam_id . '&format=json');

$data = json_decode($json);
foreach ($data->response->games as $game) { 
    if ($game->appid == $game_id) {
        echo 'Playtime for ', $game_id, ' is ' . $game->playtime_forever;
    }
}
Thursday, December 8, 2022
 
addev
 
2

I think, you may want something like this

$url = 'http://www.peopleperhour.com/freelance-seo-jobs';
$html = file_get_html($url);
foreach ($html->find('div.item-list div.item') as $div) {
    echo $div . '<br />';
};

This will give you something like this (if you add the proper style sheet, it'll be displayed nicely)

Monday, August 29, 2022
 
smghost
 
Only authorized users can answer the search term. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :