Downloading a large file using curl

Question 1

I need to download remote file using curl.

Here's the sample code I have:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$st = curl_exec($ch);
$fd = fopen($tmp_name, 'w');
fwrite($fd, $st);
fclose($fd);

curl_close($ch);

But it can't handle big files, because it reads to memory first.

Is it possible to stream the file directly to disk?

Question 2

<?php
set_time_limit(0);
//This is the file where we save the    information
$fp = fopen (dirname(__FILE__) . '/localfile.tmp', 'w+');
//Here is the file we are downloading, replace spaces with %20
$ch = curl_init(str_replace(" ","%20",$url));
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
// write curl response to file
curl_setopt($ch, CURLOPT_FILE, $fp); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
// get curl response
curl_exec($ch); 
curl_close($ch);
fclose($fp);
?>

Question 3

This is pretty easy. All you need to do is provide cURL with a callback to handle data as it comes in.

function onResponseBodyData($ch, $data)
{
    echo $data;
    return strlen($data);
}

curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'onResponseBodyData');

Returning the length of data from your callback is important. It signifies how much data you processed. If you return something other than the length of data passed in (such as 0), then the request is aborted.

Now, make sure you don't have output buffering turned on, and configure your server to not buffer the entire response before sending. It will work out of the box on most configurations.

You can find more examples here: http://curl.haxx.se/libcurl/php/examples/callbacks.html

Question 4

You could use streams.Try something like this on the client:

InputStream fileInStream = new FileInputStream(fileName);
String sContentDisposition = "attachment; filename="" + fileName.getName()+""";
WebResource fileResource = a_client.resource(a_sUrl);       
ClientResponse response = fileResource.type(MediaType.APPLICATION_OCTET_STREAM)
                        .header("Content-Disposition", sContentDisposition)
                        .post(ClientResponse.class, fileInStream);

with resource like this on the server:

@PUT
@Consumes("application/octet-stream")
public Response putFile(@Context HttpServletRequest a_request,
                         @PathParam("fileId") long a_fileId,
                         InputStream a_fileInputStream) throws Throwable
{
    // Do something with a_fileInputStream
    // etc

Question 5

O_LARGEFILE should never be used directly by applications. It's to be used internally by the 64-bit-offset-compatible version of open in libc when it makes the syscall to the kernel (Linux, or possibly another kernel with this 64-bit-offset-mode-is-a-second-class-citizen nonsense). Just make sure to always include -D_FILE_OFFSET_BITS=64 in your CFLAGS and you'll never have to worry about anything.

Question 6

Give this a go

<?php

    $output_filename = "testfile.igc";

    $host = "http://www.xcontest.org/track.php?t=2avxjsv1.igc";
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $host);
    curl_setopt($ch, CURLOPT_VERBOSE, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_AUTOREFERER, false);
    curl_setopt($ch, CURLOPT_REFERER, "http://www.xcontest.org");
    curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    $result = curl_exec($ch);
    curl_close($ch);

    print_r($result); // prints the contents of the collected file before writing..


    // the following lines write the contents to a file in the same directory (provided permissions etc)
    $fp = fopen($output_filename, 'w');
    fwrite($fp, $result);
    fclose($fp);
?>

#

or if you want to put it within a loop for parsing several links... you need some functions.. here is a rough idea....

<?php

    function collect_file($url){
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_VERBOSE, 1);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_AUTOREFERER, false);
        curl_setopt($ch, CURLOPT_REFERER, "http://www.xcontest.org");
        curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
        curl_setopt($ch, CURLOPT_HEADER, 0);
        $result = curl_exec($ch);
        curl_close($ch);
        return($result);
    }

    function write_to_file($text,$new_filename){
        $fp = fopen($new_filename, 'w');
        fwrite($fp, $text);
        fclose($fp);
    }


    // start loop here

    $new_file_name = "testfile.igc";
    $url = "http://www.xcontest.org/track.php?t=2avxjsv1.igc";

    $temp_file_contents = collect_file($url);
    write_to_file($temp_file_contents,$new_file_name)

    // end loop here
?>

Downloading a large file using curl

Answers

php,curl,download