How to POST a large amount of data within PHP curl without memory overhead?

Question 1

I'm currently using PHP curl extension to communicate with some HTTP APIs.

I use a bulk loader to perform a lot of operations at once. The bulk endpoint must be called with POST method so I use a code like :

<?php
$h = curl_init($url);
curl_setopt(CURLOPT_POST, true);
curl_setopt(CURLOPT_POSTFIELDS, $data);
curl_setopt(CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($h);
curl_close($h);

The bulk endpoint allows me to send a huge amount of data (more than 200Mo at once). For the moment I need to load the data inside a variable which require PHP to be able to use enough memory...

I need to set memory_limit to a high value just for the bulk load...

Is there a way to use a file stream to send data with the curl PHP extension ? I saw the CURLOPT_INFILE and CURLOPT_READFUNCTION but it seems to don't work with POST method...

I also saw that the curl command line tool is able to perform a --data "@/path/to/file/content" which seems to be what I need...

Any ideas ?

Question 2

Use CURLOPT_INFILE

$curl = curl_init();
curl_setopt( $curl, CURLOPT_PUT, 1 );
curl_setopt( $curl, CURLOPT_INFILESIZE, filesize($tmpFile) );
curl_setopt( $curl, CURLOPT_INFILE, ($in=fopen($tmpFile, 'r')) );
curl_setopt( $curl, CURLOPT_CUSTOMREQUEST, 'POST' );
curl_setopt( $curl, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json' ] );
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
$result = curl_exec($curl);
curl_close($curl);
fclose($in);

Question 3

# GET query goes in the URL you're hitting
$ch = curl_init('http://example.com/script.php?query=parameter');
# POST fields go here.
curl_setopt($ch, CURLOPT_POSTFIELDS, array('post' => 'parameter', 'values' => 'go here'));

PHP itself wouldn't decide to ignore the GET parameters if a POST is performed. It'll populate $_GET regardless of what kind of http verb was used to load the page - if there's query parameters in the URL, they'll go into $_GET.

If you're not getting $_POST and $_GET with this, then something is causing a redirect or otherwise killing something. e.g. have you check $_SERVER['REQUEST_METHOD'] to see if your code is actually running as a POST? PHP won't populate $_POST if a post wasn't actually performed. You may have sent a post to the server, but that doesn't mean your code will actually be executed under a POST regime - e.g. a mod_rewrite redirect.

Since you have FOLLOW_REDIRECT turned on, you're simply ASSUMING you're actually getting a post when your code executes.

Question 4

What you want to do, I believe, is something like this:

Client --> Server A

           Server A --> POST --> Server B

Client <------------------------ Server B

so that, for example, the client could login to Server B without knowing the password, which is known to Server A.

If this is so, you can do something similar, but not exactly what you want (which could be something like OpenID, though, and solvable with OpenID).

You can have Server A do the POST and receive the answer, and send the answer to Client. Unfortunately, you probably can't set cookies (they would be valid for A subdomain, and they wouldn't be sent to Server B anymore) and sessions are likely not to work for similar reasons.

You might be able to have Server A working as a complete proxy: see this answer How can I scrape website content in PHP from a website that requires a cookie login? .

Payment gateway

Most banks have API to do exactly that (Paypal, even it's not a bank, does, and so does WorldPay).

One possible workflow is to send all the data to the bank, which responds with an unique ID. You then either show all the information yourself or (much preferred by the banks) the bank shows the information to the customers, when you redirect them using a special URL and the unique ID.

The customer can change the data in his form -- but all that he obtains is to abort the transaction, for the two data copies no longer agree, and he can't touch the copy you sent (other methods and workflows of course exist).

If your system works according to this workflow (or similar) and using the bank's own API and suggested practices, please disregard and accept my apologies: you're doing it right. But just in case you are not, well, please think it over.

Trying to craft a custom workflow with cURL is maybe possible (for some banks it is definitely possible), but it is suspiciously close to rolling your own cryptography, is likely to be less supported by the bank, and might trigger some anomaly detector on the bank's part (just to cite one, lots of payments would appear to come from the same IP address or range).

Question 5

You can create an own file-like object and pass to TarFile.addfile. Your file-like object will generate the encrypted contents on the fly in the fileobj.read() method.

Question 6

Loading the entire thing into a single static readonly data-structure (it being immutable means once constructed it can be safely used from many threads) would give the greatest overall performance per lookup.

However, it would lead to a long start-up time, which may not be acceptable. In that case you may consider loading each item as accessed, but that brings concurrency issues (since you are mutating a data-structure used by multiple threads).

In between is the option of loading all indices on start-up, and then adding the rest of the information on a per-access basis, with finer-grained locks to reduce lock contention.

Or you can ignore the lot and just load from the database as needed. This does have some advantages in terms of performance just because memory is not used for rarely-used information. It'll be a whole lot easier if you ever suddenly find that you do have to allow the data to change.

No one comes out as the only reasonable way to go in general, it'll depend on specifics of application, data, and usage patterns.

How to POST a large amount of data within PHP curl without memory overhead?

Answers

Payment gateway

php,post,curl