Any way to keep curl's cookies in memory and not on disk

Question 1

I'm doing some cURL work in php 5.3.0.

I'm wondering if there is any way to tell the curl handle/object to keep the cookies in memory (assuming I'm reusing the same handle for multiple requests), or to somehow return them and let me pass them back when making a new handle.

Theres this long accepted method for getting them in/out of the request:

curl_setopt($ch, CURLOPT_COOKIEJAR, $filename); 
curl_setopt($ch, CURLOPT_COOKIEFILE, $filename);

But I'm hitting some scenarios where I need to be running multiple copies of a script out of the same directory, and they step on each others cookie files. Yes, I know I could use tempnam() and make sure each run has its own cookie file, but that leads me to my 2nd issue.

There is also the issue of having these cookie files on the disk at all. Disk I/O is slow and a bottle neck I'm sure. I dont want to have to deal with cleaning up the cookie file when the script is finished (if it even exits in a way that lets me clean it up).

Any ideas? Or is this just the way things are?

Question 2

You can use the CURLOPT_COOKIEJAR option, and set the file to "/dev/null" for Linux / MacOS X or "NULL" for Windows. This will prevent the cookies from being written to disk, but it will keep them around in memory as long as you reuse the handle and don't call curl_easy_cleanup().

Question 3

cURL is not able to verify the authenticity of the certificate being used, because the certificate for the signing authority cannot be found in the local database.

This might be symptomatic of a self signed certificate being used.

What you should do is add the certificate for the signing authority to /etc/ssl/certs/ca-certificates.crt.

What you can do is use curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);, which will disable the check that is failing.

You really should use the first option.

Question 4

you should check examples from here http://php.net/manual/en/curl.examples.php

Bellow is the code for you case,

$url = "http://build:f9280f75396f83a0@mobile-jenkins.domain.com:8080/job/android-test/buildWithParameters";     
$data = "POST_RESULTS=true&RUN_ID=".$testrun_id."&CHECK_NAME=SampleAutomatedPlan";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);  
curl_setopt($ch, CURLOPT_POSTFIELDS, $data); 

// $output contains the output string
$output = curl_exec($ch);

// close curl resource to free up system resources
curl_close($ch);

Question 5

To answer the question, here's a simple/preliminary patch to not evict Active(file)(as seen in /proc/meminfo) if it's less than 256 MiB, that seems to work ok (no disk thrashing) with linux-stable 5.2.4:

diff --git a/mm/vmscan.c b/mm/vmscan.c
index dbdc46a84f63..7a0b7e32ff45 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2445,6 +2445,13 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg,
            BUG();
        }

+    if (NR_ACTIVE_FILE == lru) {
+      long long kib_active_file_now=global_node_page_state(NR_ACTIVE_FILE) * MAX_NR_ZONES;
+      if (kib_active_file_now <= 256*1024) {
+        nr[lru] = 0; //don't reclaim any Active(file) (see /proc/meminfo) if they are under 256MiB
+        continue;
+      }
+    }
        *lru_pages += size;
        nr[lru] = scan;
    }

Note that some ~~yet-to-be-found~~ regression on kernel 5.3.0-rc4-gd45331b00ddb will cause a system freeze(without disk thrashing, and sysrq will still work) even without this patch.

(any new developments related to this should be happening here.)

Question 6

libpq, the underlying PostgreSQL client library, has the keepalives option to enable TCP keepalives.

It looks like PgAdmin-III doesn't allow you to specify arbitrary connection parameters directly, but there's a workaround.

When you look at the connection configuration in PgAdmin-III, you will see a "service" option. This refers to the connection service file. To use it, create a ~/.pg_service.conf with contents like:

[myherokudb]
user=myusername
keepalives=1
connect_timeout=20
keepalives_idle=10

and when connecting from PgAdmin-III enter myherokudb in the service field.

This will cause PgAdmin-III to use the connection parameters specified in the service file, including enabling keepalives.

(If you're on Windows, the service file may be in another location; see the documentation).

There's no environment variable in libpq to control keepalives, so you can't set it that way, you'll have to use a service file.

Adding support for additional connection parameters to PgAdmin-III, or a checkbox in the connection options to control the keepalives parameter, should be pretty trivial. I wonder if Dave understood what you were asking for re your offer to fund the work.

Update: The service file is looked up at the location specified in the PGSYSCONFDIR environment variable. If unset, it defaults to a platform-specific location, which doesn't seem to be documented properly for Windows. I'll submit a documentation patch. The documentation for .pgpass shows its path as %APPDATA%postgresqlpgpass.conf though, so ~/.pg_service.conf should be %APPDATA%postgresqlpg_service.conf ... but it doesn't seem to be.

In fact, the correct path is:

%APPDATA%postgresql.pg_service.conf

So:

Start->Run
`%APPDATA%
create directory "postgresql" if it does not exist
create file ".pg_service.conf" as a text file with the contents given above (see note below re file naming)
In PgAdmin-III, enter "localhost" in the Host name, and the service name in the service field.

I tested on Windows, and found that you can't leave the host field in PgAdmin-III blank on Windows. PgAdmin-III seems to override any host specified in the service file with what's specified in the connection dialog. So you should not include a host key in the service file. (I'll report a bug).

Make "hide file extensions for known file types" is turned off in Windows, so you don't accidentally call it .pg_service.conf.txt instead. If you're unsure whether it's named right or not, check the "Type" column in Windows Explorer in list view; it will read "Text Document" if it's incorrectly named .pg_service.conf.txt, and CONF File if it's correctly named .pg_service.conf. If you have problems renaming it, turn off "hide file extensions for known file types", or use a sensible text editor like notepad++ that'll let you create files named however you like.

Note the leading period (dot) in the filename. Yes, that's different to pgpass.conf, and yes, that's annoying, bordering on a bug.

Any way to keep curl's cookies in memory and not on disk

Answers

php,curl,diskspace