pip fails with SSLError on MacOS because of outdated certificate - how to recover?

Question 1

When searching or installing packages with pip on MacOS (10.11.6), I am getting an SSL error similar to the one reproduced below.

The problem occurs for both python2 and python3. I acquired the versions (2.7.14 and 3.6.5) via www.python.org as Frameworks. The version of pip was 9.0.1. The problem is new (April 2018) and seems related to an update of OSX.

How to recover from this problem?

$ pip2 search numpy

Exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 45, in run
    pypi_hits = self.search(query, options)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 62, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 775, in request
    headers=headers, stream=True)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 522, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)

Question 2

There are two ways to recover from this problem.

Solution 1. As can be read here, Python.org sites stopped the support for older TLS versions, which breaks pip < 9.0.3. A new release of pip was provided, but because one cannot simply update via the broken pip, one has to bootstrap the installation. This worked for me:

# For python2
curl https://bootstrap.pypa.io/get-pip.py | python2
# For python3 (haven't tested this myself)
curl https://bootstrap.pypa.io/get-pip.py | python3

Credits for these commands go back to this SO post.

Solution 2. For Python 3.6 frameworks, a script Install Certificates.command is deployed and installed under /Applications/Python 3.6/. Executing it resolved the problem, but only for pip3.

Question 3

Updated answer

If you can establish a connection to the remote server you can use the ssl standard library module:

import ssl, socket

hostname = 'google.com'
ctx = ssl.create_default_context()
with ctx.wrap_socket(socket.socket(), server_hostname=hostname) as s:
    s.connect((hostname, 443))
    cert = s.getpeercert()

subject = dict(x[0] for x in cert['subject'])
issued_to = subject['commonName']
issuer = dict(x[0] for x in cert['issuer'])
issued_by = issuer['commonName']

>>> issued_to
u'*.google.com'
>>> issued_by
u'Google Internet Authority G2'

Original answer

Use pyOpenSSL.

from OpenSSL import crypto

cert_file = '/path/to/your/certificate'
cert = crypto.load_certificate(crypto.FILETYPE_PEM, open(cert_file).read())
subject = cert.get_subject()
issued_to = subject.CN    # the Common Name field
issuer = cert.get_issuer()
issued_by = issuer.CN

You can also access additional components, e.g. organisation (subject.O/issuer.O), organisational unit (subject.OU/issuer.OU).

Your certificate file might be in another format, so you could try crypto.FILETYPE_ASN1 instead of crypto.FILETYPE_PEM.

Question 4

The problem is caused by a missing intermediate CA certificate.

By inspecting the certificate presented by the service in OpenSSL I found out that the certificate was issued by "COMODO RSA Domain Validation Secure Server CA". The CA certificate of this particular authority is actually not present in the CA bundle of the python package certifi (there are different COMODO... certificates).

Solution

Manually download the missing certificate in PEM format from the CA's webpage and add it to the CA bundle used in your code.

Also, there is a mistake in the app code: When calling the function websockets.connect(), pass a keyword argument ssl=ssl_context so the CA bundle specified earlier is actually used. The correct code looks like this:

import json
import asyncio
import websockets
import ssl
import certifi


ssl_context = ssl.create_default_context()
ssl_context.load_verify_locations(certifi.where())


query =  {
    "jsonrpc": "2.0",
    "method": "queryHeadsets",
    "params": {},
    "id": 1
    }
json = json.dumps(query)

async def query(json):
    
    async with websockets.connect("wss://emotivcortex.com:54321", ssl=ssl_context) as ws:
        await ws.send(json)
        response = await ws.recv()
        print(response)

asyncio.get_event_loop().run_until_complete(query(json))

Many thanks to larsks and Steffen Ullrich for pointing me in the right direction.

Question 5

Passing the linker version to Clang via -mlinker-version=305 resolved the issue.

For CMake this can be done like so:

-DCMAKE_CXX_FLAGS="-mlinker-version=305"

Can't help but feel this is a bug.

The linker verison can be obtained via ld -v on macOS 10.12 where the problem occurs.

A handy bash function to get the ld version for passing to Clang:

#!/bin/bash

function get_ld_version() {
  local info=$( ld -v 2>&1 > /dev/null )

  echo "${info}" | perl -wne '/.ld64-(.*?)[^0-9]/ and print "$1n"'
}

Question 6

As far as I know, the content for your stickies notes is stored in $HOME/Library/StickiesDatabase. If you are using Time Machine, or some other mechanism to do back-ups for your data that include this database, then you can try restoring an older version of this database to recover your data. Because of the way the Stickies app maintains its data (in a database), without access to some previous version of the database file, there's really no way to reliably recover Stickies notes.

pip fails with SSLError on MacOS because of outdated certificate - how to recover?

Answers

The problem is caused by a missing intermediate CA certificate.

Solution

python,ssl,pip,ssl-certificate