Error installing psycopg2 in a pypy virtualenv environment

Question

I installed the shape sequinte pypy:

wget https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.10.1-linux64.tar.bz2

tar xf pypy3-v5.10.1-linux64.tar.bz2

virtualenv -p ~/pypy3-v5.10.1-linux64/bin/pypy my-pypy-env

I activated the environment

source ~/.virtualenv/my-pypy-env/bin/activate

I installed some packages:

pip install mongo

pip install numpy

But when will I install psycopg2:

pip install psycopg2

Collecting psycopg2
  Using cached psycopg2-2.7.4.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-
info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-
info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

Error: pg_config executable not found.

pg_config is required to build psycopg2 from source.  Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:

    python setup.py build_ext --pg-config /path/to/pg_config build ...

or with the pg_config option in 'setup.cfg'.

If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.

For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).


----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-7lSMte/psycopg2/

I tried to install with sudo:

sudo pip install psycopg2

But it made another mistake:

The directory '/home/prisvo/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/prisvo/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Okay, so I should just install using -H in front of sudo.

sudo -H pip install psycopg2:

Requirement already satisfied: psycopg2 in /usr/local/lib/python2.7/dist-packages

Looks like it worked, but when testing:

python prisvo-recommender.py

  File "prisvo-recomender.py", line 1, in <module>
    import banco
  File "/home/prisvo/prisvo-recommendation/corepypy/banco.py", line 1, in <module>
    import psycopg2
ImportError: No module named psycopg2

I don’t know what I’m doing wrong or if psycopg2 doesn’t run with pypy. Could someone give me a hint?

Answer 1

A search for the packages reveals that psycopg2 is not compatible with pypy (this will happen with several Python packages that have native code together, that is, using the cPython-specific API). This information is not in a single authoritative place - I found it scattered as a response in 2 or 3 places (including the Pypy site cites that psycopg2 would be an exception to the rule of compiled packages and that it would work with pypy. Well, I tried to use it here with Pypy3 and it wasn’t)

However, they have created the psycopg2cffi package, which uses another technology (CFFI) to perform the native calls to the postres libraries. This one was installed - but not before I had to make a maneuver here.

I saw that you downloaded pypy3 already compiled independent of the distribution - from source code. You will probably also need to download his source code - https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.10.1-src.tar.bz2 to have the arches . h, if you don’t have a "include" folder there. For anyone using pypy3 from the Linux distribution, as in my case, I first had to install pypy3 development package - here is a folder, so sudo dnf install pypy3-devel - I suppose that in Debian and Ubuntu the command is apt-get install pypy3-dev (but there may be variations in the package name)

Even with that, pip install psycopg2cffi still gave error - reading the messages I realized that it had not located the include file pythonconfig.h python. On my system, this file has been placed in the folder /usr/lib64/pypy3-5.5.0/include. (On other systems it will end up in a similar folder - maybe just change the version number. Since you downloaded py3 independently from Linux, you may need to download its source code as well to have the files .h). So I created an environment variable to tell the C compiler to include this include folder. Just type in the shell:

export C_INCLUDE_PATH=/usr/lib64/pypy3-5.5.0/include

There, with that the command pip install psycopg2cffi worked. If you plan to use psycopg2 directly, that’s enough - in your Python code you can try something like this:

try:
    import psycopg2
except ImportError:
    import psycopg2cffi as psycopg2

(if you’re worried about code that works in both environments, of course - otherwise just put the second import directly).

Now, if you are going to use sqlalchemy, or some other framework that imports psycopg2 without going through your code, there is one more step: the line containing the import psycopg2 is not in code under your control, and will fail when it is executed. In the case of Sqlalchemy the error occurs when trying to create an engine. The official way of not happening is to put into the connection URL what psycopg2cffi is being used for - the URL takes the form:

postgresql+psycopg2cffi://user:password@host:port/dbname

(instead of just postgresql://). Alternatively, you can simply cheat the framework (if it’s not sqlalchemy, or if you want to keep the url unchanged). To do this, after importing psycopg2cffi, we create a "psycopg2" alias for it within sys.modules - that way, when Python finds a import psycopg2 will assume that the module is already imported, and use the ...cffi instead:

import psycopg2cffi
import sys
sys.modules["psycopg2"] = sys.modules["psycopg2cffi"]

(in time never do sudo pip install... - will not solve anything if you are in a virtualenv, and can break your system if you are not - always install Python packages with the system package manager, (apt-get on Ubuntu). Inside a virtualenv you use Pip)