Interpreting Python Interpreters with Bash
Use bash to figure out your Python and Pip version bugs
I was working on my side project yesterday and came across a python interpreter bug that took all day. I thought it was a great example of moving beyond Python scripting and trying to figure out what’s going wrong behind the scenes with a terminal.
Scenario
I am trying to download a file from a public S3 Bucket, or online storage. I have already installed the required libraries, but my code is still giving me this error!
Error:
Failed to import the required Python library (botocore or boto3) on e25745c093c5's Python /usr/bin/python2. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter
PLAY RECAP *********************************************************************
localhost : ok=20 changed=16 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Where do I start?
This failure happened while I was building a Docker image [link for Docker explanation]. I built a container based on the partially built/failed image with the following bash command:
$ docker run -it de-devtools:latest bash
And that takes us into a Docker container where we should be able to poke around and see why the installed boto libraries aren’t being used by my S3 download code!
Using `which` to debug
My first commands were to figure out which `pip
` and `python
` versions the image was using.
root@b069bb76b513:/# which pip
/usr/bin/pip
root@b069bb76b513:/# which python
/usr/bin/python
root@b069bb76b513:/# which python3
/usr/bin/python3
Let’s breakdown what happened here.
`
root
` is the user that I’m using bash as. I’m `@
`, or at, this Docker container called `b069bb76b513
`.I ran the `which` command to see which pip and python versions our image is defaulting to.
This still doesn’t answer my question as to why Python2 isn’t able to see my boto installs!
However, I noticed that all of these outputs pointed to this `
/usr/bin/
` directory or location in my container.
Also, the original error message says that `/usr/bin/python2
` is the Python version that’s failing to use the boto3 module.
So I think `/usr/bin/
` might be worth checking out.
Bash Aliases for Productivity
I have a built in bash alias which reads in `ll
` as `ls -l
`. This essentially means; print out the contents of the given directory, as a vertical list. I prefer it to using `ls
` alone because I can then see much more information like file permissions, last updated at, etc.
More bash commands: grep
I also don’t want to print out the whole contents of `/usr/bin/
`, just the python items. So I used the following command, to filter my results in bash:
root@b069bb76b513:/# ll /usr/bin | grep 'python2'
<truncated results>
9 Apr 16 2018 python2 -> python2.7*
16 Apr 16 2018 python2-config -> python2.7-config*
Nov 7 10:07 python2.7*
I won’t show the whole contents but we can talk about what happened here. I ran `ll /usr/bin
` to print out all of the contents of `/usr/bin
`, but then I pipe the results with `|
` into a new bash command.
This new bash command, `grep
`, is a text search for the given keyword, `python2
`.
Let’s Dig Into Python 2
Everything looks good in our `/usr/bin
` file location, with Python 2 pointing at a Python 2.7 executable. Python versions and pip executables are so confusing.
Let’s do some coding in Python to see if we can’t recreate this error.
root@b069bb76b513:/# python2
Python 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named boto3
So according to Python2, aka Python 2.7.17, boto3 has not been installed! I’ve already installed with both pip and pip3, so what’s going on?
What happens if I try to run the python2 version of pip?
root@b069bb76b513:/# python2-pip
bash: python2-pip: command not found
That’s not correct.
Which pip am I using again?
root@b069bb76b513:/# which pip
/usr/bin/pip
Huh. That’s not necessarily python2. Let’s look in our `/usr/bin/
` location again, but this time, let’s look at what pip executables we can work with.
root@b069bb76b513:/# ll /usr/bin | grep 'pip'
-rwxr-xr-x 1 root root 292 Apr 3 2019 pip*
-rwxr-xr-x 1 root root 292 Apr 3 2019 pip2*
-rwxr-xr-x 1 root root 293 Apr 3 2019 pip3*
I see a `pip2*
` executable that I haven’t been using.
So I attempt to download some boto software with pip2!
root@b069bb76b513:/# pip2 install boto3
Collecting boto3
...
Successfully installed boto3-1.12.39
And now to test whether Python 2 recognizes this install by running a Python 2 shell.
root@b069bb76b513:/# python2
Python 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>
No failures, no “module not found”, we are good to go!
Solution
The solution was to create another pip install step, but with pip2 as the executable so that Python 2 would have this software available, in my Docker image.
- name: install pip2 dependencies
become: yes
pip:
name: ['boto3', 'botocore']
executable: /usr/bin/pip2
Another part of the solution was to also install `botocore
`, not just `boto3
`, but that shows the importance of reading error messages carefully. You can see the added code here!
Conclusion
Once I figured this out, I got a totally new error but that’s how life goes.
I hope this post walking through different Python Language and Installation versions has clarified where Python interpreters and Pip Executables typically live. If you know where they live, what they point to, you can test them and recreate installation errors!
I hope this post exemplifies the kind of work that you can expect as you dive deeper on the engineering side of data. Thanks for reading!
Data Engineering Content
Docker Explained
About the Author and Newsletter
I automate data processes. I work mostly in Python, SQL, and bash. For more about my freelance work, check out ocelotdata.com.
At Modern Data Infrastructure, I democratize the knowledge it takes to understand an open-source, transparent, and reproducible Data Infrastructure. In my spare time, I collect Lenca pottery, walk my dog, and listen to music.
More at; What is Modern Data Infrastructure.