Cloud Python Development

Standing up a cloud development practice

We have been covering a lot of database topics in recent posts and have not been spending as much time with a flexible and extendable language like Python. In this one, we’ll cover some fundamentals in setting up a Python cloud development environment.

---

Cloud Server setup

First you want to set up a cloud server, or virtual machine, in your browser.

I have the most experience in AWS’s Elastic Cloud Compute (EC2) servers. However, GCP’s Virtual Machines (VMs) are also excellent tools.

Cloud Server Shell Access

What is a shell? Computer vocabulary can often be confusing with overlapping definitions of similar but different software. For example; shell, terminal, console, and command line.

A shell is a computer program which exposes an operating system's services to a human user or other programs - Wikipedia

With a shell, your cloud server is exposed to you for development. As a Python developer, a shell is the most straightforward way to access your cloud server.

Both of the guides above have instructions on how to connect to your newly created server. Personally, I prefer using ssh with a .pem file, as this method will remain consistent across all major Operating Systems (OSs) like Windows and Linux/Unix.

---

Shell Navigation

Once you have a brand new server to work with, the next question is, How can I create, manipulate, and delete files and directories (folders) without a Graphical User Interface (GUI), or click, drag, and drop screen?

Before getting into too much shell scripting detail, I personally like to emphasize the importance of memorizing these two most basic shell commands.

You should memorize:

• cd

• ls

cd stands for “change directory”, in which you are changing your current working folder. And ls stands for “list” in which you list out the contents of your current working directory. If you are generous with your use of these two commands, you are sure to keep mental track of your Linux file system (structure and contents).

Another Intro to Terminal guide

Sam Bail is a great person and Data manager. Her approach to teaching the terminal here is different and totally valid. It is a great supplement to my brief overview.

Other intro Linux built-in commands

This is a list of 34 basic commands, which is not wieldy in the middle of development. Therefore, here is a pared-down list of often recurring commands from my Data Engineering career.

My own vernacular comments start after each pound sign.

• pwd # print where you’re at

• cat # print out file contents

• grep # search directories with a keyword

• cp # copy and paste from source to destination

• mv # move (cut and paste) from source to destination

• mkdir # make a directory (folder)

• rm # remove a file

• chmod # change permissions of a file or directory

---

Python Virtual Environments

With so many virtual environment solutions; venv, pyenv, pyvenv, virtualenv, etc. It can be confusing on where to start.

This is my personal recommendation for beginners: start by learning virtualenv and pip, tools which work with both Python 2 and 3 and in a variety of situations, and pick up other tools once you start needing them. - Stack Overflow

The pip package installer is easy enough to understand for any budding Data Engineer with at least a few months of experience. Adding on virtualenv to your learning list is an achievable feat.

Virtual environments for Python seemed redundant to me when I first started out but they are useful in isolating software dependencies per each project on a server. Sometimes projects will need specific versions of libraries and these may conflict if all of your projects are building off of a single list of software dependencies. That’s just one example of a development roadblock you may run into, by not separating out your virtual environments per project.

---

Terminal Text Editors

Once you have an introductory grasp on

• shell scripting

  • navigating the Linux file system

  • manipulating files and directories

• suitable Python virtual environments

You need to be able to edit the contents of your Python files. The most popular terminal based editors are Vim, Emacs, Nano, and ne.

Personally, I use Vim and found this book to be helpful.

Since you’ll be going without a mouse to edit your files, it’s best to try a little bit everyday, rather than jumping in with too much enthusiasm. Doing a little bit everyday works against the discouragement that comes with slower development speed on Vim, contrasted with Pycharm. However, it pays off over time.

Using a local Integrated Development Environment (IDE) like Pycharm or Microsoft VS Code can be very handy. They are always improving their cloud connectivity offerings as well. However, terminal editors are useful because you can edit files across any piece of cloud technology, provided you have command line access. If you would like to develop within paths of least resistance to Command Line Interfaces (CLIs), a terminal text editor is currently your best option.

---

Finale

These topics aren’t particularly complicated but they are a hassle to get a grasp on. With overlapping vocabulary, outdated walkthroughs, and an understandably jarring removal from the mouse and keyboard paradigm.

I’ll leave this post with a final note; that every command, subcommand, and flag (option) string you type into any command line will have meanings. Slowing down to look up the meaning of each command, subcommand, and flag will improve your understanding of these new topics. That remains true for each topic; cloud servers, Linux file system, Python environments, and terminal editors.

Best of luck! I will follow this piece up with a second part to cover some auxiliary, yet important topics.