Helpers

Virtual environment

needs rewriting [ https://dev.to/codemouse92/dead-simple-python-virtual-environments-and-pip-5b56 venv] dev.to Part of "Dead Simple Python" series.

A virtual environment, or virtualenv as it is sometimes called, is a sandbox where you can install only the Python packages you need.

python3 -V
sudo apt-get install build-essential libssl-dev libffi-dev python-dev
sudo apt install virtualenv python3-virtualenv python3-pip
virtualenv -p python3 venv            #create
virtualenv -p /usr/bin/python3.6 venv #create venv pointing to use specific executable, could be eg. py2.7
python3 -m venv venv                  #Python3.3+ has a comamnd to create virtual environemnts, 1st venv is the command, 2nd "venv" name
source ./venv/bin/activate            #activate
(venv) ubuntu@u18gui-1:~/py$          #prompt changed
deactivate                            #deactivate

#Running venv without activating, work the same as if you had activated the virtual environment
venv/bin/python
venv/bin/pip install pylint

Note: Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.

Git

Within a virtual environment's directory are the actual packages you installed with pip, therefore working with any VCS (Version Control System) you should ignore whole venv/ directory by adding it to eg. .gitignore.

Resources

virtualenv documentation

pip - python package manager

Do not use pip it self to upgrade pip as it will break your installation. Use OS package maanger instead. Using --user option to install packages it's a good habit as well as it will use the user scheme to install the package.

sudo apt-get install python-pip  #pip for pyhton2, eg. check apt-cache show python-pip, python dependency version
sudo apt-get install python3-pip #pip for python3

pip list            #list packages, no need to use eg. pip3 within venv
pip install PySide2 #insall a package
pip install PySide2==5.11.1 #or ">=" at least this version, or greater
pip install --upgrade PySide2
pip uninstall PySide2

pip install -r requirements.txt
pip install --upgrade -r requirements.txt

Search on PyPI.org

pip search web scraping

haSH-BANG or #!

Rules for python vs python2 vs python3 are described in pep-0394

#!/usr/bin/env python3   #correct way
#!/usr/bin/python        #forces use a system-wide copy of Python, won't respect "venv"

!/usr/bin/env python3

Syntax, semantics and functions

Cheatsheet Python2 and Python3

print()

Print in v3 is a function, it uses arguments with variables like below:

%s - String (or any object with a string representation, like numbers)
%d - Integers
%f - Floating point numbers
%.<number of digits>f - Floating point numbers with a fixed amount of digits to the right of the dot.
%x/%X - Integers in hex representation (lowercase/uppercase)

Eaxmple:

print("%s is %d years old." % (name, age))

Simple HTTP Server

It provides the local files browser over http protocol and access logs

sudo python  -m SimpleHTTPServer 80 #python 2
sudo python3 -m http.server      80 #python 3

Web scraping

Install Selenium lib on Ubuntu

$ which python python3
/usr/bin/python
/usr/bin/python3
sudo apt-get install python3-pip        # install pip for python3
sudo python3 -m pip install -U selenium # install selenium for python3
sudo python3 -m pip install -U pip      # upgrade pip

Install geckodriver on Ubuntu

It's required if you use Selenium in Python, webdriver function:

default_browser = webdriver.Firefox()

Install

## Geckodriver (verified on Ubuntu 16.04 running on Vagrant)
wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar.gz
sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.21.0-linux64.tar.gz -O > /usr/bin/geckodriver'
sudo chmod +x /usr/bin/geckodriver
rm geckodriver-v0.21.0-linux64.tar.gz

## Chromedriver
wget https://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo chmod +x chromedriver
sudo mv chromedriver /usr/bin/
rm chromedriver_linux64.zip

Selenium - Website automation test to mimic human interaction - page login

Script below demonstrate usage of Selenium automation through Mozilla Geckodriver. It will open Firefox, navigate to a give URL, fill in a form, click a login button and take a screenshot.

Credentials file

cat creds.yaml
wpUser: user
wpPass: pass123

Main script

#!/usr/bin/python3
from selenium import webdriver
import yaml
driver = webdriver.Firefox()
creds = yaml.safe_load(open('creds.yaml','r'))
driver.get('http://wiki.ciscolinux.co.uk/index.php?title=Special:UserLogin&returnto=Main+Page')

user_field = driver.find_element_by_name("wpName")
pass_field = driver.find_element_by_name("wpPassword")
user_field.send_keys(creds['wpUser'])
pass_field.send_keys(creds['wpPass'])
# find login button and click on it
driver.find_element_by_name('wploginattempt').click()
driver.save_screenshot('screenshot.png')
driver.close

Python

Contents