Python

From Ever changing code
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Helpers

Python3 - Virtual environments

needs rewriting [ https://dev.to/codemouse92/dead-simple-python-virtual-environments-and-pip-5b56 venv] dev.to Part of "Dead Simple Python" series.

A virtual environment, or virtualenv as it's sometimes called, is a sandbox where you can install only the Python packages you need. 

python3 -V # -> eg. Python 3.6.8

# Install virtualenv python3 packages
sudo apt install build-essential libssl-dev libffi-dev python-dev
sudo apt install virtualenv python3-virtualenv python3-pip
sudo apt install python3-venv # Debian, Ubuntu systems, tested with U20.04

# Python3.3+ use native command to create virtual environments
python3 -m venv venv          # 1st venv is the command, 2nd "venv" is a project name

# (Depricated) legacy method to create virtualenv 
virtualenv -p python3 venv            # create
virtualenv -p /usr/bin/python3.6 venv # create venv pointing to use specific executable, could be eg. py2.7

$ source ./venv/bin/activate    # activate
(venv) ubuntu@u18gui-1:~/py$    # <- prompt updated with env name
$ deactivate                    # deactivate
vagrant@ubuntu-bionic ~  $      # <- prompt cleaned

# Running venv without activating, work the same as if you had activated the virtual environment
venv/bin/python
venv/bin/pip install pylint


Note: Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.

Git

Within a virtual environment's directory are the actual packages you installed with pip, therefore working with any VCS (Version Control System) you should ignore whole venv/ directory by adding it to eg. .gitignore.

Resources

pip - python package manager

Do not use pip it self to upgrade pip as it will break your installation. Use OS package maanger instead. Using --user option to install packages it's a good habit as well as it will use the user scheme to install the package. 

sudo apt-get install python-pip  #pip for pyhton2, eg. check apt-cache show python-pip, python dependency version
sudo apt-get install python3-pip #pip for python3

pip list            #list packages, no need to use eg. pip3 within venv
pip install PySide2 #insall a package
pip install PySide2==5.11.1 #or ">=" at least this version, or greater
pip install --upgrade PySide2
pip uninstall PySide2

pip install -r requirements.txt
pip install --upgrade -r requirements.txt


Search on PyPI.org 

pip search web scraping

haSH-BANG or #!

Rules for python vs python2 vs python3 are described in pep-0394 

#!/usr/bin/env python3   # correct way
#!/usr/bin/python        # forces use a system-wide copy of Python, won't respect "venv"

Syntax, semantics and functions

Cheatsheet Python2 and Python3

print()

Print in v3 is a function, it uses arguments with variables like below: 

%s # String (or any object with a string representation, like numbers)
%d # Integers
%f # Floating point numbers
%.<number-of-digits>f # Floating point numbers with a fixed amount of digits to the right of the dot.
%x/%X # Integers in hex representation (lowercase/uppercase)

# Example
print("%s is %d years old." % (name, age))

Simple HTTP Server

It provides the local files browser over http protocol and access logs 

sudo python  -m SimpleHTTPServer 80 #python 2
sudo python3 -m http.server      80 #python 3

Web scraping

Install Selenium lib on Ubuntu

$ which python python3
/usr/bin/python
/usr/bin/python3
sudo apt-get install python3-pip        # install pip for python3
sudo python3 -m pip install -U selenium # install selenium for python3
sudo python3 -m pip install -U pip      # upgrade pip

Install geckodriver on Ubuntu

It's required if you use Selenium in Python, webdriver function: 

default_browser = webdriver.Firefox()
Firefox geckodriver (verified on Ubuntu 16.04, 18.04 running on Vagrant)
wget https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz
sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.26.0-linux64.tar.gz -O > /usr/bin/geckodriver'
sudo chmod +x /usr/bin/geckodriver
rm geckodriver-v0.26.0-linux64.tar.gz


Chromedriver
wget https://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo chmod +x chromedriver
sudo mv chromedriver /usr/bin/
rm chromedriver_linux64.zip

Selenium - Website automation test to mimic human interaction - page login

Script below demonstrate usage of Selenium automation through Mozilla Geckodriver. It will open Firefox, navigate to a give URL, fill in a form, click a login button and take a screenshot.


Credentials file 

cat creds.yaml
wpUser: user
wpPass: pass123


Main script 

#!/usr/bin/python3
from selenium import webdriver
import yaml
driver = webdriver.Firefox()
creds = yaml.safe_load(open('creds.yaml','r'))
driver.get('http://wiki.ciscolinux.co.uk/index.php?title=Special:UserLogin&returnto=Main+Page')

user_field = driver.find_element_by_name("wpName")
pass_field = driver.find_element_by_name("wpPassword")
user_field.send_keys(creds['wpUser'])
pass_field.send_keys(creds['wpPass'])
# find login button and click on it
driver.find_element_by_name('wploginattempt').click()
driver.save_screenshot('screenshot.png')
driver.close


Now you can preview the screenshot from the terminal window.

Selenium Webdriver running in a docker


Install Selenium and run from a terminal 

sudo apt install firefox python3-pip xvfb x11-utils --yes
sudo -H pip3 install bpython selenium # better option is to install in virtual environment

export DISPLAY=:2    # fake X display
Xvfb $DISPLAY -ac &  # X virtual framebuffer

export GECKO_DRIVER_VERSION='v0.26.0'
wget https://github.com/mozilla/geckodriver/releases/download/$GECKO_DRIVER_VERSION/geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
tar -xvzf geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
rm        geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
chmod +x  geckodriver
sudo cp   geckodriver /usr/local/bin/

cat <<EOF > script.py
#!/usr/bin/env python3
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile

ff_options = FirefoxOptions()
ff_options.headless = True

ff = Firefox(options=ff_options)
ff.quit()
EOF
chmod +x script.py


Official Selenium docker images can get a lot of parameters:

  • basic run uses Xvfb X virtual framebuffer and shared memory volume -v /dev/shm:/dev/shm
  • it's possible to run in complete headless mode without Xvfb
  • can be run as a server then Python code pointing to the remote server where webdriver runs on

Serverless - AWS Lambda

The Right Way™ to do Serverless in Python

References

Learning

References

Retrieved from ‘http://wiki.ciscolinux.co.uk/index.php?title=Python&oldid=5705’