Python

From Ever changing code
Revision as of 19:35, 29 July 2020 by Pio2pio (talk | contribs) (→‎Python3 - Virtual environments)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Helpers

Python3 - Virtual environments

needs rewriting [ https://dev.to/codemouse92/dead-simple-python-virtual-environments-and-pip-5b56 venv] dev.to Part of "Dead Simple Python" series.

A virtual environment, or virtualenv as it's sometimes called, is a sandbox where you can install only the Python packages you need. 

python3 -V # -> eg. Python 3.6.8

# Install virtualenv python3 packages
sudo apt install build-essential libssl-dev libffi-dev python-dev
sudo apt install virtualenv python3-virtualenv python3-pip
sudo apt install python3-venv # Debian, Ubuntu systems, tested with U20.04

# Python3.3+ use native command to create virtual environments
python3 -m venv venv          # 1st venv is the command, 2nd "venv" is a project name

# (Depricated) legacy method to create virtualenv 
virtualenv -p python3 venv            # create
virtualenv -p /usr/bin/python3.6 venv # create venv pointing to use specific executable, could be eg. py2.7

$ source ./venv/bin/activate    # activate
(venv) ubuntu@u18gui-1:~/py$    # <- prompt updated with env name
$ deactivate                    # deactivate
vagrant@ubuntu-bionic ~  $      # <- prompt cleaned

# Running venv without activating, work the same as if you had activated the virtual environment
venv/bin/python
venv/bin/pip install pylint


Note: Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.

Git

Within a virtual environment's directory are the actual packages you installed with pip, therefore working with any VCS (Version Control System) you should ignore whole venv/ directory by adding it to eg. .gitignore.

Resources

pip - python package manager

Do not use pip it self to upgrade pip as it will break your installation. Use OS package maanger instead. Using --user option to install packages it's a good habit as well as it will use the user scheme to install the package. 

sudo apt-get install python-pip  #pip for pyhton2, eg. check apt-cache show python-pip, python dependency version
sudo apt-get install python3-pip #pip for python3

pip list            #list packages, no need to use eg. pip3 within venv
pip install PySide2 #insall a package
pip install PySide2==5.11.1 #or ">=" at least this version, or greater
pip install --upgrade PySide2
pip uninstall PySide2

pip install -r requirements.txt
pip install --upgrade -r requirements.txt


Search on PyPI.org 

pip search web scraping

haSH-BANG or #!

Rules for python vs python2 vs python3 are described in pep-0394 

#!/usr/bin/env python3   # correct way
#!/usr/bin/python        # forces use a system-wide copy of Python, won't respect "venv"

Syntax, semantics and functions

Cheatsheet Python2 and Python3

print()

Print in v3 is a function, it uses arguments with variables like below: 

%s # String (or any object with a string representation, like numbers)
%d # Integers
%f # Floating point numbers
%.<number-of-digits>f # Floating point numbers with a fixed amount of digits to the right of the dot.
%x/%X # Integers in hex representation (lowercase/uppercase)

# Example
print("%s is %d years old." % (name, age))

Simple HTTP Server

It provides the local files browser over http protocol and access logs 

sudo python  -m SimpleHTTPServer 80 #python 2
sudo python3 -m http.server      80 #python 3

Web scraping

Install Selenium lib on Ubuntu

$ which python python3
/usr/bin/python
/usr/bin/python3
sudo apt-get install python3-pip        # install pip for python3
sudo python3 -m pip install -U selenium # install selenium for python3
sudo python3 -m pip install -U pip      # upgrade pip

Install geckodriver on Ubuntu

It's required if you use Selenium in Python, webdriver function: 

default_browser = webdriver.Firefox()
Firefox geckodriver (verified on Ubuntu 16.04, 18.04 running on Vagrant)
wget https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz
sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.26.0-linux64.tar.gz -O > /usr/bin/geckodriver'
sudo chmod +x /usr/bin/geckodriver
rm geckodriver-v0.26.0-linux64.tar.gz


Chromedriver
wget https://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo chmod +x chromedriver
sudo mv chromedriver /usr/bin/
rm chromedriver_linux64.zip

Selenium - Website automation test to mimic human interaction - page login

Script below demonstrate usage of Selenium automation through Mozilla Geckodriver. It will open Firefox, navigate to a give URL, fill in a form, click a login button and take a screenshot.


Credentials file 

cat creds.yaml
wpUser: user
wpPass: pass123


Main script 

#!/usr/bin/python3
from selenium import webdriver
import yaml
driver = webdriver.Firefox()
creds = yaml.safe_load(open('creds.yaml','r'))
driver.get('http://wiki.ciscolinux.co.uk/index.php?title=Special:UserLogin&returnto=Main+Page')

user_field = driver.find_element_by_name("wpName")
pass_field = driver.find_element_by_name("wpPassword")
user_field.send_keys(creds['wpUser'])
pass_field.send_keys(creds['wpPass'])
# find login button and click on it
driver.find_element_by_name('wploginattempt').click()
driver.save_screenshot('screenshot.png')
driver.close


Now you can preview the screenshot from the terminal window.

Selenium Webdriver running in a docker


Install Selenium and run from a terminal 

sudo apt install firefox python3-pip xvfb x11-utils --yes
sudo -H pip3 install bpython selenium # better option is to install in virtual environment

export DISPLAY=:2    # fake X display
Xvfb $DISPLAY -ac &  # X virtual framebuffer

export GECKO_DRIVER_VERSION='v0.26.0'
wget https://github.com/mozilla/geckodriver/releases/download/$GECKO_DRIVER_VERSION/geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
tar -xvzf geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
rm        geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
chmod +x  geckodriver
sudo cp   geckodriver /usr/local/bin/

cat <<EOF > script.py
#!/usr/bin/env python3
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile

ff_options = FirefoxOptions()
ff_options.headless = True

ff = Firefox(options=ff_options)
ff.quit()
EOF
chmod +x script.py


Official Selenium docker images can get a lot of parameters:

  • basic run uses Xvfb X virtual framebuffer and shared memory volume -v /dev/shm:/dev/shm
  • it's possible to run in complete headless mode without Xvfb
  • can be run as a server then Python code pointing to the remote server where webdriver runs on

Serverless - AWS Lambda

The Right Way™ to do Serverless in Python

References

Learning

References

Retrieved from ‘http://wiki.ciscolinux.co.uk/index.php?title=Python&oldid=5705’