Difference between revisions of "Python"

From Ever changing code
Jump to navigation Jump to search
 
(29 intermediate revisions by the same user not shown)
Line 1: Line 1:
Learning Python scratch pad
= Helpers =
= Helpers =
== Virtual environment ==
== Python3 - Virtual environments ==
;needs rewriting  [ https://dev.to/codemouse92/dead-simple-python-virtual-environments-and-pip-5b56 venv] dev.to Part of "Dead Simple Python" series.
;needs rewriting  [ https://dev.to/codemouse92/dead-simple-python-virtual-environments-and-pip-5b56 venv] dev.to Part of "Dead Simple Python" series.
The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories.
 
;Install [https://docs.python.org/3/library/venv.html venv], requires python3.3+
A virtual environment, or virtualenv as it's sometimes called, is a sandbox where you can install only the Python packages you need.
<source lang="bash">
<source lang="bash">
python3 -V
python3 -V # -> eg. Python 3.6.8
sudo apt-get install build-essential libssl-dev libffi-dev python-dev
 
sudo apt-get install -y python3-pip
# Install virtualenv python3 packages
sudo apt install python3-venv
sudo apt install build-essential libssl-dev libffi-dev python-dev
python3 -m venv my_env            #create virtual environment
sudo apt install virtualenv python3-virtualenv python3-pip
source my_env/bin/activate         #activate env
sudo apt install python3-venv # Debian, Ubuntu systems, tested with U20.04
(my_env) piotr@ubuntu ~/scripts $  #prompt change to acknowledge activated environment
 
# Python3.3+ use native command to create virtual environments
python3 -m venv venv          # 1st venv is the command, 2nd "venv" is a project name
 
# (Depricated) legacy method to create virtualenv
virtualenv -p python3 venv            # create
virtualenv -p /usr/bin/python3.6 venv # create venv pointing to use specific executable, could be eg. py2.7
 
$ source ./venv/bin/activate   # activate
(venv) ubuntu@u18gui-1:~/py$    # <- prompt updated with env name
$ deactivate                    # deactivate
vagrant@ubuntu-bionic ~ $      # <- prompt cleaned
 
# Running venv without activating, work the same as if you had activated the virtual environment
venv/bin/python
venv/bin/pip install pylint
</source>
</source>
'''Note:''' Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.
'''Note:''' Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.
;Git
Within a virtual environment's directory are the actual packages you installed with pip, therefore working with any VCS (Version Control System) you should ignore whole <code>venv/</code> directory by adding it to eg. <code>.gitignore</code>.
=== Resources ===
*[https://virtualenv.pypa.io/en/stable/ virtualenv documentation]
== pip - python package manager ==
Do not use <code>pip</code> it self to upgrade pip as it will break your installation. Use OS package maanger instead. Using <code>--user</code> option to install packages it's a good habit as well as it will use the user scheme to install the package.
<source lang=bash>
sudo apt-get install python-pip  #pip for pyhton2, eg. check apt-cache show python-pip, python dependency version
sudo apt-get install python3-pip #pip for python3
pip list            #list packages, no need to use eg. pip3 within venv
pip install PySide2 #insall a package
pip install PySide2==5.11.1 #or ">=" at least this version, or greater
pip install --upgrade PySide2
pip uninstall PySide2
pip install -r requirements.txt
pip install --upgrade -r requirements.txt
</source>
Search on [http://PyPI.org PyPI.org]
<source lang=bash>
pip search web scraping
</source>
== haSH-BANG or #! ==
Rules for python vs python2 vs python3 are described in [https://www.python.org/dev/peps/pep-0394/ pep-0394]
<source lang=bash>
#!/usr/bin/env python3  # correct way
#!/usr/bin/python        # forces use a system-wide copy of Python, won't respect "venv"
</source>


= Syntax, semantics and functions =
= Syntax, semantics and functions =
== [https://python-future.org/compatible_idioms.html Cheatsheet Python2 and Python3] ==
== print() ==
== print() ==
Print in v3 is a function, it uses arguments with variables like below:
Print in v3 is a function, it uses arguments with variables like below:
%s - String (or any object with a string representation, like numbers)
<source lang=python>
%d - Integers
%s # String (or any object with a string representation, like numbers)
%f - Floating point numbers
%d # Integers
%.<number of digits>f - Floating point numbers with a fixed amount of digits to the right of the dot.
%f # Floating point numbers
%x/%X - Integers in hex representation (lowercase/uppercase)
%.<number-of-digits>f # Floating point numbers with a fixed amount of digits to the right of the dot.
Eaxmple:
%x/%X # Integers in hex representation (lowercase/uppercase)
print("%s is %d years old." % (name, age))
 
# Example
print("%s is %d years old." % (name, age))
</source>


= Simple HTTP Server =
= Simple HTTP Server =
Line 34: Line 88:
sudo python3 -m http.server      80 #python 3
sudo python3 -m http.server      80 #python 3
</source>
</source>
= [https://dev.to/awwsmm/web-scraping-walkthrough-with-python-85c Web scraping] =


= Install Selenium lib on Ubuntu =
= Install Selenium lib on Ubuntu =
Line 51: Line 107:
</source>
</source>


Install
;[https://github.com/mozilla/geckodriver/ Firefox geckodriver] (verified on Ubuntu 16.04, 18.04 running on Vagrant)
<source lang="bash">
<source lang="bash">
## Geckodriver (verified on Ubuntu 16.04 running on Vagrant)
wget https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz
wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar.gz
sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.26.0-linux64.tar.gz -O > /usr/bin/geckodriver'
sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.21.0-linux64.tar.gz -O > /usr/bin/geckodriver'
sudo chmod +x /usr/bin/geckodriver
sudo chmod +x /usr/bin/geckodriver
rm geckodriver-v0.21.0-linux64.tar.gz
rm geckodriver-v0.26.0-linux64.tar.gz
</source>
 


## Chromedriver
;Chromedriver
<source lang="bash">
wget https://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip
wget https://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
unzip chromedriver_linux64.zip
Line 97: Line 155:
driver.close
driver.close
</source>
</source>
Now you can [[Linux_terminal_image_preview|preview the screenshot]] from the terminal window.
= Selenium Webdriver running in a docker =
*[https://github.com/SeleniumHQ/docker-selenium Selenium docker-hub image]
*[https://www.blazemeter.com/blog/how-to-run-selenium-tests-in-docker/ how-to-run-selenium-tests-in-docker]
[https://gist.github.com/pcgeek86/a1fd9d26f8ad46b51adf9513f67b95f2 Install Selenium and run from a terminal]
<source lang=bash>
sudo apt install firefox python3-pip xvfb x11-utils --yes
sudo -H pip3 install bpython selenium # better option is to install in virtual environment
export DISPLAY=:2    # fake X display
Xvfb $DISPLAY -ac &  # X virtual framebuffer
export GECKO_DRIVER_VERSION='v0.26.0'
wget https://github.com/mozilla/geckodriver/releases/download/$GECKO_DRIVER_VERSION/geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
tar -xvzf geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
rm        geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
chmod +x  geckodriver
sudo cp  geckodriver /usr/local/bin/
cat <<EOF > script.py
#!/usr/bin/env python3
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile
ff_options = FirefoxOptions()
ff_options.headless = True
ff = Firefox(options=ff_options)
ff.quit()
EOF
chmod +x script.py
</source>
Official Selenium docker images can get a lot of parameters:
* basic run uses [https://en.wikipedia.org/wiki/Xvfb Xvfb] <tt>X virtual framebuffer</tt> and shared memory volume <code>-v /dev/shm:/dev/shm</code>
* it's possible to run in complete headless mode without <code>Xvfb</code>
* can be run as a server then Python code pointing to the remote server where webdriver runs on
= Serverless - AWS Lambda =
== [https://read.iopipe.com/the-right-way-to-do-serverless-in-python-e99535574454 The Right Way™ to do Serverless in Python] ==
* [https://read.iopipe.com/the-right-way-to-do-serverless-in-python-part-2-63430131239 part2]
* [https://read.iopipe.com/cutting-through-the-layers-aws-lamba-layers-explained-28e8a8d7bda8 Cutting Through the Layers: AWS Lambda Layers Explained]
===References===
Learning
*TODO
**[https://medium.com/@info.ankitp/going-serverless-with-aws-lambda-functions-1-edec330c2bce 1st AWS Lambda]
*Completed
**[https://aws.amazon.com/getting-started/projects/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/module-5/ Build a Serverless Web Application] 2020-02


=References=
=References=

Latest revision as of 19:35, 29 July 2020

Helpers

Python3 - Virtual environments

needs rewriting [ https://dev.to/codemouse92/dead-simple-python-virtual-environments-and-pip-5b56 venv] dev.to Part of "Dead Simple Python" series.

A virtual environment, or virtualenv as it's sometimes called, is a sandbox where you can install only the Python packages you need.

python3 -V # -> eg. Python 3.6.8

# Install virtualenv python3 packages
sudo apt install build-essential libssl-dev libffi-dev python-dev
sudo apt install virtualenv python3-virtualenv python3-pip
sudo apt install python3-venv # Debian, Ubuntu systems, tested with U20.04

# Python3.3+ use native command to create virtual environments
python3 -m venv venv          # 1st venv is the command, 2nd "venv" is a project name

# (Depricated) legacy method to create virtualenv 
virtualenv -p python3 venv            # create
virtualenv -p /usr/bin/python3.6 venv # create venv pointing to use specific executable, could be eg. py2.7

$ source ./venv/bin/activate    # activate
(venv) ubuntu@u18gui-1:~/py$    # <- prompt updated with env name
$ deactivate                    # deactivate
vagrant@ubuntu-bionic ~  $      # <- prompt cleaned

# Running venv without activating, work the same as if you had activated the virtual environment
venv/bin/python
venv/bin/pip install pylint


Note: Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.

Git

Within a virtual environment's directory are the actual packages you installed with pip, therefore working with any VCS (Version Control System) you should ignore whole venv/ directory by adding it to eg. .gitignore.

Resources

pip - python package manager

Do not use pip it self to upgrade pip as it will break your installation. Use OS package maanger instead. Using --user option to install packages it's a good habit as well as it will use the user scheme to install the package.

sudo apt-get install python-pip  #pip for pyhton2, eg. check apt-cache show python-pip, python dependency version
sudo apt-get install python3-pip #pip for python3

pip list            #list packages, no need to use eg. pip3 within venv
pip install PySide2 #insall a package
pip install PySide2==5.11.1 #or ">=" at least this version, or greater
pip install --upgrade PySide2
pip uninstall PySide2

pip install -r requirements.txt
pip install --upgrade -r requirements.txt


Search on PyPI.org

pip search web scraping

haSH-BANG or #!

Rules for python vs python2 vs python3 are described in pep-0394

#!/usr/bin/env python3   # correct way
#!/usr/bin/python        # forces use a system-wide copy of Python, won't respect "venv"

Syntax, semantics and functions

Cheatsheet Python2 and Python3

print()

Print in v3 is a function, it uses arguments with variables like below:

%s # String (or any object with a string representation, like numbers)
%d # Integers
%f # Floating point numbers
%.<number-of-digits>f # Floating point numbers with a fixed amount of digits to the right of the dot.
%x/%X # Integers in hex representation (lowercase/uppercase)

# Example
print("%s is %d years old." % (name, age))

Simple HTTP Server

It provides the local files browser over http protocol and access logs

sudo python  -m SimpleHTTPServer 80 #python 2
sudo python3 -m http.server      80 #python 3

Web scraping

Install Selenium lib on Ubuntu

$ which python python3
/usr/bin/python
/usr/bin/python3
sudo apt-get install python3-pip        # install pip for python3
sudo python3 -m pip install -U selenium # install selenium for python3
sudo python3 -m pip install -U pip      # upgrade pip

Install geckodriver on Ubuntu

It's required if you use Selenium in Python, webdriver function:

default_browser = webdriver.Firefox()
Firefox geckodriver (verified on Ubuntu 16.04, 18.04 running on Vagrant)
wget https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz
sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.26.0-linux64.tar.gz -O > /usr/bin/geckodriver'
sudo chmod +x /usr/bin/geckodriver
rm geckodriver-v0.26.0-linux64.tar.gz


Chromedriver
wget https://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo chmod +x chromedriver
sudo mv chromedriver /usr/bin/
rm chromedriver_linux64.zip

Selenium - Website automation test to mimic human interaction - page login

Script below demonstrate usage of Selenium automation through Mozilla Geckodriver. It will open Firefox, navigate to a give URL, fill in a form, click a login button and take a screenshot.


Credentials file

cat creds.yaml
wpUser: user
wpPass: pass123


Main script

#!/usr/bin/python3
from selenium import webdriver
import yaml
driver = webdriver.Firefox()
creds = yaml.safe_load(open('creds.yaml','r'))
driver.get('http://wiki.ciscolinux.co.uk/index.php?title=Special:UserLogin&returnto=Main+Page')

user_field = driver.find_element_by_name("wpName")
pass_field = driver.find_element_by_name("wpPassword")
user_field.send_keys(creds['wpUser'])
pass_field.send_keys(creds['wpPass'])
# find login button and click on it
driver.find_element_by_name('wploginattempt').click()
driver.save_screenshot('screenshot.png')
driver.close


Now you can preview the screenshot from the terminal window.

Selenium Webdriver running in a docker


Install Selenium and run from a terminal

sudo apt install firefox python3-pip xvfb x11-utils --yes
sudo -H pip3 install bpython selenium # better option is to install in virtual environment

export DISPLAY=:2    # fake X display
Xvfb $DISPLAY -ac &  # X virtual framebuffer

export GECKO_DRIVER_VERSION='v0.26.0'
wget https://github.com/mozilla/geckodriver/releases/download/$GECKO_DRIVER_VERSION/geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
tar -xvzf geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
rm        geckodriver-$GECKO_DRIVER_VERSION-linux64.tar.gz
chmod +x  geckodriver
sudo cp   geckodriver /usr/local/bin/

cat <<EOF > script.py
#!/usr/bin/env python3
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile

ff_options = FirefoxOptions()
ff_options.headless = True

ff = Firefox(options=ff_options)
ff.quit()
EOF
chmod +x script.py


Official Selenium docker images can get a lot of parameters:

  • basic run uses Xvfb X virtual framebuffer and shared memory volume -v /dev/shm:/dev/shm
  • it's possible to run in complete headless mode without Xvfb
  • can be run as a server then Python code pointing to the remote server where webdriver runs on

Serverless - AWS Lambda

The Right Way™ to do Serverless in Python

References

Learning

References