25 Apr

Installing Django on Ubuntu, uWSGI, PostgreSQL, and Celery+Redis

1. Install Python3 (If not installed)
$ sudo apt-get update
$ sudo apt-get install python3 python3-dev
2. Install PostgreSQL
$ sudo apt-get update
$ sudo apt-get install postgresql postgresql-contrib libpq-dev

During the Postgres installation, an operating system user named postgres was created to correspond to the postgres PostgreSQL administrative user.
We need to change to this user to perform administrative tasks:

$ sudo su - postgres

Log into a Postgres session by typing:

$ psql

First, we will create a database for our Django project. Each project should have its own isolated database for security reasons.
We will call our database myproject in this guide, but it’s always better to select something more descriptive:

> CREATE DATABASE myproject;

Remember to end all commands at an SQL prompt with a semicolon.

Create a database user which we will use to connect to and interact with the database.

> CREATE USER myprojectuser WITH PASSWORD 'password';

Modify a few of the connection parameters for the user just created to speed up database operations.

We are setting the default encoding to UTF-8, and timezone to UTC which Django expects.
We are also setting the default transaction isolation scheme to “read committed”, which blocks reads from uncommitted transactions.

> ALTER ROLE myprojectuser SET client_encoding TO 'utf8';
> ALTER ROLE myprojectuser SET default_transaction_isolation TO 'read committed';
> ALTER ROLE myprojectuser SET timezone TO 'UTC';

Give our database user access rights to the database we created:

> GRANT ALL PRIVILEGES ON DATABASE myproject TO myprojectuser;

Exit the SQL prompt to get back to the postgres user’s shell session:

> \q

Exit out of the postgres user’s shell session to get back to your regular user’s shell session:

$ exit

Optionally Change the

$ sudo nano /etc/postgresql/X/main/pg_hba.conf
local all postgres peer

Should be:

local all postgres md5
  • If you can’t find this file, running ‘locate pg_hba.conf’ should show you where the file is.

After altering this file, restart PostgreSQL server.

$ sudo service postgresql restart.
3. Pull the project from Git store
$ cd ~
$ git clone https://gitlab.com/path/myproject.git
$ cd myproject/

Install Virtualenv

$ python3 -m venv venv
$ source venv/bin/activate
(venv) sudo -H pip3 install --upgrade pip
(venv) pip install -r requirements.txt
4. Configure the Django Database Settings

Now that we have a project, we need to configure it to use the database we created.

Install the psycopg2 package (if not in the requirement.txt) that will allow us to use the database we configured:

$ (venv) pip install psycopg2

Open the main Django project settings file located within the child project directory:

$ nano ~/myproject/myproject/settings.py
. . .

The simplest case: just add the domain name(s) and IP addresses of your Django server

ALLOWED_HOSTS = [ 'example.com', '203.0.113.5']

To respond to ‘example.com’ and any subdomains, start the domain with a dot

ALLOWED_HOSTS = ['.example.com', '203.0.113.5', 127.0.0.1]
ALLOWED_HOSTS = ['your_server_domain_or_IP', 'second_domain_or_IP', . . .]
. . .
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

Towards the bottom of the file, you will see a DATABASES section, change it to:

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'myproject', # Or path to database file if using sqlite3.
# The following settings are not used with sqlite3:
'USER': 'myuser',
'PASSWORD': 'password',
'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
'PORT': '', # Set to empty string for default.
  }
}

When you are finished, save and close the file.

5. Migrate the Database and Test your Project

Now that the Django settings are configured, we can migrate our data structures to our database and test out the server.

We can begin by creating and applying migrations to our database. Since we don’t have any actual data yet, this will simply set up the initial database structure:

$ cd ~/myproject
$ python manage.py makemigrations
$ python manage.py migrate

After creating the database structure, we can create an administrative account by typing:

$ python manage.py createsuperuser

Select a username, provide an email address, and password for the account.

Once you have an admin account set up, you can test that your database is performing correctly by starting up the Django development server:

$ python manage.py runserver 0.0.0.0:8000

In your web browser, visit your server’s domain name or IP address followed by :8000 to reach default Django root page:

http://server_domain_or_IP:8000
6. Install and Configure uWSGI
$ cd project/path
$ sudo pip3 install uwsgi
Test the uwsgi
$ uwsgi --http :8080 --home /home/myproject --chdir /home/myproject -w myproject.wsgi

Press CTRL+C to stop

Create uwsgi INI file

$ nano /etc/uwsgi/sites/myproject.ini
[uwsgi]
chdir = /home/myproject
home = /home/myproject/venv
module = myproject.wsgi:application
master = true
processes = 5

#socket = /run/uwsgi/myproject.sock
protocol = http # Must be set to use http socket with systemd
socket    = :8080
chmod-socket = 666
vacuum = true

die-on-term = true

To have uWSGI serve HTTP (instead of the binary uwsgi protocol) under Systemd socket activation, set protocol to http; for instance, in an INI, do this:

$ sudo nano /etc/systemd/system/uwsgi.service
[Unit]
Description=uWSGI Emperor
[Service]
ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown www-data:www-data /run/uwsgi'
ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Make the file executable.

$ sudo chmod a+x /etc/systemd/system/uwsgi.service

Run the service

$ systemctl start uwsgi.service
$ systemctl status uwsgi.service
7. Install and Configure Nginx as a Reverse Proxy
$ sudo apt-get install nginx
$ sudo nano /etc/nginx/sites-available/myproject
server {
listen 80;
server_name myproject.com www.myproject.com;
    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/myproject;
    }

    location / {
        include         uwsgi_params;
        #uwsgi_pass      unix:/run/uwsgi/myproject.sock;
        proxy_pass      http://127.0.0.1:8080;
    }
}
$ sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled
$ sudo nginx -t
$ sudo service nginx restart
8. Troubleshoot

For NGINX errors

$ sudo tail -f /var/log/nginx/error.log

For others errors not in the standard log files

$ sudo tail -f /var/log/syslog
9. Using Celery and Redis for Background Task Processing

Step 1: Add celery.py

Inside the “myproject” directory, create a new file called celery.py:

$ nano ~/myproject/myproject/celery.py
from future import absolute_import
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('myproject')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Step 3: Install Redis as a Celery “Broker”

$ sudo apt update
$ sudo apt install redis-server
$ sudo nano /etc/redis/redis.conf

The supervised directive is set to no by default. Since you are running Ubuntu, which uses the systemd init system, change this to systemd

$ sudo systemctl restart redis.service
$ sudo systemctl status redis
$ redis-cli
127.0.0.1:6379> ping
PONG

Check the redis binding

$ sudo netstat -lnp | grep redis

Once Redis is up, add the following code to your settings.py file:

$ nano ~/myproject/myproject/settings.py

# CELERY STUFF
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
CELERY_BEAT_SCHEDULE = {
'task1': {
'task': 'Joliba.tasks.task1',
'schedule': crontab(minute='*/10'),
    },
}

Install the celery and redis package (if not in the requirement.txt)

(venv) $ pip install celery redis

Restart the uwsgi application

$ sudo systemctl restart uwsgi.service
10. Add SSL for the website using letsencrypt.

Adding SSL with Let’s Encrypt and NGINX

27 Jan

Adding SSL with Let’s Encrypt and NGINX

Editor – This is an update to a previous blog post about using Let’s Encrypt certificates with NGINX. This new blog is based on newly added NGINX support in certbot.

The Certbot client supports two types of plugins for obtaining and installing certificates: authenticators and installers.

Authenticators validates that you control the domain(s) you are requesting a certificate for, obtains a certificate for the specified domain(s), and places the certificate in the /etc/letsencrypt directory on your machine. The authenticator does not install the certificate (it does not edit any of your server’s configuration files to serve the obtained certificate).

Installers are Plugins used with the install command to install a certificate. These plugins can modify your webserver’s configuration to serve your website over HTTPS using certificates obtained by certbot.
`http://letsencrypt.readthedocs.io/en/latest/using.html#getting-certificates-and-choosing-plugins`

Let’s Encrypt client provides 3 plugins for different web servers.

apache – Automates obtaining and installing a certificate with Apache 2.4 on Debian-based distributions with libaugeas0 1.0+.
webroot – Obtains a certificate by writing to the webroot directory of an already running webserver.
nginx – Automates obtaining and installing a certificate with Nginx.
standalone – Uses a “standalone” webserver to obtain a certificate. This is useful on systems with no webserver, or when direct integration with the local webserver is not supported or not desired. Requires port 80 or 443 to be available.
manual – Helps you obtain a certificate by giving you instructions to perform domain validation yourself. Additionally allows you to specify scripts to automate the validation task in a customized way.

We will be using the webroot plugin for this guide

1. Add the certbot repository:

$ sudo add-apt-repository ppa:certbot/certbot

Now install certbot:

$ sudo apt-get update
$ sudo apt-get install python-certbot-nginx

2. Set up NGINX
Add the webroot on the nginx server block

$ sudo vi /etc/nginx/sites-available/example.com
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name example.com www.example.com;

#LetsEncrypt webroot
location ~ /.well-known {
allow all;
root /home/www/example;
}
}
$ sudo nginx -t
$ sudo service nginx restart
3. Obtain the SSL certificate
$ sudo certbot certonly --webroot -w /home/www/example/ -d example.com -d www.example.com

4. Update NGINX
$ sudo vi /etc/nginx/sites-available/example.com

Check if the settings are updated by the

server {
listen 80;
server_name example.com www.example.com;

# Redirect non-https traffic to https
location / { return 301 https://$host$request_uri; }
} 
server {
listen 443 ssl;
server_name example.com www.example.com;

# RSA certificate
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot

include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
}
$ sudo nginx -t
$ sudo service nginx restart

To obtain a new certificate that contains additional domains without replacing your existing certificate you must use the –duplicate option.

For example:
$ /usr/bin/certbot --duplicate certonly --webroot -w /home/www/example/ -d example.com -d www.example.com -d sub.example.com

or

$ sudo certbot --nginx -d conva.net -d home/www/example/ -d example.com -d www.example.com -d sub.example.com

5. Add R
sudo crontab -e

Add line on crontab
12 3 * * * letsencrypt renew >> /var/log/letsencrypt/renew.log

27 Apr

Bio-metric data format standards

In developing software for handling bio-metric records, several standard are used for keeping the information.

These are the INCITS 378-2004 and ISO/IEC 19794-2:2005 (finger minutiae), INCITS 381-2004 and ISO/IEC 19794-4:2004 (finger image), and INCITS 385-2004 (face recognition format). The American National Standards were developed in the InterNational Committee for Information Technology Standards (INCITS(link is external)) Technical Committee M1 – Biometrics(link is external). The International standards were developed by the ISO/IEC JTC 1/SC 37(link is external) committee.

INCITS biometric standards have been adopted for use in various government programs including the Personal Identity Verification of Federal Employees and Contractors program developed by NIST. Their use is specified in NIST Special Publication 800-76-1.

05 Feb

Add SSL (https) to a CPanel Site

After Installing the SSL certificate, (CPanels and LetsEncrypt provides free SSL Certificate), the next step is redirecting the http:// traffic to https://

Using the redirect facility will result in redirect loop, as such add the following code at the end of .htaccess file on the root of the domain

RewriteEngine On 
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]

H/T to http://www.webhostinghub.com/help/learn/website/ssl/force-website-to-use-ssl

03 Oct

SSL/TLS Certificates on Flask Application with Let’s Encrypt and NGINX

Let’s Encrypt is a new certificate authority (CA) offering free and automated SSL/TLS certificates. Certificates issued by Let’s Encrypt are trusted by most browsers in production today, including Internet Explorer on Windows Vista. Simply download and run the Let’s Encrypt client to generate a certificate.

(there are a few more steps than that, of course, though not many)
Step 1: Download LetsEncrypt

Install git if you haven’t done so yet:

# apt-get install git

Use git to get the application and store it somewhere (ie: /opt)

$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Step 2: Webroot Plugin

The Webroot plugin works by placing a special file in the /.well-known directory within your document root, which can be opened (through your web server) by the Let’s Encrypt service for validation.
Depending on your configuration, you may need to explicitly allow access to the /.well-known directory.

location /.well-known {
alias /home/user/webapps/appname/.well-known;
}

Restart NGNIX

# sudo service nginx status

 

Step 3: Generate your certificate and Strong Diffie-Hellman Group

The first time you run the command below, you will be asked to provide an e-mail address to be associated to the domain or subdomain, in case you should ever need to recover the key or something.
The next time you run the same command (to renew the certificate) it won’t be asked.

So run the following command to generate the certificate:

$ sudo /opt/letsencrypt/letsencrypt-auto certonly -a webroot –agree-tos –renew-by-default \

–webroot-path=/home/user/webapps/appname \

-d website.com [-d sub.website.com] \

–e-mail=email@website.com

Then Generate Strong Diffie-Hellman Group
This may take a few minutes but when it’s done you will have a strong DH group at /etc/ssl/certs/dhparam.pem.
$ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Step 4: Configuring Nginx

After running the command that generates the certificates, you should have several files in /etc/letsencrypt/live/website.com/ (replace website.com by your own domain).
We are going to need just two of them for Nginx: fullchain.pem and privkey.pem.
Comment out or delete the lines that configure this server block to listen on port 80.
The beginning of your server block should look like this:

server {

server_name website.com www.website.com;

listen 443 ssl;

ssl_certificate /etc/letsencrypt/live/website.com/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/live/website.com/privkey.pem;

# For Safari and iOS devices

ssl_session_cache shared:SSL:20m;

 

#Diffie-Hellman Group

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_prefer_server_ciphers on;

ssl_dhparam /etc/ssl/certs/dhparam.pem;

ssl_ciphers ‘ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’;

ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

add_header Strict-Transport-Security max-age=15768000;

Lastly, outside of the original server block (that is listening on HTTPS, port 443), add this server block to redirect HTTP (port 80) to HTTPS.

server {
listen 80;
server_name website.com www.website.com;
return 301 https://$host$request_uri;
}

Put the changes into effect by restarting Nginx:

$ sudo service nginx restart

The Let’s Encrypt TLS/SSL certificate is now in place.

At this point, you should test that the TLS/SSL certificate works by visiting your domain via HTTPS in a web browser.
You can use the Qualys SSL Labs Report to see how your server configuration scores:

https://www.ssllabs.com/ssltest/analyze.html?d=website.com

Step 5: Automate the Certificate Renewal

Edit the crontab to create a new job that will run the renewal command every week.

$ sudo crontab –e

30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log

35 2 * * 1 /etc/init.d/nginx reload

25 Feb

Migrating from Python 2.7 to Python 3.4

Last year, we made a decision to migrate to python 3.4 in 2016. This was not an easy decision as all our code base in developed using python 2.7 and it has served us very well.

Yesterday, We took the big step. This has already been delayed and was overdue.

First the Python 3.4 has to be installed, using the same process as in previous blog. However, replacing 2.7 with 3.4

the virtualenv has to be recreated for python 3.4.

pyvenv <virtual env>

thunder lock: disabled (you can enable it with --thunder-lock)
 uwsgi socket 0 bound to UNIX address xhod.sock fd 3
 Python version: 2.7.10 (default, Sep 21 2015, 12:21:01) [GCC 4.4.7 20120313 (Red Hat 4.4.7-16)]
 Set PythonHome to /home/uobis/webapps/xhod/venv
 ImportError: No module named site

I had to recompile uwsgi for python 3.4

/usr/local/bin/pip2.7 uninstall uWSGI

/usr/local/bin/pip3.4 install uWSGI

*** Starting uWSGI 2.0.12 (64bit) on [Thu Feb 25 22:31:10 2016] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-16) on 25 February 2016 22:30:37
os: Linux-2.6.32-042stab108.8 #1 SMP Wed Jul 22 17:23:23 MSK 2015

 

01 Jun

uWSGI+NGINX on Linux (Centos 6)

Step 1 – Install NGINX

sudo yum install nginx

Step 2 – Add nginx to uobis group:

sudo usermod -a -G uobis nginx
sudo chmod g+x /home && chmod g+x /home/uobis && chmod g+x home/uobis/webapps/
sudo -u nginx stat /home/uobis/webapps/

cat >> kabbu.ini

[uwsgi]
module = wsgi
master = true
processes = 5
socket = kabbu.sock
chmod-socket = 660
vacuum = true

Step 3 – The uWSGI Emperor – multi-app deployment
exec uwsgi --emperor /etc/uwsgi/vassals/ --master --logto /var/log/uwsgi.log
Step 4 – Create a Upstart File

To make sure stuff is automatically run, here’s a configuration file that needs to be placed in /etc/init/uwsgi.conf

description "uWSGI Emperor"

start on runlevel [2345]
stop on runlevel [06]
respawn
exec uwsgi --emperor /etc/uwsgi/vassals/ --master --logto /var/log/uwsgi/emperor.log

Create the directories

mkdir /etc/uwsgi/vassals/

Then either reboot your system or:

sudo initctl start uwsgi
sudo initctl reload-configuration

To check if its started
initctl list

Check the system log file or the emperor.log if uwsgi is not started

30 May

Install Python on Linux (Centos)

Most installed python on Linux systems are outdated. While python 2.6 is quite good for a project, most are advised to use python 2.7 or 3.4, so the need to upgrade it.

However, for Centos 6, you are strongly advised to avoid upgrading or reinstalling the stock version as some important system programs (notable yum), uses it.
Removing the default version will break the dependent system programs, instead you should install your targeted version and make sure your files are linked to it.

Install Python 2.7

Run the following command to update the system applications:
$ sudo yum -y update

In order to get the necessary development tools, run the following:
$ sudo yum groupinstall -y development

# sudo yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel

Download, compile and install Python:

$ cd /opt
$ sudo wget --no-check-certificate https://www.python.org/ftp/python/2.7.X/Python-2.7.X.tar.xz
$ sudo tar xf Python-2.7.X.tar.xz
$ sudo cd Python-2.7.X
$ sudo ./configure --prefix=/usr/local
$ sudo make && make altinstall

List all python programs

$ ls -ltr /usr/bin/python*
$ ls -ltr /usr/local/bin/python*

Link Python 2.7 to the default python on local environment

$ ln -s /usr/local/bin/python2.7 /usr/local/bin/python

Install easy tools and pip

$ wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
$ sudo /usr/local/bin/python2.7 ez_setup.py
$ sudo /usr/local/bin/easy_install-2.7 pip

Install virtualenv
$ pip2.7 install virtualenv

 

27 May

NGINX server blocks in Centos 6

How to setup NGINX server blocks (aka VirualHosts) on Centos
Step One — Set Up New Document Root Directories
sudo mkdir -p /var/www/example.com/html
sudo mkdir -p /var/www/test.com/html

sudo chown -R $USER:$USER /var/www/example.com/html

Step Two — Create Sample Pages for Each Site
Step Three — Create Server Block Files for Each Domain
The sites-available directory will keep all of our server block files, while the sites-enabled directory will hold symbolic links to server blocks that we want to publish.

sudo mkdir /etc/nginx/sites-available
sudo mkdir /etc/nginx/sites-enabled

sudo vi /etc/nginx/nginx.conf

Add these lines to the end of the http {} block:

include /etc/nginx/sites-enabled/*.conf;
server_names_hash_bucket_size 64;

Create the First Server Block File

sudo cp /etc/nginx/conf.d/default.conf /etc/nginx/sites-available/example.com.conf

sudo nano /etc/nginx/sites-available/example.com.conf

server {
listen 80
server_name example.com www.example.com;
location / {
root /var/www/example.com/html;
index index.html index.htm;
try_files $uri $uri/ =404;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}

Create the Second Server Block File

sudo cp /etc/nginx/conf.d/default.conf /etc/nginx/sites-available/example2.com.conf

Step Four — Enable the New Server Block Files

sudo ln -s /etc/nginx/sites-available/kabbu.ng.conf /etc/nginx/sites-enabled/kabbu.ng.conf
sudo ln -s /etc/nginx/sites-available/umscholar.org.conf /etc/nginx/sites-enabled/umscholar.org.conf

Restart NGINX
/etc/init.d/nginx restart