25 Apr

Installing Django on Ubuntu, uWSGI, PostgreSQL, and Celery+Redis

1. Install Python3 (If not installed)
$ sudo apt-get update
$ sudo apt-get install python3 python3-dev
2. Install PostgreSQL
$ sudo apt-get update
$ sudo apt-get install postgresql postgresql-contrib libpq-dev

During the Postgres installation, an operating system user named postgres was created to correspond to the postgres PostgreSQL administrative user.
We need to change to this user to perform administrative tasks:

$ sudo su - postgres

Log into a Postgres session by typing:

$ psql

First, we will create a database for our Django project. Each project should have its own isolated database for security reasons.
We will call our database myproject in this guide, but it’s always better to select something more descriptive:

> CREATE DATABASE myproject;

Remember to end all commands at an SQL prompt with a semicolon.

Create a database user which we will use to connect to and interact with the database.

> CREATE USER myprojectuser WITH PASSWORD 'password';

Modify a few of the connection parameters for the user just created to speed up database operations.

We are setting the default encoding to UTF-8, and timezone to UTC which Django expects.
We are also setting the default transaction isolation scheme to “read committed”, which blocks reads from uncommitted transactions.

> ALTER ROLE myprojectuser SET client_encoding TO 'utf8';
> ALTER ROLE myprojectuser SET default_transaction_isolation TO 'read committed';
> ALTER ROLE myprojectuser SET timezone TO 'UTC';

Give our database user access rights to the database we created:

> GRANT ALL PRIVILEGES ON DATABASE myproject TO myprojectuser;

Exit the SQL prompt to get back to the postgres user’s shell session:

> \q

Exit out of the postgres user’s shell session to get back to your regular user’s shell session:

$ exit

Optionally Change the

$ sudo nano /etc/postgresql/X/main/pg_hba.conf
local all postgres peer

Should be:

local all postgres md5
  • If you can’t find this file, running ‘locate pg_hba.conf’ should show you where the file is.

After altering this file, restart PostgreSQL server.

$ sudo service postgresql restart.
3. Pull the project from Git store
$ cd ~
$ git clone https://gitlab.com/path/myproject.git
$ cd myproject/

Install Virtualenv

$ python3 -m venv venv
$ source venv/bin/activate
(venv) sudo -H pip3 install --upgrade pip
(venv) pip install -r requirements.txt
4. Configure the Django Database Settings

Now that we have a project, we need to configure it to use the database we created.

Install the psycopg2 package (if not in the requirement.txt) that will allow us to use the database we configured:

$ (venv) pip install psycopg2

Open the main Django project settings file located within the child project directory:

$ nano ~/myproject/myproject/settings.py
. . .

The simplest case: just add the domain name(s) and IP addresses of your Django server

ALLOWED_HOSTS = [ 'example.com', '203.0.113.5']

To respond to ‘example.com’ and any subdomains, start the domain with a dot

ALLOWED_HOSTS = ['.example.com', '203.0.113.5', 127.0.0.1]
ALLOWED_HOSTS = ['your_server_domain_or_IP', 'second_domain_or_IP', . . .]
. . .
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

Towards the bottom of the file, you will see a DATABASES section, change it to:

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'myproject', # Or path to database file if using sqlite3.
# The following settings are not used with sqlite3:
'USER': 'myuser',
'PASSWORD': 'password',
'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
'PORT': '', # Set to empty string for default.
  }
}

When you are finished, save and close the file.

5. Migrate the Database and Test your Project

Now that the Django settings are configured, we can migrate our data structures to our database and test out the server.

We can begin by creating and applying migrations to our database. Since we don’t have any actual data yet, this will simply set up the initial database structure:

$ cd ~/myproject
$ python manage.py makemigrations
$ python manage.py migrate

After creating the database structure, we can create an administrative account by typing:

$ python manage.py createsuperuser

Select a username, provide an email address, and password for the account.

Once you have an admin account set up, you can test that your database is performing correctly by starting up the Django development server:

$ python manage.py runserver 0.0.0.0:8000

In your web browser, visit your server’s domain name or IP address followed by :8000 to reach default Django root page:

http://server_domain_or_IP:8000
6. Install and Configure uWSGI
$ cd project/path
$ sudo pip3 install uwsgi
Test the uwsgi
$ uwsgi --http :8080 --home /home/myproject --chdir /home/myproject -w myproject.wsgi

Press CTRL+C to stop

Create uwsgi INI file

$ nano /etc/uwsgi/sites/myproject.ini
[uwsgi]
chdir = /home/myproject
home = /home/myproject/venv
module = myproject.wsgi:application
master = true
processes = 5

#socket = /run/uwsgi/myproject.sock
protocol = http # Must be set to use http socket with systemd
socket    = :8080
chmod-socket = 666
vacuum = true

die-on-term = true

To have uWSGI serve HTTP (instead of the binary uwsgi protocol) under Systemd socket activation, set protocol to http; for instance, in an INI, do this:

$ sudo nano /etc/systemd/system/uwsgi.service
[Unit]
Description=uWSGI Emperor
[Service]
ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown www-data:www-data /run/uwsgi'
ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Make the file executable.

$ sudo chmod a+x /etc/systemd/system/uwsgi.service

Run the service

$ systemctl start uwsgi.service
$ systemctl status uwsgi.service
7. Install and Configure Nginx as a Reverse Proxy
$ sudo apt-get install nginx
$ sudo nano /etc/nginx/sites-available/myproject
server {
listen 80;
server_name myproject.com www.myproject.com;
    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/myproject;
    }

    location / {
        include         uwsgi_params;
        #uwsgi_pass      unix:/run/uwsgi/myproject.sock;
        proxy_pass      http://127.0.0.1:8080;
    }
}
$ sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled
$ sudo nginx -t
$ sudo service nginx restart
8. Troubleshoot

For NGINX errors

$ sudo tail -f /var/log/nginx/error.log

For others errors not in the standard log files

$ sudo tail -f /var/log/syslog
9. Using Celery and Redis for Background Task Processing

Step 1: Add celery.py

Inside the “myproject” directory, create a new file called celery.py:

$ nano ~/myproject/myproject/celery.py
from future import absolute_import
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('myproject')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Step 3: Install Redis as a Celery “Broker”

$ sudo apt update
$ sudo apt install redis-server
$ sudo nano /etc/redis/redis.conf

The supervised directive is set to no by default. Since you are running Ubuntu, which uses the systemd init system, change this to systemd

$ sudo systemctl restart redis.service
$ sudo systemctl status redis
$ redis-cli
127.0.0.1:6379> ping
PONG

Check the redis binding

$ sudo netstat -lnp | grep redis

Once Redis is up, add the following code to your settings.py file:

$ nano ~/myproject/myproject/settings.py

# CELERY STUFF
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
CELERY_BEAT_SCHEDULE = {
'task1': {
'task': 'Joliba.tasks.task1',
'schedule': crontab(minute='*/10'),
    },
}

Install the celery and redis package (if not in the requirement.txt)

(venv) $ pip install celery redis

Restart the uwsgi application

$ sudo systemctl restart uwsgi.service
10. Add SSL for the website using letsencrypt.

Adding SSL with Let’s Encrypt and NGINX

03 Oct

SSL/TLS Certificates on Flask Application with Let’s Encrypt and NGINX

Let’s Encrypt is a new certificate authority (CA) offering free and automated SSL/TLS certificates. Certificates issued by Let’s Encrypt are trusted by most browsers in production today, including Internet Explorer on Windows Vista. Simply download and run the Let’s Encrypt client to generate a certificate.

(there are a few more steps than that, of course, though not many)
Step 1: Download LetsEncrypt

Install git if you haven’t done so yet:

# apt-get install git

Use git to get the application and store it somewhere (ie: /opt)

$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Step 2: Webroot Plugin

The Webroot plugin works by placing a special file in the /.well-known directory within your document root, which can be opened (through your web server) by the Let’s Encrypt service for validation.
Depending on your configuration, you may need to explicitly allow access to the /.well-known directory.

location /.well-known {
alias /home/user/webapps/appname/.well-known;
}

Restart NGNIX

# sudo service nginx status

 

Step 3: Generate your certificate and Strong Diffie-Hellman Group

The first time you run the command below, you will be asked to provide an e-mail address to be associated to the domain or subdomain, in case you should ever need to recover the key or something.
The next time you run the same command (to renew the certificate) it won’t be asked.

So run the following command to generate the certificate:

$ sudo /opt/letsencrypt/letsencrypt-auto certonly -a webroot –agree-tos –renew-by-default \

–webroot-path=/home/user/webapps/appname \

-d website.com [-d sub.website.com] \

–e-mail=email@website.com

Then Generate Strong Diffie-Hellman Group
This may take a few minutes but when it’s done you will have a strong DH group at /etc/ssl/certs/dhparam.pem.
$ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Step 4: Configuring Nginx

After running the command that generates the certificates, you should have several files in /etc/letsencrypt/live/website.com/ (replace website.com by your own domain).
We are going to need just two of them for Nginx: fullchain.pem and privkey.pem.
Comment out or delete the lines that configure this server block to listen on port 80.
The beginning of your server block should look like this:

server {

server_name website.com www.website.com;

listen 443 ssl;

ssl_certificate /etc/letsencrypt/live/website.com/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/live/website.com/privkey.pem;

# For Safari and iOS devices

ssl_session_cache shared:SSL:20m;

 

#Diffie-Hellman Group

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_prefer_server_ciphers on;

ssl_dhparam /etc/ssl/certs/dhparam.pem;

ssl_ciphers ‘ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’;

ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

add_header Strict-Transport-Security max-age=15768000;

Lastly, outside of the original server block (that is listening on HTTPS, port 443), add this server block to redirect HTTP (port 80) to HTTPS.

server {
listen 80;
server_name website.com www.website.com;
return 301 https://$host$request_uri;
}

Put the changes into effect by restarting Nginx:

$ sudo service nginx restart

The Let’s Encrypt TLS/SSL certificate is now in place.

At this point, you should test that the TLS/SSL certificate works by visiting your domain via HTTPS in a web browser.
You can use the Qualys SSL Labs Report to see how your server configuration scores:

https://www.ssllabs.com/ssltest/analyze.html?d=website.com

Step 5: Automate the Certificate Renewal

Edit the crontab to create a new job that will run the renewal command every week.

$ sudo crontab –e

30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log

35 2 * * 1 /etc/init.d/nginx reload

25 Feb

Migrating from Python 2.7 to Python 3.4

Last year, we made a decision to migrate to python 3.4 in 2016. This was not an easy decision as all our code base in developed using python 2.7 and it has served us very well.

Yesterday, We took the big step. This has already been delayed and was overdue.

First the Python 3.4 has to be installed, using the same process as in previous blog. However, replacing 2.7 with 3.4

the virtualenv has to be recreated for python 3.4.

pyvenv <virtual env>

thunder lock: disabled (you can enable it with --thunder-lock)
 uwsgi socket 0 bound to UNIX address xhod.sock fd 3
 Python version: 2.7.10 (default, Sep 21 2015, 12:21:01) [GCC 4.4.7 20120313 (Red Hat 4.4.7-16)]
 Set PythonHome to /home/uobis/webapps/xhod/venv
 ImportError: No module named site

I had to recompile uwsgi for python 3.4

/usr/local/bin/pip2.7 uninstall uWSGI

/usr/local/bin/pip3.4 install uWSGI

*** Starting uWSGI 2.0.12 (64bit) on [Thu Feb 25 22:31:10 2016] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-16) on 25 February 2016 22:30:37
os: Linux-2.6.32-042stab108.8 #1 SMP Wed Jul 22 17:23:23 MSK 2015

 

01 Jun

uWSGI+NGINX on Linux (Centos 6)

Step 1 – Install NGINX

sudo yum install nginx

Step 2 – Add nginx to uobis group:

sudo usermod -a -G uobis nginx
sudo chmod g+x /home && chmod g+x /home/uobis && chmod g+x home/uobis/webapps/
sudo -u nginx stat /home/uobis/webapps/

cat >> kabbu.ini

[uwsgi]
module = wsgi
master = true
processes = 5
socket = kabbu.sock
chmod-socket = 660
vacuum = true

Step 3 – The uWSGI Emperor – multi-app deployment
exec uwsgi --emperor /etc/uwsgi/vassals/ --master --logto /var/log/uwsgi.log
Step 4 – Create a Upstart File

To make sure stuff is automatically run, here’s a configuration file that needs to be placed in /etc/init/uwsgi.conf

description "uWSGI Emperor"

start on runlevel [2345]
stop on runlevel [06]
respawn
exec uwsgi --emperor /etc/uwsgi/vassals/ --master --logto /var/log/uwsgi/emperor.log

Create the directories

mkdir /etc/uwsgi/vassals/

Then either reboot your system or:

sudo initctl start uwsgi
sudo initctl reload-configuration

To check if its started
initctl list

Check the system log file or the emperor.log if uwsgi is not started

30 May

Install Python on Linux (Centos)

Most installed python on Linux systems are outdated. While python 2.6 is quite good for a project, most are advised to use python 2.7 or 3.4, so the need to upgrade it.

However, for Centos 6, you are strongly advised to avoid upgrading or reinstalling the stock version as some important system programs (notable yum), uses it.
Removing the default version will break the dependent system programs, instead you should install your targeted version and make sure your files are linked to it.

Install Python 2.7

Run the following command to update the system applications:
$ sudo yum -y update

In order to get the necessary development tools, run the following:
$ sudo yum groupinstall -y development

# sudo yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel

Download, compile and install Python:

$ cd /opt
$ sudo wget --no-check-certificate https://www.python.org/ftp/python/2.7.X/Python-2.7.X.tar.xz
$ sudo tar xf Python-2.7.X.tar.xz
$ sudo cd Python-2.7.X
$ sudo ./configure --prefix=/usr/local
$ sudo make && make altinstall

List all python programs

$ ls -ltr /usr/bin/python*
$ ls -ltr /usr/local/bin/python*

Link Python 2.7 to the default python on local environment

$ ln -s /usr/local/bin/python2.7 /usr/local/bin/python

Install easy tools and pip

$ wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
$ sudo /usr/local/bin/python2.7 ez_setup.py
$ sudo /usr/local/bin/easy_install-2.7 pip

Install virtualenv
$ pip2.7 install virtualenv

 

02 Oct

Setting up Django on WebFaction’s Apache with mod_wsgi

If you want to run a Django application on WebFaction, you may simply use their automatic application creation scripts. Unfortunately, if you want to place your application in a Virtualenv, the automatic installer will not help you. I’m sure that WebFaction will eventually add an installer to set this up, but for now, you can use the following tutorial. In this text we set up a Django project in a Virtualenv running on WebFaction’s Apache with mod_wsgi.

Create a new application

Let’s begin by setting up a generic mod_wsgi application in your WebFaction control panel. Log into the control panel, choose the option to add a new application and specify the following settings:

  • Name: test_app
  • App category: mod_wsgi
  • App type : mod_wsgi 3.4 / Python 2.7

The new application will be created in your home directory (~) under: ~/webapps/test_app.

The directory will contain two subdirectories:

  • apache2 – contains the Apache configuration files (apache2/conf) and scripts which let you control the server (apache2/bin)
  • htdocs – contains default page files.

Configure a new website to hook up your application to a domain. Test the website by visiting it in your browser. You should be greeted by a message beginning with the following text:

Welcome to your mod_wsgi website! It uses: Python 2.7....

If you see the above, then the generic application is set up correctly and we can proceed to turn it into a Virtualenv Django application.

Remove htdocs

The htdocs directory will not be needed, so feel free to remove it.

$ cd ~/webapps/test_app
$ rm -r htdocs

Install Virtualenv

Check if Virtualenv is installed on your server:

$ virtualenv --version
-bash: virtualenv: command not found

If Virtualenv is installed, you will see a version number when running the above command. If it’s missing you’ll see a command not found error message instead.

Steps to install Vitrualenv on a WebFaction server are the following:

$ mkdir -p ~/lib/python2.7/
$ easy_install-2.7 pip
$ pip install virtualenv

If you get an permission denied error try this command to install virtualenv inside your user folder:

$  pip install --user virtualenv

Verify that installation was successful:

$ virtualenv --version
1.10.1

Create a virtual environment

Let’s proceed to turn our application directory into a virtual Python environment:

$ cd ~/webapps/test_app
$ virtualenv .

This adds the folders and scripts for a virtual environment inside of the directory which WebFaction created for our application.

You can now activate the created environment:

$ source bin/activate
(test_app) $

Install Django and other dependencies

Once the initial Virtualenv setup is complete, you can install Django inside it’s lib/python2.7/site-packages directory.

(test_app) $ pip install django

Verify that Django installed correctly:

(test_app) $ django-admin.py --version
1.5.2

Your project will probably depend on other packages. You can install those from a REQUIREMENTS.txt file, which you can generate on your development server with the pip freeze command.

(test_app) $ pip install -r REQUIREMENTS.txt

Start a Django project

Let’s create a new Django project inside the virtual environment:

(test_app) $ django-admin.py startproject test_django

Directory structure

At this stage you should have created a directory structure resembling

this:

~/webapps/test_app
|-- apache2
|   |-- bin
|   |   |-- httpd
|   |   |-- httpd.worker
|   |   |-- restart             <== Scripts which start, stop and restart Apache
|   |   |-- start
|   |   `-- stop
|   |-- conf
|   |   |-- httpd.conf          <== Apache configuration file
|   |   `-- mime.types
|   |-- lib
|   |-- logs                    <== Apache error log is here
|   `-- modules
|-- bin                         <== Virtualenv scipts and binaries
|   |-- activate                <== Virtualenv activation script
|   |-- django-admin.py
|   |-- easy_install
|   |-- easy_install-2.7
|   |-- pip
|   |-- pip-2.7
|   |-- python -> python2.7
|   |-- python2 -> python2.7
|   `-- python2.7
|-- include
|-- lib
|   `-- python2.7
|       `-- site-packages       <== Virtualenv's Python packages directory
`-- test_django                 <== Your Django project directory
    |-- manage.py
    `-- test_django
        |-- __init__.py
        |-- settings.py
        |-- urls.py
        `-- wsgi.py             <== WSGI script file which Apache runs through mod_wsgi

Configure an Apache VirtualHost

We are now ready to configure Apache to serve our Django-powered webapp. In order to do this, we’ll need to modify the contents of the Apache configuration file located under apache2/conf/httpd.conf. Copy the original file to a backup for reference and make a note of the following values:

  • port number on which Apache listens to connections. This value is located in the line with the Listen directive of the original httpd.conf. In the example below we set this to 12345,
  • name of your application (test_app),
  • domain name which your website uses (example.com),
  • complete path to your application’s virtualenv and project directory: /home/my_username/webapps/test_app and /home/my_username/webapps/test_app/test_django,
  • complete path to your application’s WSGI script: /home/my_username/webapps/test_app/test_django/test_django/wsgi.py.

Use these values to customize the configuration template below and save it as your new httpd.conf:

1 import os
2 MEDIA_ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), '../../media/').replace('\\','/'))
3 STATIC_ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), '../../static/').replace('\\','/'))

Save the configuration to ~/webapps/test_app/apache2/conf/httpd.conf and restart Apache.

$ ./apache2/bin/restart

Visit your website again and you should be presented with Django congratulating you for setting your server up correctly.

Serving static and media files

The recommended way to serve static and media files on WebFaction is to use Nginx directly.

1   ServerRoot "/home/my_username/webapps/test_app/apache2"
2
3   LoadModule dir_module        modules/mod_dir.so
4   LoadModule env_module        modules/mod_env.so
5   LoadModule log_config_module modules/mod_log_config.so
6   LoadModule mime_module       modules/mod_mime.so
7   LoadModule rewrite_module    modules/mod_rewrite.so
8   LoadModule setenvif_module   modules/mod_setenvif.so
9   LoadModule wsgi_module       modules/mod_wsgi.so
10
11  KeepAlive Off
12  Listen 12345
13  MaxSpareThreads 3
14  MinSpareThreads 1
15  ServerLimit 1
16  SetEnvIf X-Forwarded-SSL on HTTPS=1
17  ThreadsPerChild 5
18
19  WSGIRestrictEmbedded On
20  WSGILazyInitialization On
21
22  NameVirtualHost *
23  <VirtualHost *>
24      ServerName example.com
25
26      # Logging configuration
27      LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
28      CustomLog /home/my_username/logs/user/access_test_app.log combined
29      ErrorLog /home/my_username/logs/user/error_test_app.log
30
31      # Django WSGI settings
32      WSGIDaemonProcess test_app processes=5 python-path=/home/my_username/webapps/test_app/test_django:/home/my_username/webapps/test_app/lib/python2.7/site-packages:/home/my_username/webapps/test_app/lib/python2.7 threads=1
33      WSGIProcessGroup test_app
34      WSGIScriptAlias / /home/my_username/webapps/test_app/test_django/test_django/wsgi.py
35  </VirtualHost>

Let’s begin by creating the directories for static and media files.

$ cd ~/webapps/test_app
$ mkdir media static

In order to tell Django where the files should be stored, we should place the appropriate lines in the project’s settings.py file. I like to keep the location of media and static folders relative to the source code project, so I would set them in this way:

1   import os
2   MEDIA_ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), '../../media/').replace('\\','/'))
3   STATIC_ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), '../../static/').replace('\\','/'))

Let’s collect the static files from all applications to the static directory:

$ cd ~/webapps/test_app
$ source bin/activate
(test_app) $ cd test_app
(test_app) $ python manage.py collecstatic

We can now serve our static files. In the WebFaction control panel, add two new applications named test_app_media and test_app_static. Both will be defined using these settings:

  • App category: Symbolic link
  • App type: Symbolic link to static-only app
  • Extra info: the path to the file folder, i.e. /home/my_username/webapps/test_app/media or /home/my_username/webapps/test_app/static

The final step is to add these Nginx-powered folders to our website definition. On the website settings screen for your domain, in the Contents section, choose to add an application. Choose the option to reuse an existing application and set the test_app_media to serve everything under http://example.com/media and test_app_static for http://example.com/static.

webfaction-website-content

Contents section of website settings

Separating development and production settings

You will want to use slightly different settings for your development and production environments. In order to separate them you can create three separate settings files:

  • settings.py – global settings, which apply to both environments
  • settings_dev.py – your development environment specific settings
  • settings_prod.py – production environment specific settings

The settings_prod.py file should only contain the settings which are specific to this environment, but also import all the global settings. We can do this by importing global settings like this:

1   from .settings import *
2
3   # Define production-specific settings
4   DEBUG = False
5   TEMPLATE_DEBUG = DEBUG
6
7   DATABASES = {
8   # ... production server database settings ...
9   }

Django checks the environment variable named DJANGO_SETTINGS_MODULE to determine which settings file to use. If this environment variable is undefined, it will fall back to test_app.settings.

In order to use your new settings module in the shell, we can add a line to the end of the script which activates our virtual environment (bin/activate).

export DJANGO_SETTINGS_MODULE=test_app.settings_prod

Apache and mod_wsgi don’t know about our new settings yet. We can set the DJANGO_SETTINGS_MODULE dynamically inside the wsgi.py script. Create a wsgi_prod.py script which will contain the following:

1   import os
2   os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_app.settings_prod")
3   from django.core.wsgi import get_wsgi_application
4   application = get_wsgi_application()

Now instruct Apache to use this WSGI script by setting the WSGIScriptAlias directive line to:

1   WSGIScriptAlias / /home/my_username/webapps/test_app/test_django/test_django/wsgi_prod.py

Restart Apache and your application should run with production settings applied.

02 Oct

python+virtualenv on Cygwin

I have been trying to make my default environment for development to be

  1. Install Python.
  2. Download and unzip pip.
  3. Install by going into the expanded directory and running python setup.py in a command prompt.
  4. Set the %PYTHONHOME% system variable to the python base directory, (i.e. C:\Python27\) and adding the python base directory and script directory (i.e. C:\Python27\Scripts) to your %PATH% system variable.
  5. Install Cygwin WITHOUT Python. The previous step tells Cygwin to use the Windows binary.
  6. Install Cygwin-Virtualenvwrapper using pip install https://bitbucket.org/cliffxuan/virtualenvwrapper-for-cygwin-windows-python/get/tip.tar.gz
  7. Install virtualenvwrapper-win using pip install virtualenvwrapper-win
  8. Make a symlink between Cygwin’s virtualenvhome directory and Windows’s using ln -s /cygdrive/c/Users/<USER>/Envs/ ~/.virtualenvs
  9. Add the following to Cygwin’s .bashrc file:
      export VIRTUALENVWRAPPER_PYTHON=”/cygdrive/c/Python27/python.exe”
      export VIRTUALENVWRAPPER_VIRTUALENV=”/cygdrive/c/Python27/Scripts/virtualenv.exe”
      source virtualenvwrapper.sh
  10. Go to C:\User\<username>\Env (or other %VIRTUALENV_HOME% location) and use virtualenv to start a new environment. Doing this allows virtualenvwrapper-win‘s workon command to work.
23 Jul

Deploying a Flask Project on WebFaction with Git [UPDATED]

I found it tedious having to redo the setup of my application on WebFaction whenever there is a new update of the application. I started looking for way to update the content of the application, without affecting the setup configurations of the application on WebFaction.
Step 1: Add your site to Your WebFaction Account
From your WebFaction console:
1.       Log into your WebFaction Console
2.       Go to your “Websites” tab, click the “Add New Website” button
3.      Enter the website name and domains as needed for your project
4.      Under the Contents section, choose Add an Application > Create a New Application
1.      Name to “myapp” 
2.      Set App Category to “mod_wsgi”
3.      App Type should be the most recent version of mod_wsgi that supports your Python version. Make sure that your Python version matches with your mod_wsgi choice ( “mod_wsgi 3.5/Python 2.7″ for this guide).
4.      Click the Save button to add that application
5.      Click the Save button to save your website
Now your WebFaction account should have a new domain, website, and mod_wsgi application.
Step 2: SSH into your new application
First, you’ll need to SSH into your webfaction account. Once you’re there, cd into your newly created application:
cd ~/webapps/myapp
In this directory, you’ll see two folders:
– apache2/      # This contains all of your Apache config and action files
– htdocs/       # This is the folder Apache to launch your project

Step 3: Upload your Flask project using Git

Upload the project/ folder to the application directory using Git. “myapp” directory is created

git clone https://username@bitbucket.org/uobis0/myapp.git

All following instructions will assume you’re still in this directory.

 virtualenv venv --python=python2.7
$ . venv/bin/activate
$ easy_install-2.7 flask    # Installs flask package for the app
$ pip install -r /myapp/app/requirements.txt  # Installs the packages for the app
$ deactivate

Now, my ~/webapps/myapp directory list looks like the following:

drwxr-xr-x 7 uobis uobis 4096 Nov 11 14:29 apache2/
drwxrwxr-x 5 uobis uobis 4096 Nov 11 14:40 venv/
drwxrwxr-x 6 uobis uobis 4096 Nov 14 16:46 myapp/
drwxr-xr-x 2 uobis uobis 4096 Nov 11 14:29 htdocs/
-rw-rw-r-- 1 uobis uobis 292 Nov 18 15:14 wsgi.py

Step 4: Edit apache2/conf/httpd.conf

Using your favorite command line editor, open up the apache2/conf/httpd.conf file:

vim ~/webapps/myapp/apache2/conf/httpd.conf

Load Alias module (optional)

You’ll see a section where Apache Modules are being loaded. I had to manually add the Alias module to the bottom of the list (shown below).

.
.
LoadModule wsgi_module       modules/mod_wsgi.so
LoadModule alias_module      modules/mod_alias.so    #Your version of mod_wsgi might not need to add this.

Modify Alias and <Directory>

Add the following parameters to your <Directory> section:

WSGIScriptAlias / /home/username/webapps/myapp/wsgi.py

<Directory /home/username/webapps/myapp/htdocs>
    AddHandler wsgi-script .py
    RewriteEngine on
    RewriteBase /
    WSGIScriptReloading On
</Directory>

Now for the final edit of the config file.

Step 5: Make sure your main file is right 

Ensure that, if your structure is package, your project’s __init__.py file is launching your Flask application. This is what the file at ~/webapps/myapp/app/__init__.py should look like:

from flask import Flask

# Setting up the App
app = Flask(__name__)
app.config.from_object('config')

# Importing the views for the rest of our site
from app import views

if __name__ == '__main__':
    app.run()

If you are using modules, the files should contain the app instantiation

app = Flask(__name__)

This will work nicely with our wsgi.py file, which we’ll set up next.

Step 6: Modify the htdocs/wsgi.py file

WebFaction should have created this file. In it are a few scripts, but you can completely remove those. Here is what the ~/webapps/myapp/wsgi.py file should contain:

import sys

# Active your Virtual Environment, which I'm assuming you've already setup
activate_this='/home/username/webapps/myapp/venv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

# Appending our Flask project files
sys.path.append('/home/username/webapps/myapp/myapp')

# Launching our app
from main import app as application

 

Step 7: Restart Apache

The last step will be to restart apache, like so:
~/webapps/myapp/apache2/bin/restart

 

Troubleshooting

If you’re having trouble, take a look at your logs in the ~/logs/users/ directory.

Application Updates

The steps above makes updates very easy:
cd ~/webapps/myapp/myapp
git pull https://username@bitbucket.org/accout/myapp.git

~/webapps/myapp/apache2/bin/restart

30 Nov

Deploying Flask on lighttpd

Deploying Flask seems a lonely taks, becuase of dearth of articles or blogs that explain in. This is even worse when deploying on a shared server.
Most people seems to treat a python deployment as a django deployment. Since, there are cheap and available Flask hsting site, i went with Django host, but confirmed that we can host any other framework.

The good thing is that, there too many similarities, and most packages are already installed on the host.

1. Open putty and log into the hosts server (s17.wservices.ch)
2. Check that the following following packages are installed

python and python-devel: the Python interpreter and its development package
lighttpd: The Lighty web server and its development package
install postgresql postgresql-contrib: The PostgreSQL database server and its development package
git: source code version control system (we will use it to download and update the application)
gcc: the C/C++ compiler (needed to compile Python extensions)
sudo: a tool that helps users run commands as other users.

If not, install them.

sudo apt-get python python-devel lighttpd httpd-devel mysql-server mysql-devel git gcc

3. Configure passwordless login (if needed)
4. Create a directory for the application, and install the application from BitBucket
mkdir app
cd app
git clone git://bitbucket.org/peppe/peppe-ng.git

5. Check User permissions
chmod -R 777 *

6. Setup the database
Go to https://panel.djangoeurope.com/databases/
Login with a username and password, and then create a database and dump the content of the dev db
pg_dump peppedb > ppdbdump.sql

Transfer the file to live host
sftp outfile.sql peppe@s17.wservices.ch

Load the Db on the live host
psql peppedb < ppdbdump.sql

7. Setup the webserver
Paste at the end of this file ~/lighttpd/lighttpd.conf

#Peppe.com
$HTTP[“host”] =~ “(^|.)peppe.com.ng$” {
fastcgi.server = (
“/flask.fcgi” => (
“main” => (
“socket” => env.HOME + “/mysite_project/mysite.sock”,
“check-local” => “disable”,
)
),
)
alias.url = (
“/media” => env.HOME + “/mysite_project/media”,
)

url.rewrite-once = (
“^(/media.*)$” => “$1”,
“^/favicon.ico$” => “/media/favicon.ico”,
“^(/.*)$” => “/flask.fcgi$1”,
)
}

Replace mydomain.com with the name of your domain. Be sure to escape all dots of your website’s name (put a backslash before it: .). Replace your_django_project/media with the path of you media directory (relative to your home directory). Replace mysite_project/mysite.sock with the path to your fastcgi socket file

Now you can launch your lighttpd:

~/init/lighttpd start

Whenever you make changes to the configuration, you can reload the configuration or restart lighttpd:

~/init/lighttpd reload

~/init/lighttpd restart

9. Configure the db and start
10. Install application updates