REST Api with Flask-Restful and MongoDB

What is REST

REST (Representational State Transfer) is an architecture style for designing networked applications. The idea is using simple HTTP methods for making calls between machines rather than complex mechanisms such as CORBA, RPC or SOAP. 

So a RESTful system typically communicates over the Hypertext Transfer Protocol with the same HTTP verbs (GET, POST, PUT, DELETE, etc.)

Flask and Flask-RESTful

Flask is a microframework for Python based on Werkzeug, Jinja2. Flask-RESTful provides an extension to Flask for building REST APIs.

Installing packages

For the api, we need to install some packages like Flask-Restful, pymongo etc. It's good practice to install packages and modules for a project in a virtual environment. So i am going to create a virtual environment and activate it.

$ virtualenv env

$ source env/bin/activate

Now the required packages can be installed using pip. (Link: requirements.txt)

$ pip install -r requirements.txt

Api

Lets Code the API first.

	# -*- coding: utf-8 -*-
	from flask import Flask, jsonify, url_for, redirect, request
	from flask_pymongo import PyMongo
	from flask_restful import Api, Resource
	app = Flask(__name__)
	app.config["MONGO_DBNAME"] = "students_db"
	mongo = PyMongo(app, config_prefix='MONGO')
	APP_URL = "http://127.0.0.1:5000"
	class Student(Resource):
	def get(self, registration=None, department=None):
	data = []
	if registration:
	studnet_info = mongo.db.student.find_one({"registration": registration}, {"_id": 0})
	if studnet_info:
	return jsonify({"status": "ok", "data": studnet_info})
	else:
	return {"response": "no student found for {}".format(registration)}
	elif department:
	cursor = mongo.db.student.find({"department": department}, {"_id": 0}).limit(10)
	for student in cursor:
	student['url'] = APP_URL + url_for('students') + "/" + student.get('registration')
	data.append(student)
	return jsonify({"department": department, "response": data})
	else:
	cursor = mongo.db.student.find({}, {"_id": 0, "update_time": 0}).limit(10)
	for student in cursor:
	print student
	student['url'] = APP_URL + url_for('students') + "/" + student.get('registration')
	data.append(student)
	return jsonify({"response": data})
	def post(self):
	data = request.get_json()
	if not data:
	data = {"response": "ERROR"}
	return jsonify(data)
	else:
	registration = data.get('registration')
	if registration:
	if mongo.db.student.find_one({"registration": registration}):
	return {"response": "student already exists."}
	else:
	mongo.db.student.insert(data)
	else:
	return {"response": "registration number missing"}
	return redirect(url_for("students"))
	def put(self, registration):
	data = request.get_json()
	mongo.db.student.update({'registration': registration}, {'$set': data})
	return redirect(url_for("students"))
	def delete(self, registration):
	mongo.db.student.remove({'registration': registration})
	return redirect(url_for("students"))
	class Index(Resource):
	def get(self):
	return redirect(url_for("students"))
	api = Api(app)
	api.add_resource(Index, "/", endpoint="index")
	api.add_resource(Student, "/api", endpoint="students")
	api.add_resource(Student, "/api/<string:registration>", endpoint="registration")
	api.add_resource(Student, "/api/department/<string:department>", endpoint="department")
	if __name__ == "__main__":
	app.run(debug=True)

view raw api.py hosted with ❤ by GitHub

This is a basic api with only one Resource Student. First we created an instance of Flask,
app = Flask(__name__)
after that we configured app with MongoDB.

The resource Student has HTTP methods defined in it as it's own method. By calling different HTTP methods we can perform basic CRUD operations. Like post for inserting/creating, put for updating, get for reading and delete for deleting a document.

In the code post, put, delete methods are basic.  These methods perform create, update and delete operation on the collection student. 

But in the get method, by calling different url we can perform different operations. If no parameter passed with url, the method will return all the documents in the collection. If a registration number passed with the url then, it will check the parameter name and will return one document. If request is made in /api/department/<string:department> url, it will check for parameter name department in the get method and returns all the documents for a particular department.

There are several urls for same resource so we added different endpoints for them.

Testing the Api

We will use python requests module (version > 2.4.2) for testing the api. But also curl and other mechanism can be used. For get request we can view the data in browsers also.

	In [1]: from requests import get, post, put, delete
	In [2]: get("http://127.0.0.1:5000/api")
	In [3]: data = {"name": "Example Name", "registration": "123433199", "department": "cse"}
	In [4]: post("http://127.0.0.1:5000/api", json=data).json()
	In [5]: put("http://127.0.0.1:5000/api/123433199", json={"website": "www.example.com"}).json()
	In [6]: delete("http://127.0.0.1:5000/api/123433199").json()

view raw apitest.py hosted with ❤ by GitHub

requests module gives us convenient way to test our api. Enough REST. Lets take some rest ;)

PYTHONPATH and importing modules in python

What is PYTHONPATH

PYTHONPATH is an environment variable that defines additional locations of Python modules. This is the default search path for module files. This means that python interpreter will search for modules in the location of the PYTHONPATH.

How importing works in python

When a module named mymodule is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named mymodule.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

• The directory containing the input script (or the current directory).
• PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
• the installation-dependent default.

problem scenario and solve:

A

|--__init__.py

|

|--B

|--|--__init__.py

|--|--greetings.py

|

|--C

|--|--__init_.py

|--|--mymodule.py

Here A is a python package. Inside A, there are two packages B and C. 

greetings.py

def my_function():
    print "Congrats Bangladesh Cricket Team!! :D"

mymodule.py

from B.greetings import my_function
my_function()

Now, there is a problem. Package A  is not in python interpreter's search path. So it can not import greetings function from B  within C. If mymodule.py is run, it will raise an ImportError as expected.

$ python mymodule.py

Traceback (most recent call last):

  File "mymodule.py", line 3, in <module>

    from B.greetings import my_function

ImportError: No module named A.B.greetings

To solve it, we need to add the absolute path of package A in sys.path inside mymodule.py before importing greetings from B. Path will be appended in sys.path. This will not affect other python program because the path will not be modified for any other Python processes. 

mymodule.py

import os
import sys

PATH = os.path.abspath(os.path.join(os.pardir))
sys.path.append(PATH)

from B.greetings import my_function
my_function()

Now as the path of A is added, Python interpreter can find the package B inside A. Let's run again this function

$ python mymodule.py

Congrats Bangladesh Cricket Team!! :D

MongoDB model with PyMongo

MongoDB

MongoDB is a popular open-source NoSQL database. NoSQL is a database technology to store and retrieve data which are not stored in database like the traditional tabular form of RDBMS. Scalability and superior performance are main beneifts of NoSQL databases over RDBMS. Mongodb uses JSON-style documents with dynamic schema to store data. Installation instructions of MongoDB will be found in it's documentation - Install MongoDB. 

PyMongo

PyMongo is a Python distribution containing tools for working with MongoDB, and is the recommended way to work with MongoDB from Python. To install PyMongo execute the command in terminal:

$ sudo pip install pymongo

Creating a model

I am going to create student model which will be used to  get, create, update, delete objects form database.

models.py

# -*- coding: utf-8 -*-
import pymongo


class Student(object):
    def __init__(self, registration, **kwargs):
        self._id = registration
        self.first_name = kwargs.get('first_name', '')
        self.last_name = kwargs.get('last_name', '')
        self.department = kwargs.get('department', '')
        self.session = kwargs.get('session', '')
        self.mobile = kwargs.get('mobile', '')

    def __str__(self):
        return self._id

    @staticmethod
    def get_db(dbname='student_db'):
        """
        make database connection and return a database instance
        """
        client = pymongo.MongoClient(host='localhost', port=27017)
        return client[dbname]

    @classmethod
    def get(cls, registration):
        """
        get student by registration number
        """
        db = Student.get_db()
        student_info = db.students.find_one({'_id': registration}, {'_id': 0})
        if student_info:
            student = Student(registration=registration, **student_info)
        else:
            student = Student(registration=registration)
        db.connection.close()
        return student

    @classmethod
    def get_all(cls):
        """
        get all students and return a list. for many student return a cursor instead.
        """
        db = Student.get_db()
        students = list(db.students.find())
        db.connection.close()
        return students

    def save(self):
        """
        save() method will create or update a document based on the _id attribute.
        """
        db = Student.get_db()
        db.students.save(self.__dict__)
        db.connection.close()

    @classmethod
    def remove_all(cls):
        """
        remove all documents for the collection.
        """
        db = Student.get_db()
        db.students.remove()
        students = list(db.students.find())
        db.connection.close()
        return students

    def remove(self):
        """
        remove a object from the collection.
        """
        db = Student.get_db()
        db.students.remove(self._id)
        db.connection.close()
        return self._id

Now this model can be used for CRUD operations.

>>>from models import Student
>>>student = Student.get('2008331041')
>>>student.first_name = "Salman"
>>>student.last_name = "Wahed"

>>>student.save()
>>>Student.get_all()

[{u'_id': u'2008331041',
  u'department': u'',
  u'first_name': u'Salman',
  u'last_name': u'Wahed',
  u'mobile': u'',
  u'session': u''}]

Unit testing in python using unittest framework

What is Unit test

Unit test is a process of testing an unit of testable code like functions, classes, procedures etc of an application using test code. Mainly programmers write unit tests for their codes while developing software. Unit testing reflects the programmers view on how things should work. This is the main theme of TDD (Test-driven development).

Unit test in python

In python there are some frameworks to do unit testing  like unittest, nose, pytest etc. Among these, unittest module comes with default python installation.

I have written a function which will take an integer value as argument and return if that integer is even or odd.

pytesting/even_odd.py

# -*-coding: utf-8-*-
def my_even_odd(number):
    if isinstance(number, int):
        if number % 2 == 0:
            return "Even"
        return "Odd"    
    return "Integer required"

This function will return "Integer required" if it gets an argument other than int. And for each argument it should return a string. Here, I am going to test two things: one is if I pass an integer or a non integer argument what it will return and  two, if it returns anything other than string.

pytesting/unittest_even_odd.py

# -*-coding: utf-8-*-
import unittest
from even_odd import my_even_odd

test_values = [
    (1, "Odd"),    (0, "Even"),    (2147483648, "Integer required"),    (2147483647, "Odd"),    (-2147483649, "Integer required"),    (-2147483648, "Even"),    ('', "Integer required"),    ({'a': 1}, "Integer required"),    ([2, 3, 5], "Integer required"),]


class TestMyEvenOdd(unittest.TestCase):
    def test_my_even_odd(self):
        for tpl in test_values:
            self.assertEqual(my_even_odd(tpl[0]), tpl[1], msg="Failed for {}".format(tpl[0]))
            self.assertTrue(isinstance(my_even_odd(tpl[0]), str))


if __name__ == '__main__':
    unittest.main()

In unittest framework to create a test we need to create a subclass of unittest.TestCase class and add or override appropriate methods. In this case I added a method test_my_even_odd. There are two main parts. One is the test class we have written and another is test fixture. A test fixture represents the preparation needed to perform one or more tests, and any associate cleanup actions. This may involve, for example, creating temporary or proxy databases, directories, or starting a server process etc. In our case we keep it simple by using a list test_values. assertEqual and and assertTrue methods of  a TestCase class.

assertEqual(self, first, second, msg=None)

This method will raise an AssertionError and test will fail if first and second objects are unequal as determined by `==`

assertTrue(self, expr, msg=None)

This method will check if the expression expr is True and raises AssertionError if False.

Now if we run unittest_even_odd.py we will get output like this,

$ python unittest_even_odd.py

.
--------------------------------------------------------------------
Ran 1 test in 0.000s

OK

That means my little function passed the test.

Vagrant up and running on ubuntu trusty

What is vagrant?

Vagrant is a software for creating and configuring complete development environment. The environment is sandboxed in a virtual machine. The virtual machine is create by a virtualization software such as VirtualBox, VMware.

Vagrant works as a wrapper around a virtualization software and around configuration management software such as Ansible, Chef, Salt or Puppet.

Why and when use vagrant?

In software development process it's a common scenario that some software works on one developers machine and simply does not work in some other programmers machine for some dependency or some other issue. Vagrant is the answer here. It will create an identical development environment for everyone who intend to run/develop the software. Also running provisioning software like puppet, chef or ansible is a nice feature to have.

Installation

Vagrant needs virtualbox. Virtualbox is a virtualization software by oracle and its free. It can be downloaded from here. After downloading .deb file for specific ubuntu version, it can be installed by this command in the shell.

$ sudo dpkg -i <location/of/the/file>

Next step is installing vagrant. .deb file of it can be downloaded from here. Installation process is same as the virtualbox. Once vagrant get installed it will seem like noting has happened but we get some command line options for vagrant.

Up and running

I am going to create a virtual environment of a ubuntu trusty 32 bit system in my ubuntu trusty computer. Reason of choosing 32 bit system is, it takes less RAM than its 64 bit counterpart in the host machine. The host machine is my personal computer and the guest machine is the virtual machine I am going to create.

Create a directory trusty32 and cd to that directory from terminal. Directory name can be anything.

$ mkdir trusty32

$ cd trusty32

Now we add vagrant box. A box is a base image to quickly clone a virtual machine. The easiest way to download boxes is downloading it from HashiCorp's Atlas using this command,

$ vagrant box add ubuntu/trusty32

Another way is to add a box from vagrantbox.es site. In this case we follow the site's instruction and add the box like this,

$ vagrant box add trusty32 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-i386-vagrant-disk1.box

vagrant box add command will download the specified box. This will take some time. Once downloaded it can be reused in multiple projects.

The next command is,

$ vagrant init trusty32

The command above will place a Vagrantfile in trusty32 directory. Vagrantfile describes type of machine required for a project, and how to configure and provision these machines. For now it will be not changed.

To boot up our virtual machine,

$ vagrant up

Above command creates and configures guest machines according to our Vagrantfile. To check if a guest machine is running,

$ vagrant status

We can ssh to the machine and gain access to the guest machine shell,

$ vagrant ssh

To shut down a running machine we have to exit from the ssh session  and initiate the command,

$ vagrant halt

If we want to destroy the virtual machine and it's resource,

$ vagrant destroy

Python map, filter, lambda explained with example

map(function, sequence[, sequence, ...]) -> list

map(function, iterable, ...) is a python builtin function.

• It returns a list of results applying the function to the items of the argument sequence(iterable) or sequences.
• If more than one iterable arguments are passed, the function must take that many arguments. The function is called with an argument list consisting of the corresponding item of each sequence.
• If one iterable is shorter than another it is assumed to be extended with None items.

We can examine these rule with some codes.

def square(a):
    return a ** 2

Here  square is a function that makes a number square. We want the square of number 0 to 5. Using map(),

>>> map(square, range(6))

[0, 1, 4, 9, 16, 25]

So map() takes a function square and a list as arguments and returns a list of square(item).  Here item is elements in the list range(6).

If we pass two iterable in map(), the function must take two arguments.

def my_pow(a, n):
    return a ** n

Now my_pow function takes two arguments a, n and returns aⁿ.

>>> map(my_pow, [2, 3], [3, 2])
[8, 9]

But what if the two iterables are not of same length? In this particular case it will  generate a TypeError.

filter(function or None, sequence) -> list, tuple, or string

filter(function, iterable) is a python builtin function.

• It returns those items of sequence for which function(item) is True.
• If function is None, return the items that are True.
• If sequence is a tuple or string, return the same type, else return a list.

We want to filter out the even integer from a list and we have written a function for this.

def is_odd(n):
    if n % 2 == 1:
        return True    
    return False

is_odd  function takes an integer and returns True if it's odd number otherwise returns False.

>>> filter(is_odd, range(10))
[1, 3, 5, 7, 9]

What happens here is, filter() applies the is_odd function on each item of the list range(10) and returns a list of those items for which is_odd returned True. It filters out the other items.


>>> filter(None, k = [True, False, True, False])
[True, True]

If the function is None it return the items that are True

lmabda function:

Now, fun begins with lambda functions. in Python,

• lambda operator is a way to create small anonymous function. An anonymous function is a function without a name.
• lambda function can be used wherever a normal function is used but it's restricted to a single expression.
• function require multiple expression can not be expressed using lambda.

lambda functions are written as:

lambda arguments: expression (operation to perform on arguments)

lambda function returns the result of the operation.

What we have done in map() and filter() examples is, separately wrote a function and pass those in map(), filter()

We can do the same using lambda function.

>>>  map(lambda n: n**2, range(6))
[0, 1, 4, 9, 16, 25]

what happens here is same in the map() example but in this case a lambda function is used.

lambda n: n**2

On the left of : are the arguments. Here n is the argument and on right side is the expression that will be applied to the argument. The return value is the result.

>>> map(lambda a, n: a**n, [2, 3], [3, 2]) 
[8, 9]

This is equivalent to the my_pow() example.

Suppose we want to get all the digits from a string. According to our filter() and lambda knowledge,

>>> filter(lambda s: s.isdigit(), "adf9df215")
'9215'

We can write a little complex expression in lambda.

>>> filter(lambda s: False if s.isspace() else True, "This is SPARTAAAAAAA")
'ThisisSPARTAAAAAAA'

This filters out all the space from the string. Hope king  Leonidas don't mind.