Table of contents
- Introduction
- Requirements
- C Wrapper
- Compile: Ninja Module
- Compile: Makefile
- Compile: inplace
- Python test
- Run tests
- TDD without hazards
1. Introduction
Since the rise of the popularity of TDD in app development we found several attempts of TDD frameworks to use in an old language as C. It’s true that there are a lot of alternatives to C as a programming language but there still are a lot of applications and Frameworks that use C language like for example our Web Framework at Segundamano.es, called Blocket. TDD in C is not really easy and always is not possible to implement and maintain. That’s why I would like to introduce a method that could help you to use TDD in your daily work without the pains of C Test Frameworks; using the popular Python programming language instead.
The main idea is to create unit tests using the build-in test platform from Python. To do this we need to include our C source code inside Python. Since Python allow us to include C Source Code as an External Library we are going to need to do some things to test our code.
To include our source code into Python we need to create a C Wrapper that will talk between our C module to test and the Python test code. The scheme is something like this:
There are other alternatives to do this like CTypes or Cython; that It makes writing C extensions for Python as easy as Python itself. But for now Python will do the trick.
2. Requirements:
Before starting, you need to assure that you have these dependencies installed on your system.
- python
- python-devel (we need Python.h)
- numpy
- gcc (or other C compiler of your choice that allow dynamic libraries)
If you are using CentOS you should use yum to install all the dependencies.
yum install python-devel
yum install numpy
3. C Wrapper
To allow Python import your C modules you need to create a C wrapper that will communicate between your C source code and the Python test suite.
There is no better way to show how it works like using a real example. Imagine that you have C module called “sgm_redis.c” that manages “Redis” and all their calls. And you have a method called “createUserInRedis” that you would like to cover with a test before doing TDD properly. We need to prepare the system before starting with TDD and creating some covering test (scaffolding) will be a good start.
To do that you will need to create two files. (modules/pythonTests/_python_tests.c and _python_tests.h) The filename with underscore is important because the python parser need to know who are definitions and methods inside.
The first thing is to include Python.h which will allow the connection between objects, then we create a method definition called “python_tests_createUserInRedis”. The first part of the name is the module name and the second the name of the method we want link. Then we create a module definition array called “module_methods” (PyMethodDef) that will contain all the methods that will be public through the wrapper and will include our method “createUserInRedis”.
After that we implement the method “python_tests_createUserInRedis” with a method call to the real “createUserInRedis” of the C module imported. Now we got all we need to call this method in Python.
Additionally we need to include a function called “init_python_tests” to allow python initialize this external module also we need the “main” functions to bypass errors with compiler.
Note that PyArg_ParseTuple need a special parameter depending on the type incoming argument passed to the function; the same for Py_BuildValue. For example, “O” param is for Objects, used to parse a Python Structure into C Struct. And the “c” param for the output means that will return a char.
The code will be something like this:
_python_tests.c
#include <Python.h>
#include <numpy/arrayobject.h>
#include "_python_tests.h"
#include "sgm_redis.h"
static char module_docstring[] =
"This module provides an interface for testing Redis Methods.";
static char python_tests_docstring[] =
"Tanslate Redis Reply Type.";
static PyObject *python_tests_createUserInRedis(PyObject *self, PyObject *args);
static PyObject *python_tests_generateActionToken(PyObject *self, PyObject *args);
static PyMethodDef module_methods[] = {
{"createUserInRedis", python_tests_createUserInRedis, METH_VARARGS, python_tests_docstring},
{"generateActionToken", python_tests_generateActionToken, METH_VARARGS, python_tests_docstring},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC init_python_tests(void)
{
PyObject *m = Py_InitModule3("_python_tests", module_methods, module_docstring);
if (m == NULL)
return;
import_array();
}
static PyObject* python_tests_createUserInRedis(PyObject *self, PyObject *args)
{
struct bconf_node *bconf_root;
struct hash_table *data;
if (!PyArg_ParseTuple(args, "OO", &bconf_root, &data))
return NULL;
int s = createUserInRedis(bconf_root, data);
return Py_BuildValue("i",s);
}
static PyObject* python_tests_generateActionToken(PyObject *self, PyObject *args)
{
char *s = generateActionToken();
return Py_BuildValue("c",s);
}
int main()
{
return 0;
}
int init__main__()
{
return 0;
}
|
_python_tests.h
#include <Python.h>
void init_python_tests();
int init__main__();
|
4. Compile: Ninja Module
After making the C Wrapper we would need to compile the module and generate a ".so" that will be imported into Python. To do this there are some ways, in our case we use Ninja as a module dependency build system. If you are using Ninja you should create a MODULE to tests independent. The objective of doing this way is make easy to include all the dependencies to the test suite and avoid compiling it outside the system. So you could launch the tests using two bash commands.
Follow these steps:
- Create a new folder under “modules/” with the name you wish. In this case “pythonTests” (modules/pythonTests)
- Create a new Builddesc file with this config: (modules/pythonTests/Builddesc)
MODULE(_python_tests
srcs[_python_tests.c]
includes[_python_tests.h]
libs[platform_core sgmutil]
copts[$python_module_cflags -Wno-unused-function -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c -o build/temp.linux-x86_64-2.6/_python_tests.o]
)
|
Where the first parameter (“_python_tests”) is for the name of the python module that will be imported into python. It should have the underscore because the Python.h parser needed to decide which methods will be accessible to the lib.
The second and third params (“_python_tests.c/_python_tests.h) are references to the C wrapper. In the “libs” we will include all the libraries that will be tested in this case is one called “sgmutil” that includes the "sgm_redis.c" module. Finally we include all the options and flags that need the compiler to create the python module correctly.
5. Compile: Makefile
It is possible to create the module using the commonly used “makefile”. You need to assure that the .c/.h for the wrapper are included and have all the options and flags like the Ninja version.
6. Compile: inplace
As an alternative you could compile the module using the setuptools for python. The main problem is that you will need to include all the source files that need to be tested manually.
First you need to create a setup.py (modules/pythonTests/setup.py) file with this config:
from distutils.core import setup, Extension
import numpy.distutils.misc_util
setup(
ext_modules=[Extension("_python_tests", ["_python_tests.c"])],
include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs(),
)
|
To compile it you need to execute this command in your bash terminal:
python setup.py build_ext --inplace
This will create a .so file that you could import in python:
./python
import _python_tests
and you could install it in your system using:
./python setup.py install --user
7. Python Tests
Finally after we got the C Wrapper Module compiled we can import it into python. So we can start our unit testing. Create a new python file for include the tests, in this case “tests.py” (modules/pythonTests/tests.py)
The code would be like this:
import pkg_resources, imp
import unittest
import sys, os
from struct import *
from ctypes import *
import ctypes
sys.path.append(os.getcwd() + '/build/regress/modules/')
def loadTestWrapper():
global __loader__, __file__
print "\n*** loading _python_tests\n"
__file__ = pkg_resources.resource_filename('_python_tests','_python_tests.so')
__loader__ = None; del __loader__
imp.load_dynamic(__name__,__file__)
class MyTestCase(unittest.TestCase):
class hash_table(Structure):
_fields_ = [
("user_id", c_char),
("token", c_char),
]
class bconf_node(Structure):
_fields_ = [
("value", c_char_p),
("keylen", c_size_t),
("key", c_char_p),
("type", c_int),
("sublen", c_int),
("count", c_int)
]
def setUp(self):
self.createUserInRedis = ""
def tearDown(self):
self.createUserInRedis = ""
def testCreateUserInRedis(self):
import _python_tests
bconf_root = self.bconf_node('v', 1, 'k', 1, 1, 1)
data = self.hash_table('a','a')
self.createUserInRedis = _python_tests.createUserInRedis(bconf_root, data)
print "\n*** createUserInRedis: " + str(self.createUserInRedis)
self.assertTrue(self.createUserInRedis == 1)
def testGenerateActionToken(self):
import _python_tests
self.actionToken = _python_tests.generateActionToken()
print "\n*** generateActionToken: " + self.actionToken
self.assertTrue(self.actionToken)
def main():
loadTestWrapper()
unittest.main()
if __name__ == '__main__':
main()
|
Note that the line that says “sys.path.append(os.getcwd() + '/build/regress/modules/')” makes reference to where the ".so" file is placed. You should change it with the correct path, if not when you execute the "tests.py" will throw an error.
The hash_table and bconf_node classes make reference to a C struct in Python and are the object that are passed through the “createUserInRedis” method.
8. Run tests
After you complete all these steps, you can run the tests using the python bash command like:
./python modules/pythonTests/tests.py
And you should view an output like this if tests pass:
.
----------------------------------------------------------------------
Ran 2 tests in 0.002s
OK
|
9. TDD without hazards
By now, you should have this test system up and running. It’s time to make some TDD!
Maybe not everybody will be happy with this workflow but there are some benefits. The pros are that test will be faster and easy to launch and maintain plus all the benefits of programming in python plus all the test tools that python offer. The only downside is that with this method you are going to call every method two times in two different places. But is a good way to have all your code clean.
References
1. Extending Python with C or C ¶. (n.d.). Retrieved May 19, 2014, from https://docs.python.org/2.6/extending/extending.html
Chapter 15. C Extensions. (n.d.). Retrieved May 19, 2014, from http://chimera.labs.oreilly.com/books/1230000000393/ch15.html
Parsing arguments and building values¶. (n.d.). Retrieved May 19, 2014, from https://docs.python.org/2/c-api/arg.html#PyArg_Parse
Pass a C Structure pointer from Python using Ctypes. (n.d.). Retrieved May 19, 2014, from http://stackoverflow.com/questions/21022387/pass-a-c-structure-pointer-from-python-using-ctypes
