Merge pull request #6 from dputtick/dev/test

Converting tests to pytest

.gitignore

@@ -65,3 +65,10 @@ target/
 *.snm
 *.toc
 *.vrb
+
+# Project specific
+/tests/dst/*
+!/tests/logs/
+!/tests/.keepdir
+
+

.travis.yml

@@ -48,40 +48,41 @@ install:
       if [[ "$TRAVIS_PYTHON_VERSION" == "2.7_with_system_site_packages" ]]; then
         sudo pip install -U pip lxml exifread pillow
         sudo pip install -U git+https://github.com/Rafiot/officedissector.git
-        sudo pip install -U oletools olefile coveralls codecov
+        sudo pip install -U oletools olefile coveralls codecov pytest-cov
       else
         pip install -U pip lxml exifread pillow
         pip install -U git+https://github.com/Rafiot/officedissector.git
-        pip install -U coveralls codecov
+        pip install -U coveralls codecov pytest-cov
       fi
-    - python setup.py -q install
+    # Module dependencies
+    - pip install -r dev-requirements.txt
     # Testing dependencies
     - sudo apt-get install rar
-    # Prepare testings
+    # Prepare tests
     # Zoo
     - git clone https://github.com/Rafiot/theZoo.git
     - pushd theZoo/malwares/Binaries
     - python unpackall.py
     - popd
-    - mv theZoo/malwares/Binaries/out tests/src/
-    # path transversal
-    - hg clone https://bitbucket.org/jwilk/path-traversal-samples
-    - pushd path-traversal-samples
-    - pushd zip
-    - make
-    - popd
-    - pushd rar
-    - make
-    - popd
-    - popd
-    - mv path-traversal-samples/zip/*.zip tests/src/
-    - mv path-traversal-samples/rar/*.rar tests/src/
+    - mv theZoo/malwares/Binaries/out tests/src_complex/
+    # path traversal
+    # - hg clone https://bitbucket.org/jwilk/path-traversal-samples
+    # - pushd path-traversal-samples
+    # - pushd zip
+    # - make
+    # - popd
+    # - pushd rar
+    # - make
+    # - popd
+    # - popd
+    # - mv path-traversal-samples/zip/*.zip tests/src_complex/
+    # - mv path-traversal-samples/rar/*.rar tests/src_complex/
     # Office docs
     - git clone https://github.com/eea/odfpy.git
-    - mv odfpy/tests/examples/* tests/src/
-    - pushd tests/src/
+    - mv odfpy/tests/examples/* tests/src_complex/
+    - pushd tests/src_complex/
     - wget https://bitbucket.org/decalage/olefileio_pl/raw/3073963b640935134ed0da34906fea8e506460be/Tests/images/test-ole-file.doc
-    - wget http://www.officedissector.com/corpus/fraunhoferlibrary.zip
+    - wget --no-check-certificate https://www.officedissector.com/corpus/fraunhoferlibrary.zip
     - unzip -o fraunhoferlibrary.zip
     - rm fraunhoferlibrary.zip
     - 7z x 42.zip -p42
@@ -92,7 +93,7 @@ install:
     - popd
 
 script:
-    - travis_wait 60 coverage run --source=bin,kittengroomer setup.py test
+    - travis_wait 60 py.test --cov=kittengroomer --cov=bin tests/
 
 notifications:
     email:

CHANGELOG

@@ -0,0 +1,9 @@
+Changelog
+=========
+
+2.1.0
+---
+
+New features:
+
+Fixes:

CONTRIBUTING.md

@@ -0,0 +1,33 @@
+The issue tracker
+=================
+
+If you find a bug or see a problem with PyCIRCLean, please open an issue in the Github
+repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a solution
+to any of the open issues - we'll do our best to review your pull request in a timely manner.
+This project is in active development, so any contributions are welcome!
+
+
+Setting up a dev environment
+============================
+
+* First, you'll want to get a local copy of PyCIRCLean. If you'd like to make a pull request
+with your changes at some point, you should fork the project on github, and then `git clone`
+your fork.
+
+* To install the project's dependencies, you can run `python setup.py install`. Alternatively,
+you can use `pip install dev-requirements.txt` to ensure you download any testing dependencies as well.
+We recommend that you use a virtualenv when installing dependencies. Note: python-magic has a non-Python
+dependency, libmagic. It is typically included in Linux distributions, but you might have to install
+it with homebrew (`brew install libmagic`) on macOS.
+
+* Some of the example scripts have additional dependencies for handling various filetypes. You'll have to
+install these seperately if you want to try out the examples or modify them for your own purposes.
+Please open an issue if you have suggestions of good alternatives for the libraries we use for file handling
+or if you have an example you'd like to contribute.
+
+
+Running the tests
+=================
+
+* Running the tests is easy. First, make sure you've installed the project and testing dependencies.
+Then, run `python -m pytest` or just `pytest` in the top level or /tests directory.

MANIFEST.in

@@ -1 +1 @@
-include kittengroomer/data/*
+include kittengroomer/data/* README.md CONTRIBUTING.md CHANGELOG dev-requirements.txt

README.md

@@ -4,23 +4,32 @@
 
 # PyCIRCLean
 
-PyCIRCLean is the Python code used by [CIRCLean](https://www.circl.lu/projects/CIRCLean/), the USB key and document sanitizer. The code
-has been separated from the devices as PyCIRCLean software can be used for dedicated security applications to sanitize documents
-from hostile environments to trusted environments.
+PyCIRCLean is the core Python code used by [CIRCLean](https://github.com/CIRCL/Circlean/), an open-source
+USB key and document sanitizer created by [CIRCL](https://www.circl.lu/). This module has been separated from the 
+device-specific scripts and can be used for dedicated security applications to sanitize documents from hostile environments 
+to trusted environments.
 
 # Installation
 
 ~~~
-python setup.py build
 python setup.py install
 ~~~
 
+OR
+
+~~~
+pip install .
+~~~
+
 # How to use PyCIRCLean
 
-PyCIRCLean is a simple Python library to handle file checking and sanitization. PyCIRCLean purpose is to have a simple library that can be
-overloaded to cover specific checking and sanitization workflows in different organizations like industrial environment or restricted/classified ICT environment. A series of practical example are in the [./bin](./bin) directory.
+PyCIRCLean is a simple Python library to handle file checking and sanitization. PyCIRCLean is designed as a simple library
+that can be overloaded to cover specific checking and sanitization workflows in different organizations like industrial
+environments or restricted/classified ICT environments. A series of practical examples utilizing PyCIRCLean can be found
+in the [./bin](./bin) directory.
 
-The following simple example using PyCIRCLean will only copy files with .conf extension matching the 'text/plain' MIME type. If any other file is found on the original USB key (source directory), the files won't be copied to the destination directory.
+The following simple example using PyCIRCLean will only copy files with a .conf extension matching the 'text/plain' MIME
+type. If any other file is found in the source directory, the files won't be copied to the destination directory.
 
 ~~~python
 #!/usr/bin/env python
@@ -86,7 +95,7 @@ class KittenGroomerSpec(KittenGroomerBase):
                 valid = False
                 compare_ext = 'Extension: {} - Expected: {}'.format(self.cur_file.extension, ', '.join(self.valid_files.keys()))
             elif self.cur_file.mimetype != expected_mime:
-                # Unexpected mimetype => dissalowed
+                # Unexpected mimetype => disallowed
                 valid = False
                 compare_mime = 'Mime: {} - Expected: {}'.format(self.cur_file.mimetype, expected_mime)
             self.cur_file.add_log_details('valid', valid)
@@ -118,8 +127,8 @@ if __name__ == '__main__':
 # How to contribute
 
 We welcome contributions (including bug fixes, new code workflows) via pull requests. We are interested in any new workflows
-that can be used to improve security in different organizations. If you see any potential enhancement required to support
-your sanitization workflow, feel free to open an issue.
+that can be used to improve security in different organizations. If you see any potential enhancements required to support
+your sanitization workflow, please feel free to open an issue. Read [CONTRIBUTING.md](/CONTRIBUTING.md) for more information.
 
 
 # License

README.rst

@@ -1 +0,0 @@
-README.md

bin/README.md

@@ -1,13 +1,18 @@
+Example scripts
+===============
+
+These are a series of example scripts designed to demonstrate PyCIRCLean's capabilities. Feel free to
+adapt or modify any of them to suit your requirements. In order to use any of these scripts, you will need to
+install the PyCIRCLean dependencies (preferably in a virtualenv):
+
+```
+    pip install git+https://github.com/ahupp/python-magic.git   # we cannot use the PyPi package for now due to a bug
+    python setup.py install     # from the root of the repository
+```
+
 Requirements per script
 =======================
 
-*Note*: in order to use any of those script, you need to install then (in a virtualenv or system wide)
-
-```
-    pip install git+https://github.com/ahupp/python-magic.git # we cannot use the PyPi package for now due to a bug
-    python setup.py install # from the root of the repository
-```
-
 filecheck.py
 ------------
 

dev-requirements.txt

@@ -0,0 +1,4 @@
+twiggy
+python-magic
+pytest
+pytest-cov

playground/README.md

@@ -1 +1 @@
-This directory contains files that may of may not be used in the project
+This directory contains extra files that may or may not be used in the project

tests/test_examples.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import os
+import sys
+
+import pytest
+
+from bin.specific import KittenGroomerSpec
+from bin.pier9 import KittenGroomerPier9
+from bin.generic import KittenGroomer
+
+if sys.version_info.major == 2:
+    from bin.filecheck import KittenGroomerFileCheck
+
+
+skip = pytest.mark.skip
+py2_only = pytest.mark.skipif(sys.version_info.major == 3,
+                                reason="filecheck.py only runs on python 2")
+
+
+@pytest.fixture
+def src_simple():
+    return os.path.join(os.getcwd(), 'tests/src_simple')
+
+
+@pytest.fixture
+def src_complex():
+    return os.path.join(os.getcwd(), 'tests/src_complex')
+
+
+@pytest.fixture
+def dst():
+    return os.path.join(os.getcwd(), 'tests/dst')
+
+
+def test_specific_valid(src_simple, dst):
+    spec = KittenGroomerSpec(src_simple, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+
+def test_specific_invalid(src_complex, dst):
+    spec = KittenGroomerSpec(src_complex, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+
+def test_pier9(src_complex, dst):
+    spec = KittenGroomerPier9(src_complex, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+
+def test_generic(src_simple, dst):
+    spec = KittenGroomer(src_simple, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+
+def test_generic_2(src_complex, dst):
+    spec = KittenGroomer(src_complex, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+
+@py2_only
+def test_filecheck(src_complex, dst):
+    spec = KittenGroomerFileCheck(src_complex, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+
+@py2_only
+def test_filecheck_2(src_simple, dst):
+    spec = KittenGroomerFileCheck(src_simple, dst, debug=True)
+    spec.processdir()
+    dump_logs(spec)
+
+## Helper functions
+
+def dump_logs(spec):
+    print(open(spec.log_processing, 'rb').read())
+    if spec.debug:
+        if os.path.exists(spec.log_debug_err):
+            print(open(spec.log_debug_err, 'rb').read())
+        if os.path.exists(spec.log_debug_out):
+            print(open(spec.log_debug_out, 'rb').read())

tests/test_integration.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import os
+
+import kittengroomer as kg
+import bin.specific as specific
+
+PATH = os.getcwd() + '/tests/'
+
+
+def test_base():
+    assert kg.FileBase
+    assert kg.KittenGroomerBase
+    assert kg.main
+
+
+def test_help_file():
+        f = kg.FileBase('tests/src_complex/blah.conf', 'tests/dst/blah.conf')
+        f.make_unknown()
+        f.make_binary()
+        f.make_unknown()
+        f.make_dangerous()
+        f.make_binary()
+        f.make_dangerous()