CIRCL
/
PyCIRCLean
mirror of https://github.com/CIRCL/PyCIRCLean
2
0
Fork 0
Code Issues Releases Wiki Activity
379 Commits
6 Branches
5 Tags
3.9 MiB
Commit Graph

379 Commits (cc081a921d39f34624e079854f3d542e20770e81)

Author SHA1 Message Date
Dan Puttick 3c1fcda29e Add different file size units to log 
* Format file sizes with B, KB, MB, or GB depending on size
* Clean up logic and variable names in add_file()
 2017-06-27 11:54:57 -04:00
Dan Puttick e27d397496 Change is_recursive to is_archive 
* is_archive is clearer and more descriptive than is_recursive
 2017-06-27 11:54:57 -04:00
Dan Puttick 74a7244fbd Add png file to src_valid 2017-06-27 11:52:21 -04:00
Raphaël Vinot 079e8d30a3 Add support for ObjectStream in PDF 
ObjectStream isn't necessarely malicious, but can be. This patch could
be improved by unpacking the content of the stream, but it requires 3rd
party libraries we don't have for now.

Final fix for PCL-01-002
 2017-06-19 11:24:47 +02:00
Raphaël Vinot 7d38ec3d32 Remove extensions processed by the script 2017-06-16 17:57:07 +02:00
Raphaël Vinot f44719b83e Add list of malicious extensions used in Google Chrome 
Fix PCL-01-009
 2017-06-16 17:26:39 +02:00
Raphaël Vinot 40f71e758f Fix logging and symlinks 
Fix PCL-01-006
 2017-06-16 14:47:53 +02:00
Raphaël Vinot 8c007e28cf Add support for XFA structure un PDF 
Partial fix for PCL-01-002
 2017-06-16 14:47:19 +02:00
Raphaël Vinot e276f33c29 Merge pull request #15 from aschet01/norepeats 
Check value against repeats in description_string.
 2017-05-28 06:41:35 +02:00
Adam Schettenhelm 7b990df436 Check against value for duplicates when inserting into description_string 2017-05-26 20:37:36 -04:00
Raphaël Vinot b6fb4c80bf Add travis tests on python 3.6 2017-04-13 23:03:50 +02:00
Raphaël Vinot f5cc3d7533 Merge pull request #14 from dputtick/logging 
Logging format improvements
 2017-04-13 22:44:39 +02:00
Dan Puttick d470d6bb21 Open test log in bytes mode 2017-04-12 17:19:21 -04:00
Dan Puttick a8179d0688 Read test logs in bytes mode 2017-04-12 15:32:57 -04:00
Dan Puttick 45d71cb362 Fix unicode filename issues using fsencode 
* Same problem we've had before - linux filenames can have non-unicode chars
in them
* We need to write the filename as raw bytes to the log
* os.fsencode lets us convert a utf-8 encoded string to bytes and ignore those
that can't be printed as unicode
* Still not clear if the log generated this way will be human-readable
 2017-04-10 13:39:28 +02:00
Dan Puttick 053f30db93 Partial update to changelog 2017-04-10 13:33:00 +02:00
Dan Puttick 13460d643d Remove twiggy from install requirements 2017-04-10 13:25:51 +02:00
Dan Puttick 5865ddd94d Make root paths abspaths 
* src_root_path and dst_root_path are now converted to abspaths when
initializing kittengroomer to avoid any weird issues with relative or misformed
paths
 2017-04-10 13:22:36 +02:00
Dan Puttick 3e56787686 Update tests for new logger 2017-04-10 13:22:36 +02:00
Dan Puttick c43ac0697a Add comments/notes to helpers.py 2017-04-10 13:22:36 +02:00
Dan Puttick 67c90087ba Add time information to test logs 2017-04-10 13:22:36 +02:00
Dan Puttick 3f49612a23 Add new logger, move logging to filecheck 
* Wrote a new text-based logger that displays all file information in the tree
instead of using two separate logs
* Stopped using twiggy since it wasn't giving us anything useful
* Moved a lot of the logging code to filecheck, since it didn't really seem
appropriate as an API. Left a Logging stub in kittengroomer to hold methods
that might be useful for implementing other loggers.
* For the new logger, had to change the way that we traverse the items in the
source file tree.
 2017-04-10 13:22:20 +02:00
Dan Puttick f0e7607a3f Improve description strings in filecheck 
* Description strings that appear in the log improved in filecheck for various
file types
* Added various comments
 2017-04-10 13:00:34 +02:00
Dan Puttick c85ad27221 New test files 2017-04-10 12:54:07 +02:00
Dan Puttick ba407219d3 Open log file in text mode instead of bytes mode 2017-03-22 21:49:48 -04:00
Dan Puttick 52bb566cc3 Write basic log to log file 2017-03-22 12:07:43 -04:00
Dan Puttick 6f9e36a578 Change filecheck for new file description method 
* self.add_file_string -> self.add_description
 2017-03-22 12:04:22 -04:00
Dan Puttick 265f32ad6b Change the way description strings are handled 2017-03-22 10:28:00 -04:00
Dan Puttick 3e7b38c5d4 Improve doc strings on FileBase 2017-03-21 18:58:17 -04:00
Dan Puttick 6851461755 Change from two separate logs to one 2017-03-20 16:10:57 -04:00
Dan Puttick 51760ebbb1 Move default log setup back into filecheck 
* Realized that the API consumer might want to write their own logging tool.
* FileBase and KittenGroomerBase will have no logging code.
* If the API consumer likes, they can import GroomerLogger and use it in their
implementation.
 2017-03-20 16:10:57 -04:00
Dan Puttick 18857c8cf7 Change names of KittenGroomerBase root dir paths 2017-03-20 16:10:57 -04:00
Dan Puttick 27f58a0ede Put log dir creation code in separate method 2017-03-20 16:10:57 -04:00
Raphaël Vinot bfc6694cd1 Merge pull request #13 from dputtick/rtl-char 
Support right to left override character
 2017-03-18 09:47:13 +01:00
Dan Puttick 2a5decf0ad Add rtl sample file to src_invalid 2017-03-16 17:34:20 -04:00
Dan Puttick 71bcc79c20 Remove rtl override char from file dst_path 
The unicode right to left override character can be used for various attacks.
This commit:
* Detects this character in the filename on the source key
* Strips it from the path before copying it to the dest key
* Marks the file as dangerous (this character doesn't belong in a filename)
 2017-03-16 12:22:26 -04:00
Raphaël Vinot 79b15fd7da Merge pull request #12 from dputtick/dev 
API changes and (some) logging functionality
 2017-03-16 10:45:20 +01:00
Dan Puttick 1abfb432b1 Edit README.md 
* Mention that example files are not up to date with the new API changes
* Update example code for API changes
 2017-03-15 22:56:01 -04:00
Dan Puttick 4d8a1d1daf Add/update docstrings for filecheck and helpers 2017-03-15 22:56:00 -04:00
Dan Puttick ac94cf5d6d Change the way test dst dirs are handled 
* Each test folder now copies files into its own test directory
* Change gitignore due to dst dir changes
* Make sure logger.tree is called for every directory
 2017-03-15 22:56:00 -04:00
Dan Puttick 0175ee48e5 Add TODOs and clarify various logging messages 2017-03-15 22:56:00 -04:00
Dan Puttick 963a2feef4 Change various methods to properties 2017-03-15 22:55:51 -04:00
Dan Puttick 59cde8cfd5 Move safe_copy to FileBase 2017-03-15 21:06:07 -04:00
Dan Puttick e73721e95f Fix bug with safe_copy 2017-03-15 21:06:07 -04:00
Dan Puttick 484c71fc86 Turn off copying for certain mimes in filecheck 2017-03-15 21:06:07 -04:00
Dan Puttick 18857da7ca Several small bugfixes 
* Fix issue with main/subtypes in init
* Fix bug in File.check() in filecheck.py
* Fix FileBase.size for symlinks
 2017-03-15 21:06:07 -04:00
Dan Puttick 3fe8c7c223 Adjust order of property initialization 
Tests were failing due to values being set before file_props dict
was created
 2017-03-15 21:06:07 -04:00
Dan Puttick 0038d3ef66 Switch to using file properties 
* make_dangerous now takes a description string
* add_file_string takes strings describing the file
 2017-03-15 21:06:07 -04:00
Dan Puttick fc8923fddd Change Groomer private methods to public 
* Changed safe_rmtree, safe_copy, safe_remove, and safe_mkdir to public methods
* If something is being used in a subclass it probably shouldn't be a private
method
 2017-03-15 21:06:07 -04:00
Dan Puttick 12d5624b4d Change FileBase.log_details to Filebase._file_props 
* _file_props is a dict that will hold all information about the file
* Updated filecheck.py to reflect this
* Potentially will change contents of file_props to being attributes on the
file in the future. This change would be easy since all access to _file_props
is now via set_property and get_property methods.
* Add filename to _file_props
 2017-03-15 21:06:06 -04:00