CIRCL
/
PyCIRCLean
mirror of https://github.com/CIRCL/PyCIRCLean
2
0
Fork 0
Code Issues Releases Wiki Activity
215 Commits
6 Branches
5 Tags
3.9 MiB
Commit Graph

215 Commits (45d71cb362c3bd0bd5442e6dc482d22749c7aabc)

Author SHA1 Message Date
Dan Puttick 45d71cb362 Fix unicode filename issues using fsencode 
* Same problem we've had before - linux filenames can have non-unicode chars
in them
* We need to write the filename as raw bytes to the log
* os.fsencode lets us convert a utf-8 encoded string to bytes and ignore those
that can't be printed as unicode
* Still not clear if the log generated this way will be human-readable
 2017-04-10 13:39:28 +02:00
Dan Puttick 053f30db93 Partial update to changelog 2017-04-10 13:33:00 +02:00
Dan Puttick 13460d643d Remove twiggy from install requirements 2017-04-10 13:25:51 +02:00
Dan Puttick 5865ddd94d Make root paths abspaths 
* src_root_path and dst_root_path are now converted to abspaths when
initializing kittengroomer to avoid any weird issues with relative or misformed
paths
 2017-04-10 13:22:36 +02:00
Dan Puttick 3e56787686 Update tests for new logger 2017-04-10 13:22:36 +02:00
Dan Puttick c43ac0697a Add comments/notes to helpers.py 2017-04-10 13:22:36 +02:00
Dan Puttick 67c90087ba Add time information to test logs 2017-04-10 13:22:36 +02:00
Dan Puttick 3f49612a23 Add new logger, move logging to filecheck 
* Wrote a new text-based logger that displays all file information in the tree
instead of using two separate logs
* Stopped using twiggy since it wasn't giving us anything useful
* Moved a lot of the logging code to filecheck, since it didn't really seem
appropriate as an API. Left a Logging stub in kittengroomer to hold methods
that might be useful for implementing other loggers.
* For the new logger, had to change the way that we traverse the items in the
source file tree.
 2017-04-10 13:22:20 +02:00
Dan Puttick f0e7607a3f Improve description strings in filecheck 
* Description strings that appear in the log improved in filecheck for various
file types
* Added various comments
 2017-04-10 13:00:34 +02:00
Dan Puttick c85ad27221 New test files 2017-04-10 12:54:07 +02:00
Dan Puttick ba407219d3 Open log file in text mode instead of bytes mode 2017-03-22 21:49:48 -04:00
Dan Puttick 52bb566cc3 Write basic log to log file 2017-03-22 12:07:43 -04:00
Dan Puttick 6f9e36a578 Change filecheck for new file description method 
* self.add_file_string -> self.add_description
 2017-03-22 12:04:22 -04:00
Dan Puttick 265f32ad6b Change the way description strings are handled 2017-03-22 10:28:00 -04:00
Dan Puttick 3e7b38c5d4 Improve doc strings on FileBase 2017-03-21 18:58:17 -04:00
Dan Puttick 6851461755 Change from two separate logs to one 2017-03-20 16:10:57 -04:00
Dan Puttick 51760ebbb1 Move default log setup back into filecheck 
* Realized that the API consumer might want to write their own logging tool.
* FileBase and KittenGroomerBase will have no logging code.
* If the API consumer likes, they can import GroomerLogger and use it in their
implementation.
 2017-03-20 16:10:57 -04:00
Dan Puttick 18857c8cf7 Change names of KittenGroomerBase root dir paths 2017-03-20 16:10:57 -04:00
Dan Puttick 27f58a0ede Put log dir creation code in separate method 2017-03-20 16:10:57 -04:00
Raphaël Vinot bfc6694cd1 Merge pull request #13 from dputtick/rtl-char 
Support right to left override character
 2017-03-18 09:47:13 +01:00
Dan Puttick 2a5decf0ad Add rtl sample file to src_invalid 2017-03-16 17:34:20 -04:00
Dan Puttick 71bcc79c20 Remove rtl override char from file dst_path 
The unicode right to left override character can be used for various attacks.
This commit:
* Detects this character in the filename on the source key
* Strips it from the path before copying it to the dest key
* Marks the file as dangerous (this character doesn't belong in a filename)
 2017-03-16 12:22:26 -04:00
Raphaël Vinot 79b15fd7da Merge pull request #12 from dputtick/dev 
API changes and (some) logging functionality
 2017-03-16 10:45:20 +01:00
Dan Puttick 1abfb432b1 Edit README.md 
* Mention that example files are not up to date with the new API changes
* Update example code for API changes
 2017-03-15 22:56:01 -04:00
Dan Puttick 4d8a1d1daf Add/update docstrings for filecheck and helpers 2017-03-15 22:56:00 -04:00
Dan Puttick ac94cf5d6d Change the way test dst dirs are handled 
* Each test folder now copies files into its own test directory
* Change gitignore due to dst dir changes
* Make sure logger.tree is called for every directory
 2017-03-15 22:56:00 -04:00
Dan Puttick 0175ee48e5 Add TODOs and clarify various logging messages 2017-03-15 22:56:00 -04:00
Dan Puttick 963a2feef4 Change various methods to properties 2017-03-15 22:55:51 -04:00
Dan Puttick 59cde8cfd5 Move safe_copy to FileBase 2017-03-15 21:06:07 -04:00
Dan Puttick e73721e95f Fix bug with safe_copy 2017-03-15 21:06:07 -04:00
Dan Puttick 484c71fc86 Turn off copying for certain mimes in filecheck 2017-03-15 21:06:07 -04:00
Dan Puttick 18857da7ca Several small bugfixes 
* Fix issue with main/subtypes in init
* Fix bug in File.check() in filecheck.py
* Fix FileBase.size for symlinks
 2017-03-15 21:06:07 -04:00
Dan Puttick 3fe8c7c223 Adjust order of property initialization 
Tests were failing due to values being set before file_props dict
was created
 2017-03-15 21:06:07 -04:00
Dan Puttick 0038d3ef66 Switch to using file properties 
* make_dangerous now takes a description string
* add_file_string takes strings describing the file
 2017-03-15 21:06:07 -04:00
Dan Puttick fc8923fddd Change Groomer private methods to public 
* Changed safe_rmtree, safe_copy, safe_remove, and safe_mkdir to public methods
* If something is being used in a subclass it probably shouldn't be a private
method
 2017-03-15 21:06:07 -04:00
Dan Puttick 12d5624b4d Change FileBase.log_details to Filebase._file_props 
* _file_props is a dict that will hold all information about the file
* Updated filecheck.py to reflect this
* Potentially will change contents of file_props to being attributes on the
file in the future. This change would be easy since all access to _file_props
is now via set_property and get_property methods.
* Add filename to _file_props
 2017-03-15 21:06:06 -04:00
Dan Puttick 1c58a7347e If no extentions FileBase.ext is now None 2017-03-15 21:06:06 -04:00
Dan Puttick b6c01db1fb Split mimetype methods 
- Instead of one large function that mutates FileBase properties,
mimetype and main/subtype are determined by two separate methods
that return mimetypes.
- The API is not changed.
- Absence of mimetype is now None instead of an empty string.
 2017-03-15 21:06:06 -04:00
Dan Puttick 9832101c85 Identify TODOs that are log related 2017-03-15 21:06:06 -04:00
Dan Puttick 8d7dd1197f Move run_process back to Groomer object 2017-03-15 21:06:06 -04:00
Dan Puttick 781d0a76af First working version with methods in File object 
- All tests now passing with file handling methods on File object
instead of Groomer object.
- Logging functionality still isn't finished.
 2017-03-15 21:06:06 -04:00
Dan Puttick 9aafe6e518 Remove cur_file from methods in File object 2017-03-15 21:06:06 -04:00
Dan Puttick 53c1598af8 Move file processing methods into File object 
- It seems like filecheck will be easier to reason about if all of
the file processing stuff happens in the File object. The Groomer
object will now be responsible only for enumerating the files to
be processed.
- Tests won't pass for this commit, but wanted to make the diff
cleaner but committing this before making changes.
 2017-03-15 21:04:57 -04:00
Dan Puttick 3d36c90d66 Make list_all_files a public method 2017-03-15 21:04:57 -04:00
Dan Puttick 4a99ba900a Make self.get_extension return a value 2017-03-15 21:04:57 -04:00
Dan Puttick c6ecc5e3a3 Fix process_dir bug in filecheck tests 2017-03-15 21:04:57 -04:00
Dan Puttick bfa257534a Remove old data/ path stuff from helpers.py 2017-03-15 21:04:57 -04:00
Dan Puttick cfeccc2561 Move main() into filecheck 
- Also change the name of processdir to process_dir
 2017-03-15 21:04:57 -04:00
Dan Puttick 61aa14c98d Change _write_log to _print_log 2017-03-15 21:04:57 -04:00
Dan Puttick 9d2f5c7241 Add .jpg test file, testfile_directory.md 
- Also fixed a bug with metadata code
 2017-03-15 21:04:57 -04:00