CIRCL
/
Circlean
mirror of https://github.com/CIRCL/Circlean
2
0
Fork 0
Code Issues Releases Wiki Activity

Improvements to README.md, wrote CONTRIBUTING.md 

Also deleted several scripts from /shell_utils that were no longer relevant
and renamed README_filecheck.md to README_setup.md
pull/45/head
Dan Puttick 2016-12-28 21:44:33 -05:00
parent 85a4a43fc2
commit 90e4327d5a
7 changed files with 112 additions and 96 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

50
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,50 @@
Building the project
====================
To get started contributing to Circlean, first, fork the project and `git clone`
your fork. Then, follow the instructions in [README_setup.md](README_setup.md)
to build an image.
The issue tracker
=================
If you find a bug or see a problem with PyCIRCLean, please open an issue in the Github
repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a solution
to any of the open issues - we'll do our best to review your pull request in a timely manner.
This project is in active development, so any contributions are welcome!
Running the tests
=================
To emulate the Raspberry Pi hardware for testing, we'll be using
[Qemu](http://wiki.qemu.org/Main_Page), an open source machine emulator.
The "qemu" package available for Ubuntu/Debian includes all of the required
packages (including qemu-system-arm) except for qemu-user-static, which must
be installed separately.
```
sudo apt-get install qemu qemu-user-static expect
```
* Get the qemu kernel for the image you are using:
```
pushd tests; wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/kernel-qemu; popd
```
* Put some test data from tests/testFiles into tests/content_img_vfat_norm
* Comment out the other tests in tests/run.sh or populate those directories as
well
* Make sure to set the filename of the image and the kernel in `tests/run.sh`
* Run the tests:
```
sudo ./run_tests.sh
```
* If the image run processed images correctly but doesn't exit and unmount the
images cleanly, look at tests/run.exp and make sure it's waiting for the
string your qemu and kernel actually produce.

94
README.md
View File

@ -12,77 +12,75 @@ To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourcefo
And the linux way is in the command line, via dd (see in copy_to_final.sh)
If you'd like to contribute to the project or build the image yourself, see
[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md).
Why/What
========
This project aims to be used in case you got an USB key you do not know what is
contains but still want to have a look.
This project aims to be useful when you get/find a USB key that you can't trust,
and you want to have a look at its contents without taking the risk of plugging it into your
main computer directly.
Work in progress, contributions welcome:
This is a work in progress - contributions are welcome:
The content of the first key will be copyed or/and converted to the second key
following theses rules (based on the mime type):
- direct copy of plain text files (mime type: text/*)
- direct copy of audio files (mime type: audio/*)
- direct copy of image files (mime type: image/*)
- direct copy of video files (mime type: video/*)
- direct copy of example files (mime type: example/*)
- direct copy of message files (mime type: message/*)
- direct copy of model files (mime type: model/*)
- direct copy of multipart files (mime type: multipart/*)
- Copying or converting the application files this way (mime type: application/*):
- pdf => HTML
- msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument* => pdf => html
- *xml* => copy as a text file
- x-dosexec (executable) => prepend and append DANGEROUS to the filename
- x-gzip|x-tar|x-7z-compressed => compressed file
- octet-stream => direct copy
Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed):
- Unpacking of archives
- Recursively run the rules on the unpacked files
The content of the first key will be copied or/and converted to the second key
following these rules (based on the mime type, as determined by libmagic):
- Direct copy of:
- Plain text files (mime type: text/*)
- Audio files (mime type: audio/*)
- Video files (mime type: video/*)
- Example files (mime type: example/*)
- Multipart files (mime type: multipart/*)
- *xml* files, after being converted to text files
- Octet-stream files
- Copied after verification:
- Image files after verifying that they are not compression bombs (mime type: image/*)
- PDF files, after marking as dangerous if they contain malicious content
- msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument*, after
parsing with oletools/olefile and marking as dangerous if the parsing fails.
- Copied but marked as dangerous (DANGEROUS_filename_DANGEROUS)
- Message files (mime type: message/*)
- Model files (mime type: model/*)
- x-dosexec (executable)
- Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed):
- Archives are unpacked, with the unpacking process stopped after 2 levels of archives
to prevent archive bombs.
- The above rules are applied recursively to the unpacked files.
Usage
=====
0. Power off the device
1. Plug the untrusted key in the top usb slot of the Raspberry Pi
2. Plug your own key in the bottom usb slot
0. Power off the device and unplug all connections.
1. Plug the untrusted key in the top USB slot of the Raspberry Pi.
2. Plug your own key in the bottom USB slot (or use any of the other slots if
there are more than 2).
*Note*: This key should be bigger than the original one because the archives
will be copyed
*Note*: This key should be bigger than the original one because any archives
present on the source key will be expanded and copied.
3. Optional: connect the HDMI cable to a screen to see what happen
4. Connect the power to the micro USB
3. Optional: connect the HDMI cable to a screen to monitor the process.
4. Connect the power to the micro USB port.
*Note*: 5V, 700mA regulated power supply
*Note*: Use a 5V, 700mA+ regulated power supply
5. Wait until you do not see any blinking green light on the board, or if you
connected the HDMI cable, check the screen
it's slow and can take 30-60 minutes depending on how many document
conversions take place
6. Power off the device and disconnect the drives
connected the HDMI cable, check the screen. The process is slow and can take
30-60 minutes depending on how many document conversions take place.
6. Power off the device and disconnect the drives.
Helper scripts
==============
You should use them as example when you are creating a new image and probably not
run them blindly as you will most probably have to change constraints accordingly to
You should use them as examples when you are creating a new image and probably not
run them blindly as you will most probably have to change parameters accordingly to
your configuration.
IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE.
* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card)
* prepare_rPI_builder.sh: update the system, add the repositories and install all
the dependencies needed to compile poppler and pdf2htmlEX on a rPi
* update_builder.sh: compile the latest version of poppler from debian experimental,
pull and compile the latest version of pdf2htmlEX from the git repository
* prepare_rPI.sh: update the system, install the dependencies of poppler and pdf2htmlEX,
install poppler and pdf2htmlEX (the deb packages compiled in the builder)
* prepare_rPI.sh: update the system, some configuration
* create_user.sh: create the user who will run the scripts, assign the proper sudo rights.
* copy_to_final.sh: populate the content of the directory fs/ in the image,
contains a sample of dd command to write the image on the SD card.
*TAKE CARE NOT USING THE WRONG DESTINATION*
NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION

26
README_filecheck.md → README_setup.md
View File

@ -3,16 +3,16 @@ Building the image from scratch
There is always a prebuilt image available for download and installation as
described in the [README](README.md). If you'd like to build the project yourself,
there are several steps:
there are several steps involved:
* Downloading a generic Raspbian Lite image.
* Adding space to the downloaded image.
* Downloading and building the dependencies.
* Copying the project filesystem into the image.
* Downloading a generic Raspbian Lite image
* Adding space to the image
* Downloading and building the dependencies
* Copying the project filesystem into the image
This procedure will not work unless you are running Ubuntu or Debian linux. If you
only have access to MacOS or Windows, the best option is to install linux in a
VM, using something like VirtualBox.
This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or
Windows, the best option is to install Linux in a virtual machine using
something like VirtualBox.
Downloading the Raspbian image
==============================
@ -87,17 +87,17 @@ larger than it was before (6852607 vs. 2658303 in the example).
Syncing disks.
```
* Mount the image in loop mode. Edit /mount_image.sh to contain the proper values
* Mount the image in loop mode: first, edit /mount_image.sh to use the proper values
for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as
shown above. You must also change $IMAGE to the correct string. Then run:
shown above. You must also change $IMAGE to the correct path. Then run:
```
sudo ./proper_chroot.sh
```
* After mounting the image, the above script will chroot into the mounted image.
While in a chroot, the / directory is treated like it is the / directory (thus
the name, change root). To exit the chroot, run "exit" in the root directory.
While in a chroot, the / directory of the image appears as the system / directory
(thus the name, change root). To exit the chroot, run "exit" in the root directory.
Then, verify the path to the mounted partitions, and resize the filesystem
to fill the new larger partition using resize2fs:
@ -183,7 +183,7 @@ Write the image on a SD card
```
* If it has been automatically mounted, unmount the SD card (use the path you
found in the previous step:
found in the previous step):
```
umount $PATH_TO_YOUR_SD

3
fs_filecheck/etc/rc.local
View File

@ -10,6 +10,9 @@
# bits.
#
# By default this script does nothing.
#
# The above are the default comments for rc.local. For Circlean, rc.local has
# been modified to start the grooming process on boot.
clean(){
echo 'Rc Local done, quit.'

10
shell_utils/prepare_rPI.sh
View File

@ -10,16 +10,6 @@ apt-get autoremove
apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount
dpkg -i pdf2htmlex*.deb
# Make Libreoffice usable on a RO filesystem
useradd -m kitten
pushd /home/kitten
ln -s /tmp/libreoffice
popd
chown -R kitten:kitten /home/kitten
ln -s /proc/mounts /etc/mtab
# Disable swap
dphys-swapfile uninstall

14
shell_utils/prepare_rPI_builder.sh
View File

@ -1,14 +0,0 @@
#!/bin/bash
# change locales to en_US.UTF-8
dpkg-reconfigure locales
# Increase size of image. See resize_img.md
apt-get update
apt-get dist-upgrade
apt-get autoremove
# build dependencies of pdf2htmlEX
apt-get install cmake debhelper libpoppler-dev libjpeg-dev libfontforge-dev \
libspiro-dev python-dev default-jre-headless libpoppler-private-dev

11
shell_utils/update_builder.sh
View File

@ -1,11 +0,0 @@
#!/bin/bash
# We cannot use the version 0.12 because it requires fontforge 2.0
# The fork use a saner list of dependencies and a patch that allows to build on debian jessie.
wget https://github.com/Rafiot/pdf2htmlEX/archive/KittenGroomer.zip
unzip KittenGroomer.zip
cd pdf2htmlEX-KittenGroomer/
dpkg-buildpackage -uc -b