From 90e4327d5af219975090f9e32c73b87b241dcad8 Mon Sep 17 00:00:00 2001 From: Dan Puttick Date: Wed, 28 Dec 2016 21:44:33 -0500 Subject: [PATCH] Improvements to README.md, wrote CONTRIBUTING.md Also deleted several scripts from /shell_utils that were no longer relevant and renamed README_filecheck.md to README_setup.md --- CONTRIBUTING.md | 50 ++++++++++++++ README.md | 94 +++++++++++++------------- README_filecheck.md => README_setup.md | 26 +++---- fs_filecheck/etc/rc.local | 3 + shell_utils/prepare_rPI.sh | 10 --- shell_utils/prepare_rPI_builder.sh | 14 ---- shell_utils/update_builder.sh | 11 --- 7 files changed, 112 insertions(+), 96 deletions(-) create mode 100644 CONTRIBUTING.md rename README_filecheck.md => README_setup.md (87%) delete mode 100755 shell_utils/prepare_rPI_builder.sh delete mode 100755 shell_utils/update_builder.sh diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..2ed1ee4 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,50 @@ +Building the project +==================== + +To get started contributing to Circlean, first, fork the project and `git clone` +your fork. Then, follow the instructions in [README_setup.md](README_setup.md) +to build an image. + +The issue tracker +================= + +If you find a bug or see a problem with PyCIRCLean, please open an issue in the Github +repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a solution +to any of the open issues - we'll do our best to review your pull request in a timely manner. +This project is in active development, so any contributions are welcome! + +Running the tests +================= + +To emulate the Raspberry Pi hardware for testing, we'll be using +[Qemu](http://wiki.qemu.org/Main_Page), an open source machine emulator. +The "qemu" package available for Ubuntu/Debian includes all of the required +packages (including qemu-system-arm) except for qemu-user-static, which must +be installed separately. + +``` + sudo apt-get install qemu qemu-user-static expect +``` + +* Get the qemu kernel for the image you are using: + +``` + pushd tests; wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/kernel-qemu; popd +``` + +* Put some test data from tests/testFiles into tests/content_img_vfat_norm + +* Comment out the other tests in tests/run.sh or populate those directories as + well + +* Make sure to set the filename of the image and the kernel in `tests/run.sh` + +* Run the tests: + +``` + sudo ./run_tests.sh +``` + +* If the image run processed images correctly but doesn't exit and unmount the + images cleanly, look at tests/run.exp and make sure it's waiting for the + string your qemu and kernel actually produce. diff --git a/README.md b/README.md index 06eb4ce..4f9e1a0 100644 --- a/README.md +++ b/README.md @@ -12,77 +12,75 @@ To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourcefo And the linux way is in the command line, via dd (see in copy_to_final.sh) +If you'd like to contribute to the project or build the image yourself, see +[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md). + Why/What ======== -This project aims to be used in case you got an USB key you do not know what is -contains but still want to have a look. +This project aims to be useful when you get/find a USB key that you can't trust, +and you want to have a look at its contents without taking the risk of plugging it into your +main computer directly. -Work in progress, contributions welcome: +This is a work in progress - contributions are welcome: -The content of the first key will be copyed or/and converted to the second key -following theses rules (based on the mime type): -- direct copy of plain text files (mime type: text/*) -- direct copy of audio files (mime type: audio/*) -- direct copy of image files (mime type: image/*) -- direct copy of video files (mime type: video/*) -- direct copy of example files (mime type: example/*) -- direct copy of message files (mime type: message/*) -- direct copy of model files (mime type: model/*) -- direct copy of multipart files (mime type: multipart/*) -- Copying or converting the application files this way (mime type: application/*): - - pdf => HTML - - msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument* => pdf => html - - *xml* => copy as a text file - - x-dosexec (executable) => prepend and append DANGEROUS to the filename - - x-gzip|x-tar|x-7z-compressed => compressed file - - octet-stream => direct copy - -Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed): -- Unpacking of archives -- Recursively run the rules on the unpacked files +The content of the first key will be copied or/and converted to the second key +following these rules (based on the mime type, as determined by libmagic): +- Direct copy of: + - Plain text files (mime type: text/*) + - Audio files (mime type: audio/*) + - Video files (mime type: video/*) + - Example files (mime type: example/*) + - Multipart files (mime type: multipart/*) + - *xml* files, after being converted to text files + - Octet-stream files +- Copied after verification: + - Image files after verifying that they are not compression bombs (mime type: image/*) + - PDF files, after marking as dangerous if they contain malicious content + - msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument*, after + parsing with oletools/olefile and marking as dangerous if the parsing fails. +- Copied but marked as dangerous (DANGEROUS_filename_DANGEROUS) + - Message files (mime type: message/*) + - Model files (mime type: model/*) + - x-dosexec (executable) +- Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed): + - Archives are unpacked, with the unpacking process stopped after 2 levels of archives + to prevent archive bombs. + - The above rules are applied recursively to the unpacked files. Usage ===== -0. Power off the device -1. Plug the untrusted key in the top usb slot of the Raspberry Pi -2. Plug your own key in the bottom usb slot +0. Power off the device and unplug all connections. +1. Plug the untrusted key in the top USB slot of the Raspberry Pi. +2. Plug your own key in the bottom USB slot (or use any of the other slots if +there are more than 2). - *Note*: This key should be bigger than the original one because the archives - will be copyed + *Note*: This key should be bigger than the original one because any archives + present on the source key will be expanded and copied. -3. Optional: connect the HDMI cable to a screen to see what happen -4. Connect the power to the micro USB +3. Optional: connect the HDMI cable to a screen to monitor the process. +4. Connect the power to the micro USB port. - *Note*: 5V, 700mA regulated power supply + *Note*: Use a 5V, 700mA+ regulated power supply 5. Wait until you do not see any blinking green light on the board, or if you - connected the HDMI cable, check the screen - it's slow and can take 30-60 minutes depending on how many document - conversions take place -6. Power off the device and disconnect the drives + connected the HDMI cable, check the screen. The process is slow and can take + 30-60 minutes depending on how many document conversions take place. +6. Power off the device and disconnect the drives. Helper scripts ============== -You should use them as example when you are creating a new image and probably not -run them blindly as you will most probably have to change constraints accordingly to +You should use them as examples when you are creating a new image and probably not +run them blindly as you will most probably have to change parameters accordingly to your configuration. IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE. * proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card) -* prepare_rPI_builder.sh: update the system, add the repositories and install all - the dependencies needed to compile poppler and pdf2htmlEX on a rPi -* update_builder.sh: compile the latest version of poppler from debian experimental, - pull and compile the latest version of pdf2htmlEX from the git repository -* prepare_rPI.sh: update the system, install the dependencies of poppler and pdf2htmlEX, - install poppler and pdf2htmlEX (the deb packages compiled in the builder) +* prepare_rPI.sh: update the system, some configuration * create_user.sh: create the user who will run the scripts, assign the proper sudo rights. * copy_to_final.sh: populate the content of the directory fs/ in the image, contains a sample of dd command to write the image on the SD card. - *TAKE CARE NOT USING THE WRONG DESTINATION* - - - + NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION diff --git a/README_filecheck.md b/README_setup.md similarity index 87% rename from README_filecheck.md rename to README_setup.md index 9fa2413..afb74a1 100644 --- a/README_filecheck.md +++ b/README_setup.md @@ -3,16 +3,16 @@ Building the image from scratch There is always a prebuilt image available for download and installation as described in the [README](README.md). If you'd like to build the project yourself, -there are several steps: +there are several steps involved: -* Downloading a generic Raspbian Lite image. -* Adding space to the downloaded image. -* Downloading and building the dependencies. -* Copying the project filesystem into the image. +* Downloading a generic Raspbian Lite image +* Adding space to the image +* Downloading and building the dependencies +* Copying the project filesystem into the image -This procedure will not work unless you are running Ubuntu or Debian linux. If you -only have access to MacOS or Windows, the best option is to install linux in a -VM, using something like VirtualBox. +This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or +Windows, the best option is to install Linux in a virtual machine using +something like VirtualBox. Downloading the Raspbian image ============================== @@ -87,17 +87,17 @@ larger than it was before (6852607 vs. 2658303 in the example). Syncing disks. ``` -* Mount the image in loop mode. Edit /mount_image.sh to contain the proper values +* Mount the image in loop mode: first, edit /mount_image.sh to use the proper values for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as -shown above. You must also change $IMAGE to the correct string. Then run: +shown above. You must also change $IMAGE to the correct path. Then run: ``` sudo ./proper_chroot.sh ``` * After mounting the image, the above script will chroot into the mounted image. -While in a chroot, the / directory is treated like it is the / directory (thus -the name, change root). To exit the chroot, run "exit" in the root directory. +While in a chroot, the / directory of the image appears as the system / directory +(thus the name, change root). To exit the chroot, run "exit" in the root directory. Then, verify the path to the mounted partitions, and resize the filesystem to fill the new larger partition using resize2fs: @@ -183,7 +183,7 @@ Write the image on a SD card ``` * If it has been automatically mounted, unmount the SD card (use the path you -found in the previous step: +found in the previous step): ``` umount $PATH_TO_YOUR_SD diff --git a/fs_filecheck/etc/rc.local b/fs_filecheck/etc/rc.local index 6c2e691..cb0d793 100755 --- a/fs_filecheck/etc/rc.local +++ b/fs_filecheck/etc/rc.local @@ -10,6 +10,9 @@ # bits. # # By default this script does nothing. +# +# The above are the default comments for rc.local. For Circlean, rc.local has +# been modified to start the grooming process on boot. clean(){ echo 'Rc Local done, quit.' diff --git a/shell_utils/prepare_rPI.sh b/shell_utils/prepare_rPI.sh index e408bfd..a55eedc 100755 --- a/shell_utils/prepare_rPI.sh +++ b/shell_utils/prepare_rPI.sh @@ -10,16 +10,6 @@ apt-get autoremove apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount dpkg -i pdf2htmlex*.deb -# Make Libreoffice usable on a RO filesystem -useradd -m kitten -pushd /home/kitten -ln -s /tmp/libreoffice -popd - -chown -R kitten:kitten /home/kitten - -ln -s /proc/mounts /etc/mtab - # Disable swap dphys-swapfile uninstall diff --git a/shell_utils/prepare_rPI_builder.sh b/shell_utils/prepare_rPI_builder.sh deleted file mode 100755 index d9c380b..0000000 --- a/shell_utils/prepare_rPI_builder.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/bash - -# change locales to en_US.UTF-8 -dpkg-reconfigure locales - -# Increase size of image. See resize_img.md - -apt-get update -apt-get dist-upgrade -apt-get autoremove - -# build dependencies of pdf2htmlEX -apt-get install cmake debhelper libpoppler-dev libjpeg-dev libfontforge-dev \ - libspiro-dev python-dev default-jre-headless libpoppler-private-dev diff --git a/shell_utils/update_builder.sh b/shell_utils/update_builder.sh deleted file mode 100755 index 0fdfbe5..0000000 --- a/shell_utils/update_builder.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -# We cannot use the version 0.12 because it requires fontforge 2.0 -# The fork use a saner list of dependencies and a patch that allows to build on debian jessie. - -wget https://github.com/Rafiot/pdf2htmlEX/archive/KittenGroomer.zip -unzip KittenGroomer.zip - -cd pdf2htmlEX-KittenGroomer/ - -dpkg-buildpackage -uc -b