mirror of https://github.com/CIRCL/Circlean
Improvements to README.md, wrote CONTRIBUTING.md
Also deleted several scripts from /shell_utils that were no longer relevant and renamed README_filecheck.md to README_setup.mdpull/45/head
parent
85a4a43fc2
commit
90e4327d5a
|
@ -0,0 +1,50 @@
|
||||||
|
Building the project
|
||||||
|
====================
|
||||||
|
|
||||||
|
To get started contributing to Circlean, first, fork the project and `git clone`
|
||||||
|
your fork. Then, follow the instructions in [README_setup.md](README_setup.md)
|
||||||
|
to build an image.
|
||||||
|
|
||||||
|
The issue tracker
|
||||||
|
=================
|
||||||
|
|
||||||
|
If you find a bug or see a problem with PyCIRCLean, please open an issue in the Github
|
||||||
|
repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a solution
|
||||||
|
to any of the open issues - we'll do our best to review your pull request in a timely manner.
|
||||||
|
This project is in active development, so any contributions are welcome!
|
||||||
|
|
||||||
|
Running the tests
|
||||||
|
=================
|
||||||
|
|
||||||
|
To emulate the Raspberry Pi hardware for testing, we'll be using
|
||||||
|
[Qemu](http://wiki.qemu.org/Main_Page), an open source machine emulator.
|
||||||
|
The "qemu" package available for Ubuntu/Debian includes all of the required
|
||||||
|
packages (including qemu-system-arm) except for qemu-user-static, which must
|
||||||
|
be installed separately.
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo apt-get install qemu qemu-user-static expect
|
||||||
|
```
|
||||||
|
|
||||||
|
* Get the qemu kernel for the image you are using:
|
||||||
|
|
||||||
|
```
|
||||||
|
pushd tests; wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/kernel-qemu; popd
|
||||||
|
```
|
||||||
|
|
||||||
|
* Put some test data from tests/testFiles into tests/content_img_vfat_norm
|
||||||
|
|
||||||
|
* Comment out the other tests in tests/run.sh or populate those directories as
|
||||||
|
well
|
||||||
|
|
||||||
|
* Make sure to set the filename of the image and the kernel in `tests/run.sh`
|
||||||
|
|
||||||
|
* Run the tests:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo ./run_tests.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
* If the image run processed images correctly but doesn't exit and unmount the
|
||||||
|
images cleanly, look at tests/run.exp and make sure it's waiting for the
|
||||||
|
string your qemu and kernel actually produce.
|
94
README.md
94
README.md
|
@ -12,77 +12,75 @@ To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourcefo
|
||||||
|
|
||||||
And the linux way is in the command line, via dd (see in copy_to_final.sh)
|
And the linux way is in the command line, via dd (see in copy_to_final.sh)
|
||||||
|
|
||||||
|
If you'd like to contribute to the project or build the image yourself, see
|
||||||
|
[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md).
|
||||||
|
|
||||||
Why/What
|
Why/What
|
||||||
========
|
========
|
||||||
|
|
||||||
This project aims to be used in case you got an USB key you do not know what is
|
This project aims to be useful when you get/find a USB key that you can't trust,
|
||||||
contains but still want to have a look.
|
and you want to have a look at its contents without taking the risk of plugging it into your
|
||||||
|
main computer directly.
|
||||||
|
|
||||||
Work in progress, contributions welcome:
|
This is a work in progress - contributions are welcome:
|
||||||
|
|
||||||
The content of the first key will be copyed or/and converted to the second key
|
The content of the first key will be copied or/and converted to the second key
|
||||||
following theses rules (based on the mime type):
|
following these rules (based on the mime type, as determined by libmagic):
|
||||||
- direct copy of plain text files (mime type: text/*)
|
- Direct copy of:
|
||||||
- direct copy of audio files (mime type: audio/*)
|
- Plain text files (mime type: text/*)
|
||||||
- direct copy of image files (mime type: image/*)
|
- Audio files (mime type: audio/*)
|
||||||
- direct copy of video files (mime type: video/*)
|
- Video files (mime type: video/*)
|
||||||
- direct copy of example files (mime type: example/*)
|
- Example files (mime type: example/*)
|
||||||
- direct copy of message files (mime type: message/*)
|
- Multipart files (mime type: multipart/*)
|
||||||
- direct copy of model files (mime type: model/*)
|
- *xml* files, after being converted to text files
|
||||||
- direct copy of multipart files (mime type: multipart/*)
|
- Octet-stream files
|
||||||
- Copying or converting the application files this way (mime type: application/*):
|
- Copied after verification:
|
||||||
- pdf => HTML
|
- Image files after verifying that they are not compression bombs (mime type: image/*)
|
||||||
- msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument* => pdf => html
|
- PDF files, after marking as dangerous if they contain malicious content
|
||||||
- *xml* => copy as a text file
|
- msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument*, after
|
||||||
- x-dosexec (executable) => prepend and append DANGEROUS to the filename
|
parsing with oletools/olefile and marking as dangerous if the parsing fails.
|
||||||
- x-gzip|x-tar|x-7z-compressed => compressed file
|
- Copied but marked as dangerous (DANGEROUS_filename_DANGEROUS)
|
||||||
- octet-stream => direct copy
|
- Message files (mime type: message/*)
|
||||||
|
- Model files (mime type: model/*)
|
||||||
Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed):
|
- x-dosexec (executable)
|
||||||
- Unpacking of archives
|
- Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed):
|
||||||
- Recursively run the rules on the unpacked files
|
- Archives are unpacked, with the unpacking process stopped after 2 levels of archives
|
||||||
|
to prevent archive bombs.
|
||||||
|
- The above rules are applied recursively to the unpacked files.
|
||||||
|
|
||||||
Usage
|
Usage
|
||||||
=====
|
=====
|
||||||
|
|
||||||
0. Power off the device
|
0. Power off the device and unplug all connections.
|
||||||
1. Plug the untrusted key in the top usb slot of the Raspberry Pi
|
1. Plug the untrusted key in the top USB slot of the Raspberry Pi.
|
||||||
2. Plug your own key in the bottom usb slot
|
2. Plug your own key in the bottom USB slot (or use any of the other slots if
|
||||||
|
there are more than 2).
|
||||||
|
|
||||||
*Note*: This key should be bigger than the original one because the archives
|
*Note*: This key should be bigger than the original one because any archives
|
||||||
will be copyed
|
present on the source key will be expanded and copied.
|
||||||
|
|
||||||
3. Optional: connect the HDMI cable to a screen to see what happen
|
3. Optional: connect the HDMI cable to a screen to monitor the process.
|
||||||
4. Connect the power to the micro USB
|
4. Connect the power to the micro USB port.
|
||||||
|
|
||||||
*Note*: 5V, 700mA regulated power supply
|
*Note*: Use a 5V, 700mA+ regulated power supply
|
||||||
|
|
||||||
5. Wait until you do not see any blinking green light on the board, or if you
|
5. Wait until you do not see any blinking green light on the board, or if you
|
||||||
connected the HDMI cable, check the screen
|
connected the HDMI cable, check the screen. The process is slow and can take
|
||||||
it's slow and can take 30-60 minutes depending on how many document
|
30-60 minutes depending on how many document conversions take place.
|
||||||
conversions take place
|
6. Power off the device and disconnect the drives.
|
||||||
6. Power off the device and disconnect the drives
|
|
||||||
|
|
||||||
Helper scripts
|
Helper scripts
|
||||||
==============
|
==============
|
||||||
|
|
||||||
You should use them as example when you are creating a new image and probably not
|
You should use them as examples when you are creating a new image and probably not
|
||||||
run them blindly as you will most probably have to change constraints accordingly to
|
run them blindly as you will most probably have to change parameters accordingly to
|
||||||
your configuration.
|
your configuration.
|
||||||
|
|
||||||
IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE.
|
IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE.
|
||||||
|
|
||||||
* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card)
|
* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card)
|
||||||
* prepare_rPI_builder.sh: update the system, add the repositories and install all
|
* prepare_rPI.sh: update the system, some configuration
|
||||||
the dependencies needed to compile poppler and pdf2htmlEX on a rPi
|
|
||||||
* update_builder.sh: compile the latest version of poppler from debian experimental,
|
|
||||||
pull and compile the latest version of pdf2htmlEX from the git repository
|
|
||||||
* prepare_rPI.sh: update the system, install the dependencies of poppler and pdf2htmlEX,
|
|
||||||
install poppler and pdf2htmlEX (the deb packages compiled in the builder)
|
|
||||||
* create_user.sh: create the user who will run the scripts, assign the proper sudo rights.
|
* create_user.sh: create the user who will run the scripts, assign the proper sudo rights.
|
||||||
* copy_to_final.sh: populate the content of the directory fs/ in the image,
|
* copy_to_final.sh: populate the content of the directory fs/ in the image,
|
||||||
contains a sample of dd command to write the image on the SD card.
|
contains a sample of dd command to write the image on the SD card.
|
||||||
*TAKE CARE NOT USING THE WRONG DESTINATION*
|
NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -3,16 +3,16 @@ Building the image from scratch
|
||||||
|
|
||||||
There is always a prebuilt image available for download and installation as
|
There is always a prebuilt image available for download and installation as
|
||||||
described in the [README](README.md). If you'd like to build the project yourself,
|
described in the [README](README.md). If you'd like to build the project yourself,
|
||||||
there are several steps:
|
there are several steps involved:
|
||||||
|
|
||||||
* Downloading a generic Raspbian Lite image.
|
* Downloading a generic Raspbian Lite image
|
||||||
* Adding space to the downloaded image.
|
* Adding space to the image
|
||||||
* Downloading and building the dependencies.
|
* Downloading and building the dependencies
|
||||||
* Copying the project filesystem into the image.
|
* Copying the project filesystem into the image
|
||||||
|
|
||||||
This procedure will not work unless you are running Ubuntu or Debian linux. If you
|
This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or
|
||||||
only have access to MacOS or Windows, the best option is to install linux in a
|
Windows, the best option is to install Linux in a virtual machine using
|
||||||
VM, using something like VirtualBox.
|
something like VirtualBox.
|
||||||
|
|
||||||
Downloading the Raspbian image
|
Downloading the Raspbian image
|
||||||
==============================
|
==============================
|
||||||
|
@ -87,17 +87,17 @@ larger than it was before (6852607 vs. 2658303 in the example).
|
||||||
Syncing disks.
|
Syncing disks.
|
||||||
```
|
```
|
||||||
|
|
||||||
* Mount the image in loop mode. Edit /mount_image.sh to contain the proper values
|
* Mount the image in loop mode: first, edit /mount_image.sh to use the proper values
|
||||||
for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as
|
for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as
|
||||||
shown above. You must also change $IMAGE to the correct string. Then run:
|
shown above. You must also change $IMAGE to the correct path. Then run:
|
||||||
|
|
||||||
```
|
```
|
||||||
sudo ./proper_chroot.sh
|
sudo ./proper_chroot.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
* After mounting the image, the above script will chroot into the mounted image.
|
* After mounting the image, the above script will chroot into the mounted image.
|
||||||
While in a chroot, the / directory is treated like it is the / directory (thus
|
While in a chroot, the / directory of the image appears as the system / directory
|
||||||
the name, change root). To exit the chroot, run "exit" in the root directory.
|
(thus the name, change root). To exit the chroot, run "exit" in the root directory.
|
||||||
Then, verify the path to the mounted partitions, and resize the filesystem
|
Then, verify the path to the mounted partitions, and resize the filesystem
|
||||||
to fill the new larger partition using resize2fs:
|
to fill the new larger partition using resize2fs:
|
||||||
|
|
||||||
|
@ -183,7 +183,7 @@ Write the image on a SD card
|
||||||
```
|
```
|
||||||
|
|
||||||
* If it has been automatically mounted, unmount the SD card (use the path you
|
* If it has been automatically mounted, unmount the SD card (use the path you
|
||||||
found in the previous step:
|
found in the previous step):
|
||||||
|
|
||||||
```
|
```
|
||||||
umount $PATH_TO_YOUR_SD
|
umount $PATH_TO_YOUR_SD
|
|
@ -10,6 +10,9 @@
|
||||||
# bits.
|
# bits.
|
||||||
#
|
#
|
||||||
# By default this script does nothing.
|
# By default this script does nothing.
|
||||||
|
#
|
||||||
|
# The above are the default comments for rc.local. For Circlean, rc.local has
|
||||||
|
# been modified to start the grooming process on boot.
|
||||||
|
|
||||||
clean(){
|
clean(){
|
||||||
echo 'Rc Local done, quit.'
|
echo 'Rc Local done, quit.'
|
||||||
|
|
|
@ -10,16 +10,6 @@ apt-get autoremove
|
||||||
apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount
|
apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount
|
||||||
dpkg -i pdf2htmlex*.deb
|
dpkg -i pdf2htmlex*.deb
|
||||||
|
|
||||||
# Make Libreoffice usable on a RO filesystem
|
|
||||||
useradd -m kitten
|
|
||||||
pushd /home/kitten
|
|
||||||
ln -s /tmp/libreoffice
|
|
||||||
popd
|
|
||||||
|
|
||||||
chown -R kitten:kitten /home/kitten
|
|
||||||
|
|
||||||
ln -s /proc/mounts /etc/mtab
|
|
||||||
|
|
||||||
# Disable swap
|
# Disable swap
|
||||||
dphys-swapfile uninstall
|
dphys-swapfile uninstall
|
||||||
|
|
||||||
|
|
|
@ -1,14 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# change locales to en_US.UTF-8
|
|
||||||
dpkg-reconfigure locales
|
|
||||||
|
|
||||||
# Increase size of image. See resize_img.md
|
|
||||||
|
|
||||||
apt-get update
|
|
||||||
apt-get dist-upgrade
|
|
||||||
apt-get autoremove
|
|
||||||
|
|
||||||
# build dependencies of pdf2htmlEX
|
|
||||||
apt-get install cmake debhelper libpoppler-dev libjpeg-dev libfontforge-dev \
|
|
||||||
libspiro-dev python-dev default-jre-headless libpoppler-private-dev
|
|
|
@ -1,11 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# We cannot use the version 0.12 because it requires fontforge 2.0
|
|
||||||
# The fork use a saner list of dependencies and a patch that allows to build on debian jessie.
|
|
||||||
|
|
||||||
wget https://github.com/Rafiot/pdf2htmlEX/archive/KittenGroomer.zip
|
|
||||||
unzip KittenGroomer.zip
|
|
||||||
|
|
||||||
cd pdf2htmlEX-KittenGroomer/
|
|
||||||
|
|
||||||
dpkg-buildpackage -uc -b
|
|
Loading…
Reference in New Issue