CIRCL
/
Circlean
mirror of https://github.com/CIRCL/Circlean
2
0
Fork 0
Code Issues Releases Wiki Activity

Update readme, changelog, and contributing

pull/49/head
Dan Puttick 2017-02-07 20:33:22 -05:00
parent a2f2d39582
commit 58ae576343
6 changed files with 301 additions and 245 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
CHANGELOG.md
View File

@ -1,6 +1,9 @@
Version 2.1 - 2017-02-XX
- Updated to the newest version of Raspbian Jessie lite (January 11th 2017 release)
- NTFS files can be mounted
- NTFS files can now be mounted as source or destination keys
- Added udev rules that ensure the USB ports map deterministically to source and destination keys
- New debug flag and debug logging functionality to make working on Circlean without a monitor easier
- Turned off automatic display sleep
Version 2.0.2 - 2016-05-12
- Improve filename encoding

51
CONTRIBUTING.md
View File

@ -1,17 +1,52 @@
Building the project
====================
Getting started
===============
To get started contributing to Circlean, first, fork the project and `git clone`
your fork. Then, follow the instructions in [README_setup.md](README_setup.md)
to build an image.
If you'd like to work on the Python code that processes files for Circlean, you should
take a look at [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean), specifically the
filecheck.py script. To get started contributing to Circlean, first, fork the project and
`git clone` your fork. Then, follow the instructions in [setup_with_proot.md](doc/
setup_with_proot.md) to build an image. To make things easier, you can also download a
prebuilt image as mentioned in the README, and then mount and make modifications to this
image to test your changes.
The issue tracker
=================
If you find a bug or see a problem with PyCIRCLean, please open an issue in the Github
repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a solution
to any of the open issues - we'll do our best to review your pull request in a timely manner.
This project is in active development, so any contributions are welcome!
repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a
solution to any of the open issues - we'll do our best to review your pull request in a
timely manner. This project is in active development, so any contributions are welcome!
Dependencies
============
* Timidity for playing midi files
* Git for installing some Python dependencies
* 7Zip for unpacking archives
* Pmount and ntfs-3g for mounting usb key partitions
* Python 3 and pip for installing and running Python dependencies
* Python3-lxml for handling ooxml and other Office files in filecheck.py
* libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev for various image formats (
dependencies for pillow)
* Exifread for file metadata
* Pillow for handling images
* Olefile, oletools, and officedissector for handling various Office filetypes
* PyCIRCLean for main file handling code
Helper scripts
==============
Use the scripts in shell_utils/ as examples - do not run them blindly as you will most
probably have to change some constants/paths accordingly to your configuration.
IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE.
* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card)
* prepare_rPI.sh: update the system, some configuration
* create_user.sh: create the user who will run the scripts, assign the proper sudo rights.
* copy_to_final.sh: populate the content of the directory fs/ in the image,
contains a sample of dd command to write the image on the SD card.
NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION
Running the tests
=================

41
README.md
View File

@ -3,17 +3,20 @@ CIRCLean
![CIRCLean logo](https://www.circl.lu/assets/images/logos/circlean.png)
![Cleaner in action](http://www.circl.lu/assets/images/CIRCLean/CIRCLean.png)
How To
======
How To Install
==============
[Graphical how-to and pre-built image](http://circl.lu/projects/CIRCLean/).
[Graphical how-to and pre-built image download](http://circl.lu/projects/CIRCLean/).
To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourceforge.net/projects/win32diskimager/).
To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourceforge.net/
projects/win32diskimager/). On linux/macOS, use dd (see the how-to link for
instructions).
And the linux way is in the command line, via dd (see in copy_to_final.sh)
The current prebuilt image is based on the 1-11-17 release of Raspbian Jessie Lite.
The smallest SD card that Circlean can fit on is currently 4GB.
If you'd like to contribute to the project or build the image yourself, see
[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md).
[contributing](CONTRIBUTING.md) and the [setup instructions](doc/setup_with_proot.md).
This is a work in progress - contributions are welcome.
Why/What
@ -21,11 +24,15 @@ Why/What
This project aims to be useful when you get/find a USB key that you can't trust,
and you want to look at its contents without taking the risk of plugging it into
your computer directly.
your computer directly. The official project page can be found at [https://www.circl.lu/projects/CIRCLean/]
The Raspberry Pi Foundation blog has a [post](https://www.raspberrypi.org/blog/kittengroomercirclean-data-security-for-journalists-and-activists/) with more information
about an older version of the project and details of the inspiration behind it.
CIRCLean is currently tested to work with USB keys that have FAT32, NTFS, or
ext2/3/4 filesystems. Currently, exFAT is not supported due to lack of support for
this format in pmount. The vast majority of USB keys will be FAT32 or NTFS.
ext2/3/4 filesystems (ext* filesystems can only be used as source keys, not destination
keys). Currently, exFAT is not supported due to lack of support for this format in pmount.
The vast majority of USB keys will be FAT32 or NTFS.
The content of the untrusted key will be copied or/and converted to the second
(blank) key following these rules (based on the mime type as determined bylibmagic):
@ -71,19 +78,3 @@ there are more than 2).
connected the HDMI cable, check the screen. The process is slow and can take
30-60 minutes depending on how many document conversions take place.
6. Power off the device and disconnect the drives.
Helper scripts
==============
You should use them as examples when you are creating a new image and probably not
run them blindly as you will most probably have to change parameters accordingly to
your configuration.
IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE.
* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card)
* prepare_rPI.sh: update the system, some configuration
* create_user.sh: create the user who will run the scripts, assign the proper sudo rights.
* copy_to_final.sh: populate the content of the directory fs/ in the image,
contains a sample of dd command to write the image on the SD card.
NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION

196
doc/README_setup.md
View File

@ -1,196 +0,0 @@
Building the image from scratch
===============================
There is always a prebuilt image available for download and installation as
described in the [README](README.md). If you'd like to build the project yourself,
there are several steps involved:
* Downloading a generic Raspbian Lite image
* Adding space to the image
* Downloading and building the dependencies
* Copying the project filesystem into the image
This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or
Windows, the best option is to install Linux in a virtual machine using
something like VirtualBox.
Downloading the Raspbian image
==============================
* Get the most recent version of Raspbian Jessie Lite:
```
wget https://downloads.raspberrypi.org/raspbian_lite_latest
```
* Unpack it:
```
unzip XXXX-XX-XX-raspbian-jessie-lite.zip
```
Adding space to the image
=========================
* Use dd to add 2GB (2048 blocks at 1024k each). Using /dev/zero as the input
file yields an unlimited number of "0x00" bytes.
```
> dd if=/dev/zero bs=1024k count=2048 >> XXXX-XX-XX-raspbian-jessie-lite.img
```
* Grow the root partition using fdisk. The "p" command prints the current partition
table. The first partition listed is the boot partition, which shouldn't be changed.
The "d" command, when given the parameter "2", deletes the current root partition.
The "n" command then makes a new partition. It can take the default for "type"
and "number". The "First sector" should be the value that was the "start" sector of the root
partition (131072 in the example below, but this varies depending on the version of the
Raspbian image). The "Last sector" should be the default, and it should be significantly
larger than it was before (6852607 vs. 2658303 in the example).
```
> fdisk XXXX-XX-XX-raspbian-jessie-lite.img
Command (m for help): *p*
Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x244b8248
Device Boot Start End Sectors Size Id Type
XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA)
XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux
Command (m for help): *d*
Partition number (1,2, default 2): *2*
Partition 2 has been deleted.
Command (m for help): *n*
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p):
Using default response p.
Partition number (2-4, default 2):
First sector (2048-6852607, default 2048): *137216*
Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951):
Created a new partition 2 of type 'Linux' and of size 3.2 GiB.
Command (m for help): *w*
The partition table has been altered.
Syncing disks.
```
* Mount the image in loop mode: first, edit /mount_image.sh to use the proper values
for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as
shown above. You must also change $IMAGE to the correct path. Then run:
```
sudo ./proper_chroot.sh
```
* After mounting the image, the above script will chroot into the mounted image.
While in a chroot, the / directory of the image appears as the system / directory
(thus the name, change root). To exit the chroot, run "exit" in the root directory.
Then, verify the path to the mounted partitions, and resize the filesystem
to fill the new larger partition using resize2fs:
```
> df | grep /mnt/arm
/dev/loop0 3927752 1955672 1794172 53% /mnt/arm_rPi
/dev/loop1 57288 18960 38328 34% /mnt/arm_rPi/boot
> sudo resize2fs /dev/loop0
```
Installing the dependencies
===========================
* To install the dependencies, you'll have to reenter the chroot again:
```
sudo chroot /mnt/arm_rPi
```
* Change your user to root (your global variables may be broken as a result):
```
su root
```
* Change the locales (remove "en_GB.UTF-8 UTF-8", add "en_US.UTF-8 UTF-8"). The
arrow keys move the cursor, spacebar selects/deselects a locale, tab moves the cursor
to a different context, and enter lets you select "ok":
```
dpkg-reconfigure locales
```
* In the image, make sure everything is up-to-date and remove the old packages:
```
apt-get update
apt-get dist-upgrade
apt-get autoremove
apt-get install timidity git p7zip-full python-dev python-pip python-lxml pmount libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev tk-dev python-tk libxml2-dev libxslt1-dev
```
* Install the Python dependencies for PyCIRCLean. Currently, PyCIRCLean is
Python 2.7 and 3.3+ compatible, but Python 2 support might be dropped at some point.
```
pip install oletools olefile exifread Pillow
pip install git+https://github.com/Rafiot/officedissector.git
pip install git+https://github.com/CIRCL/PyCIRCLean.git
```
* Create a new user and make mounting work with a read-only filesystem.
```
useradd -m kitten
chown -R kitten:kitten /home/kitten
ln -s /proc/mounts /etc/mtab
```
* Enable rc.local, which ensures that the code in /etc/rc.local is run on boot.
This is what triggers CIRCLean to run.
```
systemctl enable rc-local.service
```
* Exit the chroot again, and copy the files from your repository into the mounted
image.
```
sudo ./copy_to_final.sh /mnt/arm_rPi/
```
Write the image on a SD card
============================
* Plug your SD card into the computer. Then, find where it is mounted using df:
```
df -h
```
* If it has been automatically mounted, unmount the SD card (use the path you
found in the previous step):
```
umount $PATH_TO_YOUR_SD
```
* Write the image to the card:
```
sudo dd bs=4M if=$PATH_TO_YOUR_IMAGE of=$PATH_TO_YOUR_SD
```

29
doc/image-setup-checklist.md
View File

@ -1,24 +1,25 @@
* Download qemu and qemu-user-static if not already installed
* Download qemu, qemu-user-static, and proot if not already installed
* Download the newest raspbian-lite image from raspberrypi.org
* Verify the sha1 hash of the downloaded .zip file
* Unzip the image
* Expand the image by 2GB using dd
* Expand the root partition using fdisk
* Mount both partitions in loop mode using /shell_utils/simple_mount_image.sh
* Mount both partitions in loop mode using /shell_utils/basic_mount_image.sh
* Use df to find the larger partition, and resize the filesystem to fill it
* Copy circlean_fs/root_partition/etc/systemd/system/rc-local.service into the equivalent location
* Use proot to enter a chroot in the image: sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash
* Run dpkg-reconfigure locales (this step + others using proot + qemu can be slow, be patient)
* Run dpkg-reconfigure locales
* apt-get update
* apt-get dist-upgrade (might have to run this and autoremove several times)
* apt-get autoremove
* apt-get install the linux dependencies:
- timidity # for playing music
- git # for installing python dependencies from github
- timidity
- git
- p7zip-full
- pmount ntfs-3g # for mounting, including ntfs
- pmount ntfs-3g
- python3 python3-pip
- python3-lxml
- libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev # dependencies for building pillow
- libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev
* Compile p7zip-rar from source
- Change your source.list file
- Make a new directory and cd to it
@ -35,21 +36,19 @@
- git+https://github.com/CIRCL/PyCIRCLean.git
* Add a user named "kitten"
* Symlink /proc/mounts to /etc/mtab
* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location
* Turn on rc-local.service `systemctl enable rc.local`
* Turn on rc-local.service `systemctl enable rc-local.service`
- If it doesn't work, read these instructions: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd
* apt-get autoclean
* apt-get autoremove
* Exit from proot
* Copy all of the project files from circlean_fs/ into the two partitions:
- rsync -vnri <source> <destination> will do a dry run of what will be copied, remove the -n to copy. See the rsync manpage for details.
- diode_controller/ if you're using the led functionality and have an external led
- midi/ files into /opt/midi/
- you might want to double check all of the permissions of the new files/directories
* apt-get autoclean
* apt-get autoremove
* Exit the chroot
* Copy the image over to the SD card: sudo dd bs=4M if=<image> of=/dev/sd<letter>
- In newer versions of dd, you can add status=progress
* Mount the image
* Optional: fsck the root partition (sudo e2fsck -f /dev/sd<letter>2).
* Test with an rpi
- FAT32 partition
- NTFS partition
- FAT32 filesystem
- NTFS filesystem

224
doc/setup_with_proot.md Normal file
View File

@ -0,0 +1,224 @@
Building the image from scratch
===============================
There is always a prebuilt image available for download and installation as
described in the [README](README.md). If you'd like to build the project yourself,
there are several steps involved:
* Downloading a generic Raspbian Lite image
* Resizing the image and partition
* Downloading and building the dependencies
* Modifying the image configuration
* Copying the project filesystem into the image
This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or
Windows, the best option is to install Linux in a virtual machine using
something like VirtualBox.
It is recommended that you make a copy of image_setup_checklist.md and √ items off
on the list as you go.
Preparation
===========
* Make sure your development environment is up to date:
```
apt-get update
apt-get dist-upgrade
```
* Install qemu, qemu-user-static, and proot if not already installed:
```
apt-get install qemu qemu-user-static proot
```
Download the Raspbian image
==============================
* Get the most recent version of Raspbian Jessie Lite from https://downloads.raspberrypi.org/raspbian_lite/images/:
```
wget https://downloads.raspberrypi.org/raspbian_lite_latest
```
* Verify the hash of the downloaded file and compare it to the hash on the server:
```
shasum XXXX-XX-XX-raspbian-jessie-lite.zip
```
* Unpack it:
```
unzip XXXX-XX-XX-raspbian-jessie-lite.zip
```
Add space to the image
=========================
* Use dd to add 2GB (2048 blocks of 1024k each). Using /dev/zero as the input
file yields an unlimited number of "0x00" bytes.
```
> dd if=/dev/zero bs=1024k count=2048 >> XXXX-XX-XX-raspbian-jessie-lite.img
```
* Expand the root (second) partition using fdisk. The first partition listed is the boot
partition, which shouldn't be changed. In the new partition, the "First sector" should be
the value that was the "start" sector of the old root partition (137216 in the example
below, but this varies depending on the version of the Raspbian image). The "Last sector"
should be the default, and it should be significantly larger than it was before (6909951 vs.
2715647 in the example).
```
> fdisk XXXX-XX-XX-raspbian-jessie-lite.img
Command (m for help): *p*
Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x244b8248
Device Boot Start End Sectors Size Id Type
XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA)
XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux
Command (m for help): *d*
Partition number (1,2, default 2): *2*
Partition 2 has been deleted.
Command (m for help): *n*
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p):
Using default response p.
Partition number (2-4, default 2):
First sector (2048-6852607, default 2048): *137216*
Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951):
Created a new partition 2 of type 'Linux' and of size 3.2 GiB.
Command (m for help): *w*
The partition table has been altered.
Syncing disks.
```
* Mount the image in loop mode: first, edit shell_utils/basic_mount_image.sh to use the
proper values for $BOOT_START and $ROOT_START, which you can obtain using fdisk and "p"
as in the previous step. You must also change $IMAGE to the correct path. Then run:
```
sudo ./shell_utils/basic_mount_image.md
```
* Verify the path to the mounted partitions in /dev, and resize the root (larger) filesystem
to fill the new larger partition using resize2fs:
```
> df | grep /mnt/arm
/dev/loop0 3927752 1955672 1794172 53% /mnt/rpi-root
/dev/loop1 57288 18960 38328 34% /mnt/rpi-boot
> sudo resize2fs /dev/loop0
```
Installing the dependencies
===========================
* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location in the image.
```
cp circlean_fs/root_partition/systemd/system/rc-local.service /mnt/rpi-root/etc/systemd/system/rc-local.service
```
* Use [proot](https://proot-me.github.io/) to enter the equivalent of a chroot inside
the mounted image.
```
sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash
```
* Change your locales (remove "en_GB.UTF-8 UTF-8", add "en_US.UTF-8 UTF-8"). The
arrow keys move the cursor, spacebar selects/deselects a locale, tab moves the cursor
to a different context, and enter lets you select "ok". This step might take some time,
be patient:
```
dpkg-reconfigure locales
```
* In the image, make sure everything is up-to-date and remove old packages. You may have to
run dist-upgrade and autoremove several times for everything to be installed, and a few
raspbian-sys-mods related installs may fail - you can ignore them:
```
apt-get update
apt-get dist-upgrade
apt-get autoremove
```
* Install the linux dependencies (see CONTRIBUTING.md for more details):
```
apt-get install timidity git p7zip-full python3 python3-pip python3-lxml pmount ntfs-3g libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev
```
* Compile p7zip-rar from source. First, uncomment out the second line in /etc/apt/sources.list. Then:
```
cd /home/pi
mkdir rar && cd rar/
apt-get build-dep p7zip-rar
dpkg -i ${path to p7zip-rar .deb file}
```
* Install the Python dependencies for PyCIRCLean/filecheck.py. PyCIRCLean is 3.3+
compatible, so use pip -V to make sure you're using the right version of pip. You might
have to edit your PATH variable or use pip3 to get the correct pip. You also might want to
verify that these dependencies are current by checking in the PyCIRCLean git repo.
```
pip install -U pip
pip install oletools exifread pillow
pip install git+https://github.com/decalage2/oletools.git
pip install git+https://github.com/Rafiot/officedissector.git
pip install git+https://github.com/CIRCL/PyCIRCLean.git
```
* Create a new user named "kitten":
```
useradd -m kitten
chown -R kitten:kitten /home/kitten
```
* Symlinking /proc/mounts to /etc/mtab is necessary because /etc/mtab cannot be edited by
pmount if root is read-only. /proc/mounts is maintained by the kernel and is guaranteed to
be accurate.
```
ln -s /proc/mounts /etc/mtab
```
* Enable rc.local, which ensures that the code in /etc/rc.local is run on boot.
This is what triggers CIRCLean to run.
```
systemctl enable rc-local.service
```
* Clean up:
```
apt-get clean
apt-get autoremove
apt-get autoclean
```
* Exit proot, and copy the files from your repository into the mounted
image. Adding a -n flag will make rsync do a dry run instead of copying. See the rsync
manpage for more details. Make sure to include the trailing slashes on the paths:
```
exit
sudo rsync -vri circlean_fs/boot/ /mnt/rpi-boot/
sudo rsync -vri circlean_fs/root_partition/ /mnt/rpi-root/
cp -rf midi /mnt/rpi-root/opt/
```
* If have an external hardware led and you're using the led functionality, copy
the led files from diode_controller/ as well.
Write the image on a SD card
============================
* Plug your SD card into the computer. Then, find where it is mounted using lsblk or df:
```
lsblk
df -h
```
* If it has been automatically mounted, unmount the SD card (use the path you
found in the previous step):
```
umount $PATH_TO_YOUR_SD
```
* Write the image to the card. Newer versions of dd include a status option to monitor the
copying process:
```
sudo dd bs=4M if=$PATH_TO_YOUR_IMAGE of=$PATH_TO_YOUR_SD status=progress
```
* Use fsck to verify the root partition:
```
sudo e2fsck -f /dev/sd<number>2
```