From 58ae57634334c27db4f067da6f0048c304e99504 Mon Sep 17 00:00:00 2001 From: Dan Puttick Date: Tue, 7 Feb 2017 20:33:22 -0500 Subject: [PATCH] Update readme, changelog, and contributing --- CHANGELOG.md | 5 +- CONTRIBUTING.md | 51 ++++++-- README.md | 41 +++---- doc/README_setup.md | 196 ------------------------------ doc/image-setup-checklist.md | 29 +++-- doc/setup_with_proot.md | 224 +++++++++++++++++++++++++++++++++++ 6 files changed, 301 insertions(+), 245 deletions(-) delete mode 100644 doc/README_setup.md create mode 100644 doc/setup_with_proot.md diff --git a/CHANGELOG.md b/CHANGELOG.md index cc12256..15dd536 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,9 @@ Version 2.1 - 2017-02-XX - Updated to the newest version of Raspbian Jessie lite (January 11th 2017 release) -- NTFS files can be mounted +- NTFS files can now be mounted as source or destination keys +- Added udev rules that ensure the USB ports map deterministically to source and destination keys +- New debug flag and debug logging functionality to make working on Circlean without a monitor easier +- Turned off automatic display sleep Version 2.0.2 - 2016-05-12 - Improve filename encoding diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2ed1ee4..b879e74 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,17 +1,52 @@ -Building the project -==================== +Getting started +=============== -To get started contributing to Circlean, first, fork the project and `git clone` -your fork. Then, follow the instructions in [README_setup.md](README_setup.md) -to build an image. +If you'd like to work on the Python code that processes files for Circlean, you should +take a look at [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean), specifically the +filecheck.py script. To get started contributing to Circlean, first, fork the project and +`git clone` your fork. Then, follow the instructions in [setup_with_proot.md](doc/ +setup_with_proot.md) to build an image. To make things easier, you can also download a +prebuilt image as mentioned in the README, and then mount and make modifications to this +image to test your changes. The issue tracker ================= If you find a bug or see a problem with PyCIRCLean, please open an issue in the Github -repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a solution -to any of the open issues - we'll do our best to review your pull request in a timely manner. -This project is in active development, so any contributions are welcome! +repo. We'll do our best to respond as quickly as possible. Also, feel free to contribute a +solution to any of the open issues - we'll do our best to review your pull request in a +timely manner. This project is in active development, so any contributions are welcome! + +Dependencies +============ +* Timidity for playing midi files +* Git for installing some Python dependencies +* 7Zip for unpacking archives +* Pmount and ntfs-3g for mounting usb key partitions +* Python 3 and pip for installing and running Python dependencies +* Python3-lxml for handling ooxml and other Office files in filecheck.py +* libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev for various image formats ( +dependencies for pillow) +* Exifread for file metadata +* Pillow for handling images +* Olefile, oletools, and officedissector for handling various Office filetypes +* PyCIRCLean for main file handling code + +Helper scripts +============== + +Use the scripts in shell_utils/ as examples - do not run them blindly as you will most +probably have to change some constants/paths accordingly to your configuration. + +IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE. + +* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card) +* prepare_rPI.sh: update the system, some configuration +* create_user.sh: create the user who will run the scripts, assign the proper sudo rights. +* copy_to_final.sh: populate the content of the directory fs/ in the image, + contains a sample of dd command to write the image on the SD card. + NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION + Running the tests ================= diff --git a/README.md b/README.md index 3ffacce..e9040e5 100644 --- a/README.md +++ b/README.md @@ -3,17 +3,20 @@ CIRCLean ![CIRCLean logo](https://www.circl.lu/assets/images/logos/circlean.png) ![Cleaner in action](http://www.circl.lu/assets/images/CIRCLean/CIRCLean.png) -How To -====== +How To Install +============== -[Graphical how-to and pre-built image](http://circl.lu/projects/CIRCLean/). +[Graphical how-to and pre-built image download](http://circl.lu/projects/CIRCLean/). -To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourceforge.net/projects/win32diskimager/). +To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourceforge.net/ +projects/win32diskimager/). On linux/macOS, use dd (see the how-to link for +instructions). -And the linux way is in the command line, via dd (see in copy_to_final.sh) +The current prebuilt image is based on the 1-11-17 release of Raspbian Jessie Lite. +The smallest SD card that Circlean can fit on is currently 4GB. If you'd like to contribute to the project or build the image yourself, see -[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md). +[contributing](CONTRIBUTING.md) and the [setup instructions](doc/setup_with_proot.md). This is a work in progress - contributions are welcome. Why/What @@ -21,11 +24,15 @@ Why/What This project aims to be useful when you get/find a USB key that you can't trust, and you want to look at its contents without taking the risk of plugging it into -your computer directly. +your computer directly. The official project page can be found at [https://www.circl.lu/projects/CIRCLean/] + +The Raspberry Pi Foundation blog has a [post](https://www.raspberrypi.org/blog/kittengroomercirclean-data-security-for-journalists-and-activists/) with more information +about an older version of the project and details of the inspiration behind it. CIRCLean is currently tested to work with USB keys that have FAT32, NTFS, or -ext2/3/4 filesystems. Currently, exFAT is not supported due to lack of support for -this format in pmount. The vast majority of USB keys will be FAT32 or NTFS. +ext2/3/4 filesystems (ext* filesystems can only be used as source keys, not destination +keys). Currently, exFAT is not supported due to lack of support for this format in pmount. +The vast majority of USB keys will be FAT32 or NTFS. The content of the untrusted key will be copied or/and converted to the second (blank) key following these rules (based on the mime type as determined bylibmagic): @@ -71,19 +78,3 @@ there are more than 2). connected the HDMI cable, check the screen. The process is slow and can take 30-60 minutes depending on how many document conversions take place. 6. Power off the device and disconnect the drives. - -Helper scripts -============== - -You should use them as examples when you are creating a new image and probably not -run them blindly as you will most probably have to change parameters accordingly to -your configuration. - -IN ALL CASES, PLEASE READ THE COMMENTS IN THE SCRIPTS AT LEAST ONCE. - -* proper_chroot.sh: uses qemu to chroot into a raspbian instance (.img or SD Card) -* prepare_rPI.sh: update the system, some configuration -* create_user.sh: create the user who will run the scripts, assign the proper sudo rights. -* copy_to_final.sh: populate the content of the directory fs/ in the image, - contains a sample of dd command to write the image on the SD card. - NOTE: TAKE CARE NOT TO USE THE WRONG DESTINATION diff --git a/doc/README_setup.md b/doc/README_setup.md deleted file mode 100644 index 6e6c157..0000000 --- a/doc/README_setup.md +++ /dev/null @@ -1,196 +0,0 @@ -Building the image from scratch -=============================== - -There is always a prebuilt image available for download and installation as -described in the [README](README.md). If you'd like to build the project yourself, -there are several steps involved: - -* Downloading a generic Raspbian Lite image -* Adding space to the image -* Downloading and building the dependencies -* Copying the project filesystem into the image - -This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or -Windows, the best option is to install Linux in a virtual machine using -something like VirtualBox. - -Downloading the Raspbian image -============================== - -* Get the most recent version of Raspbian Jessie Lite: - -``` - wget https://downloads.raspberrypi.org/raspbian_lite_latest -``` - -* Unpack it: - -``` - unzip XXXX-XX-XX-raspbian-jessie-lite.zip -``` - -Adding space to the image -========================= - -* Use dd to add 2GB (2048 blocks at 1024k each). Using /dev/zero as the input -file yields an unlimited number of "0x00" bytes. - -``` - > dd if=/dev/zero bs=1024k count=2048 >> XXXX-XX-XX-raspbian-jessie-lite.img -``` - -* Grow the root partition using fdisk. The "p" command prints the current partition -table. The first partition listed is the boot partition, which shouldn't be changed. -The "d" command, when given the parameter "2", deletes the current root partition. -The "n" command then makes a new partition. It can take the default for "type" -and "number". The "First sector" should be the value that was the "start" sector of the root -partition (131072 in the example below, but this varies depending on the version of the -Raspbian image). The "Last sector" should be the default, and it should be significantly -larger than it was before (6852607 vs. 2658303 in the example). - - -``` - > fdisk XXXX-XX-XX-raspbian-jessie-lite.img - - Command (m for help): *p* - Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors - Units: sectors of 1 * 512 = 512 bytes - Sector size (logical/physical): 512 bytes / 512 bytes - I/O size (minimum/optimal): 512 bytes / 512 bytes - Disklabel type: dos - Disk identifier: 0x244b8248 - - Device Boot Start End Sectors Size Id Type - XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA) - XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux - - Command (m for help): *d* - Partition number (1,2, default 2): *2* - - Partition 2 has been deleted. - - Command (m for help): *n* - Partition type - p primary (1 primary, 0 extended, 3 free) - e extended (container for logical partitions) - Select (default p): - - Using default response p. - Partition number (2-4, default 2): - First sector (2048-6852607, default 2048): *137216* - Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951): - - Created a new partition 2 of type 'Linux' and of size 3.2 GiB. - - Command (m for help): *w* - The partition table has been altered. - Syncing disks. -``` - -* Mount the image in loop mode: first, edit /mount_image.sh to use the proper values -for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as -shown above. You must also change $IMAGE to the correct path. Then run: - -``` - sudo ./proper_chroot.sh -``` - -* After mounting the image, the above script will chroot into the mounted image. -While in a chroot, the / directory of the image appears as the system / directory -(thus the name, change root). To exit the chroot, run "exit" in the root directory. -Then, verify the path to the mounted partitions, and resize the filesystem -to fill the new larger partition using resize2fs: - -``` - > df | grep /mnt/arm - - /dev/loop0 3927752 1955672 1794172 53% /mnt/arm_rPi - /dev/loop1 57288 18960 38328 34% /mnt/arm_rPi/boot - - > sudo resize2fs /dev/loop0 -``` - -Installing the dependencies -=========================== - -* To install the dependencies, you'll have to reenter the chroot again: - -``` - sudo chroot /mnt/arm_rPi -``` - -* Change your user to root (your global variables may be broken as a result): - -``` - su root -``` - -* Change the locales (remove "en_GB.UTF-8 UTF-8", add "en_US.UTF-8 UTF-8"). The -arrow keys move the cursor, spacebar selects/deselects a locale, tab moves the cursor -to a different context, and enter lets you select "ok": - -``` - dpkg-reconfigure locales -``` - -* In the image, make sure everything is up-to-date and remove the old packages: - -``` - apt-get update - apt-get dist-upgrade - apt-get autoremove - apt-get install timidity git p7zip-full python-dev python-pip python-lxml pmount libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev tk-dev python-tk libxml2-dev libxslt1-dev -``` - -* Install the Python dependencies for PyCIRCLean. Currently, PyCIRCLean is -Python 2.7 and 3.3+ compatible, but Python 2 support might be dropped at some point. - -``` - pip install oletools olefile exifread Pillow - pip install git+https://github.com/Rafiot/officedissector.git - pip install git+https://github.com/CIRCL/PyCIRCLean.git -``` - -* Create a new user and make mounting work with a read-only filesystem. - -``` - useradd -m kitten - chown -R kitten:kitten /home/kitten - ln -s /proc/mounts /etc/mtab -``` - -* Enable rc.local, which ensures that the code in /etc/rc.local is run on boot. -This is what triggers CIRCLean to run. - -``` - systemctl enable rc-local.service -``` - -* Exit the chroot again, and copy the files from your repository into the mounted -image. - -``` - sudo ./copy_to_final.sh /mnt/arm_rPi/ -``` - -Write the image on a SD card -============================ - -* Plug your SD card into the computer. Then, find where it is mounted using df: - -``` - df -h -``` - -* If it has been automatically mounted, unmount the SD card (use the path you -found in the previous step): - -``` - umount $PATH_TO_YOUR_SD -``` - -* Write the image to the card: - -``` - sudo dd bs=4M if=$PATH_TO_YOUR_IMAGE of=$PATH_TO_YOUR_SD -``` diff --git a/doc/image-setup-checklist.md b/doc/image-setup-checklist.md index e3d1d4e..1a40527 100644 --- a/doc/image-setup-checklist.md +++ b/doc/image-setup-checklist.md @@ -1,24 +1,25 @@ -* Download qemu and qemu-user-static if not already installed +* Download qemu, qemu-user-static, and proot if not already installed * Download the newest raspbian-lite image from raspberrypi.org * Verify the sha1 hash of the downloaded .zip file * Unzip the image * Expand the image by 2GB using dd * Expand the root partition using fdisk -* Mount both partitions in loop mode using /shell_utils/simple_mount_image.sh +* Mount both partitions in loop mode using /shell_utils/basic_mount_image.sh * Use df to find the larger partition, and resize the filesystem to fill it +* Copy circlean_fs/root_partition/etc/systemd/system/rc-local.service into the equivalent location * Use proot to enter a chroot in the image: sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash -* Run dpkg-reconfigure locales (this step + others using proot + qemu can be slow, be patient) +* Run dpkg-reconfigure locales * apt-get update * apt-get dist-upgrade (might have to run this and autoremove several times) * apt-get autoremove * apt-get install the linux dependencies: - - timidity # for playing music - - git # for installing python dependencies from github + - timidity + - git - p7zip-full - - pmount ntfs-3g # for mounting, including ntfs + - pmount ntfs-3g - python3 python3-pip - python3-lxml - - libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev # dependencies for building pillow + - libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev * Compile p7zip-rar from source - Change your source.list file - Make a new directory and cd to it @@ -35,21 +36,19 @@ - git+https://github.com/CIRCL/PyCIRCLean.git * Add a user named "kitten" * Symlink /proc/mounts to /etc/mtab -* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location -* Turn on rc-local.service `systemctl enable rc.local` +* Turn on rc-local.service `systemctl enable rc-local.service` - If it doesn't work, read these instructions: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd +* apt-get autoclean +* apt-get autoremove +* Exit from proot * Copy all of the project files from circlean_fs/ into the two partitions: - rsync -vnri will do a dry run of what will be copied, remove the -n to copy. See the rsync manpage for details. - diode_controller/ if you're using the led functionality and have an external led - midi/ files into /opt/midi/ - you might want to double check all of the permissions of the new files/directories -* apt-get autoclean -* apt-get autoremove -* Exit the chroot * Copy the image over to the SD card: sudo dd bs=4M if= of=/dev/sd - In newer versions of dd, you can add status=progress -* Mount the image * Optional: fsck the root partition (sudo e2fsck -f /dev/sd2). * Test with an rpi - - FAT32 partition - - NTFS partition + - FAT32 filesystem + - NTFS filesystem diff --git a/doc/setup_with_proot.md b/doc/setup_with_proot.md new file mode 100644 index 0000000..edd1b1b --- /dev/null +++ b/doc/setup_with_proot.md @@ -0,0 +1,224 @@ +Building the image from scratch +=============================== + +There is always a prebuilt image available for download and installation as +described in the [README](README.md). If you'd like to build the project yourself, +there are several steps involved: + +* Downloading a generic Raspbian Lite image +* Resizing the image and partition +* Downloading and building the dependencies +* Modifying the image configuration +* Copying the project filesystem into the image + +This procedure will only work on Ubuntu or Debian Linux. If you use MacOS or +Windows, the best option is to install Linux in a virtual machine using +something like VirtualBox. + +It is recommended that you make a copy of image_setup_checklist.md and √ items off +on the list as you go. + +Preparation +=========== + +* Make sure your development environment is up to date: +``` + apt-get update + apt-get dist-upgrade +``` +* Install qemu, qemu-user-static, and proot if not already installed: +``` + apt-get install qemu qemu-user-static proot +``` + +Download the Raspbian image +============================== + +* Get the most recent version of Raspbian Jessie Lite from https://downloads.raspberrypi.org/raspbian_lite/images/: + +``` + wget https://downloads.raspberrypi.org/raspbian_lite_latest +``` +* Verify the hash of the downloaded file and compare it to the hash on the server: +``` + shasum XXXX-XX-XX-raspbian-jessie-lite.zip +``` +* Unpack it: +``` + unzip XXXX-XX-XX-raspbian-jessie-lite.zip +``` + +Add space to the image +========================= + +* Use dd to add 2GB (2048 blocks of 1024k each). Using /dev/zero as the input +file yields an unlimited number of "0x00" bytes. +``` + > dd if=/dev/zero bs=1024k count=2048 >> XXXX-XX-XX-raspbian-jessie-lite.img +``` +* Expand the root (second) partition using fdisk. The first partition listed is the boot +partition, which shouldn't be changed. In the new partition, the "First sector" should be +the value that was the "start" sector of the old root partition (137216 in the example +below, but this varies depending on the version of the Raspbian image). The "Last sector" +should be the default, and it should be significantly larger than it was before (6909951 vs. +2715647 in the example). + +``` + > fdisk XXXX-XX-XX-raspbian-jessie-lite.img + + Command (m for help): *p* + Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 512 bytes + I/O size (minimum/optimal): 512 bytes / 512 bytes + Disklabel type: dos + Disk identifier: 0x244b8248 + + Device Boot Start End Sectors Size Id Type + XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA) + XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux + + Command (m for help): *d* + Partition number (1,2, default 2): *2* + + Partition 2 has been deleted. + + Command (m for help): *n* + Partition type + p primary (1 primary, 0 extended, 3 free) + e extended (container for logical partitions) + Select (default p): + + Using default response p. + Partition number (2-4, default 2): + First sector (2048-6852607, default 2048): *137216* + Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951): + + Created a new partition 2 of type 'Linux' and of size 3.2 GiB. + + Command (m for help): *w* + The partition table has been altered. + Syncing disks. +``` +* Mount the image in loop mode: first, edit shell_utils/basic_mount_image.sh to use the +proper values for $BOOT_START and $ROOT_START, which you can obtain using fdisk and "p" +as in the previous step. You must also change $IMAGE to the correct path. Then run: +``` + sudo ./shell_utils/basic_mount_image.md +``` +* Verify the path to the mounted partitions in /dev, and resize the root (larger) filesystem +to fill the new larger partition using resize2fs: +``` + > df | grep /mnt/arm + + /dev/loop0 3927752 1955672 1794172 53% /mnt/rpi-root + /dev/loop1 57288 18960 38328 34% /mnt/rpi-boot + + > sudo resize2fs /dev/loop0 +``` + +Installing the dependencies +=========================== + +* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location in the image. +``` + cp circlean_fs/root_partition/systemd/system/rc-local.service /mnt/rpi-root/etc/systemd/system/rc-local.service +``` +* Use [proot](https://proot-me.github.io/) to enter the equivalent of a chroot inside +the mounted image. +``` + sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash +``` +* Change your locales (remove "en_GB.UTF-8 UTF-8", add "en_US.UTF-8 UTF-8"). The +arrow keys move the cursor, spacebar selects/deselects a locale, tab moves the cursor +to a different context, and enter lets you select "ok". This step might take some time, +be patient: +``` + dpkg-reconfigure locales +``` +* In the image, make sure everything is up-to-date and remove old packages. You may have to +run dist-upgrade and autoremove several times for everything to be installed, and a few +raspbian-sys-mods related installs may fail - you can ignore them: +``` + apt-get update + apt-get dist-upgrade + apt-get autoremove +``` +* Install the linux dependencies (see CONTRIBUTING.md for more details): +``` + apt-get install timidity git p7zip-full python3 python3-pip python3-lxml pmount ntfs-3g libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev +``` +* Compile p7zip-rar from source. First, uncomment out the second line in /etc/apt/sources.list. Then: +``` + cd /home/pi + mkdir rar && cd rar/ + apt-get build-dep p7zip-rar + dpkg -i ${path to p7zip-rar .deb file} +``` +* Install the Python dependencies for PyCIRCLean/filecheck.py. PyCIRCLean is 3.3+ +compatible, so use pip -V to make sure you're using the right version of pip. You might +have to edit your PATH variable or use pip3 to get the correct pip. You also might want to +verify that these dependencies are current by checking in the PyCIRCLean git repo. +``` + pip install -U pip + pip install oletools exifread pillow + pip install git+https://github.com/decalage2/oletools.git + pip install git+https://github.com/Rafiot/officedissector.git + pip install git+https://github.com/CIRCL/PyCIRCLean.git +``` +* Create a new user named "kitten": +``` + useradd -m kitten + chown -R kitten:kitten /home/kitten +``` +* Symlinking /proc/mounts to /etc/mtab is necessary because /etc/mtab cannot be edited by +pmount if root is read-only. /proc/mounts is maintained by the kernel and is guaranteed to +be accurate. +``` + ln -s /proc/mounts /etc/mtab +``` +* Enable rc.local, which ensures that the code in /etc/rc.local is run on boot. +This is what triggers CIRCLean to run. +``` + systemctl enable rc-local.service +``` +* Clean up: +``` + apt-get clean + apt-get autoremove + apt-get autoclean +``` +* Exit proot, and copy the files from your repository into the mounted +image. Adding a -n flag will make rsync do a dry run instead of copying. See the rsync +manpage for more details. Make sure to include the trailing slashes on the paths: +``` + exit + sudo rsync -vri circlean_fs/boot/ /mnt/rpi-boot/ + sudo rsync -vri circlean_fs/root_partition/ /mnt/rpi-root/ + cp -rf midi /mnt/rpi-root/opt/ +``` +* If have an external hardware led and you're using the led functionality, copy +the led files from diode_controller/ as well. + +Write the image on a SD card +============================ + +* Plug your SD card into the computer. Then, find where it is mounted using lsblk or df: +``` + lsblk + df -h +``` +* If it has been automatically mounted, unmount the SD card (use the path you +found in the previous step): +``` + umount $PATH_TO_YOUR_SD +``` +* Write the image to the card. Newer versions of dd include a status option to monitor the +copying process: +``` + sudo dd bs=4M if=$PATH_TO_YOUR_IMAGE of=$PATH_TO_YOUR_SD status=progress +``` +* Use fsck to verify the root partition: +``` + sudo e2fsck -f /dev/sd2 +```