CIRCL
/
Circlean
mirror of https://github.com/CIRCL/Circlean
2
0
Fork 0
Code Issues Releases Wiki Activity

Add new documentation, move to docs/

pull/49/head
Dan Puttick 2017-01-26 17:54:37 -05:00
parent 51de40f2aa
commit a2f2d39582
12 changed files with 221 additions and 70 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
CHANGELOG
View File

@ -1,26 +0,0 @@
Version 1.2 - 2015-03-10
- Rollback the migration to Jessie and use Wheezy again: the only important dependency from Jessie was poppler, which is available in the backports
- Use the most recent security patches
- Do not wait for user input in case of password protected archive
Version 1.1.1 - 2014-10-26
- General upgrade of Debian to avoid the system to fail in case there is no HDMI cable connected.
Version 1.1 - 2014-10-01
- NTFS support added for USB key
- Updated to Debian Jessie including patches for [bash vulnerabilities CVE-2014-6271 - CVE-2014-7169](/pub/tr-27/)
- CIRCLean user are now removed from the sudoer
Version 1.0 - 2014-05-20
- Based on Raspbian Jessie
- Fully automated tests with Qemu
- Mimetype: support of PDF, Office documents, archives, windows executables
- Filesystem: USB keys have to be formated in vfat
- Support of multiple partitions
- Renaming of autorun.inf on the source key
- Operating system is read only
- Use pdf2htmlEX v0.11

47
CHANGELOG.md Normal file
View File

@ -0,0 +1,47 @@
Version 2.1 - 2017-02-XX
- Updated to the newest version of Raspbian Jessie lite (January 11th 2017 release)
- NTFS files can be mounted
Version 2.0.2 - 2016-05-12
- Improve filename encoding
Version 2.0.1 - 2016-04-26
- Re-add [timidity](http://timidity.sourceforge.net/) so the MIDI files are played properly
Version 2.0 - 2016-04-26
- No critical bugs have been identified, this release uses the latest version of Raspbian Jessie lite, with all system updates
Version 2.0-BETA - 2015-11-06
- There a new beta version of CIRCLean which is a significant improvement from the latest version in term of speed and efficiency on low-end hardware like the first version of the Raspberry Pi. The new code base of CIRCLean is now based on [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean)
Version 1.3 - 2015-05-27
- Fix a [critical security bug](https://www.circl.lu/projects/CIRCLean/security/advisory-01) related to [polyglot files](https://github.com/CIRCL/Circlean/issues/9) - thanks to the reporters ([Jann Horn](https://github.com/thejh), [seclab-solutions](http://www.seclab-solutions.com/))
- Use [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean) for conversion
- Convert PDF files to PDF/A before converting to HTML
Version 1.2 - 2015-03-10
- Rollback the migration to Jessie and use Wheezy again: the only important dependency from Jessie was poppler, which is available in the backports
- Use the most recent security patches
- Do not wait for user input in case of password protected archive
Version 1.1.1 - 2014-10-26
- General upgrade of Debian to avoid the system to fail in case there is no HDMI cable connected.
Version 1.1 - 2014-10-01
- NTFS support added for USB key
- Updated to Debian Jessie including patches for [bash vulnerabilities CVE-2014-6271 - CVE-2014-7169](/pub/tr-27/)
- CIRCLean user are now removed from the sudoer
Version 1.0 - 2014-05-20
- Based on Raspbian Jessie
- Fully automated tests with Qemu
- Mimetype: support of PDF, Office documents, archives, windows executables
- Filesystem: USB keys have to be formated in vfat
- Support of multiple partitions
- Renaming of autorun.inf on the source key
- Operating system is read only
- Use pdf2htmlEX v0.11

0
doc/NOTES-RPI2 → doc/NOTES-RPI2.md
View File

0
doc/NOTES → doc/NOTES.md
View File

12
README_setup.md → doc/README_setup.md
View File

@ -53,16 +53,16 @@ larger than it was before (6852607 vs. 2658303 in the example).
> fdisk XXXX-XX-XX-raspbian-jessie-lite.img > fdisk XXXX-XX-XX-raspbian-jessie-lite.img
Command (m for help): *p* Command (m for help): *p*
Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3508535296 bytes, 6852608 sectors Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors
Units: sectors of 1 * 512 = 512 bytes Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos Disklabel type: dos
Disk identifier: 0x6f92008e Disk identifier: 0x244b8248
Device Boot Start End Sectors Size Id Type Device Boot Start End Sectors Size Id Type
XXXX-XX-XX-raspbian-jessie-lite.img1 8192 131071 122880 60M c W95 FAT32 (LBA) XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA)
XXXX-XX-XX-raspbian-jessie-lite.img2 131072 2658303 2527232 1.2G 83 Linux XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux
Command (m for help): *d* Command (m for help): *d*
Partition number (1,2, default 2): *2* Partition number (1,2, default 2): *2*
@ -77,8 +77,8 @@ larger than it was before (6852607 vs. 2658303 in the example).
Using default response p. Using default response p.
Partition number (2-4, default 2): Partition number (2-4, default 2):
First sector (2048-6852607, default 2048): *131072* First sector (2048-6852607, default 2048): *137216*
Last sector, +sectors or +size{K,M,G,T,P} (131072-6852607, default 6852607): Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951):
Created a new partition 2 of type 'Linux' and of size 3.2 GiB. Created a new partition 2 of type 'Linux' and of size 3.2 GiB.

36
doc/TODO
View File

@ -1,36 +0,0 @@
TODO
====
* the script locations should be changed in the next version so they don't sit
next to the rPi's example development code that ships with the stock rPi
* the system isn't optimised and should be : cleanup and making it as close to
stock as possible
[Npot sure] Starting process should be more obfuscated
* strip exif data and leave it in a .txt file next to the image it came from
=> exiftool
[Done with remount] set filesystem of OS in RO (physical switch and/or remount OS)
[OK] mount source key in RO and noexec <= also nosuid and nodev
[OK] mount target key with noexec <= also nosuid and nodev
* convert spreadsheets in csv ?
[done in HTML] convert documents (pdfs/*office/...) in images ?
[Not Needed] Have a look at Ghostscript to work on PDFs (.pdf -> .eps -> .png?)
[do everything as user] do not run the conversions as root -> run in chroot
* take eth0 down in /etc/network/interfaces or in the groomer script disable the
interface before anything happens
* hdmi should stay up: solvable by poking the power management timer
(better not to disable the PM completely)
[Done] get rid of pdfbox. remove need for java
[WIP] scripts to generate a SD card automatically (win/mac/linux)
* move the scripts away from /opt/
* strip back libreoffice to minimum required packages. in particular, if possible,
remove libreoffice-java-common package
* Write the groomer log on the destination key
[Done] use /etc/mime.types and file -b --mime-type <filename> to find out the type of
the file
* Extract metadata from all the files => https://mat.boum.org/
HTML Files
==========
- disable JS
- cleanup external imports (js/css/images)

13
doc/TODO.md Normal file
View File

@ -0,0 +1,13 @@
TODO
====
* strip exif data and leave it in a .txt file next to the image it came from
=> exiftool
* Scripts to generate a SD card automatically (win/mac/linux)
* Extract metadata from all the files => https://mat.boum.org/
HTML Files
==========
- disable JS?
- cleanup external imports (js/css/images)

55
doc/image-setup-checklist.md Normal file
View File

@ -0,0 +1,55 @@
* Download qemu and qemu-user-static if not already installed
* Download the newest raspbian-lite image from raspberrypi.org
* Verify the sha1 hash of the downloaded .zip file
* Unzip the image
* Expand the image by 2GB using dd
* Expand the root partition using fdisk
* Mount both partitions in loop mode using /shell_utils/simple_mount_image.sh
* Use df to find the larger partition, and resize the filesystem to fill it
* Use proot to enter a chroot in the image: sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash
* Run dpkg-reconfigure locales (this step + others using proot + qemu can be slow, be patient)
* apt-get update
* apt-get dist-upgrade (might have to run this and autoremove several times)
* apt-get autoremove
* apt-get install the linux dependencies:
- timidity # for playing music
- git # for installing python dependencies from github
- p7zip-full
- pmount ntfs-3g # for mounting, including ntfs
- python3 python3-pip
- python3-lxml
- libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev # dependencies for building pillow
* Compile p7zip-rar from source
- Change your source.list file
- Make a new directory and cd to it
- apt-get build-dep p7zip-rar
- dpkg -i <p7zip-rar .deb file path>
* Make sure the right pip executable is called by `pip3`, change your path if necessary
* Upgrade pip: pip3 install -U pip
* pip3 install python dependencies
- exifread
- pillow
- olefile
- git+https://github.com/decalage2/oletools.git
- git+https://github.com/grierforensics/officedissector.git
- git+https://github.com/CIRCL/PyCIRCLean.git
* Add a user named "kitten"
* Symlink /proc/mounts to /etc/mtab
* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location
* Turn on rc-local.service `systemctl enable rc.local`
- If it doesn't work, read these instructions: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd
* Copy all of the project files from circlean_fs/ into the two partitions:
- rsync -vnri <source> <destination> will do a dry run of what will be copied, remove the -n to copy. See the rsync manpage for details.
- diode_controller/ if you're using the led functionality and have an external led
- midi/ files into /opt/midi/
- you might want to double check all of the permissions of the new files/directories
* apt-get autoclean
* apt-get autoremove
* Exit the chroot
* Copy the image over to the SD card: sudo dd bs=4M if=<image> of=/dev/sd<letter>
- In newer versions of dd, you can add status=progress
* Mount the image
* Optional: fsck the root partition (sudo e2fsck -f /dev/sd<letter>2).
* Test with an rpi
- FAT32 partition
- NTFS partition

24
doc/modifying_image.md Normal file
View File

@ -0,0 +1,24 @@
Modifying an already-built image
================================
One way to debug the project or test changes quickly is to modify an already built
version of the project. Once you've got an image set up on an SD card, you can mount
the image and make changes to the files directly or copy changes you've made locally
onto the mounted image. The only requirement is a linux distro such as Debian or Ubuntu.
If you're using MacOS, you can download and install VirtualBox.
Mounting an image
=================
* The steps listed in mount_image.sh are only necessary if you'd like to chroot
into and run executables from the image locally.
* To mount the image for the purpose of reading/writing to it, the process is much
* Plug the SD card into the computer.
* If you're on Virtualbox, you'll probably have to unmount the image on the host OS
(on MacOS this involves ejecting it or using diskutil unmountDisk) and then mount it
on the virtualized OS. You might have to select it under "Devices" first.
* Then, in linux, use sudo fdisk -l to find the location of the image.
* sudo mount $PATH_TO_IMAGE $PATH_TO_CHOSEN_MOUNT_POINT will mount the image.
* The path to the image will need to be the path to the partition with the OS on it,
which should be the second partition. So /dev/sdb2, not just dev/sdb.
* When you're done, sudo umount $PATH_TO_MOUNT_POINT will unmount it.
* If you get a warning about "No caching mode page found," it's safe to skip it
by pressing enter.

74
doc/qemu-notes.md Normal file
View File

@ -0,0 +1,74 @@
Various qemu startup commands
=============================
From https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=37386
qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-4.4.13-jessie -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1" -hda ~/qemu_vms/2016-09-23-raspbian-jessie-lite.img -redir tcp:5022::22
From https://github.com/dhruvvyas90/qemu-rpi-kernel
qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-4.4.13-jessie -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ~/qemu_vms/2016-09-23-raspbian-jessie-lite.img
From http://pub.phyks.me/respawn/mypersonaldata/public/2014-05-20-11-08-01/
qemu-system-arm -kernel <<<path to kernel>>> -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw init=/bin/bash" -hda <<<path to disk image>>>
Others:
qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-3.10.25-wheezy -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ~/qemu_vms/2015-02-16-raspbian-wheezy.img
qemu-system-arm -kernel qemu-rpi-kernel/kernel-qemu-3.10.25-wheezy -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda 2015-02-16-raspbian-wheezy.img
Places to get raspbian base images:
===================================
For Raspbian Wheezy image:
wget https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-02-17/2015-02-16-raspbian-wheezy.zip
For Raspbian Jessie Lite image:
wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2016-09-28/2016-09-23-raspbian-jessie-lite.zip
Traceback of the qemu failure on digitalocean
=============================================
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4738:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4738:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `lm4549.out'
Could not initialize SDL(No available video device) - exiting
Notes
=====
- The error message: it is probably not a big deal - can make them not being blocking by modifying https://github.com/CIRCL/Circlean/blob/master/tests/run.exp#L10
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=760365
- Could not initialize SDL(No available video device) - exiting <= this one is blocking
- I guess it is the vnc switch - requires x11 installed
- If you use a cloud instance, you will need to get qemu to open a port you can connect to with vnc
- The good thing of having VNC is that you can see what explodes when you're running the image

0
doc/resize_img.md → doc/resize_image.md
View File

4
doc/tests_TODO.md
View File

@ -16,14 +16,14 @@ Ideas
===== =====
Source keys: Source keys:
[DONE] Working documents, one / multiple partitions - Working documents, one / multiple partitions
- Non working documents: one / multiple partitions - Non working documents: one / multiple partitions
- different FS on different partitions - different FS on different partitions
- Non working FS - Non working FS
- Malicious documents (very slow, might break the conversions) - Malicious documents (very slow, might break the conversions)
Destinations keys Destinations keys
[DONE] empty, big enough - empty, big enough
- empty, too small - empty, too small
- broken - broken
- not empty - not empty