diff --git a/CHANGELOG b/CHANGELOG deleted file mode 100644 index 44ce0ea..0000000 --- a/CHANGELOG +++ /dev/null @@ -1,26 +0,0 @@ -Version 1.2 - 2015-03-10 - -- Rollback the migration to Jessie and use Wheezy again: the only important dependency from Jessie was poppler, which is available in the backports -- Use the most recent security patches -- Do not wait for user input in case of password protected archive - -Version 1.1.1 - 2014-10-26 - -- General upgrade of Debian to avoid the system to fail in case there is no HDMI cable connected. - -Version 1.1 - 2014-10-01 - -- NTFS support added for USB key -- Updated to Debian Jessie including patches for [bash vulnerabilities CVE-2014-6271 - CVE-2014-7169](/pub/tr-27/) -- CIRCLean user are now removed from the sudoer - -Version 1.0 - 2014-05-20 - -- Based on Raspbian Jessie -- Fully automated tests with Qemu -- Mimetype: support of PDF, Office documents, archives, windows executables -- Filesystem: USB keys have to be formated in vfat -- Support of multiple partitions -- Renaming of autorun.inf on the source key -- Operating system is read only -- Use pdf2htmlEX v0.11 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..cc12256 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,47 @@ +Version 2.1 - 2017-02-XX +- Updated to the newest version of Raspbian Jessie lite (January 11th 2017 release) +- NTFS files can be mounted + +Version 2.0.2 - 2016-05-12 +- Improve filename encoding + +Version 2.0.1 - 2016-04-26 +- Re-add [timidity](http://timidity.sourceforge.net/) so the MIDI files are played properly + +Version 2.0 - 2016-04-26 +- No critical bugs have been identified, this release uses the latest version of Raspbian Jessie lite, with all system updates + +Version 2.0-BETA - 2015-11-06 +- There a new beta version of CIRCLean which is a significant improvement from the latest version in term of speed and efficiency on low-end hardware like the first version of the Raspberry Pi. The new code base of CIRCLean is now based on [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean) + +Version 1.3 - 2015-05-27 +- Fix a [critical security bug](https://www.circl.lu/projects/CIRCLean/security/advisory-01) related to [polyglot files](https://github.com/CIRCL/Circlean/issues/9) - thanks to the reporters ([Jann Horn](https://github.com/thejh), [seclab-solutions](http://www.seclab-solutions.com/)) +- Use [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean) for conversion +- Convert PDF files to PDF/A before converting to HTML + +Version 1.2 - 2015-03-10 + +- Rollback the migration to Jessie and use Wheezy again: the only important dependency from Jessie was poppler, which is available in the backports +- Use the most recent security patches +- Do not wait for user input in case of password protected archive + +Version 1.1.1 - 2014-10-26 + +- General upgrade of Debian to avoid the system to fail in case there is no HDMI cable connected. + +Version 1.1 - 2014-10-01 + +- NTFS support added for USB key +- Updated to Debian Jessie including patches for [bash vulnerabilities CVE-2014-6271 - CVE-2014-7169](/pub/tr-27/) +- CIRCLean user are now removed from the sudoer + +Version 1.0 - 2014-05-20 + +- Based on Raspbian Jessie +- Fully automated tests with Qemu +- Mimetype: support of PDF, Office documents, archives, windows executables +- Filesystem: USB keys have to be formated in vfat +- Support of multiple partitions +- Renaming of autorun.inf on the source key +- Operating system is read only +- Use pdf2htmlEX v0.11 diff --git a/doc/NOTES-RPI2 b/doc/NOTES-RPI2.md similarity index 100% rename from doc/NOTES-RPI2 rename to doc/NOTES-RPI2.md diff --git a/doc/NOTES b/doc/NOTES.md similarity index 100% rename from doc/NOTES rename to doc/NOTES.md diff --git a/README_setup.md b/doc/README_setup.md similarity index 92% rename from README_setup.md rename to doc/README_setup.md index afb74a1..6e6c157 100644 --- a/README_setup.md +++ b/doc/README_setup.md @@ -53,16 +53,16 @@ larger than it was before (6852607 vs. 2658303 in the example). > fdisk XXXX-XX-XX-raspbian-jessie-lite.img Command (m for help): *p* - Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3508535296 bytes, 6852608 sectors + Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos - Disk identifier: 0x6f92008e + Disk identifier: 0x244b8248 Device Boot Start End Sectors Size Id Type - XXXX-XX-XX-raspbian-jessie-lite.img1 8192 131071 122880 60M c W95 FAT32 (LBA) - XXXX-XX-XX-raspbian-jessie-lite.img2 131072 2658303 2527232 1.2G 83 Linux + XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA) + XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux Command (m for help): *d* Partition number (1,2, default 2): *2* @@ -77,8 +77,8 @@ larger than it was before (6852607 vs. 2658303 in the example). Using default response p. Partition number (2-4, default 2): - First sector (2048-6852607, default 2048): *131072* - Last sector, +sectors or +size{K,M,G,T,P} (131072-6852607, default 6852607): + First sector (2048-6852607, default 2048): *137216* + Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951): Created a new partition 2 of type 'Linux' and of size 3.2 GiB. diff --git a/doc/TODO b/doc/TODO deleted file mode 100644 index 1b4fd75..0000000 --- a/doc/TODO +++ /dev/null @@ -1,36 +0,0 @@ -TODO -==== - -* the script locations should be changed in the next version so they don't sit - next to the rPi's example development code that ships with the stock rPi -* the system isn't optimised and should be : cleanup and making it as close to - stock as possible -[Npot sure] Starting process should be more obfuscated -* strip exif data and leave it in a .txt file next to the image it came from - => exiftool -[Done with remount] set filesystem of OS in RO (physical switch and/or remount OS) -[OK] mount source key in RO and noexec <= also nosuid and nodev -[OK] mount target key with noexec <= also nosuid and nodev -* convert spreadsheets in csv ? -[done in HTML] convert documents (pdfs/*office/...) in images ? -[Not Needed] Have a look at Ghostscript to work on PDFs (.pdf -> .eps -> .png?) -[do everything as user] do not run the conversions as root -> run in chroot -* take eth0 down in /etc/network/interfaces or in the groomer script disable the - interface before anything happens -* hdmi should stay up: solvable by poking the power management timer - (better not to disable the PM completely) -[Done] get rid of pdfbox. remove need for java -[WIP] scripts to generate a SD card automatically (win/mac/linux) -* move the scripts away from /opt/ -* strip back libreoffice to minimum required packages. in particular, if possible, - remove libreoffice-java-common package -* Write the groomer log on the destination key -[Done] use /etc/mime.types and file -b --mime-type to find out the type of - the file -* Extract metadata from all the files => https://mat.boum.org/ - -HTML Files -========== - -- disable JS -- cleanup external imports (js/css/images) diff --git a/doc/TODO.md b/doc/TODO.md new file mode 100644 index 0000000..ee8be35 --- /dev/null +++ b/doc/TODO.md @@ -0,0 +1,13 @@ +TODO +==== + +* strip exif data and leave it in a .txt file next to the image it came from + => exiftool +* Scripts to generate a SD card automatically (win/mac/linux) +* Extract metadata from all the files => https://mat.boum.org/ + +HTML Files +========== + +- disable JS? +- cleanup external imports (js/css/images) diff --git a/doc/image-setup-checklist.md b/doc/image-setup-checklist.md new file mode 100644 index 0000000..e3d1d4e --- /dev/null +++ b/doc/image-setup-checklist.md @@ -0,0 +1,55 @@ +* Download qemu and qemu-user-static if not already installed +* Download the newest raspbian-lite image from raspberrypi.org +* Verify the sha1 hash of the downloaded .zip file +* Unzip the image +* Expand the image by 2GB using dd +* Expand the root partition using fdisk +* Mount both partitions in loop mode using /shell_utils/simple_mount_image.sh +* Use df to find the larger partition, and resize the filesystem to fill it +* Use proot to enter a chroot in the image: sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash +* Run dpkg-reconfigure locales (this step + others using proot + qemu can be slow, be patient) +* apt-get update +* apt-get dist-upgrade (might have to run this and autoremove several times) +* apt-get autoremove +* apt-get install the linux dependencies: + - timidity # for playing music + - git # for installing python dependencies from github + - p7zip-full + - pmount ntfs-3g # for mounting, including ntfs + - python3 python3-pip + - python3-lxml + - libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev # dependencies for building pillow +* Compile p7zip-rar from source + - Change your source.list file + - Make a new directory and cd to it + - apt-get build-dep p7zip-rar + - dpkg -i +* Make sure the right pip executable is called by `pip3`, change your path if necessary +* Upgrade pip: pip3 install -U pip +* pip3 install python dependencies + - exifread + - pillow + - olefile + - git+https://github.com/decalage2/oletools.git + - git+https://github.com/grierforensics/officedissector.git + - git+https://github.com/CIRCL/PyCIRCLean.git +* Add a user named "kitten" +* Symlink /proc/mounts to /etc/mtab +* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location +* Turn on rc-local.service `systemctl enable rc.local` + - If it doesn't work, read these instructions: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd +* Copy all of the project files from circlean_fs/ into the two partitions: + - rsync -vnri will do a dry run of what will be copied, remove the -n to copy. See the rsync manpage for details. + - diode_controller/ if you're using the led functionality and have an external led + - midi/ files into /opt/midi/ + - you might want to double check all of the permissions of the new files/directories +* apt-get autoclean +* apt-get autoremove +* Exit the chroot +* Copy the image over to the SD card: sudo dd bs=4M if= of=/dev/sd + - In newer versions of dd, you can add status=progress +* Mount the image +* Optional: fsck the root partition (sudo e2fsck -f /dev/sd2). +* Test with an rpi + - FAT32 partition + - NTFS partition diff --git a/doc/modifying_image.md b/doc/modifying_image.md new file mode 100644 index 0000000..6d59f24 --- /dev/null +++ b/doc/modifying_image.md @@ -0,0 +1,24 @@ +Modifying an already-built image +================================ +One way to debug the project or test changes quickly is to modify an already built +version of the project. Once you've got an image set up on an SD card, you can mount +the image and make changes to the files directly or copy changes you've made locally +onto the mounted image. The only requirement is a linux distro such as Debian or Ubuntu. +If you're using MacOS, you can download and install VirtualBox. + +Mounting an image +================= +* The steps listed in mount_image.sh are only necessary if you'd like to chroot +into and run executables from the image locally. +* To mount the image for the purpose of reading/writing to it, the process is much +* Plug the SD card into the computer. +* If you're on Virtualbox, you'll probably have to unmount the image on the host OS +(on MacOS this involves ejecting it or using diskutil unmountDisk) and then mount it +on the virtualized OS. You might have to select it under "Devices" first. +* Then, in linux, use sudo fdisk -l to find the location of the image. +* sudo mount $PATH_TO_IMAGE $PATH_TO_CHOSEN_MOUNT_POINT will mount the image. +* The path to the image will need to be the path to the partition with the OS on it, +which should be the second partition. So /dev/sdb2, not just dev/sdb. +* When you're done, sudo umount $PATH_TO_MOUNT_POINT will unmount it. +* If you get a warning about "No caching mode page found," it's safe to skip it +by pressing enter. diff --git a/doc/qemu-notes.md b/doc/qemu-notes.md new file mode 100644 index 0000000..cc22bc6 --- /dev/null +++ b/doc/qemu-notes.md @@ -0,0 +1,74 @@ +Various qemu startup commands +============================= + +From https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=37386 +qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-4.4.13-jessie -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1" -hda ~/qemu_vms/2016-09-23-raspbian-jessie-lite.img -redir tcp:5022::22 + + +From https://github.com/dhruvvyas90/qemu-rpi-kernel +qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-4.4.13-jessie -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ~/qemu_vms/2016-09-23-raspbian-jessie-lite.img + + +From http://pub.phyks.me/respawn/mypersonaldata/public/2014-05-20-11-08-01/ +qemu-system-arm -kernel <<>> -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw init=/bin/bash" -hda <<>> + + +Others: +qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-3.10.25-wheezy -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ~/qemu_vms/2015-02-16-raspbian-wheezy.img + +qemu-system-arm -kernel qemu-rpi-kernel/kernel-qemu-3.10.25-wheezy -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda 2015-02-16-raspbian-wheezy.img + + + +Places to get raspbian base images: +=================================== + +For Raspbian Wheezy image: +wget https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-02-17/2015-02-16-raspbian-wheezy.zip + +For Raspbian Jessie Lite image: +wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2016-09-28/2016-09-23-raspbian-jessie-lite.zip + + + + +Traceback of the qemu failure on digitalocean +============================================= + +pulseaudio: pa_context_connect() failed +pulseaudio: Reason: Connection refused +pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver +ALSA lib confmisc.c:768:(parse_card) cannot find card '0' +ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory +ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings +ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory +ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name +ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory +ALSA lib conf.c:4738:(snd_config_expand) Evaluate error: No such file or directory +ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default +alsa: Could not initialize DAC +alsa: Failed to open `default': +alsa: Reason: No such file or directory +ALSA lib confmisc.c:768:(parse_card) cannot find card '0' +ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory +ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings +ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory +ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name +ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory +ALSA lib conf.c:4738:(snd_config_expand) Evaluate error: No such file or directory +ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default +alsa: Could not initialize DAC +alsa: Failed to open `default': +alsa: Reason: No such file or directory +audio: Failed to create voice `lm4549.out' +Could not initialize SDL(No available video device) - exiting + + +Notes +===== +- The error message: it is probably not a big deal - can make them not being blocking by modifying https://github.com/CIRCL/Circlean/blob/master/tests/run.exp#L10 +- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=760365 +- Could not initialize SDL(No available video device) - exiting <= this one is blocking +- I guess it is the vnc switch - requires x11 installed +- If you use a cloud instance, you will need to get qemu to open a port you can connect to with vnc +- The good thing of having VNC is that you can see what explodes when you're running the image diff --git a/doc/resize_img.md b/doc/resize_image.md similarity index 100% rename from doc/resize_img.md rename to doc/resize_image.md diff --git a/doc/tests_TODO.md b/doc/tests_TODO.md index b969896..f3e5fcf 100644 --- a/doc/tests_TODO.md +++ b/doc/tests_TODO.md @@ -16,14 +16,14 @@ Ideas ===== Source keys: -[DONE] Working documents, one / multiple partitions +- Working documents, one / multiple partitions - Non working documents: one / multiple partitions - different FS on different partitions - Non working FS - Malicious documents (very slow, might break the conversions) Destinations keys -[DONE] empty, big enough +- empty, big enough - empty, too small - broken - not empty