mirror of https://github.com/CIRCL/Circlean
Add new documentation, move to docs/
Dan Puttick
parent
51de40f2aa
commit
a2f2d39582
26
CHANGELOG
26
CHANGELOG
|
|
@ -1,26 +0,0 @@
|
|||
Version 1.2 - 2015-03-10
|
||||
|
||||
- Rollback the migration to Jessie and use Wheezy again: the only important dependency from Jessie was poppler, which is available in the backports
|
||||
- Use the most recent security patches
|
||||
- Do not wait for user input in case of password protected archive
|
||||
|
||||
Version 1.1.1 - 2014-10-26
|
||||
|
||||
- General upgrade of Debian to avoid the system to fail in case there is no HDMI cable connected.
|
||||
|
||||
Version 1.1 - 2014-10-01
|
||||
|
||||
- NTFS support added for USB key
|
||||
- Updated to Debian Jessie including patches for [bash vulnerabilities CVE-2014-6271 - CVE-2014-7169](/pub/tr-27/)
|
||||
- CIRCLean user are now removed from the sudoer
|
||||
|
||||
Version 1.0 - 2014-05-20
|
||||
|
||||
- Based on Raspbian Jessie
|
||||
- Fully automated tests with Qemu
|
||||
- Mimetype: support of PDF, Office documents, archives, windows executables
|
||||
- Filesystem: USB keys have to be formated in vfat
|
||||
- Support of multiple partitions
|
||||
- Renaming of autorun.inf on the source key
|
||||
- Operating system is read only
|
||||
- Use pdf2htmlEX v0.11
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
Version 2.1 - 2017-02-XX
|
||||
- Updated to the newest version of Raspbian Jessie lite (January 11th 2017 release)
|
||||
- NTFS files can be mounted
|
||||
|
||||
Version 2.0.2 - 2016-05-12
|
||||
- Improve filename encoding
|
||||
|
||||
Version 2.0.1 - 2016-04-26
|
||||
- Re-add [timidity](http://timidity.sourceforge.net/) so the MIDI files are played properly
|
||||
|
||||
Version 2.0 - 2016-04-26
|
||||
- No critical bugs have been identified, this release uses the latest version of Raspbian Jessie lite, with all system updates
|
||||
|
||||
Version 2.0-BETA - 2015-11-06
|
||||
- There a new beta version of CIRCLean which is a significant improvement from the latest version in term of speed and efficiency on low-end hardware like the first version of the Raspberry Pi. The new code base of CIRCLean is now based on [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean)
|
||||
|
||||
Version 1.3 - 2015-05-27
|
||||
- Fix a [critical security bug](https://www.circl.lu/projects/CIRCLean/security/advisory-01) related to [polyglot files](https://github.com/CIRCL/Circlean/issues/9) - thanks to the reporters ([Jann Horn](https://github.com/thejh), [seclab-solutions](http://www.seclab-solutions.com/))
|
||||
- Use [PyCIRCLean](https://github.com/CIRCL/PyCIRCLean) for conversion
|
||||
- Convert PDF files to PDF/A before converting to HTML
|
||||
|
||||
Version 1.2 - 2015-03-10
|
||||
|
||||
- Rollback the migration to Jessie and use Wheezy again: the only important dependency from Jessie was poppler, which is available in the backports
|
||||
- Use the most recent security patches
|
||||
- Do not wait for user input in case of password protected archive
|
||||
|
||||
Version 1.1.1 - 2014-10-26
|
||||
|
||||
- General upgrade of Debian to avoid the system to fail in case there is no HDMI cable connected.
|
||||
|
||||
Version 1.1 - 2014-10-01
|
||||
|
||||
- NTFS support added for USB key
|
||||
- Updated to Debian Jessie including patches for [bash vulnerabilities CVE-2014-6271 - CVE-2014-7169](/pub/tr-27/)
|
||||
- CIRCLean user are now removed from the sudoer
|
||||
|
||||
Version 1.0 - 2014-05-20
|
||||
|
||||
- Based on Raspbian Jessie
|
||||
- Fully automated tests with Qemu
|
||||
- Mimetype: support of PDF, Office documents, archives, windows executables
|
||||
- Filesystem: USB keys have to be formated in vfat
|
||||
- Support of multiple partitions
|
||||
- Renaming of autorun.inf on the source key
|
||||
- Operating system is read only
|
||||
- Use pdf2htmlEX v0.11
|
||||
|
|
@ -53,16 +53,16 @@ larger than it was before (6852607 vs. 2658303 in the example).
|
|||
> fdisk XXXX-XX-XX-raspbian-jessie-lite.img
|
||||
|
||||
Command (m for help): *p*
|
||||
Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3508535296 bytes, 6852608 sectors
|
||||
Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3537895424 bytes, 6909952 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
Disklabel type: dos
|
||||
Disk identifier: 0x6f92008e
|
||||
Disk identifier: 0x244b8248
|
||||
|
||||
Device Boot Start End Sectors Size Id Type
|
||||
XXXX-XX-XX-raspbian-jessie-lite.img1 8192 131071 122880 60M c W95 FAT32 (LBA)
|
||||
XXXX-XX-XX-raspbian-jessie-lite.img2 131072 2658303 2527232 1.2G 83 Linux
|
||||
XXXX-XX-XX-raspbian-jessie-lite.img1 8192 137215 129024 63M c W95 FAT32 (LBA)
|
||||
XXXX-XX-XX-raspbian-jessie-lite.img2 137216 2715647 2578432 1.2G 83 Linux
|
||||
|
||||
Command (m for help): *d*
|
||||
Partition number (1,2, default 2): *2*
|
||||
|
|
@ -77,8 +77,8 @@ larger than it was before (6852607 vs. 2658303 in the example).
|
|||
|
||||
Using default response p.
|
||||
Partition number (2-4, default 2):
|
||||
First sector (2048-6852607, default 2048): *131072*
|
||||
Last sector, +sectors or +size{K,M,G,T,P} (131072-6852607, default 6852607):
|
||||
First sector (2048-6852607, default 2048): *137216*
|
||||
Last sector, +sectors or +size{K,M,G,T,P} (131216-6909951, default 6909951):
|
||||
|
||||
Created a new partition 2 of type 'Linux' and of size 3.2 GiB.
|
||||
|
||||
36
doc/TODO
36
doc/TODO
|
|
@ -1,36 +0,0 @@
|
|||
TODO
|
||||
====
|
||||
|
||||
* the script locations should be changed in the next version so they don't sit
|
||||
next to the rPi's example development code that ships with the stock rPi
|
||||
* the system isn't optimised and should be : cleanup and making it as close to
|
||||
stock as possible
|
||||
[Npot sure] Starting process should be more obfuscated
|
||||
* strip exif data and leave it in a .txt file next to the image it came from
|
||||
=> exiftool
|
||||
[Done with remount] set filesystem of OS in RO (physical switch and/or remount OS)
|
||||
[OK] mount source key in RO and noexec <= also nosuid and nodev
|
||||
[OK] mount target key with noexec <= also nosuid and nodev
|
||||
* convert spreadsheets in csv ?
|
||||
[done in HTML] convert documents (pdfs/*office/...) in images ?
|
||||
[Not Needed] Have a look at Ghostscript to work on PDFs (.pdf -> .eps -> .png?)
|
||||
[do everything as user] do not run the conversions as root -> run in chroot
|
||||
* take eth0 down in /etc/network/interfaces or in the groomer script disable the
|
||||
interface before anything happens
|
||||
* hdmi should stay up: solvable by poking the power management timer
|
||||
(better not to disable the PM completely)
|
||||
[Done] get rid of pdfbox. remove need for java
|
||||
[WIP] scripts to generate a SD card automatically (win/mac/linux)
|
||||
* move the scripts away from /opt/
|
||||
* strip back libreoffice to minimum required packages. in particular, if possible,
|
||||
remove libreoffice-java-common package
|
||||
* Write the groomer log on the destination key
|
||||
[Done] use /etc/mime.types and file -b --mime-type <filename> to find out the type of
|
||||
the file
|
||||
* Extract metadata from all the files => https://mat.boum.org/
|
||||
|
||||
HTML Files
|
||||
==========
|
||||
|
||||
- disable JS
|
||||
- cleanup external imports (js/css/images)
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
TODO
|
||||
====
|
||||
|
||||
* strip exif data and leave it in a .txt file next to the image it came from
|
||||
=> exiftool
|
||||
* Scripts to generate a SD card automatically (win/mac/linux)
|
||||
* Extract metadata from all the files => https://mat.boum.org/
|
||||
|
||||
HTML Files
|
||||
==========
|
||||
|
||||
- disable JS?
|
||||
- cleanup external imports (js/css/images)
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
* Download qemu and qemu-user-static if not already installed
|
||||
* Download the newest raspbian-lite image from raspberrypi.org
|
||||
* Verify the sha1 hash of the downloaded .zip file
|
||||
* Unzip the image
|
||||
* Expand the image by 2GB using dd
|
||||
* Expand the root partition using fdisk
|
||||
* Mount both partitions in loop mode using /shell_utils/simple_mount_image.sh
|
||||
* Use df to find the larger partition, and resize the filesystem to fill it
|
||||
* Use proot to enter a chroot in the image: sudo proot -q qemu-arm -S /mnt/rpi-root -b /mnt/rpi-boot:/boot /bin/bash
|
||||
* Run dpkg-reconfigure locales (this step + others using proot + qemu can be slow, be patient)
|
||||
* apt-get update
|
||||
* apt-get dist-upgrade (might have to run this and autoremove several times)
|
||||
* apt-get autoremove
|
||||
* apt-get install the linux dependencies:
|
||||
- timidity # for playing music
|
||||
- git # for installing python dependencies from github
|
||||
- p7zip-full
|
||||
- pmount ntfs-3g # for mounting, including ntfs
|
||||
- python3 python3-pip
|
||||
- python3-lxml
|
||||
- libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev # dependencies for building pillow
|
||||
* Compile p7zip-rar from source
|
||||
- Change your source.list file
|
||||
- Make a new directory and cd to it
|
||||
- apt-get build-dep p7zip-rar
|
||||
- dpkg -i <p7zip-rar .deb file path>
|
||||
* Make sure the right pip executable is called by `pip3`, change your path if necessary
|
||||
* Upgrade pip: pip3 install -U pip
|
||||
* pip3 install python dependencies
|
||||
- exifread
|
||||
- pillow
|
||||
- olefile
|
||||
- git+https://github.com/decalage2/oletools.git
|
||||
- git+https://github.com/grierforensics/officedissector.git
|
||||
- git+https://github.com/CIRCL/PyCIRCLean.git
|
||||
* Add a user named "kitten"
|
||||
* Symlink /proc/mounts to /etc/mtab
|
||||
* Copy circlean_fs/root_partition/systemd/system/rc-local.service into the equivalent location
|
||||
* Turn on rc-local.service `systemctl enable rc.local`
|
||||
- If it doesn't work, read these instructions: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd
|
||||
* Copy all of the project files from circlean_fs/ into the two partitions:
|
||||
- rsync -vnri <source> <destination> will do a dry run of what will be copied, remove the -n to copy. See the rsync manpage for details.
|
||||
- diode_controller/ if you're using the led functionality and have an external led
|
||||
- midi/ files into /opt/midi/
|
||||
- you might want to double check all of the permissions of the new files/directories
|
||||
* apt-get autoclean
|
||||
* apt-get autoremove
|
||||
* Exit the chroot
|
||||
* Copy the image over to the SD card: sudo dd bs=4M if=<image> of=/dev/sd<letter>
|
||||
- In newer versions of dd, you can add status=progress
|
||||
* Mount the image
|
||||
* Optional: fsck the root partition (sudo e2fsck -f /dev/sd<letter>2).
|
||||
* Test with an rpi
|
||||
- FAT32 partition
|
||||
- NTFS partition
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
Modifying an already-built image
|
||||
================================
|
||||
One way to debug the project or test changes quickly is to modify an already built
|
||||
version of the project. Once you've got an image set up on an SD card, you can mount
|
||||
the image and make changes to the files directly or copy changes you've made locally
|
||||
onto the mounted image. The only requirement is a linux distro such as Debian or Ubuntu.
|
||||
If you're using MacOS, you can download and install VirtualBox.
|
||||
|
||||
Mounting an image
|
||||
=================
|
||||
* The steps listed in mount_image.sh are only necessary if you'd like to chroot
|
||||
into and run executables from the image locally.
|
||||
* To mount the image for the purpose of reading/writing to it, the process is much
|
||||
* Plug the SD card into the computer.
|
||||
* If you're on Virtualbox, you'll probably have to unmount the image on the host OS
|
||||
(on MacOS this involves ejecting it or using diskutil unmountDisk) and then mount it
|
||||
on the virtualized OS. You might have to select it under "Devices" first.
|
||||
* Then, in linux, use sudo fdisk -l to find the location of the image.
|
||||
* sudo mount $PATH_TO_IMAGE $PATH_TO_CHOSEN_MOUNT_POINT will mount the image.
|
||||
* The path to the image will need to be the path to the partition with the OS on it,
|
||||
which should be the second partition. So /dev/sdb2, not just dev/sdb.
|
||||
* When you're done, sudo umount $PATH_TO_MOUNT_POINT will unmount it.
|
||||
* If you get a warning about "No caching mode page found," it's safe to skip it
|
||||
by pressing enter.
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
Various qemu startup commands
|
||||
=============================
|
||||
|
||||
From https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=37386
|
||||
qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-4.4.13-jessie -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1" -hda ~/qemu_vms/2016-09-23-raspbian-jessie-lite.img -redir tcp:5022::22
|
||||
|
||||
|
||||
From https://github.com/dhruvvyas90/qemu-rpi-kernel
|
||||
qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-4.4.13-jessie -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ~/qemu_vms/2016-09-23-raspbian-jessie-lite.img
|
||||
|
||||
|
||||
From http://pub.phyks.me/respawn/mypersonaldata/public/2014-05-20-11-08-01/
|
||||
qemu-system-arm -kernel <<<path to kernel>>> -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw init=/bin/bash" -hda <<<path to disk image>>>
|
||||
|
||||
|
||||
Others:
|
||||
qemu-system-arm -kernel ~/qemu_vms/kernel-qemu-3.10.25-wheezy -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ~/qemu_vms/2015-02-16-raspbian-wheezy.img
|
||||
|
||||
qemu-system-arm -kernel qemu-rpi-kernel/kernel-qemu-3.10.25-wheezy -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda 2015-02-16-raspbian-wheezy.img
|
||||
|
||||
|
||||
|
||||
Places to get raspbian base images:
|
||||
===================================
|
||||
|
||||
For Raspbian Wheezy image:
|
||||
wget https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-02-17/2015-02-16-raspbian-wheezy.zip
|
||||
|
||||
For Raspbian Jessie Lite image:
|
||||
wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2016-09-28/2016-09-23-raspbian-jessie-lite.zip
|
||||
|
||||
|
||||
|
||||
|
||||
Traceback of the qemu failure on digitalocean
|
||||
=============================================
|
||||
|
||||
pulseaudio: pa_context_connect() failed
|
||||
pulseaudio: Reason: Connection refused
|
||||
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
|
||||
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
|
||||
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
|
||||
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
|
||||
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
|
||||
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
|
||||
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
|
||||
ALSA lib conf.c:4738:(snd_config_expand) Evaluate error: No such file or directory
|
||||
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default
|
||||
alsa: Could not initialize DAC
|
||||
alsa: Failed to open `default':
|
||||
alsa: Reason: No such file or directory
|
||||
ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
|
||||
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
|
||||
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
|
||||
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
|
||||
ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
|
||||
ALSA lib conf.c:4259:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
|
||||
ALSA lib conf.c:4738:(snd_config_expand) Evaluate error: No such file or directory
|
||||
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default
|
||||
alsa: Could not initialize DAC
|
||||
alsa: Failed to open `default':
|
||||
alsa: Reason: No such file or directory
|
||||
audio: Failed to create voice `lm4549.out'
|
||||
Could not initialize SDL(No available video device) - exiting
|
||||
|
||||
|
||||
Notes
|
||||
=====
|
||||
- The error message: it is probably not a big deal - can make them not being blocking by modifying https://github.com/CIRCL/Circlean/blob/master/tests/run.exp#L10
|
||||
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=760365
|
||||
- Could not initialize SDL(No available video device) - exiting <= this one is blocking
|
||||
- I guess it is the vnc switch - requires x11 installed
|
||||
- If you use a cloud instance, you will need to get qemu to open a port you can connect to with vnc
|
||||
- The good thing of having VNC is that you can see what explodes when you're running the image
|
||||
|
|
@ -16,14 +16,14 @@ Ideas
|
|||
=====
|
||||
|
||||
Source keys:
|
||||
[DONE] Working documents, one / multiple partitions
|
||||
- Working documents, one / multiple partitions
|
||||
- Non working documents: one / multiple partitions
|
||||
- different FS on different partitions
|
||||
- Non working FS
|
||||
- Malicious documents (very slow, might break the conversions)
|
||||
|
||||
Destinations keys
|
||||
[DONE] empty, big enough
|
||||
- empty, big enough
|
||||
- empty, too small
|
||||
- broken
|
||||
- not empty
|
||||
|
|
|
|||
Loading…
Reference in New Issue