diff --git a/README_filecheck.md b/README_filecheck.md index 4887e24..9fa2413 100644 --- a/README_filecheck.md +++ b/README_filecheck.md @@ -1,52 +1,139 @@ -Install Qemu and Expect -============ - -Install the necessary packages: - -``` - sudo apt-get install qemu qemu-user-static expect -``` - - -Create a new image from scratch +Building the image from scratch =============================== -* Download the most recent version of Raspbian Jessie lite: - https://downloads.raspberrypi.org/raspbian_lite_latest +There is always a prebuilt image available for download and installation as +described in the [README](README.md). If you'd like to build the project yourself, +there are several steps: + +* Downloading a generic Raspbian Lite image. +* Adding space to the downloaded image. +* Downloading and building the dependencies. +* Copying the project filesystem into the image. + +This procedure will not work unless you are running Ubuntu or Debian linux. If you +only have access to MacOS or Windows, the best option is to install linux in a +VM, using something like VirtualBox. + +Downloading the Raspbian image +============================== + +* Get the most recent version of Raspbian Jessie Lite: + +``` + wget https://downloads.raspberrypi.org/raspbian_lite_latest +``` * Unpack it: ``` - unzip 2016-03-18-raspbian-jessie-lite.zip + unzip XXXX-XX-XX-raspbian-jessie-lite.zip ``` +Adding space to the image +========================= -Prepare the image -================= +* Use dd to add 2GB (2048 blocks at 1024k each). Using /dev/zero as the input +file yields an unlimited number of "0x00" bytes. -It will be used for the build environment and the final image. +``` + > dd if=/dev/zero bs=1024k count=2048 >> XXXX-XX-XX-raspbian-jessie-lite.img +``` -* [Add empty space to the image](resize_img.md) +* Grow the root partition using fdisk. The "p" command prints the current partition +table. The first partition listed is the boot partition, which shouldn't be changed. +The "d" command, when given the parameter "2", deletes the current root partition. +The "n" command then makes a new partition. It can take the default for "type" +and "number". The "First sector" should be the value that was the "start" sector of the root +partition (131072 in the example below, but this varies depending on the version of the +Raspbian image). The "Last sector" should be the default, and it should be significantly +larger than it was before (6852607 vs. 2658303 in the example). -* Chroot in the image + +``` + > fdisk XXXX-XX-XX-raspbian-jessie-lite.img + + Command (m for help): *p* + Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3508535296 bytes, 6852608 sectors + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 512 bytes + I/O size (minimum/optimal): 512 bytes / 512 bytes + Disklabel type: dos + Disk identifier: 0x6f92008e + + Device Boot Start End Sectors Size Id Type + XXXX-XX-XX-raspbian-jessie-lite.img1 8192 131071 122880 60M c W95 FAT32 (LBA) + XXXX-XX-XX-raspbian-jessie-lite.img2 131072 2658303 2527232 1.2G 83 Linux + + Command (m for help): *d* + Partition number (1,2, default 2): *2* + + Partition 2 has been deleted. + + Command (m for help): *n* + Partition type + p primary (1 primary, 0 extended, 3 free) + e extended (container for logical partitions) + Select (default p): + + Using default response p. + Partition number (2-4, default 2): + First sector (2048-6852607, default 2048): *131072* + Last sector, +sectors or +size{K,M,G,T,P} (131072-6852607, default 6852607): + + Created a new partition 2 of type 'Linux' and of size 3.2 GiB. + + Command (m for help): *w* + The partition table has been altered. + Syncing disks. +``` + +* Mount the image in loop mode. Edit /mount_image.sh to contain the proper values +for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as +shown above. You must also change $IMAGE to the correct string. Then run: ``` sudo ./proper_chroot.sh ``` -* Change your user to root (your global variables may be broken) +* After mounting the image, the above script will chroot into the mounted image. +While in a chroot, the / directory is treated like it is the / directory (thus +the name, change root). To exit the chroot, run "exit" in the root directory. +Then, verify the path to the mounted partitions, and resize the filesystem +to fill the new larger partition using resize2fs: + +``` + > df | grep /mnt/arm + + /dev/loop0 3927752 1955672 1794172 53% /mnt/arm_rPi + /dev/loop1 57288 18960 38328 34% /mnt/arm_rPi/boot + + > sudo resize2fs /dev/loop0 +``` + +Installing the dependencies +=========================== + +* To install the dependencies, you'll have to reenter the chroot again: + +``` + sudo chroot /mnt/arm_rPi +``` + +* Change your user to root (your global variables may be broken as a result): ``` su root ``` -* The locales may be broken, fix it (remove `en_GB.UTF-8 UTF-8`, set `en_US.UTF-8 UTF-8`): +* Change the locales (remove "en_GB.UTF-8 UTF-8", add "en_US.UTF-8 UTF-8"). The +arrow keys move the cursor, spacebar selects/deselects a locale, tab moves the cursor +to a different context, and enter lets you select "ok": ``` dpkg-reconfigure locales ``` -* In the image, make sure everything is up-to-date, and remove the old packages +* In the image, make sure everything is up-to-date and remove the old packages: ``` apt-get update @@ -55,7 +142,8 @@ It will be used for the build environment and the final image. apt-get install timidity git p7zip-full python-dev python-pip python-lxml pmount libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev tk-dev python-tk libxml2-dev libxslt1-dev ``` -* Install python requirements +* Install the Python dependencies for PyCIRCLean. Currently, PyCIRCLean is +Python 2.7 and 3.3+ compatible, but Python 2 support might be dropped at some point. ``` pip install oletools olefile exifread Pillow @@ -63,7 +151,7 @@ It will be used for the build environment and the final image. pip install git+https://github.com/CIRCL/PyCIRCLean.git ``` -* Create the user and mtab for a RO filesystem +* Create a new user and make mounting work with a read-only filesystem. ``` useradd -m kitten @@ -71,15 +159,38 @@ It will be used for the build environment and the final image. ln -s /proc/mounts /etc/mtab ``` -* Copy the files - -``` - sudo ./copy_to_final.sh /mnt/arm_rPi/ -``` - -* Enable rc.local +* Enable rc.local, which ensures that the code in /etc/rc.local is run on boot. +This is what triggers CIRCLean to run. ``` systemctl enable rc-local.service ``` +* Exit the chroot again, and copy the files from your repository into the mounted +image. + +``` + sudo ./copy_to_final.sh /mnt/arm_rPi/ +``` + +Write the image on a SD card +============================ + +* Plug your SD card into the computer. Then, find where it is mounted using df: + +``` + df -h +``` + +* If it has been automatically mounted, unmount the SD card (use the path you +found in the previous step: + +``` + umount $PATH_TO_YOUR_SD +``` + +* Write the image to the card: + +``` + sudo dd bs=4M if=$PATH_TO_YOUR_IMAGE of=$PATH_TO_YOUR_SD +``` diff --git a/README_initial_setup.md b/README_initial_setup.md deleted file mode 100644 index a6e41c2..0000000 --- a/README_initial_setup.md +++ /dev/null @@ -1,232 +0,0 @@ -Install Qemu and Expect -============ - -Install the necessary packages: - -``` - sudo apt-get install qemu qemu-user-static expect -``` - -Create a new image from scratch -=============================== - -* Download the most recent version of Raspbian Jessie lite: - https://downloads.raspberrypi.org/raspbian_lite_latest - -* Unpack it: - -``` - unzip 2016-03-18-raspbian-jessie-lite.zip -``` - -Prepare the base image -====================== - -It will be used for the build environment and the final image. - -* [Add empty space to the image](resize_img.md) - -* Chroot in the image - -``` - sudo ./proper_chroot.sh -``` - -* Change your user to root (your global variables may be broken) - -``` - su root -``` - -* The locales may be broken, fix it (remove `en_GB.UTF-8 UTF-8`, set `en_US.UTF-8 UTF-8`): - -``` - dpkg-reconfigure locales -``` - -* In the image, make sure everything is up-to-date, and remove the old packages - -``` - apt-get update - apt-get dist-upgrade - apt-get autoremove -``` - -Setup two images -================ - -* Create two separate images: one will be used to build the deb packages that are not available in wheezy - -``` - mv raspbian-wheezy.img BUILDENV-raspbian-wheezy.img - cp BUILDENV-raspbian-wheezy.img FINAL-raspbian-wheezy.img -``` - -Build environment specifics -=========================== - -* Create a symlink to the build image - -``` - ln -s BUILDENV-raspbian-wheezy.img raspbian-wheezy.img -``` - -* Chroot in the image - -``` - sudo ./proper_chroot.sh -``` - -* Change your user to root (your global variables may be broken) - -``` - su root -``` - -* Add Wheezy backports source packages to build a poppler version compatible with pdf2htmlEX - -``` - echo 'deb-src http://ftp.debian.org/debian/ wheezy-backports main' >> /etc/apt/sources.list - apt-get instal debian-keyring debian-archive-keyring - apt-get update -``` - -* Get the required build dependencies and the sources - -``` - apt-get build-dep poppler - apt-get source poppler -``` - -* Compile the package - -``` - cd poppler-/ - dpkg-buildpackage -``` -At least on Debian 8, you may receive an error about libpoppler-glib-dev missing the gir1.2-poppler requirement; you can ignore it. - - -* Install the packages required by pdf2htmlEX - -``` - apt-get install cmake libfontforge-dev libspiro-dev python-dev default-jre-headless - cd .. - dpkg -i libpoppler-dev* libpoppler* libpoppler-private-dev* -``` - -* Download the sources of pdf2htmlEX (we cannot use anything newer than v0.11 because fontforge>=2.0 is not available) - -``` - wget https://github.com/Rafiot/pdf2htmlEX/archive/KittenGroomer.zip - unzip KittenGroomer.zip -``` - -* Compile the package - -``` - cd pdf2htmlEX-KittenGroomer/ - dpkg-buildpackage -uc -b -``` - -* Get the packages out of the building image (run it outside of the chroot) - -``` - cp /mnt/arm_rPi/libpoppler46_* /mnt/arm_rPi/pdf2htmlex_* deb/ -``` - -Final image specifics -===================== - -* Change the link to the image - -``` - rm raspbian-wheezy.img - ln -s FINAL-raspbian-wheezy.img -raspbian-wheezy.img -``` - -* Chroot in the image - -``` - sudo ./proper_chroot.sh -``` - -* Change your user to root (your global variables may be broken) - -``` - su root -``` - -* Copy the debian packages into the chroot (run it outside of the chroot) - -``` - cp deb/*.deb /mnt/arm_rPi/ -``` - -* Install repencencies required by the project - -``` - apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount ntfs-3g unoconv python-pip - dpkg -i *.deb - pip install twiggy python-magic -``` - -* Create the user, make Libreoffice and mtab working on a RO filesystem - -``` - useradd -m kitten - pushd /home/kitten - ln -s /tmp/libreoffice - mkdir .config/ - ln -s /tmp/libreoffice_config .config/libreoffice - popd - chown -R kitten:kitten /home/kitten - ln -s /proc/mounts /etc/mtab -``` - -* Copy the script to the image - -``` - sudo ./copy_to_final.sh -``` - -* Get the PyCIRCLean modules -``` - pip install git+https://github.com/CIRCL/PyCIRCLean -``` - - -* Exit the chroot - -Write the image on a SD card -============================ - -*WARNING*: Make sure you write on the right filesystem - -``` - sudo dd bs=4M if=FINAL-raspbian-wheezy.img of=/dev/ -``` - -Run the tests -============= - -* Get the qemu kernel: -``` - pushd tests; wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/kernel-qemu; popd -``` - -* Put some test data from tests/testFiles into tests/content_img_vfat_norm - -* Comment out the other tests in tests/run.sh or populate those directories as - well - -* Make sure to set the filename of the image and the kernel in `tests/run.sh` - -* Run the tests: -``` - sudo ./run_tests.sh -``` - -* If the image run processed images correctly but doesn't exit and unmount the - images cleanly, look at tests/run.exp and make sure it's waiting for the - string your qemu and kernel actually produce. diff --git a/mount_image.sh b/mount_image.sh index beb5558..e239577 100755 --- a/mount_image.sh +++ b/mount_image.sh @@ -1,13 +1,13 @@ #!/bin/bash +# This script will mount a given image or sd card in loop mode. +# Make sure to change the path and offsets for the image you use. You can get +# the correct offsets using `file $PATH_TO_IMAGE` or fdisk. +# If you want to mount an SD card, unset $IMAGE. + # To make debugging easier echo "KittenGroomer: in mount_image.sh" 1>&2 -# Notes: -# - To chroot in an existing SD card, unset IMAGE. Change the paths to the partitions if needed. -# - The offsets are thoses of 2013-02-09-wheezy-raspbian.img. It will change on an other image. -# To get the offsets, use the "file" command. - if [ "$(id -u)" != "0" ]; then echo "This script must be run as root" 1>&2 exit 1 @@ -33,21 +33,17 @@ PARTITION_BOOT='/dev/mmcblk0p1' #PARTITION_ROOTFS='/dev/sdd2' #PARTITION_BOOT='/dev/sdd1' -# If you use the img +# If you use the img... +# Double check the path and offsets as noted above! ##### Debian IMAGE='2016-05-09_CIRCLean.img' -OFFSET_ROOTFS=$((131072 * 512)) OFFSET_BOOT=$((8192 * 512)) -##### Arch -#IMAGE='archlinux-hf-2013-02-11.img' -#OFFSET_ROOTFS=$((186368 * 512)) -#OFFSET_BOOT=$((2048 * 512)) -############ +OFFSET_ROOTFS=$((131072 * 512)) CHROOT_PATH='/mnt/arm_rPi' clean(){ - mv ${CHROOT_PATH}/etc/ld.so.preload_bkp ${CHROOT_PATH}/etc/ld.so.preload + mv ${CHROOT_PATH}/etc/ld.so.preload_backup ${CHROOT_PATH}/etc/ld.so.preload rm ${CHROOT_PATH}/etc/resolv.conf rm ${CHROOT_PATH}/usr/bin/qemu*arm* @@ -79,7 +75,7 @@ elif [ -a ${PARTITION_ROOTFS} ]; then mount ${PARTITION_ROOTFS} ${CHROOT_PATH} mount ${PARTITION_BOOT} ${CHROOT_PATH}/boot else - print 'You need a SD card or an image' + echo 'You need a SD card or an image' exit fi @@ -95,7 +91,7 @@ mount -o bind /tmp ${CHROOT_PATH}/tmp cp -pf /etc/resolv.conf ${CHROOT_PATH}/etc -mv ${CHROOT_PATH}/etc/ld.so.preload ${CHROOT_PATH}/etc/ld.so.preload_bkp +mv ${CHROOT_PATH}/etc/ld.so.preload ${CHROOT_PATH}/etc/ld.so.preload_backup # To make debugging easier echo "KittenGroomer: Image mounted, executing command from mount_image.sh" 1>&2