CIRCL
/
Circlean
mirror of https://github.com/CIRCL/Circlean
2
0
Fork 0
Code Issues Releases Wiki Activity

Added information to README_filecheck.md 

Deleted README_initial_setup because it was no longer needed after the switch
from generic.py to filecheck.py
pull/45/head
Dan Puttick 2016-12-28 20:20:39 -05:00
parent dbf35536be
commit 85a4a43fc2
3 changed files with 153 additions and 278 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

173
README_filecheck.md
View File

@ -1,52 +1,139 @@
Install Qemu and Expect
============
Install the necessary packages:
```
sudo apt-get install qemu qemu-user-static expect
```
Create a new image from scratch
Building the image from scratch
===============================
* Download the most recent version of Raspbian Jessie lite:
https://downloads.raspberrypi.org/raspbian_lite_latest
There is always a prebuilt image available for download and installation as
described in the [README](README.md). If you'd like to build the project yourself,
there are several steps:
* Downloading a generic Raspbian Lite image.
* Adding space to the downloaded image.
* Downloading and building the dependencies.
* Copying the project filesystem into the image.
This procedure will not work unless you are running Ubuntu or Debian linux. If you
only have access to MacOS or Windows, the best option is to install linux in a
VM, using something like VirtualBox.
Downloading the Raspbian image
==============================
* Get the most recent version of Raspbian Jessie Lite:
```
wget https://downloads.raspberrypi.org/raspbian_lite_latest
```
* Unpack it:
```
unzip 2016-03-18-raspbian-jessie-lite.zip
unzip XXXX-XX-XX-raspbian-jessie-lite.zip
```
Adding space to the image
=========================
Prepare the image
=================
* Use dd to add 2GB (2048 blocks at 1024k each). Using /dev/zero as the input
file yields an unlimited number of "0x00" bytes.
It will be used for the build environment and the final image.
```
> dd if=/dev/zero bs=1024k count=2048 >> XXXX-XX-XX-raspbian-jessie-lite.img
```
* [Add empty space to the image](resize_img.md)
* Grow the root partition using fdisk. The "p" command prints the current partition
table. The first partition listed is the boot partition, which shouldn't be changed.
The "d" command, when given the parameter "2", deletes the current root partition.
The "n" command then makes a new partition. It can take the default for "type"
and "number". The "First sector" should be the value that was the "start" sector of the root
partition (131072 in the example below, but this varies depending on the version of the
Raspbian image). The "Last sector" should be the default, and it should be significantly
larger than it was before (6852607 vs. 2658303 in the example).
* Chroot in the image
```
> fdisk XXXX-XX-XX-raspbian-jessie-lite.img
Command (m for help): *p*
Disk XXXX-XX-XX-raspbian-jessie-lite.img: 3.3 GiB, 3508535296 bytes, 6852608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6f92008e
Device Boot Start End Sectors Size Id Type
XXXX-XX-XX-raspbian-jessie-lite.img1 8192 131071 122880 60M c W95 FAT32 (LBA)
XXXX-XX-XX-raspbian-jessie-lite.img2 131072 2658303 2527232 1.2G 83 Linux
Command (m for help): *d*
Partition number (1,2, default 2): *2*
Partition 2 has been deleted.
Command (m for help): *n*
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p):
Using default response p.
Partition number (2-4, default 2):
First sector (2048-6852607, default 2048): *131072*
Last sector, +sectors or +size{K,M,G,T,P} (131072-6852607, default 6852607):
Created a new partition 2 of type 'Linux' and of size 3.2 GiB.
Command (m for help): *w*
The partition table has been altered.
Syncing disks.
```
* Mount the image in loop mode. Edit /mount_image.sh to contain the proper values
for $OFFSET_BOOT and $OFFSET_ROOTFS, which you can obtain using fdisk and "p" as
shown above. You must also change $IMAGE to the correct string. Then run:
```
sudo ./proper_chroot.sh
```
* Change your user to root (your global variables may be broken)
* After mounting the image, the above script will chroot into the mounted image.
While in a chroot, the / directory is treated like it is the / directory (thus
the name, change root). To exit the chroot, run "exit" in the root directory.
Then, verify the path to the mounted partitions, and resize the filesystem
to fill the new larger partition using resize2fs:
```
> df | grep /mnt/arm
/dev/loop0 3927752 1955672 1794172 53% /mnt/arm_rPi
/dev/loop1 57288 18960 38328 34% /mnt/arm_rPi/boot
> sudo resize2fs /dev/loop0
```
Installing the dependencies
===========================
* To install the dependencies, you'll have to reenter the chroot again:
```
sudo chroot /mnt/arm_rPi
```
* Change your user to root (your global variables may be broken as a result):
```
su root
```
* The locales may be broken, fix it (remove `en_GB.UTF-8 UTF-8`, set `en_US.UTF-8 UTF-8`):
* Change the locales (remove "en_GB.UTF-8 UTF-8", add "en_US.UTF-8 UTF-8"). The
arrow keys move the cursor, spacebar selects/deselects a locale, tab moves the cursor
to a different context, and enter lets you select "ok":
```
dpkg-reconfigure locales
```
* In the image, make sure everything is up-to-date, and remove the old packages
* In the image, make sure everything is up-to-date and remove the old packages:
```
apt-get update
@ -55,7 +142,8 @@ It will be used for the build environment and the final image.
apt-get install timidity git p7zip-full python-dev python-pip python-lxml pmount libjpeg-dev libtiff-dev libwebp-dev liblcms2-dev tcl-dev tk-dev python-tk libxml2-dev libxslt1-dev
```
* Install python requirements
* Install the Python dependencies for PyCIRCLean. Currently, PyCIRCLean is
Python 2.7 and 3.3+ compatible, but Python 2 support might be dropped at some point.
```
pip install oletools olefile exifread Pillow
@ -63,7 +151,7 @@ It will be used for the build environment and the final image.
pip install git+https://github.com/CIRCL/PyCIRCLean.git
```
* Create the user and mtab for a RO filesystem
* Create a new user and make mounting work with a read-only filesystem.
```
useradd -m kitten
@ -71,15 +159,38 @@ It will be used for the build environment and the final image.
ln -s /proc/mounts /etc/mtab
```
* Copy the files
```
sudo ./copy_to_final.sh /mnt/arm_rPi/
```
* Enable rc.local
* Enable rc.local, which ensures that the code in /etc/rc.local is run on boot.
This is what triggers CIRCLean to run.
```
systemctl enable rc-local.service
```
* Exit the chroot again, and copy the files from your repository into the mounted
image.
```
sudo ./copy_to_final.sh /mnt/arm_rPi/
```
Write the image on a SD card
============================
* Plug your SD card into the computer. Then, find where it is mounted using df:
```
df -h
```
* If it has been automatically mounted, unmount the SD card (use the path you
found in the previous step:
```
umount $PATH_TO_YOUR_SD
```
* Write the image to the card:
```
sudo dd bs=4M if=$PATH_TO_YOUR_IMAGE of=$PATH_TO_YOUR_SD
```

232
README_initial_setup.md
View File

@ -1,232 +0,0 @@
Install Qemu and Expect
============
Install the necessary packages:
```
sudo apt-get install qemu qemu-user-static expect
```
Create a new image from scratch
===============================
* Download the most recent version of Raspbian Jessie lite:
https://downloads.raspberrypi.org/raspbian_lite_latest
* Unpack it:
```
unzip 2016-03-18-raspbian-jessie-lite.zip
```
Prepare the base image
======================
It will be used for the build environment and the final image.
* [Add empty space to the image](resize_img.md)
* Chroot in the image
```
sudo ./proper_chroot.sh
```
* Change your user to root (your global variables may be broken)
```
su root
```
* The locales may be broken, fix it (remove `en_GB.UTF-8 UTF-8`, set `en_US.UTF-8 UTF-8`):
```
dpkg-reconfigure locales
```
* In the image, make sure everything is up-to-date, and remove the old packages
```
apt-get update
apt-get dist-upgrade
apt-get autoremove
```
Setup two images
================
* Create two separate images: one will be used to build the deb packages that are not available in wheezy
```
mv raspbian-wheezy.img BUILDENV-raspbian-wheezy.img
cp BUILDENV-raspbian-wheezy.img FINAL-raspbian-wheezy.img
```
Build environment specifics
===========================
* Create a symlink to the build image
```
ln -s BUILDENV-raspbian-wheezy.img raspbian-wheezy.img
```
* Chroot in the image
```
sudo ./proper_chroot.sh
```
* Change your user to root (your global variables may be broken)
```
su root
```
* Add Wheezy backports source packages to build a poppler version compatible with pdf2htmlEX
```
echo 'deb-src http://ftp.debian.org/debian/ wheezy-backports main' >> /etc/apt/sources.list
apt-get instal debian-keyring debian-archive-keyring
apt-get update
```
* Get the required build dependencies and the sources
```
apt-get build-dep poppler
apt-get source poppler
```
* Compile the package
```
cd poppler-<VERSION>/
dpkg-buildpackage
```
At least on Debian 8, you may receive an error about libpoppler-glib-dev missing the gir1.2-poppler requirement; you can ignore it.
* Install the packages required by pdf2htmlEX
```
apt-get install cmake libfontforge-dev libspiro-dev python-dev default-jre-headless
cd ..
dpkg -i libpoppler-dev* libpoppler* libpoppler-private-dev*
```
* Download the sources of pdf2htmlEX (we cannot use anything newer than v0.11 because fontforge>=2.0 is not available)
```
wget https://github.com/Rafiot/pdf2htmlEX/archive/KittenGroomer.zip
unzip KittenGroomer.zip
```
* Compile the package
```
cd pdf2htmlEX-KittenGroomer/
dpkg-buildpackage -uc -b
```
* Get the packages out of the building image (run it outside of the chroot)
```
cp /mnt/arm_rPi/libpoppler46_* /mnt/arm_rPi/pdf2htmlex_* deb/
```
Final image specifics
=====================
* Change the link to the image
```
rm raspbian-wheezy.img
ln -s FINAL-raspbian-wheezy.img -raspbian-wheezy.img
```
* Chroot in the image
```
sudo ./proper_chroot.sh
```
* Change your user to root (your global variables may be broken)
```
su root
```
* Copy the debian packages into the chroot (run it outside of the chroot)
```
cp deb/*.deb /mnt/arm_rPi/
```
* Install repencencies required by the project
```
apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount ntfs-3g unoconv python-pip
dpkg -i *.deb
pip install twiggy python-magic
```
* Create the user, make Libreoffice and mtab working on a RO filesystem
```
useradd -m kitten
pushd /home/kitten
ln -s /tmp/libreoffice
mkdir .config/
ln -s /tmp/libreoffice_config .config/libreoffice
popd
chown -R kitten:kitten /home/kitten
ln -s /proc/mounts /etc/mtab
```
* Copy the script to the image
```
sudo ./copy_to_final.sh
```
* Get the PyCIRCLean modules
```
pip install git+https://github.com/CIRCL/PyCIRCLean
```
* Exit the chroot
Write the image on a SD card
============================
*WARNING*: Make sure you write on the right filesystem
```
sudo dd bs=4M if=FINAL-raspbian-wheezy.img of=/dev/<FILESYSTEM>
```
Run the tests
=============
* Get the qemu kernel:
```
pushd tests; wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/kernel-qemu; popd
```
* Put some test data from tests/testFiles into tests/content_img_vfat_norm
* Comment out the other tests in tests/run.sh or populate those directories as
well
* Make sure to set the filename of the image and the kernel in `tests/run.sh`
* Run the tests:
```
sudo ./run_tests.sh
```
* If the image run processed images correctly but doesn't exit and unmount the
images cleanly, look at tests/run.exp and make sure it's waiting for the
string your qemu and kernel actually produce.

26
mount_image.sh
View File

@ -1,13 +1,13 @@
#!/bin/bash
# This script will mount a given image or sd card in loop mode.
# Make sure to change the path and offsets for the image you use. You can get
# the correct offsets using `file $PATH_TO_IMAGE` or fdisk.
# If you want to mount an SD card, unset $IMAGE.
# To make debugging easier
echo "KittenGroomer: in mount_image.sh" 1>&2
# Notes:
# - To chroot in an existing SD card, unset IMAGE. Change the paths to the partitions if needed.
# - The offsets are thoses of 2013-02-09-wheezy-raspbian.img. It will change on an other image.
# To get the offsets, use the "file" command.
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
@ -33,21 +33,17 @@ PARTITION_BOOT='/dev/mmcblk0p1'
#PARTITION_ROOTFS='/dev/sdd2'
#PARTITION_BOOT='/dev/sdd1'
# If you use the img
# If you use the img...
# Double check the path and offsets as noted above!
##### Debian
IMAGE='2016-05-09_CIRCLean.img'
OFFSET_ROOTFS=$((131072 * 512))
OFFSET_BOOT=$((8192 * 512))
##### Arch
#IMAGE='archlinux-hf-2013-02-11.img'
#OFFSET_ROOTFS=$((186368 * 512))
#OFFSET_BOOT=$((2048 * 512))
############
OFFSET_ROOTFS=$((131072 * 512))
CHROOT_PATH='/mnt/arm_rPi'
clean(){
mv ${CHROOT_PATH}/etc/ld.so.preload_bkp ${CHROOT_PATH}/etc/ld.so.preload
mv ${CHROOT_PATH}/etc/ld.so.preload_backup ${CHROOT_PATH}/etc/ld.so.preload
rm ${CHROOT_PATH}/etc/resolv.conf
rm ${CHROOT_PATH}/usr/bin/qemu*arm*
@ -79,7 +75,7 @@ elif [ -a ${PARTITION_ROOTFS} ]; then
mount ${PARTITION_ROOTFS} ${CHROOT_PATH}
mount ${PARTITION_BOOT} ${CHROOT_PATH}/boot
else
print 'You need a SD card or an image'
echo 'You need a SD card or an image'
exit
fi
@ -95,7 +91,7 @@ mount -o bind /tmp ${CHROOT_PATH}/tmp
cp -pf /etc/resolv.conf ${CHROOT_PATH}/etc
mv ${CHROOT_PATH}/etc/ld.so.preload ${CHROOT_PATH}/etc/ld.so.preload_bkp
mv ${CHROOT_PATH}/etc/ld.so.preload ${CHROOT_PATH}/etc/ld.so.preload_backup
# To make debugging easier
echo "KittenGroomer: Image mounted, executing command from mount_image.sh" 1>&2