Merge pull request #61 from dputtick/dev

Release 2.3.1: fix bug with RPi 3

CHANGELOG.md

@@ -1,3 +1,7 @@
+Version 2.3.1 - 2017-10-25
+- Bugfix for Raspberry Pi 3 that was causing rc.local to fail to boot
+- Turned off several networking related services: startup is much faster
+
 Version 2.3 - 2017-09-08
 - Updated to the newest version of Raspbian Stretch lite (2017-08-16 release)
 - Using the newest version of PyCIRCLean, including several vulnerability and bug fixes

circlean_fs/root_partition/etc/rc.local

@@ -23,8 +23,6 @@ echo "GROOMER: end of boot, running rc.local."
 
 if [ -e /dev/sda ]; then
   if [ -e /dev/sdb ]; then
-    # Avoid possible misuse - turn off eth0 (ethernet port)
-    /sbin/ifconfig eth0 down
     trap clean EXIT TERM INT
     cd /opt/groomer
     /usr/sbin/led &

doc/NOTES-RPI2.md

@@ -1,12 +0,0 @@
-USB Ports
----------
-
-If you connect multiple keys to the Rpi, they will be listed that way:
-
-First: Top left
-Second: Top right
-Third: Bottom left
-Forth: Bottom right
-
-tl;dr: as long as the source key (sda) is connected to the top left port, the destination (sdb) can be connected on any other port.
-

doc/NOTES.md

@@ -1,15 +1,15 @@
-Notes
+Usage Notes
-=====
+===========
 
-* don't plug in USB devices with a hub because there's no way to tell it which
+* Don't plug in USB devices with a hub because there's no way to tell it which
-  is source and target - its the first drive enumerated (top port) that is the
+  is source and target - its the first drive detected (top port) that is the
   source and the second (bottom port) is the target
-* don't turn it off without shutting down the system, when grooming is done it
+* Don't turn it off without shutting down the system, when grooming is done it
   shuts down automatically: losing power while it's running can trash the OS
   on the SD cards because SD cards don't always like dirty shutdowns (ie power loss)
 * Using a target usb stick that has a status light as long as the device has
   power is a really useful thing as there the other status lights on the groomer
-  are less than indicative at times: because the 'OK' led on the rPi toggles on activity
+  are less than indicative at times: because the 'OK' led on the RPi toggles on activity
   it can be off for a long time while processing something and only comes back
   on when that process finishes - hence why a USB that has some sort of LED activity
   when just plugged in (even if not reading or writing but while the USB port is
@@ -18,46 +18,32 @@ Notes
   then be off on the USB device
 * Use a larger target device as all zip files get unpacked and processed onto
   the target
-* if you have an hdmi monitor plugged in you can watch what's happening for about
+* If you have an hdmi monitor plugged in you can watch what's happening for about
   30 minutes until the rPI's power saving kicks in and turns off the monitor
-* if only one usb stick is present at power up, it doesn't groom and looks like
+* If only one usb stick is present at power up, it doesn't groom and looks like
   a normal rPi
-* if you want to ssh into the rPi username is 'pi' password 'raspberry' as per defaults
+* If you want to ssh into the RPi username is 'pi' password 'raspberry' as per defaults
 
 
 Technical notes
 ===============
 
-* groomer script is in /opt/groomer/ with the other required files
+* Groomer script is in /opt/groomer/ with the other required files and the ip
-* dependencies are libre-office and OpenJRE
+  address is 192.168.1.89
-* and the ip address is 192.168.1.89
+* The groomer process is kicked off in /etc/rc.local
-* the groomer process is kicked off in /etc/rc.local
+* The heavy lifting is dispatched from /opt/groomer/groomer.sh
-* the heavy lifting takes place or is dispatched from /opt/groomer/groomer.sh
+* All files processing is in filecheck.py
-  in that script file is what file types get processed (or if not listed there,
-  get ignored)
-* there are two ways pdf's can get handled -right now they have their text extracted
-  to the target device, the other way copies it and extracts the text
-* the pdf text extraction isn't perfect and is the slowest part of it, but should
-  be able to handle unicode stuff and currently doesn't do image extraction from
-  pdf's but could do that too
 
 
-Discussion
+USB Ports
-==========
+=========
 
-* however image exports of pdf pages only have the images and no text so it's not
+If you connect multiple keys to the RPi, they will be detected in this order:
-  like saving each page to a jpg which would be a really handy and safe way of
+
-  converting pdf's
+First: Top left
-* spread sheets and presentations get converted to pdfs to kill off any embedded
+Second: Top right
-  macros and it's assumed that it's not producing evil pdf's on export but does
+Third: Bottom left
-  nothing to sanitize any embedded links within those documents
+Forth: Bottom right
-* for spreadsheets, if they are longer than a page, only a page worth from that
+
-  sheet is exported right from the middle of the sheet (ie the top and bottom of
+* As long as the source key (sda) is connected to the top left port, the
-  that sheet will get cut off and only the contents in the middle exported to pdf)
+destination (sdb) can be connected on any other port.
-  dumb but i figure if you want to go back to the source because it's interesting
-  enough on the groomed side of it, then you can take the extra precautions
-* the groomed target only copies "safe" files, and does its best to convert any
-  potential unsafe files to a safer format
-* safe files being one that I know of that can't contain malicious embedded macros
-  or other crap like that, and those than can get converted to something that wont
-  contain code after conversion

doc/TODO.md

@@ -1,13 +0,0 @@
-TODO
-====
-
-* strip exif data and leave it in a .txt file next to the image it came from
-  => exiftool
-* Scripts to generate a SD card automatically (win/mac/linux)
-* Extract metadata from all the files => https://mat.boum.org/
-
-HTML Files
-==========
-
-- disable JS?
-- cleanup external imports (js/css/images)

doc/image_setup_checklist.md

@@ -38,6 +38,7 @@
 * Symlink /proc/mounts to /etc/mtab
 * Turn on rc-local.service `systemctl enable rc-local.service`
     - If it doesn't work, read these instructions: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd
+* Disable networking.service, bluetooth.service, and dchpcd.service
 * apt-get autoclean
 * apt-get autoremove
 * Exit from proot

doc/setup_with_proot.md

@@ -185,6 +185,13 @@ This is what triggers CIRCLean to run.
 ```
     systemctl enable rc-local.service
 ```
+* Turn off several networking related services. This speeds up boot and reduces the
+attack surface:
+```
+    systemctl disable networking.service
+    systemctl disable bluetooth.service
+    systemctl disable dhcpcd.service
+```
 * Clean up:
 ```
     apt-get clean

shell_utils/copy_metadata.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+# Filename: cp-metadata
+
+myecho=echo
+src_path="$1"
+dst_path="$2"
+
+find "$src_path" |
+  while read src_file; do
+    dst_file="$dst_path${src_file#$src_path}"
+    $myecho chmod --reference="$src_file" "$dst_file"
+    $myecho chown --reference="$src_file" "$dst_file"
+    $myecho touch --reference="$src_file" "$dst_file"
+  done

shell_utils/prepare_rPI.sh

@@ -3,15 +3,9 @@
 # change locales to en_US.UTF-8
 dpkg-reconfigure locales
 
-sed -i "s/wheezy/jessie/" /etc/apt/sources.list
 apt-get update
 apt-get dist-upgrade
 apt-get autoremove
-apt-get install libreoffice p7zip-full libfontforge1 timidity freepats pmount
-dpkg -i pdf2htmlex*.deb
-
-# Disable swap
-dphys-swapfile uninstall
 
 # enable rc.local
 systemctl enable rc-local.service