lookyloo/README.md

98 lines
2.8 KiB
Markdown
Raw Normal View History

2017-09-25 15:48:29 +02:00
![Lookyloo icon](lookyloo/static/lookyloo.jpeg)
2017-07-23 19:56:51 +02:00
*Lookyloo* is a web interface allowing to scrape a website and then displays a
tree of domains calling each other.
2017-08-12 17:45:33 +02:00
# What is that name?!
2017-07-23 19:56:51 +02:00
```
1. People who just come to look.
2. People who go out of their way to look at people or something often causing crowds and more disruption.
3. People who enjoy staring at watching other peoples misfortune. Oftentimes car onlookers to car accidents.
Same as Looky Lou; often spelled as Looky-loo (hyphen) or lookylou
In L.A. usually the lookyloo's cause more accidents by not paying full attention to what is ahead of them.
```
Source: [Urban Dictionary](https://www.urbandictionary.com/define.php?term=lookyloo)
2017-07-23 19:56:51 +02:00
2018-01-05 17:09:42 +01:00
# Screenshot
![Screenshot of Lookyloo](doc/example.png)
2017-07-23 19:56:51 +02:00
2017-08-12 17:45:33 +02:00
# Implementation details
2017-07-23 19:56:51 +02:00
This code is very heavily inspired by [webplugin](https://github.com/etetoolkit/webplugin) and adapted to use flask as backend.
2017-07-23 19:56:51 +02:00
2017-08-12 17:45:33 +02:00
# Installation of har2tree
2017-07-23 19:56:51 +02:00
The core dependency is ETE Toolkit, which you can install following the guide
on the official [website](http://etetoolkit.org/download/)
2017-07-23 19:56:51 +02:00
2017-10-02 11:10:32 +02:00
Note: all the PyQt4 dependencies are optional.
2017-08-12 19:11:02 +02:00
2017-08-12 17:45:33 +02:00
# Installation of scrapysplashwrapper
You need a running splash instance, preferably on [docker](https://splash.readthedocs.io/en/stable/install.html)
2017-08-12 17:45:33 +02:00
```bash
sudo apt install docker.io
sudo docker pull scrapinghub/splash
sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui --disable-lua
# On a server with a decent abount of RAM, you may want to run it this way:
# sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-ui -s 100 --disable-lua -m 50000
2017-08-12 17:45:33 +02:00
```
# Installation of the whole thing
```bash
pip install -r requirements.txt
2017-07-23 19:56:51 +02:00
pip install -e .
2017-10-02 11:13:45 +02:00
wget https://d3js.org/d3.v4.min.js -O lookyloo/static/d3.v4.min.js
2018-02-06 01:23:00 +01:00
wget https://cdn.rawgit.com/eligrey/FileSaver.js/5733e40e5af936eb3f48554cf6a8a7075d71d18a/FileSaver.js -O lookyloo/static/FileSaver.js
2017-07-23 19:56:51 +02:00
```
# Run the app locally
```bash
export FLASK_APP=lookyloo
flask run
```
## With a reverse proxy (Nginx)
```bash
pip install uwsgi
```
### Config files
You have to configure the two following files:
* `etc/nginx/sites-available/lookyloo`
* `etc/systemd/system/lookyloo.service`
And copy them to the appropriate directories and run the following command:
2017-08-12 20:40:08 +02:00
```bash
sudo ln -s /etc/nginx/sites-available/lookyloo /etc/nginx/sites-enabled
```
If needed, remove the default site
```bash
sudo rm /etc/nginx/sites-enabled/default
```
Make sure everything is working:
```bash
sudo systemctl start lookyloo
sudo systemctl enable lookyloo
sudo nginx -t
# If it is cool:
sudo service nginx restart
```
And you can open ```http://<IP-or-domain>/```
2017-08-12 20:40:08 +02:00
Now, you should configure [TLS (let's encrypt and so on)](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04)
2017-08-12 20:40:08 +02:00