Lookyloo is a web interface allowing to scrape a website and then displays a tree of domains calling each other. https://lookyloo.circl.lu/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Raphaël Vinot dd3674c195 new: Initial support for CNAME lookups 17 hours ago
.github chg: Move issues templates to directory expected by github 1 month ago
bin fix: Force kill 3rdparty.sh 2 days ago
cache fix: Systemd service, add proper stop script 1 year ago
config new: Config option for Flask IP and Port, reorganize config loading 3 days ago
contributing Initial set up contrib guide 1 month ago
doc new: Add installation notes 1 month ago
etc chg: Properly handle proxied setups 5 months ago
indexing fix: Keep indexing database 1 month ago
known_content chg: Improve message for known malicious entries, add known file 1 week ago
known_content_user chg: Cleanups, allow to add context from ressources page 3 weeks ago
lookyloo new: Initial support for CNAME lookups 17 hours ago
tools chg: Add config files validator 3 days ago
user_agents new: Add recent UA list 3 days ago
website chg: Remove FileSaver.js from the deps, bump datatables 2 days ago
.dockerignore chg: Do not put the content of scraped in the package. 2 months ago
.gitignore chg: Cleanups, allow to add context from ressources page 3 weeks ago
.travis.yml chg: Bump dependencies 5 months ago
Dockerfile fix: Remove client directory 2 weeks ago
LICENSE chg: Bump license 3 months ago
README.md Update README.md 3 months ago
SECURITY.md Create SECURITY.md 3 months ago
code_of_conduct.md new: Add code of conduct 1 month ago
docker-compose.yml fix: Docker, capture form, error message. 2 months ago
poetry.lock new: Initial support for CNAME lookups 17 hours ago
pyproject.toml new: Initial support for CNAME lookups 17 hours ago
setup.py chg: Use capture UUID as a reference everywhere 2 months ago

README.md

Lookyloo icon

Gitter

Lookyloo is a web interface allowing to scrape a website and then displays a tree of domains calling each other.

Thank you very much Tech Blog @ willshouse.com for the up-to-date list of UserAgents.

What is that name?!

1. People who just come to look.
2. People who go out of their way to look at people or something often causing crowds and more disruption.
3. People who enjoy staring at watching other peoples misfortune. Oftentimes car onlookers to car accidents.
Same as Looky Lou; often spelled as Looky-loo (hyphen) or lookylou
In L.A. usually the lookyloo's cause more accidents by not paying full attention to what is ahead of them.

Source: Urban Dictionary

Screenshot

Screenshot of Lookyloo

Implementation details

This code is very heavily inspired by webplugin and adapted to use flask as backend.

The two core dependencies of this project are the following:

  • ETE Toolkit: A Python framework for the analysis and visualization of trees.
  • Splash: Lightweight, scriptable browser as a service with an HTTP API

Cookies

If you want to scrape a website as if you were loggged in, you need to pass your sessions cookies. You can do it the the folloing way:

  1. Install Cookie Quick Manager
  2. Click onthe icon in the top right of your browser > Manage all cookies
  3. Search for a domain, tick the Sub-domain box if needed
  4. Right clock on the domain you want to export > save to file > $LOOKYLOO_HOME/cookies.json

Then, you need to restart the webserver and from now on, every cookies you have in that file will be available for the browser used by Splash

Python client

You can use pylookyloo as a standalone script, or as a library, more details here

Installation

IMPORTANT: Use poetry

NOTE: Yes, it requires python3.7+. No, it will never support anything older.

NOTE: If you want to run a public instance, you should set only_global_lookups=True in website/web/__init__.py and bin/async_scrape.py to disallow scraping of private IPs.

Installation of Splash

You need a running splash instance, preferably on docker

sudo apt install docker.io
sudo docker pull scrapinghub/splash
sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --disable-browser-caches
# On a server with a decent abount of RAM, you may want to run it this way:
# sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash -s 100 -m 50000 --disable-browser-caches

Install redis

git clone https://github.com/antirez/redis.git
cd redis
git checkout 5.0
make
cd ..

Installation of Lookyloo

git clone https://github.com/Lookyloo/lookyloo.git
cd lookyloo
poetry install
echo LOOKYLOO_HOME="'`pwd`'" > .env

Run the app

poetry run start.py

Run the app in production

With a reverse proxy (Nginx)

pip install uwsgi

Config files

You have to configure the two following files:

  • etc/nginx/sites-available/lookyloo
  • etc/systemd/system/lookyloo.service

Copy them to the appropriate directories, and run the following command:

sudo ln -s /etc/nginx/sites-available/lookyloo /etc/nginx/sites-enabled

If needed, remove the default site

sudo rm /etc/nginx/sites-enabled/default

Make sure everything is working:

sudo systemctl start lookyloo
sudo systemctl enable lookyloo
sudo nginx -t
# If it is cool:
sudo service nginx restart

And you can open http://<IP-or-domain>/

Now, you should configure TLS (let's encrypt and so on)

Use aquarium for a reliable multi-users app

Aquarium is a haproxy + splash bundle that will allow lookyloo to be used by more than one user at once.

The initial version of the project was created by TeamHG-Memex but we have a dedicated repository that fits our needs better.

Follow the documentation if you want to use it.

Run the app with a simple docker setup

Dockerfile

The repository includes a Dockerfile for building a containerized instance of the app.

Lookyloo stores the scraped data in /lookyloo/scraped. If you want to persist the scraped data between runs it is sufficient to define a volume for this directory.

Running a complete setup with Docker Compose

Additionally you can start a complete setup, including the necessary Docker instance of splashy, by using Docker Compose and the included service definition in docker-compose.yml by running

docker-compose up

After building and startup is complete lookyloo should be available at http://localhost:5000/

If you want to persist the data between different runs uncomment the “volumes” definition in the last two lines of docker-compose.yml and define a data storage directory in your Docker host system there.