cominfra/stats
cominfra
/
stats
8
0
Fork 0
Code Issues 1 Pull Requests Activity

Update README.md

This commit is contained in:
Christian P. MOMON 2023-02-08 22:32:13 +00:00
parent a18358e3ef
commit b50c9e111f
1 changed files with 454 additions and 61 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

515
README.md
View File

@ -1,92 +1,485 @@
# Service STATS
# Service stats.a-lec.org
Service de statistiques/métriques de Libre en Communs basé sur le logiciel libre StatoolInfos : https://forge.devinsy.fr/devinsy/statoolinfos.
Équipe : Christian Momon (@cpm)
Anciens de l'équipe : n/a.
Ce service s'articule en deux parties :
- côté serveur stats.a-lec.org :
- un fichier properties décrit le Libre en Communs,
- une moulinette collecte des fichiers properties et s'en sert pour générér le site web,
- côté services LeC :
- un fichier properties décrivant le service,
- un fichier properties de métriques contenant les statistiques du services,
- un programme lancé quotidiennement pour compléter le fichier properties de métriques.
Pour déployer StatoolInfos sur un service LeC, aller directement à la [section dédiée](./README.md#ajout-dun-nouveau-service) de la documentation.
# Installation basique du site web
## Getting started
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
Actuellement, le site web est statique et généré par la moulinette `statoolinfos` toutes les heures.
Installer les paquets :
```
cd existing_repo
git remote add origin https://git.a-lec.org/a-lec/commissions/infrastructure/stats.git
git branch -M main
git push -uf origin main
apt-get install nginx python3-certbot-nginx
```
## Integrate with your tools
Ouvrir les ports http (80) et https (443) :
```
ufw allow 'Nginx HTTP'
ufw allow 'Nginx HTTPS'
```
- [ ] [Set up project integrations](https://git.a-lec.org/a-lec/commissions/infrastructure/stats/-/settings/integrations)
Configurer a minima le site web dans `/etc/nginx/sites-available/stats.a-lec.org` :
```
server
{
listen 80;
listen [::]:80;
## Collaborate with your team
server_name stats.a-lec.org;
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
access_log /var/log/nginx/stats.a-lec.org-access.log;
error_log /var/log/nginx/stats.a-lec.org-error.log;
}
```
## Test and Deploy
Activer la configuration :
```
cd /etc/nginx/sites-enabled/
ln -s ../sites-available/stats.a-lec.org
```
Use the built-in continuous integration in GitLab.
Vérifier que c'est bon et recharger :
```
nginx -t && systemctl reload nginx
```
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
Configurer le certificat SSl :
```
certbot --nginx
```
***
Créer un fichier générique `cat /etc/nginx/statoolinfos.conf` qui sera includé plus tard :
```
location /.well-known/statoolinfos/
{
charset utf-8;
types
{
text/plain properties;
}
# Editing this README
alias /srv/statoolinfos/well-known/statoolinfos/;
autoindex on;
}
```
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
Mettre beau le fichier `/etc/nginx/sites-enabled/stats.a-lec.org` :
```
server
{
listen 80;
listen [::]:80;
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
server_name stats.a-lec.org;
## Name
Choose a self-explaining name for your project.
access_log /var/log/nginx/stats.a-lec.org-access.log;
error_log /var/log/nginx/stats.a-lec.org-error.log;
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
#return 302 https://$host$request_uri;
return 302 https://stats.a-lec.org$request_uri;
}
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
server
{
set_real_ip_from 192.169.1.1;
real_ip_header proxy_protocol;
listen [::]:443 ssl ipv6only=on;
listen 443 ssl proxy_protocol;
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
server_name stats.a-lec.org;
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
access_log /var/log/nginx/stats.a-lec.org-access.log;
error_log /var/log/nginx/stats.a-lec.org-error.log;
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
ssl_certificate /etc/letsencrypt/live/stats.a-lec.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/stats.a-lec.org/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
# StatoolInfos.
include /etc/nginx/statoolinfos.conf;
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
root /var/www/stats.a-lec.org;
location = /
{
index index.xhtml;
}
}
```
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
Vérifier que c'est bon et recharger :
```
nginx -t && systemctl reload nginx
```
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
# Déploiement du générateur de site web
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## Configuration /etc/hosts
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
Le programme StatoolInfos va faire des requêtes web sur la vm donc il a besoin de la voir.
Pour éviter d'avoir un court-circuit proxy ipv4, modifier le fichier `/etc/hosts` pour déclarer stats.a-lec.org en ipv6 locale uniquement :
```
127.0.0.1 localhost
127.0.1.1 stats
## License
For open source projects, say how it is licensed.
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback stats.a-lec.org stats
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
```
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
## Configuration de Java
Java :
```
apt-get install openjdk-11-jre-headless
```
## Espace de travail
Dossiers de travail :
```
mkdir -p /srv/statoolinfos/{conf,inputs,well-known}
mkdir -p /srv/statoolinfos/well-known/statoolinfos/
ln -s /var/www/stats.a-lec.org /srv/statoolinfos/www
```
Déployer les catégories : copier le dossier https://framagit.org/chatons/chatonsinfos/-/tree/master/StatoolInfos/categories dans `/srv/statoolinfos/inputs/`.
Créer un fichier de configuration pour le générateur :
```
# [Configuration]
conf.class=federation
conf.protocol=StatoolInfos-0.5
conf.crawl.input=https://stats.a-lec.org/.well-known/statoolinfos/a-lec.properties
conf.crawl.cache=/srv/statoolinfos/cache/
conf.htmlize.categories=/srv/statoolinfos/inputs/categories/categories.properties
conf.htmlize.input=https://stats.a-lec.org/.well-known/statoolinfos/a-lec.properties
conf.htmlize.directory=/srv/statoolinfos/www/
```
Créer le fichier `properties` de type `federation` nommé `/srv/statoolinfos/well-known/statoolinfos/alec.properties` :
```
# [File]
file.class=federation
file.generator=Cpm
file.datetime=2022-01-12T02:24:35
file.protocol=StatoolInfos-0.4.0
# [Federation]
federation.name=Libre en Commnuns
federation.description=L'association Libre en Communs
federation.website=https://www.a-lec.org/
federation.logo=https://stats.a-lec.org/.well-known/statoolinfos/alec-logo.svg
federation.contact.url=
federation.contact.email=contact@a-lec.org
federation.socialnetworks.mastodon=
federation.legal.url=
federation.guide.user=
federation.guide.technical=
federation.startdate=12/2021
federation.enddate=
# [Subs]
subs.infra=https://stats.a-lec.org/.well-known/statoolinfos/infra.properties
subs.services=https://stats.a-lec.org/.well-known/statoolinfos/services.properties
# [Metrics]
metrics.members.count.2021 = 2
metrics.members.in.2021 = 2
metrics.members.out.2021 = 0
metrics.members.count.2022 = 2
metrics.members.in.2022 = 0
metrics.members.out.2022 = 0
```
Vérifier la validité des valeurs et placer dans `/srv/statoolinfos/well-known/statoolinfos/` les images de logos nécessaires.
Créer le fichier `properties` de type `organization` nommé `/srv/statoolinfos/well-known/statoolinfos/infra.properties` :
```
# [File]
file.class=organization
file.generator=Cpm
file.datetime=2022-01-12T02:22:02.096451
file.protocol=StatoolInfos-0.5
# [Organization]
organization.name=Infra
organization.description=Les services d'infrastructure de Libre en Communs
organization.website=https://www.a-lec.org/
organization.logo=https://stats.a-lec.org/.well-known/statoolinfos/alec-infra-logo.svg
organization.status.level=ACTIVE
organization.status.description=En activité
organization.owner.name=Libre en Communs
organization.owner.website=https://www.a-lec.org/
organization.owner.logo=https://stats.a-lec.org/.well-known/statoolinfos/alec-logo-carre.png
organization.contact.url=
organization.contact.email=contact@a-lec.org
organization.socialnetworks.mastodon=
organization.legal.url=
organization.guide.user=
organization.guide.technical=
organization.startdate=12/2021
organization.enddate=
organization.memberof.chalec.status.level=IDLE
organization.memberof.chalec.status.description=
organization.memberof.chalec.startdate=01/10/2021
organization.memberof.chalec.enddate=
organization.country.name=France
organization.country.code=FR
# [Subs]
subs.stats.a-lec.org=https://stats.a-lec.org/.well-known/statoolinfos/stats.a-lec.org.properties
subs.www.a-lec.org=https://www.a-lec.org/.well-known/statoolinfos/www.a-lec.org.properties
```
Vérifier la validité des valeurs et placer dans `/srv/statoolinfos/well-known/statoolinfos/` les images de logos nécessaires.
Créer le fichier `properties` de type `federation` nommé `/srv/statoolinfos/well-known/statoolinfos/services.properties` :
```
# [File]
file.class=organization
file.generator=Cpm
file.datetime=2022-01-12T02:22:02.096451
file.protocol=StatoolInfos-0.5
# [Organization]
organization.name=Services au public
organization.description=Les services au public de Chalec
organization.website=https://www.chalec.org/
organization.logo=https://stats.chalec.org/.well-known/statoolinfos/chalec-services-logo.svg
organization.status.level=ACTIVE
organization.status.description=En activité
organization.owner.name=Chalec
organization.owner.website=https://www.chalec.org/
organization.owner.logo=https://www.chalec.org/.well-known/statoolinfos/chalec-logo-carre.png
organization.contact.url=
organization.contact.email=contact@chalec.org
organization.socialnetworks.mastodon=
organization.legal.url=
organization.guide.user=
organization.guide.technical=
organization.startdate=12/2021
organization.enddate=
organization.memberof.chalec.status.level=IDLE
organization.memberof.chalec.status.description=
organization.memberof.chalec.startdate=12/2021
organization.memberof.chalec.enddate=
organization.country.name=France
organization.country.code=FR
# [Subs]
subs.audio=https://audio.chalec.org/.well-known/statoolinfos/audio.chalec.org.properties
subs.libreverse=https://libreverse.chalec.org/.well-known/statoolinfos/libreverse.chalec.org.properties
```
Vérifier la validité des valeurs et placer dans `/srv/statoolinfos/well-known/statoolinfos/` les images de logos nécessaires.
Récupérer la dernière version de StatoolInfos sur https://forge.devinsy.fr/devinsy/statoolinfos/releases. Puis placer dans `/srv/statoolinfos/bin/` les fichiers `statoolinfos.jar` et `statoolinfos.sh`. Faire un lien :
```
cd /srv/statoolinfos/bin/
ln -s statoolinfos.sh statoolinfos
```
Créer le script `/srv/statoolinfos/bin/crawl` :
```
#!/bin/bash
cd $(dirname "$0")
./statoolinfos crawl ../conf/alec.conf
```
Créer le script `/srv/statoolinfos/bin/htmlize` :
```
#!/bin/bash
cd $(dirname "$0")
./statoolinfos htmlize ../conf/alec.conf
chmod -R go+rX ../www/
```
Créer le script `/srv/statoolinfos/bin/probe` :
```
#!/bin/bash
cd $(dirname "$0")
./statoolinfos probe -previousday ../conf/
```
Créer le script `/srv/statoolinfos/bin/refresh` :
```
#!/bin/bash
cd $(dirname "$0")
./probe
./crawl
./uptime
./htmlize
```
Créer le script `/srv/statoolinfos/bin/uptime` :
```
#!/bin/bash
cd $(dirname "$0")
./statoolinfos uptime ../conf/alec.conf
```
Créer le cron `/etc/cron.d/statoolinfos` :
```
4 * * * * root /srv/statoolinfos/bin/refresh >> /srv/statoolinfos/statoolinfos-cron.log
```
# Ajout d'un nouveau service
Tout se passe sur la vm du service, sauf la dernière étape.
## Configuration d'Apache
Création d'un fichier générique `cat /etc/apache2/statoolinfos.conf` qui sera includé plus tard :
```
Alias "/.well-known/statoolinfos/" "/srv/statoolinfos/well-known/statoolinfos/"
<Directory "/srv/statoolinfos/well-known/statoolinfos/">
# Autoriser l'indexation par Apache et donc un affichage même sans index.html ou .php…
Options +Indexes
# Forcer le type de fichier correspondant aux extensions .properties avec le support de l'UTF-8.
# Permet l'affichage dans le navigateur et l'affichage correct des accents par exemple.
AddType "text/plain; charset=UTF-8" .properties
# Donner le droit d'accès à tout le monde.
Require all granted
</Directory>
```
## Configuration de Nginx
Création d'un fichier générique `cat /etc/nginx/statoolinfos.conf` qui sera includé plus tard :
```
# location ^~ /.well-known/statoolinfos/ # En cas de conflit de location.
location /.well-known/statoolinfos/
{
charset utf-8;
types
{
text/plain properties;
}
alias /srv/statoolinfos/well-known/statoolinfos/;
autoindex on;
}
```
## Espace de travail
Dossiers de travail :
```
mkdir -p /srv/statoolinfos/{conf,well-known}
mkdir -p /srv/statoolinfos/well-known/statoolinfos/
```
## Déclaration du service
Configurer l'accès well-known en ajoutant à la fin de la configuration Apache du site :
```
# StatoolInfos
Include statoolinfos.conf
</VirtualHost>
```
Ou au début du bloc `server` du fichier de configuration Nginx du site :
```
# StatoolInfos.
include /etc/nginx/statoolinfos.conf;
```
Avertissement : parfois, pour certaines applications, un conflit de `location` peut exister et alors une adaptation est à faire.
Créer le fichier properties de type service à `/srv/statoolinfos/well-known/statoolinfos/foo.chalec.org.properties`. Une documentation des propriétés est disponible là : https://framagit.org/chatons/chatonsinfos/-/blob/master/MODELES/service.properties
Vérifier les permissions de lecture :
```
chmod go+r /srv/statoolinfos/well-known/statoolinfos/*
```
## Renseigner le fichier properties d'organisation
Sur la vm `stats.alec.org`, éditer `/srv/statoolinfos/well-known/statoolinfos/services.properties` et ajouter une ligne `subs` :
```
subs.foo=https://foo.alec.org/.well-known/statoolinfos/foo.alec.org.properties
```
# Génération de métrics
## Configuration de Java
Java :
```
apt-get install openjdk-11-jre-headless
```
## Déploiement de statoolinfos
Récupérer la dernière version de StatoolInfos sur https://forge.devinsy.fr/devinsy/statoolinfos/releases. Puis placer dans `/srv/statoolinfos/bin/` les fichiers `statoolinfos.jar` et `statoolinfos.sh`. Faire un lien :
```
cd /srv/statoolinfos/bin/
ln -s statoolinfos.sh statoolinfos
```
## Configuration
Créer un fichier de configuration dans `/srv/statoolinfos/conf/foo.a-lec.org.conf` :
```
conf.class=service
conf.protocol=StatoolInfos-0.5
conf.probe.types=HttpAccessLog, HttpErrorLog
conf.probe.httpaccesslog.file=/var/log/nginx/foo.a-lec.org-access.log*
conf.probe.httperrorlog.file=/var/log/nginx/foo.a-lec.org-error.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.a-lec.org-metrics.properties
```
Pour prendre connaissance de paramètres de configuration spécifiques, consulter le [projet StatoolInfos](https://forge.devinsy.fr/devinsy/statoolinfos#generate-metrics-files).
Vérifier les permissions de lecture :
```
chmod go+r /srv/statoolinfos/well-known/statoolinfos/*
```
## Génération
Faire une première génération de métrics :
```
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/
```
Vérifier les permissions de lecture :
```
chmod go+r /srv/statoolinfos/well-known/statoolinfos/*
```
Créer un cron `/etc/cron.d/statoolinfos` :
```
2 * * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/statoolinfos-cron.log
```
Faire un lien :
```
cd /srv/statoolinfos
ln -s /etc/cron.d/statoolinfos cron
```
## Renseigner le fichier properties de service
Sur la vm du service, éditer `/srv/statoolinfos/well-known/statoolinfos/foo.a-lec.org.properties` et ajouter une ligne `subs` :
```
subs.foo=https://foo.a-lec.org/.well-known/statoolinfos/foo.a-lec.org-metrics.properties
```