Notes

BISMILLAH :)


prometheus Installation steps:-
*********************************

Reference:- https://devopscube.com/install-configure-prometheus-linux/

Below are the steps to install prometheus in linux Machine from source:-


-> Download the prometheus tar file from "https://prometheus.io/download/" into your local windows machine from there transfer it to "/u01" linux location.

-> Open the terminal and change to the location "/u01", now run the below commands


* tar -xvf prometheus-2.17.1.linux-amd64.tar.gz

* mv prometheus-2.17.1.linux-amd64 prometheus-files

* sudo useradd prometheus (set the password also as prometheus)

* sudo mkdir /etc/prometheus

* sudo mkdir /var/lib/prometheus

* sudo chown prometheus:prometheus /etc/prometheus

* sudo chown prometheus:prometheus /var/lib/prometheus

* sudo cp prometheus-files/prometheus /usr/local/bin/

* sudo cp prometheus-files/promtool /usr/local/bin/

* sudo chown prometheus:prometheus /usr/local/bin/prometheus

* sudo chown prometheus:prometheus /usr/local/bin/promtool

* sudo cp -r prometheus-files/consoles /etc/prometheus

* sudo cp -r prometheus-files/console_libraries /etc/prometheus

* sudo chown -R prometheus:prometheus /etc/prometheus/consoles

* sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

* sudo vi /etc/prometheus/prometheus.yml

=====
# my global config
global:
scrape_interval: 10s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
#evaluation_interval: 5s # Evaluate rules every 15 seconds. The default is every 1 minute.

# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
#- targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
#- "alert.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

- job_name: 'GE4_Integrator'
metrics_path: '/GE4/arcce-integrator/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

- job_name: 'Enricher'
metrics_path: '/GE4/arcce-enricher/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

- job_name: 'Generator'
metrics_path: '/GE4/arcce-generator/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

- job_name: 'Delivery'
metrics_path: '/GE4/arcce-delivery/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

- job_name: 'GE4_OmegaWrapper'
metrics_path: '/GE4/arcce-omega-wrapper/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

- job_name: 'GE4_Ack'
metrics_path: '/GE4/arcce-acknowledge/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

- job_name: 'GE4_BounceBack'
metrics_path: '/GE4/arcce-bounceback/actuator/prometheus'
scrape_interval: 10s
scheme: https
basic_auth:
static_configs:
- targets: ['cce-nonprod.us.dell.com']

remote_write:
- url: "http://10.161.12.128:9201/write"
remote_read:
- url: "http://10.161.12.128:9201/read"

=====

* sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

* sudo vi /etc/systemd/system/prometheus.service

==============================
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus
--config.file /etc/prometheus/prometheus.yml
--storage.tsdb.path /var/lib/prometheus/
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
--web.enable-lifecycle

[Install]
WantedBy=multi-user.target


==============================

* sudo systemctl daemon-reload

* sudo systemctl start prometheus

* sudo systemctl status prometheus

now, can access prometheus using the url "http://<prometheus-ip>:9090/graph"

Thats all prometheus has been sucesfully installed in your Linux machine.

Note:-

when ever there is a change in the prometheus.yml file. In order to apply those changes all we need to do is hitting the reload url as below

* curl -X POST http://localhost:9090/-/reload

In order to check the respective yml file changes, follow the below steps:
click the configuration link present in the status dropdown.



postgres Installation steps:-
******************************


Reference:-
https://www.thegeekstuff.com/2009/04/linux-postgresql-install-and-configure-from-source/

Below are the steps to install postgres in linux Machine from source:-

-> Download the prometheus tar file from "https://www.postgresql.org/ftp/source/v10.11/" into your local windows machine.

Tranfer the downloaded postgres tar file to the below loaction(i.e.,"/u01/postgresql")

* sudo mkdir /u01/postgresql

* tar postgresql-10.11.tar.gz

* cd postgresql-10.11

* ./configure --without-readline

Note:-
Here, readline is a feature which is used to capture the history of commands used. In order to simplyfy the installation process we have isabled it by using "--without-readline" flag.

* make

* make install

Thats all, postgress installation is completed.

after installation we can see folder creation under below location

* ls -l /usr/local/pgsql/

Create postgreSQL user account

* sudo adduser postgres (give the password also as postgres)

Create the postgres data directory and make postgres user as the owner.

* mkdir /usr/local/pgsql/data

* chown postgres:postgres /usr/local/pgsql/data

* ls -ld /usr/local/pgsql/data

Before we start creating any postgreSQL database, the empty data directory created in the above step should be initialized.

* su - postgres (password is postgress)

* /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data/

Make sure all postgres DB configuration files (For example, postgresql.conf) are created under the data directory as shown below.

* ls -l /usr/local/pgsql/data

now, Use the postgres postmaster command to start the postgreSQL server

* /usr/local/pgsql/bin/postmaster -D /usr/local/pgsql/data >logfile 2>&1 &

* cat logfile

The above command will give you message saying
"database system is ready to accept connections"

Create postgreSQL DB and test the installation.

* /usr/local/pgsql/bin/createdb ar_cce_monitoring_db

* /usr/local/pgsql/bin/psql ar_cce_monitoring_db

if u see "ar_cce_monitoring_db=" the you were succesfully connected to postgresDB.

To quit from DB enter the below command.

* q


Time ScaleDB Installation :
****************************

Installation of TimescaleDB requires

1. postgresSQL
2. c Compiler
3. CMake ersion 3.4 or greater


check weather postgress is installed or not.
* postgres -V

check weather C Compiler is installed or not.
* gcc --version

check weather c make is installed or not.
* cmake --version

if cmake has not installed then need to add the below dependencies

-> Download the CMake tar file from "https://github.com/Kitware/CMake/releases/download/v3.17.0/cmake-3.17.0-Linux-x86_64.tar.gz" into your local windows machine.

Tranfer the downloaded CMake tar file to the below loaction(i.e.,"/u01/postgresql")

* tar xvfz cmake-3.17.0-Linux-x86_64.tar.gz

Go to bashrc file and add the path of cmake by running below command

* vi ~/.bashrc

=========================================
export PATH=/u01/postgresql/cmake-3.17.0-rc3-Linux-x86_64/bin:$PATH

=========================================

* source ~/.bashrc

check weather cmake path is included are not by using below command

* echo $PATH

if its included now check the version of CMake

* cmake --version


Download TimeScale DB from git repository "https://github.com/timescale/timescaledb.git"

Manually copy the "timescaledb" folder to "/u01/postgresql" linux folder


* ./bootstrap

* cd build && make

* make install

You will need to edit your postgresql.conf file to include the TimescaleDB library

* su - postgres (password is postgress)

* cd /usr/local/pgsql/data

* vi postgresql.conf

add "shared_preload_libraries = 'timescaledb'" to the postgresql.conf file.

* service postgresql stop

* service postgresql start

* /usr/local/pgsql/bin/postmaster -D /usr/local/pgsql/data >logfile 2>&1 &

* cat logfile

* /usr/local/pgsql/bin/psql ar_cce_monitoring_db

* SET timescaledb.allow_install_without_preload = 'on';

* CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

That's it! now u have connected to dataBase.


Installation of pg_prometheus:
*******************************

Make sure you have PostgreSQL bin in your PATH and the postgresql-devel package for your version of PostgreSQL before compiling from source for that follow the below commands

firstly we need to install "postgresql-devel" package, but this package has inter dependent files. so, we will find install those files and then will install postgresql-devel package. Follow the below commands in same order.

* mkdir /u01/postgresql/postgresql-devel-libs

* cd /u01/postgresql/postgresql-devel-libs

Reference:

https://centos.pkgs.org/7/postgresql-10-x86_64/postgresql10-devel-10.11-2PGDG.rhel7.x86_64.rpm.html

Download the below dependencies to our local machine and the transfer them manually to "postgresql-devel-libs" package.

https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/postgresql10-devel-10.11-2PGDG.rhel7.x86_64.rpm

https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/postgresql10-libs-10.11-2PGDG.rhel7.x86_64.rpm

https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/postgresql10-10.11-2PGDG.rhel7.x86_64.rpm

http://mirror.centos.org/centos/7/os/x86_64/Packages/libicu-devel-50.2-3.el7.x86_64.rpm

http://mirror.centos.org/centos/7/os/x86_64/Packages/libicu-50.2-3.el7.x86_64.rpm

http://mirror.centos.org/centos/7/os/x86_64/Packages/postgresql-devel-9.2.24-1.el7_5.x86_64.rpm


now run the commands in the same order, as they are interdependent on each other.

* rpm -i libicu-50.2-3.el7.x86_64.rpm

* rpm -i libicu-devel-50.2-3.el7.x86_64.rpm

* rpm -i postgresql10-10.11-2PGDG.rhel7.x86_64.rpm

* rpm -i postgresql10-libs-10.11-2PGDG.rhel7.x86_64.rpm

* rpm -i postgresql-devel-9.2.24-1.el7_5.x86_64.rpm

* rpm -i postgresql10-devel-10.11-2PGDG.rhel7.x86_64.rpm

Go to bashrc file and add the path of cmake by running below command

* vi ~/.bashrc

=========================================
export PATH=/u01/CCE/jdk1.8.0_131/bin:$PATH

=========================================

* source ~/.bashrc

check weather cmake path is included are not by using below command

* echo $PATH

-> Download the pg_prometheus tar file from "https://codeload.github.com/timescale/pg_prometheus/tar.gz/0.2.1" into your local windows machine.

Tranfer the downloaded pg_prometheus tar file to the below loaction(i.e.,"/u01/postgresql")

run the below commands

* tar -xvf pg_prometheus-0.2.1.tar.gz

* cd /u01/postgresql/pg_prometheus-0.2.1

* make

* make install # Might require super user permissions

now,edit postgresql.conf to include the pg_prometheus extension

* su - postgres (password is postgress)

* cd /usr/local/pgsql/data

* vi postgresql.conf

add "shared_preload_libraries = 'pg_prometheus'" to the postgresql.conf file.

* service postgresql stop

* service postgresql start

* /usr/local/pgsql/bin/postmaster -D /usr/local/pgsql/data >logfile 2>&1 &

* cat logfile

* /usr/local/pgsql/bin/psql ar_cce_monitoring_db

* CREATE EXTENSION pg_prometheus;

Optionally grant permissions to the database user (prometheus) that will own the Prometheus data:

* CREATE ROLE prometheus WITH LOGIN PASSWORD 'secret';

* GRANT ALL ON SCHEMA prometheus TO prometheus;

now, we will create the metrics table to store our prometheus data

* SELECT create_prometheus_table('metrics',use_timescaledb=>true);


Installing prometheus-postgres Adapter:-
*****************************************

-> Download the Adapter tar file "prometheus-postgresql-adapter-0.6.0-linux-amd64.tar" from "https://github.com/timescale/prometheus-postgresql-adapter/releases" into your local windows machine.

Tranfer the downloaded Adapter tar file to the below loaction(i.e.,"/u01/postgresql")

now run the following commands

* tar -xvf prometheus-postgresql-adapter-0.6.0-linux-amd64.tar.gz

now we must tell prometheus to use this remote storage adapter by adding the following lines to prometheus.yml

* sudo vi /etc/prometheus/prometheus.yml

===================
remote_write:
- url: "http://localhost:9201/write"
remote_read:
- url: "http://localhost:9201/read"
===================

in order to reflect the prometheus.yml file changes, we need to reload the prometheus as below


* curl -X POST http://localhost:9090/-/reload

In order to check the respective yml file changes, follow the below steps:
click the configuration link present in the status dropdown.

now, start the prometheus by hitting the below command

* ./prometheus-postgresql-adapter --pg-user postgres -pg-database test -pg-password postgres -log-level "debug" -pg-copy-table metrics_copy

Thats it.! Adapter is up and running.


Grafana Instalation steps:-
****************************

Reference: https://linuxtechlab.com/install-grafana-integrate-with-prometheus-for-complete-monitoring/


-> Download the Grafana tar file from "https://dl.grafana.com/oss/release/grafana-6.4.4.linux-amd64.tar.gz" into your local windows machine.

Tranfer the downloaded Grafana tar file to the below loaction(i.e.,"/u01")

now follow the below commands

* tar xvf grafana-6.4.4.linux-amd64.tar.gz

* cd grafana-6.4.4/bin

Now to start the service we will execute the following command,

* nohup ./grafana-server &

Note:- To restart the Grafana service, kill the PID of grafana & run the above-mentioned command again.

Now that the Grafana server is up, we can configure it from our web-browser. Open the following URL from the browser

http://localhost:3000