Blackbox Exporter allows Prometheus to probe endpoints over HTTP, HTTPS, ICMP, and TCP.

Step 1: Create Blackbox Exporter User
```bash
sudo useradd --no-create-home --shell /bin/false blackbox_exporter
```

Step 2: Download and Extract Blackbox Exporter
```bash
cd /tmp
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.14.0/blackbox_exporter-0.14.0.linux-amd64.tar.gz
tar -xvf blackbox_exporter-0.14.0.linux-amd64.tar.gz
```

Step 3: Move Binaries to System Directory
```bash
sudo cp blackbox_exporter-0.14.0.linux-amd64/blackbox_exporter /usr/local/bin/
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
```

Step 4: Clean Up Installation Files
```bash
rm -rf blackbox_exporter-0.14.0.linux-amd64*
```

Step 5: Create Configuration Directory
```bash
sudo mkdir /etc/blackbox_exporter
sudo nano /etc/blackbox_exporter/blackbox.yml
```

Step 6: Configure `blackbox.yml`
Paste the following configuration:
```yaml
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [] # Defaults to 2xx
method: GET
```
Save and exit (`CTRL + X`, then `Y`, then `ENTER`).

Step 7: Update Configuration Permissions
```bash
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
```

Step 8: Create a Systemd Unit File
```bash
sudo nano /etc/systemd/system/blackbox_exporter.service
```
Step 9: Populate the Systemd Configuration
```ini
[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target
```
Step 10: Reload Systemd and Start Blackbox Exporter

```bash
sudo systemctl daemon-reload
sudo systemctl start blackbox_exporter
```

Step 11: Verify the Service is Running
```bash
sudo systemctl status blackbox_exporter
```
Step 12: Enable on Boot
```bash
sudo systemctl enable blackbox_exporter

Prometheus is used to collect and store metrics from various exporters. The Prometheus configuration file (prometheus.yml) includes two jobs: one for Prometheus itself and another for the Blackbox Exporter. The latter is configured to scrape our target website on port 9115.

Step 1: Download and Extract Prometheus
```bash
cd /tmp
wget https://github.com/prometheus/prometheus/releases/latest/download/prometheus-2.41.0.linux-amd64.tar.gz
tar -xvf prometheus-2.41.0.linux-amd64.tar.gz
```

Step 2: Move Binaries to System Directory
```bash
sudo cp prometheus-2.41.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.41.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
```

Step 3: Create Configuration Directory
```bash
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

Step 4: Configure Prometheus
```bash
sudo nano /etc/prometheus/prometheus.yml
```
Paste the following configuration:
```yaml
global:
scrape_interval: 15s

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
```
Save and exit.

Step 5: Create a Systemd Unit File
```bash
sudo nano /etc/systemd/system/prometheus.service
```
Paste the following:
```ini
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=:9090

[Install]
WantedBy=multi-user.target
```
Save and exit.

Step 6: Start and Enable Prometheus
```bash
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
```

Step 7: Verify Prometheus is Running
sudo systemctl status Prometheus

Step 13: Edit Prometheus Configuration
```bash
sudo nano /etc/prometheus/prometheus.yml
```
Append the following:
```yaml
global:
scrape_interval: 15s

external_labels:
monitor: 'prometheus'

scrape_configs:
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.google.com
- https://www.youtube.com
- http://13.127.207.158
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 13.127.207.158:9115
- target_label: __address__
replacement: 52.66.108.181:9115
```

Step 14: Restart Prometheus
```bash
sudo systemctl restart prometheus
```

Step 1: Install Dependencies
```bash
sudo apt-get install -y adduser libfontconfig1
```
Step 2: Download and Install Grafana
```bash
wget https://dl.grafana.com/oss/release/grafana_7.3.4_amd64.deb
sudo dpkg -i grafana_7.3.4_amd64.deb
```
Step 3: Start and Enable Grafana
```bash
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
sudo systemctl enable grafana-server.service
```

Step 1: Verify Blackbox Exporter is Running
```bash
curl "http://localhost:9115/probe?module=http_2xx&target=https://www.google.com"
```
Step 2: Check Prometheus Targets
- Open Prometheus UI:
```
http://<your-server-ip>:9090/targets

Creating a Grafana Dashboard

First, ensure that Grafana is installed and running. You can access the Grafana dashboard by navigating to http://<your-server-ip>:3000 in your web browser.

To begin integrating Prometheus with Grafana, navigate to the “Connections” menu located in the left sidebar. Within the “Connections” menu, select the “Data Sources” option. This will present you with a list of available data sources. From this list, choose Prometheus.

Upon selecting Prometheus, you will be presented with a section labeled “HTTP.” In this section, enter the URL of your Prometheus server. The default URL is http://localhost:9090, assuming Prometheus is running on the same machine as Grafana. However, if your Prometheus server resides on a different machine, you will need to provide the appropriate URL (if localhost doesn’t work, use IPv4 instead like “http://IPv4:9090”).

Also, you need to Allow Inbound Rule in the EC2 Security Group for destination ports: 3000(Grafana), 9115(BlackBox) and 9090(Prometheus) in order reach to the Grafana Dashboard and the Blackbox exporter to have the ability to do the PING Monitoring that we have specified.

Once you have successfully added Prometheus as a data source, you can start creating dashboards to visualize your metrics. To do so, locate the “+” icon in the left sidebar and click on it to reveal the “Create” menu. From the “Create” menu, select “Dashboard” to initiate the dashboard creation process.

After creating a new dashboard, you can start adding panels to display the metrics you want to monitor. To add a panel, simply click on the “Add Panel” button. For each panel, you will need to configure the metric query using PromQL (Prometheus Query Language). PromQL provides a powerful way to filter and aggregate metrics from your Prometheus server.

Saving the Dashboard

After meticulously crafting your dashboard, don’t forget to save it for future reference. Click on the disk icon in the top menu, provide a descriptive name for your dashboard, and click “Save.” This will ensure that your personalized dashboard layout and settings are preserved for future access.

Checking queries in Prometheus

For easier query checkups go to Prometheus UI and start typing a prompt:

Choose the prompt then hit execute:

Copy the output of the prompt and then paste in Grafana under Queries.

To add alert rules in Grafana, follow these steps:
Begin by navigating to the Alerts and Rules section within Grafana.Hover your cursor over the Grafana Cloud Alerting icon and then click on “Alerts and rules.” If you have multiple Prometheus or Loki data sources, a dropdown menu will appear at the top. Choose the desired data source for which you want to create or modify rules. Click the “Edit Rules” button to access the alert rule management interface. To create a new alert rule, click the “Add Rule” button.

In conclusion, setting up Blackbox Exporter, Prometheus, and Grafana on an AWS EC2 instance provides a powerful monitoring solution for tracking system performance and probing endpoints. By configuring Prometheus to scrape metrics from the Blackbox Exporter and visualizing them in Grafana, you can ensure comprehensive observability. Proper firewall rules and data flow configurations are crucial to enable seamless communication between these components. With this setup in place, you can proactively monitor uptime, detect issues, and maintain system reliability.