Grafana Loki

Loki is a centralised log aggregation system.

Unlike other logging systems, loki is designed to index only metadata, which is usually referred to as labels, in the loki world.

It stores the logs in chunks which are then compressed to significantly reduce the cost of loki.

Flow of Logs

The Application writes different level of logs to a log file.
Promtail is one of the agents of loki which can be configured to look for logs in a folder or a file.
Promtail Discovers the log files and monitors them and attaches labels associated with each log as specified in the configuration file of promtail.
Once promtail discovers change in the logs file, it ships the logs to Loki.
Loki stores the logs in chunks and maintains the labels in indexes. The configuration file of Loki specifies where to store these chunks of logs and indexes.
Loki provides few APIs through which we can query(LogQL) the chunks to fetch the logs.

Read path:

The querier receives the request for logs.
First, the querier passes the request to ingester for in-memory logs. Ingester returns the data to querier.
Querier lazily loads/queries the backend storage for the logs.
The querier de-duplicate service removes any redundant data as it fetches from ingester and backend store.
The querier returns the logs to the front end.

Write path:

The Distributor receives an HTTP request to store data for streams.
Distributor validates each stream of data received and sends each stream to appropriate ingesters.
Ingester will store logs in memory before writing the logs into the backend store.
Ingester will create or append logs to existing chunks for stream data.
Distributor responds with success code to the promtail.

For more information, do visit Loki Documentation.

Loki provides APIs through which we can send LogQL queries as request parameters, through which we can get log messages.

GET /loki/api/v1/query

/loki/api/v1/query allows for doing queries against a single point in time. The URL query parameters support the following values:

query: The LogQL query to perform
limit: The max number of entries to return
time: The evaluation time for the query as a nanosecond Unix epoch. Default = now.
direction: Determines the sort order of logs. Supported values are forward or backward. Default = backward.

`GET /loki/api/v1/query_range`

/loki/api/v1/query_range is used to do a query over a range of time and accepts the following query parameters in the URL:

query: The LogQL query to perform
limit: The max number of entries to return
start: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
end: The end time for the query as a nanosecond Unix epoch. Default = now.
step: Query resolution step width in duration format or float number of seconds.

Loki provides more APIs. Here, we have discussed two most frequently used APIs.

For more information, refer Loki's HTTP API.

On Mac:

We can use brew to install both loki and promtail.

brew install loki

brew install promtail

You can find the configuration file in /opt/homebrew/etc/promtail-local-config.yaml or loki-local-config.yaml.

On Linux:

<TBA>