A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from http://www.erlang.org/doc/apps/kernel/logger_chapter below:

Logging — kernel v10.3.1

Erlang provides a standard API for logging through Logger, which is part of the Kernel application. Logger consists of the API for issuing log events, and a customizable backend where log handlers, filters and formatters can be plugged in.

By default, the Kernel application installs one log handler at system start. This handler is named default. It receives and processes standard log events produced by the Erlang runtime system, standard behaviours and different Erlang/OTP applications. The log events are by default written to the terminal.

You can also configure the system so that the default handler prints log events to a single file, or to a set of wrap logs via disk_log.

By configuration, you can also modify or disable the default handler, replace it by a custom handler, and install additional handlers.

Overview

A log event consists of a log level, the message to be logged, and metadata.

The Logger backend forwards log events from the API, first through a set of primary filters, then through a set of secondary filters attached to each log handler. The secondary filters are in the following named handler filters.

Each filter set consists of a log level check, followed by zero or more filter functions.

The following figure shows a conceptual overview of Logger. The figure shows two log handlers, but any number of handlers can be installed.

---
title: Conceptual Overview
---
flowchart TD
    DB[(Config DB)]
    API ---> ML[Module Level <hr> Global Level <hr> Global Filters]
    API -.Update configuration.-> DB
    ML -.-> DB
    ML ---> HL1[Handler Level <hr> Handler Filter]
    ML ---> HL2[Handler Level <hr> Handler Filter]
    HL1 ---> HC1[Handler Callback]
    HL2 ---> HC2[Handler Callback]
    HL1 -.-> DB
    HL2 -.-> DB
    subgraph Legend
        direction LR
        start1[ ] -->|Log event flow| stop1[ ]
        style start1 height:0px;
        style stop1 height:0px;
        start2[ ] -.->|Look up configuration| stop2[ ]
        style start2 height:0px;
        style stop2 height:0px;
    end

Log levels are expressed as atoms. Internally in Logger, the atoms are mapped to integer values, and a log event passes the log level check if the integer value of its log level is less than or equal to the currently configured log level. That is, the check passes if the event is equally or more severe than the configured level. See section Log Level for a listing and description of all log levels.

The primary log level can be overridden by a log level configured per module. This is to, for instance, allow more verbose logging from a specific part of the system.

Filter functions can be used for more sophisticated filtering than the log level check provides. A filter function can stop or pass a log event, based on any of the event's contents. It can also modify all parts of the log event. See section Filters for more details.

If a log event passes through all primary filters and all handler filters for a specific handler, Logger forwards the event to the handler callback. The handler formats and prints the event to its destination. See section Handlers for more details.

Everything up to and including the call to the handler callbacks is executed on the client process, that is, the process where the log event was issued. It is up to the handler implementation if other processes are involved or not.

The handlers are called in sequence, and the order is not defined.

Logger API

The API for logging consists of a set of macros, and a set of functions of the form logger:Level/1,2,3, which are all shortcuts for logger:log(Level,Arg1[,Arg2[,Arg3]]).

The macros are defined in logger.hrl, which is included in a module with the directive

-include_lib("kernel/include/logger.hrl").

The difference between using the macros and the exported functions is that macros add location (originator) information to the metadata, and performs lazy evaluation by wrapping the logger call in a case statement, so it is only evaluated if the log level of the event passes the primary log level check.

Log Level

The log level indicates the severity of a event. In accordance with the Syslog protocol, RFC 5424, eight log levels can be specified. The following table lists all possible log levels by name (atom), integer value, and description:

Level Integer Description emergency 0 system is unusable alert 1 action must be taken immediately critical 2 critical conditions error 3 error conditions warning 4 warning conditions notice 5 normal but significant conditions info 6 informational messages debug 7 debug-level messages

Table: Log Levels

Notice that the integer value is only used internally in Logger. In the API, you must always use the atom. To compare the severity of two log levels, use logger:compare_levels/2.

Log Message

The log message contains the information to be logged. The message can consist of a format string and arguments (given as two separate parameters in the Logger API), a string or a report.

Example, format string and arguments:

logger:error("The file does not exist: ~ts",[Filename])

Example, string:

logger:notice("Something strange happened!")

A report, which is either a map or a key-value list, is the preferred way to log using Logger as it makes it possible for different backends to filter and format the log event as it needs to.

Example, report:

?LOG_ERROR(#{ user => joe, filename => Filename, reason => enoent })

Reports can be accompanied by a report callback specified in the log event's metadata. The report callback is a convenience function that the formatter can use to convert the report to a format string and arguments, or directly to a string. The formatter can also use its own conversion function, if no callback is provided, or if a customized formatting is desired.

The report callback must be a fun with one or two arguments. If it takes one argument, this is the report itself, and the fun returns a format string and arguments:

fun((logger:report()) -> {io:format(),[term()]})

If it takes two arguments, the first is the report, and the second is a map containing extra data that allows direct conversion to a string:

fun((logger:report(),logger:report_cb_config()) -> unicode:chardata())

The fun must obey the depth and chars_limit parameters provided in the second argument, as the formatter cannot do anything useful of these parameters with the returned string. The extra data also contains a field named single_line, indicating if the printed log message may contain line breaks or not. This variant is used when the formatting of the report depends on the size or single line parameters.

Example, report, and metadata with report callback:

logger:debug(#{got => connection_request, id => Id, state => State},
             #{report_cb => fun(R) -> {"~p",[R]} end})

The log message can also be provided through a fun for lazy evaluation. The fun is only evaluated if the primary log level check passes, and is therefore recommended if it is expensive to generate the message. The lazy fun must return a string, a report, or a tuple with format string and arguments.

Metadata

Metadata contains additional data associated with a log message. Logger inserts some metadata fields by default, and the client can add custom metadata in three different ways:

See the description of the logger:metadata/0 type for information about which default keys Logger inserts, and how the different metadata maps are merged.

Filters

Filters can be primary, or attached to a specific handler. Logger calls the primary filters first, and if they all pass, it calls the handler filters for each handler. Logger calls the handler callback only if all filters attached to the handler in question also pass.

A filter is defined as:

{FilterFun, Extra}

where FilterFun is a function of arity 2, and Extra is any term. When applying the filter, Logger calls the function with the log event as the first argument, and the value of Extra as the second argument. See logger:filter/0 for type definitions.

The filter function can return stop, ignore or the (possibly modified) log event.

If stop is returned, the log event is immediately discarded. If the filter is primary, no handler filters or callbacks are called. If it is a handler filter, the corresponding handler callback is not called, but the log event is forwarded to filters attached to the next handler, if any.

If the log event is returned, the next filter function is called with the returned value as the first argument. That is, if a filter function modifies the log event, the next filter function receives the modified event. The value returned from the last filter function is the value that the handler callback receives.

If the filter function returns ignore, it means that it did not recognize the log event, and thus leaves to other filters to decide the event's destiny.

The configuration option filter_default specifies the behaviour if all filter functions return ignore, or if no filters exist. filter_default is by default set to log, meaning that if all existing filters ignore a log event, Logger forwards the event to the handler callback. If filter_default is set to stop, Logger discards such events.

Primary filters are added with logger:add_primary_filter/2 and removed with logger:remove_primary_filter/1. They can also be added at system start via the Kernel configuration parameter logger.

Handler filters are added with logger:add_handler_filter/3 and removed with logger:remove_handler_filter/2. They can also be specified directly in the configuration when adding a handler with logger:add_handler/3 or via the Kernel configuration parameter logger.

To see which filters are currently installed in the system, use logger:get_config/0, or logger:get_primary_config/0 and logger:get_handler_config/1. Filters are listed in the order they are applied, that is, the first filter in the list is applied first, and so on.

For convenience, the following built-in filters exist:

Handlers

A handler is defined as a module exporting at least the following callback function:

log(LogEvent, Config) -> term()

This function is called when a log event has passed through all primary filters, and all handler filters attached to the handler in question. The function call is executed on the client process, and it is up to the handler implementation if other processes are involved or not.

Logger allows adding multiple instances of a handler callback. That is, if a callback module implementation allows it, you can add multiple handler instances using the same callback module. The different instances are identified by unique handler identities.

In addition to the mandatory callback function log/2, a handler module can export the optional callback functions adding_handler/1, changing_config/3, filter_config/1, and removing_handler/1. See logger_handler for more information about these function.

The following built-in handlers exist:

Formatters

A formatter can be used by the handler implementation to do the final formatting of a log event, before printing to the handler's destination. The handler callback receives the formatter information as part of the handler configuration, which is passed as the second argument to HModule:log/2.

The formatter information consist of a formatter module, FModule and its configuration, FConfig. FModule must export the following function, which can be called by the handler:

format(LogEvent,FConfig)
	-> FormattedLogEntry

The formatter information for a handler is set as a part of its configuration when the handler is added. It can also be changed during runtime with logger:set_handler_config(HandlerId,formatter,{Module,FConfig}) , which overwrites the current formatter information, or with logger:update_formatter_config/2,3, which only modifies the formatter configuration.

If the formatter module exports the optional callback function check_config(FConfig), Logger calls this function when the formatter information is set or modified, to verify the validity of the formatter configuration.

If no formatter information is specified for a handler, Logger uses logger_formatter as default. See the logger_formatter manual page for more information about this module.

Configuration

At system start, Logger is configured through Kernel configuration parameters. The parameters that apply to Logger are described in section Kernel Configuration Parameters. Examples are found in section Configuration Examples.

During runtime, Logger configuration is changed via API functions. See section Configuration API Functions in the logger manual page.

Primary Logger Configuration

Logger API functions that apply to the primary Logger configuration are:

The primary Logger configuration is a map with the following keys:

Handler Configuration

Logger API functions that apply to handler configuration are:

The configuration for a handler is a map with the following keys:

Notice that level and filters are obeyed by Logger itself before forwarding the log events to each handler, while formatter and all handler specific options are left to the handler implementation.

Kernel Configuration Parameters

The following Kernel configuration parameters apply to Logger:

Configuration Examples

The value of the Kernel configuration parameter logger is a list of tuples. It is possible to write the term on the command line when starting an erlang node, but as the term grows, a better approach is to use the system configuration file. See the config(4) manual page for more information about this file.

Each of the following examples shows a simple system configuration file that configures Logger according to the description.

Modify the default handler to print to a file instead of standard_io:

[{kernel,
  [{logger,
    [{handler, default, logger_std_h,  % {handler, HandlerId, Module,
      #{config => #{file => "log/erlang.log"}}}  % Config}
    ]}]}].

Modify the default handler to print each log event as a single line:

[{kernel,
  [{logger,
    [{handler, default, logger_std_h,
      #{formatter => {logger_formatter, #{single_line => true}}}}
    ]}]}].

Modify the default handler to print the pid of the logging process for each log event:

[{kernel,
  [{logger,
    [{handler, default, logger_std_h,
      #{formatter => {logger_formatter,
                        #{template => [time," ",pid," ",msg,"\n"]}}}}
    ]}]}].

Modify the default handler to only print errors and more severe log events to "log/erlang.log", and add another handler to print all log events to "log/debug.log".

[{kernel,
  [{logger,
    [{handler, default, logger_std_h,
      #{level => error,
        config => #{file => "log/erlang.log"}}},
     {handler, info, logger_std_h,
      #{level => debug,
        config => #{file => "log/debug.log"}}}
    ]}]}].
Backwards Compatibility with error_logger

Logger provides backwards compatibility with error_logger in the following ways:

Error Handling

Logger does, to a certain extent, check its input data before forwarding a log event to filters and handlers. It does, however, not evaluate report callbacks, or check the validity of format strings and arguments. This means that all filters and handlers must be careful when formatting the data of a log event, making sure that it does not crash due to bad input data or faulty callbacks.

If a filter or handler still crashes, Logger will remove the filter or handler in question from the configuration, and print a short error message to the terminal. A debug event containing the crash reason and other details is also issued.

See section Log Message for more information about report callbacks and valid forms of log messages.

Example: Add a handler to log info events to file

When starting an Erlang node, the default behaviour is that all log events on level notice or more severe, are logged to the terminal via the default handler. To also log info events, you can either change the primary log level to info:

1> logger:set_primary_config(level, info).
ok

or set the level for one or a few modules only:

2> logger:set_module_level(mymodule, info).
ok

This allows info events to pass through to the default handler, and be printed to the terminal as well. If there are many info events, it can be useful to print these to a file instead.

First, set the log level of the default handler to notice, preventing it from printing info events to the terminal:

3> logger:set_handler_config(default, level, notice).
ok

Then, add a new handler which prints to file. You can use the handler module logger_std_h, and configure it to log to file:

4> Config = #{config => #{file => "./info.log"}, level => info}.
#{config => #{file => "./info.log"},level => info}
5> logger:add_handler(myhandler, logger_std_h, Config).
ok

Since filter_default defaults to log, this handler now receives all log events. If you want info events only in the file, you must add a filter to stop all non-info events. The built-in filter logger_filters:level/2 can do this:

6> logger:add_handler_filter(myhandler, stop_non_info,
                             {fun logger_filters:level/2, {stop, neq, info}}).
ok

See section Filters for more information about the filters and the filter_default configuration parameter.

Example: Implement a handler

logger_handler describes the callback functions that can be implemented for a Logger handler.

A handler callback module must export:

It can optionally also export some, or all, of the following:

When a handler is added, by for example a call to logger:add_handler(Id, HModule, Config), Logger first calls HModule:adding_handler(Config). If this function returns {ok,Config1}, Logger writes Config1 to the configuration database, and the logger:add_handler/3 call returns. After this, the handler is installed and must be ready to receive log events as calls to HModule:log/2.

A handler can be removed by calling logger:remove_handler(Id). Logger calls HModule:removing_handler(Config), and removes the handler's configuration from the configuration database.

When logger:set_handler_config/2,3 or logger:update_handler_config/2,3 is called, Logger calls HModule:changing_config(SetOrUpdate, OldConfig, NewConfig). If this function returns {ok,NewConfig1}, Logger writes NewConfig1 to the configuration database.

When logger:get_config/0 or logger:get_handler_config/0,1 is called, Logger calls HModule:filter_config(Config). This function must return the handler configuration where internal data is removed.

A simple handler that prints to the terminal can be implemented as follows:

-module(myhandler1).
-export([log/2]).

log(LogEvent, #{formatter := {FModule, FConfig}}) ->
    io:put_chars(FModule:format(LogEvent, FConfig)).

Notice that the above handler does not have any overload protection, and all log events are printed directly from the client process.

For information and examples of overload protection, please refer to section Protecting the Handler from Overload, and the implementation of logger_std_h and logger_disk_log_h .

The following is a simpler example of a handler which logs to a file through one single process:

-module(myhandler2).
-export([adding_handler/1, removing_handler/1, log/2]).
-export([init/1, handle_call/3, handle_cast/2, terminate/2]).

adding_handler(Config) ->
    MyConfig = maps:get(config,Config,#{file => "myhandler2.log"}),
    {ok, Pid} = gen_server:start(?MODULE, MyConfig, []),
    {ok, Config#{config => MyConfig#{pid => Pid}}}.

removing_handler(#{config := #{pid := Pid}}) ->
    gen_server:stop(Pid).

log(LogEvent,#{config := #{pid := Pid}} = Config) ->
    gen_server:cast(Pid, {log, LogEvent, Config}).

init(#{file := File}) ->
    {ok, Fd} = file:open(File, [append, {encoding, utf8}]),
    {ok, #{file => File, fd => Fd}}.

handle_call(_, _, State) ->
    {reply, {error, bad_request}, State}.

handle_cast({log, LogEvent, Config}, #{fd := Fd} = State) ->
    do_log(Fd, LogEvent, Config),
    {noreply, State}.

terminate(_Reason, #{fd := Fd}) ->
    _ = file:close(Fd),
    ok.

do_log(Fd, LogEvent, #{formatter := {FModule, FConfig}}) ->
    String = FModule:format(LogEvent, FConfig),
    io:put_chars(Fd, String).
Protecting the Handler from Overload

The default handlers, logger_std_h and logger_disk_log_h, feature multiple overload protection mechanisms, which make it possible for the handlers to survive, and stay responsive, during periods of high load (when huge numbers of incoming log requests must be handled).

The mechanisms are as follows:

These mechanisms are described in more detail in the following sections.

Message Queue Length

The handler process keeps track of the length of its message queue and takes some form of action when the current length exceeds a configurable threshold. The purpose is to keep the handler in, or to as quickly as possible get the handler into, a state where it can keep up with the pace of incoming log events. The memory use of the handler must never grow larger and larger, since that will eventually cause the handler to crash. These three thresholds, with associated actions, exist:

For the overload protection algorithm to work properly, it is required that:

sync_mode_qlen =< drop_mode_qlen =< flush_qlen

and that:

drop_mode_qlen > 1

To disable certain modes, do the following:

During high load scenarios, the length of the handler message queue rarely grows in a linear and predictable way. Instead, whenever the handler process is scheduled in, it can have an almost arbitrary number of messages waiting in the message queue. It is for this reason that the overload protection mechanism is focused on acting quickly, and quite drastically, such as immediately dropping or flushing messages, when a large queue length is detected.

The values of the previously listed thresholds can be specified by the user. This way, a handler can be configured to, for example, not drop or flush messages unless the message queue length of the handler process grows extremely large. Notice that large amounts of memory can be required for the node under such circumstances. Another example of user configuration is when, for performance reasons, the client processes must never be blocked by synchronous log requests. It is possible, perhaps, that dropping or flushing events is still acceptable, since it does not affect the performance of the client processes sending the log events.

A configuration example:

logger:add_handler(my_standard_h, logger_std_h,
                   #{config => #{file => "./system_info.log",
                                 sync_mode_qlen => 100,
                                 drop_mode_qlen => 1000,
                                 flush_qlen => 2000}}).
Controlling Bursts of Log Requests

Large bursts of log events - many events received by the handler under a short period of time - can potentially cause problems, such as:

Note that these examples apply to file-based logging. If you're logging to the console the protections discussed below should be safe to disable or tweak, as long as your system can handle the load of them.

For this reason, both built-in handlers offer the possibility to specify the maximum number of events to be handled within a certain time frame. With this burst control feature enabled, the handler can avoid choking the log with massive amounts of printouts. The configuration parameters are:

A configuration example:

logger:add_handler(my_disk_log_h, logger_disk_log_h,
                   #{config => #{file => "./my_disk_log",
                                 burst_limit_enable => true,
                                 burst_limit_max_count => 20,
                                 burst_limit_window_time => 500}}).
Terminating an Overloaded Handler

It is possible that a handler, even if it can successfully manage peaks of high load without crashing, can build up a large message queue, or use a large amount of memory. The overload protection mechanism includes an automatic termination and restart feature for the purpose of guaranteeing that a handler does not grow out of bounds. The feature is configured with the following parameters:

If the handler process is terminated because of overload, it prints information about it in the log. It also prints information about when a restart has taken place, and the handler is back in action.

Note

The sizes of the log events affect the memory needs of the handler. For information about how to limit the size of log events, see the logger_formatter manual page.

Logger Proxy

The Logger proxy is an Erlang process which is part of the Kernel application's supervision tree. During startup, the proxy process registers itself as the system_logger, meaning that log events produced by the emulator are sent to this process.

When a log event is issued on a process which has its group leader on a remote node, Logger automatically forwards the log event to the group leader's node. To achieve this, it first sends the log event as an Erlang message from the original client process to the proxy on the local node, and the proxy in turn forwards the event to the proxy on the remote node.

When receiving a log event, either from the emulator or from a remote node, the proxy calls the Logger API to log the event.

The proxy process is overload protected in the same way as described in section Protecting the Handler from Overload, but with the following default values:

    #{sync_mode_qlen => 500,
      drop_mode_qlen => 1000,
      flush_qlen => 5000,
      burst_limit_enable => false,
      overload_kill_enable => false}

For log events from the emulator, synchronous message passing mode is not applicable, since all messages are passed asynchronously by the emulator. Drop mode is achieved by setting the system_logger to undefined, forcing the emulator to drop events until it is set back to the proxy pid again.

The proxy uses erlang:send_nosuspend/2 when sending log events to a remote node. If the message could not be sent without suspending the sender, it is dropped. This is to avoid blocking the proxy process.

See Also

disk_log, erlang, error_logger, logger, logger_disk_log_h, logger_filters, logger_formatter, logger_std_h, sasl(6)

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4