Adding an central authentication log to Discourse

Summary

We should add an standardised authentication log system to Discourse that stores authentication events from all of the enabled authentication methods for analysis & troubleshooting.

Details

We would provide an authentication logging provider that every authentication system would hook into and send events for. As an example, the local login provider might have the following (non-complete) list of events:

someone from [IP] failed login as the user fhqwhgads which does not exist
someone from [IP] failed login as the user supermathie, was rate limited
someone from [IP] failed login as the user supermathie, but supplied an invalid password
someone from [IP] succeeded login as the user supermathie, supplied a valid password
someone from [IP] failed login as the user supermathie, supplied a valid password, but requires MFA to continue
someone from [IP] failed login as the user supermathie, supplied a valid password, supplied an invalid TOTP code
someone from [IP] succeeded login as the user supermathie, supplied a valid password, supplied a valid TOTP code for the vault-work token

The trouble with relying on existing logs (web logs) is that with the exception of the rate limited one they all look exactly the same ([200] POST /session HTTP/1.1). There’s no meaning making its way to the log.

Concerns

There is a concern about anonymous events generating log entries and balancing that with detection of events that are interesting from a SIEM analysis point of view, for example password spraying attacks. Logging EVERYTHING means anyone on the Internet can stuff your logs… but it's debatable if this is actually a problem or not. Note retention below - perhaps failures are kept for a shorter timespan inside the application log. A similar question arises around one-time-token based logs, e.g. login-via-email. Incorrect tokens are probably not "interesting" to log, but replayed tokens absolutely are, both from the context of SIEM and user support.

Questions

Retention: keep everything forever? 30 days? 90 days? scrub anonymous events sooner? API key authentication: Should calls with bad user or admin API keys be logged here? Probably - it will assist with debugging. How much should be configurable? e.g. turning debug logs on for everything or just one provider?

Implementation Considerations

The implementation should be flexible enough to allow for another event storage provider to be added to the logging pipeline. An example of such a provider might be a raw logfile, a message queue, or an S3 bucket. The exact details of such a provider are out of scope for this topic, but implementation should be flexible enough that a plugin could register to receive logs as generated.

Background

We had a customer ask for this with the intent of feeding the events to their SIEM and we realised it was a feature gap in Discourse. During internal discussion we also discovered that this information would be useful to site admins and our own support advocates since authentication logs might be in various places and one has to know where to look to seek them out. This relies on personal knowledge and a central log would greatly ease that burden. A convenient way of viewing these logs from the rails console would be beneficial for us and site admins.

Discuss this on our forum.