Manage Multi-Tenant Notifications with Notification Manager

Notification Manager manages notifications in KubeSphere. It receives alerts or notifications from different senders and then sends notifications to different users.

Supported senders include:

  • Prometheus Alertmanager
  • Custom sender (Coming soon)

Supported receivers include:

notification-manager

Quickstart

Configure Prometheus Alertmanager to send alerts to Notification Manager

Notification Manager uses the port 19093 and API path /api/v2/alerts to receive alerts sent from Prometheus Alertmanager of KubeSphere.

To receive Alertmanager alerts, KubeSphere already added the Alertmanager webhook and route configurations like below (by editing the Secret alertmanager-main in the namespace kubesphere-monitoring-system):

Send Prometheus alerts to Notification Manager:

    "receivers":
    - "name": "prometheus"
      "webhook_configs":
      - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
    "route":
      "routes":
      - "match":
          "alerttype": ""
        "receiver": "prometheus"

Send event alerts to Notification Manager:

    "receivers":
    - "name": "event"
      "webhook_configs":
      - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
        "send_resolved": false
    "route":
      "routes":
      - "match":
          "alerttype": "event"
        "receiver": "event"
        "group_interval": "30s"

Send auditing alerts to Notification Manager:

    "receivers":
    - "name": "auditing"
      "webhook_configs":
      - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
        "send_resolved": false
    "route":
      "routes":
      - "match":
          "alerttype": "auditing"
        "receiver": "auditing"
        "group_interval": "30s"

Note

The above is the default configuration. If you do not want to receive a certain type of alert, you can delete the corresponding configuration.

Configure receivers

Notification Manager now supports three types of receivers: Email, WeChat Work and Slack. Only the administrator can configure receivers.

Email

If a tenant named test-user who wants to receive email notifications, create an email receiver as follows:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  password: dGVzdA==
kind: Secret
metadata:
  labels:
    app: notification-manager
  name: test-user-email-secret
  namespace: kubesphere-monitoring-system
type: Opaque

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: EmailConfig
metadata:
  labels:
    app: notification-manager
    type: tenant
    user: test-user
  name: test-user-config
  namespace: kubesphere-monitoring-system
spec:
  authPassword:
    key: password
    name: test-user-email-secret
  authUsername: abc1
  from: [email protected]
  requireTLS: true
  smartHost:
    host: imap.xyz.com
    port: "25"

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: EmailReceiver
metadata:
  labels:
    app: notification-manager
    type: tenant
    user: test-user
  name: test-user-receiver
  namespace: kubesphere-monitoring-system
spec:
  emailConfigSelector:
    matchLabels:
      type: tenant
      user: test-user
  to:
  - [email protected]
  - [email protected]
EOF

emailConfigSelector is a selector to select EmailConfig for the email receiver. If emailConfigSelector is not set, the receiver will use the default email configuration. You can create a default email configuration as follows:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  password: dGVzdA==
kind: Secret
metadata:
  labels:
    app: notification-manager
  name: default-email-secret
  namespace: kubesphere-monitoring-system
type: Opaque

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: EmailConfig
metadata:
  labels:
    app: notification-manager
    type: default
  name: default-email-config
  namespace: kubesphere-monitoring-system
spec:
  authPassword:
    key: password
    name: default-email-secret
  authUsername: default
  from: [email protected]
  requireTLS: true
  smartHost:
    host: imap.xyz.com
    port: "25"
EOF

Email receivers with the label type: tenant only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global email receiver with the label type: global as below:

cat <<EOF | kubectl apply -f -
apiVersion: notification.kubesphere.io/v1alpha1
kind: EmailReceiver
metadata:
  labels:
    app: notification-manager
    type: global
  name: global-email-receiver
  namespace: kubesphere-monitoring-system
spec:
  to:
  - [email protected]
EOF

Note

The global email receiver will use the default email configuration.

WeChat Work

Notification Manager supports sending notifications to WeChat Work. If a tenant named test-user who wants to receive notifications from WeChat Work, create a WeChat receiver as follows:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  wechat: dGVzdA==
kind: Secret
metadata:
  labels:
    app: notification-manager
  name: test-user-wechat-secret
  namespace: kubesphere-monitoring-system
type: Opaque

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: WechatConfig
metadata:
  name: test-user-config
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: tenant
    user: test-user
spec:
  wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
  wechatApiSecret:
    key: wechat
    name: test-user-wehat-secret
  wechatApiCorpId: wwfd76b24f06513578
  wechatApiAgentId: "1000002"

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: WechatReceiver
metadata:
  name: test-user-wechat
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: tenant
    user: test-user
spec:
  wechatConfigSelector:
    matchLabels:
      type: tenant
      user: test-user
  # optional
  # One of toUser, toParty, toParty should be specified.
  toUser: user1 | user2
  toParty: party1 | party2
  toTag: tag1 | tag2
EOF

Info

  • wechatApiCorpId is the id of your WeChat Work.
  • wechatApiAgentId is the id of the app sending messages to users in your WeChat Work.
  • wechatApiSecret is the secret of this app. You can get these two parameters in App Management of your WeChat Work.
  • Any user, party or tag who wants to receive notifications must be in the allowed users list of this app.

wechatConfigSelector is a selector to select WechatConfig for the WeChat receiver. If wechatConfigSelector is not set, the WeChat receiver will use the default WeChat configuration. You can create a default WeChat configuration as follows:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  wechat: dGVzdA==
kind: Secret
metadata:
  labels:
    app: notification-manager
  name: default-wechat-secret
  namespace: kubesphere-monitoring-system
type: Opaque

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: WechatConfig
metadata:
  name: default-wechat-config
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: default
spec:
  wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
  wechatApiSecret:
    key: wechat
    name: default-wechat-secret
  wechatApiCorpId: wwfd76b24f06513578
  wechatApiAgentId: "1000002"
EOF

WeChat receivers with the label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global WeChat receiver with the label type: global as below:

cat <<EOF | kubectl apply -f -
apiVersion: notification.kubesphere.io/v1alpha1
kind: WechatReceiver
metadata:
  name: global-wechat-wechat
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: global
spec:
  # optional
  # One of toUser, toParty, toParty should be specified.
  toUser: global
  toParty: global
  toTag: global
EOF

Note

The global WeChat receiver will use the default WeChat configuration.

Slack

Notification Manager supports sending notifications to Slack channels. If a tenant named test-user who wants to receive notifications from Slack, create a Slack receiver as follows:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  token: dGVzdA==
kind: Secret
metadata:
  labels:
    app: notification-manager
  name: test-user-slack-secret
  namespace: kubesphere-monitoring-system
type: Opaque

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: SlackConfig
metadata:
  name: test-user-config
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: tenant
    user: test-user
spec:
  slackTokenSecret: 
    key: token
    name: test-user-slack-secret

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: SlackReceiver
metadata:
  name: test-user-slack
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: tenant
    user: test-user
spec:
  slackConfigSelector:
    matchLabels:
      type: tenant
      user: test-user
  channel: alert
EOF

Info

  • The Slack token is the OAuth Access Token or Bot User OAuth Access Token when you create a Slack app.
  • This app must have the scope chat:write.
  • The user who creates the app or bot user must be in the channel to which you want to send notifications.

slackConfigSelector is a selector to select SlackConfig for the Slack receiver. If slackConfigSelector is not set, the Slack receiver will use the default Slack configuration. You can create a default Slack configuration as follows:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  token: dGVzdA==
kind: Secret
metadata:
  labels:
    app: notification-manager
  name: default-slack-secret
  namespace: kubesphere-monitoring-system
type: Opaque

---
apiVersion: notification.kubesphere.io/v1alpha1
kind: SlackConfig
metadata:
  name: default-slack-config
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: default
spec:
  slackTokenSecret: 
    key: token
    name: default-slack-secret
EOF

Slack receivers with the label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global Slack receiver with the label type: global as below:

cat <<EOF | kubectl apply -f -
apiVersion: notification.kubesphere.io/v1alpha1
kind: SlackReceiver
metadata:
  name: global-slack-slack
  namespace: kubesphere-monitoring-system
  labels:
    app: notification-manager
    type: global
spec:
  channel: global
EOF

Note

The global Slack receiver will use the default Slack configuration.