| 1 | # Implementing a new protocol |
| 2 | |
| 3 | This guide explains how to create a new protocol backend to support a new gateway/bridge in matterbridge. |
| 4 | |
| 5 | ## Step-by step list |
| 6 | |
| 7 | - [ ] Create a new catalog in [`/bridge` folder](https://github.com/42wim/matterbridge/tree/master/bridge) and a main file named after the bridge you are creating, such as `whatsapp.go` |
| 8 | - [ ] Implement a [`Bridger` interface](https://github.com/42wim/matterbridge/blob/2cfd880cdb0df29771bf8f31df8d990ab897889d/bridge/bridge.go#L11-L16) |
| 9 | - [ ] Mention your bridge exists in [`/gateway/bridgemap/bridgemap.go`](https://github.com/42wim/matterbridge/blob/master/gateway/bridgemap/bridgemap.go) |
| 10 | - [ ] Divide functionality in several files, as it is done for [slack](https://github.com/42wim/matterbridge/tree/master/bridge) |
| 11 | - `yourbridge.go` with main struct and implementation of the `Bridger` interface |
| 12 | - `handlers.go` with handling messages incoming to Bridge |
| 13 | - `helpers.go` for all the misc functions and helpers |
| 14 | - [ ] Minimal set of features is sending and receiving text messages working. |
| 15 | - [ ] Documentation |
| 16 | - [ ] Add a [sample configuration](https://github.com/42wim/matterbridge/commit/6372d599b1ca2497aa49142d10496f345041b678#diff-0fcc5f77f08a4f4106d2da34c4dcd133) of your bridge to `matterbridge.toml.sample` and explain all the custom options |
| 17 | - [ ] Add your bridge to README |
| 18 | - [ ] Document all exported functions |
| 19 | - [ ] Run `golint` and `goimports` and clean the code |
| 20 | - [ ] Send a PR |
| 21 | |
| 22 | ## Features |
| 23 | |
| 24 | Below is a feature list that you might copy to your issue. |
| 25 | |
| 26 | Features: |
| 27 | - [ ] Connect to external service |
| 28 | - [ ] Get all active chats |
| 29 | - [ ] Check if chosen channels exist externally |
| 30 | - [ ] Connect to chosen channel |
| 31 | - [ ] Show nicknames in external service |
| 32 | - [ ] Show nicknames in relayed messages |
| 33 | - [ ] Test if multiple channels are working |
| 34 | - [ ] Show profile pictures from your bridge in relayed messages |
| 35 | - [ ] Show profile picture in your bridge |
| 36 | - [ ] Handle reply/thread messages |
| 37 | - [ ] Handle deletes |
| 38 | - [ ] Handle edits |
| 39 | - [ ] Handle notifications |
| 40 | - [ ] Create a channel if it doesn't exist |
| 41 | - [ ] Sync channel metadata (name, topic, etc.) |
| 42 | - [ ] Document settings in `matterbridge.toml.sample` |
| 43 | - [ ] Document bridge in README |
| 44 | - [ ] Explain setting up the bridge process for users in the wiki |
| 45 | - [ ] Add screenshots from your bridge in the wiki |
| 46 | - [ ] Document code |
| 47 | |
| 48 | Handle messages |
| 49 | - [ ] text from the bridge |
| 50 | - [ ] text to the bridge |
| 51 | - [ ] image |
| 52 | - [ ] audio |
| 53 | - [ ] video |
| 54 | - [ ] contacts? |
| 55 | - [ ] any other? |
| 56 | |
| 57 | |
| 58 | ## FAQ |
| 59 | |
| 60 | **How can I set the default RemoteNickFormat for a protocol so users don't have to do it in a config file?** |
| 61 | |
| 62 | @42wim? |
| 63 | |
| 64 | **Why on Slack I see bot name instead of remote username?** |
| 65 | |
| 66 | Check if you: |
| 67 | - [ ] did set `Message.Username` on the message being relayed |
| 68 | - [ ] did set `RemoteNickFormat` in config file |
| 69 | |
| 70 | **Sending message to the bridge don't work** |
| 71 | |
| 72 | - [ ] Channels must match. While sending the message to the bridge make sure that you set the `config.Message.Channel` field to channel as it is mentioned in the config file. |
| 73 | |
| 74 | ### Handling HTTP requests |
| 75 | |
| 76 | > [!TIP] |
| 77 | > If your protocol doesn't do HTTP requests at all, you do not have to read this section. |
| 78 | |
| 79 | Every matterbridge bridge instance as defined in the config has its own dedicated HTTP client initiated when the program starts. It is used by |
| 80 | HTTP helpers (explained below) but may also be used directly as `b.HttpClient`. |
| 81 | |
| 82 | #### Custom HTTP client |
| 83 | |
| 84 | The HTTP client is initiated in the `NewHttpClient` method defined in [bridge/bridge.go](../../bridge/bridge.go), and can be overridden in your bridge class. |
| 85 | |
| 86 | For example, if your protocol `foo` requires custom settings, such as going through tor, you would do something like: |
| 87 | |
| 88 | ```go |
| 89 | func (b *Bfoo) NewHttpClient(http_proxy string) (*http.Client, error) { |
| 90 | // Create a new custom client here |
| 91 | } |
| 92 | ``` |
| 93 | |
| 94 | > [!WARNING] |
| 95 | > Unless your customization requires to override the `http_proxy` passed as first argument to the constructor, don't forget to respect the defined proxy setting. |
| 96 | |
| 97 | #### Custom HTTP requests |
| 98 | |
| 99 | Every HTTP request emitted by your bridge is initiated in the `NewHttpRequest` method defined in [bridge/bridge.go](../../bridge/bridge.go), and can be overridden in your bridge class: |
| 100 | |
| 101 | ```go |
| 102 | func (b *Bfoo) NewHttpRequest(method, uri string, body io.Reader) (*http.Request, error) { |
| 103 | // Create a new custom request here |
| 104 | } |
| 105 | ``` |
| 106 | |
| 107 | This is useful for protocols which require setting custom HTTP headers, such as cookies or `Authorization` headers. |
| 108 | |
| 109 | > [!INFO] |
| 110 | > This constructor is used by matterbridge's internal HTTP helpers, so by setting your custom headers in your bridge's |
| 111 | > `NewHttpRequest` method, they will be respected when using the helpers. |
| 112 | |
| 113 | #### Downloading remote files |
| 114 | |
| 115 | If your bridge needs to download files over HTTP, you can use matterbridge's internal helpers. |
| 116 | In the most common cases, you can use the two helpers `AddAvatarFromURL` (for user avatars) and `AddAttachmentFromURL`. |
| 117 | |
| 118 | > [!WARNING] |
| 119 | > In all cases, it's very important to perform such HTTP operations in the background so you don't block |
| 120 | > matterbridge on a response that may succeed or timeout. |
| 121 | > |
| 122 | > ```go |
| 123 | > if hasAttachments(m) { |
| 124 | > go func() { |
| 125 | > err := handleAttachments(rmsg, m) |
| 126 | > if err != nil { |
| 127 | > b.Log.WithError(err).Errorf("Downloading attachment failed") |
| 128 | > return |
| 129 | > } |
| 130 | > // Spreading the message (with the attachment) to other bridges takes |
| 131 | > // place in the background goroutine |
| 132 | > b.Remote <- rmsg |
| 133 | > }() |
| 134 | > // That entire message is being handled in the background, skip to the next message |
| 135 | > continue |
| 136 | > } |
| 137 | > ``` |
| 138 | |
| 139 | TODO: what happens when the filename is not set? can we guess it from the URL/content-type? should we error if |
| 140 | it's not explicit and cannot be inferred? |
| 141 | TODO: how is the ID used? is this param used at all or can we safely remove it? |
| 142 | TODO: should we take an optional hash to avoid useless requests for files we already have? since hash is already |
| 143 | calculated later in the helpers |
| 144 | |
| 145 | If you need to somehow inspect or treat the raw data bytes from a successful HTTP GET request before inserting it |
| 146 | as an attachment to the received message, you may use the `HttpGetBytes` method, which wll only succeed if the |
| 147 | returned HTTP status code is 200. |
| 148 | |
| 149 | If you need more custom logic, such as a custom HTTP verb or headers specific to this request, you may use |
| 150 | the `HttpClient` field directly, along with the `NewHttpRequest` method: |
| 151 | |
| 152 | ```go |
| 153 | // If you willingly want to avoid `http_proxy` settings and/or your bridge's request constructor, |
| 154 | // use http.NewRequest here. |
| 155 | req, err := b.NewHttpRequest("GET", uri, "") |
| 156 | if err != nil { |
| 157 | continue |
| 158 | } |
| 159 | |
| 160 | // Customise the http request |
| 161 | ... |
| 162 | |
| 163 | // Send the request |
| 164 | resp, err := b.HttpClient.Do(req) |
| 165 | ... |
| 166 | ``` |
| 167 | |
| 168 | If your protocol can know in advance the size of the remote attachment, you can compare it with the maximum |
| 169 | download size to avoid too big requests altogether: |
| 170 | |
| 171 | ```go |
| 172 | for _, attach := range m.Attachments { |
| 173 | if int64(attach.Size) > b.General.MediaDownloadSize { |
| 174 | // Ignore this specific attachment (file too big) |
| 175 | b.Log.Warnf("Attachment too big to download: %s has size %#v (MediaDownloadSize is %#v)", name, size, b.General.MediaDownloadSize) |
| 176 | continue |
| 177 | } |
| 178 | ... |
| 179 | } |
| 180 | ``` |
| 181 | |
| 182 | #### Uploading files to a remote server |
| 183 | |
| 184 | If you need to upload files to a web server, you can use the `HttpUpload` helper method. It's similar to the `HttpGetBytes` method, but takes |
| 185 | two additional arguments: |
| 186 | |
| 187 | - `headers` (`map[string][string]`), because you may need to set a specific `Content-Type` or `Authorization` header to perform the upload |
| 188 | - `ok_status` (`[]int`), because the remote server may have different success codes, eg. `200`/`201`, or even `302` for duplicate files |
| 189 | |
| 190 | > [!WARNING] |
| 191 | > Just like with HTTP downloads, it's **very important** to perform upload operations in the background. |
| 192 | |
| 193 | ### Handling file attachments |
| 194 | |
| 195 | Most protocols support sending files, such as images and other documents. How they are displayed, and how they are transferred changes |
| 196 | in every case, but there's two main approaches: |
| 197 | |
| 198 | - in-band attachments (mumble): raw content bytes are sent within a protocol message |
| 199 | - out-of-band attachments (XMPP/matrix/etc): content is uploaded to a different server, and a URL is passed along in the protocol |
| 200 | |
| 201 | For out-of-band attachments, see the HTTP upload/download section above. |
| 202 | |
| 203 | #### In-band attachments |
| 204 | |
| 205 | To receive raw bytes from in-band attachments, you can use the `AddAttachmentFromBytes` and `AddAvatarFromBytes` helper methods. They |
| 206 | both expect that you provide a filename in advance. |
| 207 | |
| 208 | > [!NOTE] |
| 209 | > All protocols currently support importing data bytes into matterbridge, but not all of them support sending raw |
| 210 | > bytes to their own network. See issue [#50](https://github.com/matterbridge-org/matterbridge/issues/50) for a comparison table |
| 211 | > and broader discussion about these limitations. |
| 212 | |
| 213 | TODO: what happens when no filename is set? Do we try to guess the mimetype to figure out an extension, and use the SHA hash as a basename? |
| 214 | |
| 215 | To send in-band attachments, you can use the `FileInfo.Data` field which contains the raw attachment bytes. |
| 216 | |
| 217 | #### Handling attachment errors |
| 218 | |
| 219 | In the upstream past, matterbridge produced a message with `msg.Event = config.EventFileFailureSize`. However, this was not really documented, |
| 220 | especially how to handle mixed successful/errored attachments. It apparently was just discarded in `gateway/handlers.go` in the `handleMessage` |
| 221 | function and not handled gracefully by specific bridges. |
| 222 | |
| 223 | At the moment, it is recommended to simply log errors from attachments, and proceed with further attachments ignoring failed ones. |
| 224 | |
