Webmentions Again (And More)

Up until recently, I had webmentions displayed using webmention.js, and I highly recommend it as the simplest way to those of you who still don’t. This solution was, however, always meant to be temporary, if only to avoid being locked into webmention.io. As a first step of getting out of the lock-in, I re-did the whole webmentions display thing this summer. It all looks mostly the same (which was one of the points), but “under the hood”…

A question that immediately arises when you start thinking about writing your own webmention endpoint: “How do I store?” You start having thoughts of databases, good and bad things about them, but with that way of thinking it’s much easier if you just self-host webmention.io (it’s open-source) and avoid all the hassle. My website is built with Hugo, and I really like the fact that everything is stored as simple text files in one git repository. It’s only logical to store webmentions the same way, isn’t it?

What pushed me was the article about doing webmentions on a Hugo-based website: why not make the server check webmention.io from time to time and save the new webmentions to a JSON, I could surely build on that. The idea of storing them in a separate file in /data/ was immediately discarded: Page Bundles were invented so that we could store each post (index.md) together with the webmentions (webmentions.json) in the same subdirectory. The webmentions that target the site as a whole (such as person mentions) I can store in a webmentions.json right in /content/ (and not yet display)…

The author of the aforementioned article does the saving in JS and runs it on Netlify. I don’t do Netlify, so I need a separate tool for saving, which fortunately has already been developed. All I needed was to add an option to save to separate subdirectories, which I did. I also added more options while I was at it (not sure if anyone needs those).¹ Also, the code that does the saving to subdirectories will likely be reused later when I write a webmention endpoint.

Here comes the first “oops”: many of the site’s pages (including almost all the Reactions) are text-only, so I was too lazy to make Bundles for them. In Hugo, the contents of a page addressed as https://evgenykuznetsov.org/posts/2021/old-comments/ can reside either in /content/posts/2021/old-comments.md, or in /content/posts/2021/old-comments/index.md; in the latter case the same subdirectory can be used to store the images and whatever is needed on the page, and this is exactly what’s called a “Page Bundle”. Since I was going to store webmentions in Bundles, the first way of storing the pages doesn’t really work, and I have already accumulated enough pages stored in this manner. I needed to re-save all of them in another way, each in its own subdirectory, and doing it manually would take all summer, especially considering different language variants. Bash to the rescue:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/bash
set -e

wd=$(pwd)

if [[ "$1" != "" ]]; then
        wd="$1"
fi

for x in "$wd"/*.md; do
        case "$x" in 
                *.ru.md)    suffix="ru.md";;
                *.en.md)    suffix="en.md";;
                *.md)       suffix="md";;
                *)          continue;;
        esac
        name="${x%%.*}"
        if [ ! -d "$name" ]; then
                mkdir "$name"
        fi
        mv "$x" "$name/index.$suffix"
done

…and long live Stack Overflow!

From there it was easier. I removed webmention.js from the templates, wrote the processing of the bundled JSON (which in many ways was the same webmention.js translated from JavaScript into Hugo templating language and customized to my needs), fixed CSS (I now know much more about CSS/SaSS than I did in June), and wrote a template for comments RSS-feeds. I also marked avatars and names for dynamic loading and uncovered a couple of bugs in indieweb-glue along the way: some things were not cached when they should have, especially the 404s that also lacked CORS headers, I fixed all that. Everything works! Admittedly, webmentions no longer appear immediately (only after the next rebuild of the site), but now I can pre-moderate them if I ever feel like it.

Yay, now I could also display older webmentions, the ones I received back when I was running Known; I did save them when I moved from Known to Hugo, and I used the same JSON structure as webmention.io! I only wonder what happened to all the avatars… Oh, it turns out that Known stored the avatars for the received webmentions, and all the links are to evgenykuznetsov.org/service/web/imageproxy/ that no longer exists. Those links need to be removed. Not by hand, of course, we have jq and the magic of find, thank Discordia for Stack Overflow!

find ./ -type f -name "webmentions.json" \
    -printf \
        "jq -c 'del \
            (..|.photo?| \
            select(contains \
            (\"evgenykuznetsov.org/service/web/imageproxy\")?) \
            )' %p | \
        sponge %p\n" \
    | \
sh

Hmm, what was the format I saved the comments from Google+, LiveJournal and diary.ru in? The same webmention.io-inspired JSON, how cute! Let’s display those, too… Oops! Likes in G+ had no date so the RSS template breaks: Hugo happily parses a date from nil and gets a value of error.Error type that can’t be further processed. Another check there, and here we go: right here on my website, the comments that are over 15 years old! So fascinating to read, especially the comments from back when the blog was much more personal… Some nicknames I can no longer remember, some have departed from this world, some I still talk to regularly, and some provoked an urge to immediately contact. A blog that is almost a half of the life old, and with comments, beats any personal journal.

Writing my own webmention endpoint is next on the roadmap.