commit 8ba7f22bec09c3e05ce2ef9c3624a5cc1f17a7c6
parent 2bdda3c080cca744ea1914ef02a14ed5894c8db4
Author: Jake Bauer <jbauer@paritybit.ca>
Date: Sun, 19 Mar 2023 15:54:43 -0400
*
Diffstat:
6 files changed, 300 insertions(+), 62 deletions(-)
diff --git a/content/blog/openbsd-on-the-dell-xps-13-9380.md b/content/blog/openbsd-on-the-dell-xps-13-9380.md
@@ -53,7 +53,7 @@ a dead battery; all for a very reasonable price and in great physical
condition.
The dead battery is probably how I was able to get such a recent, fairly
-high-end machine for only C$500 ($365 USD on 2023-05-16) before shipping and
+high-end machine for only C$500 ($365 USD on 2023-03-16) before shipping and
taxes (an additional C$90) where comparable models were at least C$150 more
expensive. I have ordered a replacement 52WHr battery (C$60), but until it
arrives I can rely on being plugged into the wall and hibernating whenever
diff --git a/content/garden/index.md b/content/garden/index.md
@@ -142,6 +142,32 @@ I have categorized my opinions to make them easier to find:
* Programming languages: [C](c), [Clojure](clojure), [Haskell](haskell), [Raku](raku), [LaTeX](latex)
* Tools: [git](git), [Vim](vim), [plan9](plan9)
+### 🖥️ System Administration
+
+Notes, configurations, and other things related to computer system administration.
+
+* General:
+ * [How I Name my Computers](computer-names)
+ * [General Tips and Tricks](general-tips-and-tricks)
+ * [Server Security](server-security)
+* OpenBSD:
+ * [IPv6 on Hetzner Cloud](openbsd-ipv6-hetzner)
+ * [MailMan3 on OpenBSD 7.1](https://xn--gckvb8fzb.com/mailman3-on-openbsd-71/)
+ * [OpenBSD HTTP and Git Server](openbsd-server)
+ * [OpenBSD NAS](openbsd-nas)
+ * [OpenBSD Router](openbsd-router)
+ * [OpenBSD Server Setup](openbsd-server-setup)
+ * [OpenBSD on the Desktop](openbsd-desktop)
+ * [Server Monitoring](server-monitoring)
+ * [Using Goaccess with OpenBSD's httpd](goaccess-with-openbsd-httpd)
+* Misc:
+ * [IPv6 Tutorial](https://metebalci.com/blog/hello-ipv6/)
+ * [JadeRune.net Admin Scripts](jaderune-admin-scripts)
+ * [MacOS Tips and Tricks](macos-tips-and-tricks)
+ * [Misskey Setup](misskey)
+ * [Tarsnap Backups With Acts](tarsnap-backups-with-acts)
+ * [UW IMAP Server Documentation](uw-imap)
+
### 🍜 Cooking
All recipes are vegan and free of tree nuts unless otherwise noted.
@@ -188,33 +214,6 @@ All recipes are vegan and free of tree nuts unless otherwise noted.
* [Sauces](sauces)
* [Oven Roasted Corn on the Cob](oven-roasted-corn-on-the-cob)
-### 🖥️ System Administration
-
-Notes, configurations, and other things related to computer system administration.
-
-* General:
- * [How I Name my Computers](computer-names)
- * [General Tips and Tricks](general-tips-and-tricks)
- * [Server Security](server-security)
-* JadeRune:
- * [JadeRune.net Admin Scripts](jaderune-admin-scripts)
-* OpenBSD:
- * [Using Goaccess with OpenBSD's httpd](goaccess-with-openbsd-httpd)
- * [OpenBSD NAS](openbsd-nas)
- * [OpenBSD Router](openbsd-router)
- * [OpenBSD Server Setup](openbsd-server-setup)
- * [OpenBSD HTTP and Git Server](openbsd-server)
- * [OpenBSD on the Desktop](openbsd-desktop)
- * [IPv6 on Hetzner Cloud](openbsd-ipv6-hetzner)
- * [MailMan3 on OpenBSD 7.1](https://xn--gckvb8fzb.com/mailman3-on-openbsd-71/)
-* Misc:
- * [Misskey Setup](misskey)
- * [Tarsnap Backups With Acts](tarsnap-backups-with-acts)
- * [UW IMAP Server Documentation](uw-imap)
- * [Issues with Linux](linux-issues)
- * [MacOS Tips and Tricks](macos-tips-and-tricks)
- * [IPv6 Tutorial](https://metebalci.com/blog/hello-ipv6/)
-
### 🍵 Tea
Notes about the various teas I've tried and about tea in general.
diff --git a/content/garden/linux-issues.md b/content/garden/linux-issues.md
@@ -1,12 +0,0 @@
-Title: Issues with Linux
-Summary: Issues with Linux
-
-# [%title]
-
-[https://lwn.net/Articles/902854/](https://lwn.net/Articles/902854/)
-
-
-> On one hand, that shows a potentially concerning concentration of power in a relatively small number of employers. On the other, this is the list of companies that are most willing to pay for maintainers to do their jobs — a good thing, given that the kernel project is short of maintainers overall.
-
-Linux kernel really is just a corporate project. That's why it's so successful generally and works well within this framework, but it's definitely not a community project and a community can't really hope to be able to maintain it
-
diff --git a/content/garden/openbsd-desktop.md b/content/garden/openbsd-desktop.md
@@ -5,15 +5,66 @@ Summary: OpenBSD on the Desktop
[← Back](./)
-Random things that I want to make note of. Usually small tutorial snippets
+Random things that I want to make note of. Usually small tutorial snippets.
+
+## Sysctl Tuning
+
+A few `sysctl.conf` tweaks to increase resource limits for a workstation
+system. There are other configurations I've seen, but I don't really know if
+they improve performance or not so I stick with the defaults unless I know that
+changing a setting will actually do something positive.
+
+```
+# Increase maximum number of open files allowed
+kern.maxfiles=262144
+# Increase the max percent of memory the buffer cache can use
+kern.bufcachepercent=90
+# Enable CTRL+ALT+DEL on console to reset the system
+machdep.kbdreset=1
+# Enable simultaneous multi-threading (hyperthreading)
+hw.smt=1
+```
+
+## Syncthing
+
+Syncthing tends to run into open file limits, especially upon first sync, so
+the above maxfiles change is needed in `/etc/sysctl.conf`, and the following is
+needed in `/etc/login.conf` (followed by running `cap_mkdb /etc/login.conf`),
+also making sure my user is in the staff group:
+
+```
+staff:\
+ ...
+ :openfiles=262144:\
+ ...
+```
+
+Then, to run syncthing as my user, in my crontab I put:
+
+```
+@reboot tmux new-session -d '/usr/local/bin/syncthing'
+```
## Multimedia
-This is a pain point with daily OpenBSD use. Audio device management is not as easy as it is with most other OSes, even if the audio system is way simpler. It can take a few commands to get things in order, and it's friendliest on laptops where you are less likely to have several peripherals plugged in.
+Ensure camera and microphone access are enabled in the OS:
+
+```
+# sysctl kern.audio.record=1
+# sysctl kern.video.record=1
+```
### Multiple audio devices
-Scan `dmesg` for `audioN` strings to figure out which audio device is mapped to which number. Then, to use, for example, audio1 when present but fall back to audio0 when not, use:
+This is a minor pain point with daily OpenBSD use. Audio device management is
+not as straightforward as it is with most other OSes, even if the audio system
+is way simpler. It can take a few commands to get things in order, and it's
+friendliest on laptops where you are less likely to have several peripherals
+plugged in.
+
+Scan `dmesg` for `audioN` strings to figure out which audio device is mapped to
+which number. Then, to use, for example, audio1 when present but fall back to
+audio0 when not, use:
```
# rcctl set sndiod flags -f rsnd/0 -F rsnd/1
@@ -22,11 +73,15 @@ Scan `dmesg` for `audioN` strings to figure out which audio device is mapped to
Audio1 can be a USB DAC or other non-default audio interface.
-(Being able to tell sndiod which audio device to use without needing to change flags with rcctl and root privileges would be very nice. As would a small utility to list out audio interfaces by name. First thing might be hard depending on how sndiod works, but second could be a simple shell script.)
+(Being able to tell sndiod which audio device to use without needing to change
+flags with rcctl and root privileges would be very nice. As would a small
+utility to list out audio interfaces by name. First thing might be hard
+depending on how sndiod works, but second could be a simple shell script.)
### Use hotplugd to reload sndiod audio devices
-When you unplug an additional audio device, sndiod won't know about it unless you tell it to reload its configuration.
+When you unplug an additional audio device, sndiod won't know about it unless
+you tell it to reload its configuration.
```
# cat > /etc/hotplug/attach
@@ -43,7 +98,8 @@ esac
### Volume Management
-`sndioctl` is the program used to change audio device volume. It can be run as a non-privileged user:
+`sndioctl` is the program used to change audio device volume. It can be run as
+a non-privileged user:
```
$ sndioctl input.mute=1
@@ -59,13 +115,25 @@ will increase output volume by 10%, for example.
### Environment Variables
-Passing the AUDIOPLAYDEVICE and AUDIORECDEVICE environment variables to a program will tell it to use the defined devices for playing or recording audio. This is useful if you have sndio set to play audio through one device, but your microphone is a different device.
+Passing the AUDIOPLAYDEVICE and AUDIORECDEVICE environment variables to
+a program will tell it to use the defined devices for playing or recording
+audio. This is useful if you have sndio set to play audio through one device,
+but your microphone is a different device.
```
$ AUDIOPLAYDEVICE=snd/1 AUDIORECDEVICE=snd/2 mumble
```
-will tell this invocation of mumble to play audio through the audio1 device but record through audio2.
+will tell this invocation of mumble to play audio through the audio1 device but
+record through audio2.
+
+### Video
+
+The user must have permission to use the `/dev/video0` device, enable with:
+
+```
+# chown $USER /dev/video0
+```
## SSH and usepam
diff --git a/content/garden/openbsd-server.md b/content/garden/openbsd-server.md
@@ -26,6 +26,11 @@ domain paritybit.ca {
}
```
+My certificate and key are called `/etc/ssl/server.crt` and
+`/etc/ssl/private/server.key` so I can avoid having to specify their locations
+in httpd.conf. If I had more than one domain I was handling on a server, I'd
+change this to reflect the explicit domain names.
+
Renewing the certificates is handled by /etc/monthly.local, which is run by
cron once a month. The output is sent to me in an email.
@@ -35,11 +40,6 @@ acme-client -v -F paritybit.ca
rcctl reload relayd httpd
```
-My certificate and key are called /etc/ssl/server.crt and
-/etc/ssl/private/server.key so I can avoid having to specify their locations in
-httpd.conf. If I had more than one domain I was handling on a server, I'd
-change this to reflect the explicit domain names.
-
## HTTP Server
The HTTP server uses OpenBSD's httpd which is very easy to configure and very
@@ -119,7 +119,7 @@ sent to the server, this is acceptable. Users are free to choose HTTPS and, in
doing so, the HSTS header will ensure a modern browser keeps using HTTPS.
Additionally, the anything in my paste directory that at least 7 days old
-will be automatically cleared with the following job in /etc/daily.local:
+will be automatically cleared with the following job in `/etc/daily.local`:
```
find /var/www/paritybit.ca/files/paste/ -type f -ctime 7 -delete
@@ -142,17 +142,16 @@ Maybe there's a better way of doing this?
### Git Server
The "git server" is really nothing more than a git daemon to handle
-cloning/fetching/pulling and stagit to generate static pages for each
-repository so code and changes can be browsed from a web browser. SSH is used
-to push changes to the server, and the git daemon is invoked using OpenBSD's
-inetd.
-
-[stagit](https://codemadness.org/git/stagit/file/README.html)
+cloning/fetching/pulling and
+[stagit](https://codemadness.org/git/stagit/file/README.html) to generate
+static pages for each repository so code and changes can be browsed from a web
+browser. SSH is used to push changes to the server, and the git daemon is
+invoked using OpenBSD's inetd.
The static pages generated by stagit are served using the configuration in
httpd.conf. Git repositories live in /var/git and updates are pushed there
-using SSH. The git daemon for cloning using the git:// protocol is invoked
-using inetd with the following configuration:
+using SSH. The git daemon for cloning using the git protocol is invoked using
+inetd with the following configuration:
```/etc/inetd.conf
git stream tcp nowait _gitdaemon /usr/local/bin/git git daemon --inetd --verbose --base-path=/var/git --export-all /var/git/
diff --git a/content/garden/server-monitoring.md b/content/garden/server-monitoring.md
@@ -0,0 +1,184 @@
+Title: Server Monitoring
+Summary: Server Monitoring
+
+# [%title]
+
+Prometheus, coupled with Grafana, is a pretty nice solution for monitoring
+a fleet of servers. Resource usage is pretty low and they're easy to configure.
+
+This guide is tailored to OpenBSD, but the steps aren't that different on other
+operating systems.
+
+## Server
+
+Prometheus and Grafana will both be running on one server. Install the
+`prometheus`, `node_exporter`, and `grafana` packages and enable all of their
+daemons.
+
+### Prometheus
+
+Configure the Prometheus daemon by editing
+`/etc/prometheus/prometheus.yml`:
+
+```
+global:
+ scrape_interval: 15s
+ evaluation_interval: 15s
+
+rule_files:
+
+scrape_configs:
+ - job_name: "prometheus"
+ static_configs:
+ - targets: ["localhost:9090"]
+ labels:
+ group: "monitoring"
+ - job_name: "node"
+ static_configs:
+ - targets: ["localhost:9100"]
+ labels:
+ name: "prometheus"
+```
+
+This is set up to display the Prometheus web interface at `localhost:9090` and
+collect data from the server that Prometheus is running on via
+a `node_exporter` on `localhost:9100`.
+
+Once this is done, start the `node_exporter` and `prometheus` daemons, check
+`/var/log/messages` for any errors, and the web interface for Prometheus should
+be available at `localhost:9090` with some collected metrics available.
+
+### Grafana
+
+Simply start the `grafana` daemon and navigate to `localhost:3000` in your web
+browser. Log in as `admin:admin` and change your password.
+
+Then, go to Configuration -> Data Sources and add a new data source. Click
+Prometheus and input `http://localhost:9090` for URL. Click save and test and
+Grafana will now be able to display collected metrics from your Prometheus
+server.
+
+Now you are free to create dashboards/panels that contain your data. The level
+of customization and options are extensive, but there are many guides on
+building dashboards plus a large library of community-made ones here:
+[Grafana.com - Dashboards](https://grafana.com/grafana/dashboards/).
+
+### Alertmanager
+
+[I have to figure out Alertmanager later]
+
+## Client
+
+On any client that you want to monitor, install `node_exporter`, enable and
+start it, and make sure it's available over port 9100. Then, in your Prometheus
+configuration, add another `static_configs` section or another target to the
+`targets` array:
+
+```
+ static_configs:
+ - targets: ["example.com:9100"]
+ labels:
+ group: "websites"
+```
+
+Once you have done this, restart the prometheus daemon to load the new
+configuration.
+
+## Login Alerts
+
+Although Prometheus has the ability to collect various metrics, it's not really
+a solution for log monitoring. I wanted something that would notify me of any
+successful login to my servers via SSH, so I wrote the following script::
+
+```
+#!/bin/sh
+
+# Watches /var/log/authlog for successful logins and sends an email
+
+email="jbauer@paritybit.ca"
+
+send_alert() {
+ date=$(echo "$1" | cut -d' ' -f1-2)
+ time=$(echo "$1" | cut -d' ' -f3)
+ host=$(echo "$1" | awk '{print $4}')
+ user=$(echo "$1" | awk '{print $9}')
+ addr=$(echo "$1" | awk '{print $11}')
+
+ subject="ALERT: Login to $user@$host from $addr on $date at $time EOM"
+
+ echo "" | mail -s "$subject" "$email"
+}
+
+count=$(grep -e "sshd.*: Accepted" /var/log/authlog | wc -l | awk '{print $1}')
+
+while true; do
+ lines="$(grep -e "sshd.*: Accepted" /var/log/authlog)"
+ newcount=$(echo "$lines" | wc -l | awk '{print $1}')
+ if [ $newcount -ne $count ]; then
+ difference=$(($newcount - $count))
+ if [ $difference -lt 0 ]; then
+ # Log rolled over
+ count=0
+ continue
+ fi
+ for i in $(seq $difference); do
+ line=$(echo "$lines" | tail -$i | head -1)
+ send_alert "$line" &
+ done
+ fi
+ count=$newcount
+ sleep 1
+done
+```
+
+Which is run like so from cron:
+
+```
+@reboot tmux new-session -d '/usr/local/bin/authalert'
+```
+
+(I should write an rc.d file for this so it can be a daemon.)
+
+This solution does use more RAM and CPU than the solution I came up with below,
+but it alerts on every login without issue and really doesn't use any
+significant amount of resources (I measured roughly a couple hundred kilobytes
+for the log data and 0.5% CPU every second).
+
+Prior to the above solution, I stumbled upon `fwa` for OpenBSD which is able to
+watch files for changes. I coupled that program with the following script:
+
+```
+#!/bin/sh
+
+# Watches /var/log/authlog for successful logins and sends an email
+
+email="mail@example.com"
+
+send_alert() {
+ date=$(echo "$1" | cut -d' ' -f1-2)
+ time=$(echo "$1" | cut -d' ' -f3)
+ host=$(echo "$1" | awk '{print $4}')
+ user=$(echo "$1" | awk '{print $9}')
+ addr=$(echo "$1" | awk '{print $11}')
+
+ subject="ALERT: Login to $user@$host from $addr on $date at $time EOM"
+
+ echo "" | mail -s "$subject" "$email"
+}
+
+/usr/local/bin/fwa /var/log/authlog | while true; do
+ lastline="$(tail -3 /var/log/authlog | grep -e "sshd.*: Accepted" | tail -1 )"
+ if [ -n "$lastline" ]; then
+ send_alert "$lastline" &
+ fi
+ read discard
+done
+```
+
+It is slightly flawed in that it will fire off an identical alert if you, for
+example, ssh into the machine in two different terminals and then exit one of
+those sessions. Note that it reads the last three lines of authlog instead of
+just the last because I found that it wouldn't alert on really quick
+operations, like when a user would `scp` a file, since the connection would
+open and close so quickly the log would already have the "Disconected" message
+instead of the "Accepted" message by the time the log was `tail`ed.
