Category Archives: English

Interactive git rebase with non-interactive editing

When working with git and especially GitHub, I often have commits on my local branch that were already submitted as a pull request. Sometimes I continue working and later notice that I have commits on the branch that have nothing to do with the next thing I am already working on. Therefore I want to remove them from the current branch.

$ git log --oneline @{upstream}..
e159d1e Commit C
70140e3 Commit B
16ed14a Commit A

Let’s say I want to get rid of Commit B. The normal way would be to use git rebase -i and then tell git to remove the commit by deleting the corresponding line or replacing pick with drop, then save the file, exit the editor and let git do the rest. However, if I already know exactly what I want to do, this seems cumbersome. Therefore I came up with the following alias:

$ git config --global alias.drop '!f() { for c in "$@"; do sha1=$(git rev-parse --short $1); [ -n "$sha1" ] && git -c rebase.abbreviateCommands=false -c sequence.editor="printf \"%s\n\" \"g/^pick $sha1 /s//drop $sha1 /\" w \"g/^#/d\" \"g/^$/d\" \"%p\" q | ed -s" rebase -i --autostash; done }; f'

I admit that maybe at some point I should have changed my plan and I should have made this into a shell script instead… But this works and requires no extra files than just ~/.gitconfig.

What does it do? This alias is not a normal alias, but as indicated by the ! at the start it is supposed to be executed in a shell. As git will append all additional arguments from the command line to the end of this shell command, I turned it into a shell function to be able to handle the arguments one after another in a loop. First, the given argument is validated as a reference to a commit and the matching sha1 is retrieved. This means it can also understand HEAD~3 and similar references. If it is not valid, git will already have printed an error message. If it is valid, this runs git rebase with some special options.

The configuration option rebase.abbreviateCommands ensures that git will show us the long command name pick and not just p in the editor. But wait, there is also sequence.editor, which will be used instead of your normal $EDITOR. This is a custom command that will get a temporary filename as an argument. As the file needs to be edited in place, I chose ed for this task. Other options like sed or awk might work as well, but as far as I know there is no standardized interface for in-place editing and I want this to work with all flavors of Linux or *BSD.

ed takes a filename as argument and then reads line-wise commands from standard input that are applied to the file. I wrapped this with a printf to avoid using lots of \n in one long string or using many echo commands instead. The first command is g/re/... which will search for a line matching the given regex and execute the commands following the second slash. As a side note, the name of the tool grep comes exactly from this ed command: g/re/p, where the p stands for print. The command executed here is s/re/replacement/ which will replace the first occurrence of the regex with the given replacement. The empty string denotes the previous match. This combination of two commands is necessary to mask errors from ed. If a s// does not replace anything, ed would return an error and exit with a non-zero status code. This behavior is masked by combining it with g//, which will only try to replace on lines that already matched.

The command w will write the file to disk. Then I also wanted to get some output of the results to quickly see what’s going on. The next two g// commands will delete comments and empty lines before %p will print the whole file to standard output. Finally, the last command q tells ed to quit.

git will then take the modified file just as if it was edited in your $EDITOR and continue with the rebase. The --autosquash option is useful to automatically stash any uncommitted changes before running the rebase and then restoring them afterwards.

Now let’s finally get to removing Commit B from the example above:

$ git drop 70140e3
pick 16ed14a Commit A
drop 70140e3 Commit B
pick e159d1e Commit C
Successfully rebased and updated refs/heads/master.

Here it goes! This was an interactive rebase, but with non-interactive editing of the sequence file using ed. Setting up a similar alias for git reword to quickly edit the commit message is left as an exercise for the reader.

Postfix with relayhost over stunnel on macOS 10.12 Sierra

I like to have a working mail setup on all machines as this allows to be notified about cronjobs that failed and also to be able to send other notifications that would otherwise be lost. It is also especially useful for things like git send-email or automatically sending GPG signatures with caff to others.

However, mails cannot just be sent from any device and mail servers on the internet usually reject mails from dial-up IPs or public WiFi networks. To fight spam, techniques like SPF have been developed that restrict the mail servers that are allowed to send mails for the domain name used in the From: field. Therefore the best way is to relay all outgoing mail through the mail server that is responsible for your domains.

While most tools also allow you to configure an external SMTP server, it is on one hand tedious to configure it everywhere and on the other hand also insecure if you have to write the username and password for authentication to many user-readable configuration files on your system. Therefore I am running a local MTA on all the computers I administrate to relay mails to a central mail server.

macOS already includes Postfix in the base installation. Luckily, this is also the mail daemon I am most experienced with, although other options such as msmtp or nullmailer exist. However, due to the System Integrity Protection (SIP) on recent macOS versions, there is no way to disable the builtin Postfix. Therefore the best option is to embrace that as a feature and use it.

On macOS, services can automatically be started by triggers configured in the launchd init system. This is also the case for Postfix, which is way there is no process running unless a mail needs to be send. As soon a mail is written to the mail spool directory by sendmail, the Postfix master will be started to pickup and process the mail as usual. This is already configured out of the box on macOS and no special configuration is needed.

Generic relayhost configuration

The following describes the necessary configuration to relay mail over another host with authentication. Note this will also work on other non-macOS machines that have a regular Postfix daemon that runs all the time.

The following additions to /etc/postfix/main.cf are required. First, tell Postfix with which name it should identify itself to other mail servers and which domain name should be appended when sending mail from local users. If your local username is foo, the mail will appear to be from foo@example.org. The relay host is the machine that will accept mails after authentication and send them to the and For the authentication to the SMTP server, SASL has to be enabled with an additional configuration file which is the password storage. The last option ensures that Postfix will only use TLS encrypted connections. Note that this setup assumes you are using port 587 with STARTTLS. Refer to the Postfix manual if you need to use SMTPS over port 465.

# /etc/postfix/main.cf

# Relay mails over mail.example.org
myhostname = foo.example.org
myorigin = example.org
relayhost = [mail.example.org]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpasswd
smtp_tls_security_level = encrypt

The password storage should look like the following example. The password is stored as plaintext, so make sure you set the permissions of the file in a way that only root can access it. It might be a good idea to create a separate user per host on your relay host for this, in order to be able to change or revoke passwords without having to reconfigure all machines.

# /etc/postfix/saslpasswd
[mail.example.org]:587 user:password

Every time you make changes to this file, you have to remember to execute the postmap command to update the .db file right next to it.

$ sudo postmap /etc/postfix/saslpasswd

And that’s it already. Now you are able to send mails via the local sendmail interface. Use the following command with your own email address to test the setup.

$ echo "Foo Bar Baz" | mail -s Test foo@example.org

Broken Postfix on macOS 10.12.6 Sierra

This setup worked fine for years until macOS 10.12.6 Sierra or the following security update arrived, with which Apple broke the TLS client of Postfix. Apparently this was fixed on macOS 10.13 High Sierra, so the following section will not apply to you if you have already upgraded.

As there are no traditional log files on macOS anymore, it took me quite a while to figure out how to debug it as the various Postfix processes all have different names. Use the following command to see everything related to Postfix:

$ log stream --predicate  '(process == "master" OR process == "qmgr" OR process == "pickup" OR process == "cleanup" OR process == "smtp")' --level debug
Filtering the log data using "process == "master" OR process == "qmgr" OR process == "pickup" OR process == "cleanup" OR process == "smtp""
Timestamp                       Thread     Type        Activity             PID
...
2018-03-21 23:41:28.629231+0100 0x158e938  Default     0x0                  82375  smtp: warning: Digest algorithm "md5" not found
2018-03-21 23:41:28.629310+0100 0x158e938  Default     0x0                  82375  smtp: warning: disabling TLS support
...

It looks like that EVP_get_digestbyname(const char *name) from libcrypto will always return NULL and cannot find any digests. I guess Apple patched something and it is now missing a call to OpenSSL_add_all_algorithms(); or similar. One of those classic mistakes when writing your first program against the OpenSSL API.

Using stunnel as an on demand TLS wrapper for Postfix

As the TLS client in Postfix is not usable, I had to find a new solution as I really do not want to send mails over an unencrypted connection. Therefore I chose stunnel as a TLS wrapper acting as a client and relaying connections from a local listening port to a remote host.

$ sudo port install stunnel certsync  # or curl-ca-bundle

Once again, launchd can be used with a new LaunchDaemon in inetd mode to only start the process when it is needed. Choose a free port to run this on, in this example I am using port 555. I recommend to use a port <1024 to ensure it cannot be bound by any other user except root on the same machine. This is using the stunnel3 launch script, as stunnel 5.x only works with configuration files, but I really do not want to keep a file lying around.

<!-- /Library/LaunchDaemons/org.example.mail.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>org.example.mail</string>
	<key>ProgramArguments</key>
	<array>
		<string>/opt/local/bin/stunnel3</string>
		<string>-c</string>
		<string>-r</string>
		<string>mail.example.org:587</string>
		<string>-n</string>
		<string>smtp</string>
                <string>-v</string>
                <string>2</string>
		<string>-A</string>
		<string>/opt/local/etc/openssl/cert.pem</string>
	</array>
        <key>UserName</key>
        <string>nobody</string>
	<key>Sockets</key>
	<dict>
		<key>Listeners</key>
		<dict>
                        <key>SockNodeName</key>
                        <string>localhost</string>
			<key>SockServiceName</key>
			<string>555</string>
			<key>SockType</key>
			<string>stream</string>
		</dict>
	</dict>
	<key>inetdCompatibility</key>
	<dict>
		<key>Wait</key>
		<false/>
	</dict>
</dict>
</plist>

After creating the file, load the new LaunchDaemon and mark it to be loaded automatically each time the system boots. In case you need it, you can permanently disable it again with unload -w. Afterwards test the connection with netcat and see how stunnel connects to the remote host and handles the STARTTLS sequence to transparently encrypt this connection with TLS.

$ sudo launchctl load -w /Library/LaunchDaemons/org.example.mail.plist
$ nc localhost 555
220 mail.example.org ESMTP Postfix

Now all that is left is to edit the Postfix configuration in /etc/postfix/main.cf to connect to localhost:555 instead of the original mail host. As TLS is broken, the security level needs to be reduced to allow unencrypted connections. You will also have to adapt the SASL password storage accordingly and remember to run postmap afterwards.

# /etc/postfix/main.cf

...
relayhost = [localhost]:555
smtp_tls_security_level = may
...

Now test it again by sending yourself an email with the mail command. Enjoy your local mail server setup!

Goodbye Sailfish OS and Jolla

Already in September 2017 I migrated away from Sailfish OS and I am now now using an Android phone. For me, this is the first time I left the niche market of alternative phone operating systems to go mainstream. Originally I used a Nokia dumb phone and then in 2009 got the Nokia N900 as my first smartphone with Maemo as OS, which was so much joy, easily extensible due to the open platform, and had a sliding keyboard. After a few years I continued with the Nokia N9 with the now abandoned Meego. The N9 was released shortly before the Nokia phone department was switched to produce Windows phones for Microsoft. However, a group of former Nokia developers formed Jolla, a company producing an alternative phone operating system called SailfishOS.

Continue reading

Upgrading a VM from macOS 10.12 Sierra to macOS 10.13 High Sierra in VirtualBox

For testing purposes, I have a VM in VirtualBox currently runnning macOS 10.12 Sierra. Now that macOS 10.13 High Sierra is in Beta, I wanted to upgrade my VM to this new release. However, this proved to be difficult with the usual ways. This blog post will describe how to upgrade a Sierra VM to High Sierra.

Continue reading

How to run rsync on remote host with sudo

Sometimes I want to transfer files including ownership. This is not possible as normal user as the chown(2) system call requires special privileges, that is: uid == 0. However, I do not want to open ssh access for root, but go with the usual way to elevate my privileges: sudo.

I will go through common solutions presented on the web and explain why these do not work at all without significant modifications on the remote host and then present a working solution using X11-Forwarding that is less invasive.

Continue reading