I decided to run the server as an XCP service, because it’s a nice easy way to make little sub-processes in macOS. All of that actually worked quite nicely, except for one fundamental part of the application: Apple Events, which have to be called on the main thread. Unfortunately in an XPC service the main thread is blocked by the service listener, meaning the Apple Events can’t run.
The solution is going to be moving logic (and Apple Events) back into the main app, still running the socket server XPC service, but forwarding messages to the server which remote clients can receive. Luckily most of the code will still work, I just need to change how it all fits together.