Mike Macgirvin is a personal friend, and a force to be reckoned with. Living on a farm in the hills of Australia, he has quietly spent his days carving out an elaborate system from scratch, doing the bulk of the work himself.
Over the past decade in particular, he has pushed the boundaries of a next-generation communication platform, restlessly experimenting and taking notes, often inventing his own solutions to complex problems.
This interview is the first part of our interview series, Faces of the Federation.
Thanks for joining me, Mike! Please describe yourself and what you do.
I’m a software professional currently living in Australia. I’ve created a number of open source projects (most notably Friendica and Hubzilla) and been involved in the related fields of electronic communications and software development for around 40 years.
Could you also fill us in a little bit on your technical background?
I spent my younger professional career fixing machines and contraptions for the semiconductor industry. We had machines fabricating things at atomic scales on semi-automated or fully-automated production lines. I spent a lot of time fixing robotic wafer handlers and high-vacuum equipment and writing custom software to automate how all these machines interacted together to reliably reproduce nano-scale devices. This got me deeply involved in communication theory and what infrastructure it took to make devices (any devices) interact.
You have been involved in the federated web space for a considerably long time. What initially got you involved with developing federated communication apps?
In the early 80s, I created and federated several BBS apps (dialup bulletin board services) with some of the emerging network protocols. We provided (over analog phone lines) a lot of the services that online social spaces provide today (messages, forums, mail, games, media downloads, etc.) , and federated messages with FidoNet, Bitnet and ARPAnet. I also wrote some email clients and forum software for NASA to link some of the contractors involved in a space physics satellite launch.
Eventually I ended up running Unix servers at Stanford and became an in-house expert in email technologies — just prior to the explosion of the World-Wide-Web. I developed IMAP servers and clients and developed a couple of open source email applications which integrated multiple protocols, filtered views, and end-to-end encyption. This ultimately led to several years as a lead developer building messaging systems for a little internet startup called Netscape. Then I went on to get sucked into Sun/iPlanet and AOL and managed dev teams building groupware and large scale communications servers.
What ultimately led to you writing DFRN, your first protocol?
That was not the first protocol I wrote. I was on several messaging standards ad hoc committees and was directly involved in standards and spec review for LDAP mail delivery, IMAPS (SSL IMAP), IMAP4REV1, and authored a half dozen drafts for things like email alias management services and special purpose IMAP extensions.
There’s nothing magic about a protocol. It’s basically just a gentleman’s agreement about how to implement something. There are a number of levels or grades of protocols from simple in-house conventions all the way to internet specifications. The higher quality protocols have some interesting characteristics. Most importantly, these are intended as actual technical blueprints so that if two independent developers in isolated labs follow the specifications accurately, their implementations should interact together perfectly. This is an important concept.
The level of specification needed to produce this higher quality protocol is a double-edged sword. If you specify things too rigidly, projects using this protocol cannot grow or extend beyond the limits and restrictions you have specified. If you do not specify the implementation rules tightly enough, you will end up with competing products or projects that can both claim to implement the specification, yet are unable to interoperate at a basic level.
“I discovered that in 2010 there was no decentralized solution on the ‘free web’ to replace Facebook — in even the most basic sense. “
In early 2010, I left Facebook because I was increasingly concerned about the directions they were taking related to personal privacy. At that time I was looking for decentralized alternatives, as my feeling was that centralization and increased reliance on single providers was a dangerous trend. I discovered that in 2010 there was no decentralized solution on the “free web” to replace Facebook — in even the most basic sense.
There was Status.Net (a Twitter workalike), and some kids in New York were building a “Facebook killer” (Diaspora) but were doing it in private and had no prior experience with communications technologies. There was also a project called Appleseed and another called 6d; but both were barely useable and neither offered anything remotely resembling the interactivity of the Facebook experience.
“There’s nothing magic about a protocol. It’s basically just a gentleman’s agreement about how to implement something.”
I looked at the so-called “Open Stack” that Status.Net used, but it offered absolutely nothing in the way of privacy. There weren’t any existing internet protocols that specifically dealt with this kind of a use-case; sharing “web resources” privately over a decentralized architecture. So I created DFRN to fill in some of these missing pieces:
- There was no established protocol for “friending”.
- Existing “social” communications protocols offered little or no support for privacy.
- There was no mechanism for Alice to “write on Bob’s wall” if Bob was using another server or service.
- The only realistic solution for private media was to ship a copy to everybody. This doesn’t scale if you’re sharing a gigabyte video with 1000 friends. It also presents even more privacy issues because now you have 1000 copies of your private video floating around.
DFRN (Distributed Friends and Relations Network) provided some simple but workable solutions in all of these areas. It is still being used in Friendica, but the technology is pretty out-dated now and some parts are clumsy. I believe I did a fair job on the specification and an independent developer should be able to federate with it based entirely on reading the spec.
There are two primary components. The first is an authenticated “pipe” that we create between Alice and Bob. Alice can prove that Bob is at one end and Bob can prove that Alice is at the other end. Then you can put most anything you want in that pipe and send it between Bob and Alice. We initially sent XML ActivityStreams, because in its day it was a pretty good model for a wide array of social content.
The other thing I did was spend a long time on the “wall-to-wall” problem and media access, and realized they were essentially the same problem. My server needs to know who is looking at it, even if the viewer doesn’t have an account on my server. This resulted in development of something I call “magic-auth”. You visit Bob’s web page and provide a claim that says “I’m Alice” and Bob’s server goes out and contacts Alice’s server and does a little dance to see if this is in fact Alice. Then it can let Alice write on Bob’s wall or see his private videos. The crypto behind it was based on essentially the same mechanism that “password-less SSH” uses; but modified for the web.
Within a month of the first release, I had implemented federation with StatusNet (“OStatus”, although we couldn’t send private messages to that network), and also we added RSS feeds and email in our social stream. Later we added Twitter and Facebook and Google and had a pretty awesome social experience.
It sucked memory like a pig, but dang, if you were on the internet — I could see you in my stream and in most cases could comment on your posts and you’d see it on your own service.
At the time, OStatus was a decent specification, despite its complete lack of privacy and spam potential. Technically I only had issues with implementing salmon magic signatures, because that specification changed and some projects were using the older spec while I implemented the newer one. Once we all got on the same page, federation between us and Status.Net was pretty good.
But that was then. GNU-Social and Mastodon’s current versions of OStatus are almost unrecognizable and have diverged a long way both from the original written spec and from each other. If you implement the OStatus spec faithfully today based purely on the written specification, you will likely be unable to communicate with either project without a lot of additional effort.
Sometime around 2012 / 2013, the Friendica project implemented support for the Diaspora federation protocol, allowing for cross-platform communication in the federated space. What sort of challenges did you find yourself up against?
A very basic high level description of the Diaspora protocol appeared in the Diaspora wiki in 2011. Initially, I had problems implementing even basic communications. The protocol wasn’t very well defined, but more importantly, the specification was incorrect, and therefore barely useful. The published description was completely different than what I saw “on the wire”.
I looked at the source code and examined packets for several months before I finally was able to put together what they were *actually* sending. Once I figured out how the protocol really worked, I communicated with Ilya (who was more or less in charge of protocol stuff for Diaspora). I explained what I found and pointed out a number of mistakes that were made. He was very cooperative and within a week or two he fixed most of the protocol issues which had surfaced.
I found several more protocol issues (some of catastrophic severity), but most of my time was spent smoothing over “policy” issues. These things aren’t even discussed in the protocol specifications, but if you don’t adopt the same set of policies you can’t interact. For example what characters are allowed in a username, and do you link hashtags on the sending side or the receiving side? These seem like trivial matters but they can really stuff things up for a service which does them differently.
Unfortunately, a few months after the initial federation module was completed Ilya’s life was cut short; and from that point forward I’ve found the Diaspora developers in general to be very difficult to work with. There is a more recent version of the protocol, released just this last year. This version has a useful specification. There are some bits lacking, but I was able to read the spec and come up with an independent implementation which interoperates.
My complaints with the current Diaspora protocol are simply that the protocol is too restrictive to express the kinds of data and relationships that my projects need to express, and the media privacy support is somewhat weak.
What is Hubzilla, and how is it different from Friendica?
By 2012 it was apparent that unseating Facebook with a decentralised free web project was not going to happen. Anybody who appeared on the scene with a new “social network” was immediately compared/contrasted with Facebook and then laughed at. There were a lot of these and most have vanished as quickly as they appeared.
Subsequently we stopped focusing on “social networking” as a project mission. We concentrated on providing a range of privacy respecting services that were decentralised, yet highly integrated. Our focus now is more towards content management, cloud services, and groupware than social networking. You can use the same interface to share a wiki with your basketball team as you can to share private videos with your girlfriend.
We added blogging features, WebDAV, CalDAV and CardDAV, and also a range of content management tools so you can build a company website using the same interface you use to provide customer assistance. You can have loyalty coupons and only those who have the coupon can see your special offers page.
What important lessons did you take from developing DFRN when you decided to go back to the drawing board and develop Zot?
The first big thing I did was solidify the “magic-auth” protocol. In Friendica it was somewhat limited in what it could do. In Zot and Hubzilla magic-auth is really the core of the service.
Access control and privacy can be attached to any resource on your server and restricted to viewers from across the web — no matter what server they have their account on. Your private videos are only visible to select people; and they don’t need an account on your server to view them. Authentication is invisible and you aren’t asked for passwords or tokens. It just happens as you browse your social stream and visit the websites you find therein.
The second important thing we did is provide “nomadic identity”, which is also built into the protocol. In 2010–2012, the free web lost *hundreds of thousands* of early adopters because we had no way to easily migrate from server to server; and lots of early server administrators closed down with little or no warning. This set the free web back at least five years, because you couldn’t trust your account and identity and friendships and content to exist tomorrow. Most of the other free web projects decided that this problem should be solved by import/export tools (which we’re still waiting for in some cases).
I saw an even bigger problem. Twitter at the time was over capacity and often would be shut down for hours or a few days. What if you didn’t really want to permanently move to another server, but you just wanted to post something and stay in touch with friends/family when your server was having a bad day? This was the impetus for nomadic identity. You could take a thumbdrive and load it into any other server; and your identity is intact and you still have all your friends. Then we allowed you to “clone” your identity so you could have these backup accounts available at any time you needed them. Then we started syncing stuff between your clones so that on server ‘A’ you still have the same exact content and friends that you do on server ‘B’. They’re clones. You can post from either. If one shuts down forever, no big deal. If it has a cert issue that takes 24 hours to fix, no big deal. Your online life can continue, uninterrupted — no matter what happens to individual servers.
As far as quality protocol specifications, Zot is at the low end. It started off as an experimental specification and I haven’t seen much interest from other projects in working together on anything — especially something that was invented by a different project. Subsequently Zot is currently only being used in a handful of forked projects. (There was an alternate Zot implementation in Java early on but that project has long been abandoned.) There is enough documentation available that somebody could implement the basic services independently; but might have issues with some of the more esoteric communications like key change notifications and clone sync packets for datasets which they don’t support locally.
Are there any new exciting developments with Hubzilla/red that you’d like to tell people about? What kinds of development are you currently focused on?
We’re currently working on the first major protocol upgrade in 5 years (titled ‘Zot VI’). We will still be able to interact just fine with older zot servers, but most of the services and APIs have been streamlined and separated so that other projects can implement individual pieces of the protocol without requiring them to implement the whole ball of wax including nomadic identity, magic-auth, and messaging. You can write a plugin for NextCloud or WordPress or Diaspora and access Hubzilla private data (or offer private data of your own to share with others) using a Python script, or add just the communications module to your own service without forcing you to implement nomadic identity.
“The way I look at it is that the free web is like family. Everybody has a dysfunctional family.”
We’re also working on federated forums which work seamlessly across all the federated protocols. Currently only ActivityPub has a problem with third-party federation. Diaspora is slightly problematic but I believe we can make it work.
The communications layer will probably add ActivityStreams-JSON as a possible data serialization. When work began on Zot in 2012, this specification was incomplete and changing daily. Hence we used our own internal data format that is mostly compatible, but uses some slightly different naming conventions. Internally it’s just a different driver and Zot can support any number of alternate serializations — subject to the availability of a driver for the chosen format.
At one point, we were close to abandoning social network federation completely (most other services aren’t compatible with either nomadic identity or magic-auth; and those represent our core features/strengths). Then several months ago I upgraded all the federation modules and started working on ActivityPub integration. The way I look at it is that the free web is like family. Everybody has a dysfunctional family. You have black sheep and relatives you really just want to strangle sometimes. Thanksgiving dinner always turns into a shitfight. They’re all fundamentalist Christians and you’re more Zen Buddhist. You can’t carry on a conversation without arguing about who has the more successful career or chastising cousin Harry for his drug use.
But when you get right down to it — none of this matters. They’re family. We’re all in this together. That’s how it is with the free web, even if some projects like to think that they are the only ones that matter. Everybody matters. Each of our projects brings a unique value proposition to the table, and provides a different set of solutions and decentralised services. You can’t ignore any of them or leave any of them behind. We’re one family and we’re all busy creating something incredible. If you look at only one member of this family, you might be disappointed in the range of services that are being offered. You’re probably missing out completely on what the rest of the family is doing. Together we’re all creating a new and improved social web. There are some awesome projects tackling completely different aspects of decentralisation and offering completely different services. If we could all work together we could probably conquer the world — though that’s unlikely to happen any time soon. The first step is just to all sit down at Thanksgiving dinner without killing each other.
This is why I’ve gone back to providing and improving interoperability with other protocols and projects — even if they don’t work perfectly with our core features and services. Hubzilla is currently alone when it comes to being able to see most everybody in the free web family from one place. Friendica isn’t far behind, and I think postActive and Pleroma might get there in the coming months. Then we might have a chance to do something useful with our crazy creations.
In the past, you have been somewhat critical about the development of ActivityPub. Could you illustrate on what your criticisms are, and why you feel that way?
As far as protocols go, ActivityPub is a poor quality specification. This is unfortunate for a protocol with such high expectations, coming from an organization whose only function is to produce working specifications for the web. The places where we needed guidance to produce interoperable products (such as privacy and encryption and signatures and even allowed message content) were left undefined or poorly defined.
Many of the things that were specified with any precision turned out to be things that are critical to interoperation and cross-federation of existing web services and the specification restricts the ability for them to interact in some fundamental ways. If the protocol solved any existing problems there might be a good reason for projects to adopt it, but it doesn’t solve any problems that we don’t already have better solutions for.
“While it doesn’t solve any specific problems for me, I enjoy being able to interact with the rest of the free web no matter what protocol they use.”
I was asked early on in the protocol development what specific use cases I’d like to see covered. I answered with several use cases involving nomadic identity and cross-domain decentralized access control. I was met with “Right. That isn’t going to happen. Come up with some realistic use cases.” The point is, I already have solutions for my use cases. If my customers’ needs don’t matter in the design of a protocol and somebody is soliciting my project as a “customer” of that protocol, I can only suggest that this is a complete marketing failure on their part. At the end of the day, I need to provide solutions for the needs of my customers. The same thing applies to every other free web project.
“Eugen is a clever developer and I respect him, but he simply does not have the experience and insights into creating global protocol decisions that might potentially affect billions of people and will work equally well for Twitter clones and newspaper publishers and bloggers.”
While it doesn’t solve any specific problems for me, I enjoy being able to interact with the rest of the free web no matter what protocol they use. So I implemented ActivityPub as a cross-federation service according to the published spec, at about the same time as Eugen implemented it for Mastodon. We both claim to follow the spec in the areas that matter, yet interoperation in some cases cannot be easily achieved without breaking interoperability with some other service.
The W3C/ActivityPub editors decided that since Mastodon has more potential “users” than any other ActivityPub adopters, Mastodon’s implementation is being praised as the gold standard for interoperability.
Therefore everybody else must bend to the way Mastodon does things, and this shuts off about half of the existing free social web from being able to interoperate over ActivityPub. Mastodon uses some pretty aggressive HTML filtering and has made other implementation decisions which are now being forced on everybody else.
Eugen is a clever developer and I respect him, but he simply does not have the experience and insights into creating global protocol decisions that might potentially affect billions of people and will work equally well for Twitter clones and newspaper publishers and bloggers.
“People on different projects tend to refuse to listen to anybody outside their chosen project, or treat them as an enemy, without looking at what the others bring to the table and what core strengths other projects provide and figuring out how to work with them.”
I’m not saying I do either, but the W3C editors haven’t provided a level playing field and I truly believe the specification is now worthless as a unifying force for the free web. We’re probably stuck with supporting multiple competing protocols for some time (years) into the future. This is OK — it is what it is, but any opportunity for free web unification using a common stack has probably been lost. Ironically, I believe this was ActivityPub’s primary goal, and that makes the specifications which restrict the ability to federate seamlessly with other services flawed — critically.
What’s the most frustrating thing about developing software in this space?
People on different projects tend to refuse to listen to anybody outside their chosen project, or treat them as an enemy, without looking at what the others bring to the table and what core strengths other projects provide and figuring out how to work with them.
As a result, every project re-implements their own incompatible solutions to every federation problem and ridicules any other solutions that others have provided without so much as logging into the service and having a look at how it works. They believe their own project is “special” and someday the masses of the internet will leave the walled gardens and come crawling to their awesome project, begging to use their awesome services. So far even the largest wave of migrations hasn’t come close to 0.05% of the large service populations, and many of those folks tried the free web once and quickly went back to the walled gardens.
Many times I’ve heard things like “I’d use the free web if they had a working events system and forums”. There are projects in the free web that offer these things, but many people on Mastodon and Diaspora do not know this; and so not only do those services not retain the member, but the free web services which actually have these services don’t get a chance to offer their solutions. The person goes away thinking that the free web doesn’t have enough basic features for general use; when in fact only specific projects may be unable to meet their needs.
Because several projects now federate to varying degrees, your choice of software isn’t all that important; just as it doesn’t really matter whether you read your email with K9 or Thunderbird or Outlook.
We have some very diverse projects with even more diverse profile spaces and even when you consider the fact that we’re all speaking completely different languages and protocols, we still have managed to create a space where all can co-exist and to some extent share socially across project boundaries. But we still need to work together, because there can be no special snowflakes. We are *all* the free web. We all have compelling features which may appeal to different groups of people.
What development are you most proud of?
Two things. Nomadic identity because it shows how we can offer resilience in decentralized services, and (ironically) the first iteration of [Friendica’s] Diaspora protocol. It was bloody hard to reverse engineer and disassemble their crypto packets, and this challenged me in ways I haven’t been challenged before or since.
What do you do when you’re not coding?
I’m fortunate to be living in a beautiful and isolated part of the world and I can sit for hours and watch the night sky, or the fog lifting over the Blue Mountains Wilderness (this is literally across the street); or just watch the kangaroos and cows as they go about their lives. Also playing guitar, and spending quiet time with my lovely wife, family, and friends.
https://wedistribute.org/2017/10/got-zot-mike-macgirvin-on-building-your-own-apps-and-protocols/
#DFRN #Friendica #Hubzilla #MikeMacgirvin #Streams #Zot
Got Zot — Mike Macgirvin on building your own apps and protocols
Mike Macgirvin is a personal friend, and a force to be reckoned with. Living on a farm in the hills of Australia, he has quietly spent his days carving out an elaborate system from scratch, doing theSean Tilley (We Distribute)