Vulnerability disclosure: Hijacking Synapse's port 8008 after triggering systemd-oomd with a Synapse memory leak #172

New issue

Closed

opened 2022-08-04 17:45:38 +00:00 by a · 1 comment

a commented

2022-08-04 17:45:38 +00:00

Owner

This security vulnerability has been fixed and was not known to have been exploited.

This vulnerability was reported by @ersei in a private email:

So my plan is to crash Synapse and steal the port (8008). Here's how:

Maximise memory use in Synapse by exploiting a bug[0] that causes a memory leak (you're running 1.63.1, the bug was fixed 1.64.0). This will put Synapse on the top of the list of memory use.

Next, I will create a memory leak myself with many small program runs and fill up the RAM. systemd-oomd will kill the program using the most memory (Synapse). Have a program that constantly tries to steal the port in the background.

Once the port is stolen, use it to sniff HTTP connections and gather credentials. I know you're active on Matrix and it would be trivial to sniff the credentials by returning 401 until the client reauths with the username/password in plaintext (there's no SSL since Nginx handles it and I'm acting behind Nginx). It wouldn't be too difficult to create a shim program to return 401 to all endpoints and only accept login endpoints and print the JSON posted to it.

And since I think you're probably an admin user and the passwords are all the same between programs due to the magic of LDAP, I can get root access to the machine that way and throw a stealthkit on there. Maybe it'll be my new Monero rig :)

This has the downside of maybe crashing your server, using all the memory, and definitely crashing Synapse... which I'm sure you want to avoid. Maybe you should update Synapse to make my job a little harder.

The vulnerability was not as serious as Ersei suggested and an attacker using this would not have been able to get root access, although they could probably still steal Matrix credentials. The attack would be obvious and noticeable, so we're very confident that no one ever carried it out. This was my response:

That sounds like a smart idea, but I'm not sure how well it would actually work out since I've anticipated attacks like these and have some mitigation measures in place for each of the steps:

I updated Synapse to 1.64.0 as you suggested 😀 (technically not a mitigation already in place, but we do in general update pretty frequently)

As you said, this uses all the memory and crashes the server so it would definitely be noticeable. Also, Matrix would stop working so I'd notice the attack for that reason as well (unless you start a patched Synapse server that steals credentials).

For Synapse, there's not much we can do about this, since I don't think Synapse supports running on a Unix socket instead of the TCP port 8008, but almost all of the other services use Unix sockets when possible.

My account doesn't use LDAP (in case the LDAP server malfunctions) so you probably wouldn't be able to get root access.

I'm interested in figuring out whether this same attack works with JupyterHub, since we also use a TCP socket for it and the hub process runs as root! However, if you try to cause an OOM, for instance by running a memory leak in a JupyterLab terminal, I think systemd-oomd would kill your process instead of the hub process.

Anyways, nice work finding this attack method!

For disclosing future vulnerabilities, Gitea get a private issues feature soon, so you can use that, or send an email to help@exozy.me.

**This security vulnerability has been fixed and was not known to have been exploited.** This vulnerability was reported by @ersei in a private email: > So my plan is to crash Synapse and steal the port (8008). Here's how: > > 1. Maximise memory use in Synapse by exploiting a bug[0] that causes a memory leak (you're running 1.63.1, the bug was fixed 1.64.0). This will put Synapse on the top of the list of memory use. > > 2. Next, I will create a memory leak myself with many small program runs and fill up the RAM. systemd-oomd will kill the program using the most memory (Synapse). Have a program that constantly tries to steal the port in the background. > > 3. Once the port is stolen, use it to sniff HTTP connections and gather credentials. I know you're active on Matrix and it would be trivial to sniff the credentials by returning 401 until the client reauths with the username/password in plaintext (there's no SSL since Nginx handles it and I'm acting behind Nginx). It wouldn't be too difficult to create a shim program to return 401 to all endpoints and only accept login endpoints and print the JSON posted to it. > > 4. And since I think you're probably an admin user and the passwords are all the same between programs due to the magic of LDAP, I can get root access to the machine that way and throw a stealthkit on there. Maybe it'll be my new Monero rig :) > > This has the downside of maybe crashing your server, using all the memory, and definitely crashing Synapse... which I'm sure you want to avoid. Maybe you should update Synapse to make my job a little harder. The vulnerability was not as serious as Ersei suggested and an attacker using this would not have been able to get root access, although they could probably still steal Matrix credentials. The attack would be obvious and noticeable, so we're very confident that no one ever carried it out. This was my response: > That sounds like a smart idea, but I'm not sure how well it would actually work out since I've anticipated attacks like these and have some mitigation measures in place for each of the steps: > > 1. I updated Synapse to 1.64.0 as you suggested 😀 (technically not a mitigation already in place, but we do in general update pretty frequently) > > 2. As you said, this uses all the memory and crashes the server so it would definitely be noticeable. Also, Matrix would stop working so I'd notice the attack for that reason as well (unless you start a patched Synapse server that steals credentials). > > 3. For Synapse, there's not much we can do about this, since I don't think Synapse supports running on a Unix socket instead of the TCP port 8008, but almost all of the other services use Unix sockets when possible. > > 4. My account doesn't use LDAP (in case the LDAP server malfunctions) so you probably wouldn't be able to get root access. > > I'm interested in figuring out whether this same attack works with JupyterHub, since we also use a TCP socket for it and the hub process runs as root! However, if you try to cause an OOM, for instance by running a memory leak in a JupyterLab terminal, I think systemd-oomd would kill your process instead of the hub process. > > Anyways, nice work finding this attack method! For disclosing future vulnerabilities, Gitea get a private issues feature soon, so you can use that, or send an email to help@exozy.me.

a added the

bug

security

labels 2022-08-04 17:45:38 +00:00

a added this to the (deleted) project 2022-08-04 17:45:39 +00:00

a closed this issue

2022-08-04 17:45:44 +00:00

a referenced this issue

2022-08-04 17:53:54 +00:00

Vulnerability disclosure: JupyterHub's proxied code-server sessions have no authentication and can be taken over #173

a commented

2022-08-05 02:04:57 +00:00

Author

Owner

@ersei I looked at the Synapse 1.64 release log, and it said that the memory leak is in frozendict, not Synpase itself. Since we updated frozendict to a non-vulnerable version on August 23, I doubt that you could have hijacked the Synapse port even if we hadn't updated Synapse to 1.64.

@ersei I looked at the [Synapse 1.64 release log](https://matrix.org/blog/2022/08/03/synapse-1-64-released), and it said that the memory leak is in frozendict, not Synpase itself. Since we updated frozendict to a non-vulnerable version on August 23, I doubt that you could have hijacked the Synapse port even if we hadn't updated Synapse to 1.64.

a referenced this issue

2022-08-05 20:28:40 +00:00

Ideas for improving exozyme security #175