It is useful to understand what the terms mean: a “slice” is just a namespace for controlling resource allocation to a group of processes (see man:/systemd.slice for more details) and it doesn’t start stuff by itself - when stuff starts, it gets put into a slice. System services get put into the system.slice while user stuff gets put into the user.slice that is then sub-sliced into a slice per user, in it there’s the app.slice, the background.slice, the session.slice, and probably a few others.
If you do systemctl --user status ssh-agent.service and it actually says in the Loaded: line - disabled (not for the vendor preset, the one before that), then something must have requested systemd to start it.
You can look at the dependency list and see if something depends on ssh-agent in the systemd unit tree by running systemctl --user list-dependencies - this will draw a tree and you can see if ssh-agent.service is listed there somewhere. For example:
$ systemctl --user list-dependencies
default.target
● ├─activity-monitor.service
…
● └─basic.target
…
● └─timers.target
● ├─google-chrome-backup.timer
● ├─restic-backup.timer
My backup timers are started by the “timers target”.
The other thing that you can do is to look at the systemd log for your user and see if there are any logs that explain why ssh-agent was started, or at least when it was started and you can see what happened before and after: journalctl --user _COMM=systemd