One thing about time tracking and using scrum and stories I am noticing is that they create secret knowledge

Because people are discouraged from solving problems (especially minor ones) without a card, there's friction to solving minor problems, and so you build secret knowledge
Urusan mastodon (AP)
I end up keeping a parallel to do list for stuff too small to put up on the board officially.

I don't think this is actually a negative thing, most of the secret knowledge generated this way is irrelevant and the remainder can be documented, but it sure doesn't jive with a scrum-centric worldview.
What do you mean that it is irrelevant? Small tasks that are done a lot but hard to fit into tickets are almost all pretty critical. And commonly leads to valuable processes thrashing back and forth constantly.
Urusan mastodon (AP)
Hmm, it does rely a lot on the work being done.

Most of what I'm thinking about are either things like doing required trainings or management-lite stuff involving collaboration with other people. That plus intermediate design work for systems and processes. It's important stuff when it's in flight, but once it's wrapped up it either goes away or has a final document-able form.
Urusan mastodon (AP)
Also, anything routine should be automated.

Even if complete automation is not feasible, a tool can be created to make the process as frictionless as possible for domain experts.

So, once a good foundation is laid down, aside from perhaps a few unavoidable regular tasks and some infrequent but hard to automate work, everything should be novel work.
See, "anything that routine should be automated" is a reasonable statement. And scrum definitely stop people from automating a bunch of common important tasks that PMs don't understand.
Urusan mastodon (AP)
A project manager is a modern day secretary, they don't really know either domain and shouldn't be running the show.
A PM's job is to manage tasks and flow of project work, and often to settle interteam friction or process. It is their job to manage how projects are complete. And generally, it means that their pull is strong on managers and interteam managers who define how work is managed.

Have you worked in an Agile company?
Urusan mastodon (AP)
I have yes.

In my situation though, we have strong engineering management as well as domain experts that aren't business people, so the business folks have to take somewhat of a back seat.

What you're talking about is a common problem, I'm just saying that there's a different way and that overly powerful PMs are a sign of problems.
I think the problem I am having is that you seem to be focusing on Agile-In-Theory and not Agile-In-Practice.

I am glad you have a great employer, but I have not seen one in any of the several places I've worked.