Custom actions can take several different forms the official list, and you will eventually need to add a custom action for only site administrators or site collection administrators. As you might suspect, the XML schema for defining custom actions absolutely supports both of those cases.
The Custom Action specification has RequireSiteAdministrator and Rights attributes that you can use to tailor access to the custom action. Note that this shouldn't be your only layer of security. If you're linking to a custom application page, you should also check the identity of the logged in user in the custom page, in case someone knows the URL to your page.
Limiting to Site Collection Administrators
Use the RequireSiteAdministrator field. It's a boolean, so you'll use
RequireSiteAdministrator="true"
or
RequireSiteAdministrator="false"
"Site Administrator", eh? Is that site level or SPSite level? Spoiler: SPSite level (see section heading). If you specify RequireSiteAdministrator=true, only users who are site collection administrators will see the link.
Limiting to Site (i.e., SPWeb) Administrators
Use
Rights="ManageWeb" RequireSiteAdministrator="false"
to restrict the custom action to users who are site administrators. There are tons of options that you can specify in the Rights attribute, so if you're looking for something more nuanced than "site administrator or not", there might be an option waiting for you.
Unlimited Access
As you might expect, if you want to let any any user see the link, just leave "ManageWeb" out of the rights attribute, and specify
RequireSiteAdministrator="false"
A Conclusion (Of Sorts)
In your project in Visual Studio, ad a new item of the type "Empty Element". In the element file, add a CustomAction node and specify the combination of attributes you need. Double check to make sure that the element file is included in one of the features (and double check to make sure that the feature is included in the package, 'cause you never know).
As I mentioned in the overview, the only thing controlled by specifying these attributes is whether the custom action link is displayed to the user. It doesn't do anything to secure the page itself, so if users know the URL of your page, they can just browse directly to it. So, check the user in your page as well.