Your questions cannot be answered directly, since they do not take into account the main point - this is COM technology.
From VBA, you can work with libraries that implement COM object specifications and cannot directly work with .NET objects. Thus, the .NET library needs to implement the COM specifications so that its functionality can be used from VBA.
On the basis of what to choose digit capacity? Based on the bitness of Office or OS?
Digit capacity is selected based on the digit capacity of the process in which the COM object will be launched. The fact is that COM objects can be In-Proc or Out-Proc, i.e. launched in the process of the caller or in a separate.
Is it possible to compile libu as AnyCpu and not wrestle with the first item?
It is possible, but you need to remember about the correct marshaling of data.
The meaning of regasm is that just the lib would be visible in the references, that is, you can do without this step and just point the way to it in the VBA Office?
VBA sees COM objects that are written in the Windows registry according to the COM specification. regasm needed in order to register data on the .NET library functionality in the registry according to the COM specification. But it can not be used constantly, if you tick the Register for COM interop in the project properties, see here .