The implementation of the vtable could be just a nested structure at the end of the heap with a constant reference at the beginning. There are 2 ways to implement the heap/base of the library:
per-opener or static.
Per-opener makes a new heap allocation every time the library is opened. This is how Posix OS’s generally do it as do Mac and Windows.
Static is faster and more Rust-like. It’s how Amiga and similar OS designs like MorphOS and AROS do most of their shared libraries. Static library implementations still allow box allocations so it’s generally the best of both worlds. I would suggest supporting both kinds of library.
There is one final matter for a dlopen/Posix style library compatibility: querying which functions are available and how many parameters they each take.
Since the AmigaOS offshoots had a revision number constant publicly available to insure that their OpenLibrary command had a new enough version to implement all the jump table entries, there was no need for DLL Hell.
The more mainstream OS designs were not so lucky or insightful and had to put versions into the filenames of their .so/.dylib/.dll to avoid version incompatibility. Thus if a function becomes obsolete a whole new library had to be created while an Amiga-style .library just redirected the old jump table entry to an error code and put the new implementation at the end of the jump table as a new entry in the structure.