1) Have one C function that opens a socket and returns its descriptor, a positive integer, to the Lua environment.
2) Have additional, separate C functions that take the socket descriptor from 1) and that bind, connect, etc.
I guess that as soon as the scope of C function 1) in ArcaOS is left and the interpreter returns to the Lua environment, any info on the socket descriptor is lost and the socket is automatically closed so that all the other functions 2) reject it ?
What exactly do you want to say with "the interpreter returns to the Lua environment" ? Your agena.exe code sample is a self-contained executable. I cannot see anything returning to anywhere (granted that the code sample is not complete ...).
I could image that if the "socket" implementation maps socket handles to "virtual" file handles that then handle inheritance and such come into play. That would mean that you will have to know what process or processes access these handles. I have no clue how you would make a socket handle created in process A known to a process B but I suppose that there are ways to do that.
Another thing that I could think of (intentionally or erroneously) is if there is some "tcperrno" variable that might get globally set and that you might have to reset before a call.
I hope that you will only ever get a socket handle >= 3. Because 0 would be for stdin, 1 for stdout and 2 for stderr if we follow the logic that socket handles are mapped to file handles.