This blog post is related to my Google Summer of Code 2023 project .
When the Stats and Achievements
API is done, I continued to work on the
Leaderboard
API. I started learning about CallResults
because I am
going to use it in the Leaderboard
API.
Actually, learning about CallResults
was not that hard as I know how to
use Callbacks
which has a similar concept.
The real problem that I faced is with ENIGMA's Async
system. The
Async
system will need some modifications as it is not Thread safe which
means that I can't depend on it. The
Universal_System/Extensions/Asynchronous
extension is the one we are talking about here. As Robert said, this extension
will need some work as it's not thread safe. I will have to create an issue for that
to be fixed later.
The Async
system is used to send and store data inside a user variable
called async_load
. The game developer can then read that variable and
get the value for each key inside it using the DataStructures
extension.
I will leave the testing for later. Creating SOGs and make the CI handles them
is tricky because we need Steam.exe
to be running in the background.
we'll follow up in the second half of this project about sogs, if we'll make it to doing them or can even get it to work
R0bert — 18/07/2023 07:37
The changes to the Async
system are not final yet.
@Saif later we will change your queue to be a an std::pair with first item being event string name, and second item being the posted event map, then we can switch (event_name) and make your queue processing fire more events than just the myevent_asyncsteam
R0bert — 18/07/2023 19:57
I had a long discussion with Josh about Robert's requested changes to the
Async
System and Josh suggested to use the old system for some
reason:
I think I'm gonna need to chat about this with him, first
Josh — 22/07/2023 07:53
Actually, after this discussion, I learned about the code generation system that ENIGMA uses:
it has to be the bottommost one, ObjectLocals
however, that would mean that you need to build any routine that's calling the event against the generated ObjectLocals sources
if you check your codegen folder, you should see an entry by that name in ObjectLocals
~/.enigma
or on Windows, {APP_DATA}/.enigma
specifically,
~/.enigma/Preprocessor_Environment_Editable/IDE_EDIT_objectdeclarations.h
to give a little better detail before I pass out, the reason we have those extension cast routines is so that we don't need to know about object_locals in the extension source code
otherwise, we'd have to rebuild every extension every time the game changed
no, it's just the name of the bottommost parent class
it's generated code
iiuc, if you open that header file, you should see myevent_steam declared in struct object_locals
if you instead go the extension route, you can declare your own event by that name and use the generated cast routine from a separate source
so uncomment the stuff you showed me earlier, but rename the event to match what's in events.ey
Josh — 22/07/2023 08:20
The code works fine when I added the changes to PFMain.h
and
PFmain.cpp
files.
Notice the mutex implementation stuff? Just keep reading, I removed them later haha.
Bascially, this queue will contain any event from any extension we have, which means we might face Race Condition problems. That's why I need to protect the queue from reading and writing at the same time. Actually, ENIGMA is single threaded so I will wait for confirmation from Robert that this is not needed.
This is how the new Async system works now and the problem is with oraganizing the code.
probably make the layer class core/universal system or something like add a way to register event types in an extension by string name with pointer function that the process queue uses to dispatch, idk
R0bert — 23/07/2023 20:12
Now let's keep that aside and talk about another changes requested (if possible) by Robert
regarding Steam Callbacks
. The problem is ENIGMA is single threaded and we don't
know if Steam Callbacks
are using threads or not.
normally nothing in ENIGMA is threaded, so it doesn't matter when we check for things
but in this case, some 3-p API is using threads, and we want to adapt that to our single-thread system
Josh — 23/07/2023 20:25
idk that it's using threads, Saif can't prove that yet, it's not in any Steamworks docs or anything but, yeh
R0bert — 23/07/2023 20:26
if there's a single-threaded routine that checks if something has happened, then we don't need the system you're proposing at all
Josh — 23/07/2023 20:25
if it weren't using threads, it would be using semaphores and locks itself or something to have IPC internally
hypothetically ofc
R0bert — 23/07/2023 20:26
that is 100% conducive to our completely standard way of doing all events
Josh — 23/07/2023 20:27
that is 100% conducive to our completely standard way of doing all events
Josh — 23/07/2023 20:27
Josh just mentioned, you can look for a non-async method
R0bert — 23/07/2023 20:27
the only reason we'd need new/custom infrastructure here is if a 3p API was forcing us to use a thread
Josh — 23/07/2023 20:27
@Saif look for a poll_ leaderboard function that's not async
R0bert — 23/07/2023 20:27
i.e, the only way the API gives us to know that something has happened is to give it a routine to run in a background thread
because we have a way to just check, in the main thread, if something has occurred?
in that case, no need for new infrastructure; continue using the system I created for extensions which iiuc Saif has already successfully done 😛
Josh — 23/07/2023 20:27
@Saif look for a non callback version, one that lets you poll it directly is what Josh is asking me now
then you can just poll it once a frame and say "Hey steam, have the leaderboards loaded yet?"
it may not have something like that
R0bert — 23/07/2023 20:27
So the problem is basically without using Callbacks
, I need to handle the State
Management code myself and the reason I said possible above is that I don't know if that's
possible with Steamworks API or not. I told Robert that it's not a good time for that change but
I will add it as a TODO for later.
To wrap what will be done after the midterms evaluation:
- Implement the rest APIs.
- Creating SOGs and trying to make it work as I mentioned earlier that it's tricky because we need Steam to be running in order the tests to pass.
- Cleaning up and organizing the new Async System to match Robert's needs.
- Checking if
Callbacks
uses Threads. If so then try to handle the State Management manually.
I will just leave the following conversation here for reference:
callbacks are very important because for example when the user opens the Overlay i can set a variable to true but what happens when close it, i don't call a function to close the overlay the user can close it from the X button at the top right so no way i can know if it's closed or not that's why callbacks are very important, any event happens like closing or opening the overlay the callback function is called and i know the event type as well, how im gonna build that behavior without callbacks idk
Saif — 23/07/2023 20:32
have you used React?
have you made a dialog in React?
R0bert — 23/07/2023 20:32
no Flutter maybe
it's the same just explain ur point
Saif — 23/07/2023 20:32
typically you do this...
this sounds detached but i'm telling you abstractly the answer to your question above
just like how modern OpenGL and Vulkan don't give you ANY state management, you have to do it yourself
did you ever do any older OpenGL with immediate mode glBegin and stuff?
R0bert — 23/07/2023 20:32
unfortunately no
Saif — 23/07/2023 20:35
ok, well those older GL apis are deprecated
because they required a stack to implement, aka state management
all the Graphics APIs, D3D, GL, Vulkan (it started with AMD Mantle and Apple Metal) moved to being low-level
that dropped everything but triangle primitives (no quads, fans etc) and all the upper-level state management and provided only programmable graphics shaders
so basically people/game engines have had to reimplement state machines on top of that
@Saif and i am using that as an analogy for why the steam callbacks are "wrong"
they are low-level, which is very efficient, but they are NOT providing that state management for us (we don't know, that was Josh's question!), but we end up having to write a queue and do synchronization ourselves
@Saif which again as i said is totally normally expected with libraries
it's actually a Unix principle do something "simple" and do it "really well"
"This is the Unix philosophy: Write programs that do one thing and do it well."
let me know if that's clear explanatio or not
@Saif let me give pseudo code example (ignore ENIGMA/GM, think only C++) Async version:
Async polling version:
@Saif the latter is like how we do input in XInput extension for Xbox 360 controller, extension polls the state of the controller 1-time every single frame, caches that, and all the functions are just returning state from that cached value, ideally if Steam had an API to do that for leaderboards it would be great and that's what Josh was questioning if existed
again idk if Steam supports that way of doing it or not, but feel free to look into the docs and let us know
the reason Josh favors that way is because, albeit it looks slightly more verbose than the other way, but it's not when you consider the other way required us to add a queue and synchronization primitive which are exactly the things Josh feels "wrong" about muddying up our main loop with
R0bert — 23/07/2023 20:39
So the new Async system basically is for me to work using Thread safe code
right, and agreed, for now we just want Saif's to work and be thread safe
R0bert — 23/07/2023 22:07
The final structure to the new Async system:
This is my last day before the midterms evaluation so I will just focus on cleaning up
Leaderboard
API and document all the implemented APIs.
الحَمْدُ لِله دائما وأبدا
My POE ChatGPT account is now become paid. The POE was just starting out so when they became stable, they started to charge for their services. I tried to find a way to register but couldn't. My father helped me alot with that as his cousin lives in Poland. He gave me his number to register an account with it. It was very easy that way. Actually, if he knew what I did to activate my account... I don't even want to think about it. Anyway, enough with the drama, let's get ready for Midterm Evaluations.