This blog post is related to my Google Summer of Code 2023 project: Steam Workshop SDK/Third Party Integration For ENIGMA .

When the Stats and Achievements API is done, I continued to work on the Leaderboard API. I started learning about CallResults because I am going to use it in the Leaderboard API.

Actually, learning about CallResults was not that hard as I know how to use Callbacks which has a similar concept.

    // c_leaderboards.h
                        
    // Called when SteamUserStats()->FindOrCreateLeaderboard() returns asynchronously
    void on_find_leaderboard(LeaderboardFindResult_t* pFindLearderboardResult, bool bIOFailure);
    CCallResult m_callResultFindLeaderboard;
    
    // Set the CallResult to call on_find_leaderboard function on finished
    c_leaderboards::m_callResultFindLeaderboard.Set(steam_api_call, this, &c_leaderboards::on_find_leaderboard);
    
    void c_leaderboards::on_find_leaderboard(LeaderboardFindResult_t* pFindLeaderboardResult, bool bIOFailure) {
        c_leaderboards::loading_ = false;
    
        if (!pFindLeaderboardResult->m_bLeaderboardFound || bIOFailure) return;

        c_leaderboards::current_leaderboard_ = pFindLeaderboardResult->m_hSteamLeaderboard;

        DEBUG_MESSAGE("Calling FindOrCreateLeaderboard succeeded.", M_INFO);
    }
                            
                        

    // c_overlay.h

    STEAM_CALLBACK(c_overlay, on_game_overlay_activated, GameOverlayActivated_t, m_CallbackGameOverlayActivated);

    // The constructor be something like:
    c_overlay::c_overlay() : m_CallbackGameOverlayActivated(this, &c_overlay::on_game_overlay_activated) {}
    
    void c_overlay::on_game_overlay_activated(GameOverlayActivated_t* pCallback) {
        if (pCallback->m_bActive) {
            c_overlay::overlay_activated_ = true;
            DEBUG_MESSAGE("Overlay activated successfully.", M_INFO);
        } else {
            c_overlay::overlay_activated_ = false;
            DEBUG_MESSAGE("Overlay deactivated successfully.", M_INFO);
        }
    }
                            
                        

The real problem that I faced is with ENIGMA's Async system. The Async system will need some modifications as it is not Thread safe which means that I can't depend on it. The Universal_System/Extensions/Asynchronous extension is the one we are talking about here. As Robert said, this extension will need some work as it's not thread safe. I will have to create an issue for that to be fixed later.

The Async system is used to send and store data inside a user variable called async_load. The game developer can then read that variable and get the value for each key inside it using the DataStructures extension.

I will leave the testing for later. Creating SOGs and make the CI handles them is tricky because we need Steam.exe to be running in the background.

we'll follow up in the second half of this project about sogs, if we'll make it to doing them or can even get it to work

R0bert — 18/07/2023 07:37

The changes to the Async system are not final yet.

@Saif later we will change your queue to be a an std::pair with first item being event string name, and second item being the posted event map, then we can switch (event_name) and make your queue processing fire more events than just the myevent_asyncsteam

R0bert — 18/07/2023 19:57

I had a long discussion with Josh about Robert's requested changes to the Async System and Josh suggested to use the old system for some reason:

I think I'm gonna need to chat about this with him, first

Josh — 22/07/2023 07:53

Actually, after this discussion, I learned about the code generation system that ENIGMA uses:

it has to be the bottommost one, ObjectLocals

however, that would mean that you need to build any routine that's calling the event against the generated ObjectLocals sources

if you check your codegen folder, you should see an entry by that name in ObjectLocals

~/.enigma

or on Windows, {APP_DATA}/.enigma

specifically, ~/.enigma/Preprocessor_Environment_Editable/IDE_EDIT_objectdeclarations.h

to give a little better detail before I pass out, the reason we have those extension cast routines is so that we don't need to know about object_locals in the extension source code

otherwise, we'd have to rebuild every extension every time the game changed

no, it's just the name of the bottommost parent class

it's generated code

iiuc, if you open that header file, you should see myevent_steam declared in struct object_locals

if you instead go the extension route, you can declare your own event by that name and use the generated cast routine from a separate source

so uncomment the stuff you showed me earlier, but rename the event to match what's in events.ey

Josh — 22/07/2023 08:20

The code works fine when I added the changes to PFMain.h and PFmain.cpp files.

    // PFmain.cpp

    namespace enigma {

        namespace extension_cast {
            extension_steamworks* as_extension_steamworks(object_basic*);
        }
        
        }  // namespace enigma
        
        namespace enigma {
        
            std::queue> posted_async_events;
        
            int extension_steamworks_mutex {1}; // Must be initialized to 1.
            
            void wait(int* mutex) {
                while (*mutex <= 0);
                (*mutex)--;
            }
        
            void signal(int* mutex) {
                (*mutex)++;
            }
        
            void fireEventsFromQueue(int* mutex) {
                // wait(mutex);
                while (!posted_async_events.empty()) {
                    enigma_user::ds_map_clear(enigma_user::async_load);
    
                    std::map event = posted_async_events.front();
    
                    posted_async_events.pop();
    
                    for (auto& [key, value] : event) {
                        enigma_user::ds_map_add(enigma_user::async_load, key, value);
                    }
    
                    instance_event_iterator = &dummy_event_iterator;
                    for (iterator it = instance_list_first(); it; ++it) {
                        object_basic* const inst = ((object_basic*)*it);
                        extension_steamworks* const inst_steamworks = extension_cast::as_extension_steamworks(inst);
                        inst_steamworks->myevent_steam();
                    }
                }
            }
            
            int enigma_main(int argc, char** argv) {
                    enigma_user::async_load = enigma_user::ds_map_create();
        
                    ...
        
                    while (!game_isending) {
        
                        ...
                        
                        fireEventsFromQueue(&extension_steamworks_mutex);
                        ...
                    }
                
                    ...
            }
            
        }
        
        namespace enigma_user {
        
            int async_load;
            
        }
                                
                            

    // PFmain.h

    namespace enigma {

        // This queue will contain any event that will be fired inside the main loop.
        extern std::queue> posted_async_events;
    
        // The following semaphore code is for the new Async system to protect the posted_async_events queue
        // from reading and writing at the same time. Steamworks extension writes to the queue.
        // mutex to protect the async queue from race conditions when the Steamworks extension is writing into it.
        extern int extension_steamworks_mutex;
        void wait(int* mutex);
        void signal(int* mutex);
    
    }

    namespace enigma_user {
    
        // When firing any event inside posted_async_events queue, this variable will contain the data that will be sent to the game.
        extern int async_load;
    
    }
                            
                        

Notice the mutex implementation stuff? Just keep reading, I removed them later haha.

Bascially, this queue will contain any event from any extension we have, which means we might face Race Condition problems. That's why I need to protect the queue from reading and writing at the same time. Actually, ENIGMA is single threaded so I will wait for confirmation from Robert that this is not needed.

This is how the new Async system works now and the problem is with oraganizing the code.

probably make the layer class core/universal system or something like add a way to register event types in an extension by string name with pointer function that the process queue uses to dispatch, idk

R0bert — 23/07/2023 20:12

Now let's keep that aside and talk about another changes requested (if possible) by Robert regarding Steam Callbacks. The problem is ENIGMA is single threaded and we don't know if Steam Callbacks are using threads or not.

normally nothing in ENIGMA is threaded, so it doesn't matter when we check for things

but in this case, some 3-p API is using threads, and we want to adapt that to our single-thread system

Josh — 23/07/2023 20:25

idk that it's using threads, Saif can't prove that yet, it's not in any Steamworks docs or anything but, yeh

R0bert — 23/07/2023 20:26

if there's a single-threaded routine that checks if something has happened, then we don't need the system you're proposing at all

Josh — 23/07/2023 20:25

if it weren't using threads, it would be using semaphores and locks itself or something to have IPC internally

hypothetically ofc

R0bert — 23/07/2023 20:26

that is 100% conducive to our completely standard way of doing all events

Josh — 23/07/2023 20:27

that is 100% conducive to our completely standard way of doing all events

Josh — 23/07/2023 20:27

Josh just mentioned, you can look for a non-async method

R0bert — 23/07/2023 20:27

the only reason we'd need new/custom infrastructure here is if a 3p API was forcing us to use a thread

Josh — 23/07/2023 20:27

@Saif look for a poll_ leaderboard function that's not async

R0bert — 23/07/2023 20:27

i.e, the only way the API gives us to know that something has happened is to give it a routine to run in a background thread

because we have a way to just check, in the main thread, if something has occurred?

in that case, no need for new infrastructure; continue using the system I created for extensions which iiuc Saif has already successfully done 😛

Josh — 23/07/2023 20:27

@Saif look for a non callback version, one that lets you poll it directly is what Josh is asking me now

then you can just poll it once a frame and say "Hey steam, have the leaderboards loaded yet?"

it may not have something like that

R0bert — 23/07/2023 20:27

So the problem is basically without using Callbacks, I need to handle the State Management code myself and the reason I said possible above is that I don't know if that's possible with Steamworks API or not. I told Robert that it's not a good time for that change but I will add it as a TODO for later.

To wrap what will be done after the midterms evaluation:

• Implement the rest APIs.
• Creating SOGs and trying to make it work as I mentioned earlier that it's tricky because we need Steam to be running in order the tests to pass.
• Cleaning up and organizing the new Async System to match Robert's needs.
• Checking if Callbacks uses Threads. If so then try to handle the State Management manually.

I will just leave the following conversation here for reference:

callbacks are very important because for example when the user opens the Overlay i can set a variable to true but what happens when close it, i don't call a function to close the overlay the user can close it from the X button at the top right so no way i can know if it's closed or not that's why callbacks are very important, any event happens like closing or opening the overlay the callback function is called and i know the event type as well, how im gonna build that behavior without callbacks idk

Saif — 23/07/2023 20:32

have you used React?

have you made a dialog in React?

R0bert — 23/07/2023 20:32

no Flutter maybe

it's the same just explain ur point

Saif — 23/07/2023 20:32

typically you do this...

    myOpenDialogIsOpen = useState<boolean>(false)

        <MyDialog
            isOpen={myOpenDialogIsOpen}>
            ...
        </MyDialog>
                                

this sounds detached but i'm telling you abstractly the answer to your question above

just like how modern OpenGL and Vulkan don't give you ANY state management, you have to do it yourself

did you ever do any older OpenGL with immediate mode glBegin and stuff?

R0bert — 23/07/2023 20:32

unfortunately no

Saif — 23/07/2023 20:35

ok, well those older GL apis are deprecated

because they required a stack to implement, aka state management

all the Graphics APIs, D3D, GL, Vulkan (it started with AMD Mantle and Apple Metal) moved to being low-level

that dropped everything but triangle primitives (no quads, fans etc) and all the upper-level state management and provided only programmable graphics shaders

so basically people/game engines have had to reimplement state machines on top of that

@Saif and i am using that as an analogy for why the steam callbacks are "wrong"

they are low-level, which is very efficient, but they are NOT providing that state management for us (we don't know, that was Josh's question!), but we end up having to write a queue and do synchronization ourselves

@Saif which again as i said is totally normally expected with libraries

it's actually a Unix principle do something "simple" and do it "really well"

"This is the Unix philosophy: Write programs that do one thing and do it well."

let me know if that's clear explanatio or not

@Saif let me give pseudo code example (ignore ENIGMA/GM, think only C++) Async version:

    load_me_a_leaderboard_async(my_leaderboard_callback);

    // this might happen on a thread
    void my_leaderboard_callback(LeaderboardData leaderboard_data) {
        // do something with leaderboard data
    }  
                                

Async polling version:

    load_me_a_leaderboard();

    // you should poll on the main thread
    // inside main loop....
    leaderboard_data = poll_leaderboard_data_async();
    if (leaderboard_data != nullptr) {
        // do something with leaderboard data
    } else {
        // leaderboard data not ready yet, try again next frame
    }   
                                

@Saif the latter is like how we do input in XInput extension for Xbox 360 controller, extension polls the state of the controller 1-time every single frame, caches that, and all the functions are just returning state from that cached value, ideally if Steam had an API to do that for leaderboards it would be great and that's what Josh was questioning if existed

again idk if Steam supports that way of doing it or not, but feel free to look into the docs and let us know

the reason Josh favors that way is because, albeit it looks slightly more verbose than the other way, but it's not when you consider the other way required us to add a queue and synchronization primitive which are exactly the things Josh feels "wrong" about muddying up our main loop with

R0bert — 23/07/2023 20:39

So the new Async system basically is for me to work using Thread safe code

right, and agreed, for now we just want Saif's to work and be thread safe

R0bert — 23/07/2023 22:07

The final structure to the new Async system:

    // PFmain.cpp

    #include "Universal_System/Extensions/Steamworks/steamworks.h"

    namespace enigma {

        std::queue<std::map<std::string, variant>> posted_async_events;

        std::mutex posted_async_events_mutex;

        void fireEventsFromQueue() {
            // Acquire lock and release it when out of scope of fireEventsFromQueue().
            std::lock_guard<std::mutex> guard(posted_async_events_mutex);
            while (!posted_async_events.empty()) {
        
                ...

                enigma::fireSteamworksEvent();
            }
        }

        int enigma_main(int argc, char** argv) {
            enigma_user::async_load = enigma_user::ds_map_create();

            ...

            while (!game_isending) {

                ...
        
                fireEventsFromQueue(&extension_steamworks_mutex);

                ...

            }

            ...
        }

    }

    namespace enigma_user {

        int async_load;

    }
                                
                            

    // PFmain.h

    namespace enigma {

        extern std::queue<std::map<std::string, variant>> posted_async_events;

        extern std::mutex posted_async_events_mutex;

        void fireEventsFromQueue();

    }

    namespace enigma_user {

        extern int async_load;

    }

                                
                            

This is my last day before the midterms evaluation so I will just focus on cleaning up Leaderboard API and document all the implemented APIs.

الحَمْدُ لِله دائما وأبدا

My POE ChatGPT account is now become paid. The POE was just starting out so when they became stable, they started to charge for their services. I tried to find a way to register but couldn't. My father helped me alot with that as his cousin lives in Poland. He gave me his number to register an account with it. It was very easy that way. Actually, if he knew what I did to activate my account... I don't even want to think about it. Anyway, enough with the drama, let's get ready for Midterm Evaluations.