This blog post is related to my Google Summer of Code 2023 project .
-
Patch #2352 has been merged
which provides:
- Implementation for
json_encode()user function. - SOG (Single Object Game) file is written for testing the function.
- Implementation for
Solving 7+ days problem...
There is a problem with string passed to the event map. it contains weird stream of text. This
text is actually a DEBUG messages that I created for debugging purposes.
INFO: Steam ASYNC: { "entries":"{wnload scores from lDownloaded scores from leaderboaleaderboard_downSteamClient020[
AlEwersal_System/Extensions/Steamworks/game_client/c_main.cpp:196198_downSteamClient0201m/Extensions/Steamworks/
game_client/c_main.cpp:76561198448852929�,El Piceersal_System/Extensions/Steamworks/game_client/c_main.cpp:
58158_downSteamClient0202m/Extensions/Steamworks/game_client/c_main.cpp:76561198376924302�,saifersal_System/Extensions
/Steamworks/game_client/c_main.cpp:505_downSteamClient0203m/Extensions/Steamworks/game_client/c_main.cpp:
76561199167979105�,Bucciarati (Couter55)ersal_System/Extensions/Steamworks/game_client/c_main.cpp:336_downSteamClient0204m
/Extensions/Steamworks/game_client/c_main.cpp:76561198239127226�,Haiyatinersal_System/Extensions/Steamworks/game_client/
c_main.cpp:236_downSteamClient0205m/Extensions/Steamworks/game_client/c_main.cpp:76561199032162694�,Kaitoersal_System/
Extensions/Steamworks/game_client/c_main.cpp:137_downSteamClient0206m/Extensions/Steamworks/game_client/c_main.cpp:
76561198968916658�,gamesadasteamersal_System/Extensions/Steamworks/game_client/c_main.cpp:96_downSteamClient0207m/
Extensions/Steamworks/game_client/c_main.cpp:76561199289756099�,Bashkash2013ersal_System/Extensions/Steamworks/game_client
/c_main.cpp:7_downSteamClient0208m/Extensions/Steamworks/game_client/c_main.cpp:76561198193841849�,silveradagioersal_System
/Extensions/Steamworks/game_client/c_main.cpp:0_downSteamClient0209m/Extensions/Steamworks/game_client/c_main.cpp:
76561199095741658�,HappyStarXiXiersal_System/Extensions/Steamworks/game_client/c_main.cpp:0_downSteamClient02010m/
Extensions/Steamworks/game_client/c_main.cpp:76561198963365650�]}","event_type":"leaderboard_download","id":1,"lb_name":"",
"num_entries":10,"status":0 }
I actually need only the blue data but the rest is undefined behavior. Look what is the interesting part, before accumulating anything inside the string, it's fine and empty. But after accumulating even a single bracket, it starts to contains that concatenated DEBUG messages.
I didn't faced something like this before, so Jsoh resommended to use
Valgrind, TSAN, and ASAN to find
the problem.
two possibilities:
- Someone is stashing a pointer to that string somewhere out-of-thread
- You have a buffer overflow somewhere and are facing memory corruption
try building your code with TSAN/ASAN enabled to see if you can detect that
if you're on Linux, another option is Valgrind; it will find everything wrong, but will slow the code to a crawl, so make sure you can reproduce the problem quickly and easily
these are good tools to have under your belt and will help you later
cool; make sure the game is built with debugging symbols (-g) so Valgrind output will be useful
set the "mode" var to DEBUG for ENIGMA's make system
Josh — 13/08/2023 18:15
Now that's a higher level of debugging and we are diverging. I started using Valgrind and by the way found a small memory leak in ENIGMA, that can be very annoying when working for long periods which explains why it was freezing my 8GB Virtual Machine.
I have got this output right after the RequestCurrentStats succeeded.
==23675== ----------------------------------------------------------------
==23675==
==23675== Possible data race during read of size 4 at 0x8F0A564 by thread #1
==23675== Locks held: none
==23675== at 0x78F6C2D: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x78F6F10: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x78BCD92: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x7841ADF: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x73B6FE8: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x761CCC1: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x754EC0C: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x487A074: ??? (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x4885F4C: ??? (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x4887A6A: ??? (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x4878114: glXSwapBuffers (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x13F4AE: enigma::ENIGMA_events() (in /tmp/egm5186009574818550152.tmp)
==23675==
==23675== This conflicts with a previous write of size 4 by thread #2
==23675== Locks held: none
==23675== at 0x730FEB4: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x50B5F67: __pthread_once_slow (pthread_once.c:116)
==23675== by 0x730DE07: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x735D7CA: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x485396A: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so)
==23675== by 0x50B0B42: start_thread (pthread_create.c:442)
==23675== by 0x5141BB3: clone (clone.S:100)
==23675== Address 0x8f0a564 is in the BSS segment of /usr/lib/x86_64-linux-gnu/dri/iris_dri.so
==23675==
==23675== ----------------------------------------------------------------
==23675==
==23675== Possible data race during read of size 4 at 0x8F0A584 by thread #1
==23675== Locks held: none
==23675== at 0x78F6C34: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x78F6F10: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x78BCD92: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x7841ADF: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x73B6FE8: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x761CCC1: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x754EC0C: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x487A074: ??? (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x4885F4C: ??? (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x4887A6A: ??? (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x4878114: glXSwapBuffers (in /home/saif/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so)
==23675== by 0x13F4AE: enigma::ENIGMA_events() (in /tmp/egm5186009574818550152.tmp)
==23675==
==23675== This conflicts with a previous write of size 8 by thread #2
==23675== Locks held: none
==23675== at 0x730FEB1: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x50B5F67: __pthread_once_slow (pthread_once.c:116)
==23675== by 0x730DE07: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x735D7CA: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x485396A: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so)
==23675== by 0x50B0B42: start_thread (pthread_create.c:442)
==23675== by 0x5141BB3: clone (clone.S:100)
==23675== Address 0x8f0a584 is in the BSS segment of /usr/lib/x86_64-linux-gnu/dri/iris_dri.so
==23675==
Notice that the start_thread? and the enigma::ENIGMA_events()? That
looks like the problem is here. Another thing is that the gameoverlayrenderer.so
library which is responsible for rendering the Steam overlay is also involved. Actually this
enigma::ENIGMA_events function is generated by the compiler, I mean I can't
debug it directly. I have to debug the code that generates it which is the compiler itself.
The next output is after closing the running exmaple game, it's the summary of the
Valgrind output. Let's keep it here for reference.
==23675== ---Thread-Announcement------------------------------------------
==23675==
==23675== Thread #9 was created
==23675== at 0x5141BA3: clone (clone.S:76)
==23675== by 0x5142A9E: __clone_internal (clone-internal.c:83)
==23675== by 0x50B0758: create_thread (pthread_create.c:295)
==23675== by 0x50B127F: pthread_create@@GLIBC_2.34 (pthread_create.c:828)
==23675== by 0x4853767: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so)
==23675== by 0x735D9E3: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x730EAC7: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x730D9AC: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x730E27D: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x842AE42: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x8436D32: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x8038587: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675==
==23675== ----------------------------------------------------------------
==23675==
==23675== Possible data race during write of size 4 at 0x1FFEFFF998 by thread #1
==23675== Locks held: none
==23675== at 0x730DE43: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x5CAAC1F: ???
==23675== by 0x730E3F5: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x842AEE8: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x8027DC2: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x789875B: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x73B4C2D: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x73B62DB: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x7301592: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x7128382: ??? (in /usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0.0.0)
==23675== by 0x7119FDD: ??? (in /usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0.0.0)
==23675== by 0x711A06C: ??? (in /usr/lib/x86_64-linux-gnu/libGLX_mesa.so.0.0.0)
==23675==
==23675== This conflicts with a previous read of size 4 by thread #9
==23675== Locks held: none
==23675== at 0x50AE9BD: pthread_barrier_wait@@GLIBC_2.34 (pthread_barrier_wait.c:113)
==23675== by 0x4854FC0: pthread_barrier_wait (in
/usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so)
==23675== by 0x730EDAC: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x730DA5C: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x730DBC7: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x735D7CA: ??? (in /usr/lib/x86_64-linux-gnu/dri/iris_dri.so)
==23675== by 0x485396A: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so)
==23675== by 0x50B0B42: start_thread (pthread_create.c:442)
==23675== Address 0x1ffefff998 is on thread #1's stack
==23675==
AL lib: (EE) alc_cleanup: 1 device not closed
==23675==
==23675== Use --history-level=approx or =none to gain increased speed, at
==23675== the cost of reduced accuracy of conflicting-access information
==23675== ERROR SUMMARY: 5370 errors from 124 contexts (suppressed: 9793 from 119)
Even though I don't know what is the problem and can't understand what is going on in
Valgrind output, but the problem is with concatenating
strings over and over to create the Async output format. Josh then recommended to use
stringstream instead of std::string to concatenate strings. I tried
this but as always didn't work.
An idea came to mind to try to concatenate outside the function, just after the
sstream definition. I started with only writing a single bracket:
leaderboard_entries_buffer << '{';
I viewed the string using leaderboard_entries_buffer.str() and it's very
clean!!!. I tried then concatenating the whole string outside the function and it again
failed. I remember that even the first line was failing before and now it's working!!!
Why? Wait a second that's my first trial inside the function:
(*leaderboard_entries_buffer) += '{';
Notice the difference? It's the concatenation operator. When using sstream,
you have to use <<. However when using normal std::string you
can use += or .append() method. They both have the same effect.
Now I know what to do, let's test my theory. Let's remove every += and
replace it with <<. I did that and it worked!!! 🎉 🎉
🎉.
This is not reasonable anyway but it's a solution. I will wait for an explanation from my mentors about this. Here's the final output:
INFO: Steam ASYNC: { "entries":"{"entries":[{"name":"AlEw","score":196198,"rank":1,"userID":"76561198448852929"},
{"name":"El Pice","score":58158,"rank":2,"userID":"76561198376924302"},{"name":"saif","score":505,"rank":3,"userID":"76561199167979105"},
{"name":"Bucciarati (Couter55)","score":336,"rank":4,"userID":"76561198239127226"},{"name":"Haiyatin","score":236,"rank":5,"userID":"76561199032162694"},
{"name":"Kaito","score":137,"rank":6,"userID":"76561198968916658"},{"name":"gamesadasteam","score":96,"rank":7,"userID":"76561199289756099"},
{"name":"Bashkash2013","score":7,"rank":8,"userID":"76561198193841849"},{"name":"silveradagio","score":0,"rank":9,"userID":"76561199095741658"},
{"name":"HappyStarXiXi","score":0,"rank":10,"userID":"76561198963365650"}]}","event_type":"leaderboard_download","id":1,"lb_name":"","num_entries":10,"status":0 }
It's just the lb_name is empty but it's a simple issue we can fix later. Let's move
to the next phase.
More diverging...
There are two important TODOs that must be done here which are:
-
Updating my patch #2352
to use
sstreaminstead ofstd::string. -
For functions such as
steam_download_scores, withidto be saved for each download request, I will use the new ENIGMA's approach for that with is:AssetArrayfromUniversal System. - Implementing a new draw function for drawing leaderboards.
-
Get this done:
@Saif print that debug message three times and I'll bet you it looks different every time
you seem to be seeing stream contamination, not actual memory corruption; I'm guessing your memory is fine
print the string to a file instead if you want to see it unmolested
Josh — 14/08/2023 17:50
This patch #2353 solves the first TODO.
Let's move to the second TODO. I will use the AssetArray to save the id for each
download request made by the player. In order to do that, I will follow simple approach of
preventing any requests as long as we have a pending request. The tricky part here is that
when we are ready to accpet a new request? Well, when we save the entries we got in the
AssetArray, we are ready. So set the loading_ to false
after saving the entries.
Working with AssetArray is very easy. I guess now we are done with the second
TODO.
Another thing that I need to mention here is that GMS has a function called
instance_create_depth(), while ENIGMA has instance_create() and
depth variable. I may provide a patch to combine them both as GMS does to
maintain compatibility. But for now I will just use instance_create() and
set the depth manually.
Here's the TODO list again:
- Implementing a new draw function for drawing leaderboards.
-
Get this done:
@Saif print that debug message three times and I'll bet you it looks different every time
you seem to be seeing stream contamination, not actual memory corruption; I'm guessing your memory is fine
print the string to a file instead if you want to see it unmolested
Josh — 14/08/2023 17:50 - Implement
instance_create_depth()user function.
Now the problem is with the json_decode() user function. It took me the whole day
to find the problem which is bascially Json extension doesn't convert the Json to
DS map correctly. It parses Jsons correctly and also the DataStructures extension
working fine. The layer between them both is broken.
I provided Patch #2355 for issue #2354:
- Fixes
Jsonextension not converting to DS map properly. - Wrote SOG file to test the changes.
- Renamed a SOG file which belongs to short Json PR (I renamed it to better naming while I am here).
After fixing the json_decode() user function, the leaderboard entries started to
view some data. Here is the output:
Here is the GMS output:
Robert finished my 3rd TODO in patch #2356. I tested it and it's working fine. Besides that he talked to me about attaching the id of the download request to the Callback function somehow instead of saving it statically so I think we have got another TODO here.
- Implementing a new draw function for drawing leaderboards.
-
Get this done:
@Saif print that debug message three times and I'll bet you it looks different every time
you seem to be seeing stream contamination, not actual memory corruption; I'm guessing your memory is fine
print the string to a file instead if you want to see it unmolested
Josh — 14/08/2023 17:50 - Changing the way I save download requests' ids.
There is something wrong with Json extension again!!! The keys are there, but there values are contain garbage values like weird integers. I debugged the code and fixed the problem for the 100 time in patch #2357.
Now the Leaderboards API looks complete:
Here is the TODO for the next week:
- Implementing a new draw function for drawing leaderboards.
-
Get this done:
@Saif print that debug message three times and I'll bet you it looks different every time
you seem to be seeing stream contamination, not actual memory corruption; I'm guessing your memory is fine
print the string to a file instead if you want to see it unmolested
Josh — 14/08/2023 17:50 - Changing the way I save download requests' ids.
- Implement the rest APIs.
- Getting an
AppIdand upload a game on Steam for testing. - Creating SOGs and trying to make it work as I mentioned earlier that it's tricky because we need Steam to be running in order the tests to pass.
- Cleaning up and organizing the new Async System to match Robert's needs.
-
Checking if
Callbacksuses Threads. If so then try to handle the State Management manually. - CPP source files need to be documented.
- Check the license for committing DLLs and headers.
- Check Steamworks's official example comments.
- Check GMS Manual for documenting user function's layer.
- Check Robert's PR #2349
-
Creating stub for Steamworks extension to take its place when disabled.
you need a stub for when the extension isnt enabled
its just an empty function
gfundies — 16/08/2023 10:44
add - Document and maintain
Jsonextension. - Document and maintain
Asynchronoussystem. - Explain the Steamworks extension's structure and what goes where.
instance_create_depth() user function to the WIKI.