View source for User:NoloZero01/Guides/First Aid For Crashes(SourceMod)


This page is aimed at helping beginner SourceMod operators deal with the enigma that Source Engine server crashes can sometimes be.

This article assumes that you have some basic SourceMod experience already - what plugins/extensions are and how to install and remove them. Deduction will also be necessary.

Do not expect this to be a magical "all-in-one" solution - rather, treat this as a starting point.

== What happened? ==

	
Many SourceMod-related crashes fall into the following two categories:

	'''Incorrect plugin or extension logic.'''
		The Source engine can be fragile at times.
		Your plugins/extensions could be passing garbage data to the engine, changing entities in unusual and unsupported ways,
		or generally doing something that *probably* shouldn't be done in the first place.
		Some might simply be not compatible with new builds of your game - for example, they could be expecting a different entity property layout.
		The vast majority of such crashes require plugin/extension source code fixes. 
	

	'''Outdated and incorrect gamedata.'''
		Gamedata, in general terms, is information that SourceMod can use to directly target the game server's memory.
		Simply put it contains instructions and clues in the form of function signatures and virtual function table offsets that SourceMod can use to call or hook internal server functions.
		Gamedata can be no longer valid after server updates and is very likely to cause recurring crashes if is it incorrect.
	
		You can find gamedata files in the **gamedata** SourceMod directory. It can be split into two broad categories: Core and Third-party.
			Core gamedata ships with SourceMod itself as part of the download bundle and is vital to SourceMod's operation. It is maintained by the SourceMod development team and is automatically updated on server start (if necessary).
			Third-party gamedata can be shipped with plugins and extensions that you install. It should be maintained by the author and will not update automatically.
			
		Game updates, be they big or small, can invalidate existing gamedata. If your server started experiencing crashes after an update, chances are your gamedata is outdated. 
		
		
== First steps ==


Your actions may differ depending on the background of the crash - whether there's been a recent game update or you installed/changed something in your plugins/configs.
	

* Recurring crashes started happening after an update

:This is common behavior for broken gamedata. Check your SourceMod error log and try to find anything suspicious. In many situations SourceMod is able to detect invalid gamedata
:and refuse to load the offending plugin with a corresponding error log, however signature collisions are not impossible. In this case a different function will be at the old gamedata address,
:leading to undefined behavior and, in most situations, a crash. If the update happened just now, chances are there won't be an immediate fix - monitor the repositories/threads of your plugins
:for updates and see if others are experiencing crashes. Be sure to run the latest stable builds of SM/MM:S too.


* There was no update, and the crashes started happening out of nowhere
:Crashes don't happen out of nowhere (except rare unexplainable Source magic, but that's a known diagnosis anyway). Try to remember what you did to the server - maybe you installed a new plugin or edited a configuration file.
:Since you're a good server operator, you should always take backups before changing anything - try to revert the changes and see if that fixes it.
:If you didn't take backups and/or rolling changes back doesn't fix the issue then it's time to start looking for the offender manually.
		

First of all, remove ALL plugins and third-party extensions from your install and see if that fixes it (most of the time it does). If so, follow this scheme:
		
* Identify what kind of code causes the crash - first readd all extensions, see if it's a plugin causing it.
* Readd ALL plugins and remove one half of them (avoid keeping plugins with removed dependencies)
* If the crash persists remove half of the remaining plugins, otherwise replace with the removed first half.
* Keep narrowing down the list of potentially faulty plugins by repeating the previous step.
		
This will leave you with the root cause of your crash, a single (or sometimes multiple!) plugin.	
		

== Extracting and analyzing ==


First of all, install '''Accelerator'''. Its a third-party crash dump processing platform maintained by one of the core SourceMod developers, and it is invaluable when it comes to analyzing server crashes.

::Once it is installed and configured, trigger the crash and restart the server. Accelerator will upload the crash dump to its processing backend and add a new entry to its log file, addons/sourcemod/logs/accelerator.log.
::In there you'll find your crash ID - a series of characters that looks like '''XXX-XXX-XXX'''. Once it's there navigate to [https://crash.limetech.org]https://crash.limetech.org and sign in with your Steam account (you did configure Accelerator with your SteamID, right?). Go to the '''Dashboard''' - you'll see your latest crash there.
	
The main page of the crash contains the '''stack trace''' - the list of functions executed right before the crash occured, where the last call is at the top of the stack.
::See if you can notice any custom extension or plugin names in it - if so, it may indicate that the crash happened in one. Most extensions and all plugins are built with debug symbols, giving you clear function names.
::Accelerator is also able to detect common crash patterns and provide probable crash causes - for example, an SDKCall with incorrect gamedata.
	
If the stack trace doesn't have any clues, click the "'''View Raw'''" button.
::Once loaded you'll see the hexadecimal representatation of the stack trace with ASCII-decoded strings on the right side. Take a look at the first line of its text.
::It can be something like '''EXCEPTION_ACCESS_VIOLATION_READ''' or '''SIGABRT''' - this is the crash error. You may use a search engine to look up the meaning of these, as it may help you identify the root cause.
::Now, lets take a look at the hexadecimal stack trace. Calls are sorted in reverse, with the last call being at the top.
::Every frame starts with the library/executable name, followed either by a function name or an offset. In the case of SourceMod calls there will also be a link to its corresponding part of source code.
::You can open these links and try to understand the underlying code - coupled with the crash reason we've talked about previously, you may be able to decipher the root cause.
	

Let's take a look at an example. Visit [https://crash.limetech.org/lha6z6lmccdk]https://crash.limetech.org/lha6z6lmccdk - this a crash from a Windows TF2 server.


<code>
	'''0 sourcemod.2.tf2.dll!UTIL_FindDataMapInfo(datamap_t *,char const *,SourceMod::sm_datatable_info_t *) [HalfLife2.cpp:381 + 0x2]'''

	'''1 sourcemod.2.tf2.dll!CHalfLife2::FindDataMapInfo(datamap_t *,char const *,SourceMod::sm_datatable_info_t *) [HalfLife2.cpp:504 + 0x10]'''

	'''2 sourcemod.2.tf2.dll!GetEntPropString [smn_entities.cpp:2292 + 0x4c]'''

	'''3 sourcepawn.jit.x86.dll!sp::Environment::Invoke(sp::PluginContext *,ke::RefPtr<sp::MethodInfo> const &,int *) [environment.cpp:346 + 0x18]'''

</code>


From this we can rebuild the function call chain, it looks something like this: <code>'''Environment::Invoke() -> GetEntPropString() -> CHalfLife2::FindDataMapInfo() -> UTIL_FindDataMapInfo()'''</code>,
with the last function being the one that crashed.
::Hover your mouse over each stack entry - you'll see a small icon appear at the right edge of the line. This will open its exact line of SourceMod source code.
::Lets open the link to the source code of '''UTIL_FindDataMapInfo''' (https://github.com/alliedmodders/sourcemod/blob/6cac489fce5ef2b3400fd92d3781aac21331d4e7/core/HalfLife2.cpp#L381).
	
::::You'll see the following construct:
::::<code>'''if (pMap->dataDesc[i].fieldName == NULL)'''</code>
		
::::We can see that its accessing a child member of pMap called dataDesc, and that its an array of objects with their own member values.
::::Remember the crash reason that we looked at earlier? In our case its EXCEPTION_ACCESS_VIOLATION_READ - it means that the program tried to read memory outside of its allocation (going out of bounds).
::::C++ array logic is just memory math, so pMap->dataDesc[i].fieldName unwraps into
		
::::<code>'''((<pMap address>) + (<dataDesc member offset>) + (i * <size of each object referenced by a dataDesc cell>) + (<offset of fieldName from the address of referenced object>))'''</code>.
		
::::Knowing that the program crashed due to memory access violation we can conclude that this memory address calculation resulted in an invalid address.
::::Looking up UTIL_FindDataMapInfo we find that its a SourceMod Plugin API function (https://sm.alliedmods.net/new-api/entity/FindDataMapInfo).
		
::::Checking more stack trace frames we see this:

<code>
'''3 jit_code_15400960_1048576 + 0xda4fd [ plugin_name.smx::.11644.GetEntityClassname ]'''

'''4 jit_code_15400960_1048576 + 0xba8a5 [ plugin_name.smx::OnPluginStart ]'''
</code>

::::This gives us more clues as to what plugin caused the crash and what functions preceded the crashed call.
::::In our case a plugin named plugin_name.smx started (OnPluginStart) and called GetEntityClassname somewhere within OnPluginStart code, followed by a call to sourcemod.2.tf2.dll's GetEntPropString function.
::::We can also find its SourcePawn API equivalent (https://sm.alliedmods.net/new-api/entity/GetEntPropString).
		
::::Combining all of the above we can track down the call to UTIL_FindDataMapInfo in our plugin's source code and start analyzing its surroundings to find the faulty logic.
	

== Crashes with no useful symbols	 ==

	
Now, chances are you've noticed strings like '''engine.dll + 0x1d9700''' in your stack trace.
::This indicates that the binary in question has no debug symbols, and Accelerator is able to give you the location of the function as an offset from the binary's base address, but not the function name nor its arguments.
::SourceMod, MM:S and most extensions come with debug symbols, but the base game server binaries may not.
::This is the case for most Windows builds of Source servers, for example TF2. Such crashes are harder to analyze and it will be covered in a future update to this page.
	
	
	

== How to seek help ==


The first rule is to always try to help yourself before asking others.
::Keep your plugins/extensions and gamedata up to date, try to analyze your crash with the method mentioned above, read through your plugin's AlliedModders thread/GitHub Issues.
::If you're still stuck and unable to find the crash cause, collect as much information about your server as possible, prepare the Accelerator crash dump links.
	
::Run the following line as one command in your server's console (NOT the client game console):
<code>
'''version; plugin_print; meta version; meta list; sm version; sm exts list; sm plugins list'''
</code>

::Post this on a SourceMod support forum or in the AlliedModders Discord server in the #sourcemod chat.
::Be sure to explain how the crash occurs, how (and if) you can reproduce it, and, as always, '''be respectful'''! SourceMod is a community effort and noone is forced to help you.