Hi,
We recently observed a deadlock in one of our applications that is reaching out to Redis. In both cases, there was an issue on one of the Redis replicas for several minutes that took a replica offline. When it came back, the services did not recover gracefully and, after getting a crash dump, DebugDiag highlighted the following deadlock (I added some notes on the lock locations in bold):
Thread 1 -
[[GCFrame]]
[[GCFrame]]
[[HelperMethodFrame] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object) Attempts to lock on _writtenAwaitingResponse, deadlocks
StackExchange.Redis.PhysicalConnection.GetHeadMessages(StackExchange.Redis.Message ByRef, StackExchange.Redis.Message ByRef)+47
StackExchange.Redis.ExceptionFactory.AddCommonDetail(System.Collections.Generic.List1>, System.Text.StringBuilder, StackExchange.Redis.Message, StackExchange.Redis.ConnectionMultiplexer, StackExchange.Redis.ServerEndPoint)+78 StackExchange.Redis.ExceptionFactory.Timeout(StackExchange.Redis.ConnectionMultiplexer, System.String, StackExchange.Redis.Message, StackExchange.Redis.ServerEndPoint, System.Nullable1)+4b3
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[[StackExchange.Redis.RedisValue, StackExchange.Redis]](StackExchange.Redis.Message, StackExchange.Redis.ResultProcessor1, StackExchange.Redis.ServerEndPoint, StackExchange.Redis.RedisValue)+199 **Grabs lock on result box** StackExchange.Redis.RedisBase.ExecuteSync[[StackExchange.Redis.RedisValue, StackExchange.Redis]](StackExchange.Redis.Message, StackExchange.Redis.ResultProcessor1, StackExchange.Redis.ServerEndPoint, StackExchange.Redis.RedisValue)+af
StackExchange.Redis.RedisDatabase.StringGet(StackExchange.Redis.RedisKey, StackExchange.Redis.CommandFlags)+e5
Thread 2:
[[HelperMethodFrame] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object) Trying to lock the result box
StackExchange.Redis.SimpleResultBox.StackExchange.Redis.IResultBox.ActivateContinuations()+2f
StackExchange.Redis.Message.Complete()+4c
StackExchange.Redis.PhysicalConnection.RecordConnectionFailed(StackExchange.Redis.ConnectionFailureType, System.Exception, System.String, Boolean, System.IO.Pipelines.IDuplexPipe)+e57 - This line grabs a lock on _writtenAwaitingResponse
StackExchange.Redis.PhysicalConnection+d__115.MoveNext()+8c9
[[HelperMethodFrame]]
mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+20
System.IO.Pipelines.PipeCompletion.ThrowLatchedException()+e
System.IO.Pipelines.Pipe.GetReadResult(System.IO.Pipelines.ReadResult ByRef)+46
System.IO.Pipelines.Pipe.GetReadAsyncResult()+e0
StackExchange.Redis.PhysicalConnection+d__115.MoveNext()+558
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+172
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15
mscorlib_ni!System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()+6f
Pipelines.Sockets.Unofficial.DedicatedThreadPoolPipeScheduler.Execute(System.Action`1, System.Object)+1e
Pipelines.Sockets.Unofficial.DedicatedThreadPoolPipeScheduler.RunWorkLoop()+11a
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+172
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+55
mscorlib_ni!System.Threading.ThreadHelper.ThreadStart(System.Object)+60
In the second thread, it looks like that thread grabs a lock on _writtenAwaitingResponse in PhsyicalConnection.cs, then gets stuck on the lock in SimpleResultBox.ActivateContinuations. The first thread meanwhile has already grabbed the continuations lock and is trying to grab the _writtenAwaitingResponse in GetHeadMessages.
We are currently on library version 2.6.66, and the application is a .NET Framework 4.8 Windows Service.
Thank you!
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4