I incorrectly configured a Redis server with "cluster-enabled yes" and that instance did not belong to a cluster. Connecting to that instance has been functioning without any visible issue. After looking over some metrics, I found the .NET service connected to that Redis server was generating ~320 first-chance exceptions per minute and about 4-5 Gen2 garbage collections per minute.
Using dotnet-dump and inspecting the heap, I found instances of these exceptions:
Exception type: StackExchange.Redis.RedisConnectionException
Message: UnableToConnect on :6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.4.27433
InnerException: <none>
StackTrace (generated):
<none>
Exception type: System.Net.Sockets.SocketException
Message: Name or service not known
InnerException: <none>
StackTrace (generated):
Pipelines.Sockets.Unofficial.dll!Pipelines.Sockets.Unofficial.Internal.Throw.Socket(Int32)+0x2a
Pipelines.Sockets.Unofficial.dll!Pipelines.Sockets.Unofficial.SocketAwaitableEventArgs.GetResult()+0x3e
StackExchange.Redis.dll!StackExchange.Redis.PhysicalConnection+<BeginConnectAsync>d__23.MoveNext()+0x3b1
Exception type: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException
Message: Name or service not known
InnerException: <none>
StackTrace (generated):
System.Net.NameResolution.dll!System.Net.Dns.GetHostEntryOrAddressesCore(System.String, Boolean)+0x16b
System.Net.NameResolution.dll!System.Net.Dns+<>c.<GetHostEntryOrAddressesCoreAsync>b__27_2(System.Object)+0x30
System.Private.CoreLib.dll!System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].InnerInvoke()+0xac
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x41
System.Private.CoreLib.dll!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1c
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)+0x1c4
System.Private.CoreLib.dll!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1c
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)+0xcc
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x46
System.Net.NameResolution.dll!System.Threading.Tasks.TaskToApm.End[[System.__Canon, System.Private.CoreLib]](System.IAsyncResult)+0x88
System.Net.Sockets.dll!System.Net.Sockets.MultipleConnectAsync.DoDnsCallback(System.IAsyncResult, Boolean)+0x94
Exception type: System.Net.Sockets.SocketException
Message: Transport endpoint is not connected
InnerException: <none>
StackTrace (generated):
System.Net.Sockets.dll!System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(System.Net.Sockets.SocketError, System.String)+0xb8
System.Net.Sockets.dll!System.Net.Sockets.Socket.Shutdown(System.Net.Sockets.SocketShutdown)+0x17a
StackExchange.Redis.dll!StackExchange.Redis.PhysicalConnection.Shutdown()+0x174
From the above RedisConnectionException message: UnableToConnect on :6379/Interactive, it looks as if my application was told there was another member of the Redis cluster to connect to with no host address. So a proposed solution could be to add check to ensure the host address is not empty.
After I fixed the Redis server by disabling cluster-mode, these all went away and the Gen2 garbage collections went down to a nominal level. So the problem is solved in my case, but I was surprised that this surfaced as first-chance exceptions. Maybe a second proposed solution could be to eventually trigger a circuit breaker pattern when the failed connection to a host reaches a threshold and let it bubble up to the application.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4