Semian is a library for controlling access to slow or unresponsive external services to avoid cascading failures.
When services are down they typically fail fast with errors like ECONNREFUSED and ECONNRESET which can be rescued in code. However, slow resources fail slowly. The thread serving the request blocks until it hits the timeout for the slow resource. During that time, the thread is doing nothing useful and thus the slow resource has caused a cascading failure by occupying workers and therefore losing capacity. Semian is a library for failing fast in these situations, allowing you to handle errors gracefully. Semian does this by intercepting resource access through heuristic patterns inspired by Hystrix and Release It:
Resource drivers are monkey-patched to be aware of Semian, these are called Semian Adapters. Thus, every time resource access is requested Semian is queried for status on the resource first. If Semian, through the patterns above, deems the resource to be unavailable it will raise an exception. The ultimate outcome of Semian is always an exception that can then be rescued for a graceful fallback. Instead of waiting for the timeout, Semian raises straight away.
If you are already rescuing exceptions for failing resources and timeouts, Semian is mostly a drop-in library with a little configuration that will make your code more resilient to slow resource access. But, do you even need Semian?
For an overview of building resilient Ruby applications, start by reading the Shopify blog post on Toxiproxy and Semian. For more in depth information on Semian, see Understanding Semian. Semian is an extraction from Shopify where it's been running successfully in production since October, 2014.
The other component to your Ruby resiliency kit is Toxiproxy to write automated resiliency tests.
Install by adding the gem to your Gemfile and require the adapters you need:
gem 'semian', require: %w(semian semian/mysql2 semian/redis)
We recommend this pattern of requiring adapters directly from the Gemfile. This ensures Semian adapters are loaded as early as possible and also protects your application during boot. Please see the adapter configuration section on how to configure adapters.
Semian works by intercepting resource access. Every time access is requested, Semian is queried, and it will raise an exception if the resource is unavailable according to the circuit breaker or bulkheads. This is done by monkey-patching the resource driver. The exception raised by the driver always inherits from the Base exception class of the driver, meaning you can always simply rescue the base class and catch both Semian and driver errors in the same rescue for fallbacks.
The following adapters are in Semian and tested heavily in production, the version is the version of the public gem with the same name:
semian/mysql2 (~> 0.3.16)semian/redis (~> 3.2.1)semian/net_httpsemian/activerecord_trilogy_adaptersemian-postgresTo create a Semian adapter you must implement the following methods:
include Semian::Adapter. Use the helpers to wrap the resource. This takes care of situations such as monitoring, nested resources, unsupported platforms, creating the Semian resource if it doesn't already exist and so on.#semian_identifier. This is responsible for returning a symbol that represents every unique resource, for example redis_master or mysql_shard_1. This is usually assembled from a name attribute on the Semian configuration hash, but could also be <host>:<port>.connect. The name of this method varies. You must override the driver's connect method with one that wraps the connect call with Semian::Resource#acquire. You should do this at the lowest possible level.query. Same as connect but for queries on the resource.ResourceBusyError and CircuitOpenError. These are raised when the request was rejected early because the resource is out of tickets or because the circuit breaker is open (see Understanding Semian. They should inherit from the base exception class from the raw driver. For example Mysql2::Error or Redis::BaseConnectionError for the MySQL and Redis drivers. This makes it easy to rescue and handle them gracefully in application code, by rescueing the base class.The best resource is looking at the already implemented adapters.
There are some global configuration options that can be set for Semian:
# Maximum size of the LRU cache (default: 500) # Note: Setting this to 0 enables aggressive garbage collection. Semian.maximum_lru_size = 0 # Minimum time in seconds a resource should be resident in the LRU cache (default: 300s) Semian.minimum_lru_time = 60 # If true, raise exceptions in case of a validation / constraint failure # Otherwise, log in output Semian.default_force_config_validation = false
Note: minimum_lru_time is a stronger guarantee than maximum_lru_size. That is, if a resource has been updated more recently than minimum_lru_time it will not be garbage collected, even if it would cause the LRU cache to grow larger than maximum_lru_size.
Note: default_force_config_validation set to true is a potentially breaking change. Misconfigured Semians will raise errors, so make sure that this is what you want. See more in Configuration Validation.
When instantiating a resource it now needs to be configured for Semian. This is done by passing semian as an argument when initializing the client. Examples built in adapters:
# MySQL2 client
# In Rails this means having a Semian key in database.yml for each db.
client = Mysql2::Client.new(host: "localhost", username: "root", semian: {
name: "master",
tickets: 8, # See the Understanding Semian section on picking these values
success_threshold: 2,
error_threshold: 3,
error_timeout: 10,
force_config_validation: false
})
# Redis client
client = Redis.new(semian: {
name: "inventory",
tickets: 4,
success_threshold: 2,
error_threshold: 4,
error_timeout: 20
})
Semian now provides a flag to specify log-based and exception-based configuration validation. To explicitly force the Semian to validate it's configurations, pass force_config_validation: true into your resource. This will raise an error in the case of a misconfigured or illegal Semian. Otherwise, if it is set to false, it will log misconfigured parameters verbosely in output.
If not specified, it will use Semian.default_force_config_validation as the flag.
When migrating to use force_config_validation: true, follow these steps:
force_config_validation: false in your configuration[SEMIAN_CONFIG_WARNING]: prefix. These logs will indicate misconfigured Semian resourcesforce_config_validation: true to enable strict validationExample log entries to look for:
[SEMIAN_CONFIG_WARNING]: Missing required arguments for Semian: [:success_threshold, :error_threshold, :error_timeout]
[SEMIAN_CONFIG_WARNING]: Both bulkhead and circuitbreaker cannot be disabled.
[SEMIAN_CONFIG_WARNING]: Bulkhead configuration require either the :tickets or :quota parameter, you provided neither
Semian's circuit breaker implementation is thread-safe by default as of v0.7.0. If you'd like to disable it for performance reasons, pass thread_safety_disabled: true to the resource options.
Bulkheads should be disabled (pass bulkhead: false) in a threaded environment (e.g. Puma or Sidekiq), but can safely be enabled in non-threaded environments (e.g. Resque and Unicorn). As described in this document, circuit breakers alone should be adequate in most environments with reasonably low timeouts.
Internally, semian uses SEM_UNDO for several sysv semaphore operations:
The intention behind SEM_UNDO is that a semaphore operation is automatically undone when the process exits. This is true even if the process exits abnormally - crashes, receives a SIG_KILL, etc, because it is handled by the operating system and not the process itself.
If, however, a thread performs a semop, the SEM_UNDO is on its parent process. This means that the operation will not be undone when the thread exits. This can result in the following unfavorable behavior when using threads:
SEM_UNDO, but since it's a thread there is the potential for ticket starvation. This can result in deadlock on the resource.SEM_UNDO, but for threads the worker count will continue to increment, only being undone when the parent process dies. This can cause the number of tickets to dramatically exceed the quota.Moreover, a strategy that utilizes SEM_UNDO is not compatible with a strategy that attempts to the semaphores tickets manually. In order to support threads, operations that currently use SEM_UNDO would need to use no semaphore flag, and the calling process will be responsible for ensuring that threads are appropriately cleaned up. It is still possible to implement this, but it would likely require an in-memory semaphore managed by the parent process of the threads. PRs welcome for this functionality.
You may now set quotas per worker:
client = Redis.new(semian: {
name: "inventory",
quota: 0.51,
success_threshold: 2,
error_threshold: 4,
error_timeout: 20
})
Per the above example, you no longer need to care about the number of tickets. Rather, the tickets shall be computed as a proportion of the number of active workers.
In this case, we'd allow 50% of the workers on a particular host to connect to this redis resource. So long as the workers are in their own process, they will automatically be registered. The quota will set the bulkhead threshold based on the number of registered workers, whenever a new worker registers.
This is ideal for environments with non-uniform worker distribution, and to eliminate the need to manually calculate and adjust ticket counts.
Note:
tickets or quota.Semian.unregister_all_resources before/after forking.For the Net::HTTP specific Semian adapter, since many external libraries may create HTTP connections on the user's behalf, the parameters are instead provided by associating callback functions with Semian::NetHTTP, perhaps in an initialization file.
To give Semian parameters, assign a proc to Semian::NetHTTP.semian_configuration that takes a two parameters, host and port like 127.0.0.1,443 or github_com,80, and returns a Hash with configuration parameters as follows. The proc is used as a callback to initialize the configuration options, similar to other adapters.
SEMIAN_PARAMETERS = { tickets: 1,
success_threshold: 1,
error_threshold: 3,
error_timeout: 10 }
Semian::NetHTTP.semian_configuration = proc do |host, port|
# Let's make it only active for github.com
if host == "github.com" && port.to_i == 80
SEMIAN_PARAMETERS.merge(name: "github.com_80")
else
nil
end
end
# Called from within API:
# semian_options = Semian::NetHTTP.semian_configuration("github.com", 80)
# semian_identifier = "nethttp_#{semian_options[:name]}"
The name should be carefully chosen since it identifies the resource being protected. The semian_options passed apply to that resource. Semian creates the semian_identifier from the name to look up and store changes in the circuit breaker and bulkhead states and associate successes, failures, errors with the protected resource.
We only require that the semian_configuration be set only once over the lifetime of the library.
If you need to return different values for the same pair of host/port value, you must include the dynamic: true option. Returning different values for the same host/port values without setting the dynamic option can lead to undesirable behavior.
A common example for dynamic options is the use of a thread local variable, such as ActiveSupport::CurrentAttributes, for requests to a service acting as a proxy.
SEMIAN_PARAMETERS = {
# ...
dynamic: true,
}
class CurrentSemianSubResource < ActiveSupport::CurrentAttributes
attribute :sub_name
end
Semian::NetHTTP.semian_configuration = proc do |host, port|
name = "#{host}_#{port}"
if (sub_resource_name = CurrentSemianSubResource.sub_name)
name << "_#{sub_resource_name}"
end
SEMIAN_PARAMETERS.merge(name: name)
end
# Two requests to shopify.com can use two different semian resources,
# as long as `CurrentSemianSubResource.sub_name` is set accordingly:
# CurrentSemianSubResource.set(sub_name: "sub_resource_1") { Net::HTTP.get_response(URI("http://shopify.com")) }
# and:
# CurrentSemianSubResource.set(sub_name: "sub_resource_2") { Net::HTTP.get_response(URI("http://shopify.com")) }
For most purposes, "#{host}_#{port}" is a good default name. Custom name formats can be useful to grouping related subdomains as one resource, so that they all contribute to the same circuit breaker and bulkhead state and fail together.
A return value of nil for semian_configuration means Semian is disabled for that HTTP endpoint. This works well since the result of a failed Hash lookup is nil also. This behavior lets the adapter default to whitelisting, although the behavior can be changed to blacklisting or even be completely disabled by varying the use of returning nil in the assigned closure.
Since we envision this particular adapter can be used in combination with many external libraries, that can raise additional exceptions, we added functionality to expand the Exceptions that can be tracked as part of Semian's circuit breaker. This may be necessary for libraries that introduce new exceptions or re-raise them. Add exceptions and reset to the default list using the following:
# assert_equal(Semian::NetHTTP.exceptions, Semian::NetHTTP::DEFAULT_ERRORS) Semian::NetHTTP.exceptions += [::OpenSSL::SSL::SSLError] Semian::NetHTTP.reset_exceptions # assert_equal(Semian::NetHTTP.exceptions, Semian::NetHTTP::DEFAULT_ERRORS)Mark Unsuccessful Responses as Failures
Unsuccessful responses (e.g. 5xx responses) do not raise exceptions, and as such are not marked as failures by default. The open_circuit_server_errors Semian configuration parameter may be set to enable this behaviour, to mark unsuccessful responses as failures as seen below:
SEMIAN_PARAMETERS = { tickets: 1,
success_threshold: 1,
error_threshold: 3,
error_timeout: 10,
open_circuit_server_errors: true }
Semian supports Active Record adapter trilogy. It can be configured in the database.yml:
semian: &semian
success_threshold: 2
error_threshold: 3
error_timeout: 4
half_open_resource_timeout: 1
bulkhead: false # Disable bulkhead for Puma: https://github.com/shopify/semian#thread-safety
name: semian_identifier_name
default: &default
adapter: trilogy
username: root
password:
host: localhost
read_timeout: 2
write_timeout: 1
connect_timeout: 1
semian:
<<: *semian
Example cases for activerecord-trilogy-adapter can be run using BUNDLE_GEMFILE=gemfiles/activerecord_trilogy_adapter.gemfile bundle exec rake examples:activerecord_trilogy_adapter
Semian is a library with heuristics for failing fast. This section will explain in depth how Semian works and which situations it's applicable for. First we explain the category of problems Semian is meant to solve. Then we dive into how Semian works to solve these problems.
Semian is not a trivial library to understand, introduces complexity and thus should be introduced with care. Remember, all Semian does is raise exceptions based on heuristics. It is paramount that you understand Semian before including it in production as you may otherwise be surprised by its behaviour.
Applications that benefit from Semian are those working on eliminating SPOFs (Single Points of Failure), and specifically are running into a wall regarding slow resources. But it is by no means a magic wand that solves all your latency problems by being added to your Gemfile. This section describes the types of problems Semian solves.
If your application is multithreaded or evented (e.g. not Resque and Unicorn) these problems are not as pressing. You can still get use out of Semian however.
This is better illustrated with a real world example from Shopify. When you are browsing a store while signed in, Shopify stores your session in Redis. If Redis becomes unavailable, the driver will start throwing exceptions. We rescue these exceptions and simply disable all customer sign in functionality on the store until Redis is back online.
This is great if querying the resource fails instantly, because it means we fail in just a single roundtrip of ~1ms. But if the resource is unresponsive or slow, this can take as long as our timeout which is easily 200ms. This means every request, even if it does rescue the exception, now takes an extra 200ms. Because every resource takes that long, our capacity is also significantly degraded. These problems are explained in depth in the next two sections.
With Semian, the slow resource would fail instantly (after a small amount of convergence time) preventing your response time from spiking and not decreasing capacity of the cluster.
If this sounds familiar to you, Semian is what you need to be resilient to latency. You may not need the graceful fallback depending on your application, in which case it will just result in an error (e.g. a HTTP 500) faster.
We will now examine the two problems in detail.
In-depth analysis of real world exampleIf a single resource is slow, every single request is going to suffer. We saw this in the example before. Let's illustrate this more clearly in the following Rails example where the user session is stored in Redis:
def index @user = fetch_user @posts = Post.all end private def fetch_user user = User.find(session[:user_id]) rescue Redis::CannotConnectError nil end
Our code is resilient to a failure of the session layer, it doesn't HTTP 500 if the session store is unavailable (this can be tested with Toxiproxy). If the User and Post data store is unavailable, the server will send back HTTP 500. We accept that, because it's our primary data store. This could be prevented with a caching tier or something else out of scope.
This code has two flaws however:
These two problems in turn have two related problems associated with them: response time and capacity.
Requests that attempt to access a down session storage are all gracefully handled, the @user will simply be nil, which the code handles. There is still a major impact on users however, as every request to the storage has to time out. This causes the average response time to all pages that access it to go up by however long your timeout is. Your timeout is proportional to your worst case timeout, as well as the number of attempts to hit it on each page. This is the problem Semian solves by using heuristics to fail these requests early which causes a much better user experience during downtime.
When your single-threaded worker is waiting for a resource to return, it's effectively doing nothing when it could be serving fast requests. To use the example from before, perhaps some actions do not access the session storage at all. These requests will pile up behind the now slow requests that are trying to access that layer, because they're failing slowly. Essentially, your capacity degrades significantly because your average response time goes up (as explained in the previous section). Capacity loss simply follows from an increase in response time. The higher your timeout and the slower your resource, the more capacity you lose.
It should be clear by now that timeouts aren't enough. Consistent timeouts will increase the average response time, which causes a bad user experience, and ultimately compromise the performance of the entire system. Even if the timeout is as low as ~250ms (just enough to allow a single TCP retransmit) there's a large loss of capacity and for many applications a 100-300% increase in average response time. This is the problem Semian solves by failing fast.
Semian consists of two parts: Circuit Breaker and Bulkheading. To understand Semian, and especially how to configure it, we must understand these patterns and their implementation.
Disable Semian via environment variable SEMIAN_DISABLED=1.
The circuit breaker pattern is based on a simple observation - if we hit a timeout or any other error for a given service one or more times, we’re likely to hit it again for some amount of time. Instead of hitting the timeout repeatedly, we can mark the resource as dead for some amount of time during which we raise an exception instantly on any call to it. This is called the circuit breaker pattern.
When we perform a Remote Procedure Call (RPC), it will first check the circuit. If the circuit is rejecting requests because of too many failures reported by the driver, it will throw an exception immediately. Otherwise the circuit will call the driver. If the driver fails to get data back from the data store, it will notify the circuit. The circuit will count the error so that if too many errors have happened recently, it will start rejecting requests immediately instead of waiting for the driver to time out. The exception will then be raised back to the original caller. If the driver’s request was successful, it will return the data back to the calling method and notify the circuit that it made a successful call.
The state of the circuit breaker is local to the worker and is not shared across all workers on a server.
Circuit Breaker ConfigurationThere are four configuration parameters for circuit breakers in Semian:
true if not set.error_threshold_timeout amount of time before opening the circuit, that is to start rejecting requests instantly.error_threshold errors must occur to open the circuit. Defaults to error_timeout seconds if not set.error_timeout is still used and will reset the circuit. Defaults to true if not set.It is possible to disable Circuit Breaker with environment variable SEMIAN_CIRCUIT_BREAKER_DISABLED=1.
For more information about configuring these parameters, please read this post.
For some applications, circuit breakers are not enough. This is best illustrated with an example. Imagine if the timeout for our data store isn't as low as 200ms, but actually 10 seconds. For example, you might have a relational data store where for some customers, 10s queries are (unfortunately) legitimate. Reducing the time of worst case queries requires a lot of effort. Dropping the query immediately could potentially leave some customers unable to access certain functionality. High timeouts are especially critical in a non-threaded environment where blocking IO means a worker is effectively doing nothing.
In this case, circuit breakers aren't sufficient. Assuming the circuit is shared across all processes on a server, it will still take at least 10s before the circuit is open. In that time every worker is blocked (see also "Defense Line" section for an in-depth explanation of the co-operation between circuit breakers and bulkheads) this means we're at reduced capacity for at least 20s, with the last 10s timeouts occurring just before the circuit opens at the 10s mark when a couple of workers have hit a timeout and the circuit opens. We thought of a number of potential solutions to this problem - stricter timeouts, grouping timeouts by section of our application, timeouts per statement—but they all still revolved around timeouts, and those are extremely hard to get right.
Instead of thinking about timeouts, we took inspiration from Hystrix by Netflix and the book Release It (the resiliency bible), and look at our services as connection pools. On a server with W workers, only a certain number of them are expected to be talking to a single data store at once. Let's say we've determined from our monitoring that there’s a 10% chance they’re talking to mysql_shard_0 at any given point in time under normal traffic. The probability that five workers are talking to it at the same time is 0.001%. If we only allow five workers to talk to a resource at any given point in time, and accept the 0.001% false positive rate—we can fail the sixth worker attempting to check out a connection instantly. This means that while the five workers are waiting for a timeout, all the other W-5 workers on the node will instantly be failing on checking out the connection and opening their circuits. Our capacity is only degraded by a relatively small amount.
We call this limitation primitive "tickets". In this case, the resource access is limited to 5 tickets (see Configuration). The timeout value specifies the maximum amount of time to block if no ticket is available.
graph TD;
Start[Start]
CheckConnection{Request Connection to Resource}
AllocateTicket[Allocate Ticket if Available]
BlockTimeout[Block until Timeout or Ticket Available]
AccessResource[Access Resource]
ReleaseTicket[Release Ticket]
FailRequest[Fail Request]
OpenCircuit[Open Circuit Breaker]
Start --> CheckConnection
CheckConnection -->|Ticket Available| AllocateTicket
AllocateTicket --> AccessResource
AccessResource --> ReleaseTicket
ReleaseTicket --> CheckConnection
CheckConnection -->|No Ticket Available| BlockTimeout
BlockTimeout -->|Timeout| FailRequest
BlockTimeout -->|Ticket Available| AccessResource
FailRequest --> OpenCircuit
OpenCircuit --> CheckConnection
Loading
How do we limit the access to a resource for all workers on a server when the workers do not directly share memory? This is implemented with SysV semaphores to provide server-wide access control.
There are two configuration values. It's not easy to choose good values and we're still experimenting with ways to figure out optimal ticket numbers. Generally something below half the number of workers on the server for endpoints that are queried frequently has worked well for us.
true if not set.0 unless you have very few workers running (i.e. less than ~5).It is possible to disable Bulkhead with environment variable SEMIAN_BULKHEAD_DISABLED=1.
Note that there are system-wide limitations on how many tickets can be allocated on a system. cat /proc/sys/kernel/sem will tell you.
Bulkhead debugging on linuxSystem-wide limit on the number of semaphore sets. On Linux systems before version 3.19, the default value for this limit was 128. Since Linux 3.19, the default value is 32,000. On Linux, this limit can be read and modified via the fourth field of
/proc/sys/kernel/sem.
Note: It is often helpful to examine the actual IPC resources on the system. Semian provides an easy way to get the semaphore key:
irb> require 'semian'
irb> puts Semian::Resource.new(:your_resource_name, tickets: 42).key # do this from a dev machine
"0x48af51ea"
This key can then be used to easily inspect the semaphore on any host machine:
ipcs -si $(ipcs -s | grep 0x48af51ea | awk '{print $2}')
Which should output something like:
Semaphore Array semid=5570729
uid=8192 gid=8192 cuid=8192 cgid=8192
mode=0660, access_perms=0660
nsems = 4
otime = Thu Mar 30 15:06:16 2017
ctime = Mon Mar 13 20:25:36 2017
semnum value ncount zcount pid
0 1 0 0 48
1 42 0 0 48
2 42 0 0 27
3 31 0 0 48
In the above example, we can see each of the semaphores. Looking at the enum code in ext/semian/sysv_semaphores.h we can see that:
The finished defense line for resource access with circuit breakers and bulkheads then looks like this:
The RPC first checks the circuit; if the circuit is open it will raise the exception straight away which will trigger the fallback (the default fallback is a 500 response). Otherwise, it will try Semian which fails instantly if too many workers are already querying the resource. Finally the driver will query the data store. If the data store succeeds, the driver will return the data back to the RPC. If the data store is slow or fails, this is our last line of defense against a misbehaving resource. The driver will raise an exception after trying to connect with a timeout or after an immediate failure. These driver actions will affect the circuit and Semian, which can make future calls fail faster.
A useful way to think about the co-operation between bulkheads and circuit breakers is through visualizing a failure scenario graphing capacity as a function of time. If an incident strikes that makes the server unresponsive with a 20s timeout on the client and you only have circuit breakers enabled--you will lose capacity until all workers have tripped their circuit breakers. The slope of this line will depend on the amount of traffic to the now unavailable service. If the slope is steep (i.e. high traffic), you'll lose capacity quicker. The higher the client driver timeout, the longer you'll lose capacity for. In the example below we have the circuit breakers configured to open after 3 failures:
If we imagine the same scenario but with only bulkheads, configured to have tickets for 50% of workers at any given time, we'll see the following flat-lining scenario:
Circuit breakers have the nice property of re-gaining 100% capacity. Bulkheads have the desirable property of guaranteeing a minimum capacity. If we do addition of the two graphs, marrying bulkheads and circuit breakers, we have a plummy outcome:
This means that if the slope or client timeout is sufficiently low, bulkheads will provide little value and are likely not necessary.
Ok, great, we've got a way to fail fast with slow resources, how does that make my application more resilient?
Failing fast is only half the battle. It's up to you what you do with these errors, in the session example we handle it gracefully by signing people out and disabling all session related functionality till the data store is back online. However, not rescuing the exception and simply sending HTTP 500 back to the client faster will help with capacity loss.
It's important to understand that the exceptions raised by Semian Adapters inherit from the base class of the driver itself, meaning that if you do something like:
def posts Post.all rescue Mysql2::Error [] end
Exceptions raised by Semian's MySQL2 adapter will also get caught.
We do not recommend mindlessly sprinkling rescues all over the place. What you should do instead is writing decorators around secondary data stores (e.g. sessions) that provide resiliency for free. For example, if we stored the tags associated with products in a secondary data store it could look something like this:
# Resilient decorator for storing a Set in Redis.
class RedisSet
def initialize(key)
@key = key
end
def get
redis.smembers(@key)
rescue Redis::BaseConnectionError
[]
end
private
def redis
@redis ||= Redis.new
end
end
class Product
# This will simply return an empty array in the case of a Redis outage.
def tags
tags_set.get
end
private
def tags_set
@tags_set ||= RedisSet.new("product:tags:#{self.id}")
end
end
These decorators can be resiliency tested with Toxiproxy. You can provide fallbacks around your primary data store as well. In our case, we simply HTTP 500 in those cases unless it's cached because these pages aren't worth much without data from their primary data store.
With Semian::Instrumentable clients can monitor Semian internals. For example to instrument just events with statsd-instrument:
# `event` is `success`, `busy`, `circuit_open`, `state_change`, or `lru_hash_gc`.
# `resource` is the `Semian::Resource` object (or a `LRUHash` object for `lru_hash_gc`).
# `scope` is `connection` or `query` (others can be instrumented too from the adapter) (is nil for `lru_hash_gc`).
# `adapter` is the name of the adapter (mysql2, redis, ..) (is a payload hash for `lru_hash_gc`)
Semian.subscribe do |event, resource, scope, adapter|
case event
when :success, :busy, :circuit_open, :state_change
StatsD.increment("semian.#{event}", tags: {
resource: resource.name,
adapter: adapter,
type: scope,
})
else
StatsD.increment("semian.#{event}")
end
end
How does Semian work with containers? Semian uses SysV semaphores to coordinate access to a resource. The semaphore is only shared within the IPC. Unless you are running many workers inside every container, this leaves the bulkheading pattern effectively useless. We recommend sharing the IPC namespace between all containers on your host for the best ticket economy. If you are using Docker, this can be done with the --ipc flag.
Why isn't resource access shared across the entire cluster? This implies a coordination data store. Semian would have to be resilient to failures of this data store as well, and fall back to other primitives. While it's nice to have all workers have the same view of the world, this greatly increases the complexity of the implementation which is not favourable for resiliency code.
Why isn't the circuit breaker implemented as a host-wide mechanism? No good reason. Patches welcome!
Why is there no fallback mechanism in Semian? Read the Failing Gracefully section. In short, exceptions is exactly this. We did not want to put an extra level on abstraction on top of this. In the first internal implementation this was the case, but we later moved away from it.
Why does it not use normal Ruby semaphores? To work properly the access control needs to be performed across many workers. With MRI that means having multiple processes, not threads. Thus we need a primitive outside of the interpreter. For other Ruby implementations a driver that uses Ruby semaphores could be used (and would be accepted as a PR).
Why are there three semaphores in the semaphore sets for each resource? This has to do with being able to resize the number of tickets for a resource online.
Can I change the number of tickets freely? Yes, the logic for this isn't trivial but it works well.
What is the performance overhead of Semian? Extremely minimal in comparison to going to the network. Don't worry about it unless you're instrumenting non-IO.
Semian requires a Linux environment for Bulkheading. We provide a docker-compose file that runs MySQL, Redis, Toxiproxy and Ruby in containers. Use the steps below to work on Semian from a Mac OS environment.
# install docker-for-desktop $ brew cask install docker # install latest docker-compose $ brew install docker-compose # install visual-studio-code (optional) $ brew cask install visual-studio-code # clone Semian $ git clone https://github.com/Shopify/semian.git $ cd semian
reopen in container (first boot might take about a minute)See https://code.visualstudio.com/docs/remote/containers for more details
If you make any changes to .devcontainer/ you'd need to recreate the containers:
Rebuild Container from the command paletteRunning Tests:
$ bundle exec rake Run with SKIP_FLAKY_TESTS=true to skip flaky tests (CI runs all tests)To use the interactive debugger on vscode:
.env file (if it doesn't exist)DEBUG ENV variable (ex; DEBUG=true).vscode/ subdirectory, create a launch.json file, and include the following:{
"configurations": [
{
"type": "rdbg",
"name": "Attach to Ruby rdbg",
"request": "attach",
"debugPort": "12345",
}
]
}
_test.rb file (under test/), include the following snippet near the line of interest:require "debug" binding.break if ENV["DEBUG"]
Note: unless you are using an vscode extension such as Dev Container, do not use the built-in vscode breakpoints -- they will not work!
$ docker-compose -f .devcontainer/docker-compose.yml --profile test up -d
When the process indicates that it is waiting for the debugger connection, go to the Run and Debug tab, and execute the Attach to Ruby rdbg debugger
Use the vscode debugging tools (such as step in, step out, pause, resume) as normal
Test semian in containers:
$ docker-compose -f .devcontainer/docker-compose.yml up -d$ docker exec -it semian bashIf you make any changes to .devcontainer/ you'd need to recreate the containers:
$ docker-compose -f .devcontainer/docker-compose.yml up -d --force-recreateRun tests in containers:
$ docker-compose -f ./.devcontainer/docker-compose.yml --profile test run --rm test
Running Tests:
$ bundle exec rake Run with SKIP_FLAKY_TESTS=true to skip flaky tests (CI runs all tests)$ bundle exec rake test:parallel TEST_WORKERS=5 TEST_WORKER_NUM=1
Build a semian native extension with debug information.
$ bundle exec rake clean --trace $ export DEBUG=1 $ bundle exec rake build $ bundle install
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4