Hystrix Circuit Breaker and Execution Timeouts

Hystrix is a latency and fault tolerance tool developed by Netflix OSS. Included in the library is an implementation of the very useful Circuit Breaker pattern that can be easily folded into Java applications.

Since Hystrix provides much more than just circuit breaker functionality, it can be easy to overlook the impact that execution timeouts can have on @HystrixCommand methods.

Let's take the following example:

public class Cashbox
{
  @Autowired
  private DepositService service;
  
  @HystrixCommand(
      fallbackMethod = "depositLater"
      commandProperties = { 
        @HystrixProperty(name="circuitBreaker.errorThresholdPercentage", value="1"),
        @HystrixProperty(name="circuitBreaker.requestVolumeThreshold", value="1"),
        @HystrixProperty(name="circuitBreaker.sleepWindowInMilliseconds", value="10000"),
      }
  )
  public boolean deposit(BigDecimal amount)
  {
    return service.makeDeposit(amount);
  }

  public boolean depositLater(BigDecimal amount)
  {
    return service.queueForDeposit(amount);
  }
}

While the circuit is closed, a call to deposit will attempt the make a deposit via a backing service. As you might expect, should the service throw an exception, the circuit will open. In this case the depositLater method will be called and an attempt made to have the service queue the deposit for later. Since the method with the @HystrixCommand annotation does not return an AsyncResult, all of this execution will be done synchronously.

So, you may think that you can put on your single-threaded hat and ignore any potential concurrency problems. Think again! All HystrixCommands are executed on their own thread, despite the synchronous nature of the call.

What may not be immediately obvious here is that if the service's makeDeposit method takes longer than 1 second, the depositLater method will be called as well. This is because all @HystrixCommand methods have an execution timeout of 1000 millis enabled by default.

To disable this execution timeout, you can add the following Hystrix configuration property:

@HystrixProperty(name="execution.timeout.enabled", value="false")

Hystrix is a fantastic open source contribution and provides a wealth of functionality for handling latency and fault-tolerance. When adding a circuit breaker to a method that mutates a shared resource (e.g. account, file, record), don't ignore the complexities of concurrency.