Low memory, and the rainy day fund

Just like any other piece of software, Orchids consumes memory.  Monitoring signatures can involve using a growing amount of memory over time.

You can get some information on memory consumption by running run Orchids with the -v1 flag.  In that case, Orchids will tell you what signatures are costly in terms of number of threads created.  A signature that creates n threads (at most) after reading n events can be expected to also use memory proportional to the number of events.

However, machines have finite memory.  If Orchids just ran its algorithm blindly, it would eventually use up all the memory the machine has.  An attacker, knowing that, could just feed events to the machine so that Orchids would consume as much memory as possible, leading to a so-called denial-of-service attack.

Orchids has two ways to mitigate that.

  • First, Orchids runs within a memory limit.  That is set by the MaxMemorySize directive in orchids.conf.  Orchids will never consume more that the memory you set this way.
  • Second, Orchids uses a rainy day fund.  The problem with a mere memory limit is that, once you have used up all the memory allowed by the MaxMemorySize directive, there is simply no memory left to do any work, and Orchids would in principle abort.
    The idea of the rainy day fund is as follows.  On start up, Orchids preallocates a dummy block of memory.  The size of that block of memory is specified by the RainyDayFund directive in orchids.conf.
    Whenever Orchids reaches its memory limit, it will enter a so-called low memory mode.  In that mode, Orchids will starting eating up some memory from the rainy day fund, while measuring how much memory each thread will consume.
    If Orchids ever consumes more than a fixed amount of the rainy day fund (fixed to 75% for now), then it enters critical memory mode.  In that mode, Orchids deallocates the index tables that help it execute fast, and kills enough threads to recuperate enough memory.  Once that is done, it reallocates as much of the rainy day fund as it can, attempting to go back to normal mode.

Critical memory mode does a lot of nasty things.  By deallocating index tables, Orchids is slowed down.  By killing threads,  some attacks will go undetected.  However, this is better than just letting Orchids abort—and no longer detect anything.

The strategy that Orchids uses to kill threads in critical memory mode is experimental, and subject to change.  The risk is that an attacker might know that strategy, and exploit it so as to hide some attacks to Orchids.  Keeping the strategy secret makes no sense, if only because the Orchids source is public.

For now, the strategy is as follows: kill the most recent threads.  That is it.

The important point is that that strategy does not kill the threads that monitor long-standing attack candidates.  The opposite strategy, killing the oldest threads, would be all too easy to exploit for attackers: start some activity that Orchids can monitor, then drown Orchids into a flow of events, so as to make sure to induce a critical memory situation: if Orchids killed the oldest threads, the attacker could be sure he could proceed undetected.