What should we build Cache? / Infopuls Ukraine Blog

Not a few good articles have been written on the topic "What, How, and Where to Cache." So why once again procrastinate this topic? And because the topic is quite important, and many, until they encounter specific problems, do not consider it necessary to deal with it. So the audience that I am counting on is those who, by the time the existing articles were published, were not interested in them, but now there is interest, and they will not pass by.

I will try to briefly highlight the main points in organizing caching, after which I will consider the innovations of the .Net Framework 4.0, which should simplify the life of developers (we will talk about the In-memory cache outside the ASP.NET infrastructure).

Introduction

Often, when it comes to performance, it is quite difficult to do without the use of caching techniques. But before we can effectively apply it, we need to answer the following questions:

• What to cache : which data should be stored in the cache;
• How to cache : what is the maximum amount we can allocate for the cache to work; whether the maximum permissible time is established during which the element will not be considered obsolete; whether the relevance of our elements in the cache will depend on some external factors or will there be dependencies between the elements themselves inside the cache; whether the order in which we will remove items from the cache when the memory limit is reached will be important; etc…
• Where to cache : what will act as a cache - in devices, it can be a hardware cache, in programs, as a rule, we resort to a ready-made or self-written cache implementation that can satisfy the requirements in the "how" question;

Interestingly, the answers must be given in the exact order in which these questions are listed. For it is difficult to say “where to cache”, not understanding, “what and how” we cache. Still, it is highly advisable to take care of caching in the early stages of system design. Since contrary to popular belief that "Cache can always be added at the last moment," this is often not the case. Without thinking about caching at the initial stage, then adding and testing it can be extremely difficult. Let's try to find the answers to the questions asked above, but I want to clarify right away that most of the thoughts below will be given regarding the general-purpose cache inside the .Net application stored in RAM, i.e. it is not a processor cache or a browser cache. In addition, under one article,

What? How? Where?

When we decide that we will cache, we need to understand that the cache is not free and will not be useful for all data types. Regardless of the chosen cache strategy and cache implementation, one way or another, we will have problems with data obsolescence and the amount of memory they occupy. And since caching adds complexity during the creation / testing / maintenance of our application, we definitely do not need to cache everything in a row. You need to take a balanced approach to choosing those places where adding a cache, in addition to problems, will also bring benefits.

Do not cache data that is only useful in real time. For example, currency quotes in the trading system, outdated by 30 seconds, can significantly affect the correctness of decisions. But if we have a summary of sales statistics for the last week in our system on the home page, then this data is perfect for caching.

Also, it makes no sense to cache what you can get from a data source so quickly or calculate on the fly. But if the data source is far and very slow, and the specificity of the data allows them to be used with some delay, then these are good candidates for caching.

As another example, let's look at data that is calculated on the fly, it requires a lot of processor time, but the result is quite large in volume. When we try to cache this data, we can quickly fill up all available memory, placing only a few results in it. In these conditions, it will be difficult to achieve effective cache operation, as it is likely that just a few new elements will lead to overwriting of the just calculated values ​​and the percentage of successful hits (finding the desired element in the cache) will be extremely low. In this case, you should rather think about speeding up the calculation algorithm, rather than caching the result. When choosing the right candidates to save to the cache, always think about the effectiveness of the cache. Those. we must strive to ensure that the data we select

Thinking about how to properly store our data in the cache, we should pay attention to the following points:

• Timely data obsolescence
• Correct order of deleting items when reaching the maximum available memory
• Data coherence (if the cache is distributed, then the same object may differ in different cache instances and thereby lead to negative consequences)

Some problems related to the question “how to store” can be so complex that separate projects are created to solve them and specialists with relevant experience are distinguished. I hope this does not apply to your project, because, as I said, this article will not affect the depths associated with caching issues.

So, after receiving answers to the questions “What” and “How”, it may turn out that our answer to the question where will the Dictionary becreated in our application. If so, then we are very lucky. But, as a rule, everything is a little more complicated and we still have to write a full implementation of the cache or choose one of the ready-made solutions.
Note: there is no consensus on whether a Dictionary-based implementation will be considered a cache or not. Personally, I prefer to consider this a special case, which apart "stands aside." At the same time, I even came across a term describing such a cache as “static”, i.e. a cache in which data is not deleted and considered infinitely relevant.

Handwritten cache

I won’t tell you how to write my cache. On the contrary, I will try to protect you from the false impression that this is easy and simple. Unless the Dictionary-like implementation perfectly covers our needs, writing a full cache is quite a challenge.

The first difficulty that comes to mind is working in a multi-threaded environment. After all, if we use the cache, then for sure the system is not small and it will be extremely inefficient to work in one thread. Those. all data write / read / invalidate operations must be thread safe. Without extensive experience working with threads, we are guaranteed deadlocks or slow operation due to the not optimally selected approach for synchronizing threads.

If you think about how data obsolescence occurs, then a number of far from the simplest scenarios that our cache should support can come to mind. Taking the Cartesian product of all possible options, we get that many states in which our system can be. This will become a sufficient basis to make the task of debugging and testing the self-written cache simply unbearable.

Many caching examples that can be found on the Internet use the Weak references mechanism. An uncontrollable desire to apply them in their implementation may appear. But without sufficient experience in the relevant field, we do not slightly increase the chances of getting a code that is not only that most of the team will not understand, so it is unlikely to work after the first 5 times of rewriting.
I think that I could continue this list for a long time, but I hope that even the reasons already listed are enough for you to lose the desire to test yourself for strength and perseverance. If not, then I can only wish "Strength, mind and patience (C)."

Now, realizing that your cache is far from easy, I propose moving on to the final part of the article, which will tell you that there is already something ready in the .NET Framework to simplify our lives.

Life before .Net Framework 4.0

Caching has always been an integral part of ASP.NET web applications and the .Net Framework has provided excellent tools for ASP.NET applications. Therefore, historically, all classes for working with the cache were located in the System.Web assembly. When the cache was required outside the web (for example, the Windows service), many developers sacrificed the beauty of their solutions and added a link to the System.Web assembly. This allowed us to take advantage of the cache, but it attracted a huge amount of unnecessary code. This problem remained unresolved for a long time, but fortunately, in the .NET Framework 4.0 they nevertheless returned to it. As a result, we got the System.Runtime.Caching namespace , in which, among other things, there is an abstract ObjectCache class and its only implementation isThe MemoryCache . It is with them that I would like to introduce you.

Objectcache

ObjectCache is an abstract class that enables us to standardize approaches when working with various cache implementations. Having the same interface (API) for working with the cache, we do not have to study in detail each new cache implementation. Indeed, from the point of view of the user, implementations will have to look the same and behave according to well-known expectations expressed in the form of APIs of this class. The main methods, properties and their purpose are given below.

Properties :

• DefaultCacheCapabilities - bit flags (enum, with the Flags attribute) that determine what capabilities a particular implementation provides (deleting an item at a specific time, supporting regions, having a callback mechanism, etc.)
• Name - name of the cache instance; in the case of using MemoryCache, it can be useful if we want to save data in isolated areas of memory and create more than one cache instance (this feature is called “regions” and is not implemented in MemoryCache)
• this - indexer to access items by key

Methods :

• Add (...) , AddOrGetExisting (...) , Set (...) - add data to the cache
• Get (...) , GetCacheItem (...) , GetValues ​​(...) - return data from the cache
• GetCount () - returns the current cache size
• Contains (...) - checks for the existence of an element by key
• Remove (...) - removes an element by key

I hope now, in general terms, it’s already clear how you can create your own cache implementation. But there are a couple of points that I would like to consider in more detail, namely, methods for adding data to the cache. I propose to do this, using the example of methods in an existing cache implementation - the MemoryCache class.

Memory cache

As the name implies, MemoryCache is an implementation that stores data in RAM. At the moment, this is the only class in the .Net Framework that inherits ObjectCache, but there are Nuget packages that offer other implementations (for example, you can use the SqlCache Nuget package to store data in the Sql server ). Below we will consider only those methods whose operation may not be immediately obvious. To demonstrate how the methods work, unit test listings written using xUnit will be shown .

AddOrGetExisting Method (...)

Adds an item only if the key has not yet been used, otherwise it ignores the new value and returns the existing value.

Add method (...)

It is a wrapper over AddOrGetExisting (...) and works almost identically, with the only difference being that it returns True if the item has been successfully added, and False if the key already exists (i.e., adding a value does not occur).

Set method (...)

Adds a new or replaces an existing item without checking existing keys. Those. unlike the Add and AddOrGetExisting methods, the passed value to the Set method will always appear in the cache.

Regions in MemoryCache

All methods for adding data to MemoryCache have overloads that accept the region parameter ( example1 , example2 and example3 ). But when we try to pass any non-NULL value to it, we get a NotSupportedException. Someone may say that this violates the Lisk substitution principle (so L id), but it is not. After all, before taking advantage of the possibility of regions, the client code must make sure that they are implemented in a specific implementation. This is done by checking the DefaultCacheCapabilities property for the presence of the corresponding bit flag (DefaultCacheCapabilities.CacheRegions), and it is just not set for MemoryCache.

CacheItemPolicy

To demonstrate all the methods of adding data, the simplest versions of overloads were selected that take the key, value and time until which the value will be considered relevant. But they all also have a version that accepts a parameter of type CacheItemPolicy. It is thanks to this parameter that we have quite rich capabilities for managing the lifetime of an element in the cache, which makes the implementation of MemoryCache extremely useful.

Most of the properties of this type look understandable, but in practice we will encounter many unexpected surprises. Strictly speaking, many of them will not be in the CacheItemPolicy class itself, but in the logic of the MemoryCache methods that accept this type. But since these types are often used together, I propose to consider them together too.

AbsoluteExpiration and SlidingExpiration Properties

From the name it is clear what these properties are responsible for. But curious people may ask the following question: “How will the cache behave if you simultaneously set values ​​for both properties”? Someone may suggest that AbsoluteExpiration has a higher priority and the object will be deleted at the time of AbsoluteExpiration, even if it is regularly requested from the cache (more often than SlidingExpiration). Someone on the contrary will assume that the value of SlidingExpiration will allow the object to survive AbsoluteExpiration. But the Microsoft developers considered that there was no truly correct answer and acted differently - they throw an ArgumentException at the stage of adding an element to the cache. Therefore, we can choose only one temporary (time-dependent) disability strategy for each element.

The second surprise awaits us if we decide to write tests for functionality that uses the cache. Surely, to speed up the test run, we want to set a fairly small value for SlidingExpiration (less than 1 second). In this case, our tests will behave unstably and will often fall. This is all due to the fact that to optimize the cache, at the time of reading the element (Get method and its derivatives), the new Expires value will be set only if it differs from the old one by at least one second. I could not find confirmation of this in the documentation, but you can verify this by decompiling the MemoryCache class and studying the UpdateSlidingExp (...) method of the internal class MemoryCacheEntry.

Priority Property

When I saw this property, I expected it to have a low / medium / high value to set the order in which items should be removed from the cache when the maximum volume was reached. But it can only have 2 values: CacheItemPriority.Default or CacheItemPriority.NotRemovable.
MSDN states that setting CacheItemPriority.NotRemovable will cause the item to never be removed from the cache. Personally, I took this fact as the fact that, adding all the elements with such a priority, we get a Dictionary-like implementation, but this is far from the case. Elements will still be deleted if they “fade” (AbsoluteExpiration occurs or SlidingExpiration passes), but unlike the default mode, they will not be deleted from memory when the limit on the amount of occupied memory is reached. By the way, the limit can be set through the CacheMemoryLimit property (in bytes) or through the PhysicalMemoryLimit property (percent of the total memory in the system).

RemovedCallback and UpdateCallback

Another surprise. Both properties are accepted by delegates, which will be called, both in case of updating, and in case of removing an item from the cache.

If you think about it, an update is essentially a delete operation, immediately followed by the operation of adding a new value. This explains why RemovedCallback fires when an item is updated. And the fact that UpdateCallback is triggered upon deletion is simply a fact from MSDN.

The difference in properties is that RemovedCallback should be called after, and UpdateCallback - before the actual removal of the item from the cache. The delegates stored in these properties accept a parameter that contains a link to the cache, a link to the item to be deleted, and the reason for the item to be deleted.

Another gift is stored in the implementation of MemoryCache. This class has a slightly strange validation logic for the parameter passed to CacheItemPolicy. First, it checks that both delegates are not set at the same time, otherwise we will get an ArgumentException at the stage of adding an element to the cache.



And all would be fine if for the UpdateCallback property to work correctly it would be enough to make sure that there is no value in the RemovedCallback property. But in fact, we always get an ArgumentException at the stage of adding an element when setting a non-empty value in UpdateCallback.


As a result, the only valid property for setting the delegate signaling changes in the cache is RemovedCallback (valid only for the implementation of MemoryCache).

ChangeMonitors Property

This property can store a collection of objects of type ChangeMonitor, each of which can add conditions under which the item will be removed from the cache.

Besides the fact that we can create our own implementations of the abstract ChangeMonitor class, the following classes exist in the .Net Framework:

• CacheEntryChangeMonitor - monitors changes to other elements in the same cache instance
• FileChangeMonitor (HostFileChangeMonitor) - monitors file and folder changes in the file system
• SqlChangeMonitor - monitors changes in the database (rather slow and rarely applicable in practice).

It is important to remember that this property in the CacheItemPolicy object must be set before adding the item to the cache. Setting or changing it for an already added item has no effect.

Conclusion

Despite a number of not very obvious features in the implementation of MemoryCache, this class is still an extremely useful tool in the developer's arsenal, as it allows you to get a thread-safe "concrete" working cache implementation with good features for managing element invalidation policies. I am sure that an attempt to write your own analogue will be time consuming and certainly will not be as effective.