with message queue for keep data up to date
https://dontasktoask.com/
Well it's kind of consultation, not a yes no question. So I need advice and background about mechanism for keeping cache data up to date. I read something about MQ but couldn't understand it deeply
Let's imaging you're having your consultation right now. What would be the question you ask first? ——— For some background: https://doc.postsharp.net/caching-pubsub https://bluzelle.com/blog/things-you-should-know-about-database-caching https://redis.io/topics/pubsub https://redis.io/topics/client-side-caching https://redis.io/commands/client-caching (!!!! supported from version 6 !!!! ) https://github.com/redisson/redisson/blob/master/redisson/src/main/java/org/redisson/RedissonLocalCachedMap.java
wow that a hella high version
That is very new feature. An it is not exactly what you asked about, it's not about message queues, but it solves the same problem. The last link points to old school implementation with "pub/sub". Probably it's what you need.
what? I just meant that 6!!!! version is hella high
Ah, sorry, I thought you're the person who asked the original question))) Yep, that's very new feature. It adds out-of-the-box support on redis side for what people were doing for a long time on their own.
you didn't get the joke haha
Wow, first thanks a lot for those links. Second, now I can be more specific actually. Say I have a list of objects(say property object) stored in mssql DB. Now, the clients want to get this list. One client added to the DB a new property and 2 minutes later another client ask for the list. How can I garantee that the list will be updated. I understand that if I update the cache right after each insertion it will slow down the operation. What is the proper way to keep the cache value up to date?
Would the data be updated frequently?
Ok, forget about those links I gave. Except the second one - there you'll need to read about "Cache Aside" strategy. You should have been more specific from the very beginning. This is known as "XY problem". I guessed it wrong, you needed other thing; and links I gave are for more complex 1st-level + 2nd-level cache configuration. Message queue is NOT needed to solve the problem you described. https://xyproblem.info/ I completely agree with @ColonelMustang , you don't need complicated stuff. At least at the moment. Based on your questions, you're stepping into unknown waters for you, and believe me it's very complex to design, and even more complex to debug. Cache related issues is the worst nightmare. So I highly recommend to start with simplest cache aside, 2nd-level cache only, synchronous operations, and strong consistency. I highly recommend to not go with more complicated caching implementations before you get 1-2 years of in-field experience working with simple cache configurations. Designing cache solutions is always a game of trade-offs. You have to think of several things: - what is data access pattern (in your example it's write seldom/read often) - what are consistency requirements - what is SLA - what is production peak load, what throughput is needed - can your data be effectively sharded And usually based on those requirements you are choosing balance between consistency and performance. In most cases strong consistency is WAY MORE important. And this is exactly what I suggest for you to start with. The simplest case would be: - use only redis (2nd-level cache) - synchronously write entry into redis on each data modification AFTER successful DB write operation - set TTL on write only, do not update TTL on read And for the problem you described there's one simple solution - never write data directly into database omitting the cache. If you want to update properties (as in your example), create an API for it, or whatever other mechanism, which updates cache accordingly. Never make UPDATE queries directly to database. Otherwise, if you'll be facing cache inconsistency problems, and there's no way to avoid those. If you update value directly in database, all the clients reading that properties value will use stale value from cache until it expires.
Let me draw it, one min ...
Thank you very much for the very detailed response. Well, I do have an a CRUD endpoints for property. So I will avoid direct queries to DB. But the CRUD operation are for single property, in case of update a single property or adding a new one to DB(via endpoint of course) Redis will update the list itself?
There's no magic that will make "redis update the list itself". See where blue arrows come from on the diagram sketch
Do you have an example for an operation which illustrate the blue arrows?
@Service class AppPropertiesDataAccessService { private AppPropertiesRepository repo; private RedisClient redis; private Duration ttl; // ALWAYS read your properties with this method <T> T getAppProperty(String propName) { var propValue = redis.get("app-props:" + propName); if (propValue == null) { var valueFromDb = repo.getAppProp(propName); if (valueFromDb != null) { redis.setex("app-props:" + propName, valueFromDb, ttl); } } return propValue; } // ALWAYS update your properties with this method void setAppProperty(String propName, Object propValue) { repo.saveProperty(propName, propValue); redis.setex("app-props:" + propName, propValue, ttl); }
well this is the simple scenario of store and get single property, what will happened if getAllProperites() method will be added? , could you implement this as well please?
This is a sample code, I think you'll need to extend it based on your requirement
I really dont find any example for my requirement, although my case is very simple haha. For someone who already implemented redis cache it should be the simplest thing i think. Just maintain a list... do i need to add extra code to addProperty method? do a complicated code at the getAllProperties method?..
you can just wrap it inside a loop and return a list... but unless you know all the names of keys in redis, you cannot fetch the values anyway
As @ColonelMustang mentioned, it's just sample code snippet. I even not checked it for syntax. Its main purpose is to demonstrate order of actions on db and cache you need for write and read operations. What's your use-case and what are data access flows? You're saying that you need to retrieve ALL the list of application settings? But why? And more important - how often do you need it, do you really need to cache the full list?
Обсуждают сегодня