- I've directly inspected calls for pro/Max plans and as of today they have 1hr cache expiries. This has definitely degraded to 5 min in the past but that's the behavior today.
If you're paying API rates, you can choose 5m or 1hr yourself (and pay different rates).
Keeping a 1hr cache warm could still be useful, sure, but outside that, I don't see much use of this today.
- When using API keys the default is still 5 minutes. Setting 1 hour for the entire session is actually a lot more wasteful considering the 2x rate that it comes with
- Is the 5-minute expiration correct? I thought it was more like ~1 hour.
- You can set the retention globally (whole session) to 1 hour which will actually make it more expensive. The default is 5 minutes.
*UPD:* actually it appears the default is authentication-dependent. API key gets 5 minutes, subscriptions - 1 hour.
- Feels like will be shut down real quick
- Yes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.
- Can you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?
- Yes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.
- Why? Seems fine if people want to burn more cache reads.
- Because it makes Anthropic earn less money
- > Because it makes Anthropic burn more money
Fixed that for you.
- "Earn"?
Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..
- Earn is synonymous with profit, and this hurts their profits. Introducing latency can benefit much more people and actually make usage more efficient. Reducing latency can sieze up a functioning system.
Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.
- Glad this exists. It will force anthropic to fix their flawed cache mechanism.
- How should they fix it?
- There are a lot of workflows where cache is likely to be consumed exactly once (e.g. anything chat-like where a single thread is extended; if a prefix cache hit is found, and a set of new longer prefixes is inserted in the cache, it is unlikely that the original shorter prefix will be consumed in later turns). I could see allowing each session key a small number (1?) of 24-hour cache entries, where inserting a new cache entry (perhaps the maximum-length prefix, perhaps an API-tagged specified-length prefix) consumes that slot and either boots the rest of the items from cache, or demotes them to 5 minutes or something. Basically: workflow awareness, not uniform handling.
- Make the cache TTL more adaptive. Have more tiers than just 5m and 1h. Long-running subagents become increasingly ubiquitous. There's no reason for Anthropic to not do a better job for this use case.
- Don't we pay for cache input though?
- You can send and receive 1 byte and refresh the cache. Keeping the cache alive isn't free, but it's close.
- The idea is this keeps it in cache so you don’t have to pay to re-input.
- This is a good idea if it could manage it in an acceptable way.
For example, there might be something I intended to complete in one sitting, but took two sittings in the same day unexpectedly. Maybe it could just be a few cache delays per day or something, tagged in advance somehow.
- [dead]
29 points by s0ck_r4w 3 hours ago | 23 comments