Cold hard cache.

Time to crawl the interwebs. I’m looking for something relatively small and lightweight, a binary blob cache that I can drop into place, import a module in python and have relatively easy access to.

The keys are likely to be large, the blobs may be several MB. I don’t care a great-deal about persistence.

What I’m looking to achieve is something like ‘distcc’ for asset conversion. The backend doesn’t need to know that, it’s just going to get semi-opaque key values that ultimately serve to compartmentalize hash spaces.

In more detail tho, I’m going to be doing something like:

# Check if we have already built it locally.
if locally_cached(params) then  # include timestamp compare
  return
endif

# Do a manual local conversion if we have local changes to the
# asset or any of the dependencies, or if the the tool is different
# than official version (e.g when tools developers are testing)

if file_size <= LOCAL_ONLY_SIZE or
      have_local_modifications(dependencies) or
      tool_version != active_version[tool] or
  convert(params)
  return
endif

# Send a request to the server for it's copy of the data, and then
# continue looking at local assets.

key = tool + tool_ver + filename + file_timestamp + file_size
queued = queue_server_query(remote_key, callback (response) {
  if response.answer is 'yes' and response.data then
    save_data(params, data)
    mark_resolved(params)
  else
    # data isn't on the asset server; do the conversion ourselves
    # and then forward the result so the next person doesn't have
    # to do a conversion.
    convert(params)
    send_asset(key, params)
  endif
)

# It's possible that we might not be able to queue the request if
# we can't reach the asset server, or maybe it doesn't allow you
# to pull resources over the vpn or something. In anyway, we just
# go ahead and work locally.

if not queued then
  convert(params)
endif

Additionally, I might Tell the server which assets we’re going to convert and let the server act as a kind of scheduler that provides *hints* as to which conversions should be preferred by different clients.

Alternatively: have the server send a random seed in the response and have each client use it to randomly sort the list of non-cached assets it will convert, and have each client pick one asset to convert while, at the same time, asking the server if it has copies of the others yet until the pending queue is empty.