get all keys set in memcached
GetMemcachedKeyTelnetGet Problem Overview
How can I get all the keys set in my memcached instance(s)?
I tried googling, but didn't find much except that PHP
supports a getAllKeys
method, which means it is actually possible to do this somehow. How can I get the same within a telnet session?
I have tried out all the retrieval related options mentioned in memcached cheat sheet and Memcached telnet command summary, but none of them work and I am at a loss to find the correct way to do this.
Note: I am currently doing this in development, so it can be assumed that there will be no issues due to new keys being set or other such race conditions happening, and the number of keys will also be limited.
Get Solutions
Solution 1 - Get
Found a way, thanks to the link here (with the original google group discussion here)
First, Telnet
to your server:
telnet 127.0.0.1 11211
Next, list the items to get the slab ids:
stats items STAT items:3:number 1 STAT items:3:age 498 STAT items:22:number 1 STAT items:22:age 498 END
The first number after ‘items’ is the slab id. Request a cache dump for each slab id, with a limit for the max number of keys to dump:
stats cachedump 3 100 ITEM views.decorators.cache.cache_header..cc7d9 [6 b; 1256056128 s] ENDstats cachedump 22 100 ITEM views.decorators.cache.cache_page..8427e [7736 b; 1256056128 s] END
Solution 2 - Get
memdump
There is a memcdump
(sometimes memdump
) command for that (part of libmemcached-tools
), e.g.:
memcdump --servers=localhost
which will return all the keys.
memcached-tool
In the recent version of memcached
there is also memcached-tool
command, e.g.
memcached-tool localhost:11211 dump | less
which dumps all keys and values.
See also:
Solution 3 - Get
Base on @mu 無 answer here. I've written a cache dump script.
The script dumps all the content of a memcached server. It's tested with Ubuntu 12.04 and a localhost memcached, so your milage may vary.
#!/usr/bin/env bash
echo 'stats items' \
| nc localhost 11211 \
| grep -oe ':[0-9]*:' \
| grep -oe '[0-9]*' \
| sort \
| uniq \
| xargs -L1 -I{} bash -c 'echo "stats cachedump {} 1000" | nc localhost 11211'
What it does, it goes through all the cache slabs and print 1000 entries of each.
Please be aware of certain limits of this script i.e. it may not scale for a 5GB cache server for example. But it's useful for debugging purposes on a local machine.
Solution 4 - Get
If you have PHP & PHP-memcached installed, you can run
$ php -r '$c = new Memcached(); $c->addServer("localhost", 11211); var_dump( $c->getAllKeys() );'
Solution 5 - Get
Bash
To get list of keys in Bash, follow the these steps.
First, define the following wrapper function to make it simple to use (copy and paste into shell):
function memcmd() {
exec {memcache}<>/dev/tcp/localhost/11211
printf "%s\n%s\n" "$*" quit >&${memcache}
cat <&${memcache}
}
Memcached 1.4.31 and above
You can use lru_crawler metadump all
command to dump (most of) the metadata for (all of) the items in the cache.
> As opposed to cachedump
, it does not cause severe performance problems and has no limits on the amount of keys that can be dumped.
Example command by using the previously defined function:
memcmd lru_crawler metadump all
See: ReleaseNotes1431.
Memcached 1.4.30 and below
Get list of slabs by using items statistics command, e.g.:
memcmd stats items
For each slub class, you can get list of items by specifying slub id along with limit number (0
- unlimited):
memcmd stats cachedump 1 0
memcmd stats cachedump 2 0
memcmd stats cachedump 3 0
memcmd stats cachedump 4 0
...
Note: You need to do this for each memcached server.
To list all the keys from all stubs, here is the one-liner (per one server):
for id in $(memcmd stats items | grep -o ":[0-9]\+:" | tr -d : | sort -nu); do
memcmd stats cachedump $id 0
done
Note: The above command could cause severe performance problems while accessing the items, so it's not advised to run on live.
Notes:
> stats cachedump
only dumps the HOT_LRU
(IIRC?), which is managed by a background thread as activity happens. This means under a new enough version which the 2Q algo enabled, you'll get snapshot views of what's in just one of the LRU's.
>
> If you want to view everything, lru_crawler metadump 1
(or lru_crawler metadump all
) is the new mostly-officially-supported method that will asynchronously dump as many keys as you want. you'll get them out of order but it hits all LRU's, and unless you're deleting/replacing items multiple runs should yield the same results.
Source: GH-405.
Related:
-
Writing a Redis client in pure bash (it's Redis, but very similar approach)
-
Check other available commands at https://memcached.org/wiki
-
Check out the
protocol.txt
docs file.
Solution 6 - Get
The easiest way is to use python-memcached-stats package, https://github.com/abstatic/python-memcached-stats
The keys() method should get you going.
Example -
from memcached_stats import MemcachedStats
mem = MemcachedStats()
mem.keys()
['key-1', 'key-2', 'key-3', ... ]
Solution 7 - Get
I was using Java's spyMemcached, and used this code. It is based on Anshul Goyal's answer
@Autowired
@Qualifier("initMemcachedClient")
private MemcachedClient memcachedClient;
public List<String> getCachedKeys(){
Set<Integer> slabIds = new HashSet<>();
Map<SocketAddress, Map<String, String>> stats;
List<String> keyNames = new ArrayList<>();
// Gets all the slab IDs
stats = memcachedClient.getStats("items");
stats.forEach((socketAddress, value) -> {
System.out.println("Socket address: "+socketAddress.toString());
value.forEach((propertyName, propertyValue) -> {
slabIds.add(Integer.parseInt(propertyName.split(":")[1]));
});
});
// Gets all keys in each slab ID and adds in List keyNames
slabIds.forEach(slabId -> {
Map<SocketAddress, Map<String, String>> keyStats = memcachedClient.getStats("cachedump "+slabId+" 0");
keyStats.forEach((socketAddress, value) -> {
value.forEach((propertyName, propertyValue) -> {
keyNames.add(propertyName);
});
});
});
System.out.println("number of keys: "+keyNames.size());
return keyNames;
}