Quantcast
Channel: Geo program design suggestions using Redis - Stack Overflow
Viewing all articles
Browse latest Browse all 6

Answer by systemjack for Geo program design suggestions using Redis

$
0
0

One idea I take away from the description is that you'll know the current location of a given "robot" and you'd like to find other mobile users near it in real time, but you'd also like some bit of historical location information.

I like to think of redis as exposing raw building blocks for making a higher level database. With that in mind you realize you need to build and maintain your own higher level database features like indexes, etc.

Since this type of data will primarily be accessed when you have a specific robot in mind, I'd recommend storing the location history and metadata for a bot in a key based on the bot's unique identifier and not its location.

Then maintain it's relative location (or any other grouping) to others by managing its ID in sets or hashes that group bots in a given location. You can use multiple sets or nested data structures for a sort of level of detail capability.

Keep your data integrity by updating the bot record and the location info as part of a redis transaction. Use pipelining for efficiency.

You don't need to use expire for old posts as you can manage your database size by limiting the count of historical entries in the bot's main record. When you go to update a bot, just do some kind of cleanup operation on it when it gets over a certain length (llen, slen, hlen, etc.) to give you a predictable/adjustable aggregate data size.

If there's any hope what you're doing might become production I highly recommend considering partitioning out of the gate. Any level of success will require it so might as well do it up front. Trust me. For this case I'd partition functionally (location vs. robot state...different databases on different replication groups) as well as by key (hash or whatever...to break down your 500M into reasonable chunks).

Partitioning makes transactions tough but for your use case I doubt that's a deal breaker. Using redis messaging in conjunction with transactions can allow you to keep your integrity by executing various updates programatically.

Finally, I'd consider something besides redis (elasticache in your case I'm assuming). In the spectrum of support for concurrency and ability to do complex queries, redis is great for the former. For that reason it's perfect for keeping track of sessions or similar state.

You'll require a lot of concurrency but it's mostly appending and not updates. So not like an evolving state machine. And you need at least some ability to search.

If you need to relate objects to each other (queries), be able to support analytics, etc., with 500M users you'd be able to afford a big redshift cluster, dynamo or similar. Could put kinesis in front to help with concurrency by batching small messages up for bulk loading. Both redshift and dynamo benefit from special loading paths from kinesis.

Definitely want to stay away from RDS, but there's other options that would be simpler to implement and would help you avoid that inevitable day when you have to iterate your redis cluster (for which you'd use scan obviously).

(Old post I know but interesting question and the answer applies to lots of situations.)


Viewing all articles
Browse latest Browse all 6

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>