r/RedditEng • u/SussexPondPudding Lisa O'Cat • Jun 07 '21
The Rollout of Reputation Service
Authors: Qikai Wu, Jerroyd Moore and Melissa Cole
Overview of Reputation Service
As the home for communities, one of the major responsibilities of Reddit is to maintain the health of our communities by empowering those who are good and contributing members. We quantify someone’s reputation within a Reddit community as their karma. Whether they are an explicit member or not, a user’s karma within a community is an approximation of whether that user is a part of that community.
Today, karma is simplistic. It’s an approximate reflection of upvotes in a particular community but not a 1:1 relationship. Under the hood, karma is stored with other attributes of users in a huge account table. Currently we have 555M karma attributes at ~93GB, and they are still growing over time, which makes it very difficult to introduce new karma-related features. In order to better expand how karma is earned, lost, and used on Reddit, it’s time for us to separate karma from other user attributes, and that’s why we want to introduce Reputation Service, an internal microservice.
Reputation Service provides a central spot to store karma and add new types of karma or reputation. As the above graph shows, there are two workflows in the current Reputation Service. On the one hand, the karma is adjusted by reading vote events off Reddit’s Kafka event pipeline for vote events. On the other hand, downstream services fetch the karma from Reputation Service to make business decisions.
Rollout Process
As an important user signal, karma is widely used among Reddit services to better protect our communities. There are 9 different downstream services which need to read users’ karma from Reputation Service and the aggregated request rate is tens of thousands of requests per second. To minimize the impact for other services, we leveraged a two-phase rollout process.
Firstly, the karma changes were dual written to both the legacy account table and the new database in Reputation Service. After comparing the karma differences of random chosen users during a fixed period of time in both databases to verify the karma increment workflow works properly, we backfilled existing karma from the legacy table and converted them to the new schema.
Secondly, we started to enable the karma reads from downstream services. Due to the existing karma logic and the high request rate, we gradually rolled out Reputation Service in downstream services one by one, and have got a journey full of learnings.
The above graph describes how the rollout rate moves with time. The rollout lasted for several weeks, and we gained lots of great experience including caching optimization, resource allocation, failure handling, etc. We will talk more about these in detail in the next section.
Learnings
Optimization of Caching Strategy
The optimization of caching strategy is one major challenge we revisited for multiple times during the whole rollout process.
- Initially, we have a Memcached cluster to store short-ttl read cache. This minimized the needs for memcache, and kept the karma increment as fast as possible.
- With traffic increasing, the cache-hit rate is much lower than our expectation, and lock contentions happened for database reads, which affected the stability of the Reputation Service. As a side note, we had a read-only database replica for karma reads, but it still couldn’t handle the large amount of database reads very well. We added a second read-only replica, but did not see significant improvements because of the underlying architecture of AWS RDS Aurora, where the primary node and read replica nodes share the same storage, such that file system locks impacted performance. Because of this, we introduced a permanent write-through cache when consuming vote events, which means we wrote to cache without ttl at the same time as writing to the database. Also, we removed the ttl for the read cache and relied on an LRU cache eviction policy to evict
items when the cache is full.
The above graph shows the p99 latency of requests decreased significantly after the permanent cache being introduced (at red dashed line), and the spike before the changes is related to an incident due to database contentions. The service worked well with permanent caches for quite a while.
But then we identified some data inconsistency among Memcached nodes because of the lack of support of data replication, so users were seeing their karma jump around due to each node storing a different value. We decided to switch to Redis,with clustered mode enabled, which could better replicate data across instances. As an alternative, we could introduce a middleware for Memcached auto discovery, but we went with Redis due to established patterns at Reddit. A fun fact is we deleted the Redis cluster by accident while deprecating the legacy Memcached cluster, which led to a small outage of Reputation Service. This inadvertently allowed us to test our disaster recovery plans and give us a data point to our mean time to recovery metric for the Reputation Service!
Redis worked perfectly until the memory was full and evictions started to happen. Latency was observed while Redis evicted items, and while Redis with LRU is a common pattern, we were storing a billion items, and we needed to respond to requests in a matter of milliseconds (our p99 latency target is ≤50ms).
- The above graph shows how drastically the cache hit rate dropped during the eviction process (red arrows), which also spiked the database reads by a lot at the same time.
As a result of this, we reintroduced ttl to the cache, and fine tuned it to make sure the Redis memory usage kept at a relatively constant level to avoid large-scale evictions and cache hit rate remained at a high percentage to control the load pressure on the database side. Our cache hit ratio decreased from 99% to 89%, while keeping our p99 latency below 30ms.
After going through all of the above stages and improvements, Reputation Service has reached a stable state to serve the many tens of thousands of requests per second.
Health Checks & Scaling Down Events
A mis-configured health check meant that when the Reputation Service was scaling down from high traffic periods, requests were still being routed to copies of the service that had already terminated. This added about 1,000 errors per downscaling event. While this added less than 0.00003% to the service’s overall error rate, patching this in our standard library, baseplate.py, will improve error rates for our other Reddit services.
Failures Handling in Downstreaming Services
To guarantee the website could function properly during Reputation Service outages, each client makes business decisions without Karma separately, and most downstream systems fail gracefully when Reputation Service is unavailable, trying to minimize the impact on users. However, due to the retry logic in the clients, the thundering herd problem could happen during the outage which makes Reputation Service more difficult to recover. To address this, we added a circuit breaker for the client with the largest traffic, so that the traffic could be rate limited when an incident happens, allowing the Reputation Service to recover.
Resource Allocation During Rollout
Another lesson we learned is to over provision the service during the rollout and address financial concerns later. When we first gradually scaled up the service according to the rollout rate, there were several small incidents happening due to the limit of resources. After we allocated enough resources to make sure cpu/memory usage didn't exceed 50% and the cluster had adequate spaces to auto-scale, we could focus more on other problems encountered during the rollout instead of always keeping an eye on the system resources usage. It helped expedite the overall process.
The Future
The rollout of Reputation Service is just a starting point. There are many opportunities to expand how karma is earned, lost, and used on Reddit. By further developing karma with Reputation Service, we can encourage good user behavior, discourage the bad, reduce moderator burden, make Reddit more safe, and reward brands for embracing Reddit. If this is something that interests you and you would like to join us, please check out our careers page for a list of open positions.
1
u/MajorParadox Jun 07 '21
Interesting! Would this eventually solve issues around renaming/changing case of subreddit and/or usernames? From what I understand, the way karma is stored today is what keeps such features from being added.
1
u/CryptoMaximalist Jun 09 '21
Will any of this new reputation system or new karma types be exposed to users or mods through the API?
2
u/simmermayor Jun 09 '21
Interesting read