Month: November 2021

How to tailor a 401(k) plan you and your employees will love

Designing a 401(k) plan is like building a house. It takes care, attention, and the help of a few skilled professionals to create a plan that works for both you and your employees. In fact, thoughtful plan design can help motivate even reluctant retirement savers to start investing for their future.

As you embark on the 401(k) design process, there are many options to consider. In this article, we’ll take you through the most important choices so you can make well-informed decisions. Since certain choices may not be available on the various pricing models of any given provider, make sure you understand your options and the trade-offs you’re making.

Let’s get started!

401(k) eligibility

When would you like employees to be eligible to participate in the plan? You can opt to have employees become eligible:

Immediately – as soon as they begin working for your companyAfter a specific length of service – for example, a period of hours, months, or years of service

It’s also customary to have an age requirement (for example, employees must be 18 years or older to participate in the plan). Plus, you may want to add an “employee class exclusion” to prevent part-time, seasonal, or temporary employees from participating in the plan.

Once employees become eligible, they can immediately enroll – or, you can restrict enrollment to a monthly, quarterly, or semi-annual basis. If you have immediate 401(k) eligibility and enrollment, in theory, more employees could participate in the plan. However, if your company has a higher rate of turnover, you may want to consider adding service length requirements to alleviate the unnecessary administrative burden of having to maintain many small accounts of employees who are no longer with your organization.

Enrollment

Enrollment is another important feature to consider as you structure your plan. You may simply allow employees to enroll on their own, or you can add an automatic enrollment feature. Automatic enrollment (otherwise known as auto-enrollment) allows employers to automatically deduct elective deferrals from employees’ wages unless they elect not to contribute.

With automatic enrollment, all employees are enrolled in the plan at a specific contribution rate when they become eligible to participate in the plan. Employees have the freedom to opt out and change their contribution rate and investments at any time.

As you can imagine, automatic enrollment can have a significant impact on plan participation. In fact, according to research by The Pew Charitable Trusts, automatic enrollment 401(k) plans have participation rates greater than 90%! That’s in stark contrast to the roughly 50% participation rate for plans in which employees must actively opt in.

If you decide to elect automatic enrollment, consider your default contribution rate carefully. A 3% default contribution rate is still the most popular; however, more employers are electing higher default rates because research shows that opt-out rates don’t appreciably change even if the default rate is increased. Many financial experts recommend a savings rate of at least 10%, so using a higher automatic enrollment default rate gets employees even more of a head start.

Compensation

You’re permitted to exclude certain types of compensation for plan purposes, including compensation earned prior to plan entry and fringe benefits for purposes of compliance testing and allocating employer contributions. You may choose to define your compensation as:

W2 (box 1 wages) plus deferrals – Total taxable wages, tips, prizes, and other compensation3401(a) wages – All wages taken into account for federal tax withholding purposes, plus the required additions to W-2 wages listed aboveSection 415 Safe Harbor – All compensation received from the employer which is includible in gross income

Employer contributions

Want to encourage employees to enroll in the plan? Free money is a great place to start! That’s why more employers are offering profit sharing or matching contributions.

In fact, EBRI and Greenwald & Associates’ found that nearly 73% of workers said they were likely to save for retirement if their contributions were matched by their employer.

Some of the more common employer contributions are:

Safe harbor contributions – With the added bonus of being able to avoid certain time-consuming compliance tests, safe harbor contributions often follow one of these formulas:>Basic safe harbor match—Employer matches 100% of employee contributions, up to 3% of their compensation, plus 50% of the next 2% of their compensation.Enhanced safe harbor match—Employer matches 100% of employee contributions, up to 4% of their compensation.Non-elective contribution—Employer contributes 3% of each employee’s compensation, regardless of whether they make their own contributions.Discretionary matching contributions – You decide what percentage of employee 401(k) deferrals to match and the maximum percentage of pay to match. For example, you could elect to match 50% of contributions on up to 6% of compensation. One advantage of having a discretionary matching contribution is that you retain the flexibility to adjust the matching rate as your business needs change.Non-elective contributions – Each pay period, you have the option of contributing to your employees’ 401(k) accounts, regardless of whether they contribute. For example, you could make a profit sharing contribution (one type of non-elective contribution) at the end of the year as a percentage of employees’ salaries or as a lump-sum amount.

In addition to helping your employees build their retirement nest eggs, employer contributions are also tax deductible (up to 25% of total eligible compensation), so it may cost less than you think.  Plus, offering an employer contribution can play a key role in recruiting and retaining top employees. In fact, a Betterment for Business study found that more than 45% of respondents considered a 401(k) match to be a factor when deciding whether to accept a job.

401(k) vesting

If you elect to make an employer contribution, you also need to decide on a vesting schedule (an employee’s own contributions are always 100% vested). Note that all employer contributions made as part of a safe harbor plan are immediately and 100% vested.

The three main vesting schedules are:

Immediate – Employees are immediately vested in (or own) 100% of employer contributions as soon as they receive them.Graded – Vesting takes place in a gradual manner. For example, a six-year graded schedule could have employees vest at a rate of 20% a year until they are fully vested.Cliff – The entire employer contribution becomes 100% vested all at once, after a specific period of time. For example, if you had a three-year cliff vesting schedule and an employee left after two years, they would not be able to take any of the employer contributions (only their own).

Like your eligibility and enrollment decisions, vesting can also have an impact on employee participation. Immediate vesting may give employees an added incentive to participate in the plan. On the other hand, a longer vesting schedule could encourage employees to remain at your company for a longer time.

Service counting method

If you decide to use length of service to determine your eligibility and vesting schedules, you must also decide how to measure it. Typically, you may use:

Elapsed time – Period of service as long as employee is employed at the end of periodActual hours – Actual hours worked. With this method, you’ll need to track and report employee hoursActual hours/equivalency – A formula that credits employees with set number of hours per pay period (for example, monthly = 190 hours)

401(k) withdrawals and loans

Naturally, there will be times when your employees need to withdraw money from their retirement accounts. Your plan design will have rules outlining the withdrawal parameters for:

TerminationIn-service withdrawals (at attainment of age 59 ½; rollovers at any time)HardshipsQualified Domestic Relations Orders (QDROs)Required Minimum Distributions (RMDs)

Plus, you’ll have to decide whether to allow participants to take 401(k) plan loans (and the maximum amount of the loan). While loans have the potential to derail employees’ retirement dreams, having a loan provision means employees can access their money if they need it and employees can pay themselves back plus interest. If employees are reluctant to participate because they’re afraid their savings will be “locked up,” then a loan provision can help alleviate that fear.

Investment options

When it comes to investment methodology, there are many strategies to consider. Your plan provider can help guide you through the choices and associated fees. For example, at Betterment, we believe that ETFs offer investors significant diversification and flexibility at a low cost. Plus, we offer ETFs in conjunction with personalized, unbiased advice to help today’s retirement savers pursue their goals.

Get help from the experts

Your 401(k) plan provider can walk you through your plan design choices and help you tailor a plan that works for your company and your employees. Once you’ve settled on your plan design, you will need to codify those features in the form of a formal plan document to govern your 401(k) plan. At Betterment, we draft the plan document for you and provide it to you for review and final approval.

Your business is likely to evolve—and your plan design can evolve, too. Drastic increase in profits? Consider adding an employer match or profit sharing contribution to share the wealth. Plan participation stagnating? Consider adding an automatic enrollment feature to get more employees involved. Employees concerned about access to their money in an uncertain world? Consider adding a 401(k) loan feature.

Need a little help figuring out your plan design? Talk to Betterment. Our experts make it easy for you to offer your employees a better 401(k) quickly and easily—all for a fraction of the cost of most providers.

Want a better 401(k)?
Learn More>

You can also visit our other websites and post your article.

Digital Pudding, Garden Center Blog, Business Lawyer Of Milwaukee, 123 Design Studio, Authentic Maple Leaf’s Shop, Gigle, Tech Ready Mix , Best Baby Items, Symposium 18, Jocok Iwanis, Dennis Kitchens, No Agents, Colo Computer Clinic, Adventures In Business Communications, Art Van Bode Graven, Brics 2015, Oyster Sentinel, House Calls Observe, Cedar Homes Chile, Chinook Garage Doors, Contractors Centerpoint, Down Home Living, New West Institute, US Home Based Business, BCN Equipamientos, Professionals PH, Thomas Angel, Coroa Homes, Stewart Homes Inc, 24 Hour Plumbing DFW, F1 Racing Tech, Point Payment Center, Raise Ohio, Castors Avignon, Coyle Greer, Something To Celebrate, Tor Option, Cook Equipment, Packers NFL Official Online Store, Atlantic Retzalisations

Smart savers

401(k) considerations for highly compensated employees

A 401(k) plan should help every employee – from senior executives to entry-level workers – save for a more comfortable future. To help ensure highly compensated employees (HCEs) don’t gain an unfair advantage through the 401(k) plan, the IRS implemented certain rules that all plans must follow. Wondering how to navigate these special considerations for HCEs? Read on for answers to commonly asked questions.

1. What is an HCE?

According to the IRS, an HCE is an individual who:

Owned more than 5% of the interest in the business at any time during the year or the preceding year, regardless of how much compensation that person earned or received, orReceived compensation from the business of more than $130,000 (if the preceding year is 2020 or 2021), and, if the employer so chooses, was in the top 20% of employees when ranked by compensation.

2. Why are there special considerations for HCEs?

Does your plan offer a company match? If so, consider this example: Joe is a senior manager earning $200,000 a year. He can easily afford to max out his 401(k) plan contributions and earn the full company match (dollar-for-dollar up to 6%). Thomas is an entry-level administrative assistant earning $35,000 a year. He can only afford to contribute 2% of his paycheck to the 401(k) plan, and therefore, isn’t eligible for the full company match. Not only that, Joe can contribute more – and earn greater tax benefits – than Thomas. It doesn’t seem fair, right? The IRS doesn’t think so either.

To ensure HCEs don’t disproportionately benefit from the 401(k) plan, the IRS requires annual compliance tests known as non-discrimination tests.

3. What is non-discrimination testing?

In order to retain tax-qualified status, a 401(k) plan must not discriminate in favor of key owners and officers, nor highly compensated employees. This is verified annually by a number of tests, which include:

Coverage tests  – These tests review the ratio of HCEs benefitting from the plan (i.e., of employees considered highly compensated, what percent are benefiting) against the ratio of non-highly compensated employees (NHCEs) benefiting from the plan. Typically, the NHCE percentage benefiting must be at least 70% or 0.7 times the percentage of HCEs considered benefiting for the year, or further testing is required. These tests are performed across employee contributions, matching, and after-tax contributions, and non-elective (employer, non-matching) contributions.ADP and ACP tests – The Actual Deferral Percentage (ADP) Test and the Actual Contribution Percentage (ACP) Test help to ensure that HCEs are not saving significantly more than the employee base. The tests compare the average deferral (traditional and Roth) and employer contribution (matching and after-tax) rates between HCEs and NHCEs.Top-heavy test – A plan is considered top-heavy when the total value of the Key employees’ plan accounts is greater than 60% of the total value of the plan assets. (The IRS defines a key employee as an officer making more than $185,000, an owner of more than 5% of the business, or an owner of more than 1% of the business who made more than $150,000 during the plan year.)

4. What if my plan doesn’t pass non-discrimination testing?

You may be surprised to learn that it’s actually easier for large companies to pass the tests because they have many employees at varying income levels contributing to the plan. However, small and mid-size businesses may struggle to pass if they have a relatively high number of HCEs. If HCEs contribute a lot to the plan, but NHCEs don’t, there’s a chance that the 401(k) plan will not pass nondiscrimination testing.

If your plan fails, you’ll need to fix the imbalance by returning 401(k) plan contributions to your HCEs or increasing contributions to your NHCEs. If you have to refund contributions, affected employees may fall behind on their retirement savings—and that money may be subject to state and federal taxes! Not to mention the fact that you may upset several top employees, which could have a detrimental impact on employee satisfaction and retention.

5. How can I avoid this headache-inducing situation?

If you want to bypass compliance tests, consider a safe harbor 401(k) plan. A safe harbor plan is like a typical 401(k) plan except it requires you to:

Contribute to the plan on your employees’ behalf, sometimes as an incentive for them to save in the planEnsure the mandatory employer contribution vests immediately – rather than on a graded or cliff vesting schedule – so employees can always take these contributions with them when they leave

To fulfill safe harbor requirements, you can elect one of the following employer contribution formulas:

Basic safe harbor match—Employer matches 100% of employee contributions, up to 3% of their compensation, plus 50% of the next 2% of their compensationEnhanced safe harbor match—Employer matches 100% of employee contributions, up to 4% of their compensation.Non-elective contribution—Employer contributes 3% of each employee’s compensation, regardless of whether they make their own contributions.

Want to contribute more? You absolutely can – the above percentages are only the minimum required of a safe harbor plan.

6. How can a safe harbor plan benefit my top earners?

With a safe harbor 401(k) plan, you can ensure that your HCEs will be able to max out your retirement contributions (without the fear that contributions will be returned if the plan fails nondiscrimination testing).

7. What are the upsides (and downsides) of a safe harbor plan?

Beyond ensuring your HCEs can max out their contributions, a safe harbor plan can help you:

Attract and retain top talent—Offering your employees a matching or non-elective contribution is a powerful recruitment tool. Plus, an employer contribution is a great way to reward your current employees (and incentivize them to save for their future).Improve financial wellness—Studies show that financial stress impacts employees’ ability to focus on work. By helping your employees save for retirement, you help ease that burden and potentially improve company productivity and profitability.Save time and stress—Administering your 401(k) plan takes time—and it can become even more time-consuming and stressful if you’re worried that your plan may not pass nondiscrimination testing. Bypass certain tests altogether by electing a safe harbor 401(k).Reduce your taxable income—Like any employer contribution, safe harbor contributions are tax deductible! Plus, you can receive valuable tax credits to help offset the costs of your 401(k) plan.

Of course, these benefits come with a cost; specifically the expense of increasing your overall payroll by 3% or more. So be sure to evaluate whether your company has the financial capacity to make employer contributions on an annual basis.

8. Are there other ways for HCEs to save for retirement?

If you decide against a safe harbor plan, you can always encourage your HCEs to take advantage of other retirement-saving avenues, including:

Health savings account (HSA) – If your company offers an HSA – typically available to those enrolled in a high-deductible health plan (HDHP) – individuals can contribute up to $3,600, families can contribute up to $7,200, and employees age 55 or older can contribute an additional $1,000 in 2021. The key benefits are:>Contributions are tax free, earnings grow tax-free, and funds can be withdrawn tax-free anytime they’re used for qualified health care expenses.The HSA balance carries over and has the potential to grow unlike a “use-it-or-lose-it” FSA.Once employees turn 65, they can withdraw money from an HSA for any purpose – not just medical expenses – without penalty. However, they will have to pay income tax, so they may want to consider reserving it for medical expenses in retirement.Traditional IRA – If employees make after-tax contributions to a traditional IRA, all earnings and growth are tax-deferred. For 2021, the IRA contribution maximum is $6,000 and employees age 50 or older can make an additional $1,000 catch-up contribution.Roth IRA – HCEs may still be eligible to contribute to a Roth IRA, since Roth IRAs have their own separate income limits. But even if an employee’s income is too high to contribute to a Roth IRA, they may be able to convert a Traditional IRA into a Roth IRA via the “backdoor” IRA strategy. To do so, they would make non-deductible contributions to their Traditional IRA, open a Roth IRA, and perform a Roth IRA conversion. This is a more advanced strategy, so for more information, your employees should consult a financial advisor.Taxable Account – A taxable account is a great way to save beyond IRS limits. If employees are maxed out their 401(k) and IRA and want to keep saving, they can invest extra cash in a taxable account.

Want to learn more? Betterment can help.

Helping HCEs navigate retirement planning can be a challenge. If you’re considering a safe harbor plan or want to explore new ways to enhance retirement savings for all your employees, talk to Betterment today.

Want a better 401(k)?
Learn More>

Did you miss our previous article…
https://www.toroption.co/?p=276

In the past 24 hours, a Ruby on Rails application at Betterment performed somewhere on the order of 10 million asynchronous tasks.

While many of these tasks merely sent a transactional email, or fired off an iOS or Android push notification, plenty involved the actual movement of money—deposits, withdrawals, transfers, rollovers, you name it—while others kept Betterment’s information systems up-to-date—syncing customers’ linked account information, logging events to downstream data consumers, the list goes on.

What all of these tasks had in common (aside from being, well, really important to our business) is that they were executed via a database-backed job-execution framework called Delayed, a newly-open-sourced library that we’re excited to announce… right now, as part of this blog post!

And, yes, you heard that right. We run millions of these so-called “background jobs” daily using a SQL-backed queue—not Redis, or RabbitMQ, or Kafka, or, um, you get the point—and we’ve very intentionally made this choice, for reasons that will soon be explained! But first, let’s back up a little and answer a few basic questions.

Why Background Jobs?

In other words, what purpose do these background jobs serve? And how does running millions of them per day help us?

Well, when building web applications, we (as web application developers) strive to build pages that respond quickly and reliably to web requests. One might say that this is the primary goal of any webapp—to provide a set of HTTP endpoints that reliably handle all the success and failure cases within a specified amount of time, and that don’t topple over under high-traffic conditions.

This is made possible, at least in part, by the ability to perform units of work asynchronously. In our case, via background jobs. At Betterment, we rely on said jobs extensively, to limit the amount of work performed during the “critical path” of each web request, and also to perform scheduled tasks at regular intervals. Our reliance on background jobs even allows us to guarantee the eventual consistency of our distributed systems, but more on that later. First, let’s take a look at the underlying framework we use for enqueuing and executing said jobs.

Frameworks Galore!

And, boy howdy, are there plenty of available frameworks for doing this kind of thing! Ruby on Rails developers have the choice of resque, sidekiq, que, good_job, delayed_job, and now… delayed, Betterment’s own flavor of job queue!

Thankfully, Rails provides an abstraction layer on top of these, in the form of the Active Job framework. This, in theory, means that all jobs can be written in more or less the same way, regardless of the job-execution backend. Write some jobs, pick a queue backend with a few desirable features (priorities, queues, etc), run some job worker processes, and we’re off to the races! Sounds simple enough!

Unfortunately, if it were so simple we wouldn’t be here, several paragraphs into a blog post on the topic. In practice, deciding on a job queue is more complicated than that. Quite a bit more complicated, because each backend framework provides its own set of trade-offs and guarantees, many of which will have far-reaching implications in our codebase. So we’ll need to consider carefully!

How To Choose A Job Framework

The delayed rubygem is a fork of both delayed_job and delayed_job_active_record, with several targeted changes and additions, including numerous performance & scalability optimizations that we’ll cover towards the end of this post. But first, in order to explain how Betterment arrived where we did, we must explain what it is that we need our job queue to be capable of, starting with the jobs themselves.

You see, a background job essentially represents a tiny contract. Each consists of some action being taken for / by / on behalf of / in the interest of one or more of our customers, and that must be completed within an appropriate amount of time. Betterment’s engineers decided, therefore, that it was critical to our mission that we be capable of handling each and every contract as reliably as possible. In other words, every job we attempt to enqueue must, eventually, reach some form of resolution.

Of course, job “resolution” doesn’t necessarily mean success. Plenty of jobs may complete in failure, or simply fail to complete, and may require some form of automated or manual intervention. But the point is that jobs are never simply dropped, or silently deleted, or lost to the cyber-aether, at any point, from the moment we enqueue them to their eventual resolution.

This general property—the ability to enqueue jobs safely and ensure their eventual resolution—is the core feature that we have optimized for. Let’s call it resilience.

Optimizing For Resilience

Now, you might be thinking, shouldn’t all of these ActiveJob backends be, at the very least, safe to use? Isn’t “resilience” a basic feature of every backend, except maybe the test/development ones? And, yeah, it’s a fair question. As the author of this post, my tactful attempt at an answer is that, well, not all queue backends optimize for the specific kind of end-to-end resilience that we look for. Namely, the guarantee of at-least-once execution.

Granted, having “exactly-once” semantics would be preferable, but if we cannot be sure that our jobs run at least once, then we must ask ourselves: how would we know if something didn’t run at all? What kind of monitoring would be necessary to detect such a failure, across all the features of our app, and all the types of jobs it might try to run? These questions open up an entirely different can of worms, one that we would prefer remained firmly sealed.

Remember, jobs are contracts. A web request was made, code was executed, and by enqueuing a job, we said we’d eventually do something. Not doing it would be… bad. Not even knowing we didn’t do it… very bad. So, at the very least, we need the guarantee of at-least-once execution.

Building on at-least-once guarantees

If we know for sure that we’ll fully execute all jobs at least once, then we can write our jobs in such a way that makes the at-least-once approach reliable and resilient to failure. Specifically, we’ll want to make our jobs idempotent—basically, safely retryable, or resumable—and that is on us as application developers to ensure on a case-by-case basis. Once we solve this very solvable idempotency problem, then we’re on track for the same net result as an “exactly-once” approach, even if it takes a couple extra attempts to get there.

Furthermore, this combination of at-least-once execution and idempotency can then be used in a distributed systems context, to ensure the eventual consistency of changes across multiple apps and databases. Whenever a change occurs in one system, we can enqueue idempotent jobs notifying the other systems, and retry them until they succeed, or until we are left with stuck jobs that must be addressed operationally. We still concern ourselves with other distributed systems pitfalls like event ordering, but we don’t have to worry about messages or events disappearing without a trace due to infrastructure blips.

So, suffice it to say, at-least-once semantics are crucial in more ways than one, and not all ActiveJob backends provide them. Redis-based queues, for example, can only be as durable (the “D” in “ACID”) as the underlying datastore, and most Redis deployments intentionally trade-off some durability for speed and availability. Plus, even when running in the most durable mode, Redis-based ActiveJob backends tend to dequeue jobs before they are executed, meaning that if a worker process crashes at the wrong moment, or is terminated during a code deployment, the job is lost. These frameworks have recently begun to move away from this LPOP-based approach, in favor of using RPOPLPUSH (to atomically move jobs to a queue that can then be monitored for orphaned jobs), but outside of Sidekiq Pro, this strategy doesn’t yet seem to be broadly available.

And these job execution guarantees aren’t the only area where a background queue might fail to be resilient. Another big resilience failure happens far earlier, during the enqueue step.

Enqueues and Transactions

See, there’s a major “gotcha” that may not be obvious from the list of ActiveJob backends. Specifically, it’s that some queues rely on an app’s primary database connection—they are “database-backed,” against the app’s own database—whereas others rely on a separate datastore, like Redis. And therein lies the rub, because whether or not our job queue is colocated with our application data will greatly inform the way that we write any job-adjacent code.

More precisely, when we make use of database transactions (which, when we use ActiveRecord, we assuredly do whether we realize it or not), a database-backed queue will ensure that enqueued jobs will either commit or roll back with the rest of our ActiveRecord-based changes. This is extremely convenient, to say the least, since most jobs are enqueued as part of operations that persist other changes to our database, and we can in turn rely on the all-or-nothing nature of transactions to ensure that neither the job nor the data mutation is persisted without the other.

Meanwhile, if our queue existed in a separate datastore, our enqueues will be completely unaware of the transaction, and we’d run the risk of enqueuing a job that acts on data that was never committed, or (even worse) we’d fail to enqueue a job even when the rest of the transactional data was committed. This would fundamentally undermine our at-least-once execution guarantees!

We already use ACID-compliant datastores to solve these precise kinds of data persistence issues, so with the exception of really, really high volume operations (where a lot of noise and data loss can—or must—be tolerated), there’s really no reason not to enqueue jobs co-transactionally with other data changes. And this is precisely why, at Betterment, we start each application off with a database-backed queue, co-located with the rest of the app’s data, with the guarantee of at-least-once job execution.

By the way, this is a topic I could talk about endlessly, so I’ll leave it there for now. If you’re interested in hearing me say even more about resilient data persistence and job execution, feel free to check out Can I break this?, a talk I gave at RailsConf 2021! But in addition to the resiliency guarantees outlined above, we’ve also given a lot of attention to the operability and the scalability of our queue. Let’s cover operability first.

Maintaining a Queue in the Long Run

Operating a queue means being able to respond to errors and recover from failures, and also being generally able to tell when things are falling behind. (Essentially, it means keeping our on-call engineers happy.) We do this in two ways: with dashboards, and with alerts.

Our dashboards come in a few parts. Firstly, we host a private fork of delayed_job_web, a web UI that allows us to see the state of our queues in real time and drill down to specific jobs. We’ve extended the gem with information on “erroring” jobs (jobs that are in the process of retrying but have not yet permanently failed), as well as the ability to filter by additional fields such as job name, priority, and the owning team (which we store in an additional column).

We also maintain two other dashboards in our cloud monitoring service, DataDog. These are powered by instrumentation and continuous monitoring features that we have added directly to the delayed gem itself. When jobs run, they emit ActiveSupport::Notification events that we subscribe to and then forward along to a StatsD emitter, typically as “distribution” or “increment” metrics. Additionally, we’ve included a continuous monitoring process that runs aggregate queries, tagged and grouped by queue and priority, and that emits similar notifications that become “gauge” metrics. Once all of these metrics make it to DataDog, we’re able to display a comprehensive timeboard that graphs things like average job runtime, throughput, time spent waiting in the queue, error rates, pickup query performance, and even some top 10 lists of slowest and most erroring jobs.

On the alerting side, we have DataDog monitors in place for overall queue statistics, like max age SLA violations, so that we can alert and page ourselves when queues aren’t working off jobs quickly enough. Our SLAs are actually defined on a per-priority basis, and we’ve added a feature to the delayed gem called “named priorities” that allows us to define priority-specific configs. These represent integer ranges (entirely orthogonal to queues), and default to “interactive” (0-9), “user visible” (10-19), “eventual” (20-29), and “reporting” (30+), with default alerting thresholds focused on retry attempts and runtime.

There are plenty of other features that we’ve built that haven’t made it into the delayed gem quite yet. These include the ability for apps to share a job queue but run separate workers (i.e. multi-tenancy), team-level job ownership annotations, resumable bulk orchestration and batch enqueuing of millions of jobs at once, forward-scheduled job throttling, and also the ability to encrypt the inputs to jobs so that they aren’t visible in plaintext in the database. Any of these might be the topic for a future post, and might someday make their way upstream into a public release!

But Does It Scale?

As we’ve grown, we’ve had to push at the limits of what a database-backed queue can accomplish. We’ve baked several improvements into the delayed gem, including a highly optimized, SKIP LOCKED-based pickup query, multithreaded workers, and a novel “max percent of max age” metric that we use to automatically scale our worker pool up to ~3x its baseline size when queues need additional concurrency.

Eventually, we could explore ways of feeding jobs through to higher performance queues downstream, far away from the database-backed workers. We already do something like this for some jobs with our journaled gem, which uses AWS Kinesis to funnel event payloads out to our data warehouse (while at the same time benefiting from the same at-least-once delivery guarantees as our other jobs!). Perhaps we’d want to generalize the approach even further.

But the reality of even a fully “scaled up” queue solution is that, if it is doing anything particularly interesting, it is likely to be database-bound. A Redis-based queue will still introduce DB pressure if its jobs execute anything involving ActiveRecord models, and solutions must exist to throttle or rate limit these jobs. So even if your queue lives in an entirely separate datastore, it can be effectively coupled to your DB’s IOPS and CPU limitations.

So does the delayed approach scale?

To answer that question, I’ll leave you with one last takeaway. A nice property that we’ve observed at Betterment, and that might apply to you as well, is that the number of jobs tends to scale proportionally with the number of customers and accounts. This means that when we naturally hit vertical scaling limits, we could, for example, shard or partition our job table alongside our users table. Then, instead of operating one giant queue, we’ll have broken things down to a number of smaller queues, each with their own worker pools, emitting metrics that can be aggregated with almost the same observability story we have today. But we’re getting into pretty uncharted territory here, and, as always, your mileage may vary!

Try it out!

If you’ve read this far, we’d encourage you to take the leap and test out the delayed gem for yourself! Again, it combines both DelayedJob and its ActiveRecord backend, and should be more or less compatible with Rails apps that already use ActiveJob or DelayedJob. Of course, it may require a bit of tuning on your part, and we’d love to hear how it goes! We’ve also built an equivalent library in Java, which may also see a public release at some point. (To any Java devs reading this: let us know if that interests you!)

Already tried it out? Any features you’d like to see added? Let us know what you think!

Did you miss our previous article…
https://www.toroption.co/?p=273