Now that we have covered unit testing and background jobs with Sidekiq, let's
add a new feature to send reminder emails to users with outstanding tasks.
In this chapter we will discuss concepts like idempotency, mail delivery
windows, setting user preferences, and more.
Features
These are the basic requirements of the feature :
- Send reminder emails to users who have pending tasks.
- Users should have an option to enable/disable the email notifications.
- User can set preferred time to receive this pending tasks notification email.
- If a user has disabled the email notification then that user should not be
able to set preferred time to receive email.
Technical design
To implement this feature, we need to introduce the following changes:
On the backend
-
For scheduling periodic Sidekiq jobs, we can use the
sidekiq-cron gem, which will be sending the mail.
-
In the development environment, instead of sending an email to the user's
email address, we can preview the email in our browser using the
letter_opener gem.
-
By default, if the user has set a preferred time then the mail will be sent at
that time else mail will be sent at 10 AM.
-
We store this preferred time in a table called preferences. This
preferences table consists of
notification_delivery_hour,should_receive_email, and user columns.
-
Create user_notifications table with last_notification_sent_date and
user columns.
-
In UserNotification model, add validations to ensure that the
last_notification_sent_date attribute is present, has a valid date format,
and the date is not in the past.
-
We will be creating a default preference for the user whenever a new user is
created.
-
We will be saving the default preference value in a constant
DEFAULT_NOTIFICATION_DELIVERY_HOUR, which will be loaded automatically from
config/initializers/constants.rb, by Rails, after it loads the framework
plus any gems and plugins in our application.
-
Add PreferencesController which will add the ability for the user to update
their preference for if and when to receive the email.
Inside the PreferencesController, we will add a custom action called mail
which will handle enabling and disabling the email notification.
-
Mail sending can often fail due to network errors, among others. When that
happens, Sidekiq will retry based on rules we set in the initializer.
On the frontend
-
Define preference related axios connectors.
-
We will add a Preferences component which will handle the preferences of
the user.
-
Delegate the job of taking in the preferred time for mail delivery to a Form
component within the Preferences folder.
-
Add a function fetchPreferenceDetails in the Preferences container
component which will fetch preference details like
notification_delivery_hour and should_receive_email.
-
Add a function updateEmailNotification in the Preferences component
which will handle enabling/disabling email notifications.
-
Similarly, add a function updatePreference in the Preferences container
component which will update notification_delivery_hour.
Sidekiq Job
To send each batch of reminder emails, we will schedule a
TodoNotificationsJob to run every hour and delegate the job of finding the
recipients to a TodoNotificationsService.
This service will ultimately invoke separate UserNotificationJob's for each
recipient it finds.
Let's say we use a single job for sending all the emails. If this job
fails in the middle of sending the emails the whole email delivery system will
be down. The rest of the users will not get any email. This is known as the
single point of failure. To avoid this single point of failure we are using
separate jobs for each recipient. If a separate job fails then it does not
affect the rest of the jobs and they can continue sending emails. Thus we can
use separate jobs and avoid a single point of failure and make our system
more reliable.
We will also create a scheduler file to store cron values that will help
schedule our TodoNotificationsJob at the desired time interval for each
environment.
The parent Sidekiq Job will then invoke TodoNotificationsService, which
will find all the users with pending tasks but no entries in a new table called
user_notifications.
What is idempotency?
We need our service to be idempotent, ie: duplicate mail shouldn't be sent every
time we retry.
Let's say our email delivery service started and successfully sends 200 emails.
But on the delivery of the 201st email, it failed. Now, if we retry sending the
failed emails, then it's possible that those 200 emails which were already
delivered, will be sent again to the recipient. This only occurs if our service
is not idempotent. We can say our service is idempotent only when no duplicate
emails are sent each and every time we retry/rerun the service.
To make our service idempotent we need to add a mechanism to keep track of who
has received the email and who has not. If we have this data, then we can figure
out to whom all we have already sent the emails successfully. This ensures that
even if the service fails in between and is restarted, no user should will get
the same email twice.
From a RESTful service standpoint, an operation or service call is idempotent
when clients can perform a duplicate request harmlessly. In other words, making
multiple identical requests has the same effect as making a single request.
Note that while idempotent operations produce the same result on the server, the
response itself may not be the same; e.g. a resource's state may change between
requests.
To keep track of sent emails we will use the user_notifications table from
earlier to associate each user with a last_notification_sent_date.
If a user has an entry for today in user_notifications, it means there's no
need to try again.
Action Mailer
Action Mailer allows you to send emails from your application using mailer
classes and views. They inherit from ActionMailer::Base and live in
app/mailers.
To make sure that the email has truly been sent to the user, we will make use of
Action Mailer's after_action callback to safely set the
last_notification_sent_date value only after the mail has been sent
successfully.
Mailers have methods called "actions" and they use views to structure their
content.
A controller generates content like say HTML, which will be sent back to the
client, whereas a Mailer creates a message to be delivered via email.
We will be creating the TaskMailer mailer and task_mailer view.
We are now ready to start coding. Let us dive in.
Setting up base
Open the Gemfile.rb and add the following lines:
For the letter_opener gem to open the mails automatically in our browser, we
need to set the following in config/environments/development.rb:
Now any email delivery will pop up in our browser instead of being sent via the
internet. The messages will be stored in ./tmp/letter_opener.
Creating models
Go to db/migrate folder and paste the following lines of code for the
CreateUserNotifications migration:
Also, paste the following lines of code for the CreatePreferences migration:
Adding validations
Now let's setup UserNotification model and add validations for
last_notification_sent_date.
Open app/models/user_notification.rb and add the following lines to it:
Now let's setup our Preference model along with validations in
app/models/preference.rb:
Currently, we are only providing the ability to set the preference for when to
receive an email notification.
Thus in the above code, we have added the validations for the
notification_delivery_hour attribute which makes sure that it's present, and
it only accepts integers ranging from 0-23.
Add the following line in config/initializers/constants.rb file::
We are setting the DEFAULT_NOTIFICATION_DELIVERY_HOUR in our constants.rb
file instead of a specific model. This is because the constant is not limited to
a specific context and will be used across different files. Thus it makes sense
to add it to a global Constants context instead.
Now let's set the associations correctly in our User model too:
The private method, build_default_preference, will create a default preference
for the user whenever a new user is created.
Inside the build_default_preference method we have used the build_preference
method. Let's take a look at where it came from.
When we declare a belongs_to or has_one association, the declaring class
automatically will have access to a build_association method related to the
association. In this case, because of the has_one :preference association in
User model, Rails adds the build_preference method to the User class.
In case of has_many or has_and_belongs_to_many associations, we should use
the association.build method instead of the build_association method. For
example, if we were to declare a has_many :preferences association in the User
model, we would have to use the preferences.build method given that Rails
won't add the build_preferences method for an association that is not
has_one.
Now that we have established the associations in the User model, let's enhance
its readability. The below lines illustrate the associations defined in our
model:
Upon reviewing this code snippet, we can observe that the first two associations
share a common option of class_name: "Task". Furthermore, the remaining
associations include the option dependent: :destroy. Therefore, it would be
beneficial if we group the associations by their shared options. This is where
the with_options method can aid us. Let's replace the aforementioned
associations with the code below:
The with_option method provides an elegant way of factoring out duplication
from options passed to a series of method calls. Each method called in the
block, with the block variable as the receiver, will have its options merged
with the default options hash provided. You can learn more about this method in
the official docs.
Adding preferences for existing users
The build_preference method we had added in the previous section will only add
a default preference for new users. The preference for all existing users will
remain nil. There are two ways by which we can fix this issue:
-
Deleting all users from the console and creating new users. Newly added users
will have a default preference but we shouldn't take this approach because:
-
It's considered a bad practice.
-
Once the application is deployed, we shouldn't delete users from console in
production environment.
-
A better way to do this would be to add a migration which will add default
preference for all existing users where preference is nil.
If you recall, we have done something similar in the
Adding slug to task chapter
where we had used a migration to add slugs to existing task records.
Before moving on towards adding a migration to fix the issue, you should attempt
to do this by yourself. If you need help you can refer to the
Adding slug to task chapter.
Now, let's us add a migration to fix this issue. Run the following command to
generate a migration:
The above command with generate a migration file. Inside the migration file add
the following lines of code:
We have used the missing method which performs a left outer join operation on
parent and child tables then uses a where clause to identify the missing
relations. In this case we are querying all user records which have a missing
preference relation and then adding a default preference for all those user
records.
You can read more about the missing method from
official documentation.
Run the following command to execute the migration:
Creating the controller
Create a new controller for the preference model by running the following
command:
Open app/controllers/preferences_controller.rb and add the following lines to
it:
We have added a mail action in the PreferencesController which will handle
the logic to enable and disable the email notifications. Now, the question
arises that why we need another action when updating a record should be handled
by the update action. We want to do so because the mail action is only
concerned with updating the should_receive_email attribute of a preference
record and the response for that will also be different from the update
action. Therefore, it is better to add a custom action.
We have created a load_preference method to load the preference before the
show, update and mail actions are called. Since each user has one
associated preference record we can query it using current_user.preference
inside the load_preference method. Hence appending the preference id to the
preference routes or sending it as a request param while making a request is not
required in this case.
Now, let's create the route for the preference. We should declare preference
as a singular resource because we do not need a resource identifier in the
routes. Open the routes.rb and append the following line:
We have added a collection route called mail for the preference resource.
Any patch request on preference/mail will invoke the mail action inside the
PreferencesController.
Note that we have used a patch verb for the mail route over the put verb.
This is so because a put verb is used when the entire entity needs to be
updated whereas a patch request is used when the entity needs to be updated
partially. In our case, we are only updating the should_receive_email
attribute of a preference record thus the use of patch verb.
Why we do not need a preference policy?
Policies prevent any unauthorized access of database records. In case of
preferences, a user should not be able to access the preferences of another
user. We do not need a policy to ensure this because we are using the
preferences association of the User model to load the preference inside the
PreferencesController.
Doing so will only load the preference of the current_user thus eliminating
the need for authorization using a policy.
Building the UI for user preference
Since we have already added APIs for handling user preference, let's now build
the views for it.
Let's create a new file to define all the preference related APIs:
Open the apis/preferences.js and paste the following lines:
Now, let's create the preference component:
Paste the following lines into index.jsx:
Here, fetchPreferenceDetails function uses the show axios API call from
preference related axios APIs. Similarly, the updatePreference function will
be using the update axios API.
As you can see we have used a Form component which is not yet created. Let's
create it:
Paste the following lines in the Preferences/Form.jsx:
The Form component contains a checkbox for enabling and disabling the email
notifications. When the checkbox value changes, updateEmailNotification
function defined inside the Preferences component will be called and it will
in turn make an API for mail action in the PreferencesController.
There is a updatePreference function as well which will be called upon
submitting the form and it will make an API request to update the logged in
user's preference.
We need to provide an Input component from where the user can set their
preferred time for mail delivery.
We have to constraint the values that this component can accept, when its
type is set to be number.
Thus let's update our reusable Input component to provide this functionality:
Finally, our Preference component is ready to use. Let's add a route for this
component in App.jsx and let's also create a link to it in our navbar:
Now open the Navbar component from components/NavBar/index.jsx and add the
following line to create a Preferences item in navbar:
Implementing task mailer
Let's create the task mailer by running the following Rails command:
Open the application_mailer.rb from app/mailers/application_mailer.rb and,
provide the default email from which the mail will be send:
Mailers have methods called "actions" and they use views to structure their
content. A controller generates content like say HTML, which will be sent back
to the client, whereas a Mailer creates a message to be delivered via email.
Now, let's edit the task_mailer.rb which we have created.
Paste the following lines in task_mailer.rb:
As you can see in the above code we have used after_action which is an
Action Mailer Callback. Using an after_action callback enables you to
perform operations after successfully sending the mail.
Once we ensure that mail is successfully sent, we will create an entry in
user_notifications with today's date, for the current user.
Our TaskMailer is ready. Let's create a mailer view, which will contain the
content and styling of the email.
Open app/views and you should be able to see the task_mailer directory.
Let's create a pending_tasks_email.html.erb file into it:
Paste the following lines into it:
Previewing emails using ActionMailer::Preview
During development, we will need to see how the layout and content of an email will look like in the UI without having to go through steps of generating an email, by sending a request to a controller action, triggering a service, or waiting for a job to execute. Rails provides the ActionMailer::Preview class for this specific purpose. It allows us to preview emails by visiting a URL.
Let's see how we can utilize ActionMailer::Preview to preview the email for the TaskMailer.
First, we need to create a class that inherits ActionMailer::Preview in the test/mailers/previews folder. Rails identifies a mailer as a preview if the file name and class name is suffixed with the keyword preview in appropriate casing. So in the case of TaskMailer, the class should be named TaskMailerPreview, and the file should be name task_mailer_preview.rb. Since this file is already generated as part of the mailer generate command, we can skip this step.
Next, to see the preview of pending_tasks_email, implement a method that has the same name and call TaskMailer.pending_tasks_email from that method. We follow this convention of naming mailer actions and preview methods with the same name. We can pass the ID of the first user as the receiver_id. Add the following code to task_mailer_preview.rb:
While previewing emails, we do not want to create database entries for user notifications. For that, we have passed a param named preview with the value true to denote that we are previewing the mail.
We can use this param to skip the after_action callback in TaskMailer, like so:
In the above code, we are using the safe navigation operator(&) to access the preview key of the params hash. It checks if the value of params is nil. If it's not nil, it proceeds to access the value associated with the key :preview in the params hash. If params is nil, the expression returns nil without raising an error.
Now, you can preview the pending task email at http://localhost:3000/rails/mailers/task_mailer/pending_tasks_email.
You can see the list of URLs of all email previews associated with TaskMailer at http://localhost:3000/rails/mailers/task_mailer.
Use case of secrets.yml file
Rails supports the secrets.yml file to store your application's secrets. It is
used to store secrets such as access keys for external APIs or our Redis URL
etc.
This will provide reusability and also will help provide us easier access.
Let's create this file:
Open the file and paste the following lines:
Creating scheduled_jobs.yml
We will create a scheduler file containing the cron syntax for different
environments and jobs.
Cron is a standard Unix utility that is used to schedule commands for automatic
execution at specific intervals.
Here we are using cron to send an email notification to the users every day at
the default time, i.e., at 10 AM or preferred time, if set by the user.
Let's create the scheduler file:
Open the file and paste the following lines into it:
The default and development are two important keys in the YAML file. Each of
those keys invoke a job.
In the above code, you can see that the cron value for default and
development are different. The reason is that in development we don't need to
wait. We need the cron to get triggered as soon as possible.
The cron value for development executes a cron job every minute, whereas for
default the same is executed every hour.
Here in our YAML file, the default key is the main key because its value will
be used in other environments like test, staging, and production.
We create an alias for the default key using an anchor marked with the
ampersand & and then reference that alias using the asterisk *. This is
purely a "YAML" way of inheriting existing keys.
Adding logic to schedule cron jobs
We have already written down when and what needs to be enqueued, in our
scheduled_jobs.yml file.
Thus in the Sidekiq initializer, we have to add logic that will read the content
from our scheduler file and will enqueue based on the cron defined for the
specific environment.
Also, we have to update the REDIS_URL with the URL we have specified in our
secrets file.
Open sidekiq.rb from config/initializers.
Add the following lines to this file:
We have currently not enabled mail sending in production environment.
If possible, we will be handling this case in the upcoming chapters.
Adding TodoNotifications functionality
Let's create the TodoNotifications job:
The sole purpose of the TodoNotificationsJob is process a
TodoNotificationsService, which will take care of other core functionalities.
Open app/jobs/todo_notifications_job.rb and paste the following:
Let's create our TodoNotificationService to handle the heavy work. This service will take care of finding users with pending tasks, whose delivery
time falls with the current execution time range, and invoke workers which will
send the mail to each user.
Open the todo_notification_service.rb file and paste the following:
Let's discuss the initialize method before moving ahead. initialize method
in Ruby is basically a constructor which we use to initialize instance variables
during object creation. The initialize method gets invoked implicitly even if
no instance variables are initialized.
If there is an attribute which you need to access in multiple methods defined
inside the class then you should declare it as an instance variable and
initialize it in the initialize method. At BigBinary we use the
attr_accessor macro to access the instance methods inside the class.
Let's also create a separate job, named UserNotificationsJob.
As per our technical design, we need to ensure that mail delivery for one single
user, doesn't break the whole delivery system.
The first line of defense against this case, is to invoke a separate job,
which is UserNotificationsJob, for each of the user.
This way failure of one job won't affect the other jobs, given that
Sidekiq runs them in separate threads.
Let's create UserNotificationsJob:
Open app/jobs/user_notifications_job.rb and paste the following:
deliver_later enqueues the email to be delivered, into a queue called
mailers, through Active Job. Since we want Sidekiq to use the mailers queue
along with the default queue, let us set them in the Sidekiq configuration file
sidekiq.yml by replacing its contents with the following:
Please note that in order for the deliver_later method to utilize sidekiq,
we also need to set config.active_job.queue_adapter = :sidekiq in
config/application.rb, as we had already done in the previous chapter.
Phew! That was a lot of content with loads of good stuff.
That completes our feature and mail delivery should be working in a development
environment.
We will be writing tests for the same, in the next chapter.
Moving messages to i18n en.locales
Recall that we already have a Task not found. translation definition inside
en.yml file and it is very similar to Preference not found. which is one of
the error translation definitions we have to add.
Rather than having two similar translations for messages which look like
record not found, we can use the variable interpolation feature provided by
i18n to pass the name of the entity which is not found. This will prevent code
duplication.
Fully replace en.yml with the following lines of code:
Now, the not_found translation in en.yml, will expect a value to be passed
for the entity variable. So let's go through our JSON responses and pass that
variable with appropriate entity name, wherever we are using the not_found
translation key.
Update the following highlighted line in the load_preference method in
PreferencesController:
We have also used a Task not found translation inside
test_not_found_error_rendered_for_invalid_task_slug in TasksControllerTest
where we are testing if the correct error is being rendered for an invalid task
slug. Let's update the test case to use the updated i18n translation string.
Update the test_shouldnt_create_comment_without_task inside the
CommentsControllerTest like this:
Update the test_should_respond_with_not_found_error_if_user_is_not_present
inside the SessionsControllerTest like this:
Update the handle_api_exception method inside the ApplicationController like
so:
Let's also add the error translations in UserNotification and Preference
models.
Update UserNotification model with the following lines of code:
Update the following highlighted line in the Preference model:
You may have observed that, in contrast to our other models, we employ I18n.t() instead of t() for internationalization handling in the Preference model. If we were to use the helper method t() here, it would result in the following error:
As this error message suggests, the t() method would not be accessible within the code block passed to the validates method. The validates method is defined at the class level, meaning that any code placed inside the validates block doesn't have direct access to the specific instance being validated. Consequently, instance-specific methods like t, found in TranslationHelper, are not directly accessible within the validates block. This is why we have opted to use I18n.t, which is the translate method available through the I18n module.
There is an alternative approach that allows for direct access to the t method from TranslationHelper. To access instance-specific data or methods, such as t, you can define custom validation methods that are invoked within the context of an instance.
Testing PreferencesController
To test the preferences controller, we do not need to create a preference
factory since a preference association is created automatically for each user
when that new user is created.
Now, create a new file test/controllers/preferences_controller_test.rb and add
the following content:
Now, let us test if the correct preferences are rendered for a valid user.
Append the following test case to the PreferencesControllerTest:
We should also add a test case to check if correct response is being rendered if
the preference is not found. Add the following test case to the
PreferencesControllerTest:
Let us test if the update action is working as intended. Add the following test
cases to the PreferencesControllerTest:
Let us test the mail action. Add the following test case to the
PreferencesControllerTest:
Now you can try running all the preference test cases and they should be
passing:
Now let's commit these changes:
