Monitoring Cron Jobs with Whenever & AppSignal

Hey guys, this episode we're going to be doing some advanced configuration in the whenever gem to monitor cron jobs. So whenever is awesome. It's a little Ruby script that basically generates cron jobs and the cron syntax for us. So we have the ability in our schedule.rb RB file to say every 3 hours or one day or one month whatever we want and it will generate the cron syntax from that and we can also use some of these shortcuts to say let's run a rake task let's run a runner from rails um all that kind of stuff is wired up behind the scenes and we can use this and customize it in order to do monitoring for our cron jobs which is something we often will want to have. There's a cool little utility from AppSign signal called AppSignal Wrap that you can use to run a command and it will capture the output and the exit codes. So you can tell if that uh command was successful, when it ran and capture the logs from it. So we can wrap our cron jobs with this uh using whenever. So let's dive in. Let's go to our terminal. I've sshed into a little Ubuntu virtual machine I've got here. We're in uh logged in as root. So we're going to install app signal wrap there. And then we're going to become the deploy user where we will do our setup here. So I'm going to source bashrc to load our environment. And we should have ruby 4.0.2 the latest version. And we're going to gem install whenever. And whenever is going to give us a whenever command where we can call whenever this will give us config schedule.rb RB and then we can write our own rake task in here. So we're going to do a very simple one. We'll say every 1 minute we want to run a command that prints out the current working directory. That's as simple as it can be. And when we save this we can run whenever it's going to print out the generated command which you will see is wrapped with bin bash-l c is a flag saying this is a login shell. So, when we open up the bash shell, we're going to load the bash profile, which will also load uh typically um like your bash rc, but it might add things like your environment variables. And you can drop in your app signal app push API key as an environment variable in that bash profile and export that so that your cron jobs will have access to it. And then, of course, - c says just run this command. So, that is what it's going to generate. But we need to actually change this so we can use this app signal wrap utility instead. So let's go back to our file here and let's create a new job type. So command is a built-in job type, but we could call one like wrap and give it a command. So with this we could say app signal uh wrap and the name option that we' pass into the command. So we'll say name pwd. We'll need a d-cron flag and then we can do dash and then run the task and set the output as well. So if you ever go up here and uncomment this and set the output to a specific file location, that is where the output of your command will be located. Task is actually a variable that represents um the command that you pass or the task that you pass into the command helper. And then this is going to be used um like so. So we'll say wrap and pwd. So we've basically replaced the built-in command function uh or job type with our own wrap one that wraps with apps signal wrap. And as long as that environment variable with our API key is set up, this will work great. Another thing we can do for testing is we can set output to say our home deploy cron.log file in that directory. And so we could ever we could always poke around at this and make sure it's loading our environment variable. Um but now if we run whenever we'll see this is running bin bash LC um like normal but inside it's going to run apps signal wrap with the name of pwd the cron flag so it reports as a checkin and then this says everything after this will be the command and its options. So pwd will run with the output of standard out and standard error being written to home deploy cron.log. Perfect. So what we can do now is we can say whenever update cron tab and this is going to then write to the cron tab. Then we can go into app signal and create a check-in with a matching name. So we'll say pwd our schedule is in cron syntax. So we want it to run every minute just like we decided in our example uh in the schedule.rb. And we can create our check-in here. So this is going to then basically look for every minute uh check-in from app signal wrap or from their API or whatever. You could use curl to send notifications over. You could use the libraries to do that too. We're going to use apps signal wrap to do this. Then also when this is running, we should see in the log section on app signal that uh the output of this command pwd if you're not familiar just writes the uh current working directory uh prints the working directory is probably what it stands for pwd. So we should see that output here in the logs as well and that's what I've already done in the past with some testing. But let's refresh this and we'll see that the first check-in here was a success which is good. And if we refresh our logs um we'll see at 1512 the same time um that it was also successful. So that's great. So basically the logs are being captured. We are using the command to send a message to app signal saying this is started. We're also sending a command over um to their API that says this is finished at this time and it keeps track of that and if this varies for example you might be like doing something with the database or loading something from a network things can take various amounts of times and you can have a notifier that says hey if this ever takes more than a minute notify us and let us know that something went wrong uh and we should look into it. So that is all we have to do to enable app signal wrap as a custom um job type for whenever. Now this is fine but what if we want to also support command and we want to have app signal wrap work for the rails runner and the rake task examples as well. Well, instead of doing this as a custom uh job type, we can use job template and override that instead. So, we'll say set job template and this becomes a string where very similarly to how it is built in, which I will show you. We'll go to lib whenever setup.rb. There's some stuff in here that sets the environment variable rails m. Uh if that is not set then it will um set it or default it to production. It sets a path variable as well. Some options for chronic for parsing your cron job syntax. And then here we find the job template. So this is really what we want to customize. So here we can do a couple different things. We could rely on uh setting the app signal wrap API key in the cron tab itself or we can use it the same way we kind of defined up here where we have app signal wrap name-cron d- job. So rather than having to define the task in the output that's already defined by these shortcuts the job types themselves. So now we can set this up to just have the job passed in. What will work nicely about this is we're going to load the bash login shell. So in your bash profile when you set your app signal API key, it's going to make that accessible. So we don't actually have to put it in your cron tab um itself. So this is going to work nicely and we can then run whenever uh to see the output of that. First we'll see that it is going to run bin bash lc just like we had before. name, cron flag, and then the rest of our job. Um, and if we want to, you know, test and see something like a rake example or a runner example, um, user delete all. That'll be a fun one. Don't do that every minute. Um, this is going to then print out those full commands, but they get added after the double hyphen here. So we see scriptr runner or rails end production rake example- silent and the output flags. So let's get rid of those terrible examples. Um and then run whenever update cron tab and we just want to confirm back in app signal that the new updates for this um are going to work correctly. So this was if we say cron tab-l one of the cool things about this is it adds these comments and it includes this is the identifier. The identifier defaults to the location of your cron schedule. Um but you can also give it a name to make it a bit more descriptive if you have multiple of these. Then also it shows you the exact timestamp. So we ran this at 151851. So the next time this would actually check in is at 19. and it was also a success. So that is good. We know that the new version of this is also working. So it started at 1901, finished at 1901 as well. Uh cuz it runs super fast. It's not really doing any actual work, but uh that would be the next time that this would actually run. So we'll see that here as well in our logs. 151901.739. Awesome. So, this is a really handy little piece of functionality. You could use this with any other tools that allow you to check in um to monitor cron jobs or whatever. And all you have to do is customize either the job template, which will apply to every single different job type that you have. Or you can have ones that specifically implement that functionality. Um, and you could also add, you know, the, uh, if you wanted to do this version of it, you could say job type, um, runner or rake, and you can have, um, wrap rake as your example. So, you would have app signal wrap, and then you would do the regular rake task stuff here at the end, and you could set the bundle command and all that um, if you wanted to do the custom stuff. But really, I think I'm going to want to monitor every single cron job in this application. So, it makes more sense to have the job template set rather than building out our own custom um job types. But this is some of the cool stuff that you can do with whenever that you may have never touched before and having your uh your tooling around that. So, that this is now built in. um you've probably already got the the app signal library um you know running in your app application collecting errors and things like that. So now you have this as well for your cron jobs which I think is really convenient and helpful to have. So that is it for this episode. I really really love whenever uh there's some so much cool stuff and it is a very straightforward library and it's just one of those things I aspire to build these cool little tools that are very flexible yet welldesigned so you can extend them and do custom things as needed like you see here. Um but it also is very small and concise and does exactly what you needed to do and nothing else. So yeah that is it. That's how you can configure your own custom whenever kron job formats.