Project

General

Profile

Bug #3653

Scheduled jobs stop executing

Added by Vadim 6 months ago. Updated about 1 month ago.

Status:
Marked for Closure
Priority:
Normal
Assignee:
Target version:
RapidDeploy - 5.0-FIX
Start date:
10/15/2020
Due date:
% Done:

0%

Estimated time:
Affects Version:
RapidDeploy - 5.0-FIX
Timesheet Code:
Additional version details:
5.0.27

Description

Hello.

I setup special project for server clean up and scheduled it for each day to run. It was working for some time and stopped. I see project in scheduler calendar, but I don't see it in executed project list. And server looks like it is not maintained as planned. :)

Will You need any additional info? I won't change anything in existing configuration, so if You need I could show You everything.

Will be waiting for You reply.

Have a nice day!
Best regards,
Vadim.


Files

rapiddeploy-web-app.zip (821 KB) rapiddeploy-web-app.zip Vadim, 10/27/2020 03:28 PM
rapiddeploy-web-app.zip (822 KB) rapiddeploy-web-app.zip Vadim, 10/28/2020 07:16 AM
firefox_KglQoLJCL4.png (24.3 KB) firefox_KglQoLJCL4.png Vadim, 11/19/2020 03:39 PM
firefox_RYhKu3jq4C.png (94.2 KB) firefox_RYhKu3jq4C.png Vadim, 11/19/2020 03:39 PM
firefox_Fv9PGICRVX.png (49.6 KB) firefox_Fv9PGICRVX.png Vadim, 11/19/2020 03:39 PM
firefox_j7h8DtQLFn.png (25.6 KB) firefox_j7h8DtQLFn.png Vadim, 11/19/2020 03:39 PM
firefox_TeYAXVxhxT.png (182 KB) firefox_TeYAXVxhxT.png Vadim, 11/20/2020 01:35 PM
firefox_adrKiW8LZm.png (257 KB) firefox_adrKiW8LZm.png Vadim, 11/23/2020 12:35 PM
#1

Updated by Mariano 6 months ago

  • Status changed from New to Feedback
  • Assignee set to Vadim

Hi Vadim,

Based on what you described, for some weird reason the scheduled job was removed from QUARTZ but it is still in future jobs.

So, the quick solution is rescheduling the job, please try:

1 - Go to Future Jobs
2 - Go to Pipeline Jobs or Project Jobs tab depending of job type.
3 - Find the job that you have this problem and press the button Reschedule in Actions column.
4 - Check if everything is correct and then press "Accept".

This would synchronize QUARTZ and the job to be executed.

Let me know if you have any issue or doubt.

Please close this ticket if this was solved.

Regards,
MP

#2

Updated by Vadim 6 months ago

Hello Mariano.

Thank You for the reply. It works. But schedule stops working after some time. And I don't understand what it depends on. System works stably, does not restart, no any other tasks or plans are being scheduled. But after several executions job plan is not executed automatically anymore. Maybe there is any possibility to fix it permanently? :)

Best regards,
Vadim.

#3

Updated by Rafael 6 months ago

  • Target version set to 5.0-FIX

Hi Vadim,

Have you identified any reason for this desynchronisation like the number of executions of the scheduled job, a failure on a deployment...?

As we haven't really been able to reproduce this, it would be very helpful for us any information you can find out.

Something useful for us could be the server logs when a job was supposed to be executed but didn't run, along with some information about what was the job and when was it supposed to be executed.

As this is very particular and very random, I'm not really sure how can I help you better. :-S

And please let me know if we can help you in any other way.

Cheers!

Rafa

#4

Updated by Vadim 6 months ago

Hi Rafael.

No, there are no any problems in system when task should be executed. No any failures. Job just does not start.
Which logs do You need web-app?
Project is quite simple - it runs single command to prune Docker images. Single command, three targets. If I run it manually - no any error occurs. Command can fail only if Docker is not working properly. Task scheduler to run daily at 23:00.

I attach web-app log as see there more errors that I expected. :) Take a look please and let me know what's wrong.

Best regards,
Vadim.

#5

Updated by Vadim 6 months ago

Forgot to write project name: system-server_cleanup_project(Id=25).

#6

Updated by Rafael 6 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Vadim to Rafael
#7

Updated by Vadim 6 months ago

Hi Rafael.

I rescheduled job plan yesterday, but it didn't run this night. Attached please find updated log.

Best regards,
Vadim.

#8

Updated by Rafael 5 months ago

  • Assignee changed from Rafael to Mariano
#9

Updated by Mariano 5 months ago

  • Assignee changed from Mariano to Rafael

Hey Rafa,

Could you please arrange a meeting with Vadim to see this issue? Any time is fine to me. I am 4 hours behind you both.

I could not get anything from logs.

Cheers,
MP

#10

Updated by Rafael 5 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Rafael to Vadim

Hi Vadim,

Please let me know when is a good time for you (Mariano is in Buenos Aires, 4 hours behind as he said) and I'll schedule a call for the three of us.

Cheers!

Rafa

#11

Updated by Vadim 5 months ago

Hi Rafael.

Sorry for the delay in reply. We can meet tomorrow (Tuesday) at about 14 or 15 o'clock by local time. Please let me know the best for You time.

Best regards,
Vadim.

#12

Updated by Mariano 5 months ago

  • Assignee changed from Vadim to Mariano
#13

Updated by Vadim 5 months ago

Hello.

Am not sure if I found a problem. But one error in all that setup I found. :) And that's really very fun thing. :) The scheduled job plan is not presented in a job plan list. I don't know how, but I remember that it disappeared after some update. I thought then that it was erased or something else hapend to it and created a new one. The difference is in id and one target. :) Schedule has link to an old job plan, but not a new one. Maybe that affects somehow the scheduler? I'll try to scheduler another one.

Best regards,
Vadim.

#14

Updated by Mariano 5 months ago

  • Assignee changed from Mariano to Vadim

Hi Vadim,

You are right, As I mentioned, for scheduled jobs, you have two main types, one is the schedule job that it meant to be executed once and the other type are the recurrent ones where same job is executed N times.

In the first case, when is time to execute the job and if something goes wrong you can see everything in the same job record (ID). But for recurrent jobs the history is completely different. When is time to execute a recurrent jobs, the scheduler clone the original scheduled job with same definition data and runs it as a new job record which have a different job id and the scheduled job. If any error happens on this new job, everything is logged into this new job record.

In other words, in your case, you have a recurrent scheduled job which runs a pipeline job. When is time to execute the job, the scheduler creates a new job based on the recurrent scheduled one which runs the pipeline persisted in the job. The new job will have their one log and record in RD. But the recurrent scheduled job will be in the same state as before.

Let me know if it is not clear enought or you have any doubt.

Cheers,
MP

#15

Updated by Vadim 5 months ago

Hi Mariano.

Thank you. Everything is clear.

Both job plans (old "1" and new one "2") were deployed successfully on a scheduled time. Will check on Monday if they still run. :)

Best regards,
Vadim.

#16

Updated by Vadim 5 months ago

Hi Mariano.

One of two scheduled jobs didn't run today. :( On a screenshot You can see that both were running over a weekend, but today one didn't start.

Best regards,
Vadim.

#17

Updated by Rafael 5 months ago

  • Assignee changed from Vadim to Mariano
#18

Updated by Mariano 5 months ago

  • Assignee changed from Mariano to Vadim

Hey Vadim, it is very weird.

Could you please check if there is any error in RD log by the expecterd execution time of those jobs?

Thanks,
MP

#19

Updated by Rafael 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Vadim to Mariano
#20

Updated by Rafael 2 months ago

Let's also check the mismatch in the timing shown in the scheduler and the system information.

#21

Updated by Mariano about 2 months ago

  • Tracker changed from Support to Bug
  • Status changed from In Progress to Resolved
  • Assignee changed from Mariano to Vadim

Hi Vadim,

The good news is that I found the bug, I could reproduce it. The prolem comes when you schedule or reschedule a job and the start date is before current datetime. This is a limitation of quartz framework. I amended our code to avoid this issue and this fix will be available on next release 5.0.27.

Meanwhile, to make your current cron scheduled jobs, you can reschedule them but set the start datetime in the future (one minute from the current datetime it perfect) and leave the same values to the rest of fields.

Let me know if it worked for you. If not, please assign this to me and add a note.

Regards,
MP

#22

Updated by Rafael about 2 months ago

  • Additional version details set to 5.0.27
#23

Updated by Rafael about 1 month ago

  • Status changed from Resolved to Marked for Closure

Hi Vadim,

Finally, version 5.0.27 of RapidDeploy is out with this fix included! :-)

Please let us know if you still see this issue with the upgraded version.

As this is a problem with scheduled jobs, it will take longer to check the fix, so I'll let more days to time this ticket out, but if we don't hear from you in a considerably longer period I'll happily close the ticket as resolved. :-)

Cheers!

Rafa

Also available in: Atom PDF