Feature #3577

Question regarding deployment organization

Added by Vadim about 1 month ago. Updated 28 days ago.

Status:ResolvedStart date:06/22/2020
Priority:NormalDue date:
Assignee:Vadim% Done:

0%

Category:-
Target version:RapidDeploy - 5.0-FIX
Affects Version:5.0-FIX Additional version details:5.0.22
Timesheet Code:

Description

Hello.

Were discussing with colleague the deployment pipeline and deployment organization in common. And found that cannot imagine how to configure deployment plan in a way we'd like to have it.

At this time our plan looks like this:
1. Automatic deployment to Development environment. Triggered from the outside (GitLab -> Jenkins -> RapidDeploy).
2. Manual deployment to Test environment. Should be triggered manually from RapidDeploy user interface by developer.
3. Manual package promotion to production. Should be triggered manually from RapidDeploy user interface by developer.
4. Automatic deployment to Live environment with approval from responsible person.
I attach diagram to the ticket.
Everything looks fine. But... Deployment could normally be applied on time when most of the workers are offline. And approvers also could not work on that time. That's why we thought that it could be better to have next plan:
1. Automatic deployment to Development environment. Triggered from the outside (GitLab -> Jenkins -> RapidDeploy).
2. Manual deployment to Test environment. Should be triggered manually from RapidDeploy user interface by developer.
3. Manual step for deployment to Live environment approval initiation. Should be triggered manually from RapidDeploy user interface by developer. Responsible person should get message that approval is needed for certain project build.
4. Approval from responsible person.
5. Automatic package promotion to production.
6. Manual deployment to Live environment. Should be triggered by some person, that is responsible to software updates, etc.

Can You please check if such scheme can be implemented? :)

Best regards,
Vadim.

firefox_LT1jLO3bYr.png (39.3 KB) Vadim, 06/22/2020 10:26 am

firefox_wENkVE3r8H.png (37.2 KB) Vadim, 06/25/2020 06:22 am

History

#1 Updated by Vadim about 1 month ago

Hi again.

Forgot to add some more thoughts.

If we have everything configured as now and Live environment deployment has to get approval, then it automatically requires approval for rollback. That's Ok when deployment finished successfully and somebody would like to get back previous version. Then it's normal to get an approval. But what if deployment to Live fails? As I understand in this case if I configure a failure branch with automatic ROLLBACK version deployment, then it will be necessary to get an approval for it. And if all responsible people are busy that leads to a broken Live environment that waits for somebody to approve a deployment. Or I'm nor right?

Of cause, I describe a very simple and therefore unstable system. Live environment should be duplicated at least and updates to both systems should be done separately. But the problem with rolling back on failure still exists. :) Even when having two parallelized Live system we can get one being waiting for approval.

I will tell You truth: I didn't try this. :) I just am analyzing existing plan diagram. Rollback to production has "Approval" block. Both: Rollback pipeline and Failure branch.

Will be waiting for Your comments. :)

Best regards,
Vadim.

#2 Updated by Vadim about 1 month ago

And one more question: is there any way to be informed about approval pending, approval and canceling? I mean, is it possible to send e-mail to:
1. Responsible person, who has to make approval. Mail should arrive when an appropriate step reached.
2. To user that initiated deployment. Mail should arrive when approved or canceled.
3. To systems administrator. Could be nice to get all copies. :)

Best regards,
Vadim.

#3 Updated by Mariano about 1 month ago

  • Status changed from New to Feedback
  • Assignee set to Vadim

Hi Vadim,

Well... at first impression it looks like the scenario described is doable. Let us know if you find anything stops you from doing it.

Regarding approval notification, it is currently implemented as you described, It sends emails to all groups users involved into approval workflow in every workflow step. If you have correctly configured the SMTP settings in rapiddeploy properties and the right email in users, they should receive all notifications. Please check always spam folder, in my case when I test these notifications, always them go directly to spam :S.

Regarding ROLLBACK version and approval workflow, I am afraid that when an environment is set with an approval workflow, always it needs to be approved, in normal, failure or rollback ways. But your point of view it is a very good point. On one hand as you said, when it comes to rollback, looks like the approval should be avoided due to the target will be rolled back to the latest successfully deployment. But on the other hand, the approval workflow is always set in critical environments where someone should be notified to any change agaist it, even for new features or rollback. So it looks like a controvertial topic. I will discuss about this with the team and let's see if we can improve it in some way.

Hope I could help you.

Thanks for contacting us, let us know if there is any further enquiry or issue.

Regards,
MP

#4 Updated by Vadim about 1 month ago

Hi Mariano,

Thank You very much for the reply!

Regarding the scenario we need - yes, I cannot implement it. :) The second one, where approval is get first and physically deployment goes later manually. So if shortly, I'd like to insert an additional manual step between approval and deployment.

That's great that mail works by default. I just forgot to setup SMTP in server settings. I'll do that and hope that everything will start working. :)

Regarding the Rollback. Yes, as You say, any changes in critical environments should be confirmed and all responsible persons should be informed. But the main point here is rollback on failure. An automatic operation. I do an approved deployment and something goes wrong, deployment fails and new version is not deployed. It could be great that in this case system takes pre-configured Failure branch and deploys previous (rollback) version automatically, without additional approval, as this version was already approved earlier. :) Just like pushing Ctrl+Z. :) All changes are removed and previous system version is running again. I think this is quite logic.
Rollback branch, as it was described somewhere in Your manuals, differs from a Failure branch. It is running when deployment was successfully implemented, but something is working in a wrong way. I think that this is Ok to get an approval here to get back to the previous version. This is change that human gets into as human decides that something is working not as planned. To minimize human factor it is necessary to get an approval. But when automatic process fails, then, I think, automatic process should also repair everything. Automatically. :)

Thank You one more time for Your help. Will be thankful to get instructions of how to make approval separately from deployment.

Best regards,
Vadim.

#5 Updated by Mariano about 1 month ago

Hi Vadim,

Sorry for not getting the issue. The approval workflow is currently supported by project jobs (deployments) running and target dictionary values changing.

So If I understood correctly, you need to use the approval workflow before package promotion and then continue a deployment job manually.

I think you can split the job plan, for example Run a job plan where you have set steps where run another job plans and project/promotion jobs between steps. Also you have tools to run job plans and project jobs out of a job plan context. You can create a project which uses approval workflow and set in the project tasks that run another project job or another job plan, and also there is a halt task simmilar to manual step in job plans but in project context.

So, based on your needs, I think you can create a project with a target that involves an approval workflow, there you can add a task that runs a job plan that do the package promotion with a manual step the waits for a manual confirmation to continue and run a project job to production.

Regarding failure handling without approval workflow, you have failure branches feature in project orchestration context where you can set tasks in the failure branch to amend or rollback changes made before. This does not involve approval workflow due to it happens in the same project target context.

Hope this helps, if not, I would recommend set a web meeting with you, Rafa and I to clarifies all doubts.

Regards,
MP

#6 Updated by Rafael about 1 month ago

  • Target version set to 5.0-FIX

Hi Vadim,

You're really pushing our tool to the limit! Thank you very much for all your feedback! :-)

I'm trying to understand the problem but I'm not really sure if I do. I think the best thing here would be to hold a meeting and try to explain us the problem directly, what do think?

Just let me know your availability for tomorrow and I'll set it up. And please remember Mariano is in Buenos Aires, so any time after 3 PM would be better. :-)

Cheers!

Rafa

#7 Updated by Vadim about 1 month ago

Hi Mariano, hi Rafael. :)

Thank You very much for Your support! We are just trying to implement everything we need using everything You provide. :)

Sorry that didn't reply earlier - we had a national holiday on Wednesday and I was very busy at home.

Regarding the Mariano's suggestions. I think I understand why it's not so simple to separate approval from a deployment. And as I understand, Mariano offers to move most of the deployment works to the project plan (or several plans). Don't know if that's the best solution. Maybe just don't see all possibilities of such approach.
I tried to configure an additional project that have to get an approval before the package promotion and deployment to a critical environment. That could solve the problem but in this case nobody knows which project is really being deployed as approval information contains data about the project that should be approved. :) It can be hard to explain.
I mean I have project A, which is a real project with a real deployment operations. And I created also project B which does nothing, it just has to get an approval. So I deploy project A to Test environment, then deploy project B to critical environment and this step has to get an approval. And next I just deploy project A without any additional approvals as a whole pipeline is approved. But in this case approvers get an information about the project B only.
Ok, that's quite tricky really. :) We can discuss this online. Really we could have a call today at 4 PM (local time, so it's 3 PM by Madrid time). Or on Monday at the same time, as on Friday we have a shortened day.

Regarding the failures handling. I'd like not to handle failures in a project plan. Our projects are just single service version deployment. So rollback is just a previous version deployment and nothing else. Think that it has no sense to allow project to do too much (direct and reverse deployment). Also there are problems with getting out a rollback version inside a project. But all these problems are automatically handled by RapidDeploy on a job plan layer. The only problem I see with approval on a Failure branch. Just as I wrote it earlier. If we have a job plan as attached then it is necessary to approve direct deployment (what is Ok), but rollback on failure also should be approved (what is not Ok).
We can discuss this also. I think that this place can be even more critical.

By the way, Rafael, we can have a short meeting without Mariano (due to huge time difference) and I could explain what I try to describe here so that You could talk to Your colleagues and Mariano as well. And next week we can have a common call already.

Please let me know what's the best strategy to move forward. :)

Have a nice day!
Best regards,
Vadim.

#8 Updated by Rafael about 1 month ago

Hi Vadim,

Let's have a call in 10 minutes if you're available!

Here are the details:

Topic: Catch up with Vadim
Time: Jun 25, 2020 03:30 PM Madrid

Join Zoom Meeting
https://us02web.zoom.us/j/89326812443

Meeting ID: 893 2681 2443
One tap mobile
+13017158592,,89326812443# US (Germantown)
+13126266799,,89326812443# US (Chicago)

Dial by your location
+1 301 715 8592 US (Germantown)
+1 312 626 6799 US (Chicago)
+1 346 248 7799 US (Houston)
+1 646 558 8656 US (New York)
+1 669 900 6833 US (San Jose)
+1 253 215 8782 US (Tacoma)
+44 203 051 2874 United Kingdom
+44 203 481 5237 United Kingdom
+44 203 481 5240 United Kingdom
+44 131 460 1196 United Kingdom
+54 341 512 2188 Argentina
+54 343 414 5986 Argentina
+54 112 040 0447 Argentina
+48 22 398 7356 Poland
+48 22 307 3488 Poland
+34 91 787 0058 Spain
+34 917 873 431 Spain
+34 84 368 5025 Spain
Meeting ID: 893 2681 2443
Find your local number: https://us02web.zoom.us/u/ketPuKYPNR

Cheers!

Rafa

#9 Updated by Rafael about 1 month ago

Hi Vadim,

Sorry for this hurried invitation! I guess you were already busy with something else! :-)

Let's keep a conversation tomorrow morning at any time even though Mariano will probably not be able to join. I already had a chat with him and I think we have a solution for what you need.

I've defaulted to 13:00 (your local time), but let me know if any other time is better for you! Of if you want to change it to Monday. Just use the same call details in the comment above.

Cheers!

Rafa

#10 Updated by Vadim about 1 month ago

Hello Rafael.

Sorry, was a little bit busy and didn't see Your invitation. Maybe we can have short conversation tomorrow?

Best regards,
Vadim.

#11 Updated by Vadim about 1 month ago

Hi Rafael.

That's Ok. 13 by local time is good. Will join Your call. :)

Have a nice day!

Best regards,
Vadim.

#12 Updated by Rafael about 1 month ago

  • Tracker changed from Support to Feature
  • Status changed from Feedback to In Progress
  • Assignee changed from Vadim to Mariano

Little summary about today's conversation

  • We need to debate about having a new "approval" step in a job plan. It would be something similar to a "manual" step but with approval history logging as well. It is like having a "generic" approval of the environment in question, not an approval to deploy, but an approval of consistency of the current environment state by the responsible person.
    • We need to determine the difficulty and feasibility of adding an email notification for a "manual" step.
  • We also need a discussion about the need of approval for a failure or a rollback branch in a job plan for certain environments that need approval. It is true you can handle a failure scenario inside a project, but you can not handle different package versions inside the project, that needs to be done at job plan scope.

#13 Updated by Mariano about 1 month ago

  • Status changed from In Progress to Feedback

#14 Updated by Mariano about 1 month ago

Hi Vadim,

We have discussed about this. We decided to not to change the currnt approval workflow for jobs due to this feature must prevent users from making any change directly to the environment.

But we agreed that we can add a new approval step feature in job plans, so users can manage to set an approval workflow as we do with jobs but in the job plan context.

This will be developed and included in next release 5.0.22.

#15 Updated by Vadim about 1 month ago

Hi Mariano,

Great! Extremely good solution. System will have two different approval possibilities. I think that there can be situations when both approvals are in a single pipeline. :)

Really nice! Thank You very much for a good news!

Best regards,
Vadim.

#16 Updated by Rafael about 1 month ago

  • Status changed from Feedback to In Progress

#17 Updated by Mariano about 1 month ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Mariano to Rafael

Rafa, this new feature is added into the HEAD. Please update and rebuild your workspace and do any testing to detect any possible bug.

Saludos y gracias.
MP

#18 Updated by Mariano about 1 month ago

  • Status changed from Resolved to In Progress
  • Assignee changed from Rafael to Mariano

#19 Updated by Mariano 29 days ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Mariano to Rafael

Hi Rafa,

Please update your workspace and try this out all new features for approval step.

I added also in deployment approval, a new feature to add individual approval request reason for each job.

Cheers,
MP

#20 Updated by Rafael 28 days ago

  • Assignee changed from Rafael to Vadim
  • Additional version details set to 5.0.22

Hi Mariano,

After a thoroughly test this seems to be working as a charm! :-)

Vadim, this feature will be released with RapidDeploy 5.0.22 along with a few other new features and fixes.

I'll leave this ticket open until the new version is out and we can get a proper feedback from you.

Cheers!

Rafa

Also available in: Atom PDF