Shavlik Protect 9.2 Deployment Process Workflow and Troubleshooting

Version 16

    Purpose

     

    This document outlines the 9.2 deployment workflow and provides common troubleshooting tips.

     

    Deployment Workflow

     

    The following happens as soon as you choose to deploy patches from the Protect console:

     

    Verifying Patch Downloads:

    • Protect verifies all required patches are downloaded.
      • Protect will automatically download patches if needed.
      • If you are using a Distribution Server, Protect will not verify the patches are there. You can manually sync your Distribution Servers by Tools > Operations > Distribution Servers > Run Now.

     

    Creating Deployment Packages:

    • Protect will create deployment packages for each machine (deployPackage-<%DeployID%>.zip File) under the user's temp folder. %temp%\<%Generated GUID%
      • The deployment package comprises of patch information from WindowsPatchData.zip and the deployment instruction from the deployment template.
      • A brief explanation of the deployPackage-<%DeployID%>.zip structure:
        • coreDeploymentOptions.xml: Contains general tab information.
        • externalIdentifiers.xml: The machine and deployment ID and core version to use.
        • machineManifest.xml: Machine Name and what patches to install.
        • nonCoreDeploymentOptions.xml: Contains Shavlik Protect Tracker information, Reboot and Distribution Server options.
        • patchSpecifiers.xml: The patch to deploy information. You can match up the patches from the machineManifest,xml by using the uid

     

    Copying Files To The Target:

    • Protect copies the deployment package and all supporting binaries out to the target machine's C:\Windows\ProPatches\Staged folder.
    • Patches are copied to the C:\Windows\ProPatches\Patches folder if you are not using a distribution server. If you are using a distribution server, the patches are copied to the Patches folder when the deployment occurs.
    • Target machine's folder structure explanation: C:\Windows\ProPatches\
      • \Installation: Where the deployment actual runs from. A installation sandbox folder is create when the deploy executes. It appends the date and time when the deployment started.
      • \Logs: Where all the logs are located. The only exception is STDeploy.log. It starts out in the Staged folder and contains the information about creating installation subfolders and moving STDeploy.exe to the installation subfolder. Once that has been completed, it move the tracing to the Logs folder and starts logging there.
      • \Patches: Where all the patches are located and executed from with the exception of patches that require a dependent action install scripting. Those patches are copied to the Installation sandbox.
      • \Scheduler: Where the Shavlik and Microsoft interface executable and log are located.
      • \Staged: Where deployment files are first copied to. The subfolder name is the date time the deployment was created. It will different from the installation sandbox folder date time.

     

    Deployment Scheduling:

    • All deployment jobs are scheduled on the target machine, even if ran immediately.
      • Deployments are scheduled using the Shavlik Scheduler by default or Microsoft Scheduler depending on options set in Tools > Options > Scheduling in the Protect console.
      • The Microsoft Scheduler is used as a fail-over if scheduling with the Shavlik Scheduler fails.
      • If you want to manually start the deployment on the target machine, execute the InstallPatches-<%DeployID%>.bat file from the Staged folder.  It will schedule the deployment to run in 3 minutes.

     

    The Deployment Process On The Target Machine:

    • All deployment jobs start from the Staged folder. Files are moved from the Staged folder into the Installation Sandbox folder.
      • STDeploy.exe starts the deployment.
      • The Installation sandbox subfolder is created and STDeploy.exe is copied to it.
      • STDeploy.exe is launched from the Installation sandbox folder.
      • All files from the Staged folder  are copied to the Installation sandbox folder.
      • The deployment package deployPackage-<%DeployID%>.zip is loaded.
      • All deployment related tasks for this deployment are removed from the Shavlik Scheduler and Microsoft Scheduler.  This is to prevent duplicate tasks from running.
      • dplyevts.dll is loaded, this is used to send Deployment Tracker information to the console.
      • STDeployerCore.dll, this is what actually does the patch installations.
      • nonCoreDeploymentOptions.xml is checked to see if Distribution Server are being used. If so, it downloads the patches.
      • STDeploy.exe starts deploying patches by looping through the list of patches. It also sends a Deployment Tracker start message to the console and calling STDeployerCore with the patch information.
      • STDeployerCore executes the patch, records the raw return code and cooked result and returns to STDeploy.exe. The cooked result is what is set in the Shavlik Breadcrumb.
      • STDeploy.exe sends a Tracker message with the appropriate status and error codes back to the console.
      • Once all patches have completed, STDeploy.exe determines if a reboot is needed based off of the STDeployerCore results and nonCoreDeploymentOptions.xml  options.
      • If a reboot is required:
        • After patching is completed, the post reboot jobs and safe reboot are scheduled.
          • STDeploy.exe creates the PostBootTasks.xml which contains patch post boot jobs, this updates the Shavlik Breadcrumb and sends a completed status to the Deployment Tracker.
          • Schedules the Post Boot Job to run at start up.
          • Schedules the reboot to occur 3 minutes out.
        • After the reboots occurs, the post boot job starts, this executes the STDeploy.exe to execute everything in the PostBootTask.xml.
      • If there is no reboot required:
        • The Shavlik Breadcrumbs are updated. (breadcrumbs are Deployment Tracker statuses)
        • A completed status is sent to the Deployment Tracker.
      • If there is still a sub folder in the Staged folder then most likely there is a scheduler issue.
    • After reboot is complete a re-scan request is sent to the console to verify the patch install was successful.
    • At this point the Patches are installed (or failed) and the Deployment Tracker is updated with the final status.

     

    Troubleshooting Deployment Issues

     

    Patch Download Issues:

    • For any patch download errors you see in the UI, you can see more information (including the download URL) in the ST.Protect.managed.username@ProtectServerName.log located in the C:\ProgramData\LANDESK\Shavlik Protect\Logs folder.
      • You must have logging set to All in Tools > Options > Logging to see download errors.
    • Searching for Downloading From in the log will make locating these issues much easier.
      • You can attempt to manually download the patch on the Protect server to see if any helpful errors are returned.
    • Many download issues can be attributed to Firewall and Proxies blocking traffic.  Exceptions are often needed to allow the downloads: Shavlik Protect Firewall and Proxy Exceptions Url List

     

    Common Deployment Failures:

    • There are times when a Patch Scan completes, but the Deployment fails: The most common fix for this issue is outlined in this document: Scan Complete Successfully but Deployment Fails Due to Bad Credentials Set in Machine Properties
    • Routine routing issues and common causes:  Here are some test to perform from the Protect server to the target machine:
      • nslookup by FQDN, NetBIOS and IP Address.
      • net use \\machine\IPC$ from a CMD Prompt.  If prompted, supply the credentials used for the Scan\Deployment (use FQDN, NetBIOS and IP Address and record what works)
    • More error information on deployment issue can be found in the ST.Protect.managed.username@ProtectServerName.log located in the C:\ProgramData\LANDESK\Shavlik Protect\Logs folder.

     

    Deployment Issue Caused By Scheduler Issues:

     

    Deployment Issues From the Deployment Tracker:

    • The Deployment Tracker dialog provides at-a-glance information pertaining to patch deployment status. Each line in the dialog indicates a single task and includes information about the task's current state, the machine affected by the task, a description of the task, when the task is scheduled to be started, and the time that the last status information was collected.
    • You can use the check boxes near the top of the dialog to specify what state information is displayed. The state information can help you to begin troubleshooting possible problems.
      • Show failures: A patch deployment didn't fully take and more research is necessary. The Operations Monitor may provide additional information if one of the main steps in the deployment process failed.
        • One of the more common reasons for seeing a "Failed" item in Deployment Tracker is because a patch that requires a reboot to complete was deployed but 'Do Not Reboot' was specified in the deployment template. If you receive a "Failed" status in Deployment Tracker, check the Patch Details for the patch in question to see if a reboot is required to complete the installation of this patch.
      • Show in progress: A patch has not completed installation.  If the status remains yellow, it could be an indication that the remote machine cannot communicate back to the Deployment Tracker.
      • Show successfully completed: The task was successfully implemented.
    • Deployment Tracker Stays at Executed or Scheduled Status:  Deployment Tracker Stays at Executed or Scheduled Status
    • Deciphering Shavlik Protect Deployment Tracker Status Messages:  Deciphering Shavlik Protect Deployment Tracker Status Messages
    • Deployment Tracker Status - Unable to Verify:  Deployment Tracker Status - Unable to Verify or Complete (not verified)

     

    Client Side Logs:

    • The client side deployment logs are located in the C:\Windows\ProPatches\Logs folder:
      • CL5.log: Operations of CL5.exe, used for handling deployment/patch installation process.
      • deplyevts.log: Operations of the Deployment Tracker events.
      • SafeReboot.log: Operations of SafeReboot.exe:
        • Determines if a user dialog is displayed.
        • Logs commands during deployment process.
      • STDeployerCore.log:  Operations of install switches and patch install return codes.
      • STDeploy.log  Operations of various information about the deployment.

     

    Additional Items To Consider:

    • Patches often require a reboot to be considered installed, verify the machine rebooted.
      • A failure to reboot could indicate a Scheduler issue.
        • For Scheduler issues, take a look inside the Scheduler.log located in C:\Windows\ProPatches\Scheduler folder.
          • Do you see errors around the time the deployment job was suppose to run? (the time is in UTC)
      • It's possible a hung patch isn't allowing the deployment to complete. Look in the Task Manager of the target machine for any patch install processes.
        • Killing the process will allow the deployment to continue so be careful doing this during production hours

     

    Affected Product(s)

     

    Shavlik Protect 9.2.x