February 2020

Department: Information Security

Document ID:  05.4.0.1




Proprietary Statement

This document was developed specifically for and by Christopher Newport University.  The concepts and methodologies contained herein are proprietary to Christopher Newport University.  Duplication, reproduction, or disclosure of information in this document without the express written consent of Christopher Newport University is prohibited.

All Trademarks, Registered Trademarks, Service Marks, and brand and product names used in this document are the property of their respective owners.

© Copyright 2020 Christopher Newport University. All rights reserved.

Review and Revision History

Date

Version

Description of Change

(Affected Sections)

Author

February 2020

1.0

Initial Release

Wendy L. Corrice

February 20211.0Annual ReviewWendy Corrice









CNU ITS Server/Desktop OS Patching Procedures

Introduction

Windows Desktop Patching Procedures

Windows Server Patching Procedures

Linux Server Patching Procedures

Emergency Patching Procedures

Banner Application Patching Procedures

Tomcat Patching Procedures

References

Screenshots

Desktop Patch Management Console Screenshot

Desktop Patch Management Approval Screenshot

Desktop Patch Management Documentation Screenshot

Introduction

The purpose of the Server Operating System (OS) and Desktop OS Patching Procedures are to document the management and installation of OS patches. Proper patch management reduces the risk of security breaches and possible system disruptions.

Windows Desktop Patching Procedures


Communication: 

  • Production: A regular reminder message is posted by a member of the Systems & Support team via CNU Connect and email, when necessary. Windows Patching Announcement 
  • Testing: The patch manager (usually a member of the Systems & Support team) uses Slack messages or an email list to communicate when patches are deployed for testing.
  • Problems: Users are expected to create LANDesk tickets reporting issues with patches. A member of the Systems & Support team will follow up to gather more information. 


Patch Management Console:


Test Workstation Patch Deployment:

  • On Monday after patches are released by Microsoft, patches are approved by the patch manager in ITS (usually a member of the Systems & Support team) for deployment to test systems in the WSUS group “WSUS Testing ITS” (internal ITS workstations and other machines identified by campus volunteers willing to observe and report problems with Windows workstation patching).
  • Patches install on test systems Thursday morning at 4 AM, including reboot (if required). Any issues identified by ITS staff or testing volunteers are identified to ITS staff for troubleshooting and remediation and tracked via LANDesk helpdesk tickets. 


Production Workstation Patch Deployment:

  • The following Monday, patches are approved by ITS (AD for Systems and Support) for deployment to all workstations on the domain. WSUS reflects patch approval (it is self-documenting). 
  • As per Active Directory group policy, patches install on all remaining applicable workstations Thursday morning at 4 AM, including reboot (if required).


Verification/Rollback

  • WSUS verification that patches have been deployed and data supporting installation on workstations is retained for at least three years (successful/unsuccessful deployments).  
  • If an update fails/proves unsuccessful on a workstation, a ticket is created in LANDesk to track the issue/resolution.
  • Rollback is done on a case-by-case basis and usually involves uninstalling the patch in Windows safe mode, by a member of the Systems & Support  staff. It is important to note that not all patches support rollback and a failed patch is not necessarily indicative of a fleet-wide problem. 


Windows Workstation Patch Documentation

Windows Server Patching Procedures


Communication: 

  • Production: A regular reminder message is posted by a member of the Systems & Support team via CNU Connect email, when necessary. 
  • Testing: The Systems team uses Slack messages or email mailing lists to communicate when patches are deployed for testing. 
  • Problems: Users are expected to create tickets reporting issues with patches to be investigating by a member of the Systems & Support team.


Patch Management Console:

  • Windows patches are released by Microsoft and appear in Windows Server Update Services (WSUS - see description below) located on AS55PV.cnu.edu.


Test Server Patch Deployment:

  • On Monday after patches are released by Microsoft, patches are approved by the Systems team for deployment to test servers “WSUS Testing ITS” (non-customer facing servers maintained by ITS for testing purposes and other internal non-production purposes).
  • Patches install on test servers Friday mornings at 4 AM, including reboot (if required). Any issues identified by ITS staff for troubleshooting and remediation. 


Production Server Patch Deployment:

  • The following Tuesday, patches that are installed on the development servers are automatically deployed to all remaining ITS servers Friday morning at 3 AM, including reboot (if required). 


Verification/Rollback:

  • WSUS records evidence and verification that patches have been deployed and reports are retained for at least three years (successful/unsuccessful deployments).  
  • Rollback is done on a case-by-case basis and usually involves uninstalling/repairing the patch in Windows safe mode, by a member of the Systems & Support staff. It is important to note that not all patches support rollback and a failed patch is not necessarily indicative of a fleet-wide problem. 


Exception procedure:

  • A member of the Systems team applies patches manually using the individual server’s Windows Update application and reboots manually. Examples would include systems on segregated networks (PCI compliance) or systems where the standard reboot time isn’t acceptable. 
  • On systems where patching is not advised or prohibited by the vendor, those servers are segregated in WSUS to avoid patches accidentally applied. Staff supporting the system(s) affected are expected to follow vendor advice and alert the Systems & Support team when patches are confirmed safe to apply.


Windows Server Patch Documentation:

  • All patch deployments are logged in the WSUS console.
  • Exceptions to the server patching procedures will be reviewed by the ISO and Director of Systems and Support and documented accordingly upon approval.


Linux Server Patching Procedures


Patch Management Console:

  • Patches are released by Red Hat and appear in the Red Hat Satellite update interface. The Red Hat Satellite system continually monitors all RHEL systems for patch availability and provides administrators an overview of patching needs on individual RHEL systems. 


Test Linux Server Patch Deployment:

  • If patches are available, they are installed on test servers hosting test or development versions of applications. After installation, the servers are rebooted, if applicable. The Systems team monitors the testing server group and advises the AD for Systems and Support, if there are any issues. Without any reported issues, the patches are ready to be applied in production.


Production Linux Server Patch Deployment:

  • On the fourth Friday of the month, the same patches installed in the test group are installed on production servers. After installation, the Systems team reboots the servers, if necessary. These reboots take place during the ITS maintenance window of Fridays from 6-9 PM. 


Verification/Rollback:

  • ITS systems staff will review patch installation reports locally on the patched RHEL server.
  • Rollback is done on a case-by-case basis by a member of the Systems team. 


Linux Server Patch Documentation:

  • All Linux server patch events are tracked and logged in the Red Hat Satellite system. 


Emergency Patching Procedures

Based on consultation with the ISO and based on probably of exploit/risk, the Systems team may apply emergency patches outside of the standard patch cycle. These patches follow an abbreviated version of the standard process.

  • Communication: Impacted users are notified via email that test and production systems will be patched out of cycle.
  • Test Server Patch Deployment: Test systems are patched immediately. 
  • Production Server Patch Deployment: Production system is patched at the earliest available time. 


Verification/Rollback:

  • ITS systems staff will review patch installation reports locally on the server or WSUS depending on the OS and update the server patching.

Rollback is done on a case-by-case basis by a member of the Systems team. It is important to note that not all patches support rollback and a failed patch is not necessarily indicative of a fleet-wide problem. 

Patch Documentation:

  • All Linux server patch events are tracked in Red Hat Satellite by ITS staff, which is monitored monthly by the AD for Systems and Support.


Banner Application Patching Procedures


When Banner patches are requested by data stewards or a critical patch is released, they are installed in the test instance of the application as soon as practical. The requesting or affected user group is then notified and it is requested they thoroughly test their business processes to ensure that application or the new functionality is working as intended.  Once two weeks have passed with no issues reported, the patches are applied in production. Internal documentation regarding this process is maintained on confluence.

  • Communication: Impacted users are notified via email that test and production systems will be patched out of cycle.
  • Test Server Patch Deployment: Test systems are patched at the earliest available time after users have been notified. 
  • Production Server Patch Deployment: Production system is patched at the earliest available time, at least two weeks after lower environments. 


Verification/Rollback:

  • By necessity two different types of verification must occur with each Banner patch or upgrade.
    • ITS Systems staff verifies that logs, applications, and Ellucian Solution Manager all reflect the patch has been installed correctly and component version numbers are correct
    • End users must verify appropriate functionality within their purview. 
  • Rollback is done on a case-by-case basis by a member of the Systems team. 


Patch Documentation:

  • Patch and upgrade events for Banner and its components are tracked via the Ellucian Solution Manager and are also available via querying the databases and the GUAABOT form within the Banner application itself.


Apache Tomcat Application Patching Procedures


When Tomcat patches are requested by data stewards or a critical patch is released, they are installed in the test instance of the application as soon as practical. The requesting or affected user group is then notified and it is requested they thoroughly test their business processes to ensure that application or the new functionality is working as intended.  Once two weeks have passed with no issues reported, the patches are applied in production. Internal documentation regarding this process is maintained on confluence.

  • Communication: Impacted users are notified via email that test and production systems will be patched out of cycle.
  • Test Server Patch Deployment: Test systems are patched at the earliest available time after users have been notified. 
  • Production Server Patch Deployment: Production system is patched at the earliest available time, at least two weeks after lower environments. 


Verification/Rollback:

  • By necessity two different types of verification must occur with each Tomcat patch or upgrade.
    • ITS Systems staff verifies that logs, applications, and Ellucian Solution Manager all reflect the patch has been installed correctly and component version numbers are correct
    • End users must verify appropriate functionality within their purview. 
  • Rollback is done on a case-by-case basis by a member of the Systems team. 


Patch Documentation:

  • Patch and upgrade events for Tomcat and its components are tracked by the Systems Team.


Update Tomcat Step-by-Step

Here is how to update Apache Tomcat on a server already running it.

These instructions use the Banner ESM server as an example. Update the referenced commands as needed for your server.

***If you already have separate configs, jump to Step 5.

  1.   Take a snapshot of Virtual Machine as a backup if applicable.
  2.   Create new directories to act as config directories (your server may already have this step completed): a.  #>mkdir -p /u01/app/tomcat/config/esm
  3.   To ensure ease of maintenance, move config from binaries that will act as CATALINA_BASE
  4.   Again this may already be done but if it's not, move Tomcat configs to your new folder.
  5.   #>cp r /u01/app/tomcat/apache tomcat-8.5.30/conf /u01/app/tomcat/config/esm
  6.   #>cp r /u01/app/tomcat/apache tomcat-8.5.30/logs/u01/app/tomcat/config/esm
  7.   #>cp r /u01/app/tomcat/apache tomcat-8.5.30/temp/u01/app/tomcat/config/esm
  8.   #>cp r /u01/app/tomcat/apache tomcat-8.5.30/webapps/u01/app/tomcat/config/esm
  9.   #>cp r /u01/app/tomcat/apache tomcat-8.5.30/work/u01/app/tomcat/config/esm
  10.   Rename old dirs to avoid confusion (not neccessary but could help you not look in the wrong directory):
  11.   #>mv /u01/app/tomcat/apache-tomcat-8.5.30/conf /u01/app/tomcat/apache-tomcat-8.5.30/oldconf_goto_u01apptomcatconfig
  12.   Create link from /u01/app/tomcat/latest to updated version of tomcat to make upgrades to tomcat easier since now the configs will be separate:
  13.   Unzip new Apache Tomcat zip into /u01/app/tomcat (for this example we are upgrading Tomcat 8.5.30 to 8.5.55 b.  #>rm /u01/app/tomcat/latest
  14. #>ln -s /u01/app/tomcat/apache-tomcat-8.5.50 /u01/app/tomcat/latest
  15.   Change /home/tomcat/.bash_profile to set environment variables, add the following:
  16.   #>export CATALINA_HOME=/u01/app/tomcat/latest
  17.   #>export CATALINA_BASE=/u01/app/tomcat/config/esm
  18.   Edit any tomcat start/stop scripts to point to new $CATALINA_HOME (if needed):
  19.   Add export CATALINA_HOME and CATALINA_BASE statements
  20.   Change paths if needed such as if referencing any specific instance of Tomcat (apache-tomcat-8.5.30) to point to your symbolic link /u01 /app/tomcat/latest






Notes

Windows Server Update Services (WSUS), previously known as Software Update Services (SUS), is a computer program developed by Microsoft Corporation that enables administrators to manage the distribution of updates and hotfixes released for Microsoft products to computers in a corporate environment. WSUS downloads these updates from the Microsoft Update website and then distributes them to computers on a network. WSUS runs on Windows Server and is free to licensed Microsoft customers.

Patching procedures will be reviewed on an annual basis.


References

Screenshots

Desktop Patch Management Console Screenshot

Desktop Patch Management Approval Screenshot

Desktop Patch Management Documentation Screenshot 

Individual Windows Server Patch History Screenshot


Next Review: April 2022