SOP's for Deployment and Management of Web Applications for Government Departments of Khyber Pakhtunkhwa

1 Introduction

This document defines the Standard Operations Procedures (SOPs) for the deployment and management of all government websites & customized applications used by the government of Khyber Pakhtunkhwa. This includes any applications which are used to interact between Government to Government, Government to Public or Public to Government. Document addresses applications previously developed & deployed or any new applications government departments intended to develop, deploy, manage or use.

This document will be used as a guideline for software development, deployment, and maintenance team. This will include operating procedures and requirement which each stakeholder will fulfill. This document also includes SOPs which needs to be furnished in the development, deployment, and management of the applications. The detailed SOPs for Integration of Cyber Security for web application development can be found in Appendix.

This will include a prerequisite checklist filled and signed by the development team to ensure the compliance with standards. This document also contains a list of requirement identified by the development team which needs to be furnished by the hosting provider and a checklist for the maintenance and operation team for the safety and security of the applications. SOPs for Datacenter have also included to benchmark the minimum compliance for each application.

Guidelines for maintenance teams are identified, listed and it is expected that the maintenance team will ensure that all these procedures are carried out on regular basis. Government websites are hosted at KP Datacenter and third-party hosting providers in and outside Pakistan. KP Datacenter is established as a common Web hosting infrastructure for the Govt. web applications.

As a standard guideline, it is recommended that all the applications/websites should be hosted within Pakistan preferably at KP Datacenter.

Confidential: The contents of this document are confidential and intended solely for the recipient. Reproduction of or forwarding to anyone is strictly forbidden.

1.1 Definitions

Unless the context otherwise requires, the following terms shall have the meanings hereby respectively assigned to them, that is to say,

“SOPs” means the Standard Operation Procedures.
“Datacenter” means the KP Data Center.
‘’Department” means the department of the government of Khyber Pakhtunkhwa.
“KPITB” means the Khyber Pakhtunkhwa Information Technology Board.
“KPCERC” means the KP Cyber Emergency Response Center.
“DIO” means Departmental Information Officer designated as a focal person of the respective department, who shall be managing the applications.
“Application” means the Website, Web Application, and MIS, Portal or Mobile application.
“ADR” means Application Deployment Request.

2. Pre Deployment

To ensure compliance with standard SDLC, it is recommended that an application should accompany all required documentation and is deployed only after it has been gone through preliminary phases of requirement, analysis, development, and testing.

Software deployment is a set of multi-dimensional activities related to benchmarking the deployment technologies, parameters, configuration and installation of the tested application.

It is expected that all required analysis, design and development documentation along with the code is available with the respective department, moreover it is also expected that the department holds and owns the code along with all other environmental technologies, this will include APIs, SDKs, libraries, licenses (if included in TORs), plugins (if purchased from third-party marketplace) or any other internal or external technologies which are required for the successful deployment and operations of the application.

Maintenance and operation team will ensure that all such requirements are furnished by the software development team.

Before requesting for the deployment of the application at KP Datacenter or external hosting within Pakistan, the department IT team should ensure the availability of above-mentioned items.

2.1 Testing

After the completion of development, the addressed software application must undergo a rigorous testing phase before it is deployed for public access. Recommended tests expected before the deployment are enlisted below:

Functional Testing: Ensures proper functioning of all the features of the application.
Usability Testing: Ensures user-centered interaction design of the application.
Code Review: Ensures the better code quality (readability, uniformity, understandability) and absence of any defect that can cause any performance problems, security vulnerabilities or injected malware.
Data Conversion Test: Ensures the accurate migration of appropriate legacy data.
Interfaces Test: Ensures proper functioning with all associated applications.
Security Test: Ensures the confidentiality, integrity, and availability of the application.
Performance Test: Ensures responsiveness under projected average and peak processing loads.
Restoration Test: Ensures full functioning of the application following an infrastructure rollback/restoration.
Regression Test: Applies exclusively to modifications of existing applications. Ensures that the new version does not compromise existing functionality.
Operating Platform Test: Ensures proper functioning of the application across all combinations of relevant hardware and software components.

2.2 Data Protection

The application must follow standard best practices for data protection preferably complaint with national/international data protection regulations.

2.3 Domain Name - Nomenclature

Departments shall align their domain names in support of top-level government domain (kp.gov.pk). It is recommended to use .gov.pk TLD for a better SEO & visibility. Moreover, if only static website or application is intended for information decimation and no MIS is hosted, it is recommended to use “departmentname.kp.gov.pk” or “application.kp.gov.pk” as preferred nomenclature.

For dedicated URLs, a set of guidelines for selecting a suitable domain are given below:

Choose a single short name (word) that establishes the identity of the department, provided that the name is not a common/generic name shared by two or more departments (e.g. establishment, health, labor etc.).
Choose abbreviations that are easy to remember. This is more appropriate for a department with permanent existence.
Prefix or suffix D or dept. or dir. (to denote department or directorate) and insert O (to denote of) within an abbreviation to make it more memorable.
Use shorten forms if it makes sense (e.g. meteo - meteorology, exams - examinations)
When there are unrelated/multiple subjects assigned to the department, choose the most important or well-known subject (which is less likely to change) for the domain name.
In the case of Divisions, Districts and Local Government entities where the area name is longer, shorten it in a meaningful and memorable manner.
Dash or underscore is not recommended.
Characters like $, @, % etc. are not allowed and shall not be used.
Any domain name that is not aligned with the above naming conventions will be subject to the review of the concerned department and KPITB.
The naming convention for applications hosted by departments should follow either of the following
“dept.kp.gov.pk/application” or
“mis.dept.kp.gov.pk”

2.4 Formal Request

A formal Application Deployment Request (ADR) in a prescribed form (Annexure A) will be submitted by the department.

3 Deployment

3.1 IP Pooling

The IP pool (a set of available IP addresses) will be set up as per requirement and will be shared with the DIO. IP addresses may be designated as either dedicated, meaning that the department becomes the only owner of this address, or shared, meaning that this address is shared among different departments or applications and services are mapped against the unique sockets.

3.2 Hosting

Web servers will be set up for hosting and managing of applications. Servers will be configured according to the requirements, which include a set of server modules, switching on / off web service and other application pool. A new virtual host may be added to the web server and new site’s directory structure may be created.

These directories are located in the corresponding virtual host directories:

On Linux: /var/www/vhosts/<domain_name>
On Windows: C:\inetpub\vhosts\<domain_name>

3.3 VPS / VPN

Access Virtual Private Server may be provided to the departments who required to host multiple application on the server or they required root access to the operating system. A separate VPS may be set up with required Operation System, CPU, RAM, and other system requirements.

Datacenter will be responsible for making sure VPS is running and is connected to the network. Operation and maintenance of the machine will be a responsibility of DIO. DIO will ensure that the machine is updated operating systems, software version and other dependent software. The data center may enforce to update the OS version or any other patch based on the technology/vulnerability status of the software.

Datacenter may provide the access to the VPS through VPN, credentials for which will be shared by the data center, it is the responsibility of DIO to ensure the privacy, security of the VPN credentials and datacenter will ensure the accessibility of the VPN services.

3.4 C-Panel

Access C-Panel will be used for the management of application at a web server. It provides a graphical interface and automation tools for hosting of the web application.

C-Panel can be accessed from following URLs or any other URL shared by Datacenter.

Beside many other features, C-Panel is used for uploading files, setting up domain names to hosting, Installing CMS and other third-party application, backing up application, checking bandwidth usage and setting up an Email account.

DIO will document all the settings enabled/provisioned on the C-Panel and may share these with KPCERC / Datacenter on request. If specialized bandwidth for each subdomain/application is needed, the required permits will be set up at C-Panel.

3.5 Content Management Systems (CMSs)

A reliable and updated version of the Content Management System (CMS) should be chosen along with updated and tested plug-ins. Ensure that the plug-ins are actively updated against the latest vulnerabilities or technological enhancements. Some plug-ins might be compatible with the CMS but are no longer in active development - this means that security patches are not performed and may increase the risk of vulnerability. These types of a plugin are not recommended to be used.

DIO along with the software development team must ensure that CMS or any other platform application is updated against the known vulnerabilities or software threats. KP CERC may issue advisories from time to time to address existing or upcoming vulnerabilities. To address the exiting software threats and vulnerabilities it is recommended to consult OWSAP top 10 and Common Weakness Enumeration CWE list.

3.6 Directory Structure

Proper file permissions should be maintained on site folder structure.

3.7 Admin credentials

Generate and safe keep complex credentials (must be more than 8 characters long, and must contain at least one uppercase letter, a number, and a symbol).

3.8 Third-Party Hosting

Few contents will require to be hosted outside the government IT infrastructure for example video on YouTube etc. It is recommended that all the government website /services/applications should be hosted in Pakistan only.

In the case of critical applications with PII and other critical data must ensure that the hosting provider is compliant with national data protection laws.

3.9 Web application firewall

Departments may opt for the Web application firewall (WAF) services; Datacenter may deploy the service if required. In the case of third part WAF, their CDN should be within Pakistan.

WAF is used to filter, monitor, and block HTTP traffic to and from a web application. By inspecting HTTP traffic, it can prevent attacks stemming from web application security flaws, such as SQL injection, cross-site scripting (XSS), file inclusion, and security misconfigurations.

3.10 Confidentiality, Integrity and Availability

As a policy, it is mandatory for all the applications to enable TLS/SSL services such as HTTPS. Ensure strong encryption of critical data at rest, especially for backups. Implement an in-depth vulnerability management program to identify and respond to threat.

It is recommended that at time of deployment and during the monitoring and management of the software application, DIO must ensure that application is updated against the possible software threats and vulnerabilities listed by OWASP top 10, Common Weakness Enumeration or CVE databases.

These frameworks compile the list of most common/successful hardware, software & platform vulnerabilities, along with their mitigation plan.

DIO is expected to execute or discuss the mitigation strategy against the potential vulnerability. DIO will ensure the integrity of data by periodically reviewing the application code, data, and logs to find an injection of bad data into the application.

It is recommended to use DDoS mitigation services designed to block attacks at the edge of the network. Hosting infrastructure should be robust & scalable and will be able to support if a traffic spike occurs.

4 Application Management

4.1 Logs

Logs are a critical part of any system, they provide the deep insights of application, what the application is doing and what caused the error if something wrong happens.

Application logs are invaluable data for:

Identifying security incidents
Monitoring policy violations
Establishing performance baselines
Assisting non-repudiation controls
Providing information about problems and unusual conditions
Contributing additional application-specific data for incident investigation which is lacking in other log sources
Helping defend against vulnerability identification and exploitation through attack detection Events are logged at an application level, system level and during the transaction at a network level, all level of logs should be enabled and accessible of monitoring.

#	Logs	Description	Responsibility
1	Application Logs	An application log is a file of events that should be logged by the software application. It should be set up during the development by a development team. It should contain errors, informational events, and warnings. The basic data of visitors including username and IP address should be logged for profiling. If an open-source application or third-party plugin is used, the logs should be enabled before use.	DIO, Development team
2	Database Logs	Database logs / Transaction are captured by Database Management Systems (DBMS), to establish integrity to data. It also maintains the history of actions executed by a database and crashes or hardware failures.	DIO
3	Application Server Logs	The server logs, captured by IIS log files for windows and Apache Access Logs for Linux and any other web server, should be enabled-by-default. They mainly capture and process the Information about the HTTP requests.	Datacenter, DIO (In case of VPS)
4	System Logs	System logs should be enabled and available when required. It records events that occur in an operating system or other software runs or messages between different users of communication software.	Datacenter, DIO (In case of VPS)
5	Network Logs	Network logs are maintained by datacenter and will be available on request. This records information about a network, including usage, performance, and utilization. It also tracks user`s activity, IP address over the network at different timestamps	Datacenter

All the responsible entities will retain the logs for the minimum 6-month duration unless otherwise specified. Log data must not be destroyed before the duration of the required data retention period. If logs are not accessible for monitoring, the trigger should be available to access the logs.

4.2 Monitoring

Web security strategies should be put in place. Real-time event log monitoring for critical security incidents and periodic analysis should be included as well to detect suspicious activity to provide a quick response.

Mechanisms to monitor security-relevant policies (e.g., authentication, authorization, etc.), activity (e.g., privileged user activity) and applications (e.g., IDS, IPS, firewall, etc.) in real time should also be put in place.

A specialized application performance monitoring software may be integrated within the primary application that is being monitored. It will provide runtime metrics of system/application performance, which are provided to the application administrator.

4.3 Updating

DIO is required to make sure everything on the application is still working properly. This can be achieved by means conducting a thorough audit of application (going through each page if possible) and checking for issues such as:

Plugins or themes that need to be updated.
Missing and/or poor-quality images.
Incorrect and/or outdated user information.
Broken links.
Broken features and/or plugins.
Formatting and/or style issues.
Missing and/or out-of-date content.

4.4 Backups

There shall be regular back-ups of all hosted content for the purpose of ensuring business continuity in case of failure. Regular offline backups of websites and databases should be performed at both onsite and offsite.

4.5 Business continuity and disaster recovery

Comprehensive business continuity and disaster recovery plan should be developed and reviewed periodically. DIO is expected to ensure BC & DR plan for its department software services.

4.6 Incident Management

Incident Management depending on the situation, incidents are handled according to several categories:

Cause Origin

Internal (equipment failure, data corruption, human error)
External (DDOS attack, site defacement, natural disasters, etc.

Method of Detection

Monitoring systems deployed by the department/datacenter.
Reported by persons inside of the department
Reported by persons outside of the department

Severity of Incident

Mild
Severe

Security-related incidents consist of, but are not limited to the following:

Site defacement: unauthorized content or pages are inserted upon viewing one or more pages of the website. This includes content that poses as a legitimate site of another but may contain malicious code.
SQL Injection: unauthorized content, functionality or users are added through the site's database due to an exploited vulnerability in the system.
Distributed Denial of Service: site response is slow or does not load at all due to a large amount of intentional traffic accessing the site.
Crypto-mining: unauthorized use of server resources to validate or add transactions in a blockchain digital ledger, usually harnessed through an exploited vulnerability in the system.
E-mail spamming: using the server's resources to send unsolicited emails to recipients through an exploited vulnerability in the system.
Root access: unauthorized remote access and control of server resources and processes through an exploited vulnerability in the system.

4.7 Vulnerability Assessment and Penetration Testing

The initial vulnerability scan does not guarantee that the website is 100% free from vulnerabilities - this only means that the site has been cleared of known threats and vulnerabilities from detection tools and methods available during the time scanning. Vulnerabilities and threats may still exist but have not been discovered and exploited yet as of scan time.