Checklist of Questions for an Enterprise Software Development Vendor

If you are interviewing a vendor or issuing an RFP to develop custom software for the Enterprise, there are numerous program, training, support, architecture, and security requirements that any Enterprise-grade system should have. Here is a checklist of questions to probe any vendor about the maturity of their enterprise software development approach end-to-end.

Project Plan

• Provide a project plan for the complete implementation, with proposed schedule indicating all phases, milestones and deliverables.
• Identify key project resources that would be assigned, describing their role, skills, relevant project experience, and certifications.
• Clearly state dependencies on our (client) resources.
• Identify any tools that would be shared between the service provider and the client for the purpose of tracking project progress (for example, project management, time tracking, bug tracking, task lists, collaboration spaces, scrum boards, etc)
• Describe the the proposed governance structure for project implementation.
• Describe your project management methodology and how it would guide the implementation of this project between the service provider and the client.
• Provide an overview of your scope management and change control process that allows for a flexible, "fail fast" approach in the face of significant expected change.
• Provide a risk register of the top 5 risks your project manager will be tracking from the start of the implementation project, along with the likelihood, impact, and mitigation strategy for each one.
• In your proposal, describe where the project resources would ideally be located (e.g. co-located at the client's head office, or elsewhere globally).
• The client’s preference is that proposed resources be available for the duration of the project. Describe how resourcing risk will be mitigated and, if necessary, how changes to resourcing will be made in the event that key resources become unavailable.

Support and Training

• Describe your warranty, including which team members from implementation are involved in warranty, and what the defect resolution times would be.
• Once the IT system is in production, the service provider will be responsible for tier 1-3 support. The client's own call centre would not be involved at any level of support other than redirecting calls about the IT system that they receive to the service provider.
• A live call centre must be open from XX to YY eastern time on all Client working days. Incident resolution including resolution of any code defects must meet the following SLAs once the IT system is live:
  • 4h resolution - Outage or degradation of critical service affecting many users.
  • 12h resolution - Single user or small number of users’ productivity is affected, they cannot work. 
  • 48h resolution - User's productivity not affected, a workaround is available and work can continue.
• What inputs and considerations would you use to determine how to stand up the Support initially, to meet the above SLAs and at the same time minimize costs to the client?
  
• What KPIs are monitored regularly to ensure support (call centre, incident resolution, defect resolution) SLAs are met, while minimizing cost to the client?
  
• What governance structure is used for the Support phase? Note this may be different from Governance for implementation.
  
• What other support mechanisms do you recommend beyond live call centre support? (IVR, online chat with an agent, chatbots) Please list any optional costs separately.
  
• How does the model scale to meet increased and decreased volume of inquiries and defects?
  
• What processes and/or agreements are in place with subcontractors to ensure subcontractors adhere to SLAs described?
  
• If a critical defect is ever at risk of not being resolved within the SLA, what processes are followed to ensure the fastest resolution time possible?
  
• When a code defect is fixed in support, what testing is done to ensure no regression is introduced in the IT system?
  

Release Management

• For each software package, how frequently are updates pushed to the client's IT system?
  What processes do you follow to ensure users of the IT system are informed of upcoming changes to the IT system, and trained on them?
  What processes do you follow to ensure no defects are introduced by new releases?
• Describe your approach to implementing training. How long will it typically take for a user to become proficient in using the IT system? Please discuss all aspects of the training and knowledge transfer plan.

Roadmaps

• Please provide the product roadmap for all software packages included in the IT system, including software components. Are you aware of plans to sunset any of the proposed components? If so, please explain the reason for using them and what will be done to replace them once they are no longer supported.
• Please describe how you track these components’ lifecycle so that you can proactively replace ones that are going out of support or end-of-life.

QA/UAT Plan

• Describe the QA infrastructure you will provide to ensure formal acceptance of each component of the IT solution.

Transition and Decommissioning

• Describe the high-level process you would follow at the end of the contract if a new contract were awarded to another service provider. How would you prevent a user outage during this time?
• Considering all data stored in the IT system, all software packages involved, any custom code, custom scripting and configuration, training material and other assets, what intellectual property would you consider to be the Proponent's, the client's, or a 3rd party's?
• What intellectual property is not the client's but could be transferred or licensed if the contractual relationship between the client and the Proponent ends?
• What parts of the ongoing engagement could be transferred to the client or another service provider? For example, once implementation is complete, could hosting and support be transferred to another Service Provider?
• Describe the high-level process you would implement to decommission the IT system if the client were to request it.

Multi-Language

• Users should be able to switch between different supported languages at any time and all relevant UX elements (e.g. menus, buttons, labels, breadcrumbs) will change to the selected language. Explain how the user's language choice will apply to all integrated components of the system and will not have to be specified by the user per component.

User Experience and Usability

• Does the IT system allow for a custom-designed user interface? In general, please describe the degree of flexibility for the UI elements to meet complex data interactions.
• Describe any user experience design patterns, principles, or standards you plan to adhere to while designing the IT system.
• Describe any accessibility design patterns, principles or standards (eg. WCAG 2.0) that you plan to adhere to while designing the IT system.
• Usability is key to adoption of the IT system by internal and external users. What capabilities and processes would you introduce to the implementation project to ensure that usability is tested? (interaction design, visual design, paper prototyping, formative or summative usability testing, etc.). Describe.
• The client will make available real-world users and SMEs over the course of the project to provide input . Describe how the project could incorporate these stakeholders in the usability planning and testing process.
• The IT solution should adopt the client’s branding standards. Please describe how you can implement these standards in your solution?
• All web UI presented to the user should use a responsive design that can adapt to smaller form factors like tablets without requirements for software changes. Please confirm.

Application Auditing and Logging

• The IT system must be capable of logging both business and technical events, and error conditions, for audit and operational purposes. (Note that simple page reads, viewing a document or other entity in the system is not considered a business event and doesn't need to be logged.)
  
• Describe how the IT system provides logging of technical error information to support troubleshooting.
  
• Business events must be logged in a way that can be searched and reported on. For example, if a legal dispute arises over communication sent to a USER, it must be necessary to show that a certain message was sent in the system, the user who sent it, and the date and time it was sent. All other business-meaningful events must be auditable in the same way. Describe how the IT system maintains a full audit trail of all business events that can be queried later if necessary.

System Scalability

• Consistent performance of the IT system is essential during normal business conditions. (An example of a relevant performance metric would be "all page loads from the IT solution for Client staff should be completed within 2 seconds 80% of the time"). Please describe how you monitor and ensure consistent performance of the various elements of the IT system. How will you be able to report against mutually agreed-upon performance metrics?

Data

• The data in the IT system should be consistent and accurate as viewed by any user at any time.  Please describe how your solution achieves this. For example, does your solution require data to be copied (in the background) from one component to another to ensure seamless integration? If so, does this copying introduce any risk of a user seeing outdated information?
• Describe how the client would implement data extracts of operational data for downstream systems such as a data warehouse. Can these be partial or incremental or must they be complete each time?
• Describe the separation of transaction, configuration, and master data in your solution.
• The data in the IT system should be of high integrity. Please describe how the solution ensures data integrity? Are there measures in place to prevent against SQL injections, cross-site scripting, etc?
• Please describe how you will structure the data model to ensure AI/ML algorithms are able to make effective use of the data?

Solution Architecture

• Please describe the overall system Architecture (layers, components, network, client), including a visual diagram of the application architecture and the supporting platforms.
  
• The IT system must be Web-based (for both internal and external users) and work on modern Web browsers running on a desktop or laptop. There is no requirement for a native mobile interface, although all interfaces must have a responsive design that can be viewed on a tablet. Please specify which browsers and versions are supported by your solution? Do you plan to eliminate support for any of the major browsers in the near future? The client will require the latest versions of Chrome, Firefox and Microsoft Edge (chromium version) to be supported.
  
• The IT system should not require any supporting software on the end user’s workstation. If this is not the case, please specify any components that require deployment on the client's network, or on the end user's computer?
  
• Client users should be signed in to the IT system automatically using their corporate login, without having to explicitly enter login credentials. Even if the IT system consists of multiple integrated technology platforms, users should not have to re-authenticate if moving from one component of the IT system to another. Describe how this would be achieved (AD federation, etc.) and what would be required of the client, noting any risks and limitations.

Customization and Configuration

• Explain how each software package proposed as part of your solution can be configured for the purposes of the client. What coding would be required as part of the implementation? Would any of the modifications require customization of a COTS product that would prevent future upgrades?

Deployment Environments

• Traditional deployment environments - development, testing, staging, production - are assumed to be managed by the service provider behind the scenes as part of release management. Please describe any additional environments the client may require. For example, would the client need its own development, testing and staging environments to configure workflows, worksheets, and UI web forms on its own? Would the client need its own testing environment to perform acceptance testing of Service Provider deliverables?

Availability, Resilience and Disaster Recovery
The client expects the IT system to adhere to the following availability and resilience metrics:
Recovery Time Objective (RTO): 24 hours
Recovery Point Objective (RPO): When data loss has occurred, the recovery point will be from the last available backup: 24 hours
Retention of Backup: 30 days

• Can the above SLAs be achieved? Please describe your proposed annual target availability metric. A suggested formula for calculating this metric is:
  
• How would you approach backup and recovery?
  
• How will you ensure resilience of your solution in the face of a disastrous event?
  
• What is your plan for each of the following scenarios:
  
  • Loss of the primary service instance
    
  • Loss of partner and/or public network connectivity/internet
    
  • If there were some form of Data corruption, how can we recover to a last known good copy of the data.

Hosting Model

• The IT system should be hosted either by the service provider or in a public cloud procured by the service provider. Please describe where you are proposing to host the IT system. Will your production environment be hosted in the same place as your pre-production environments?
• How will you ensure that software is patched, maintained and upgraded?
• How will you monitor for outages? How will you notify the client of these disruptions? What is your process for restoring the IT system to users when an outage occurs?

Log Management
The client requires evidence that you are recording security events occurring on your technology components (wired or wireless) including:
* Servers (Database, Web and Midtier (‘web/app’), DNS, Proxy, File, etc…)
* Network Infrastructure (Storage, Switch (physical and virtual), Router, Software Defined Network Devices (‘SDN’), network appliances, etc…)
* Security Infrastructure (Firewall, Central Security Log Management solution, etc…)

• What type of information will be in the log messages sent to your Security Operations Team / Security Incident and Event Management platform?
  
• Please describe the logs and/or reports that would be available to the client within the IT system and how the client can retrieve them.

Security Monitoring

• Technology based solutions process, transport, and store data that is important to the client’s business operations. The solutions must be continuously monitored to: detect and mitigate cyber security threats and vulnerabilities; to ensure the IT security controls are effective; and to prevent the loss of information assets.
  
• The client is expecting that with the IT system being proposed, the Proponent will be providing continuous security monitoring and incident response of all the underlying infrastructure (servers, applications, DNS, network and security infrastructure, etc.) including the IT system and third party suppliers/providers.
  
• Please list the use cases that are covered by your recommended security monitoring.
  
• What is the Service Level Agreement the client can expect from the managed service (e.g., monitored hourly, daily, weekly, and/or monitored by human vs AI)

Operational Security
Operational Security requires oversight of functional mechanisms related to security within the IT system and the related technical infrastructure supporting it. The client is expecting a plan for responding to a cybersecurity incident methodically. 

• Please attach the cyber incident response plan used when an incident is detected as part of the security monitoring service of the underlying infrastructure (servers, application, DNS, network and security infrastructure, etc.) that supports the IT system. 
  
• Please attach/include any relevant certifications for the IT system you are proposing (ie SOC 2 type 1 and 2).  Please also indicate the scope of the certification.  If you do not have any certifications yet, please share your road map/plan for the certifications you intend to qualify for.

Vulnerability Management
A technical flaw or weakness (‘vulnerability’) in the hardware, software, or firmware required by technology-based solutions can compromise the IT security measures if accidentally triggered or intentionally exploited. Technical vulnerabilities must be continuously mitigated to ensure the IT security controls are effective, and to prevent the loss of the client’s digital information assets.

• When new vulnerabilities are detected, what are the SLAs to mitigating the vulnerabilities?
  
• What is your plan to address zero-day vulnerabilities? 
  
• How frequently are vulnerability scanning, penetration testing and code scanning performed?

Encryption and Integrity Protection

• Encryption and integrity protection mechanisms are used to prevent disclosure and modification, protecting confidentiality and integrity of the client’s data assets. Please describe how you will protect the client's data in transit and at rest.

Secure Component Configuration
The Client’s expectation is that this solution will use fully patched, up-to-date operating systems, BIOS, and firmware; disable unnecessary services; use Endpoint protection (ie anti-virus / anti-malware); change default passwords; and use Network Time Protocol.

• Please describe your approach to reducing the ‘attack surface’ based on the Client's security component configuration expectations.

Secure Code Development

• Please describe the processes and technology used to ensure custom source code integrity and confidentiality.
• Please describe the input validation and output validation controls.