This is documented in Omegabit Internal Operations Wiki.

This is documented in Omegabit Disaster Recovery Handbook, Section 1.1 to 1.4 and Section 2.3

ref: Omegabit Disaster Recovery Plan TOC

Yes. Per agreed upon SLA. 

Yes. ref: Omegabit Disaster Recovery Plan TOC

Yes. ref: Omegabit Disaster Recovery Plan TOC

Yes. ref: Omegabit Disaster Recovery Plan TOC, Omegabit Operations Portal, and Training curriculums

Yes. The DR plan was recently exercised and updated in Q2 of 2017. A certified statement can be provided by executive management certifying this, provided the vetting proceeds to the next round.

ref: Omegabit Disaster Recovery Plan TOC

● Logical and physical redundancy at the VMWare, JVM, repository and other critical layers of the runtime environment stack

● Warm-spare redundant Liferay architecture (proposed)

● Server failover capability

● Rapid nearline backup recovery

● Comprehensive off site DR for catastrophic failure

In the event that a high-availability portal configuration is required, redundant nodes of the HA configuration will be purposefully isolated to discrete server and backend infrastructure as a complement to that logical HA configuration, to the benefit of higher reliability and faster recovery under various logical/physical architecture failure scenarios.

Omegabit operates comprehensive SNMP and service level monitoring of all configured hosts and services.  Triggers are adjustable and set by default to detect failures as well as symptoms of imminent failure.  Monitor alerts are responded to by live personnel, 24x7x365, and acted upon according to severity, per the terms of our SLA.

The core physical host infrastructure is inherently HA in terms of disk arrays, storage and network paths, physical servers, switching, etc.  Omegabit operates a modern VMWare based infrastructure.  In the case of most physical failures services are designed to continue transparently with no observable interruption to operations.  In the case of logical failures, the VM, JVM, and Liferay backend service configuration is proposed as an HA setup, to practical limits.  If a higher level of resilience is required than is proposed, we are able to accommodate that as additional scope.  Disaster Recovery (DR) is an inherent component of the regular day-to-day operations performed by Omegabit, as a core function of the hosting operations is supplied for all tenants.

Omegabit offers multiple redundant layers of protection including but not limited to:

● Logical and physical redundancy at the VMWare, JVM, repository and other critical layers of the runtime environment stack

● Warm-spare redundant Liferay architecture (proposed)

● Server failover capability

● Rapid nearline backup recovery

● Comprehensive off-site DR for catastrophic failure

Backups snapshots of the entire VM stack are performed every 2hrs, and the offsite archives of those backups are continuous to a second physical location.  Retention for 2hr snaps for 48hrs, dailys for 30 days, and weeklys for 16 weeks.  We can accommodate longer retention if necessary.  Some of these retention policies impact RPO.  

For PCI, you may want logs to last up to 1yr but, that can be accomplished through application design or by depending on our backups.  We recommend using both strategies depending on your reporting needs.

Backups should be considered for disaster recovery purposes only.  Our retention policy is variable and based upon data volume.  Depending upon the environment, rollbacks to the previous day, several days, weeks are available, but with sporadic snapshots between periods.  Therefore, a specific point-in-time recovery may not be possible.  We are typically able to restore backward up to several weeks depending upon the total size of your store.

Omegabit can provide additional backup and archival services to meet specific requirements on a needs basis.  Please contact your sales representative for more information.

Omegabit features a comprehensive alternate-site DR recovery plan that includes regular off-site archives using Omegabit owned and managed equipment.  Backup to the public cloud (e.g. Amazon), is optional but requires special arrangement and may not be compatible with some PII/HIPAA requirements.  Specific features for disaster recovery vary by tier of service; please see the SOW for complete details on RTO/RPO times and obligations.

Omegabit offers a completely HA infrastructure with no single-point-of-failure in the chain systems.  And utilizes the latest VMWare, Nimble Storage, and high-speed network architecture available to ensure no bottlenecks to systems operations.  

The proposed solution also includes a fully clustered Liferay application, which provides additional redundancy at the critical logical layers of the software.  As with most clouds, all servers are redundant and software will automatically fail to another node in the event of a physical system failure. Moreover, Omegabit is able to provide additional "hot" fault-tolerance as an optional feature.

Omegabit also operates extensive backups including on-site runtime snapshots every 2hrs and continuous "hot" offsite archives to a secondary DR location.

Liferay can be clustered for HA at the application (JVM) level.  In that case we also ensure that your nodes are distributed across separate physical hardware.  This does provide the convenience of hot-redundancy at the application server layer, and the ability to support things like rotating outages for Liferay.  It does increase the cost both in terms of infrastructure and DXP licensing (2x, typically).  

For true HA, it would also be appropriate to cluster the search services (Elastic), although, they are less prone to breakage and change, relative to the app servers. And, in a perfect world, the DB as well.  However, that is more complicated:  MySQL does not cluster well; there are alternate strategies avaialbe (including hot-HA at the VMWare/cloud layer).  That said, the DB and search are typically the most stable layers and are least likely to benefit from HA in terms of practical ROI and actual risk vs. cost.  The alternative option is to employ a commercial RDBMS like Oracle in a cluster, which has licensing implications, but we are happy to host/accommodate.  If this is a near term concern, let's have a call to discuss options.  If this is a hypothetical, the answer is, yes, we can absolutely support it with enough resources in play and assuming it is cost-justified to the business. 

Core cloud and network infrastructure:  99.999%

Liferay application infrastructure:  greater than 99.98%

Omegabit frames uptime in terms of application vs. core infrastructure stability as a more practical measure of efficacy.  Beware of providers that only indicate core infrastructure uptime vs. Liferay application uptime, which is typically higher, and can be misleading.

Omegabit commits that Client's hosted Web applications will have a monthly availability of 99.95% or greater. Monthly application availability is calculated using the following equation:
     
Availability Percentage = 100-((Md/43800)*100)
Where Md = The total minutes of unplanned downtime for the month
And 43800 is the average number of minutes in a calendar month
(22 minutes of downtime in a month would result in a total Network Availability for the month of 99.94%)
     

Planned Downtime     

Omegabit makes every effort to perform maintenance to its private cloud services transparently in the background without interrupting Client services in any capacity.  On rare occasion, a brief outage is required to complete maintenance or repair/replace piece of equipment, or in case of emergency security remediation. Omegabit makes every effort to provide at least 5 business days advanced notice for scheduled maintenance, and will provide Client with no less than 48 hours advanced notification concerning planned outages, which will include the following information:
● Affected systems and services
● Reason for outage
● Expected downtime
● Contact information
       
Downtime due to planned outages where the outage is within the expectation set in the notification and where Omegabit has provided notification of the event to the Client at least 48 hours in advance is not used in calculating monthly application availability.
       
In the event that Web application availability falls below 99.95% in any calendar month, Client will be credited on the next billing cycle with an amount equal to the percentage of the affected month's total hosting charges as indicated per the SLA terms.
       

Schedules and approvals are managed in direct coordination with Client teams to ensure changes are controlled and do not cause breakage. Actions are scheduled according to policies defined in the Omegabit SOW/SLA, except where explicitly overridden by special policy or Client requirement.