The advent of cloud computing has revolutionised how businesses operate, promising unparalleled scalability, flexibility, and efficiency. However, with these advancements come significant data management challenges that can undermine the very benefits the cloud aims to deliver. This article explores the complexities that arise in data management due to the promises of cloud technology, focusing on SaaS applications, data access, retention, backup, business continuity, security, data privacy, and shared responsibility models.
1. The Allure of Scalability
Cloud computing’s promise of scalability allows businesses to expand their operations without the need for substantial physical infrastructure investments. While this elasticity is a boon, it introduces complexities in data management. As organisations scale their operations, the volume of data grows exponentially, which can lead to challenges in maintaining data integrity, performance, and security. For instance, a retail company experiencing rapid growth may face difficulties managing vast amounts of customer data across various cloud services, potentially leading to inconsistencies and increased vulnerability to data breaches.
Many organisations have data centre exit strategies, largely and in some cases completely closing down existing owned and operated compute facilities. As part of such plans, it’s worthwhile assessing the current posture of data contained on storage arrays, file-shares and object stores, however, this is often not undertaken. What was on-premise often ends up shifted to the likes of S3 and Blob or SharePoint Online and OneDrive, compounding data management challenges.
Genesys Data recommendation: Consider moving to Cloud and SaaS providers – only the minimum level of data, containing the minimum level of detail that your organisation needs to access and process. Analyse and rationalise your data footprint and employ tools to classify and label your data as you go forward in the cloud.
2. Flexibility and SaaS and PaaS
Software as a Service (SaaS) and Platform as a Service (PaaS) offer incredible flexibility by allowing users to access software over the internet without managing the underlying infrastructure. However, this flexibility can complicate data management. SaaS platforms often come with their own data storage and management practices, which may not align with an organisations internal policies or regulations. For example, a business using multiple SaaS tools for CRM, HR, and project management may struggle with integrating data from these disparate sources, leading to fragmented information and increased difficulty in achieving a unified view of organisational information.
Businesses must clearly understand the terms of their SaaS agreements, including how they can access, retrieve, and delete their data if needed, and the implications of data ownership during vendor transitions. In some cases our clients find that they can obtain an extract or a copy of their data extracted from SaaS platforms, however it’s only to be used for recovery into the same SaaS provider or it needs significant manipulation and transformation work to be made usable elsewhere.
Clearly defining who has access to what data within a SaaS application can be challenging, especially as organisations grow and use more tools. Role-based access control (RBAC) must be carefully designed to ensure that sensitive data is only accessible to authorised users. Additionally, monitoring data access and maintaining audit trails are critical for both security and compliance purposes.
Genesys Data recommendation: Ensure your SaaS provider can provide formal and annual attestation against security controls, such ISO27001, IRAP, SOC2 Type II. Ensure there is clarity around scope of delivery along with where key obligations start and stop. Have a clear plan around data governance and vendor management for critical data assets.
3. Security Posture
The cloud’s promise of remote access and collaboration introduces new challenges in managing data access and security. Ensuring that sensitive information is accessible only to authorised personnel while being protected from unauthorised access requires robust access control mechanisms. For instance, a private school using cloud-based collaboration tools must implement stringent access controls to prevent data leaks or breaches, which can be challenging given the dynamic nature of cloud environments and the complexity of managing permissions across multiple platforms. File transfers, meeting recordings, transcripts from the likes of Microsoft Teams chats end up spread across OneDrive, SharePoint Online and Exchange Online. These relatively new technologies are creating new forms of data sprawl.
Configuration drift, coding errors and bad management practices all occur in the cloud too. It wasn’t too long ago that the data of more than nine million customers was leaked due to a vulnerability in an API (Application Programming Interface) that was left exposed to the internet. It appears that this API did not require authentication, allowing unauthorised access to personal information such as customer data, including names, addresses, email addresses, phone numbers, passport numbers, and driver’s license details.
Some might argue that traditional data centres, are more reliable as they are not as susceptible to the scale of external threats being tested against Cloud Service Providers. The counter argument here is that the cloud offers a level of security that far surpasses traditional data centres and comes with enterprise grade expertise, resources and advanced technologies. Whatever the case, fundamentals about data handling, access and governance need focus and improvement.
Genesys Data recommendation: Understand and track data flows. Give consideration to Data Los Prevention technologies, Muti-Factor Authentication must be mandated, regularly review how files and folders are shared, mask or obfuscate data for testing purposes as test environments often have less stringent controls around them.
4. Data Retention
Organisations must ensure that they have clear policies in place for data retention and deletion. Over-retaining data can lead to excessive storage costs, increased security risks, and potential non-compliance. On the other hand, under-retaining data may result in the loss of important records needed for audits, legal defence, or business continuity. For example, a healthcare provider must adhere to strict regulations regarding patient data retention.
Cloud providers may offer different retention options, making it essential to align data management practices with company standards. Many companies leave retention decisions to the likes of IT administrators, meaning that there’s often no consultation with business owners and executive decision makers.
Effective data lifecycle management involves defining how data is stored, accessed, archived, and eventually deleted. In cloud environments, managing the data lifecycle across multiple services (e.g., databases, object storage, file storage) can be difficult, especially when each service has its own retention settings and controls.
Genesys Data recommendation: Policies should define how long different types of data should be retained, based on classification such as sensitivity, recovery point and recovery time requirements, purpose and applicable regulations. Multiple copies of data in cloud, in separate clouds, copies that are immutable and offline should also be factored into data retention design.
5. Backup and Recovery
To varying degrees, the cloud offers the convenience of automated backups and disaster recovery solutions, but managing these boring and uninteresting practices remain a challenge to most organisations and are usually poorly managed.
Organisations must ensure that their backup strategies align with their data protection needs and recovery objectives. A media company that relies on cloud storage for its digital assets must have a robust backup plan to prevent data loss in case of accidental deletion or corruption. Failure to do so can result in significant operational disruptions and financial losses.
You can architect for redundant storage with multiple data copies within a cloud region, multiple geographic redundant storage with data replication across cloud-regions such as between Sydney and Melbourne. You can have storage tiers with different costs and performance characteristics and add immutability. Cloud servers can write snapshots to redundant storage supporting varying degrees of full and granular recovery. SaaS and PaaS providers may undertake some form of backup, or may not, citing that data within their services are your responsibility!
If you have multiple cloud providers and multiple critical SaaS services and IT delivery on-premises, without a backup and recovery plan, you’re in a pickle and the costs of managing data-at-rest can be eye watering!
Genesys Data recommendation: Snapshots are not backups, yet you need both for speed of recoverability, granularity and cost management via the likes of deduplication. The 3+2+1 principle is still a valid data backup strategy in the cloud, however, it’s not being well followed – Keep at least three copies of your data, store these copies on at least two different types of media – Ensure that at least one copy is stored offsite, off-tenancy or off-cloud, in a air-gapped tamper-proof environment. Don’t rely on replication in lieu of backups as data can become corrupt in transit. Have an organisation wide backup strategy with monitoring, reporting and data-security inbuilt. Also, understand your management costs.
6. Business Continuity and Disaster Recovery
Maintaining business continuity in a cloud environment requires careful planning and management. Cloud flexibility means that data and applications can be easily moved or replicated, but ensuring that these processes are well defined and do not disrupt business operations is crucial. For instance, a global e-commerce platform must have strategies in place to ensure that data replication and failover processes do not impact customer experience or result in downtime, which could lead to lost sales and damaged reputation.
Business Continuity is sometimes overlooked in cloud environments due to an overreliance on cloud providers’ inherent redundancy, a misunderstanding of where the likes of SaaS and PaaS services start and stop, and the complexity and costs associated with implementing disaster recovery plans. In some organisations data sovereignty and compliance issues can complicate continuity planning.
The division of responsibilities between cloud service providers and customers can lead to ambiguity about who handles disaster recovery, resulting in gaps in DR planning. Additionally, the rapid pace of technological advancements and frequent updates in cloud services can make it challenging for organisations to keep their disaster recovery strategies aligned with the current cloud environment, potentially leading to outdated or ineffective plans.
Genesys Data recommendation: To ensure cloud business continuity and data availability, develop a comprehensive continuity plan that includes disaster recovery and regular backups. Understand the shared responsibility model with your cloud provider, utilise multi-region and multi-zone deployments, and regularly test your DR plan. Also, stay informed about cloud provider updates and request information about the cloud providers own security posture though evidence of vulnerability, penetration and security assessments.
7. Data Privacy Challenges
Ensuring data privacy in the cloud can be particularly challenging due to the global nature of cloud services and the varying privacy laws across different jurisdictions. Multinational companies must navigate complex regulatory landscapes, such as the General Data Protection Regulation (GDPR) in Europe along with the Privacy Act in Australia. For instance, an international company storing personal data of EU citizens must ensure compliance with GDPR requirements, including data subject rights and cross-border data transfer regulations.
Contracts with SaaS providers should include clauses detailing the provider’s obligations for data protection and privacy, specifying security measures, access controls, and breach notification procedures. They should clarify data ownership, usage limitations, and terms for data retention, deletion, and transfers, especially across borders. The contract should address the use of sub-processors, audit rights, and support for fulfilling data subject rights. Additionally, it should outline procedures for data return or destruction upon contract termination. These provisions ensure that data privacy and compliance are effectively managed.
The Australian Privacy Act requires that personal data be collected only for specific purposes and is retained only as long as necessary. Ensuring that cloud providers comply with these principles and do not retain data longer than required can be challenging. This is a specific area to watch given proposed changes to the Privacy Act.
Genesys Data recommendation: Organisations should conduct thorough due diligence when selecting cloud providers, implement strong contractual agreements, maintain clear data management practices, and regularly review and monitor compliance with Australian privacy requirements.
8. Shared Responsibility Models
One of the fundamental aspects of cloud security is understanding shared responsibility models. In this model, the cloud provider is responsible for securing the underlying infrastructure, while cloud consumers or customers are responsible for managing their own data and where in scope, applications. This division of responsibility can sometimes lead to confusion. For example, while a cloud provider might ensure the physical and network security of their data centers, the Customer must manage their own data encryption, access controls, and compliance with relevant regulations along with data backup and disaster recovery. Misunderstanding these responsibilities can lead to gaps in security and data protection, leaving organisations significantly exposed.
It is unfortunately common for SaaS providers to enforce that they bear no responsibility for tenant or company data placed into their platforms. Here’s an excerpt from a SaaS contract for a SaaS platform used to in the manufacturing sector :
“You, and not <CLOUD SERVICE PROVIDER>, are responsible for maintaining and protecting all of your information. <CLOUD SERVICE PROVIDER> will not be liable for any loss or corruption of your information, or for any costs or expenses associated with backing up or restoring any of your information.”
It’s also important to note that SaaS providers often change their standard terms and agreements, adding new features, sometimes removing others, varying terms of use and changing obligations to their customers.
Genesys Data recommendation: Ensure you obtain expert advice relating to cloud responsibility models and contracts. Present the risks of SaaS and ensure these risks are either mitigated or are understood. Have clarity about how you will manage and maintain your data in a SaaS platform and define an appropriate exit strategy that ensures data access, migration or destruction. Review agreements and contracts regularly and understand the broader business implications such as availability and continuity expectations.
The cloud’s promises of scalability, flexibility, and efficiency come with inherent complexities in data management. Organisations must navigate these challenges by implementing robust data management strategies, including integrating SaaS applications, managing iterative development processes, enforcing strict access controls, and ensuring compliance with data retention and backup policies. By addressing these challenges proactively, businesses can fully leverage the cloud’s benefits while mitigating the risks associated with its dynamic and evolving nature.
While cloud computing offers remarkable advantages, it also presents significant data management challenges that organisations must address to ensure they fully realise the benefits of this technology. By understanding and managing these complexities, businesses can better navigate the cloud landscape and maintain effective data governance, security, and privacy.