Big data migration plan

Paper details:

Assignment: Big data systems enterprise deployment, integration, scalability and security issues In view of the need to handle increasingly larger amounts of data and to better cope with the ensuing big data V-characteristics (Volume, Velocity, Variety, Veracity etc.), several large organisations have begun migrating their standard legacy Enterprise Data Warehouse (EDW) systems to Big-Data-Driven-Enterprise (BDDE) schemes, which subsume the EDW. Many of these schemes utilise the Hadoop ecosystem supported by a variety of additional data mining and business analytics sub-systems. In this Assignment, you will look at how big organisations migrate from standard EDW to BDDE schemes (that incorporate the EDW) and will investigate the key issues that often have to be dealt with in such migrations. To complete this Assignment: Complete out the following tasks: (300 words)Describe a real company use case that want to transform to the BDDE type. Select the industry sector and your particular business activity. Your company’s operations must include a Web presence—you must define what service this Web presence provides to its customers and additionally explain how data will be used and collected. (200 words)(include a Data flows map diagram)Describe data flows in your company. Identify where you will use data collected from the Web and other sources of information such as sell statistics, user activity or social media data. Describe how you plan to integrate new types of big data with current EDW data workflows that primarily use relational data. Do not forget about operational aspects such as data backups and, if applicable, long-term data storage. Data provenance may be needed in addition to regular activity and communication logs. (200 words)Select a suitable platform for general big data management (e.g. in-house or cloud based infrastructure) and Big Data Management System (BDMS) platform with vendor (e.g. Hadoop supplied by Cloudera CDH, AWS EMR or HortonWorks). You should provide sufficient detail about the BDMS components and tools that you intend to use within the BDDE scheme. (200 words)Provide suggestions for how you will address security and privacy issues when managing your customer data, and your company’s data. Again, do not forget about regular backups and secure backup storage for the BDDE. (200 words)Define a data management policy, including data protection and access control. Briefly address a majority of the CSA top ten security and privacy challenges. (200 words)Suggest what big data analytics and visualization methods you will use, including specific commercial tools and platforms. (75 words)Conclusion By Day 7 (Wednesday) Submit a report containing your responses for tasks 1 – 6 written up as an “EDW-to-BDDE Migration Plan” for the specified company. Your report should address the issues raised in a reasonably concise and practical manner.