Architecture of Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 (often referred to as ADL Gen2) is a comprehensive and intricate platform that has been intricately designed to handle vast datasets, especially when big data analytics is in play. To truly harness its capabilities, understanding its architecture is fundamen-tal. So, let’s guide you through the architectural framework of ADLS Gen2:

• Foundational principles: Azure Data Lake Storage Gen2 extends the capabilities of

Azure Blob storage. Essentially, it combines the scalable and cost-efficient nature of

Blob storage with big data analytics capabilities, resulting in a powerful hybrid.

• Hierarchical namespace: At the heart of ADLS Gen2’s architecture is the hierarchi-cal namespace. You can organize objects (such as files and directories) into a hierarchy, similar to a file system. Operations such as renaming or deleting directories become atomic, which optimizes big data workloads.

• Storage account: This is the apex of the organizational structure in ADLS Gen2. Every storage account possesses a unique namespace that is used to address both Blob and data lake storage data in that account.

• Containers: Within storage accounts are containers, functioning almost like directo-ries in traditional file systems. They group sets of blobs and can be used to organize data based on projects, departments, or data types.

• Blob: Every data piece in Azure Data Lake Storage Gen2 is essentially a blob. You’ll deal with large files broken into chunks, with each chunk being a blob. These blobs can be further categorized into block blobs and append blobs.

• Azure Active Directory (Azure AD): With Azure AD, you get seamless integration, allowing you to set up authentication and authorization mechanisms for your data.

82 CHAPTER 3     Describe considerations for working with non-relational data on Azure

• Role-based access control (RBAC): levels, ensuring that the right people

Assign roles and manage access at granular have the right permissions.

• POSIX-compliant ACLs: Further refine data access with POSIX-compliant ACLs. This is a game-changer for big data processing needs, offering precision access control.
• Data integration: Azure Data Lake Storage Gen2 smoothly integrates with various Azure processing solutions, such as Azure Databricks, Azure HDInsight, and Azure Data Factory. This integration means that once your data is stored, processing and analyzing it become more streamlined.

• Performance: Being built upon the foundational principles of Azure Blob storage, ADLS Gen2 offers massive scalability, high availability, and robust data analytics perfor-mance. Features like multiprotocol access on the same account ensure that you can use the Blob object store and the data lake store in a congruent manner, optimizing your data analytics tasks.

Figure 3-8 shows the architecture of Azure Data Lake Storage Gen2.

FIGURE 3-8 Azure Data Lake Storage Gen2 architecture

The architecture of Azure Data Lake Storage Gen2 is a testament to Azure’s commitment to offering a structured yet flexible environment for big data analytics. As you immerse yourself deeper into the intricacies of ADLS Gen2, you’ll discover that its architectural choices serve both your present and future data needs.