Architecture Introduction

架构总览

Apache drill是在大规模数据集场景下,可以低延迟地进行结构和半结构化/嵌套数据结构查询的一个分布式查询引擎。受到谷歌公司的Dremel的启发,Drill被设计出来以支持几千个节点和PB级别的数据规模下,支持交互响应级别的商务智能分析和查询。 Drill也适用到在大规模数据集场景下进行简单而迅速的查询.Drill能够查询像是JSON或是Parquet这种嵌套的数据,也能动态地发现schema.Drill并不需要一个中央的元数据库.

Apache Drill is a low latency distributed query engine for large-scale datasets, including structured and semi-structured/nested data. Inspired by Google’s Dremel, Drill is designed to scale to several thousands of nodes and query petabytes of data at interactive speeds that BI/Analytics environments require.

Drill is also useful for short, interactive ad-hoc queries on large-scale data sets. Drill is capable of querying nested data in formats like JSON and Parquet and performing dynamic schema discovery. Drill does not require a centralized metadata repository.

顶层架构(High-Level Architecture)

Drill包含一个专门为了处理大规模数据的分布式执行环境。Apache Drill的核心是一个叫做“钻头”(drillbit)的服务,它负责从客户端接受请求,处理该查询,并将结果返回给客户端。一个drillbit服务可以在Hadoop集群中所有有需要的节点上安装和运行,形成一个分布式的集群环境。当drillbit运行在集群中的数据节点上时,drillbit可以查询执行过程中最大限度地使数据本地调用,而无需在网络上或是节点之间移动数据。Drill使用ZooKeeper来记录集群成员和健康检查信息。虽然钻工作在Hadoop集群环境中,Drill并不紧紧地与hadoop绑死,而是可以运行于任何分布式集群。Drill唯一的依赖是zookeeper. 请查阅Drill Query Execution

Drill 客户端

你可以通过下面的客户端来访问drill:

  1. Drill shell

  2. Drill Web Console

  3. ODBC/JDBC

  4. C++ API

    Drill includes a distributed execution environment, purpose built for large- scale data processing. At the core of Apache Drill is the “Drillbit” service, which is responsible for accepting requests from the client, processing the queries, and returning results to the client.

    A Drillbit service can be installed and run on all of the required nodes in a Hadoop cluster to form a distributed cluster environment. When a Drillbit runs on each data node in the cluster, Drill can maximize data locality during query execution without moving data over the network or between nodes. Drill uses ZooKeeper to maintain cluster membership and health-check information.

    Though Drill works in a Hadoop cluster environment, Drill is not tied to Hadoop and can run in any distributed cluster environment. The only pre-requisite for Drill is Zookeeper.

动态Schema发现

Drill并不需要一份数据schema或是类型定义就可以开始执行查询。Drill是分批次地开妈数据处理的。自描述的数据格式,像Parquet,JSON,AVRO,还有一些Nosql 数据库,格式描述是数据的一部分,Drill在处理的过程中会根据需求加以利用。

Drill does not require schema or type specification for data in order to start the query execution process. Drill starts data processing in record-batches and discovers the schema during processing. Self-describing data formats such as Parquet, JSON, AVRO, and NoSQL databases have schema specified as part of the data itself, which Drill leverages dynamically at query time. Because the schema can change over the course of a Drill query, many Drill operators are designed to reconfigure themselves when schemas change.

灵活的数据模型

Drill允许访问嵌套的数据属性,就好像它们是SQL列一样,并提供了直观的扩展以轻松地操作它们。从架构的角度来看,Drill提供了一个复杂的级联式的列式数据模型,用来描述复杂的,高度动态且不断变化的数据模型。在Drill里,关系数据被视为复合/多结构数据的一个简化处理。

Drill allows access to nested data attributes, as if they were SQL columns, and provides intuitive extensions to easily operate on them. From an architectural point of view, Drill provides a flexible hierarchical columnar data model that can represent complex, highly dynamic and evolving data models. Relational data in Drill is treated as a special or simplified case of complex/multi-structured data.

去中央元数据设计

Drill不要求一个集中的元数据。你并不需要创建一个元数据库来存储表和视图,或依赖于一个有这种功能的元数据管理组件。Drill的元数据来源于那些跟源数据打交道的存储插件。存储插件能提供全部元数据中的一系列子区间(例如Hive),或是元数据的一部分(如HBase),或者就没有元数据(针对文件类)。去中央元数据意味着Drill不依赖于一个单一的Hive库,您可以一次查询多个Hive库,然后把结果与HBase的表或分布式文件系统中的文件信息组装起来。您也可以在Drill中使用SQL DDL语句来创建元数据,这些元数据就像传统的关系数据库中管理的一样。Drill的元数据也可以通过ANSI标准的INFORMATION_SCHEMA数据库来访问。

Drill does not have a centralized metadata requirement. You do not need to create and manage tables and views in a metadata repository, or rely on a database administrator group for such a function. Drill metadata is derived through the storage plugins that correspond to data sources. Storage plugins provide a spectrum of metadata ranging from full metadata (Hive), partial metadata (HBase), or no central metadata (files). De-centralized metadata means that Drill is NOT tied to a single Hive repository. You can query multiple Hive repositories at once and then combine the data with information from HBase tables or with a file in a distributed file system. You can also use SQL DDL statements to create metadata within Drill, which gets organized just like a traditional database. Drill metadata is accessible through the ANSI standard INFORMATION_SCHEMA database.

可扩展的设计

Drill在所有层都提供了一个可扩展的架构,包括存储插件,查询,查询优化/执行器以及客户端API层。您可以定制任意层来满足您的机构的特定需求,也可以把这一层延伸到更广泛的用途。Drill使用类路径扫描来查找和加载插件,并用最少的配置来添加额外的存储插件,功能和操作支持。

Drill provides an extensible architecture at all layers, including the storage plugin, query, query optimization/execution, and client API layers. You can customize any layer for the specific needs of an organization or you can extend the layer to a broader array of use cases. Drill uses classpath scanning to find and load plugins, and to add additional storage plugins, functions, and operators with minimal configuration.