This was extracted (@ 2024-04-17 22:10) from a list of minutes
which have been approved by the Board.
Please Note
The Board typically approves the minutes of the previous meeting at the
beginning of every Board meeting; therefore, the list below does not
normally contain details from the minutes of the most recent Board meeting.
WARNING: these pages may omit some original contents of the minutes.
Meeting times vary, the exact schedule is available to ASF Members and Officers, search for "calendar" in the Foundation's private index page (svn:foundation/private-index.html).
Report was filed, but display is awaiting the approval of the Board minutes.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Project Status: Current project status:Ongoing (high activity) Issues for the board:None ## Membership Data: Apache Arrow was founded 2016-01-19 (8 years ago) There are currently 103 committers and 52 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - Jonathan Keane was added to the PMC on 2023-10-13 - Raúl Cumplido was added to the PMC on 2023-11-12 - Curt Hagenlocher was added as committer on 2023-10-14 - Felipe Oliveira Carvalho was added as committer on 2023-12-06 - James Duong was added as committer on 2023-11-16 - Xuwei Fu was added as committer on 2023-10-22 ## Project Activity: ### Sub Project Updates Arrow has several subprojects, as listed on https://arrow.apache.org/ ### ADBC ADBC 0.8.0 was released on 9 November 2023. Highlights: C#/.NET has added a BigQuery driver. Also, it now can be used through ADO.NET. The R bindings now use the ADBC 1.1.0 specification, and more packages are now available on CRAN. The Snowflake driver now has an option to control whether to return decimal types or attempt to convert to integers/floats. The PostgreSQL driver now uses COPY for bulk ingestion, which Pandas has found is approximately 35x faster than their previous method. The SQLite driver can now load extensions and supports more data types, including Arrow binary (SQL BLOB), and supports binding some dictionary-encoded types (which will be unpacked). ### Arrow Flight No update ### Arrow Flight SQL There is community interest in implementing and extending an ODBC driver for Arrow Flight SQL, and we are in the process of accepting an ODBC driver[1] to support that interest. [1] https://lists.apache.org/thread/t109wsn86cnk5kbc390snco0s751qhpv ### Arrow Flight SQL adapter for PostgreSQL There were some feedback after the first release and we solved most of them. We’ll release the next version in a few months. ### DataFusion & Ballista DataFusion continues releasing regularly. We have submitted a proposed paper describing the system to the ACM SIGMOD conference and in general are trying to scale the project as it grows in popularity. There is a draft proposal for promoting DataFusion (including Ballista) to a top-level ASF project, and we hope to have a proposed board resolution for the April board meeting. Ballista is not very active but continues to receive occasional contributions. ### nanoarrow Arrow nanoarrow continues its ~quarterly release cadence with active improvement to the C, R, and Python implementations scheduled for release in mid-January as nanoarrow 0.4. ### Language Area Updates Arrow has at least 13 different language implementations, as explained in https://arrow.apache.org/overview/ Arrow 14.0.0 was released from the monorepo: https://arrow.apache.org/blog/2023/11/01/14.0.0-release/ ### C++ Fixed shape tensor extension type has been added as a new canonical extension type in release 12.0.0 and variable shape tensor extension is now being under review for the next release (16.0.0). #### Dataset & Parquet Added support for reading and writing the newly added Parquet float16 logical type. Added support for Parquet modular encryption. #### Acero & Compute Improvements were made to several compute functions. Added support for serializing and deserializing compute expressions using Substrait. #### Gandiva Migrated LLVM JIT engine from MCJIT to ORC v2/LLJIT. Added support for the latest LLVM (17). Added support for registering external function registries. Added support for registering external C functions. ### C# The C# implementation has been steadily improving its compatibility with the standard. Since the last report, it has gained support for duration and interval types, as well as the new types utf8 view, binary view and list view. Dictionaries now work correctly in file and memory implementations, and there are only four explicit exclusions for C# in the Archery integration tests. ### Go Integration testing for the C Data API has been added to the CI for Go and other implementations. The Parquet implementation continues to gain fixes for different encoding/decoding types and bug fixes. Also fixed Go release verification for arm64. ### Java We are working on adding nullability annotations and enabling module support for Java 9+. ### JavaScript No update ### Julia Version 2.7.0 was released on 10 December 2023 ### Rust The parquet implementation continues to mature, for example supporting new statistics metadata. The FFI bindings have been improved as well, and are now integration tested against arrow-cpp (among others) The object_store module, developed as part of this project, allows for generically interacting with object store systems such as AWS S3, Google Cloud Storage, and Azure Blob Storage. This crate has seen significant adoption outside of the arrow community, for example the crates.io service itself. ### C (GLib) No update ### MATLAB We are currently working on integrating with the project release tooling to make it possible to distribute pre-built MLTBX files for easy installation of the MATLAB interface. ### Python There has been ongoing work on improving interoperability with other Python projects for example adding C Data Interface PyCapsule protocol and implementing the usage of capsules in ADBC and nanoarrow-python. We have also implemented the DLPack protocol on Arrow Arrays that is used to move the data to ML libraries. A critical security vulnerability was discovered in PyArrow versions 0.14.0 to 14.0.0 that allowed arbitrary code execution when loading a malicious Arrow IPC, Feather, or Parquet data file (CVE-2023-47248). The vulnerability was patched in PyArrow version 14.0.1. A hotfix package was released to patch the vulnerability in all other versions of PyArrow for users unable to immediately upgrade. ### R Completed large parts of a major rework of the Arrow R package build system. These changes aim to reduce maintenance burden and streamline new-contributor experience e.g. by automating the use of nightly builds which enables contributions to the R package without having to setup a C++ development environment. ### Ruby Added some convenient APIs. ### Swift Improved Flight SQL implementation. ## Community Health: Community communication continues to be strong. There have been 5 blog posts published to https://arrow.apache.org/blog/ in the last 3 months. The mailing lists are active
WHEREAS, the Board of Directors heretofore appointed Andrew Lamb (alamb) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Andrew Lamb from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Andrew Grove (agrove) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Andrew Lamb is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Andrew Grove be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7E, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. More information can be found at https://arrow.apache.org/overview/ ## Project Status: Current project status: Ongoing (high activity) Issues for the board: None ## Membership Data: Apache Arrow was founded 2016-01-19 (8 years ago) There are currently 99 committers and 50 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Dewey Dunnington on 2023-06-22. - Oleks V. was added as committer on 2023-09-30 - Metehan Yildirim was added as committer on 2023-08-29 ## Project Activity: We added a new array layout, [Utf8View] to the Arrow spec, which allows more efficient variable length string handling. There has also been much [Utf8ViewDiscussion] about how to use this new layout to interoperate better with the broader ecosystem We have started adding cross-language integration tests for the [C Data Interface], ensuring that implementations comply with the spec and are concretely interoperable. This is in addition to the existing integration tests for the [IPC format] and for the [Flight RPC] protocol. We registered our DOAP file: https://projects.apache.org/project.html?arrow [Utf8View]: https://s.apache.org/ldd1b [Utf8ViewDiscussion]: https://s.apache.org/a1v8o [C Data Interface]: https://arrow.apache.org/docs/format/CDataInterface.html [IPC format]: https://s.apache.org/5uwhb [Flight RPC]: https://arrow.apache.org/docs/format/Flight.html [ListView and LargeListView]: We have also finalized adding alternative layouts to variable-length list arrays that require an extra buffer with sizes, a layout found in other systems such as DuckDB and Velox. This in turn allows offsets to be out of order. Elements of the child array do not have to be stored in the same order they logically appear in the list elements of the parent array. [C Data Interface]: ArrowDeviceArray and ArrowDeviceArrayStream structs were added to the spec to allow support for Arrow Arrays whose buffers are on a non-CPU device. ## Sub Project Updates Arrow has several subprojects, as listed on https://arrow.apache.org/ ### ADBC Version 0.7.0 was released, along with a 1.1.0 revision of the API specification it implements. ### Arrow Flight We added support for long-running queries use case. ### Arrow Flight SQL There have been a few discussions about small improvements as systems implement FlightSQL more broadly ### Arrow Flight SQL adapter for PostgreSQL This is a new sub project that adds Arrow Flight SQL endpoint to PostgreSQL. We have released the first version: https://arrow.apache.org/blog/2023/09/13/flight-sql-postgresql-0.1.0-release/ ### DataFusion DataFusion continues releasing regularly. We are working on a paper describing the system for ACM SIGMOD, and in general are trying to scale the project as it grows in popularity. We have added User Defined Window functions, much improved List and Struct support, Spillable Hashing, better join performance, parallel writes, and a new library user guide. ### Acero No Update ### nanoarrow We released version 0.3.0, which focused on solidifying the C library and IPC implementation. ## Language Area Updates Arrow has at least 13 different language implementations, as explained in https://arrow.apache.org/overview/ Arrow 13.0.0 was released from the monorepo: https://arrow.apache.org/blog/2023/08/24/13.0.0-release/ ### C++ StringView and ListView data types were added to the C++ implementation, along with ArrowDeviceArray and ArrowDeviceArrayStreams for non-CPU devices. ### C# The C# implementation gained support for Flight SQL servers and clients. It also added support for fixed-size lists, dense and sparse unions and maps, and is now complete enough to express the types used in ADBC schemas. Its C API support has been hooked up to the formal C Data interface integration testing. ### Go StringView and ListView data types are added to the Go implementation. ### Java Support for Java 21 has been added. ### JavaScript ### Julia No highlight. ### Rust The Rust implementation reworked the computation kernels, and is in the process of adding StringView. ### C (GLib) We have added new bindings continually as usual. ### MATLAB We started to use GitHub Projects to organize the MATLAB bindings tasks: https://github.com/orgs/apache/projects/289. We have also continued to expand the MATLAB bindings by adding tabular types (i.e. RecordBatch and Table), more Array types (struct, date, time, timestamp, …), and CSV file I/O. We are actively working towards cutting an initial “0.1” release of the MATLAB bindings in the coming weeks. ### Python ### R Added bindings to the C++ functionality for reading ND-JSON, and continued to work to ensure dplyr compatibility. ### Ruby We have added new bindings continually as usual. ### Swift No highlight. ## Recent Releases: * NANOARROW-0.3.0 was released on 2023-09-29. * RS-OS-0.7.1 was released on 2023-09-29. * ADBC-0.7.0 was released on 2023-09-23. * RS-47.0.0 was released on 2023-09-22. * RS-DATAFUSION-PYTHON-31.0.0 was released on 2023-09-18. * FLIGHT-SQL-POSTGRESQL-0.1.0 was released on 2023-09-13. * RS-DATAFUSION-31.0.0 was released on 2023-09-11. * ADBC-0.6.0 was released on 2023-08-28. * RS-DATAFUSION-30.0.0 was released on 2023-08-25. * RS-46.0.0 was released on 2023-08-24. * 13.0.0 was released on 2023-08-23. * RS-OS-0.7.0 was released on 2023-08-18. * RS-DATAFUSION-29.0.0 was released on 2023-08-14. * RS-45.0.0 was released on 2023-08-02. * RS-DATAFUSION-28.0.0 was released on 2023-07-25. * RS-44.0.0 was released on 2023-07-18. ## Community Health: Community communication continues to be strong. There have been 5 blog posts published to https://arrow.apache.org/blog/ in the last 3 months. The mailing lists are active * dev@arrow.apache.org had a 32% decrease in traffic in the past quarter (545 emails compared to 798) * user@arrow.apache.org had a 37% decrease in traffic in the past quarter (76 emails compared to 120) For the mono repo: * 2266 commits in the past quarter (-2% change) * 249 code contributors in the past quarter (no change) * 1788 PRs opened on GitHub, past quarter (-12% change) * 1682 PRs closed on GitHub, past quarter (-16% change) * 1499 issues opened on GitHub, past quarter (-8% change) * 1211 issues closed on GitHub, past quarter (-11% change)
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. More information can be found at https://arrow.apache.org/overview/ ## Project Status: Current project status: Ongoing (high activity) Issues for the board: None ## Membership Data: Apache Arrow was founded 2016-01-19 (7 years ago) There are currently 97 committers and 50 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - Ben Baumgold was added to the PMC on 2023-06-19 - Jie Wen was added to the PMC on 2023-06-10 - Dewey Dunnington was added to the PMC on 2023-06-22 - Matthew Topol was added to the PMC on 2023-05-02 - Gang Wu was added as committer on 2023-05-15 - Kevin Gurney was added as committer on 2023-07-04 - Marco Neumann was added as committer on 2023-05-11 - Mehmet Ozan Kabak was added as committer on 2023-06-10 - Ruihang Xia was added as committer on 2023-04-15 ## Project Activity: There has been healthy debate about adding new formats, [StringArray] and [ListView], focused on increasing Arrow’s appeal in high performance computation engines. We have completed the transition from JIRA to using GitHub issues for the mono repo and that appears to be going well. The DataFusion subproject is considering applying to become its own top level Apache project (see DataFusion update below) [StringArray]: https://lists.apache.org/thread/c6frlr9gcxy8qdhbmv8cn3rdjbrqxb1v [ListView]: https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb ## Sub Project Updates Arrow has several subprojects, as listed on https://arrow.apache.org/ ### ADBC We have released 2 new minor versions. They include new drivers and new implementations. ### Arrow Flight We have added new features to the Arrow Flight specification: 1. Ordered data support: https://github.com/apache/arrow/issues/34852 2. Resultset expiration support: https://github.com/apache/arrow/issues/35500 ### Arrow Flight SQL We have updated the Arrow Flight SQL specifications based on the above Arrow Flight update. ### DataFusion DataFusion continues to grow and mature. The community added many new features as described in the latest [blog] post, and discussed and came to consensus on the [goals] of the project and is discussing a [move to its own top level Apache project]. Current development focus is on performance and adding better support for structured types such as Lists and Structs. We expect more work on improving documentation and communicating externally over the next quarter. [blog]: https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/ [goals]: https://github.com/apache/arrow-datafusion/discussions/6441 [move to its own top level Apache project]: https://github.com/apache/arrow-datafusion/discussions/6475 ## Language Area Updates Arrow has at least 12 different language implementations, as explained in https://arrow.apache.org/overview/ Arrow 12.0.0 was released from the monorepo: https://arrow.apache.org/blog/2023/05/02/12.0.0-release/ ### C++ PRs have been created with example implementations of two new layouts, Array View and String View. These layouts are motivated by Arrow-compatible engines which found these layouts to be more efficient for their workflows. As mentioned in the previous report, the C++ compute engine Acero was broken out into a separate module and Arrow-C++ can now be built without it, allowing for more modular feature configuration. ### C# C# now has a complete implementation of the C data interface, allowing for efficient intra-process communication between C# and other languages. In addition, there has been some early discussion ### Go PRs were created with the example implementation of StringView for Go to be the second implementation in order to vote on the layout. Changes were introduced to improve compatibility with x86 (32-bit) systems and TinyGo builds for WebAssembly builds, along with corresponding CI builds. A default Arrow Flight middleware was added for handling Cookies via gRPC headers. Usage of the Go implementation continues to grow and expand in the community. ### Java Ongoing maintenance of the Arrow Java implementation remains steady. ### JavaScript ### Julia We have released new versions rapidly when we fix a problem. New PMC member who focuses on Julia has joined. There are 2 PMC members who focus on Julia now. ### nanoarrow The 0.2.0 release of nanoarrow featured support for decoding the Arrow IPC format and included a number of interface improvements and bugfixes resulting from early usage. Ongoing work includes support for non-CPU data via the Arrow C Device interface and documentation improvements suggested by early users of the library. ### Rust The Rust implementation has been focused on improving the UX of the API, the speed, consistency and correctness (timezones!) of the kernels. ### C (GLib) We have added new bindings continually as usual. ### MATLAB New committer who focuses on MATLAB has joined. The committer is the first committer who focuses on MATLAB. We’ll expand the MATLAB community. Integrated support for mathworks/libmexclass, enabling streamlined development of the MATLAB interface. As a result, significant progress has been made on public MATLAB APIs, including support for Array and RecordBatch construction from equivalent MATLAB types (e.g. table). Recently merged Windows and ccache CI support, bridging the platform gap for MATLAB qualification. This will help ensure quality of PRs and improve developer confidence when making changes. Next steps for the MATLAB interface include working on compound / nested data types and tabular file I/O workflows. ### Python The python community is embracing “protocols”, which allow for library-agnostic interchange and duck-typing. Pyarrow has added support for the dataframe interchange protocol which maps to pyarrow’s Table class. In addition, some early discussion has begun around a dataset protocol based on pyarrow’s datasets API. ### R The R bindings now support JSON Datasets and continue to benefit from ongoing performance enhancements and feature additions in the C++ library. ### Ruby Ruby related questions/issue reports were increased. It shows that user base of the Ruby bindings is increasing. ### Swift We have started implementing Arrow Flight. ## Community Health: Community communication continues to be strong. There have been 9 blog posts published to https://arrow.apache.org/blog/ in the last 3 months, including two from community members on their use of Arrow The mailing lists are active * dev@arrow.apache.org had a 10% decrease in traffic in the past quarter (779 emails compared to 858) * jira@arrow.apache.org had a 100% decrease in traffic in the past quarter (0 emails compared to 10778) For the mono repo: * 2275 commits in the past quarter (5% increase) * 254 code contributors in the past quarter (1% increase) * 1986 PRs opened on GitHub, past quarter (-6% change) * (1954 PRs closed on GitHub, past quarter (-11% change) * 1573 issues opened on GitHub, past quarter (-11% change) * 1342 issues closed on GitHub, past quarter (-5% change)
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. More information can be found at https://arrow.apache.org/overview/ ## Issues: There are no specific issues to report ## Membership Data: Apache Arrow was founded 2016-01-19 (7 years ago) There are currently 92 committers and 46 PMC members in this project. The Committer-to-PMC ratio is 2:1. Community changes, past quarter: - Will Jones was added to the PMC on 2023-03-12 - Mustafa Akur was added as committer on 2023-04-05 - Brent Gardner was added as committer on 2023-01-17 - Wang Mingming was added as committer on 2023-02-21 ## Project Activity: The main mono repo moved to github and from JIRA for issues reporting, to lower the friction for new contributors to participate. This friction was partly caused by the new requirement for PMC to create JIRA accounts, as well as the need for 2 separate accounts (JIRA and github). A run-end encoded (REE) layout has been added to the Arrow Columnar Format: https://arrow.apache.org/docs/format/Columnar.html#run-end-encoded-layout REE is a variation of run-length encoding (RLE) and is well-suited for representing data containing sequences of the same value in a way that allows efficient random access from a logical index using binary search. The Arrow Columnar Format now defines canonical extension types: https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html#official-list This provides a way to extend Arrow data types by including the specification in the official Arrow documentation. This way the extension type is specified as a canonical extension type. The first canonical extension type to be added to the list is Fixed Shape Tensor Extension Type. The implementation of which will also be added into the Arrow C++. ## Community Health: In response to several queries about becoming a committer, the PMC agreed upon a public description of the criteria used, which is now posted posted at https://arrow.apache.org/committers/ There have been 8 blog posts published to https://arrow.apache.org/blog/ in the last 3 months, including one from the community https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/ about mapping telemetry data into Arrow. ## Language Area Updates Arrow has at least 12 different language implementations, as explained in https://arrow.apache.org/overview/ Arrow 11.0.0 was released from the monorepo: https://arrow.apache.org/blog/2023/01/25/11.0.0-release/ ### ADBC Arrow ADBC was released as part of a larger effort to foster integration at a higher level. See the relevant blog entries https://arrow.apache.org/blog/2023/02/16/adbc-0.2.0-release/ and https://arrow.apache.org/blog/2023/03/21/adbc-0.3.0-release/ for more detail A proposal for Rust support is underway, and there are discussions about extending the API specification as well. ### C++ The new run-end encoded array type has been added to the C++ implementation. This array type was inspired by Arrow-compatible engines. Work will continue in the subsequent releases to integrate this data type into additional parts of the C++ codebase, such as the compute kernels. The Plasma subproject will be removed in the 12.0.0 release. It was originally donated by the Ray project, but the original maintainers from Ray forked it and reintegrated it as a private utility for Ray, since a tighter coupling allowed them to make better optimizations. The Arrow subproject has been mostly unmaintained since. However, in the meantime Arrow has added its own homegrown protocols for exchanging data: Flight RPC and the C Data Interface. The Acero compute engine, currently embedded within the core libarrow shared library, is being pulled out into a separate library. This will allow the C++ distribution to be more modular and also help support splitting the higher-level bindings (such as Python) into smaller sub-packages. ### C# Work has started on adding the C Data Interface to the C# implementation. This will allow this implementation to create C# wrappers around Arrow libraries implemented in other languages, such as the C++ Parquet reader or the DataFusion query engine. ### Go The Go implementation has released various compute functionality natively in Go, leveraging SIMD where able. This includes various arithmetic functions along with filter mechanisms for Arrays, Records and Tables. Work has also started on integrating Substrait with this compute functionality. The new run-end encoded type array was added to the Go implementation, inspired by Arrow-compatible engines. This has been integrated into additional parts of the code-base including scalars, and parts of the compute package. ADBC has a native Go definition via interfaces along with native Go implementations for the Flight SQL driver and CGO implementations to load any other desired ADBC drivers. There is also an adapter to leverage any ADBC drivers via Go’s standard database/sql package. The Go Flight SQL driver is utilized and wrapped to be deployed with the Python ADBC package as a shared library with almost no dependencies, making it ideal for use in the python wheels. There’s been a significant uptick in community contributions and interest in the Go implementation. In particular, significant contributions have been coming from CloudQuery (https://github.com/cloudquery/cloudquery) as they shift to leverage Arrow as their primary type system and move their codebase to utilizing Arrow. We’ve also seen OpenTelemetry utilize the Go Arrow implementation to prototype shifting their telemetry memory format to using Arrow for communication benefits, leading to a series of blog posts (the first of which has been posted: https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/). ### Java Publish modules also as a Software Bill of Materials SBOM artifacts thru CycloneDX plugin. Expose writer compression supports Lz4/Zstd for ArrowFileWriter/ArrowStreamWriter. Split flight-sql-jdbc-driver module to facilitate reuse. Working to incorporate Substrait consumer into Dataset module thru JNI binding to call Acero. ### JavaScript ### Julia ### nanoarrow The nanoarrow C library and associated R bindings had their first release: https://arrow.apache.org/blog/2023/03/07/nanoarrow-0.1.0-release/. Additionally, support for reading Arrow IPC streams is mostly complete and will be included in the forthcoming 0.2 release ### Rust Rust has several projects: arrow-rs (arrow, parquet, arrow-flight object_store implementations) arrow-datafusion: Rust query engine arrow-ballista: Distributed query engine The arrow-rs Rust implementation has been working on improving the lower level array implementations both for better use of ergonomics as well as interoperability with another, non ASF governed Rust Arrow implementation, arrow2. The communities of arrow and arrow2 are working on a high level plan to merge to combine resources. See https://github.com/jorgecarleitao/arrow2/issues/1429 . We also made substantial performance improvements to CSV parsing, JSON parsing, and date / timestamp parsing and hope to blog about that soon The DataFusion project gained several new public users, and contributions to it have grown substantially. We continue to struggle to with finding enough review capacity to keep up with contributions. On the upside there are several maintainers who regularly review and merge code regularly. We are also working on improving our documentation site https://arrow.apache.org/datafusion/ which is now more regularly updated. We also continue calendar based (every 2 weeks) release train which has served the projects well. ### C (GLib) We added more bindings of the C++ implementation. ### MATLAB 1. In response to helpful feedback from the community [1], a significant refactor [2] was done to mathworks/libmexclass [3] (a framework which makes it possible to implement MATLAB classes in terms of corresponding C++ classes using MEX [4]) in order to support tighter integration with the upstream Apache Arrow codebase. There is an open pull request to merge in the refactored libmexclass code here: https://github.com/apache/arrow/pull/34563 2. The first arrow.array.<Array> class (i.e. arrow.array.Float64Array) is also included as part of this pull request. After this pull request has been merged, development efforts will continue to be focused on building out the MATLAB Interface to Arrow, starting with the rest of the core arrow.array.<Array> types. [1] https://github.com/apache/arrow/pull/34563#discussion_r1137910630 [2] https://github.com/mathworks/libmexclass/pull/55 [3] https://github.com/mathworks/libmexclass [4] https://www.mathworks.com/help/matlab/matlab_external/structure-of-c-mex-function.html ### Python The Dataframe Interchange Protocol (https://data-apis.org/dataframe-protocol/latest/index.html) is now implemented for pyarrow Table and RecordBatch. The protocol enables conversion for basic data types between Python dataframe libraries that have the protocol implemented and therefore serves the same purpose as the Arrow C Data Interface but with less data type coverage. As it is a Python protocol it is a bit easier for the Python libraries to implement. With it PyArrow will be able to exchange data with the libraries that do not support Arrow format. Some examples of the libraries that currently support the protocol are: cuDF, modin, pandas, polars and vaex. ### R The nanoarrow project had its first CRAN release; this provides users with a minimal set of bindings to work with Arrow C Data and/or Arrow C Stream structures in circumstances where they wish to work with Arrow but cannot or do not want to work with the full library. There has been a substantial rewrite of the docs, which should make it easier for users to find help with the R package, and we've written some new functions to simplify the process of working with datasets. Various updates have been done to ensure that the package is compatible with the latest version of dplyr, the API which the arrow R package implements.. ### Ruby We added more Ruby-ish APIs. ### Swift We started implementing the Swift implementation. ## Release activity Recent releases: * RS-37.0.0 was released on 2023-04-10. * RS-OS-0.5.6 was released on 2023-04-03. * RS-DATAFUSION-21.1.0: was released on 2023-04-02. * RS-36.0.0 was released on 2023-03-28. * RS-DATAFUSION-21.0.0 was released on 2023-03-27. * ADBC-0.3.0 was released on 2023-03-21. * RS-DATAFUSION-PYTHON-20.0.0 was released on 2023-03-20. * JULIA-2.5.0 was released on 2023-03-16. * RS-DATAFUSION-20.0.0 was released on 2023-03-14. * RS-35.0.0 was released on 2023-03-13. * NANOARROW-0.1.0 was released on 2023-03-07. * RS-OS-0.5.5 was released on 2023-03-02. * RS-34.0.0 was released on 2023-02-27. * ADBC-0.2.0 was released on 2023-02-15. * RS-33.0.0 was released on 2023-02-13. * RS-DATAFUSION-18.0.0 was released on 2023-02-13. * RS-OS-0.5.4 was released on 2023-02-07. * JULIA-2.4.3 was released on 2023-02-04. * RS-32.0.0 was released on 2023-01-30. * RS-DATAFUSION-17.0.0 was released on 2023-01-30. * 11.0.0 was released on 2023-01-25. * RS-31.0.0 was released on 2023-01-16. * JULIA-2.4.2 was released on 2023-01-15. * RS-DATAFUSION-16.0.0 was released on 2023-01-15.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: Lack of ASF sponsored invite-free chat service is a minor source of friction for community building. Most subprojects now use github for tickets to lower the barrier to entry for new / casual contributors, but we still have fragmented stories for group chat. ASF Slack requires an invite and some sub communities use other chat-like services. ## Membership Data: Apache Arrow was founded 2016-01-20 (7 years ago) There are currently 89 committers and 45 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - Kun Liu was added to the PMC on 2022-11-13 - Jacob Quinn was added to the PMC on 2022-10-25 - Nicola Crane was added to the PMC on 2022-10-25 - Jacob Wujciak was added as committer on 2022-12-19 - Ben Baumgold was added as committer on 2022-10-26 - Bogumił Kamiński was added as committer on 2022-10-24 - Eric Hanson was added as committer on 2022-10-26 - Jie Wen was added as committer on 2023-01-08 - Jarrett Revels was added as committer on 2022-11-02 - Curtis Vogt was added as committer on 2022-11-02 - Raúl Cumplido was added as committer on 2022-12-05 - Will Jones was added as committer on 2022-10-28 - Yang Jiang was added as committer on 2022-11-02 ## Project Activity: * Switching from JIRA to github issues in order to keep the overhead for new contributors low (no need to register for an ASF JIRA account) * [ADBC] (Arrow Database Connectivity) first release: * Community voted to add RLE to the specification * Additional subproject updates are below * We continue to release several different products and releases per quarter [ADBC]: https://arrow.apache.org/blog/2023/01/05/introducing-arrow-adbc/ Recent releases: ADBC-0.1.0 was released on 2023-01-10. RS-30.0.1 was released on 2023-01-08. RS-OS-0.5.3 was released on 2023-01-08. RS-30.0.0 was released on 2023-01-03. RS-29.0.0 was released on 2022-12-12. RS-OS-0.5.2 was released on 2022-12-07. RS-DATAFUSION-15.0.0 was released on 2022-12-05. DATAFUSION-PYTHON-0.7.0 was released on 2022-11-29. RS-28.0.0 was released on 2022-11-28. 10.0.1 was released on 2022-11-22. RS-BALLISTA-0.10.0 was released on 2022-11-21. JULIA-2.4.1 was released on 2022-11-18. RS-27.0.0 was released on 2022-11-15. RS-DATAFUSION-14.0.0 was released on 2022-11-07. RS-26.0.0 was released on 2022-11-03. 10.0.0 was released on 2022-10-26. JULIA-2.4.0 was released on 2022-10-26. RS-BALLISTA-0.9.0 was released on 2022-10-26. RS-25.0.0 was released on 2022-10-17. ## Community Health: The community health appears good, discussions on the mailing lists and github are productive. We recently had a nice discussion on the State of the Project: https://lists.apache.org/thread/r8gl3wvjgy9k8n2t194r0bbdbxx6ksqc and discussed various ways to keep encouraging the community. ## Language Area Updates Arrow has at least 12 different language bindings, as explained in https://arrow.apache.org/overview/ Arrow 10.0.0 release: https://arrow.apache.org/blog/2022/10/31/10.0.0-release/ ### C++ ### C# ### Go We’re seeing significant increases in interest and usage of the Arrow Go library. From startups like Spice.AI to being incorporated and used in Google BigQuery’s quickstart example and more. 2022 was a big year of updates, fixes, and drumming up interest for the Go module that we hope to continue for increased adoption and usage. The Go module, along with C++, is used as the initial implementation for the Run-End Encoding array implementation. Future development plans are to continue to expand the compute capabilities of the Go module and extend integration with Substrait. ### Java ### JavaScript ### Julia We’ve worked again on simplifying and streamlining the administrative side for the Julia implementation; adding additional committers, simplifying the release process, etc. This has increased the rate of contributions, as expected. There’s interest in finishing the C data/stream interfaces for the Julia implementation soon. ### Rust Rust has several projects: arrow-rs (arrow, parquet, arrow-flight object_store implementations) arrow-datafusion: rust query engine arrow-ballista: distributed query engine We are working to incorporate substrait into DataFusion Working on external communication with several blog posts about technology on sorting Fast and Memory Efficient Multi-Column Sorts in Apache Arrow Rust, Part 1 and Querying Parquet with Millisecond Latency We also continue calendar based release train with good results. ### C (GLib) We’ve added support for 16-bit float type. ### MATLAB 1. We have been focusing development efforts on implementing an "object dispatch layer" that uses MEX to "connect" MATLAB objects with corresponding C++ objects. This code is being actively developed at github.com/mathworks/libmexclass. See the following Arrow mailing list discussion for more context. We hope to upstream the changes needed to make the MATLAB Interface to Arrow use libmexclass under the hood in the coming months. This should enable the MATLAB interface to wrap relevant Arrow C++ objects (e.g. arrow::Array, arrow::Table) and expose them to MATLAB. 2. We have been continuing to investigate Windows CI support for the MATLAB interface. Currently, only Linux and macOS are supported. ### Python ### R ### Ruby We’ve added support for 16-bit float type. There is a new contributor who develops a new data frame library based on the Ruby bindings. The new contributor upstreamed some improvements implemented in the downstream data frame library.
WHEREAS, the Board of Directors heretofore appointed Kouhei Sutou (kou) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Kouhei Sutou from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Andrew Lamb (alamb) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Kouhei Sutou is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Andrew Lamb be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7D, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Arrow was founded 2016-01-20 (7 years ago) There are currently 79 committers and 42 PMC members in this project. The Committer-to-PMC ratio is roughly 5:3. Community changes, past quarter: - Raphael Taylor-Davies was added to the PMC on 2022-09-20 - L. C. Hsieh was added to the PMC on 2022-09-03 - Weston Pace was added to the PMC on 2022-09-05 - Zhong Yanghong was added as committer on 2022-09-08 - Remzi Yang was added as committer on 2022-09-09 - Daniel Harris was added as committer on 2022-09-23 ## Project Activity: - RS-24.0.0 was released on 2022-10-03. - RS-BALLISTA-0.8.0 was released on 2022-09-20. - RS-23.0.0 was released on 2022-09-19. - RS-DF-12.0.0 was released on 2022-09-16. - RS-OS-0.5.0 was released on 2022-09-11. - RS-22.0.0 was released on 2022-09-05. - RS-21.0.0 was released on 2022-08-30. - RS-DF-11.0.0 was released on 2022-08-19. - RS-OS-0.4.0 was released on 2022-08-13. - RS-20.0.0 was released on 2022-08-09. - RS-19.0.0 was released on 2022-07-26. - 6.0.2 was released on 2022-07-19. - 7.0.1 was released on 2022-07-19. - 8.0.1 was released on 2022-07-19. - RS-DF-10.0.0 was released on 2022-07-15. - RS-18.0.0 was released on 2022-07-11. ## Community Health: - We're voting a new Apache Arrow Database Connectivity (ADBC) interface developing at https://github.com/apache/arrow-adbc . - We'll deprecate unmaintained Apache Arrow Plasma object store. - We've accepted Apache Arrow Flight SQL JDBC driver donation. - We've switched required C++ specification to C++17 from C++11. - We've changed issue assign policy on Jira: All users can assign themselves as assignee for easy to contribute. - We've accepted Apache Arrow Rust Object Store donation. - We've backed Apache Arrow DataFusion Python bindings to ASF.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Arrow was founded 2016-01-20 (6 years ago) There are currently 76 committers and 39 PMC members in this project. The Committer-to-PMC ratio is roughly 5:3. Community changes, past quarter: - No new PMC members. Last addition was QP Hou on 2022-02-17. - Alenka Frim was added as committer on 2022-06-23 - Dewey Dunnington was added as committer on 2022-06-15 - Rok Mihevc was added as committer on 2022-06-15 - L. C. Hsieh was added as committer on 2022-04-28 ## Project Activity: Recent releases: - RS-17.0.0 was released on 2022-06-28. - RS-16.0.0 was released on 2022-06-13. - RS-15.0.0 was released on 2022-05-30. - RS-14.0.0 was released on 2022-05-16. - 8.0.0 was released on 2022-05-07. - RS-13.0.0 was released on 2022-05-03. - RS-12.0.0 was released on 2022-04-19. - RS-DF-7.1.0 was released on 2022-04-18. - JULIA-2.3.0 was released on 2022-04-12. ## Community Health: - A book about Apache Arrow was published from a community member. https://www.packtpub.com/product/in-memory-analytics-with-apache-arrow/9781801071031 - We've split Apache Arrow Ballista to https://github.com/apache/arrow-ballista from https://github.com/apache/arrow-datafusion . - We've marked Apache Arrow C Streaming Interface stable. - A conference that focuses on Apache Arrow was held by community members. https://thedatathread.com/ - We're improving our release process to reduce release management cost. - We've accepted donation of Apache Arrow Flight SQL JDBC driver.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Arrow was founded 2016-01-20 (6 years ago) There are currently 72 committers and 39 PMC members in this project. The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - QP Hou was added to the PMC on 2022-02-17 - Kun Liu was added as committer on 2022-03-09 - Jacob Quinn was added as committer on 2022-02-23 - Raphael Taylor-Davies was added as committer on 2022-03-24 - Xudong Wang was added as committer on 2022-03-16 - Yijie Shen was added as committer on 2022-03-09 ## Project Activity: Recent releases: - RS-11.1.0 was released on 2022-04-04. - RS-11.0.0 was released on 2022-03-21. - JULIA-2.2.1 was released on 2022-03-08. - RS-10.0.0 was released on 2022-03-08. - RS-9.1.0 was released on 2022-02-22. - RS-DF-7.0.0 was released on 2022-02-17. - RS-9.0.2 was released on 2022-02-13. - 7.0.0 was released on 2022-02-03. - RS-8.0.0 was released on 2022-01-25. - RS-7.0.0 was released on 2022-01-12. ## Community Health: - We've released a new version of Apache Arrow Julia after it was re-donated. - We'll start IP clearance for Apache Arrow Flight SQL JDBC driver donation. - We may register Apache Arrow as a Digital Public Good like Apache Fineract: https://digitalpublicgoods.net/registry/apache-fineract.html - We'll rename the "master" branch to "main": https://issues.apache.org/jira/browse/ARROW-15689 We'll be asking INFRA for support on this when we're ready. - We've rotated the PMC chair.
WHEREAS, the Board of Directors heretofore appointed Wes McKinney (wesm) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Wes McKinney from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Kouhei Sutou (kou) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Wes McKinney is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Kouhei Sutou be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7E, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * https://github.com/apache/arrow-julia is blocked by Infra on CI: * https://issues.apache.org/jira/browse/INFRA-22676 isn’t triaged. * https://github.com/apache/arrow-julia wants to run CI before merging pull requests. ## Membership Data: Apache Arrow was founded 2016-01-19 (6 years ago) There are currently 67 committers and 38 PMC members in this project. The Committer-to-PMC ratio is roughly 9:5. Community changes, past quarter: - Daniël Heres was added to the PMC on 2021-12-20 - Joris Van den Bossche was added to the PMC on 2021-11-21 - Yibo Cai was added to the PMC on 2022-01-03 - Alessandro Molina was added as committer on 2022-01-04 - Rémi Dettai was added as committer on 2021-12-07 ## Project Activity: Recent releases: * RS-6.5.0 was released on 2021-12-29. * RS-6.4.0 was released on 2021-12-14. * RS-6.3.0 was released on 2021-11-29. * 6.0.1 was released on 2021-11-18. * RS-DF-6.0.0 was released on 2021-11-17. * RS-6.2.0 was released on 2021-11-15. * RS-6.1.0 was released on 2021-11-03. * 6.0.0 was released on 2021-10-26. * RS-6.0.0 was released on 2021-10-17. Planned releases: * 7.0.0 in mid-January ## Community Health: * Apache Arrow Julia was re-donated https://s.apache.org/arrow-jl-donation * We’ve created a repository for collaborative prototyping + algorithms / performance research https://github.com/apache/arrow-experiments * Discussions around adding new columnar memory layouts to Arrow (in-memory, IPC, C ABI) * Discussions around community maintained extension repos for Apache Arrow Datafusion https://github.com/datafusion-contrib * We’ll rotate the PMC chair in 2022-01 * We’ve started Apache Arrow Cookbook to accept more new users. We’ll create similar documentation for new contributors. https://arrow.apache.org/cookbook/ * Rust continues releases every other week and is trying to balance open group collaboration and fast iterations: https://github.com/apache/arrow-rs/issues/1120 * Rust Arrow2 project, https://github.com/jorgecarleitao/arrow2, implementation of Arrow developed outside ASF, continues to generate community questions about if/when it will be brought into ASF.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (6 years ago) There are currently 64 committers and 35 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - Neville Dipale was added to the PMC on 2021-07-28 - QP Hou was added as committer on 2021-07-26 - Nicola Crane was added as committer on 2021-09-08 - Weston Pace was added as committer on 2021-07-09 - Matthew Topol was added as committer on 2021-08-27 - Jiayu Liu was added as committer on on 2021-10-07 ## Project Activity: - Planned upcoming 6.0 release in mid-October - Rust migration to multiple repositories (arrow-rs, arrow-datafusion) has been successful and the community has been consistently making releases. ## Community Health: - We’ve re-engaged with maintainers of the Julia port and are working on implementing the same development process that Rust has adopted (separate apache Repo). IP clearance will need to be redone. - Discussions around using Github issues as a support mechanism in conjunction or to replace user@
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * Most issues from previous board reports have been addressed though we would benefit as discussed from more mentorship in the Apache Way for new contributors. * Long GitHub Actions queueing times for CI jobs have been an occasional concern. We are aware that we are one of the most greedy consumers of GHA minutes in the ASF, so we will continue to think about future strategies to supplement GitHub Actions with dedicated build infrastructure. Our CI/CD needs are some of the most complex that we've observed in mainstream open source software. ## Membership Data: Apache Arrow was founded 2016-01-19 (5 years ago) There are currently 61 committers and 34 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - Ben Kietzman was added to the PMC on 2021-05-20 - David Li was added to the PMC on 2021-06-20 - Daniël Heres was added as committer on 2021-04-26 - Dominik Moritz was added as committer on 2021-06-02 - Ian Cook was added as committer on 2021-04-27 - Jonathan Keane was added as committer on 2021-04-27 - Kazuaki Ishizaki was added as committer on 2021-06-06 - Weston Pace was added as committer on 2021-07-09 ## Project Activity: * The Rust projects have successfully migrated to independent arrow-* repositories and begun making biweekly releases on a schedule. This progress seems to be going reasonably smoothly and the prior issues have been adequately addressed. * We are progressing as usual toward making our ~quarterly major release, now 5.0.0. ## Community Health: * We've succeeded at significantly expanding the committership and PMC this quarter, and our community growth in general continues to be robust.
## Description:
The mission of Apache Arrow is the creation and maintenance of software related
to columnar in-memory processing and data interchange
## Issues:
* CI Issues continue to be a problem. With the increase in utilization of
ASF’s allotment of Github Actions we’ve seen CI build time go from 20-30
minutes to as long as 5 hours
* There have been several issues around understanding exact requirements for
what it means to follow the Apache way and developing code within the
community:
* Several Rust projects were developed outside of the ASF repos and then
had proposals to be donated back.
* After accepting the donation of Julia developers continued to develop in
the original Julia repo and only make PRs back to the Arrow repo on an
intermittent basis. Since the Arrow community has been growing rapidly,
we had not done a thorough enough job of educating newcomers what it
means to develop “inside the community”.
* It appears a development model similar to what Rust is proposing
(see community health) will mitigate technical concerns for why this
was the development model chosen.
* It is unclear if JIRA use is required / encouraged for Apache projects
or if using a non Apache hosted system such as GitHub issues for parts
of the project is sufficiently community friendly
* In both cases it seems exact requirements of the ASF are not clear.
Through discussion it became apparent that the preferred method for
revolutionary changes to the existing code bases is by creating separate
development branches for prototyping with updates back to the community.
* ASF’s Artifactory isn’t ready yet. We don’t know the current status. See
also https://lists.apache.org/thread.html/r9200fe
d3fa812f8c7de07a2500425f258db3231baa8e05f288175e4a%40%3Cbuilds.apache.org%3E
## Membership Data:
Apache Arrow was founded 2016-01-19 (5 years ago)
There are currently 55 committers and 32 PMC members in this project.
The Committer-to-PMC ratio is roughly 7:4.
Community changes, past quarter:
- Andrew Lamb was added to the PMC on 2021-03-07
- Jorge Leitão was added to the PMC on 2021-03-07
- Yibo Cai was added as committer on 2021-03-05
## Project Activity:
We are aiming to have a 4.0 release this month.
## Community Health:
* Rust members of the community have proposed to follow a new development
workflow using separate Arrow github repos specifically for Rust components
and migrating away from JIRA to Github for issue tracking. Rust will also
start releasing on a different cadence then the rest of the language
bindings.
* The releases will still follow Apache Arrow governance (voting for
releases, etc)
* Depending on the experience of the Rust community, the Julia community may
follow a similar path in the future
@Roy: Follow up with board issues
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * After our past issues around scarce CI resources, we had managed to stabilize our CI setup around the relatively generous execution facilities provided by Github Actions, and a bunch of additional builds on other platforms (mostly for nightly builds). That equilibrium was unfortunately shattered after INFRA suddenly disallowed most third-party Github Actions helpers in response to security concerns. Our remaining Github Actions builds also seem to be queued longer. This has now been resolved (https://issues.apache.org/jira/browse/INFRA-21239) but it was hugely disruptive to our community. Concurrently, Travis-CI also started to be much stricter with resource consumption, which is currently affecting several of our nightly builds until we manage to migrate them to other CI services. These factors compounded mean we'll finally need to provide and set up our own CI resources. We hope to be able to migrate a significant number of builds to Buildkite, using execution resources paid by our sponsors. We also hope that INFRA rules around repository access and configuration won't stifle that endeavour. ## Membership Data: Apache Arrow was founded 2016-01-19 (5 years ago) There are currently 54 committers and 30 PMC members in this project. The Committer-to-PMC ratio is 9:5. Community changes, past quarter: - No new PMC members. Last addition was Francois Saint-Jacques on 2020-03-04. - Andrew Lamb was added as committer on 2020-11-06 ## Project Activity: - We'd like to release Arrow 3.0 around mid-January. The issues discussed above (diminished CI coverage due to disallowed third-party Github Actions, blocked nightly builds on Travis-CI, complex release procedure) make it more difficult than we'd like. - There seems to be growing interest in zero-copy zero-dependency cross-runtime interoperability using our "C data interface". It is now concretely possible to share data in-process between R, Python and Rust, without any shared code and without going through IPC mechanisms (the latter still possible, of course, using the Arrow IPC format). Other projects such as DuckDB have started using this interface. - The Arrow in-memory format is regularly being adopted and/or adapted to in other projects, especially in the dataframe and database communities. ## Community Health: * The project is overall very active. The Rust implementation is especially thriving, with numerous pull requests being submitted and merged in due time. The C++ implementation is advancing at similar speed as in previous reports. A Julia implementation was recently contributed. * Our pull request backlog has also grown slightly as a result of increased activity, but it does not seem to be posing a real problem for now. * The weight of our release procedure puts a burden on our capacity to release often. The "weight" is a function of the project complexity but also limitations in our CI/CD resources and feedback cycles (e.g. builds that run only once a day overnight often fail and we end up blocked on releasing until those failures are resolved). Our goal is to cut a new feature release roughly once every two months, which often ends up challenging. This situation also makes us reluctant to put out bugfix releases in-between, except for critical fixes. Improved CI/CD along with improvements to streamline the release process itself (for example, addressing procedural issues with preparing the Java project to release) would help a great deal. We also need more PMC members to step up and help with release management. Most of the release management over the lifetime of the project has fallen on the shoulders of only 4 out of our 30 PMCs.
WHEREAS, the Board of Directors heretofore appointed Jacques Nadeau (jacques) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Jacques Nadeau from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Wes McKinney (wesm) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Jacques Nadeau is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Wes McKinney be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7D, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues needing board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (5 years ago) There are currently 53 committers and 30 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Francois Saint-Jacques on 2020-03-04. - Jorge Leitão was added as committer on 2020-09-29 ## Project Activity: The project has good activity with between five and ten commits most days. Dev email list activity was down in August and September (~50% over earlier in the year) but some of that is likely due to all the energy leading up to the 1.0 release. We should continue to monitor the mail trends to see if there are any systemic changes or concerns. ## Community Health: Github actions have substantially relieved the CI pain within the project. Arrow continues to see a diverse set of contributions with the total number of code contributors approaching 500. The project decided to move to a rotating PMC chair, with the first rotation planned for this month.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. The project has some level of support for 11 different programming languages. ## Issues: - There are no issues requiring board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (4 years ago) There are currently 52 committers and 30 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Francois Saint-Jacques on 2020-03-04. - Liya Fan was added as committer on 2020-06-09 - Ji Liu was added as committer on 2020-06-09 ## Project Activity: - We made the 0.17.0 and 0.17.1 releases since the last board report. The community is readying a 1.0.0 release which will formally mark stability in the Arrow columnar format binary protocol and a move to semantic versioning of the Arrow libraries. - We moved new JIRA issue notifications off of the dev@ mailing list to issues@ and created a new jira@ mailing list to capture the full JIRA firehose. The reasoning is that this could help encourage more participating in mailing list discussions. - We made changes in the codebase to remove uses of potentially non-neutral language that has been changed in many other open source projects. ## Community Health: The project and contributor base continues to grow in size and scope. We now have over 500 unique contributors since the creation of the project.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. The project has some level of support for 11 different programming languages. ## Issues: - We are continuing to work with INFRA on issues related to self-hosted CI machines integrated with our GitHub-based pull request workflows. There are two avenues we are exploring (and we may well use both of them), GitHub Actions Self-hosted and Buildkite. Per INFRA-19217 Buildkite has just been approved for the @apache GitHub organization and we have validated that we can successfully use this with the free Arrow organization that Buildkite has provided us. CI/CD is likely to require an ongoing significant investment of time, and we are doing the best we can try avoid overburdening ASF Infra with requests. ## Membership Data: Apache Arrow was founded 2016-01-19 (4 years ago) There are currently 50 committers and 30 PMC members in this project. The Committer-to-PMC ratio is 5:3. Community changes, past quarter: - Francois Saint-Jacques was added to the PMC on 2020-03-04 - Neal Richardson was added to the PMC on 2020-03-04 - No new committers. Last addition was Joris Van den Bossche on 2019-12-06. ## Project Activity: - 0.16.0 was released at the end of January. We are close to releasing 0.17.0, with a 1.0.0 release hopefully sometime in 2020. - Three months ago, Apache Arrow was accepted for continuous fuzzing in the OSS-Fuzz infrastructure. We have now finally stabilized the situation by fixing all detected issues in the Arrow C++ IPC implementation, and are actively fixing issues in the Arrow C++ Parquet reader. - We just adopted a "C Data Interface" for the project which will open many new opportunities for integrations with third party projects. ## Community Health: The project and contributor base continues to grow in size and scope. We now have over 400 unique contributors since the creation of the project.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (4 years ago) There are currently 50 committers and 28 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Micah Kornfield on 2019-08-21. - Eric Erhardt was added as committer on 2019-10-18 - Joris Van den Bossche was added as committer on 2019-12-06 ## Project Activity: * We have completed our initial migration away from Travis CI for continuous integration and patch validation to use the new GitHub Actions (GHA) service. We are much happier with the compute resource allocation provided by GitHub but longer term we are concerned that the generous free allocation may not continue and would be interested to know what kinds of guarantees (if any) GitHub may make to the ASF regarding GHA. * We are not out of the woods on CI/CD as there are features of Apache Arrow that we cannot test in GitHub Actions. We are still considering options for running these optional test workloads as well as other kinds of periodic workloads like benchmarking * We hope to make a 1.0.0 release of the project in early 2020. We had thought that our next major release after 0.15.0 would be 1.0.0 but we have not yet completed some necessary work items that the community has agreed are essential to graduate to 1.0.0 Recent releases: 0.15.0 was released on 2019-10-05. 0.14.1 was released on 2019-07-21. 0.14.0 was released on 2019-07-04. ## Community Health: The developer community is healthy and continues to grow.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * There are no issues requiring board attention at this time. ## Membership Data: * Apache Arrow was founded 2016-01-19 (4 years ago) * There are currently 48 committers and 28 PMC members in this project. * The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - Micah Kornfield was added to the PMC on 2019-08-21 - Sebastien Binet was added to the PMC on 2019-08-21 - Ben Kietzman was added as committer on 2019-09-07 - David Li was added as committer on 2019-08-30 - Kenta Murata was added as committer on 2019-09-05 - Neal Richardson was added as committer on 2019-09-05 - Praveen Kumar was added as committer on 2019-07-14 ## Project Activity: * The project has just made a 0.15.0 release. * We are discussing ways to make the Arrow libraries as accessible as possible to downstream projects for minimal use cases while allowing the development of more comprehensive "standard libraries" with larger dependency stacks in the project * We plan to make a 1.0.0 release as our next major release, at which time we will declare that the Arrow binary protocol is stable with forward and backward compatibility guarantees ## Community Health: * The community is continuing to grow at a great rate. We see good adoption among many other projects and fast growth of key metrics. * Many contributors are struggling with the slowness of pre-commit CI. Arrow has a large number of different platforms and components and a complex build matrix. As new commits come in, they frequently take a long time to complete. The community is trying several ways to solve this. There is bubbling frustration in the community around the GitHub repo rules for using third party services. This is especially challenging when there are free solutions to relieve the community pressure but the community is unable to access these resources. This frustration is greatest among people who work on many non-asf OSS projects which don't have such restrictive rules around GitHub. Some examples of ways the community has tried to resolve these have included: * Try to use CircleCI, rejected in INFRA-15964 * Try to use Azure Pipelines, rejected in INFRA-17030 * Try to resolves Issues with Travis CI capacity: INFRA-18533 & https://s.apache.org/ci-capacity (no resolution beyond "find donations") * The creation of new infrastructure design (in progress but a huge amount of thankless work) * While the community has seen great growth in contribution (more than 300 unique contributors at this point), the vast majority are casual contributors. The daily active committers (the workhorses of the project that bear the load committing the constant PRs, more than 5000 closed at this point) have been growing slower than adoption. This is despite the fact that the community has been very aggressive at being inclusive of new committers (with likelihood to have more than 50 in the next week). The community is still continuing to try to brainstorm ways to improve this.
@David: close the loop with Arrow and other communities with needs for CI service improvements
## Description:
Apache Arrow is a cross-language development platform for in-memory
data. It specifies a standardized language-independent columnar memory
format for flat and hierarchical data, organized for efficient
analytic operations on modern hardware. It also provides computational
libraries and zero-copy streaming messaging and interprocess
communication. Languages currently supported include C, C++, C#, Go,
Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The community is discussing a 1.0.0 release featuring
forward-looking binary format stability guarantees. Given the
nature of the project, this is obviously an important milestone
for adoption and user support
- Since the last report, a new Buildbot-based CI system has been
connected to apache/arrow to provide additional build capacity, with
a bot system called "ursabot" to provide on demand builds, benchmark
comparisons, and other tools to assist the developer community
## Health report:
- We have been having significant problems with CI build times and are
discussing strategies to decouple our de velopment process from the
shared pool of ASF-managed cloud CI resources like Travis CI and
Appveyor
- The community is healthy, though there were some concerns
around the 0.14.0 release vote and we are discussing
conventions around handling issues raised during release
candidate vetting.
## PMC changes:
- Currently 26 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Andrew Grove on Sun Feb 03 2019
## Committer base changes:
- Currently 43 committers.
- New committers:
- Francois Saint-Jacques was added as a committer on Wed Jun 12 2019
- Neville Dipale was added as a committer on Mon May 13 2019
- Praveen Kumar has also been invited to be a committer and accepted.
Acct request is in process.
## Releases:
- 0.14.0 was released on Wed Jul 03 2019
## JIRA activity:
- 735 JIRA tickets created in the last 3 months
- 690 JIRA tickets closed/resolved in the last 3 months
@Rich: Follow up about release vote concerns
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported
include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The project received a donation of DataFusion, a Rust-based query
engine for Apache Arrow
## Health report:
- The project is very healthy, with a growing number and diversity of
contributors
## PMC changes:
- Currently 26 PMC members.
- Andrew Grove was added to the PMC on Sun Feb 03 2019
## Committer base changes:
- Currently 41 committers.
- New commmitters:
- Micah Kornfield was added as a committer on Fri Mar 08 2019
- Deepak Majeti was added as a committer on Thu Jan 31 2019
- Paddy Horan was added as a committer on Fri Feb 08 2019
- Ravindra Pindikura was added as a committer on Fri Feb 01 2019
- Sun Chao was added as a committer on Fri Feb 22 2019
## Releases:
- 0.12.0 was released on Sat Jan 26 2019
- 0.12.1 was released on Sun Feb 24 2019
- 0.13.0 was released on Sun Mar 31 2019
- JS-0.4.0 was released on Tue Feb 05 2019
- JS-0.4.1 was released on Sat Mar 23 2019
## JIRA activity:
- 969 JIRA tickets created in the last 3 months
- 861 JIRA tickets closed/resolved in the last 3 months
## Description:
- Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported
include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The project received 3 code donations in the last quarter: a native C#
library, a Parquet integration library for Ruby, and a native Parquet
implementation in Rust
- The community is considering a donation of DataFusion an in-memory query
engine built on top of the Rust Arrow library
- Community is actively working on 0.12 release
- Community has discussed starting to do time-based releases approximately
every two months
## Health report:
- The project is very healthy, with a growing number and diversity of
contributors
## PMC changes:
- Currently 25 PMC members.
- Krisztian Szucs was added to the PMC on Wed Nov 07 2018
## Committer base changes:
- Currently 36 committers.
- New commmitters:
- Romain Francois was added as a committer on Fri Nov 09 2018
- Sebastien Binet was added as a committer on Thu Nov 08 2018
- Yosuke Shiro was added as a committer on Thu Nov 08 2018
## Releases:
- 0.11.1 was released on Mon Oct 22 2018
## JIRA activity:
- 712 JIRA tickets created in the last 3 months
- 601 JIRA tickets closed/resolved in the last 3 months
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported include
C, C++, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The Arrow and Parquet communities resolved by vote to merge their respective
C++ codebases in the Apache Arrow repository. This work was completed this
quarter
- The project received two code donations via IP clearance: a GLib interface to
the Parquet C++ libraries, and the Gandiva LLVM vectorized Arrow expression
compiler
- Work has commenced on R language integration with the C++ libraries
- An initial MATLAB binding to the C++ libraries was contributed
- The community is discussing receiving a proposed native implementation of
Arrow in C# .NET
## Health report:
- The project is very healthy, though rapid user and contributor growth has
stressed the limits of our developer tooling and put a great deal of burden
on the active project maintainers
## PMC changes:
- Currently 24 PMC members.
- Antoine Pitrou was added to the PMC on Mon Aug 20 2018
## Committer base changes:
- Currently 33 committers.
- New commmitters:
- Andrew Grove was added as a committer on Tue Aug 07 2018
- Krisztian Szucs was added as a committer on Thu Aug 16 2018
## Releases:
- 0.10.0 was released on Sun Aug 05 2018
- 0.11.0 was released on Sun Oct 07 2018
## JIRA activity:
- 649 JIRA tickets created in the last 3 months
- 476 JIRA tickets closed/resolved in the last 3 months
## Description: Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, Go, Java, JavaScript, Python, Ruby, and Rust. ## Issues: - There are no issues requiring board attention at this time ## Activity: - We have not released since March as we work to improve our release and build automation. We plan to include binary artifacts in our next release vote, where we have only had source artifacts in past releases. - We received a code donation of a Ruby interface via the usual IP Clearance process - There is a new Arrow + LLVM analytics project (Gandiva) happening outside of the Arrow community. We discussed incorporating this work into Apache Arrow but have not made any decisions yet. ## Health report: The project's user and contributor base is growing rapidly. We are struggling a bit with maintainer bandwidth. As an example, 2 committers have merged 84% of patches (where there have been nearly 2000) since the project's inception. We are discussing ways to grow the maintainer base on the mailing list. ## PMC changes: - Currently 23 PMC members. - Siddharth Teotia was added to the PMC on Thu May 17 2018 ## Committer base changes: - Currently 31 committers. - No new committers added in the last 3 months - Last committer addition was Antoine Pitrou at Tue Apr 03 2018 ## Releases: - Last release was 0.9.0 on Mon Mar 19 2018 ## JIRA activity: - 392 JIRA tickets created in the last 3 months - 303 JIRA tickets closed/resolved in the last 3 months
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported include
C, C++, Go, Java, JavaScript, Python, Ruby, and Rust.
## Issues:
There are no issues requiring board attention at this time
## Activity:
- We received a donation of a Go implementation since the last report
- New contributors have started developing native Rust bindings. Given its
early development, community will likely start by doing regular Rust-only
releases at a higher velocity than primary releases (similar to JS)
- Given the number of different artifacts developed, the build and release
complexity has gotten too difficult (too many manual steps). The
community is discussing ways to improve the process to simplify development.
## Health report:
Project is very healthy with a growing developer and user community.
Overall development activity has increased since the prior quarter
with the addition of two new programming languages (Go, Rust) to the
project.
## PMC changes:
- Currently 22 PMC members.
- New PMC members:
- Phillip Cloud was added to the PMC on Thu Mar 15 2018
- Philipp Moritz was added to the PMC on Thu Mar 15 2018
## Committer base changes:
- Currently 31 committers.
- New commmitters:
- Antoine Pitrou was added as a committer on Tue Apr 03 2018
- Brian Hulette was added as a committer on Mon Feb 12 2018
- Robert Nishihara was added as a committer on Mon Feb 19 2018
## Releases:
- 0.9.0 was released on Mon Mar 19 2018
- JS-0.3.0 was released on Wed Feb 21 2018
- JS-0.3.1 was released on Fri Mar 16 2018
## JIRA activity:
- 464 JIRA tickets created in the last 3 months
- 358 JIRA tickets closed/resolved in the last 3 months
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported include
C, C++, Java, JavaScript, Python, and Ruby.
## Issues:
There are no issues requiring board attention at this time
## Activity:
- Steady development activity from previous quarter and continued growth in
contributor base
- Added 5 new committers
- First JavaScript-only release (0.2.0) made on December 1
## Health report:
Project is very healthy with a growing developer and user community.
## PMC changes:
- Currently 20 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Kouhei Sutou on Fri Sep 15 2017
## Committer base changes:
- Currently 28 committers.
- New commmitters:
- Phillip Cloud was added as a committer on Tue Oct 03 2017
- Bryan Cutler was added as a committer on Wed Oct 04 2017
- Li Jin was added as a committer on Fri Oct 06 2017
- Paul Taylor was added as a committer on Fri Oct 06 2017
- Siddharth Teotia was added as a committer on Wed Oct 04 2017
## Releases:
- 0.8.0 was released on Sat Dec 16 2017
- JS-0.2.0 was released on Fri Dec 01 2017
## JIRA activity:
- 323 JIRA tickets created in the last 3 months
- 300 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. Arrow contains implementations or bindings in various stages in Java, C++, C, Ruby, Python, and JavaScript. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - We have added 6 new committers to the project - Continued development activity and growing community since the last board report. We have made 4 releases, with the next release 0.8.0 coming in a month or so - Development activity is increasing in JavaScript, and we are also looking to grow a community of R language developers in the coming quarters. ## Health report: - Arrow activity continues to increase with community interest and adoption. ## PMC changes: - Currently 20 PMC members. - Kouhei Sutou was added to the PMC on Fri Sep 15 2017 ## Committer base changes: - Currently 28 committers. - New commmitters: - Phillip Cloud was added as a committer on Tue Oct 03 2017 - Bryan Cutler was added as a committer on Wed Oct 04 2017 - Li Jin was added as a committer on Fri Oct 06 2017 - Philipp Moritz was added as a committer on Mon Aug 14 2017 - Paul Taylor was added as a committer on Fri Oct 06 2017 - Siddharth Teotia was added as a committer on Wed Oct 04 2017 ## Releases: - 0.5.0 was released on Sun Jul 23 2017 - 0.6.0 was released on Mon Aug 14 2017 - 0.7.0 was released on Sat Sep 16 2017 - 0.7.1 was released on Sun Oct 01 2017 ## JIRA activity: - 460 JIRA tickets created in the last 3 months - 382 JIRA tickets closed/resolved in the last 3 months
## Description:
Arrow is a columnar in-memory analytics layer designed to accelerate big data.
It houses a set of canonical in-memory representations of flat and hierarchical
data along with multiple language-bindings for structure manipulation. It also
provides IPC and common algorithm implementations.
## Issues:
- There are no issues requiring board attention at this time.
## Activity:
- Heavy development activity and growing community since the last board
report. We have made 3 releases, with the next release 0.5.0 coming soon.
- The Arrow 0.3.0 release on May 2 included C and Ruby bindings for the Arrow
C++ libraries. We have also seen a native JavaScript (TypeScript)
implementation appear for use.
- The TurbODBC C++ and Python project released version 2.0.0
which included support for converting ODBC data to Apache
Arrow. This was enabled by an internal C++ API to the Python
Arrow bindings, and will help provide a blueprint for future
thirdparty Python libraries that use Arrow.
- The Ray project for machine learning from the UC Berkeley RISELab contributed
a large software component, a shared memory object store ("Plasma"), to the
Apache Arrow project.
- We have made significant progress toward completing compatibility between the
Java and C++ implementations of the Arrow memory format. As soon as we
achieve reasonable completeness, we should consider leaping to Arrow 1.0.0 to
communicate to the rest of the open source world that Arrow is no longer as
much of a work-in-progress and ready for more widespread use.
- We have created the arrow-dist git repo to assist with cross-language and
cross-platform packaging.
- Apache Spark has merged its first Arrow integration, SPARK-13534
- The external GPU Open Analytics Initiative is using Apache Arrow as its data
interchange format
## Health report:
- Arrow is seeing an uptick in community interest and adoption. The increase in
activity reflects the project's scope expanding (i.e. more programming
languages) and increase in use in other projects. We expect this trend to
continue as Arrow's perception changes to be deemed more production-ready and
stable.
## PMC changes:
- Currently 19 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Uwe Korn on Wed Apr 12 2017
## Committer base changes:
- Currently 22 committers.
- Kouhei Sutou was added as a committer on Wed May 10 2017
## Releases:
- 0.3.0 was released on Thu May 04 2017
- 0.4.0 was released on Mon May 22 2017
- 0.4.1 was released on Thu Jun 08 2017
## Mailing list activity:
- We changed our JIRA notification schema to send only issue *creation*
e-mails to the primary mailing list, with further comments and edits going
to issues@
- dev@arrow.apache.org:
- 547 subscribers (up 17 in the last 3 months):
- 622 emails sent to list (1098 in previous quarter)
- issues@arrow.apache.org:
- 11 subscribers (up 0 in the last 3 months):
- 1985 emails sent to list (1255 in previous quarter)
- reviews@arrow.apache.org:
- 9 subscribers (up 9 in the last 3 months)
## JIRA activity:
- 395 JIRA tickets created in the last 3 months
- 333 JIRA tickets closed/resolved in the last 3 months
@Phil: follow up to ensure that PMC knows that IP clearance is required
## Description:
Arrow is a columnar in-memory analytics layer designed to accelerate big data.
It houses a set of canonical in-memory representations of flat and
hierarchical data along with multiple language-bindings for structure
manipulation. It also provides IPC and common algorithm implementations.
## Issues:
- There are no issues requiring board attention at this time.
## Activity:
- CodeBase/Format:
- Substantial progress and a 0.2 release since last report, close to 0.3
- Example additions include: Large contribution of GLIB from support from new
contributor adding support for Ruby, Lua, Go, Enhancements to HDFS support
including partitioned directories, clarification & improvements to Time
types, Tensor Flow compatibility, support for fixed with binary types,
Python read enhancements, Incorporation of Feather file format, and many
other items.
- Spark integration (https://s.apache.org/arrowspark) looks promising and
will hopefully expose Arrow to a large group of additional users.
- Awareness and evangelism:
- Talks at conferences and meetups including:
Spark Summit East https://s.apache.org/arrowss17
Strata San Jose https://s.apache.org/arrowstrata17
Dataworks Munich https://s.apache.org/arrowdataworks17
- Community:
- Continued influx of new contributors. Some PMC members have been
especially effective at engaging new communities, through discussions on
Twitter as well as other means.
## Health report:
- Double the number of dev and issue emails over the previous quarter mean
that the people who are active in the community are very active.
- At the same time, the PMC just started a discussion about how to continue to
grow the team. There have been various casual contributions which is good but
the core group of prolific contributors is growing slowly.
- We need to continue to make a concerted effort to provide example use cases
to help more people understand and appreciate Arrow benefits.
- We're seeing demand for this type of solution by other groups of people,
some inside the foundation, some outside. We're doing community outreach to
try to engage others but always worry about NIH thinking. Our open and
collaborative approach to building and extending the Arrow format and
software will hopefully convince more people to join the project rather than
creating competing technologies. Only time will tell in each case.
## PMC changes:
- Currently 19 PMC members.
- Last PMC addition was Uwe Korn on Thu Apr 13 2017
## Committer base changes:
- Currently 21 committers.
- No new committers added in the last 3 months
- Last committer addition was Uwe Korn at Thu Oct 27 2016
## Releases:
- 0.2.0 was released on Sat Feb 18 2017
## JIRA activity:
- 332 JIRA tickets created in the last 3 months
- 282 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Arrow has made great progress since the last report. - The community has actively been driving towards a set of cross-language compatibility tests. These are now complete. - The compatibility tests were a key gate identified to seeing the specification as solidified. Now that it is, the community will be starting work on our second release. - This release will show the arrow projects java arrow, arrow-cpp, py-arrow and Parquet's parquet-cpp all working nicely together. ## Health report: - A core group of community members continue to make good progress on various aspects of both the Java, C++ and python projects. - We're seen a small number of casual contributors arrive and provide additional patches to the project. - Multiple people have been doing community outreach through the various blog posts, meetups and conference presentations. Examples include - Upcoming talk at Strata San Jose in March - Upcoming talk at Dataworks Summit Munich in April - Arrow and Pandas vision: https://s.apache.org/arrow_1701_01 - Python Data Wrangling talk: https://s.apache.org/arrow_1701_02 - We continue to see nice growth in mailing list and jira activity. ## PMC changes: - Currently 18 PMC members. - Wes McKinney was added to the PMC on Wed Oct 19 2016 ## Committer base changes: - Currently 21 committers. - Uwe Korn was added as a committer on Thu Oct 27 2016 ## Releases: - Last release was 0.1.0 on Tue Oct 11 2016 ## JIRA activity: - 140 JIRA tickets created in the last 3 months - 117 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Arrow made its first release. - In preparation of the release, multiple discussions were focused on formalizing various Arrow specification details. - Discussion was good and we reworked some integration to invert the dependency model between the Parquet project and the Arrow project. - A new Arrow file format was defined and implemented in both Java and C++. It is also available from Python. - Community members covered Arrow at multiple conferences including Strata NYC. - Arrow <> Parquet interchange has been made available in C++. - The new Arrow file format is planned to be used to move forward on both cross-language IPC implementations and enabling cross-language compatibility tests. - We've seen good growth in the Arrow developer mailing list, having increased to 467 subscribers (up 43 in the last 3 months): ## Health report: - The first release is a good step in engaging a broader range of contributors and users. Having bits for use, albeit alpha, allows us to engage a wider range of engineers. - We need to continue to add new examples and more documentation to better describe how to use and extend Arrow. ## PMC changes: - Currently 17 PMC members. - No new PMC members added in the last 3 months - Last PMC addition was Abdel Hakim Deneche on Tue Jan 19 2016 ## Committer base changes: - Currently 20 committers. - No new committers added in the last 3 months - Last committer addition was Ippokratis Pandis at Thu Feb 18 2016 ## Releases: - 0.1.0 was released on Wed Oct 12 2016 ## JIRA activity: - 95 JIRA tickets created in the last 3 months - 73 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Awareness continues to increase with the community having done presentations at various meetups as well as the following conferences: Pydata Paris, Hadoop Summit Ireland, Hadoop Summit San Jose and Berlin Buzzwords. - The CPP work has made good progress. - The cross-project work with Parquet has seen substantial work (both in the Parquet project and the Arrow project). This should be a great first example proof-of-concept integration showing the benefits of in-memory columnar layer. - There has been substantial progress on development of for the IPC / memory sharing. - Java development has slowed some but appears to be picking up again. - A new independent project called Feather is using Arrow as a format for writing to disk. This has also increased engagement with Arrow itself and we have a number excited communities including R & Python (and the Julia community experimenting). ## Health report: - We've seen good discussion and development activity since the last report. - We need to get to a first release. - Prior to doing so, the community is working on rudimentary integration tests between Java and C++ and more formal format specification. - More work can be done to make the project approachable to newly interested parties by creating additional documentation and quickstart. A sample application will also help. ## PMC changes: - Currently 17 PMC members. - No new PMC members added in the last 3 months. - Last PMC addition was Abdel Hakim Deneche on Tue Jan 20 2016 ## Committer base changes: - Currently 20 committers. - No new committers added in the last 3 months - Last committer addition was Ippokratis Pandis at Thu Feb 18 2016 ## Releases: - No releases yet. ## JIRA activity: - 71 JIRA tickets created in the last 3 months - 40 JIRA tickets closed/resolved in the last 3 months
@Marvin: Links to unreleased source code must be removed from the Arrow home page.
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - A number of public presentations have been done about Arrow including at Strata SJ, Hadoop Summit Europe and various meetups. Response at each was strong and we saw subsequent increased interaction on the mailing list. - We've seen a great new project Feather, a collaboration between the Python and R communities build on top of Arrow to provide an ephemeral cross system format that performs faster and has better typing than the traditionally used CSV format. - A number of interested organizations have posted blogs about their interest in and support for Arrow. ## Health report: - We continue to see new community members engage. - Public discussions and contributions from both committers and casual contributors continue to improve the Arrow specification. - We're working with the incubating Mnemonic community to support alternatives to ephemeral memory for storing Arrow vectors. This will likely first appear as an optional extension module of the Java api. - We're still negotiating the final Arrow switch over in the Drill community. The goal is to do it as part of the 2.0 branch to avoid any disruption to the active stable branch (master). ## PMC changes: - Currently 17 PMC members. - No new PMC members added in the last 3 months ## Committer base changes: - Currently 20 committers. - Most recently added committers: - Ippokratis Pandis was added as a committer on Thu Feb 18 2016 - David Alves was added as a committer on Wed Feb 17 2016 - Wes McKinney was added as a committer on Mon Feb 01 2016 ## Releases: - No releases yet. ## JIRA activity: - 100 JIRA tickets created in the last 3 months - 67 JIRA tickets closed/resolved in the last 3 months
## Description: - Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - A number of articles were posted about Arrow in the last month. This has brought in a large number of interested parties and we've seen nice increase in community engagement. - There has been solid ideation and design discussions around IPC, metadata and shared memory semantics. - Development of the Python bindings is underway with a number of JIRAs focused on that component. - Since people have become aware of the project, we've seen a nice increase in activity on the mailing list. Within that last month we've gone from 17 subscribers to 263 on the dev list and have seen 136 messages on the list. ## Health report: - JIRAs are being opened and closed at a solid rate given the freshness of the project. - A number of design discussions have included great feedback and engagement people outside the initial PMC/committers. - We've seen several code contributions from first-time contributors. - The final separation of code from the Drill codebase is up for review and will likely move forward after the 1.6 Drill release (voting nearly underway). ## PMC changes: - Currently 17 PMC members. - No new PMC members since project was established. ## Committer base changes: - Currently 20 committers. - New committers: - Ippokratis Pandis was added as a committer on Thu Feb 18 2016 - Wes McKinney was added as a committer on Mon Feb 01 2016 ## Releases: - No releases yet. ## JIRA activity: - 41 JIRA tickets created in the last 3 months - 20 JIRA tickets closed/resolved in the last 3 months
## Description: - Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - there are no issues requiring board attention at this time ## Activity: - The project was established at the last board meeting. - Mailing lists, repositories and issue tracking have been established by infrastructure. - An initial website is underway and should be available by the time of the board meeting. - The community also has worked with Sally from press@ to announce the project via a press release on February 17th. - Various community members are working on putting together better documentation and communication around Arrow. ## Health report: - The project is just getting started as an independent project. - One of the key initial efforts is finalizing the extraction of code from the Drill codebase. This is tracking well and we hope to complete this before the next board report. - A number of community talks are being submitted to upcoming conferences and meetups to make more people aware of Arrow. - We need to start spending more time growing the community beyond the initial PMC and committers. - The Project voted to add 5 additional committers to the core project due to their involvement initial Arrow discussions. ## PMC changes: - Currently 17 PMC members. - No new PMC members added since the project was established. ## Committer base changes: - Currently 18 committers. - Wes McKinney was added as a committer on Mon Feb 01 2016 - Offers are out to four other community members to become committers. ## Releases: - No Arrow releases have yet been made. ## Mailing list activity: - dev@arrow.apache.org: - 17 subscribers (up 17 in the last 3 months) - issues@arrow.apache.org: - 4 subscribers (up 4 in the last 3 months)
WHEREAS, the Board of Directors deems it to be in the best interests of the Foundation and consistent with the Foundation's purpose to establish a Project Management Committee charged with the creation and maintenance of open-source software, for distribution at no charge to the public, related to columnar in-memory processing and data interchange NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to be known as the "Apache Arrow Project", be and hereby is established pursuant to Bylaws of the Foundation; and be it further RESOLVED, that the Apache Arrow Project be and hereby is responsible for the creation and maintenance of software related to columnar in-memory processing and data interchange; and be it further RESOLVED, that the office of "Vice President, Apache Arrow" be and hereby is created, the person holding such office to serve at the direction of the Board of Directors as the chair of the Apache Arrow Project, and to have primary responsibility for management of the projects within the scope of responsibility of the Apache Arrow Project; and be it further RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Arrow Project: * Todd Lipcon <todd@apache.org> * Ted Dunning <tdunning@apache.org> * Michael Stack <stack@apache.org> * P. Taylor Goetz <ptgoetz@apache.org> * Reynold Xin <rxin@apache.org> * Julian Hyde <jhyde@apache.org> * Julien Le Dem <julien@apache.org> * Jacques Nadeau <jacques@apache.org> * James Taylor <jamestaylor@apache.org> * Jake Luciani <jake@apache.org> * Parth Chandra <parthc@apache.org> * Alex Levenson <alexlevenson@apache.org> * Marcel Kornacker <marcel@apache.org> * Steven Phillips <smp@apache.org> * Hanifi Gunes <hg@apache.org> * Jason Altekruse <json@apache.org> * Abdel Hakim Deneche <adeneche@apache.org> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Jacques Nadeau be appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. RESOLVED, that the Apache Arrow Project be and hereby is tasked with the migration and rationalization of the Apache Drill Arrow sub-project; and be it further RESOLVED, that all responsibilities pertaining to the Apache Drill Arrow sub-project encumbered upon the Apache Drill Project are hereafter discharged. Special Order 7A, Establish the Apache Arrow Project, was approved by Unanimous Vote of the directors present.