This was extracted (@ 2023-02-28 12:10) from a list of minutes
which have been approved by the Board.
Please Note
The Board typically approves the minutes of the previous meeting at the
beginning of every Board meeting; therefore, the list below does not
normally contain details from the minutes of the most recent Board meeting.
WARNING: these pages may omit some original contents of the minutes.
Meeting times vary, the exact schedule is available to ASF Members and Officers, search for "calendar" in the Foundation's private index page (svn:foundation/private-index.html).
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: Lack of ASF sponsored invite-free chat service is a minor source of friction for community building. Most subprojects now use github for tickets to lower the barrier to entry for new / casual contributors, but we still have fragmented stories for group chat. ASF Slack requires an invite and some sub communities use other chat-like services. ## Membership Data: Apache Arrow was founded 2016-01-20 (7 years ago) There are currently 89 committers and 45 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - Kun Liu was added to the PMC on 2022-11-13 - Jacob Quinn was added to the PMC on 2022-10-25 - Nicola Crane was added to the PMC on 2022-10-25 - Jacob Wujciak was added as committer on 2022-12-19 - Ben Baumgold was added as committer on 2022-10-26 - Bogumił Kamiński was added as committer on 2022-10-24 - Eric Hanson was added as committer on 2022-10-26 - Jie Wen was added as committer on 2023-01-08 - Jarrett Revels was added as committer on 2022-11-02 - Curtis Vogt was added as committer on 2022-11-02 - Raúl Cumplido was added as committer on 2022-12-05 - Will Jones was added as committer on 2022-10-28 - Yang Jiang was added as committer on 2022-11-02 ## Project Activity: * Switching from JIRA to github issues in order to keep the overhead for new contributors low (no need to register for an ASF JIRA account) * [ADBC] (Arrow Database Connectivity) first release: * Community voted to add RLE to the specification * Additional subproject updates are below * We continue to release several different products and releases per quarter [ADBC]: https://arrow.apache.org/blog/2023/01/05/introducing-arrow-adbc/ Recent releases: ADBC-0.1.0 was released on 2023-01-10. RS-30.0.1 was released on 2023-01-08. RS-OS-0.5.3 was released on 2023-01-08. RS-30.0.0 was released on 2023-01-03. RS-29.0.0 was released on 2022-12-12. RS-OS-0.5.2 was released on 2022-12-07. RS-DATAFUSION-15.0.0 was released on 2022-12-05. DATAFUSION-PYTHON-0.7.0 was released on 2022-11-29. RS-28.0.0 was released on 2022-11-28. 10.0.1 was released on 2022-11-22. RS-BALLISTA-0.10.0 was released on 2022-11-21. JULIA-2.4.1 was released on 2022-11-18. RS-27.0.0 was released on 2022-11-15. RS-DATAFUSION-14.0.0 was released on 2022-11-07. RS-26.0.0 was released on 2022-11-03. 10.0.0 was released on 2022-10-26. JULIA-2.4.0 was released on 2022-10-26. RS-BALLISTA-0.9.0 was released on 2022-10-26. RS-25.0.0 was released on 2022-10-17. ## Community Health: The community health appears good, discussions on the mailing lists and github are productive. We recently had a nice discussion on the State of the Project: https://lists.apache.org/thread/r8gl3wvjgy9k8n2t194r0bbdbxx6ksqc and discussed various ways to keep encouraging the community. ## Language Area Updates Arrow has at least 12 different language bindings, as explained in https://arrow.apache.org/overview/ Arrow 10.0.0 release: https://arrow.apache.org/blog/2022/10/31/10.0.0-release/ ### C++ ### C# ### Go We’re seeing significant increases in interest and usage of the Arrow Go library. From startups like Spice.AI to being incorporated and used in Google BigQuery’s quickstart example and more. 2022 was a big year of updates, fixes, and drumming up interest for the Go module that we hope to continue for increased adoption and usage. The Go module, along with C++, is used as the initial implementation for the Run-End Encoding array implementation. Future development plans are to continue to expand the compute capabilities of the Go module and extend integration with Substrait. ### Java ### JavaScript ### Julia We’ve worked again on simplifying and streamlining the administrative side for the Julia implementation; adding additional committers, simplifying the release process, etc. This has increased the rate of contributions, as expected. There’s interest in finishing the C data/stream interfaces for the Julia implementation soon. ### Rust Rust has several projects: arrow-rs (arrow, parquet, arrow-flight object_store implementations) arrow-datafusion: rust query engine arrow-ballista: distributed query engine We are working to incorporate substrait into DataFusion Working on external communication with several blog posts about technology on sorting Fast and Memory Efficient Multi-Column Sorts in Apache Arrow Rust, Part 1 and Querying Parquet with Millisecond Latency We also continue calendar based release train with good results. ### C (GLib) We’ve added support for 16-bit float type. ### MATLAB 1. We have been focusing development efforts on implementing an "object dispatch layer" that uses MEX to "connect" MATLAB objects with corresponding C++ objects. This code is being actively developed at github.com/mathworks/libmexclass. See the following Arrow mailing list discussion for more context. We hope to upstream the changes needed to make the MATLAB Interface to Arrow use libmexclass under the hood in the coming months. This should enable the MATLAB interface to wrap relevant Arrow C++ objects (e.g. arrow::Array, arrow::Table) and expose them to MATLAB. 2. We have been continuing to investigate Windows CI support for the MATLAB interface. Currently, only Linux and macOS are supported. ### Python ### R ### Ruby We’ve added support for 16-bit float type. There is a new contributor who develops a new data frame library based on the Ruby bindings. The new contributor upstreamed some improvements implemented in the downstream data frame library.
WHEREAS, the Board of Directors heretofore appointed Kouhei Sutou (kou) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Kouhei Sutou from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Andrew Lamb (alamb) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Kouhei Sutou is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Andrew Lamb be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7D, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Arrow was founded 2016-01-20 (7 years ago) There are currently 79 committers and 42 PMC members in this project. The Committer-to-PMC ratio is roughly 5:3. Community changes, past quarter: - Raphael Taylor-Davies was added to the PMC on 2022-09-20 - L. C. Hsieh was added to the PMC on 2022-09-03 - Weston Pace was added to the PMC on 2022-09-05 - Zhong Yanghong was added as committer on 2022-09-08 - Remzi Yang was added as committer on 2022-09-09 - Daniel Harris was added as committer on 2022-09-23 ## Project Activity: - RS-24.0.0 was released on 2022-10-03. - RS-BALLISTA-0.8.0 was released on 2022-09-20. - RS-23.0.0 was released on 2022-09-19. - RS-DF-12.0.0 was released on 2022-09-16. - RS-OS-0.5.0 was released on 2022-09-11. - RS-22.0.0 was released on 2022-09-05. - RS-21.0.0 was released on 2022-08-30. - RS-DF-11.0.0 was released on 2022-08-19. - RS-OS-0.4.0 was released on 2022-08-13. - RS-20.0.0 was released on 2022-08-09. - RS-19.0.0 was released on 2022-07-26. - 6.0.2 was released on 2022-07-19. - 7.0.1 was released on 2022-07-19. - 8.0.1 was released on 2022-07-19. - RS-DF-10.0.0 was released on 2022-07-15. - RS-18.0.0 was released on 2022-07-11. ## Community Health: - We're voting a new Apache Arrow Database Connectivity (ADBC) interface developing at https://github.com/apache/arrow-adbc . - We'll deprecate unmaintained Apache Arrow Plasma object store. - We've accepted Apache Arrow Flight SQL JDBC driver donation. - We've switched required C++ specification to C++17 from C++11. - We've changed issue assign policy on Jira: All users can assign themselves as assignee for easy to contribute. - We've accepted Apache Arrow Rust Object Store donation. - We've backed Apache Arrow DataFusion Python bindings to ASF.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Arrow was founded 2016-01-20 (6 years ago) There are currently 76 committers and 39 PMC members in this project. The Committer-to-PMC ratio is roughly 5:3. Community changes, past quarter: - No new PMC members. Last addition was QP Hou on 2022-02-17. - Alenka Frim was added as committer on 2022-06-23 - Dewey Dunnington was added as committer on 2022-06-15 - Rok Mihevc was added as committer on 2022-06-15 - L. C. Hsieh was added as committer on 2022-04-28 ## Project Activity: Recent releases: - RS-17.0.0 was released on 2022-06-28. - RS-16.0.0 was released on 2022-06-13. - RS-15.0.0 was released on 2022-05-30. - RS-14.0.0 was released on 2022-05-16. - 8.0.0 was released on 2022-05-07. - RS-13.0.0 was released on 2022-05-03. - RS-12.0.0 was released on 2022-04-19. - RS-DF-7.1.0 was released on 2022-04-18. - JULIA-2.3.0 was released on 2022-04-12. ## Community Health: - A book about Apache Arrow was published from a community member. https://www.packtpub.com/product/in-memory-analytics-with-apache-arrow/9781801071031 - We've split Apache Arrow Ballista to https://github.com/apache/arrow-ballista from https://github.com/apache/arrow-datafusion . - We've marked Apache Arrow C Streaming Interface stable. - A conference that focuses on Apache Arrow was held by community members. https://thedatathread.com/ - We're improving our release process to reduce release management cost. - We've accepted donation of Apache Arrow Flight SQL JDBC driver.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Arrow was founded 2016-01-20 (6 years ago) There are currently 72 committers and 39 PMC members in this project. The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - QP Hou was added to the PMC on 2022-02-17 - Kun Liu was added as committer on 2022-03-09 - Jacob Quinn was added as committer on 2022-02-23 - Raphael Taylor-Davies was added as committer on 2022-03-24 - Xudong Wang was added as committer on 2022-03-16 - Yijie Shen was added as committer on 2022-03-09 ## Project Activity: Recent releases: - RS-11.1.0 was released on 2022-04-04. - RS-11.0.0 was released on 2022-03-21. - JULIA-2.2.1 was released on 2022-03-08. - RS-10.0.0 was released on 2022-03-08. - RS-9.1.0 was released on 2022-02-22. - RS-DF-7.0.0 was released on 2022-02-17. - RS-9.0.2 was released on 2022-02-13. - 7.0.0 was released on 2022-02-03. - RS-8.0.0 was released on 2022-01-25. - RS-7.0.0 was released on 2022-01-12. ## Community Health: - We've released a new version of Apache Arrow Julia after it was re-donated. - We'll start IP clearance for Apache Arrow Flight SQL JDBC driver donation. - We may register Apache Arrow as a Digital Public Good like Apache Fineract: https://digitalpublicgoods.net/registry/apache-fineract.html - We'll rename the "master" branch to "main": https://issues.apache.org/jira/browse/ARROW-15689 We'll be asking INFRA for support on this when we're ready. - We've rotated the PMC chair.
WHEREAS, the Board of Directors heretofore appointed Wes McKinney (wesm) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Wes McKinney from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Kouhei Sutou (kou) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Wes McKinney is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Kouhei Sutou be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7E, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * https://github.com/apache/arrow-julia is blocked by Infra on CI: * https://issues.apache.org/jira/browse/INFRA-22676 isn’t triaged. * https://github.com/apache/arrow-julia wants to run CI before merging pull requests. ## Membership Data: Apache Arrow was founded 2016-01-19 (6 years ago) There are currently 67 committers and 38 PMC members in this project. The Committer-to-PMC ratio is roughly 9:5. Community changes, past quarter: - Daniël Heres was added to the PMC on 2021-12-20 - Joris Van den Bossche was added to the PMC on 2021-11-21 - Yibo Cai was added to the PMC on 2022-01-03 - Alessandro Molina was added as committer on 2022-01-04 - Rémi Dettai was added as committer on 2021-12-07 ## Project Activity: Recent releases: * RS-6.5.0 was released on 2021-12-29. * RS-6.4.0 was released on 2021-12-14. * RS-6.3.0 was released on 2021-11-29. * 6.0.1 was released on 2021-11-18. * RS-DF-6.0.0 was released on 2021-11-17. * RS-6.2.0 was released on 2021-11-15. * RS-6.1.0 was released on 2021-11-03. * 6.0.0 was released on 2021-10-26. * RS-6.0.0 was released on 2021-10-17. Planned releases: * 7.0.0 in mid-January ## Community Health: * Apache Arrow Julia was re-donated https://s.apache.org/arrow-jl-donation * We’ve created a repository for collaborative prototyping + algorithms / performance research https://github.com/apache/arrow-experiments * Discussions around adding new columnar memory layouts to Arrow (in-memory, IPC, C ABI) * Discussions around community maintained extension repos for Apache Arrow Datafusion https://github.com/datafusion-contrib * We’ll rotate the PMC chair in 2022-01 * We’ve started Apache Arrow Cookbook to accept more new users. We’ll create similar documentation for new contributors. https://arrow.apache.org/cookbook/ * Rust continues releases every other week and is trying to balance open group collaboration and fast iterations: https://github.com/apache/arrow-rs/issues/1120 * Rust Arrow2 project, https://github.com/jorgecarleitao/arrow2, implementation of Arrow developed outside ASF, continues to generate community questions about if/when it will be brought into ASF.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (6 years ago) There are currently 64 committers and 35 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - Neville Dipale was added to the PMC on 2021-07-28 - QP Hou was added as committer on 2021-07-26 - Nicola Crane was added as committer on 2021-09-08 - Weston Pace was added as committer on 2021-07-09 - Matthew Topol was added as committer on 2021-08-27 - Jiayu Liu was added as committer on on 2021-10-07 ## Project Activity: - Planned upcoming 6.0 release in mid-October - Rust migration to multiple repositories (arrow-rs, arrow-datafusion) has been successful and the community has been consistently making releases. ## Community Health: - We’ve re-engaged with maintainers of the Julia port and are working on implementing the same development process that Rust has adopted (separate apache Repo). IP clearance will need to be redone. - Discussions around using Github issues as a support mechanism in conjunction or to replace user@
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * Most issues from previous board reports have been addressed though we would benefit as discussed from more mentorship in the Apache Way for new contributors. * Long GitHub Actions queueing times for CI jobs have been an occasional concern. We are aware that we are one of the most greedy consumers of GHA minutes in the ASF, so we will continue to think about future strategies to supplement GitHub Actions with dedicated build infrastructure. Our CI/CD needs are some of the most complex that we've observed in mainstream open source software. ## Membership Data: Apache Arrow was founded 2016-01-19 (5 years ago) There are currently 61 committers and 34 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - Ben Kietzman was added to the PMC on 2021-05-20 - David Li was added to the PMC on 2021-06-20 - Daniël Heres was added as committer on 2021-04-26 - Dominik Moritz was added as committer on 2021-06-02 - Ian Cook was added as committer on 2021-04-27 - Jonathan Keane was added as committer on 2021-04-27 - Kazuaki Ishizaki was added as committer on 2021-06-06 - Weston Pace was added as committer on 2021-07-09 ## Project Activity: * The Rust projects have successfully migrated to independent arrow-* repositories and begun making biweekly releases on a schedule. This progress seems to be going reasonably smoothly and the prior issues have been adequately addressed. * We are progressing as usual toward making our ~quarterly major release, now 5.0.0. ## Community Health: * We've succeeded at significantly expanding the committership and PMC this quarter, and our community growth in general continues to be robust.
## Description:
The mission of Apache Arrow is the creation and maintenance of software related
to columnar in-memory processing and data interchange
## Issues:
* CI Issues continue to be a problem. With the increase in utilization of
ASF’s allotment of Github Actions we’ve seen CI build time go from 20-30
minutes to as long as 5 hours
* There have been several issues around understanding exact requirements for
what it means to follow the Apache way and developing code within the
community:
* Several Rust projects were developed outside of the ASF repos and then
had proposals to be donated back.
* After accepting the donation of Julia developers continued to develop in
the original Julia repo and only make PRs back to the Arrow repo on an
intermittent basis. Since the Arrow community has been growing rapidly,
we had not done a thorough enough job of educating newcomers what it
means to develop “inside the community”.
* It appears a development model similar to what Rust is proposing
(see community health) will mitigate technical concerns for why this
was the development model chosen.
* It is unclear if JIRA use is required / encouraged for Apache projects
or if using a non Apache hosted system such as GitHub issues for parts
of the project is sufficiently community friendly
* In both cases it seems exact requirements of the ASF are not clear.
Through discussion it became apparent that the preferred method for
revolutionary changes to the existing code bases is by creating separate
development branches for prototyping with updates back to the community.
* ASF’s Artifactory isn’t ready yet. We don’t know the current status. See
also https://lists.apache.org/thread.html/r9200fe
d3fa812f8c7de07a2500425f258db3231baa8e05f288175e4a%40%3Cbuilds.apache.org%3E
## Membership Data:
Apache Arrow was founded 2016-01-19 (5 years ago)
There are currently 55 committers and 32 PMC members in this project.
The Committer-to-PMC ratio is roughly 7:4.
Community changes, past quarter:
- Andrew Lamb was added to the PMC on 2021-03-07
- Jorge Leitão was added to the PMC on 2021-03-07
- Yibo Cai was added as committer on 2021-03-05
## Project Activity:
We are aiming to have a 4.0 release this month.
## Community Health:
* Rust members of the community have proposed to follow a new development
workflow using separate Arrow github repos specifically for Rust components
and migrating away from JIRA to Github for issue tracking. Rust will also
start releasing on a different cadence then the rest of the language
bindings.
* The releases will still follow Apache Arrow governance (voting for
releases, etc)
* Depending on the experience of the Rust community, the Julia community may
follow a similar path in the future
@Roy: Follow up with board issues
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * After our past issues around scarce CI resources, we had managed to stabilize our CI setup around the relatively generous execution facilities provided by Github Actions, and a bunch of additional builds on other platforms (mostly for nightly builds). That equilibrium was unfortunately shattered after INFRA suddenly disallowed most third-party Github Actions helpers in response to security concerns. Our remaining Github Actions builds also seem to be queued longer. This has now been resolved (https://issues.apache.org/jira/browse/INFRA-21239) but it was hugely disruptive to our community. Concurrently, Travis-CI also started to be much stricter with resource consumption, which is currently affecting several of our nightly builds until we manage to migrate them to other CI services. These factors compounded mean we'll finally need to provide and set up our own CI resources. We hope to be able to migrate a significant number of builds to Buildkite, using execution resources paid by our sponsors. We also hope that INFRA rules around repository access and configuration won't stifle that endeavour. ## Membership Data: Apache Arrow was founded 2016-01-19 (5 years ago) There are currently 54 committers and 30 PMC members in this project. The Committer-to-PMC ratio is 9:5. Community changes, past quarter: - No new PMC members. Last addition was Francois Saint-Jacques on 2020-03-04. - Andrew Lamb was added as committer on 2020-11-06 ## Project Activity: - We'd like to release Arrow 3.0 around mid-January. The issues discussed above (diminished CI coverage due to disallowed third-party Github Actions, blocked nightly builds on Travis-CI, complex release procedure) make it more difficult than we'd like. - There seems to be growing interest in zero-copy zero-dependency cross-runtime interoperability using our "C data interface". It is now concretely possible to share data in-process between R, Python and Rust, without any shared code and without going through IPC mechanisms (the latter still possible, of course, using the Arrow IPC format). Other projects such as DuckDB have started using this interface. - The Arrow in-memory format is regularly being adopted and/or adapted to in other projects, especially in the dataframe and database communities. ## Community Health: * The project is overall very active. The Rust implementation is especially thriving, with numerous pull requests being submitted and merged in due time. The C++ implementation is advancing at similar speed as in previous reports. A Julia implementation was recently contributed. * Our pull request backlog has also grown slightly as a result of increased activity, but it does not seem to be posing a real problem for now. * The weight of our release procedure puts a burden on our capacity to release often. The "weight" is a function of the project complexity but also limitations in our CI/CD resources and feedback cycles (e.g. builds that run only once a day overnight often fail and we end up blocked on releasing until those failures are resolved). Our goal is to cut a new feature release roughly once every two months, which often ends up challenging. This situation also makes us reluctant to put out bugfix releases in-between, except for critical fixes. Improved CI/CD along with improvements to streamline the release process itself (for example, addressing procedural issues with preparing the Java project to release) would help a great deal. We also need more PMC members to step up and help with release management. Most of the release management over the lifetime of the project has fallen on the shoulders of only 4 out of our 30 PMCs.
WHEREAS, the Board of Directors heretofore appointed Jacques Nadeau (jacques) to the office of Vice President, Apache Arrow, and WHEREAS, the Board of Directors is in receipt of the resignation of Jacques Nadeau from the office of Vice President, Apache Arrow, and WHEREAS, the Project Management Committee of the Apache Arrow project has chosen by vote to recommend Wes McKinney (wesm) as the successor to the post; NOW, THEREFORE, BE IT RESOLVED, that Jacques Nadeau is relieved and discharged from the duties and responsibilities of the office of Vice President, Apache Arrow, and BE IT FURTHER RESOLVED, that Wes McKinney be and hereby is appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. Special Order 7D, Change the Apache Arrow Project Chair, was approved by Unanimous Vote of the directors present.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues needing board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (5 years ago) There are currently 53 committers and 30 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Francois Saint-Jacques on 2020-03-04. - Jorge Leitão was added as committer on 2020-09-29 ## Project Activity: The project has good activity with between five and ten commits most days. Dev email list activity was down in August and September (~50% over earlier in the year) but some of that is likely due to all the energy leading up to the 1.0 release. We should continue to monitor the mail trends to see if there are any systemic changes or concerns. ## Community Health: Github actions have substantially relieved the CI pain within the project. Arrow continues to see a diverse set of contributions with the total number of code contributors approaching 500. The project decided to move to a rotating PMC chair, with the first rotation planned for this month.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. The project has some level of support for 11 different programming languages. ## Issues: - There are no issues requiring board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (4 years ago) There are currently 52 committers and 30 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Francois Saint-Jacques on 2020-03-04. - Liya Fan was added as committer on 2020-06-09 - Ji Liu was added as committer on 2020-06-09 ## Project Activity: - We made the 0.17.0 and 0.17.1 releases since the last board report. The community is readying a 1.0.0 release which will formally mark stability in the Arrow columnar format binary protocol and a move to semantic versioning of the Arrow libraries. - We moved new JIRA issue notifications off of the dev@ mailing list to issues@ and created a new jira@ mailing list to capture the full JIRA firehose. The reasoning is that this could help encourage more participating in mailing list discussions. - We made changes in the codebase to remove uses of potentially non-neutral language that has been changed in many other open source projects. ## Community Health: The project and contributor base continues to grow in size and scope. We now have over 500 unique contributors since the creation of the project.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange. The project has some level of support for 11 different programming languages. ## Issues: - We are continuing to work with INFRA on issues related to self-hosted CI machines integrated with our GitHub-based pull request workflows. There are two avenues we are exploring (and we may well use both of them), GitHub Actions Self-hosted and Buildkite. Per INFRA-19217 Buildkite has just been approved for the @apache GitHub organization and we have validated that we can successfully use this with the free Arrow organization that Buildkite has provided us. CI/CD is likely to require an ongoing significant investment of time, and we are doing the best we can try avoid overburdening ASF Infra with requests. ## Membership Data: Apache Arrow was founded 2016-01-19 (4 years ago) There are currently 50 committers and 30 PMC members in this project. The Committer-to-PMC ratio is 5:3. Community changes, past quarter: - Francois Saint-Jacques was added to the PMC on 2020-03-04 - Neal Richardson was added to the PMC on 2020-03-04 - No new committers. Last addition was Joris Van den Bossche on 2019-12-06. ## Project Activity: - 0.16.0 was released at the end of January. We are close to releasing 0.17.0, with a 1.0.0 release hopefully sometime in 2020. - Three months ago, Apache Arrow was accepted for continuous fuzzing in the OSS-Fuzz infrastructure. We have now finally stabilized the situation by fixing all detected issues in the Arrow C++ IPC implementation, and are actively fixing issues in the Arrow C++ Parquet reader. - We just adopted a "C Data Interface" for the project which will open many new opportunities for integrations with third party projects. ## Community Health: The project and contributor base continues to grow in size and scope. We now have over 400 unique contributors since the creation of the project.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention at this time. ## Membership Data: Apache Arrow was founded 2016-01-19 (4 years ago) There are currently 50 committers and 28 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Micah Kornfield on 2019-08-21. - Eric Erhardt was added as committer on 2019-10-18 - Joris Van den Bossche was added as committer on 2019-12-06 ## Project Activity: * We have completed our initial migration away from Travis CI for continuous integration and patch validation to use the new GitHub Actions (GHA) service. We are much happier with the compute resource allocation provided by GitHub but longer term we are concerned that the generous free allocation may not continue and would be interested to know what kinds of guarantees (if any) GitHub may make to the ASF regarding GHA. * We are not out of the woods on CI/CD as there are features of Apache Arrow that we cannot test in GitHub Actions. We are still considering options for running these optional test workloads as well as other kinds of periodic workloads like benchmarking * We hope to make a 1.0.0 release of the project in early 2020. We had thought that our next major release after 0.15.0 would be 1.0.0 but we have not yet completed some necessary work items that the community has agreed are essential to graduate to 1.0.0 Recent releases: 0.15.0 was released on 2019-10-05. 0.14.1 was released on 2019-07-21. 0.14.0 was released on 2019-07-04. ## Community Health: The developer community is healthy and continues to grow.
## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: * There are no issues requiring board attention at this time. ## Membership Data: * Apache Arrow was founded 2016-01-19 (4 years ago) * There are currently 48 committers and 28 PMC members in this project. * The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - Micah Kornfield was added to the PMC on 2019-08-21 - Sebastien Binet was added to the PMC on 2019-08-21 - Ben Kietzman was added as committer on 2019-09-07 - David Li was added as committer on 2019-08-30 - Kenta Murata was added as committer on 2019-09-05 - Neal Richardson was added as committer on 2019-09-05 - Praveen Kumar was added as committer on 2019-07-14 ## Project Activity: * The project has just made a 0.15.0 release. * We are discussing ways to make the Arrow libraries as accessible as possible to downstream projects for minimal use cases while allowing the development of more comprehensive "standard libraries" with larger dependency stacks in the project * We plan to make a 1.0.0 release as our next major release, at which time we will declare that the Arrow binary protocol is stable with forward and backward compatibility guarantees ## Community Health: * The community is continuing to grow at a great rate. We see good adoption among many other projects and fast growth of key metrics. * Many contributors are struggling with the slowness of pre-commit CI. Arrow has a large number of different platforms and components and a complex build matrix. As new commits come in, they frequently take a long time to complete. The community is trying several ways to solve this. There is bubbling frustration in the community around the GitHub repo rules for using third party services. This is especially challenging when there are free solutions to relieve the community pressure but the community is unable to access these resources. This frustration is greatest among people who work on many non-asf OSS projects which don't have such restrictive rules around GitHub. Some examples of ways the community has tried to resolve these have included: * Try to use CircleCI, rejected in INFRA-15964 * Try to use Azure Pipelines, rejected in INFRA-17030 * Try to resolves Issues with Travis CI capacity: INFRA-18533 & https://s.apache.org/ci-capacity (no resolution beyond "find donations") * The creation of new infrastructure design (in progress but a huge amount of thankless work) * While the community has seen great growth in contribution (more than 300 unique contributors at this point), the vast majority are casual contributors. The daily active committers (the workhorses of the project that bear the load committing the constant PRs, more than 5000 closed at this point) have been growing slower than adoption. This is despite the fact that the community has been very aggressive at being inclusive of new committers (with likelihood to have more than 50 in the next week). The community is still continuing to try to brainstorm ways to improve this.
@David: close the loop with Arrow and other communities with needs for CI service improvements
## Description:
Apache Arrow is a cross-language development platform for in-memory
data. It specifies a standardized language-independent columnar memory
format for flat and hierarchical data, organized for efficient
analytic operations on modern hardware. It also provides computational
libraries and zero-copy streaming messaging and interprocess
communication. Languages currently supported include C, C++, C#, Go,
Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The community is discussing a 1.0.0 release featuring
forward-looking binary format stability guarantees. Given the
nature of the project, this is obviously an important milestone
for adoption and user support
- Since the last report, a new Buildbot-based CI system has been
connected to apache/arrow to provide additional build capacity, with
a bot system called "ursabot" to provide on demand builds, benchmark
comparisons, and other tools to assist the developer community
## Health report:
- We have been having significant problems with CI build times and are
discussing strategies to decouple our de velopment process from the
shared pool of ASF-managed cloud CI resources like Travis CI and
Appveyor
- The community is healthy, though there were some concerns
around the 0.14.0 release vote and we are discussing
conventions around handling issues raised during release
candidate vetting.
## PMC changes:
- Currently 26 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Andrew Grove on Sun Feb 03 2019
## Committer base changes:
- Currently 43 committers.
- New committers:
- Francois Saint-Jacques was added as a committer on Wed Jun 12 2019
- Neville Dipale was added as a committer on Mon May 13 2019
- Praveen Kumar has also been invited to be a committer and accepted.
Acct request is in process.
## Releases:
- 0.14.0 was released on Wed Jul 03 2019
## JIRA activity:
- 735 JIRA tickets created in the last 3 months
- 690 JIRA tickets closed/resolved in the last 3 months
@Rich: Follow up about release vote concerns
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported
include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The project received a donation of DataFusion, a Rust-based query
engine for Apache Arrow
## Health report:
- The project is very healthy, with a growing number and diversity of
contributors
## PMC changes:
- Currently 26 PMC members.
- Andrew Grove was added to the PMC on Sun Feb 03 2019
## Committer base changes:
- Currently 41 committers.
- New commmitters:
- Micah Kornfield was added as a committer on Fri Mar 08 2019
- Deepak Majeti was added as a committer on Thu Jan 31 2019
- Paddy Horan was added as a committer on Fri Feb 08 2019
- Ravindra Pindikura was added as a committer on Fri Feb 01 2019
- Sun Chao was added as a committer on Fri Feb 22 2019
## Releases:
- 0.12.0 was released on Sat Jan 26 2019
- 0.12.1 was released on Sun Feb 24 2019
- 0.13.0 was released on Sun Mar 31 2019
- JS-0.4.0 was released on Tue Feb 05 2019
- JS-0.4.1 was released on Sat Mar 23 2019
## JIRA activity:
- 969 JIRA tickets created in the last 3 months
- 861 JIRA tickets closed/resolved in the last 3 months
## Description:
- Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported
include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The project received 3 code donations in the last quarter: a native C#
library, a Parquet integration library for Ruby, and a native Parquet
implementation in Rust
- The community is considering a donation of DataFusion an in-memory query
engine built on top of the Rust Arrow library
- Community is actively working on 0.12 release
- Community has discussed starting to do time-based releases approximately
every two months
## Health report:
- The project is very healthy, with a growing number and diversity of
contributors
## PMC changes:
- Currently 25 PMC members.
- Krisztian Szucs was added to the PMC on Wed Nov 07 2018
## Committer base changes:
- Currently 36 committers.
- New commmitters:
- Romain Francois was added as a committer on Fri Nov 09 2018
- Sebastien Binet was added as a committer on Thu Nov 08 2018
- Yosuke Shiro was added as a committer on Thu Nov 08 2018
## Releases:
- 0.11.1 was released on Mon Oct 22 2018
## JIRA activity:
- 712 JIRA tickets created in the last 3 months
- 601 JIRA tickets closed/resolved in the last 3 months
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported include
C, C++, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
## Issues:
- There are no issues requiring board attention at this time
## Activity:
- The Arrow and Parquet communities resolved by vote to merge their respective
C++ codebases in the Apache Arrow repository. This work was completed this
quarter
- The project received two code donations via IP clearance: a GLib interface to
the Parquet C++ libraries, and the Gandiva LLVM vectorized Arrow expression
compiler
- Work has commenced on R language integration with the C++ libraries
- An initial MATLAB binding to the C++ libraries was contributed
- The community is discussing receiving a proposed native implementation of
Arrow in C# .NET
## Health report:
- The project is very healthy, though rapid user and contributor growth has
stressed the limits of our developer tooling and put a great deal of burden
on the active project maintainers
## PMC changes:
- Currently 24 PMC members.
- Antoine Pitrou was added to the PMC on Mon Aug 20 2018
## Committer base changes:
- Currently 33 committers.
- New commmitters:
- Andrew Grove was added as a committer on Tue Aug 07 2018
- Krisztian Szucs was added as a committer on Thu Aug 16 2018
## Releases:
- 0.10.0 was released on Sun Aug 05 2018
- 0.11.0 was released on Sun Oct 07 2018
## JIRA activity:
- 649 JIRA tickets created in the last 3 months
- 476 JIRA tickets closed/resolved in the last 3 months
## Description: Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, Go, Java, JavaScript, Python, Ruby, and Rust. ## Issues: - There are no issues requiring board attention at this time ## Activity: - We have not released since March as we work to improve our release and build automation. We plan to include binary artifacts in our next release vote, where we have only had source artifacts in past releases. - We received a code donation of a Ruby interface via the usual IP Clearance process - There is a new Arrow + LLVM analytics project (Gandiva) happening outside of the Arrow community. We discussed incorporating this work into Apache Arrow but have not made any decisions yet. ## Health report: The project's user and contributor base is growing rapidly. We are struggling a bit with maintainer bandwidth. As an example, 2 committers have merged 84% of patches (where there have been nearly 2000) since the project's inception. We are discussing ways to grow the maintainer base on the mailing list. ## PMC changes: - Currently 23 PMC members. - Siddharth Teotia was added to the PMC on Thu May 17 2018 ## Committer base changes: - Currently 31 committers. - No new committers added in the last 3 months - Last committer addition was Antoine Pitrou at Tue Apr 03 2018 ## Releases: - Last release was 0.9.0 on Mon Mar 19 2018 ## JIRA activity: - 392 JIRA tickets created in the last 3 months - 303 JIRA tickets closed/resolved in the last 3 months
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported include
C, C++, Go, Java, JavaScript, Python, Ruby, and Rust.
## Issues:
There are no issues requiring board attention at this time
## Activity:
- We received a donation of a Go implementation since the last report
- New contributors have started developing native Rust bindings. Given its
early development, community will likely start by doing regular Rust-only
releases at a higher velocity than primary releases (similar to JS)
- Given the number of different artifacts developed, the build and release
complexity has gotten too difficult (too many manual steps). The
community is discussing ways to improve the process to simplify development.
## Health report:
Project is very healthy with a growing developer and user community.
Overall development activity has increased since the prior quarter
with the addition of two new programming languages (Go, Rust) to the
project.
## PMC changes:
- Currently 22 PMC members.
- New PMC members:
- Phillip Cloud was added to the PMC on Thu Mar 15 2018
- Philipp Moritz was added to the PMC on Thu Mar 15 2018
## Committer base changes:
- Currently 31 committers.
- New commmitters:
- Antoine Pitrou was added as a committer on Tue Apr 03 2018
- Brian Hulette was added as a committer on Mon Feb 12 2018
- Robert Nishihara was added as a committer on Mon Feb 19 2018
## Releases:
- 0.9.0 was released on Mon Mar 19 2018
- JS-0.3.0 was released on Wed Feb 21 2018
- JS-0.3.1 was released on Fri Mar 16 2018
## JIRA activity:
- 464 JIRA tickets created in the last 3 months
- 358 JIRA tickets closed/resolved in the last 3 months
## Description:
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication. Languages currently supported include
C, C++, Java, JavaScript, Python, and Ruby.
## Issues:
There are no issues requiring board attention at this time
## Activity:
- Steady development activity from previous quarter and continued growth in
contributor base
- Added 5 new committers
- First JavaScript-only release (0.2.0) made on December 1
## Health report:
Project is very healthy with a growing developer and user community.
## PMC changes:
- Currently 20 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Kouhei Sutou on Fri Sep 15 2017
## Committer base changes:
- Currently 28 committers.
- New commmitters:
- Phillip Cloud was added as a committer on Tue Oct 03 2017
- Bryan Cutler was added as a committer on Wed Oct 04 2017
- Li Jin was added as a committer on Fri Oct 06 2017
- Paul Taylor was added as a committer on Fri Oct 06 2017
- Siddharth Teotia was added as a committer on Wed Oct 04 2017
## Releases:
- 0.8.0 was released on Sat Dec 16 2017
- JS-0.2.0 was released on Fri Dec 01 2017
## JIRA activity:
- 323 JIRA tickets created in the last 3 months
- 300 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. Arrow contains implementations or bindings in various stages in Java, C++, C, Ruby, Python, and JavaScript. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - We have added 6 new committers to the project - Continued development activity and growing community since the last board report. We have made 4 releases, with the next release 0.8.0 coming in a month or so - Development activity is increasing in JavaScript, and we are also looking to grow a community of R language developers in the coming quarters. ## Health report: - Arrow activity continues to increase with community interest and adoption. ## PMC changes: - Currently 20 PMC members. - Kouhei Sutou was added to the PMC on Fri Sep 15 2017 ## Committer base changes: - Currently 28 committers. - New commmitters: - Phillip Cloud was added as a committer on Tue Oct 03 2017 - Bryan Cutler was added as a committer on Wed Oct 04 2017 - Li Jin was added as a committer on Fri Oct 06 2017 - Philipp Moritz was added as a committer on Mon Aug 14 2017 - Paul Taylor was added as a committer on Fri Oct 06 2017 - Siddharth Teotia was added as a committer on Wed Oct 04 2017 ## Releases: - 0.5.0 was released on Sun Jul 23 2017 - 0.6.0 was released on Mon Aug 14 2017 - 0.7.0 was released on Sat Sep 16 2017 - 0.7.1 was released on Sun Oct 01 2017 ## JIRA activity: - 460 JIRA tickets created in the last 3 months - 382 JIRA tickets closed/resolved in the last 3 months
## Description:
Arrow is a columnar in-memory analytics layer designed to accelerate big data.
It houses a set of canonical in-memory representations of flat and hierarchical
data along with multiple language-bindings for structure manipulation. It also
provides IPC and common algorithm implementations.
## Issues:
- There are no issues requiring board attention at this time.
## Activity:
- Heavy development activity and growing community since the last board
report. We have made 3 releases, with the next release 0.5.0 coming soon.
- The Arrow 0.3.0 release on May 2 included C and Ruby bindings for the Arrow
C++ libraries. We have also seen a native JavaScript (TypeScript)
implementation appear for use.
- The TurbODBC C++ and Python project released version 2.0.0
which included support for converting ODBC data to Apache
Arrow. This was enabled by an internal C++ API to the Python
Arrow bindings, and will help provide a blueprint for future
thirdparty Python libraries that use Arrow.
- The Ray project for machine learning from the UC Berkeley RISELab contributed
a large software component, a shared memory object store ("Plasma"), to the
Apache Arrow project.
- We have made significant progress toward completing compatibility between the
Java and C++ implementations of the Arrow memory format. As soon as we
achieve reasonable completeness, we should consider leaping to Arrow 1.0.0 to
communicate to the rest of the open source world that Arrow is no longer as
much of a work-in-progress and ready for more widespread use.
- We have created the arrow-dist git repo to assist with cross-language and
cross-platform packaging.
- Apache Spark has merged its first Arrow integration, SPARK-13534
- The external GPU Open Analytics Initiative is using Apache Arrow as its data
interchange format
## Health report:
- Arrow is seeing an uptick in community interest and adoption. The increase in
activity reflects the project's scope expanding (i.e. more programming
languages) and increase in use in other projects. We expect this trend to
continue as Arrow's perception changes to be deemed more production-ready and
stable.
## PMC changes:
- Currently 19 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Uwe Korn on Wed Apr 12 2017
## Committer base changes:
- Currently 22 committers.
- Kouhei Sutou was added as a committer on Wed May 10 2017
## Releases:
- 0.3.0 was released on Thu May 04 2017
- 0.4.0 was released on Mon May 22 2017
- 0.4.1 was released on Thu Jun 08 2017
## Mailing list activity:
- We changed our JIRA notification schema to send only issue *creation*
e-mails to the primary mailing list, with further comments and edits going
to issues@
- dev@arrow.apache.org:
- 547 subscribers (up 17 in the last 3 months):
- 622 emails sent to list (1098 in previous quarter)
- issues@arrow.apache.org:
- 11 subscribers (up 0 in the last 3 months):
- 1985 emails sent to list (1255 in previous quarter)
- reviews@arrow.apache.org:
- 9 subscribers (up 9 in the last 3 months)
## JIRA activity:
- 395 JIRA tickets created in the last 3 months
- 333 JIRA tickets closed/resolved in the last 3 months
@Phil: follow up to ensure that PMC knows that IP clearance is required
## Description:
Arrow is a columnar in-memory analytics layer designed to accelerate big data.
It houses a set of canonical in-memory representations of flat and
hierarchical data along with multiple language-bindings for structure
manipulation. It also provides IPC and common algorithm implementations.
## Issues:
- There are no issues requiring board attention at this time.
## Activity:
- CodeBase/Format:
- Substantial progress and a 0.2 release since last report, close to 0.3
- Example additions include: Large contribution of GLIB from support from new
contributor adding support for Ruby, Lua, Go, Enhancements to HDFS support
including partitioned directories, clarification & improvements to Time
types, Tensor Flow compatibility, support for fixed with binary types,
Python read enhancements, Incorporation of Feather file format, and many
other items.
- Spark integration (https://s.apache.org/arrowspark) looks promising and
will hopefully expose Arrow to a large group of additional users.
- Awareness and evangelism:
- Talks at conferences and meetups including:
Spark Summit East https://s.apache.org/arrowss17
Strata San Jose https://s.apache.org/arrowstrata17
Dataworks Munich https://s.apache.org/arrowdataworks17
- Community:
- Continued influx of new contributors. Some PMC members have been
especially effective at engaging new communities, through discussions on
Twitter as well as other means.
## Health report:
- Double the number of dev and issue emails over the previous quarter mean
that the people who are active in the community are very active.
- At the same time, the PMC just started a discussion about how to continue to
grow the team. There have been various casual contributions which is good but
the core group of prolific contributors is growing slowly.
- We need to continue to make a concerted effort to provide example use cases
to help more people understand and appreciate Arrow benefits.
- We're seeing demand for this type of solution by other groups of people,
some inside the foundation, some outside. We're doing community outreach to
try to engage others but always worry about NIH thinking. Our open and
collaborative approach to building and extending the Arrow format and
software will hopefully convince more people to join the project rather than
creating competing technologies. Only time will tell in each case.
## PMC changes:
- Currently 19 PMC members.
- Last PMC addition was Uwe Korn on Thu Apr 13 2017
## Committer base changes:
- Currently 21 committers.
- No new committers added in the last 3 months
- Last committer addition was Uwe Korn at Thu Oct 27 2016
## Releases:
- 0.2.0 was released on Sat Feb 18 2017
## JIRA activity:
- 332 JIRA tickets created in the last 3 months
- 282 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Arrow has made great progress since the last report. - The community has actively been driving towards a set of cross-language compatibility tests. These are now complete. - The compatibility tests were a key gate identified to seeing the specification as solidified. Now that it is, the community will be starting work on our second release. - This release will show the arrow projects java arrow, arrow-cpp, py-arrow and Parquet's parquet-cpp all working nicely together. ## Health report: - A core group of community members continue to make good progress on various aspects of both the Java, C++ and python projects. - We're seen a small number of casual contributors arrive and provide additional patches to the project. - Multiple people have been doing community outreach through the various blog posts, meetups and conference presentations. Examples include - Upcoming talk at Strata San Jose in March - Upcoming talk at Dataworks Summit Munich in April - Arrow and Pandas vision: https://s.apache.org/arrow_1701_01 - Python Data Wrangling talk: https://s.apache.org/arrow_1701_02 - We continue to see nice growth in mailing list and jira activity. ## PMC changes: - Currently 18 PMC members. - Wes McKinney was added to the PMC on Wed Oct 19 2016 ## Committer base changes: - Currently 21 committers. - Uwe Korn was added as a committer on Thu Oct 27 2016 ## Releases: - Last release was 0.1.0 on Tue Oct 11 2016 ## JIRA activity: - 140 JIRA tickets created in the last 3 months - 117 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Arrow made its first release. - In preparation of the release, multiple discussions were focused on formalizing various Arrow specification details. - Discussion was good and we reworked some integration to invert the dependency model between the Parquet project and the Arrow project. - A new Arrow file format was defined and implemented in both Java and C++. It is also available from Python. - Community members covered Arrow at multiple conferences including Strata NYC. - Arrow <> Parquet interchange has been made available in C++. - The new Arrow file format is planned to be used to move forward on both cross-language IPC implementations and enabling cross-language compatibility tests. - We've seen good growth in the Arrow developer mailing list, having increased to 467 subscribers (up 43 in the last 3 months): ## Health report: - The first release is a good step in engaging a broader range of contributors and users. Having bits for use, albeit alpha, allows us to engage a wider range of engineers. - We need to continue to add new examples and more documentation to better describe how to use and extend Arrow. ## PMC changes: - Currently 17 PMC members. - No new PMC members added in the last 3 months - Last PMC addition was Abdel Hakim Deneche on Tue Jan 19 2016 ## Committer base changes: - Currently 20 committers. - No new committers added in the last 3 months - Last committer addition was Ippokratis Pandis at Thu Feb 18 2016 ## Releases: - 0.1.0 was released on Wed Oct 12 2016 ## JIRA activity: - 95 JIRA tickets created in the last 3 months - 73 JIRA tickets closed/resolved in the last 3 months
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Awareness continues to increase with the community having done presentations at various meetups as well as the following conferences: Pydata Paris, Hadoop Summit Ireland, Hadoop Summit San Jose and Berlin Buzzwords. - The CPP work has made good progress. - The cross-project work with Parquet has seen substantial work (both in the Parquet project and the Arrow project). This should be a great first example proof-of-concept integration showing the benefits of in-memory columnar layer. - There has been substantial progress on development of for the IPC / memory sharing. - Java development has slowed some but appears to be picking up again. - A new independent project called Feather is using Arrow as a format for writing to disk. This has also increased engagement with Arrow itself and we have a number excited communities including R & Python (and the Julia community experimenting). ## Health report: - We've seen good discussion and development activity since the last report. - We need to get to a first release. - Prior to doing so, the community is working on rudimentary integration tests between Java and C++ and more formal format specification. - More work can be done to make the project approachable to newly interested parties by creating additional documentation and quickstart. A sample application will also help. ## PMC changes: - Currently 17 PMC members. - No new PMC members added in the last 3 months. - Last PMC addition was Abdel Hakim Deneche on Tue Jan 20 2016 ## Committer base changes: - Currently 20 committers. - No new committers added in the last 3 months - Last committer addition was Ippokratis Pandis at Thu Feb 18 2016 ## Releases: - No releases yet. ## JIRA activity: - 71 JIRA tickets created in the last 3 months - 40 JIRA tickets closed/resolved in the last 3 months
@Marvin: Links to unreleased source code must be removed from the Arrow home page.
## Description: Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - A number of public presentations have been done about Arrow including at Strata SJ, Hadoop Summit Europe and various meetups. Response at each was strong and we saw subsequent increased interaction on the mailing list. - We've seen a great new project Feather, a collaboration between the Python and R communities build on top of Arrow to provide an ephemeral cross system format that performs faster and has better typing than the traditionally used CSV format. - A number of interested organizations have posted blogs about their interest in and support for Arrow. ## Health report: - We continue to see new community members engage. - Public discussions and contributions from both committers and casual contributors continue to improve the Arrow specification. - We're working with the incubating Mnemonic community to support alternatives to ephemeral memory for storing Arrow vectors. This will likely first appear as an optional extension module of the Java api. - We're still negotiating the final Arrow switch over in the Drill community. The goal is to do it as part of the 2.0 branch to avoid any disruption to the active stable branch (master). ## PMC changes: - Currently 17 PMC members. - No new PMC members added in the last 3 months ## Committer base changes: - Currently 20 committers. - Most recently added committers: - Ippokratis Pandis was added as a committer on Thu Feb 18 2016 - David Alves was added as a committer on Wed Feb 17 2016 - Wes McKinney was added as a committer on Mon Feb 01 2016 ## Releases: - No releases yet. ## JIRA activity: - 100 JIRA tickets created in the last 3 months - 67 JIRA tickets closed/resolved in the last 3 months
## Description: - Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - A number of articles were posted about Arrow in the last month. This has brought in a large number of interested parties and we've seen nice increase in community engagement. - There has been solid ideation and design discussions around IPC, metadata and shared memory semantics. - Development of the Python bindings is underway with a number of JIRAs focused on that component. - Since people have become aware of the project, we've seen a nice increase in activity on the mailing list. Within that last month we've gone from 17 subscribers to 263 on the dev list and have seen 136 messages on the list. ## Health report: - JIRAs are being opened and closed at a solid rate given the freshness of the project. - A number of design discussions have included great feedback and engagement people outside the initial PMC/committers. - We've seen several code contributions from first-time contributors. - The final separation of code from the Drill codebase is up for review and will likely move forward after the 1.6 Drill release (voting nearly underway). ## PMC changes: - Currently 17 PMC members. - No new PMC members since project was established. ## Committer base changes: - Currently 20 committers. - New committers: - Ippokratis Pandis was added as a committer on Thu Feb 18 2016 - Wes McKinney was added as a committer on Mon Feb 01 2016 ## Releases: - No releases yet. ## JIRA activity: - 41 JIRA tickets created in the last 3 months - 20 JIRA tickets closed/resolved in the last 3 months
## Description: - Arrow is a columnar in-memory analytics layer designed to accelerate big data. It houses a set of canonical in-memory representations of flat and hierarchical data along with multiple language-bindings for structure manipulation. It also provides IPC and common algorithm implementations. ## Issues: - there are no issues requiring board attention at this time ## Activity: - The project was established at the last board meeting. - Mailing lists, repositories and issue tracking have been established by infrastructure. - An initial website is underway and should be available by the time of the board meeting. - The community also has worked with Sally from press@ to announce the project via a press release on February 17th. - Various community members are working on putting together better documentation and communication around Arrow. ## Health report: - The project is just getting started as an independent project. - One of the key initial efforts is finalizing the extraction of code from the Drill codebase. This is tracking well and we hope to complete this before the next board report. - A number of community talks are being submitted to upcoming conferences and meetups to make more people aware of Arrow. - We need to start spending more time growing the community beyond the initial PMC and committers. - The Project voted to add 5 additional committers to the core project due to their involvement initial Arrow discussions. ## PMC changes: - Currently 17 PMC members. - No new PMC members added since the project was established. ## Committer base changes: - Currently 18 committers. - Wes McKinney was added as a committer on Mon Feb 01 2016 - Offers are out to four other community members to become committers. ## Releases: - No Arrow releases have yet been made. ## Mailing list activity: - dev@arrow.apache.org: - 17 subscribers (up 17 in the last 3 months) - issues@arrow.apache.org: - 4 subscribers (up 4 in the last 3 months)
WHEREAS, the Board of Directors deems it to be in the best interests of the Foundation and consistent with the Foundation's purpose to establish a Project Management Committee charged with the creation and maintenance of open-source software, for distribution at no charge to the public, related to columnar in-memory processing and data interchange NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to be known as the "Apache Arrow Project", be and hereby is established pursuant to Bylaws of the Foundation; and be it further RESOLVED, that the Apache Arrow Project be and hereby is responsible for the creation and maintenance of software related to columnar in-memory processing and data interchange; and be it further RESOLVED, that the office of "Vice President, Apache Arrow" be and hereby is created, the person holding such office to serve at the direction of the Board of Directors as the chair of the Apache Arrow Project, and to have primary responsibility for management of the projects within the scope of responsibility of the Apache Arrow Project; and be it further RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Arrow Project: * Todd Lipcon <todd@apache.org> * Ted Dunning <tdunning@apache.org> * Michael Stack <stack@apache.org> * P. Taylor Goetz <ptgoetz@apache.org> * Reynold Xin <rxin@apache.org> * Julian Hyde <jhyde@apache.org> * Julien Le Dem <julien@apache.org> * Jacques Nadeau <jacques@apache.org> * James Taylor <jamestaylor@apache.org> * Jake Luciani <jake@apache.org> * Parth Chandra <parthc@apache.org> * Alex Levenson <alexlevenson@apache.org> * Marcel Kornacker <marcel@apache.org> * Steven Phillips <smp@apache.org> * Hanifi Gunes <hg@apache.org> * Jason Altekruse <json@apache.org> * Abdel Hakim Deneche <adeneche@apache.org> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Jacques Nadeau be appointed to the office of Vice President, Apache Arrow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed. RESOLVED, that the Apache Arrow Project be and hereby is tasked with the migration and rationalization of the Apache Drill Arrow sub-project; and be it further RESOLVED, that all responsibilities pertaining to the Apache Drill Arrow sub-project encumbered upon the Apache Drill Project are hereafter discharged. Special Order 7A, Establish the Apache Arrow Project, was approved by Unanimous Vote of the directors present.