This was extracted (@ 2025-02-19 17:10) from a list of minutes
which have been approved by the Board.
Please Note
The Board typically approves the minutes of the previous meeting at the
beginning of every Board meeting; therefore, the list below does not
normally contain details from the minutes of the most recent Board meeting.
WARNING: these pages may omit some original contents of the minutes.
Meeting times vary, the exact schedule is available to ASF Members and Officers, search for "calendar" in the Foundation's private index page (svn:foundation/private-index.html).
## Description:
The mission of Apache Airflow is the creation and maintenance of software
related to workflow automation and scheduling that can be used to author and
manage data pipelines
## Project Status:
Current project status: *Ongoing* with *high* activity
Issues for the board:
No issues for the board.
## Membership Data:
Apache Airflow was founded 2018-12-19 (6 years ago)
There are currently 68 committers and 35 PMC members in this project.
The Committer-to-PMC ratio is roughly 9:5.
Community changes, past quarter:
- Vikram Koka was added to the PMC on 2024-10-19
- Gopal Dirisala was added as committer on 2024-10-28
- Pavan Kumar was added as committer on 2024-10-29
- Karthikeyan Singaravelan was added as committer on 2024-10-28
## Project Activity:
- Development for Airflow 3 continues at a rapid pace, with several key
features and enhancements actively being worked on. The target release is
still set for March/April 2025. A good summary of the progress:
https://lists.apache.org/thread/r6hcyx8h023h42zz2xz94sz524v1z81w
- The community continues to maintain and release updates for the 2.x series
alongside the development of 3.0 to ensure stability and support for users
during the transition. Airflow 2.10.2 and 2.10.3, alongwith 7 batch releases
of providers were released in this period and
2.10.4 is being voted on currently. Full release activity:
- Provider packages 2024-11-24 was released on 2024-11-27.
- Provider packages 2024-11-14 was released on 2024-11-18.
- Apache Airflow 2.10.3 was released on 2024-11-05.
- Provider packages 2024-11-03 was released on 2024-11-05.
- Provider packages 2024-10-27 was released on 2024-10-31.
- Provider packages 2024-10-10 was released on 2024-10-14.
- Provider packages 2024-09-27 was released on 2024-10-01.
- Provider packages 2024-09-21 was released on 2024-09-24.
- Apache Airflow 2.10.2 was released on 2024-09-20.
- Security:
We are proceeding with “Airflow Beach Cleaning” project with Alpha-Omega
and Python Software Foundation with a goal to review and help all Airflow
Python dependencies (700+) in terms of their security processes,
willingness to cooperate and “cleaning” our part of the security beach. We
are engaged with - currently < 20 initial projects and based on those
conversations we will scale our efforts in the coming months and years -
with the goal of sharing our practices and teaching others in the
ecosystem to follow up similar patterns. We speak at FOSDEM and submitted
talks to FOSS Backstage and Pycon US next year where we want to share our
experience and enthuse others to follow.
- Meetups, Conferences & other community events
- Fortnightly Airflow 3 dev calls have had ~30 attendees from various
stakeholders: Astronomer, AWS, Bosch, Cloudera, Google. They have been
very productive for Airflow 3 development. Meeting notes at
https://cwiki.apache.org/confluence/x/8ApeEg . This is more focussed
towards developers; we have a separate monthly Airflow Town Hall call for
users which attract ~50 attendees too, e.g
https://lists.apache.org/thread/015ormvpbj0172gfh2ftvyytj2vwrt5m
- Videos from Airflow Summit 2024 have been uploaded on Airflow's YouTube
channel: https://www.youtube.com/@ApacheAirflow/playlists
- 100+ attendees in local meetups in London, NYC in the last quarter.
Upcoming local meetups in Vancouver (in Feb 2025):
https://www.meetup.com/paris-apache-airflow-meetup/
- Planning is underway for Airflow Summit 2025
## Community Health:
- Commit velocity has shown a great increase (14% QoQ) owing to Airflow 3
developement with a 28% increase in new PRs opened.
- The Dev mailing list activity has reasonably decreased as we have moved on
from planning phase to execution phase for Airflow 3.
- Committers have done a great job in creating a large number of "good first
issues" to attract more contributors who could also help in Airflow 3
development. This shows up in the numbers too: 14% increase with 233 code
conitrbutors this quarter:
https://github.com/apache/airflow/contribute
A good example of one such project here which had tons of new contributors:
https://github.com/apache/airflow/issues/42370
https://lists.apache.org/thread/bkkww0lfj7qhsqlxlyywynzglng2v346
- Downloads of the main Apache Airflow package have jumped to over 31
million/month (https://pypistats.org/packages/apache-airflow)
- GitHub stars: 31,552
- GitHub Contributors: 3,157
- Airflow Slack workspace members: 55,000
- Two new books have been published on Airflow
- Data Pipelines with Apache Airflow, Second Edition from Manning
Publication
- Apache Airflow Best Practices: A practical guide to orchestrating data
workflow with Apache Airflow from Packt Publishing
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Project Status: Current project status: On Going Issues for the board: None ## Membership Data: Apache Airflow was founded 2018-12-19 (6 years ago) There are currently 65 committers and 34 PMC members in this project. The Committer-to-PMC ratio is roughly 9:5. Community changes, past quarter: - Jens Scheffler was added to the PMC on 2024-08-06 - Ryan Hatter was added as committer on 2024-06-19 - Rom Sharon was added as committer on 2024-07-15 - Shahar Epstein was added as committer on 2024-07-15 ## Project Activity: Major work underway for Airflow 3. Airflow 3 summary https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0 - 15 AIPs agreed in scope for 3.0, target release date Mar/Apr next year. * Apache Airflow 2.10.1 was released on 2024-09-06. ## Community Health: Airflow Summit (which is being held in the Bay Area now): 637 attendees from 21 countries, 105 talks from 142 speakers. Videos from the talk will be on the Airflow YouTube channel in a couple of weeks. We have over 3000 contributors to the project. The project has received a grant of $10K by Bloomberg FOSS Contributor Fund.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Project Status: Current project status: ongoing, high activity Issues for the board: none ## Membership Data: Apache Airflow was founded 2018-12-19 (5 years ago) There are currently 62 committers and 33 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - No new PMC members. Last addition was Andrey Anshin on 2024-01-15. - Wei Lee was added as committer on 2024-04-04 ## Project Activity: The Airflow project has kicked off a project to plan and release the next major version, v3.0[1] and is currently having frequent "dev calls" to organize and align the community. Minutes are available in our Confluence Space[2]. Release activity shows our regular cadence of minor and bug fix releases, and regular "provider" packages": Apache Airflow 2.9.2 was released on 2024-06-10. Provider packages 2024-06-07 was released on 2024-06-09. Provider packages 2024-05-30 was released on 2024-06-03. Provider packages 2024-05-26 was released on 2024-05-30. Provider packages 2024-05-12 was released on 2024-05-17. Apache Airflow 2.9.1 was released on 2024-05-06. Provider packages 2024-05-01 was released on 2024-05-06. Apache Airflow Python Client 2.9.0 was released on 2024-05-02. Provider packages 2024-04-16 was released on 2024-04-21. Provider packages 2024-04-13 was released on 2024-04-16. Provider packages 2024-04-10 was released on 2024-04-13. Apache Airflow 2.9.0 was released on 2024-04-08. Provider packages 2024-03-25 was released on 2024-04-03. Apache Airflow 2.8.4 was released on 2024-03-25. Apache Airflow Helm Chart 1.13.1 was released on 2024-03-25. [1]: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0 [2]: https://cwiki.apache.org/confluence/x/8ApeEg ## Community Health: Activity on mailing lists and issues is down slightly over the last quarter but there is a lot of engagement in the first two Airflow 3 Dev/planning calls with around 30 participants in each call so far. The second in-person Airflow Summit is happening in San Francisco in September and had 200 submissions from 190 unique speakers and 78 talks accepted/scheduled. There were five talks about Apache Airflow in to CoC EU summit (3 in the Data Engineering track, 2 elsewhere), and only one was given by a PMC member.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Project Status: Current project status: Ongoing (High activity) Issues for the board: No issues worth board attention ## Membership Data: Apache Airflow was founded 2018-12-19 (5 years ago) There are currently 61 committers and 33 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - Andrey Anshin was added to the PMC on 2024-01-15 - No new committers. Last addition was Utkarsh Sharma on 2023-11-29. ## Project Activity: * We continue the steady release of bug fixes (2.8.*) and new feature releases (2.9) as well as regular Provider releases, Helm Chart and Python client. * Security: We achieved 100% artifacts build reproducibility - for Airflow, 90 providers, Helm Chart and Python client. We continue processing the security issues in a reasonable time, we have a strong response team that reacts quickly for severe issues (including one false alert where we were mislead by reporter description that project is much more severe, but we’ve been cooperating with Apache Superset (same component was vulnerable) and we managed to jointly work out good and reasonable response without unnecessary alerting our users. Work is ongoing on finishing what had been started with Sovereign Tech Fund with the goal of setting an example of being “CRA ready” long before CRA will even be defined. We introduced rotation of security team members and some of the new members are actively engaged in solving and triaging the issues already. * Upcoming 2.9 release is packed with features - we continue to evolve Airflow and do not let it stagnate, even with its 10 years history and incumbent status, we have quite a few new AIP(s) Airflow Improvement Proposals that we actively discuss and implement, and we continuously modernize both internal (CI/Development tools, new Contributor’s experience) as well as user-facing side of Airflow. We’ve almost completed modern UI rewrite of Airflow resulting with slick and coherent, responsive application, we are also strengthening our Data-Driven Scheduling Capabilities, Several improvements we developed over last few years got to the point that they allow for great synergies between then and with new improvements coming, and we seem to continue innovating with increasing speed - which seems like quite an achievement for 10 years old software. We started to address some of the problems that we had no capacity or energy for but they were highly requested by our users (e.g. DAG versioning) * The most active initiatives - interestingly enough a number of those are driven by individuals with very diverse affiliations - including some newcomers, but we also have a few people who return to active contributions after being away or less active for a while. * AIP-49: OTEL Traces * AIP-57: Refactor SLA feature * AIP-58 Airflow ObjectStore (AS) * AIP-61: Hybrid Execution * AIP-62: Getting Lineage from Hook instrumentation * AIP-63: DAG Versioning (3 sub-DAGs AIP-64,65,66) * AIP-67: Multi-Tenancy in Airflow * We started regular Town-Hall online meetings (three meetings already). We still have to see the effect of it but Briana Okyere (Astronomer employee) who took the role of Community Manager is very visible and impactful in the community leading a number of initiatives - like Town Hall, PR of the month, Meetups, working on Airflow Summit, Monthly newsletters, proposing code of conduct and much more. * PMC is aware of the “Astronomer Champion for Apache Airflow” that Astronomer proposed and discussed as potentially a program run by and with the PMC. Following our advice, Astronomer’s team reached out to trademarks@a.o and after considering various options and avoiding potential contention with “Apache Way” of running the project, they decided to run it as entirely “Astronomer’s” program. It’s being mentioned in community communications (including Town Hall Meetings and newsletters) but with clear Astronomer’s affiliation and without PMC involvement nor endorsement. * We cleaned up some of the “baggage” - remove out and outdated providers from active maintenance (except security fixes) and we continue extracting old and “legacy” part of Airflow to separated providers and replace those with new, modern solutions (Flask Application Builder which brings a lot of security issues, gets in direction of being optional part of Airflow). * We accepted contributions of a few providers - which we are usually quite reluctant about - from well respected entities (Teradata, Qdrant - popular vector database) which puts Airflow squarely in the middle of the AI revolution - our users seems to use Airflow more and more to orchestrate their AI related tasks (seems they fit nicely into Airflow’s architecture and strengths - which we also plant to strengthen even more). ## Events * The Airflow Summit 2024 will be held on September 10-12, 2024, in San Francisco. The event will mark the 10th Anniversary of Airflow (since the first commit) and this is going to be the main theme of Airflow Summit “Celebrating 10 years of Airflow”. Organizers aim for 800 - 1000 people, we already secured quite a good funding for that and we have requests from new Sponsors who are interested in sponsoring the event where Airflow users are in big numbers. Organizers of the Airflow Summit (Google, Astronomer and AWS) work closely with Producers (Software Guru) and the PMC to make this a successful event, where we will be able to join the expectations of Stakeholders but also reserve a big part of the event for Community driven efforts. Despite initial delays (reported in the previous report), with confirming date and getting permission from the Conferencing team we opened CFP and we are about to finish first round targeting people who need Visas to get to the US, so that we can confirm it in March, giving them at least 5 months to be able to obtain Visas which we consider as very important aspect of our community efforts. * We had a number of local Airflow Meetups: London, Cincinnati, and New York, São Paulo, Prague, Hyderabad, Bangalore, planning new events in Warsaw * Our community members gave talks all over the world - from Las Vegas, to Brussels (two talks at FOSDEM), Bangalore and more. ## Community Health: - We exceeded the milestone of 2800 contributors on GitHub this month (2700 in the last report). We see continuous interest in new contributors joining Airflow We have regular monthly newsletter and voting on PR of the month that regularly gets somewhat enthusiastic reactions from the community - We continue getting new publications at our medium publication https://medium.com/apache-airflow - whooping 10 new publications since last report by various contributors - We came back to Mentorship - we have MLH (Major League Hacking) sponsoring two interns (by Royal Bank of Canada) who work together (and showing good progress) on important component of Airflow (API/Connexion) that requires a major upgrade to free us from being depended on old dependencies with known security vulnerabilities. They are mentored by few mentors - some more and some less experienced people in our contributors / committers group.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Project Status: Current project status: Ongoing Issues for the board: See the events section ## Membership Data: Apache Airflow was founded 2018-12-19 (5 years ago) There are currently 61 committers and 32 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - No new PMC members. Last addition was Hussein Awala on 2023-07-29. - Amogh Desai was added as committer on 2023-09-13 - Jens Scheffler was added as committer on 2023-11-07 - Pankaj Koti was added as committer on 2023-09-13 - Utkarsh Sharma was added as committer on 2023-11-29 ## Project Activity: - We continue the steady release of bug fixes (2.7.*) and new feature releases (2.8) as well as regular Provider releases, Helm Chart and Python client. - The 2.8 release (Release Candidate is up for voting) is about to be released with new, important user-visible features: most notably fsspec integration became first-class citizen for Airflow (Python ecosystem de-facto standard for Object Storage interaction), consolidating logs from multiple components, new listener for datasets, ability to cache virtualenvs created dynamically. - There is an on-going work including efforts led by various stakeholders and individual community members: Internal API (AIP-44) and External Authentication and Authorisation framework (AIP-56 - part of Multi-Tenancy) - both step towards enabling Airflow to multi-tenant deployments, Next steps of the Open Telemetry integration (AIP-49), Strengthening Open Lineage integration (AIP-53). Work is ongoing on connecting Datasets, Object Storage, Event driven scheduling into a coherent modern way of interacting with data and datasets. Most of these efforts are supposed to land in Airflow 2.9. - The funding from Sovereign Tech Fund for the 4 individuals resulted in serious improvements and on-going work continues in that area. We have already a complete set of tools to prepare and publish SBOM information for Airflow and all 80+ providers. We are yet to publish information about it to our users. We have implemented reproducible builds for all providers which significantly improved reliability and robustness of verification of releases (work is on-going to implement them also for core Airflow in the way to not impact the development environment). We’ve added proactive scanning for security issues. Most importantly we reviewed and significantly improved security of the release process of our release/CI to add extra layers of protection - following a security report we received in October, where a typo/flaw in our CI process could make it potentially possible for malicious users to inject code into the release process of ours. We checked it did not happen and we introduced changes in the release process to make sure that similar issues will not happen again. Blog post about it (TODO: link to it) has been posted on Airflow Medium publication. We also iterated on our security model and security policies based on our interaction with security issue reporters and work on communicating better with our users and reporters with regards to security. - We are planning to improve the setup of the security team of ours by introducing rotation among the members. - We are planning to start regular Town-Hall online meetings (first planned in January). During the Airflow Summit we had an informal meeting (using the opportunity that a number of stakeholders, PMC members, committers were physically present, and one of the takeaways there was that synchronous (even remote ones) might work great (of course providing that no decisions will be made there, minutes will be posted, meetings will be recorded and agenda published well in advance) - and we are attempting to experiment with it and start regular meetings. - We started to actively pin-point some of the out-dated/not used providers and remove them from active maintenance. We proposed and approved the complete maintenance lifecycle for providers, with the goal of making sure we do not have to maintain the code and that it does not slow us down, but at the same time users will still be able to use the old providers and we are ready to address security issues for them. We are doing it with `daskexecutor`, `plexus` and `apache.sqoop` provider first following discussions and lazy consensus. This is part of our continuous effort to keep Airflow lean and adapt to changing reality - We accepted a contribution (from Astronomer) of 5 LLM-related providers - which is part of our continuous effort to adapt to the changing environment and make Airflow a viable platform for LLM based workflows. ## Community Health: - We exceeded the milestone of 2700 contributors on GitHub this month (2600 in the last report). We see continuous interest in new contributors joining Airflow - We have regular monthly newsletter and voting on PR of the month that regularly gets somewhat enthusiastic reactions from the community - We continue getting new publications at our medium publication https://medium.com/apache-airflow - whooping 18 new publications since last report by various contributors ## Events The Airflow Summit 2023 September 19-21 in Toronto, Canada was a great success. We had around 500 attendees, the event broke-even financially and sponsors, speakers and attendees satisfaction rate was high. We recorded and published all talks on the Airflow Summit youtube channel. We gave free passes to all the committers, we funded 15 scholarships for under-represented attendees and our sponsors and organizers are looking forward to the 2024 event that will mark the 10th anniversary of Airflow. The organizers work closely with the PMC, they’ve already started preparation and planning and the date is set for the week 9th-11th of September 2024 in the Bay Area in the US (details to be finalized), the plan is to have at least 1000 attendees. The organizers obtained permission from the ASF trademarks, the efforts continue in cooperation with Software Guru who are producers of the event. The event is organized by representatives of the biggest stakeholders (Amazon, Google, Astronomer), PMC members and other community members).
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines. ## Project Status: Current project status: Ongoing, High Activity Issues for the board: No issues worth board attention ## Membership Data: There are currently 57 committers and 32 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: * Hussein Awala was added to the PMC on 2023-07-29 * Dennis Ferruzzi was added as committer on 2023-07-04 * Maciej Obuchowski was added as committer on 2023-06-28 * Pankaj Singh was added as committer on 2023-06-13 * Phani Kumar was added as committer on 2023-06-28 * Vincent Beck was added as committer on 2023-06-26 We are also just about to announce two new - just voted-in - committers: * Pankaj Koti * Amogh Desai We plan to announce them at the Airflow Summit during our “The Faces of Airflow” discussion panel where ampng others we will praise the community of Airflow, committers, contributors and everyone who makes Airflow such a great community ## Project Activity: • We continue the steady release of bugfixes (2.6.*) and new feature releases (2.7) as well as regular Provider releases, Helm Chart and Python client. • The 2.7 release (https://airflow.apache.org/blog/airflow-2.7.0/) was packed with new, important user-visible features: Setup/Teardown, New Cluster activity and Gantt Chart UI, but also a number of changes that move Airflow in the general direction of becoming “Workflow Platform” rather than “Workflow solution”. Those changes include completion of Executor Coupling removal (AIP-51) culminated by separation of Celery, Kubernetes and Dask executors to providers (including configuration separation), allowing to extend airflow with fully-featured 3rd-party executors OpenLineage integration, but also making it clear what is and what is not Public Interface of Airflow covered by our SemVer promises. • There is an on-going work including efforts led by various stakeholders and individual community members: Internal API (AIP-44) and External Authentication and Authorisation framework (AIP-56 - part of Multi-Tenancy) - both step towards enabling Airflow to multi-tenant deployments, Next steps of the Open Telemetry integration (AIP-49), Strengthening Open Lineage integration (AIP-53)). • With the help and involvement of the ASF Security team, we managed to establish a highly functional security team that operates in a much more efficient way and delivered great results: we managed to fix all outstanding issues and we have a working process to manage influx of new issues in a much better organized and efficient way. We also implemented structural changes that eliminated a wide range of potential security issues as well as improved our communication around the Security Model of Airflow towards users and researchers. Findings and learnings from building the team will be shared at the Community Over Code conference in Halifax at the talk of Jarek Potiuk who led the effort and Arnount Engelen - who is part of the ASF security team https://communityovercode.org/schedule-list/#CY011. Work continues on improving the process and tooling. Releases: * Provider packages 2023-08-29 was released on 2023-09-01. * Provider packages 2023-08-26 was released on 2023-08-29. * Apache Airflow Python Client 2.7.0 was released on 2023-08-24. * Apache Airflow 2.7.0 was released on 2023-08-18. * Provider packages 2023-08-11 was released on 2023-08-14. * Provider packages google 2023-08-09 was released on 2023-08-10. * Provider packages redis 2023-08-08 was released on 2023-08-09. * Provider packages 2023-08-05 was released on 2023-08-08. * Provider packages 2023-07-29 was released on 2023-08-01. * Provider package elasticsearch 2023-07-17 was released on 2023-07-21. * Provider packages 2023-07-12 was released on 2023-07-16. * Apache Airflow 2.6.3 was released on 2023-07-10. * Provider packages ad hoc release 2023-07-09 was released on 2023-07-10. * Provider packages 2023-07-06 was released on 2023-07-09. * Apache Airflow Helm Chart 1.10.0 was released on 2023-06-27. * Apache Airflow Python Client 2.6.2 was released on 2023-06-25. * Provider packages 2023-06-20 was released on 2023-06-23. * Apache Airflow 2.6.2 was released on 2023-06-17. ## Community Health: * The Airflow Summit 2023 is happening September 19-21 in Toronto, Canda. And it’s just round the corner. We have a full program and the number of attendees that already registered surpassed our expectations (we are estimating 470 - 500 attendees). We gave free passes to all the committers, we funded 15 scholarships for under-represented attendees and we have a healthy set of sponsors and we are increasing our promotional efforts in cooperation with Software Guru who are producers of the event. The event is organized by representatives of the biggest stakeholders (Amazon, Google, Astronomer), PMC members and other community members). * We had a number of local Airflow Meetups: Washington DC, Toronto, New York, and Portland * We just reached the milestone of 2600 contributors on GitHub this month (2500 in the last report). We see continuous interest in new contributors joining Airflow * We have regular monthly newsletter and voting on PR of the month that regularly gets somewhat enthusiastic reactions from the community * Despite the holiday period, we continue getting new publications at our medium publication https://medium.com/apache-airflow - 7 new publications since last report
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Project Status: Current project status: Ongoing Issues for the board: No issues worth board attention ## Membership Data: Apache Airflow was founded 2018-12-19 (4 years ago) There are currently 53 committers and 31 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - Brent Bovenzi was added to the PMC on 2023-03-15 - Hussein Awala was added as committer on 2023-04-12 - Pankaj Singh was added as committer on 2023-06-13 ## Project Activity: * We reached some milestones in the last quarters that indicate strong community and healthy activity: 30K stars on GitHub, 2500 contributors who contributed, 20K commits overall. * We released the 2.6 feature release - as planned. It included new interesting features - notification support that seems to be anecdotally (and somewhat surprisingly) highly valued and prised by our users, improvements to Grid View, solving a long time “tasks stuck in Queue” problems and making Async logs available in in the integrated logging, continuous timetable * We are steadily releasing Airflow core patchlevel/bugfix releases (with roughly monthly cadence) and gearing up to Airflow 2.7 feature release in the coming month. The 2.7 will contain AIP-52 (setup/teardown) and if we manage to implement it also AIP-44 (internal API), improved UI, and a number of other features we work on. * We are also steadily releasing 80+ provider packages that extend Airflow capabilities and allow for integration with external services/software with roughly bi-weekly cadence. * The important events in the last quarter was getting rid of Python 3.7 support (Python 3.7 reaches end-of-life status in June 2023 and many of our dependencies already dropped support for it. We proposed, discussed, voted and updated our policies, deciding on a bit more relaxed timing for dropping the support before the actual end-of-life date (we already dropped support in `main` which will apply to the upcoming Airflow 2.7 version). * We’ve implemented a series of optimizations and stabilizations of our CI builds, leading to lower build time utilization as well as faster feedback and more stability for committers and contributors. We have not measured the overall impact but we can talk roughly about 20-30% improvements in feedback time, 20-30% improvements in build time utilization and (hopefully as a result) dramatic improvements in stability of the builds and decreasing flakiness of the CI. We are looking forward to future improvements especially coming from some of the Github releases (K8S Controller they released and ASF Infrastructure options like self-hosted runners). * Big news (voted PR of the month) was adding Python 3.11 support. We worked closely with our dependencies (for example Apache Beam, Papermill), and stakeholders of libraries (Google) that held us back from supporting Python 3.11 (Released in October 2023) and finally we managed to bring Python 3.11 support to `main` - this opened up for a huge bump in versions of our dependencies that were held back so far, increasing the security of the supply chain of ours. The upcoming 2.6.2 release already supports Python 3.11. Also we worked out, discussed and voted on policies for suspending providers that hold us back and we applied it to “Yandex” provider (we informed Yandex library maintainers that we are suspending new releases until they fix their dependency issues). The policy is documented here: https://github. com/apache/airflow/blob/main/PROVIDERS.rst#suspending-releases-for-providers * Our provider released bumped minimum version of Airflow supported to 2.4 following our policies which are part of the effort of incentivising our users to upgrade to newer versions of Airlfow quicker https://github.com/apache/airflow/blob/main/PROVID ERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers * There is an on-going work including efforts led by the stakeholders and community members that have not been as active before: Open Telemetry integration (AIP-49), Open Lineage integration (AIP-53), External Authentication and Authorisation framework (AIP-56 - part of Multi-Tenancy), Automatic Setup/Teardown (AIP-52, Airflow Internal API (AIP-44 part of Multi-Tenancy), Removing Executor Coupling (AIP-51). Most of the efforts have a common theme of making Airflow more of an extensible workflow platform than just an end-user solution. * Since we got substantially more security issues reported to us by security researchers - result of Airflow being part of Hacker One bounty, we are in the process of improving and redefining a process for handling security issues. This process is somewhat experimental (we work closely with the ASF security team) as it involves not only PMC members and committers, but also other stakeholders and external security researchers who are not (yet) committers. This is somewhat at odds with the established policy of who should be on the security team but we got approval of VP Security of the ASF for the process and we are working on putting that into practice to evaluate if that can help in handling the reports in a more timely fashion, at the same time allowing external stakeholders to participate in the process following very well defined process. The process has been discussed, voted and applied already, we are working on making a routine. The security team is established, and working on the issues, we are going to intensify the work and discussions there. Part of the discussion is also general improvement in the process including adding SBOM exports and generally improving the way how we communicate security issues to our users. More information about the process we are following is here: https://github.com/apach e/airflow/blob/main/.github/SECURITY.rst#handling-security-issues-in-airflow ## Community Health: * We have regular monthly newsletter and voting on PR of the month that regularly gets somewhat enthusiastic reactions from the community We have a number of discussion redirected from PR/Slack chats to the devlist, we actively redirect “substantial” discussions requiring architectural changes to devlist to make sure we have an inclusive community * The rules on what we consider breaking changes that we started to communicate with the upcoming 2.6 release already had effect and helped to facilitate some discussion on “what is breaking” - with 3rd-parties/downstream projects that use Airflow (external Helm Chart). Hopefully they will make expectations better aligned. * We have more people publishing in our medium “Airflow” Publication https://medium.com/apache-airflow that seems to pick up the steam. We have 13 new blog posts published by authors from all-over-the community (including authors from various stakeholders) since March 21 (2 months). We also have almost 800 followers. This is a significant uptick (there are in total 34 articles since Feb 2021). * We are reaching 2500 contributors on GitHub this month (2495 currently so just 16 more contributors left). We continue to increase our lead over Spark which is the second biggest project of Apache Software Foundation on GitHub regarding the number of contributors . We see continuous interest in new contributors joining Airflow ## Events The community is getting ready for Airflow Summit 2023 that is happening September 19-21 in Toronto, Canda. We are already past the selection of 80% of the content and the Early Birds ticket ended on 10th of June. We have a substantial audience already who purchased tickets. We decided to give free passes to all the committers and we have a healthy set of sponsors and we are increasing our promotional efforts in cooperation with Software Guru who are producers of the event. The event is organized by representatives of the biggest stakeholders (Amazon, Google, Astronomer), PMC members and other community members). We had a number of local Airflow Meetups: London, New York, Paris, Charlotte, NC, New York and others.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines. ## Membership Data: There are currently 52 committers and 31 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - Brent Bovenzi was added to the PMC on 2023-03-15 - Pierre Jeambrun was added to the PMC on 2023-03-14 - Hussein Awala was added as committer on 2023-04-12 ## Issues: No issues worth board attention ## Project Activity: - We are speeding up release cadence - providers are released roughly twice a month and we are working on faster release of minor versions of Airflow on top of regular patchlevel releases. Improving release cadence has a number of benefits: better automation of release process, more incremental and smaller changes. - We included API client releases in the airflow release process to avoid the problem of missing API clients for new released versions of Airflow. - In order to streamline our release process, we introduced (following the ASF rules) an accelerated RC2+ voting process, where small individual problems detected during testing of previous RC would previously cause us to restart the whole voting time. Now small fixes to RC candidates do not automatically extend the voting time. - We are gearing up for 2.6 release which we accelerated to improve the release cadence as we saw that some bigger planned features (Setup/Teardown AIP-52, Internal API for multi-tenancy AIP-44 need more time to be ready. We introduced feature flags to release 2.6 with those features (partially implemented) disabled. We got substantially more security issues reported to us by security researchers - very likely as a result of Airflow being put on a bug-bounties as important supply chain component, we try to handle them with urgency in some areas, though there are still a few of those issues that need more engagement for the areas that do not get handled quickly - There's an on-going work including efforts led by the stakeholders and community members that have not been as active before: Open Telemetry integration (AIP-49), Open Lineage integration (AIP-53), External Authentication and Authorisation framework (AIP-56 - part of Multi-Tenancy), Automatic Setup/Teardown (AIP-52, Airflow Internal API (AIP-44 part of Multi-Tenancy), Removing Executor Coupling (AIP-51). Most of the efforts have a common theme of making Airflow more of an extensible workflow platform than just an end-user solution. ### Releases: Provider packages 2023-04-02 was released on 2023-04-06. Apache Airflow 2.5.3 was released on 2023-04-01. Apache Airflow 2.5.2 was released on 2023-03-15. Provider packages 2023-03-07 hashicorp was released on 2023-03-10. Provider packages 2023-03-03 was released on 2023-03-06. Provider packages 2023-02-18 was released on 2023-02-21. Provider packages 2023-02-08 was released on 2023-02-11. Apache Airflow Helm Chart 1.8.0 was released on 2023-02-06. Apache Airflow Go Client 2.5.0 was released on 2023-02-03. Apache Airflow Python Client 2.5.1 was released on 2023-02-01. Provider packages 2023-01-23 papermill, docker, apache.cassandra was released on 2023-01-26. Apache Airflow 2.5.1 was released on 2023-01-20. Provider packages 2023-01-14 was released on 2023-01-17. ### Events: The community is getting ready for Airflow Summit 2023 that is happening September 19-21 in Toronto, Canda. ## Community Health: - We have regular monthly newsletter and voting on PR of the month that regularly gets somewhat enthusiastic reactions from the community - We have a number of discussion redirected from PR/Slack chats to the devlist, we actively redirect “substantial” discussions requiring architectural changes to devlist to make sure we have an inclusive community - We continuously improve the triage process to provide faster and more accurate feedback to users raising issues, we introduced “needs-triage” label for GH issues that require triaging and we attempt to track those and possibly in the future work out ways to improve those. - We have quite an engagement from our users and committers on release preparation and release candidates testing. We publish “Status of the RC release” github issue where the users are testing and verifying our RC candidates, users are also actively engaged in testing those releases and seems that their engagement helps - we had a number of cases where additional testing of Release Candidates revealed some issues. This led us to introducing the “accelerated RC voting” process where we could shorten the voting period for RC2+ candidates when only a very small change was found and fixed in a previous RC. - Several of our users complained that they had no information on how to unsubscribe from users@ list - we decided to enable the footer with the information included following the advice from the VP privacy of the ASF to be more compliant with GDPR and similar regulations. - We introduced clearer rules on what we consider breaking changes and we will start communicating them with the upcoming 2.6 release - those rules clarify what 3rd-parties and our users can expect from integrating their custom code with Airflow and what they can rely on, further strengthening the “Airflow-as-a-Platform” approach. - We are tightening our policies around Providers (integrations of Airflow with external services and applications). We are clarifying the policies on when we accept new providers and when we prefer that they are developed outside of the Apache Software Foundation code as external integrations - following our well-defined Provider interfaces. We have also introduced rules that will allow us to suspend new releases of some niche providers that are holding us back due to using old and out-dated dependencies, which should improve security/supply chain healthiness as well as allow us to move faster with our release process. - We are working closer with the stakeholders (Amazon, Google, Databricks and others) where providers are historically part of the Apache Software foundation code and following the mixed-governance model we introduced (following the Apache rules) - We collaborate on upgrading the outdated dependencies of those providers and working out automated testing schemes which allow us to move faster without compromising quality and respond to changing remote APIS. That includes introducing automated system test dashboards managed by the stakeholders based on our system tests (currently works for Amazon, in the works are Google and Databricks ones). - The main stakeholder - Astronomer - through the OSS team members of Astronomer provides the backbone of the steady improvements and making Airflow strong and healthy when it comes to stability and continuously adding of new features and fixing issues, while there is also an effort of other stakeholders’ supported contributors to add new features to Airflow. Often people supported by those various stakeholders cooperate on the new Airflow features even if they are competing usually (leading to healthy “coopetition” ). We also have a continuous influx of new contributors, who are independent and not affiliated with the bigger stakeholders and they continue to raise to the ranks of committers and PMC members
No report was submitted.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Airflow was founded 2018-12-19 (4 years ago) There are currently 51 committers and 29 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Daniel Standish on 2022-09-22. - Niko Oliveira was added as committer on 2022-12-19 - Andrey Anshin was added as committer on 2022-11-28 ## Project Activity: We are on a pretty steady release cadence. Next release (2.5.1) is being voted upon as we are submitting this report. - Provider packages 2023-01-02 was released on 2023-01-05. - Provider packages apache-hive, microsoft-azure 2022-12-14 was released on 2022-12-19. - Apache Airflow 2.5.0 was released on 2022-12-02. ## Community Health: We are a very lively community with lots of activity. Work is underway for another Airflow Summit that for the first time will be in-person. - dev@airflow.apache.org had a 8% increase in traffic in the past quarter (390 emails compared to 358) - users@airflow.apache.org had a 31% decrease in traffic in the past quarter (53 emails compared to 76) - 1499 commits in the past quarter (8% increase) 223 code contributors in the past quarter (3% increase) - 1075 PRs opened on GitHub, past quarter (-6% change) - 1073 PRs closed on GitHub, past quarter (-5% change) - 422 issues opened on GitHub, past quarter (-7% change) - 418 issues closed on GitHub, past quarter (-6% change)
No report was submitted.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: There are no issues requiring board attention ## Membership Data: Apache Airflow was founded 2018-12-19 (4 years ago) There are currently 48 committers and 27 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - No new PMC members. Last addition was Jedidiah Cunningham on 2022-01-04. - No new committers. Last addition was Ping Zhang on 2022-05-31. We are in the process of inviting a couple of new committers and PMC members ## Project Activity: Apache Airflow 2.4.0 was released 2022-09-19 and was a big release for us as it adds some exciting new capabilities (that we are calling "Data-aware scheduling"). A write up of this new release with more detail is available at https://airflow.apache.org/blog/airflow-2.4.0/ Main package Releases: - Apache Airflow 2.4.0 was released on 2022-09-19. - Apache Airflow 2.3.3 was released on 2022-07-09. - Apache Airflow 2.3.4 was released on 2022-08-23. Sub/smaller package releases. - Provider packages 2022-08-15: 2022-09-08 were released on 2022-09-08. - Provider packages 2022-08-15 was released on 2022-08-18. - Provider packages 2022-08-10 was released on 2022-08-14. - Apache Airflow Python Client v2.3.0 was released on 2022-08-03. - Provider packages 2022-07-17 was released on 2022-07-20. - Provider packages 2022-07-13 was released on 2022-07-16. - Provider packages 2022-06-15 was released on 2022-06-22. Each of these "provider packages" are a sub-component of Airflow that we have split out into their own packages to a) allow easier upgrading for our users without affecting "Core" scheduling, and b) to have quicker release cycles of those components (they are more isolated and require less testing than the core scheduler/workflow engine.) ## Community Health: dev@airflow.apache.org had a 8% increase in traffic in the past quarter (390 emails compared to 358) users@airflow.apache.org had a 31% decrease in traffic in the past quarter (53 emails compared to 76) 1479 commits in the past quarter (-31% change) 234 code contributors in the past quarter (12% increase) 1093 PRs opened on GitHub, past quarter (-18% change) 1141 PRs closed on GitHub, past quarter (-14% change) 425 issues opened on GitHub, past quarter (-17% change) 465 issues closed on GitHub, past quarter (-22% change)
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Airflow was founded 2018-12-19 (3 years ago) There are currently 48 committers and 27 PMC members in this project. The Committer-to-PMC ratio is roughly 8:5. Community changes, past quarter: - No new PMC members. Last addition was Jedidiah Cunningham on 2022-01-04. - Ping Zhang was added as committer on 2022-05-31 ## Project Activity: We released the long-awaited 2.3.0 release with some exciting features. This was the biggest release so far after introducing Airflow 2 and it it involved quick turnaround of 2.3.1 and 2.3.2 (already released) with fixing teething problems encountered - but we got a very good involvement of our users who not only reported problems but also were willing to take part in testing of those issues and were very helpful in test phase. Seems like the community expectations and engagement is pretty well aligned with our approach with new releases. No big dramas even with such a big release. The list of features implemented: * Dynamic task mapping - AIP-42 (huge improvement in the way how users can build Dags) * New GridView replacing Task View that modernizes the “first” experience the users have with Airflow DAG monitoring * Tooling to purge historical data for Airflow Database * LocalKubernetesExecutor that allows for simpler deployments for users who wish to optimize their experiences when it comes to resource usage * DagProcessorManager as standalone process (AIP-43) which is the first of many steps towards improved security and full multi-tenant setup * JSON serialization for connections that simplifies credentials management for our users * Offline generation of DB migration scripts and “downgrade” feature - that make migration process reversible and allows power users to optimize their migration experience * ARM architecture support - for users who wish to run Airflow on ARM for both development and deployment (experimental) including docker image support After 2.3.* effort we focus on further, incremental steps of improving Airflow features - through Airflow Improvement Proposals (AIP) as part of the upcoming Airflow 2.4.* effort: * Improving security and multi-tenancy of Airflow AIP-44 Airflow Internal API * Implementing DAG versioning AIP-36 DAG Versioning * Adding data dependency management to Airflow AIP-48 Data Dependency Management and Data Driven Scheduling * Improving System Testing for Airflow for external integrations AIP-47 New design of Airflow System Tests Also smaller improvements and optimisations: * AIP-45 Remove double dag parsing in airflow run * AIP-46 Runtime isolation for airflow tasks and dag parsing ### Releases: * Apache Airflow 2.3.2 was released on 2022-06-04. * Apache Airflow 2.3.1 was released on 2022-05-25. * Apache Airflow Helm Chart 1.6.0 was released on 2022-05-20. * Provider packages 2022-05-12 was released on 2022-05-16. * 2.3.0 was released on 2022-04-30. * Provider packages 2022-04-30 was released on 2022-04-30. * Provider packages 2022-04-13 was released on 2022-04-22. * Provider packages 2022-04-07 was released on 2022-04-11. * Apache Airflow 2.2.5 was released on 2022-04-04. * Provider packages 2022-03-23 was released on 2022-03-26. * Provider packages 2022-03-15 was released on 2022-03-19. * Apache Airflow Helm Chart 1.5.0 was released on 2022-03-11. * Provider packages 2022-03-07 was released on 2022-03-10. # Events The Airflow Summit has been successfully completed. We got more than 7500 online users and more than 400 local participants in local events, via something that we consider as a first-of-a-kind distributed (not hybrid) event. We held local events in 13 location and we had 5 informal watch parties: * Local events: London, Tokyo, Lagos, New York, Bengaluru, SF Bay Area, Paris, Warsaw, Melbourne, Sydney, Sao-Paulo, Tel-Aviv, Seattle. * Watch parties: Atlanta, Chenai, Lisbon, Pune, Skopje. The event was completely free to attend. Talks were transmitted online and made immediately available to all registered users. We had 53 talks during the week and 5 interactive workshops with ~ 1900 USD of income (we will eventually donate all the workshop proceedings to the Apache Software Foundation - all workshops were done by volunteers) ## Community Health: Airflow continues to be the top ASF-contributed project (in terms of number of contributors) - we surpassed 2000 contributors overall (2066 on 4th of June 2022). We continue to attract new contributors - with seemingly accelerating speed. We put on hold some of the new discussions and features because of the - already packed - 2.3 release and Airflow Summit, but we are resuming the effort to complete the discussions on new features and lead some of them to completion in 2.4.0 release.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: ## Membership Data: Apache Airflow was founded 2018-12-19 (3 years ago) There are currently 47 committers and 27 PMC members in this project. The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - Jedidiah Cunningham was added to the PMC on 2022-01-04 - Josh Fell was added as committer on 2022-02-18 - Malthe Borch was added as committer on 2022-02-18 ## Project Activity: ### In-progress work We are gearing up for a new MINOR release that should address AIP-42 Dynamic Task Mapping important feature mentioned in the previous report. The earlier investment and stabilization of the Airflow codebase enabled us to start actively working on bringing more improvements to Airflow in future versions. What is important, those improvements are coming from multiple stakeholders and coming from needs and often contributed code that has been or is being implemented by the stakeholders. * Improving security and multi-tenancy of Airflow * AIP-43 DAG Processor separation * AIP-44 Airflow Internal API * Implementing DAG versioning * AIP-36 DAG Versioning * Adding data dependency management to Airflow * AIP-48 Data Dependency Management and Data Driven Scheduling * Improving System Testing for Airflow for external integrations * AIP-47 New design of Airflow System Tests * Also smaller improvements and optimisations: * AIP-45 Remove double dag parsing in airflow run * AIP-46 Runtime isolation for airflow tasks and dag parsing ### Releases 2.2.4 was released on 2022-02-22. Provider packages 2022-02-14 was released on 2022-02-18. Provider packages 2022-02-09 was released on 2022-02-13. Provider packages 2022-01-08 was released on 2022-01-12. Apache Airflow Helm Chart 1.4.0 was released on 2022-01-10. Provider packages 2021-12-31 was released on 2022-01-06. 2.2.3 was released on 2021-12-21. Provider packages 2021-12-07 was released on 2021-12-15. ### Events The Airflow Summit Steering Committee continues preparation to the Airflow Summit which is supposed to be both online and local offsite event for the Airflow community where local communities will have a chance to meet in-person and take part in global event at the same time ## Community Health: * Airflow continues to be the top ASF-contributed project (in terms of number of contributors) - we are getting close to 2000 contributors overall (1966 on 6th of March 2022). Well ahead 2nd Apache Spark (1791). * In the first week of March 2022 Airflow exceeded 10.000 Forks and 25.000 Stars in Github * We continue to have a steady flow of contributions from both seasoned contributors and new contributors * We effectively utilize Github Discussions to separate Bugs/Features from discussions. We have around 200 unique new / 600 unique regular monthly users of Discussions * Some of the topics focus around special interest groups (multi-tenancy) where we semi-regularly meet and discuss the progress (video chats which we record, publish and summarize for all community to follow) - this is especially when we have multiple stakeholders interested in the areas
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: ## Membership Data: Apache Airflow was founded 2018-12-19 (3 years ago) There are currently 45 committers and 26 PMC members in this project. The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - No new PMC members. Last addition was Ephraim Anierobi on 2021-08-31. - No new committers. Last addition was Brent Bovenzi on 2021-08-27. ## Project Activity: Recent releases: Provider packages 2021-11-30 was released on 2021-12-06. 2.2.2 was released on 2021-11-15. Provider packages (Amazon) 2021-11-04 was released on 2021-11-11. Airflow Helm Chart Check 1.3.0 was released on 2021-11-08. Provider packages 2021-10-30 was released on 2021-11-04. 2.2.1 was released on 2021-10-29. 2.2.0 was released on 2021-10-11. Provider packages 2021-10-08 was released on 2021-10-11. Provider packages 2021-09-30 was released on 2021-10-05. Apache Airflow Helm Chart 1.2.0 was released on 2021-09-28. 2.1.4 was released on 2021-09-18. Provider Packages 2021-09-04 was released on 2021-09-08. ## Community Health: - There's quite a lot of interesting discussions going on, those include introducing multi-tenancy or redesigning Airflow’s executors as well as possible drop in support for MySQL. All those discussions show that Apache Airflow is actively used and developed and there are still things that can be improved. - To reward some of Airflow users and contributors we are running an end-of-year campaign to send swags to the most active and creative participants in the campaign. - Community voted a crucial Airflow Improvement Proposals that will have huge impact on future of Airflow: [AIP-42 Dynamic Task Mapping][1] ### Events: The Airflow Summit Steering Committee has started planning 2022 summits for the Airflow community. [1]: https://s.apache.org/od4lx
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines. ## Membership Data: There are currently 45 committers and 26 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - Elad Kalif was added to the PMC on 2021-08-30 - Ephraim Anierobi was added to the PMC on 2021-08-31 (Just to mention it here: Ephraim is an alumni of the Outreachy program that Apache participated in. This is a cool story!) - Aneesh Joseph was added as committer on 2021-07-06 - Brent Bovenzi was added as committer on 2021-08-27 - Jed Cunningham was added as committer on 2021-06-25 - Tzu-ping Chung was added as committer on 2021-06-25 ## Project Activity: ### Releases: - Provider Packages 2021-09-04 was released on 2021-09-08. - Provider Packages 2021-08-30 was released on 2021-09-03. - Provider Packages 2021-08-24 (cncf.kubernetes) was released on 2021-08-27. - 2.1.3 was released on 2021-08-23. - Provider packages 2021-07-27 was released on 2021-08-02. - Apache Airflow Helm Chart 1.1.0 was released on 2021-07-26. - 2.1.2 was released on 2021-07-14. - 2.1.1 was released on 2021-07-02. - Provider packages 2021-06-26 was released on 2021-07-02. - upgrade-check-1.4.0 was released on 2021-06-26. - Provider packages 2021-06-18 was released on 2021-06-23. As of June 17, 2021 the 1.10.X version of Apache Airflow reached end of life. ### CVEs published - CVE-2021-38540: Apache Airflow: Variable Import endpoint missed authentication check < https://lists.apache.org/thread.html/rb34c3dd1a815456355217eef34060789f771b6f77c3a3dec77de2064%40%3Cusers.airflow.apache.org%3E > - CVE-2021-35936: Apache Airflow: No Authentication on Logging Server < https://lists.apache.org/thread.html/r53d6bd7b0a66f92ddaf1313282f10fec802e71246606dd30c16536df%40%3Cusers.airflow.apache.org%3E > ### Events: Airflow Summit 2021 <https://airflowsummit.org> took place July 8-16, 2021 which was attended by more than 10,000 participants from all over the world. That was up from 6000 attendees in Airflow 2020 and brought much higher quality content. We engaged meetup groups from 8 different cities around the globe, and hosted in 2 different time blocks to accommodate attendees from Europe, Americas and Asia. We are looking forward to the 2022 edition which - hopefully - will bring back the physical component, keeping the “free to attend” option for online conference. ## Community Health: The community seems to be very active despite the decrease in traffic on dev@airflow (probably due to the summer season). The number of commits and code contributions increased by 12% and 24% correspondingly. While the current big stakeholders - mainly Astronomer - grow stronger when it comes to a number of committers and PMC members, we also added committers and PMC members coming from different backgrounds and companies. There is also an increased activity from other stakeholders - Google and Amazon.
## Description:
The mission of Apache Airflow is the creation and maintenance of software
related to workflow automation and scheduling that can be used to author and
manage data pipelines.
## Membership Data:
There are currently 41 committers and 24 PMC members in this project. The
Committer-to-PMC ratio is roughly 2:1.
Community changes, past quarter:
- No new PMC members. Last addition was Xiaodong Deng on 2020-12-25.
- Xinbin Huang was added as committer on 2021-04-01
- Daniel Standish was added as committer on 2021-03-01
- Yu Qian was added as committer on 2021-04-06
## Project Activity:
### Releases:
- Apache Airflow Python Client v2.1.0 was released on 2021-05-31.
- Apache Airflow Helm Chart 1.0.0 was released on 2021-05-19.
- Provider packages 2021-05-01 was released on 2021-05-06.
- Apache Airflow Python Client v2.0.0 was released on 2021-04-23.
- 2.0.2 was released on 2021-04-19.
- Provider packages 2021-04-7 was released on 2021-04-13.
- Backport Provider Packages 2021.4.10 was released on 2021-04-10.
- Backport Provider Packages 2021.3.17 was released on 2021-03-18.
- 1.10.15 was released on 2021-03-17.
- Apache Airflow Elasticsearch Provider 1.0.3 was released on 2021-03-16.
- upgrade-check-1.3.0 was released on 2021-03-14.
- Backport Provider Packages 2021.3.13 was released on 2021-03-13.
- Provider packages 2021-03-08 was released on 2021-03-11.
- Backport Provider Packages 2021.3.3 was released on 2021-03-07.
- Provider packages 2021-02-27 was released on 2021-03-07.
With June 17, 2021 the 1.10.X version of Apache Airflow will reach end of
life. We are also proud that the first ever official version of the Airflow
Helm chart was released. We believe that this is a big step in further
adoption of Airflow.
### CVEs published
- CVE-2021-28359: Apache Airflow Reflected XSS via Origin Query Argument in
URL
<https://lists.apache.org/thread.html/ra8ce70088ba291f358e077cafdb14d174b7a1ce9a9d86d1b332d6367%40%3Cusers.airflow.apache.org%3E>
### Events:
- We are getting ready for Airflow Summit 2021
<https://airflowsummit.org> which is going to take place July 8-16,
2021. There’s going to be a lot of interesting talks from diverse speakers,
first-time contributor workshops, dedicated first-time contributor and
first-time users days with talks.The summit is done with cooperation
with 8 meetups: Bangalore, Bay Area, London, Melbourne, New York City,
Telaviv, Tokyo, Warsaw
- There was an Airflow Tel-Aviv meetup 10th of March,
https://www.meetup.com/pl-PL/tel-aviv-apache-airflow-meetup/events/276448622/
- There was an Airflow virtual meetup on the 29th of April that was hosted in
2 languages: English and Spanish. This is a larger effort to bring Airflow
content to Spanish speaking communities.
https://www.crowdcast.io/e/j3pqcxkf/register
## Community Health:
- The community is very active (dev@airflow had a 34% increase in mails). We
believe it’s strongly connected with the long awaited Airflow 2.0 release
and also with huge support from companies like Astronomer who widley promote
Airflow. While we observed a decrease in the number of commits and opened
PRs (-24% and -22%) we think it’s nothing to worry about as it can be
related with cooldown after preparing and polishing Airflow 2.0.
- Community voted on three crucial Airflow Improvement Proposals that will
have huge impact on future of Airflow:
- AIP-38: Modern Web Application
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application>
- AIP-39: Richer scheduler_interval
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-39+Richer+scheduler_interval>
- AIP-40: Deferrable ("Async") Operators
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929>
- As we are firm believers in inclusivity we’ve completed renaming of the HEAD
branch of airflow repository to “main”. We are going to continue with
keeping our eye on non-inclusive language in our projects
- The conflict between two members of the PMC has been solved and they both
cooperate including submitting joint talk at ApacheCon
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines. ## Membership Data: There are currently 39 committers and 24 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - Xiaodong Deng was added to the PMC on 2020-12-25 - Daniel Standish was added as committer on 2021-03-01 - Elad Kalif was added as committer on 2021-02-26 - Ephraim Anierobi was added as committer on 2021-02-17 - James Timmins was added as committer on 2021-02-26 - Vikram Koka was added as committer on 2021-01-05 ## Project Activity: ### Releases: - 2.0.1 was released on 2021-02-08. - 2.0.0 was released on 2020-12-17. - 1.10.14 was released on 2020-12-10 - upgrade-check-1.2.0 was released on 2021-02-12. - A batch of provider packages (a "sub"/plugin package of apache-airflow) were released on 2021-02-08 and again on 2021-03-08. - Backport Provider Packages (the same code as the normal provider packages, but compatible with 1.10.x series) were released on 2021-02-10 and again on 2021-03-08 ### CVEs published Airflow published four CVEs this past quarter: - CVE-2020-17513: Apache Airflow Server-Side Request Forgery (SSRF) in Charts & Query View - CVE-2021-26697: Apache Airflow: Lineage API endpoint for Experimental API missed authentication check - CVE-2021-26559: CWE-284 Improper Access Control on Configurations Endpoint for the Stable API - CVE-2020-17526: Apache Airflow Incorrect Session Validation in Airflow Webserver with default config - CVE-2020-17511: Apache Airflow Airflow admin password gets logged in plain text - CVE-2020-17515: Apache Airflow Reflected XSS via Origin Parameter ### Events: - “Airflow Virtual Meetup” was organized on 2021-01-28 - “Bangalore Airflow Virtual Meetup” was organized on 2021-02-12 - Airflow Virtual meet-up will be held on 2021-04-08 - We are working on organizing a virtual Airflow Summit 2021 conference which will be held on May 20-28 2021. The call for papers is already open and will close on April 12. ## Community Health: Despite the decrease in activity we are pretty healthy. - We created a community-managed Airflow publication on medium.com and dev.to to gather Airflow-related blog posts under one umbrella to promote the project. - We have two recurring community calls organized every fortnight: - Airflow Dev Call - Airflow Issue Triage process - Apache Airflow project started using Github Discussions and we observed a constant flow of users' questions. - We are also releasing a “Airflow Newsletter” to dev@airflow once a month. - We conducted a community survey among Airflow users and the results are here: http://airflow.apache.org/blog/airflow-survey-2020/ - In Q4 2020, there was a very large effort to make major upgrades in preparation for 2.0. This effort, while necessary, was also an unsustainable effort (lots of nights and weekends from committers/PMC members). Since 2.0 is now released, it is both reasonable and healthy for there to be a drop in the commit and PR pace (48% decrease in commits vs the previous quarter). However overal commits are still continuing at a healthy pace - 1535 commits form 199 contributors in in Q1 - We have seen a slight decrease in issues being opened (543, down 16%) possibly as a result of enabling GitHub Discussions, meaning users have an alternative means of asking for help rather than opening GitHub issues or posting on the lists.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: ## Membership Data: Apache Airflow was founded 2018-12-19 (2 years ago) There are currently 35 committers and 24 PMC members in this project. The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - No new PMC members. Last addition was Daniel Imberman on 2020-07-07. - Ryan Hamilton was added as committer on 2020-10-29 ## Project Activity: Apache Airflow 1.10.13 was released on Nov 25. Although it contains a bug related to task_concurrency and depends_on_past: Issue Link. The fix for it has been merged and 1.10.14rc1 would be cut this week and if the VOTE passes, 1.10.14 will be released early next week apache-airflow-upgrade-check 1.0.0 was released on Nov 25 too and is available for download via PIP: pip install apache-airflow-upgrade-check We have received credits from AWS to speed up our CI! [PAST EVENTS] Nov 17 — https://www.meetup.com/NYC-Apache-Airflow-Meetup/events/274289245/ [To Be Confirmed]: We’ve just closed all the Airflow Summit proceedings and we are going to donate $8,863.95 profit from Airflow Summit to the Apache Software Foundation. Many thanks to Software Guru for ]managing the conference efforts] - or something similar! 1.10.13 was released on 2020-11-25. upgrade-check-1.0.0 was released on 2020-11-25. Backport Provider Packages 2020.11.23 was released on 2020-11-24. Backport Provider Packages 2020.11.13 was released on 2020-11-13. Backport Provider Packages 2020.10.29 was released on 2020-11-02. Backport Provider Packages 2020.10.5 was released on 2020-10-06. ## Community Health: We are pretty healthy :-). The numbers speak for themselves we think: dev@airflow.apache.org had a 23% increase in traffic in the past quarter (669 emails compared to 540) 2825 commits in the past quarter (53% increase) 201 code contributors in the past quarter (-13% decrease) 1422 PRs opened on GitHub, past quarter (21% increase) 1410 PRs closed on GitHub, past quarter (21% increase) 633 issues opened on GitHub, past quarter (23% increase) 491 issues closed on GitHub, past quarter (63% increase)
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: We would like guidance from the Board about what constitutes a release when it comes to Docker images and Helm Charts. For context: https://lists.apache.org/x/thread.html/rcb608739206d788785081073a0deb417ffa9981634975fc5525dc769@%3Cdev.community.apache.org%3E The ASF release policy says: "Every ASF release must contain a source package, which must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tool" Our specific question: - Does the reproducibility of a Helm chart include the images it depends on? - Or do we count the Helm chart as "stand-alone" (it is just text files/text templates and references which docker files to use. We (Airflow PMC) weren't able to reach a conclusion ourselves https://github.com/apache/airflow/issues/10753 ## Membership Data: Apache Airflow was founded 2018-12-19 (2 years ago) There are currently 34 committers and 24 PMC members in this project. The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - Daniel Imberman was added to the PMC on 2020-07-07 - Kamil Breguła was added to the PMC on 2020-07-07 - Tomasz Urbaszek was added to the PMC on 2020-07-07 - QP Hou was added as committer on 2020-07-13 - Leah Cole was added as committer on 2020-07-20 - Ry Walker was added as committer on 2020-07-16 ## Project Activity: - 1.10.12 was released on 2020-08-25. - 1.10.11 was released on 2020-07-10. - "Backport Provider Packages" was released on 2020-06-24. (This is a early release of some components that will be available in Airflow 2.0 but available on the 1.10 series.) - Weekly dev calls for "Releasing 2.0" are happening (minutes in Confluence and on list), strong focus on releasing 2.0 this year. ## Community Health: - AirflowSummit 2020 was a huge success - with over 6,000 people registering and 3,000 watching the (virtual) sessions live.
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Airflow was founded 2018-12-19 (a year ago) There are currently 31 committers and 21 PMC members in this project. The Committer-to-PMC ratio is roughly 4:3. Community changes, past quarter: - No new PMC members. Last addition was Kengo Seki on 2019-12-01. - Jiajie Zhong was added as committer on 2020-04-03 ## Project Activity: Recent releases: 1.10.10 was released on 2020-04-09. 1.10.8 was released on 2020-02-07. 1.10.9 was released on 2020-02-07. The Airflow Summit is now scheduled for early July (6-17) and will of course be virtual. https://airflowsummit.org/ ## Community Health: The community is very healthy. We are in the process of moving from JIRA to Github issues. 868 PRS have been closed and 881 PRs have been filed. Its always hard to keep up with reviewing although this has been made a lot easier with some of the development tools that have been contributed. Mailinglists are healthy and the split between user and dev seems to work well. dev@airflow.apache.org had a 11% decrease in traffic in the past quarter (746 emails compared to 831) users@airflow.apache.org had a 64% decrease in traffic in the past quarter (79 emails compared to 214) 53 issues opened in JIRA, past quarter (-93% decrease) 129 issues closed in JIRA, past quarter (-80% decrease) 1211 commits in the past quarter (-27% decrease) 202 code contributors in the past quarter (1% increase) 881 PRs opened on GitHub, past quarter (-5% decrease) 868 PRs closed on GitHub, past quarter (-10% decrease) 602 issues opened on GitHub, past quarter (3662% increase) 344 issues closed on GitHub, past quarter (6780% increase)
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: Due to Corona virus concerns the first ever Airflow Summit was delayed until further notice. ## Membership Data: Apache Airflow was founded 2018-12-19 (a year ago) There are currently 30 committers and 21 PMC members in this project. The Committer-to-PMC ratio is roughly 5:4. Community changes, past quarter: - No new PMC members. Last addition was Kengo Seki on 2019-12-01. - Tomasz Urbaszek was added as committer on 2019-12-20 ## Project Activity: 1.10.9 was released on 2020-02-07 (quick fix release). 1.10.8 was released on 2020-02-07. 1.10.7 was released on 2019-12-24. ## Community Health: We have a very healthy community with lots of activity. Apache Airflow jobs rank 4 for data engineering jobs. Airflow's Slack channels are active. dev@airflow.apache.org had a 12% increase in traffic in the past quarter (823 emails compared to 733) users@airflow.apache.org had a 115% increase in traffic in the past quarter (198 emails compared to 92) 801 issues opened in JIRA, past quarter (6% increase) 654 issues closed in JIRA, past quarter (35% increase) 1607 commits in the past quarter (36% increase) 199 code contributors in the past quarter (11% increase) 910 PRs opened on GitHub, past quarter (7% increase) 944 PRs closed on GitHub, past quarter (12% increase)
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: ## Membership Data: Apache Airflow was founded 2018-12-19 (a year ago) There are currently 29 committers and 21 PMC members in this project. The Committer-to-PMC ratio is roughly 4:3. Community changes, past quarter: - Aizhamal Nurmamat kyzy was added to the PMC on 2019-11-21 - Kevin Yang was added to the PMC on 2019-11-21 - Jarek Potiuk was added to the PMC on 2019-10-18 - Kengo Seki was added to the PMC on 2019-12-01 - Kengo Seki was added as committer on 2019-12-01 ## Project Activity: Recent releases: 1.10.6 was released on 2019-10-28. 1.10.5 was released on 2019-09-04. 1.10.4 was released on 2019-08-06. ## Community Health: We have quite a few active discussions on Airflow Improvement Proposals (AIP) which are more and more being picked up by the community to steer large changes. Mailinglist and PRs remain very active as well. The growth of committers and PMC members is therefore very welcome. dev@airflow.apache.org had a 8% decrease in traffic in the past quarter (735 emails compared to 791) users@airflow.apache.org had a 360% increase in traffic in the past quarter (92 emails compared to 20) 751 issues opened in JIRA, past quarter (7% increase) 474 issues closed in JIRA, past quarter (no change) 1117 commits in the past quarter (no change) 176 code contributors in the past quarter (-12% decrease) 841 PRs opened on GitHub, past quarter (21% increase) 826 PRs closed on GitHub, past quarter (13% increase)
## Description: The mission of Apache Airflow is the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines ## Issues: * There are no issues requiring board attention ## Membership Data: Apache Airflow was founded 2018-12-19 (9 months ago) There are currently 28 committers and 17 PMC members in this project. The Committer-to-PMC ratio is roughly 7:5. Community changes, past quarter: - No new PMC members. Last addition was Alex Van Boxel on 2018-12-19. - Felix Uellendall was added as committer on 2019-09-05 - Chao-Han Tsai was added as committer on 2019-08-17 ## Project Activity: * Two new releases have been made since the last report * A new logo has been designed and voted upon ## Community Health: * We started the user mailing list to reduce the amount of message on dev. It is not very active yet * The slack community is very active across many channels * We have grown to 922 contributors, with 198 active in the last quarter * A ridiculous 1081 commits have been made in the past quarter
## Description:
- Apache Airflow (or simply Airflow) is a platform to programmatically
author, schedule, and monitor workflows. When workflows are defined as
code, they become more maintainable, versionable, testable, and
collaborative. Use Airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The Airflow scheduler executes your tasks on an array of
workers while following the specified dependencies. Rich command line
utilities make performing complex surgeries on DAGs a snap. The rich user
interface makes it easy to visualize pipelines running in production, monitor
progress, and troubleshoot issues when needed.
## Issues:
- there are no issues requiring board attention at this time
## Activity:
- Increasing community engagement from several companies (e.g. Astronomer,
Google)
- Airflow has been accepted as part of Google's Summer of Docs. This is
effort coordinated by Aizhamal on Airflow's side.
- We received and addressed three vulnerability reports (two were the same
issue manifesting in different ways plus one duplicate.) in the 1.10.3
release: CVE-2019-0216, CVE-2019-0229. We also received a duplicate report
after the release.
## Health report:
- We have grown from 743 to 807 contributors since our last report. Many only
submit one small fix and don't commit further.
- We have invited a 7 new committers (and confirmed) since our last report.
- 158 commits have made to the 1.10 release branch since our last release[1]
- 339 commits to master branch (which will become Airflow 2.0.0) since our
last report
- We have started planning a roadmap for Airflow 2.0
## PMC changes:
- Currently 17 PMC members.
- No new PMC members added in the last 3 months
- Last PMC addition was Alex Van Boxel on Wed Dec 19 2018
## Committer base changes:
- Currently 26 committers.
- New commmitters:
- Aizhamal Nurmamat kyzy was added as a committer on Thu May 30 2019
- Bas Harenslak was added as a committer on Mon Apr 22 2019
- Daniel Imberman was added as a committer on Tue Mar 19 2019
- Joshua Carp was added as a committer on Tue Apr 30 2019
- Kamil Breguła was added as a committer on Fri May 03 2019
- Kevin Yang was added as a committer on Tue Apr 30 2019
- Jarek Potiuk was added as a committer on Thu Apr 18 2019
## Releases:
- 1.10.3 was released on Wed Apr 10 2019
## Mailing list activity:
- users mailinglist was just opened up
- users@airflow.apache.org:
- 36 subscribers (up 36 in the last 3 months):
- 4 emails sent to list (0 in previous quarter)
- dev@airflow.apache.org:
- 598 subscribers (up 19 in the last 3 months):
- 884 emails sent to list (616 in previous quarter)
## JIRA activity:
- 644 JIRA tickets created in the last 3 months
- 507 JIRA tickets closed/resolved in the last 3 months
## Description:
- Apache Airflow (or simply Airflow) is a platform to programmatically
author, schedule, and monitor workflows. When workflows are defined as
code, they become more maintainable, versionable, testable, and
collaborative. Use Airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The Airflow scheduler executes your tasks on an array of
workers while following the specified dependencies. Rich command line
utilities make performing complex surgeries on DAGs a snap. The rich user
interface makes it easy to visualize pipelines running in production, monitor
progress, and troubleshoot issues when needed.
## Issues:
- there are no issues requiring board attention at this time
## Activity:
- New committer added: Xiaodong Deng
- Healthy discussions on the mailinglist around DAG serialization
## Health report:
- We have grown from 712 to 743 contributors since our last report. We do
monitor for potential committers, but we also see a lot of "hit and run"
contributors that particularly add to the 'periphery' of our code.
- Since latest release 1058 commits have been made
## PMC changes:
- Currently 17 PMC members.
- No new PMC members added in the last 3 months
## Committer base changes:
- Currently 19 committers.
- Xiaodong Deng was added as a committer on Sat Mar 09 2019
## Releases:
- 1.10.2 was released on Tue Jan 22 2019
## Mailing list activity:
- dev@airflow.apache.org:
- 578 subscribers (up 0 in the last 3 months):
- 665 emails sent to list (757 in previous quarter)
## JIRA activity:
- 578 JIRA tickets created in the last 3 months
- 420 JIRA tickets closed/resolved in the last 3 months
## Description:
- Apache Airflow (or simply Airflow) is a platform to programmatically
author, schedule, and monitor workflows. When workflows are defined as
code, they become more maintainable, versionable, testable, and
collaborative. Use Airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The Airflow scheduler executes your tasks on an array of
workers while following the specified dependencies. Rich command line
utilities make performing complex surgeries on DAGs a snap. The rich user
interface makes it easy to visualize pipelines running in production,
monitor progress, and troubleshoot issues when needed.
## Issues:
## Activity:
- Most of our assets have been moved to TL. We might have some remnants here
and there, which we try to track down
- We released a new version (1.10.2)
- We are in the process (or have done so) in promoting one of the
contributors to committer.
- We are, measured by GitHub stars, the most popular project for data
orchestration comparing Apache Oozie, Spotify's Luigi, Pinterest's Pinball
and LinkedIn's Azkaban
- Note: Apologies for not responding to the question by posting comments, it
was unclear how this worked
## Health report:
- We have grown from 674 to 712 contributors since our last report. We do
monitor for potential committers, but we also see a lot of "hit and run"
contributors that particularly add to the 'periphery' of our code.
- Since the 1.10.2 release 914 commits have been made
## PMC changes:
- Currently 17 PMC members.
- No new PMC members added in the last 3 months
## Committer base changes:
- Currently 18 committers.
- In process of adding one.
## Releases:
- 1.10.2 was released on Tue Jan 22 2019
## Mailing list activity:
- dev@airflow.apache.org:
- 576 subscribers (up 5 in the last 3 months):
- 660 emails sent to list (883 in previous quarter)
## JIRA activity:
- 545 JIRA tickets created in the last 3 months
- 383 JIRA tickets closed/resolved in the last 3 months
## Description:
- Apache Airflow (or simply Airflow) is a platform to programmatically
author, schedule, and monitor workflows. When workflows are defined as
code, they become more maintainable, versionable, testable, and
collaborative. Use Airflow to author workflows as directed acyclic graphs
(DAGs) of tasks. The Airflow scheduler executes your tasks on an array of
workers while following the specified dependencies. Rich command line
utilities make performing complex surgeries on DAGs a snap. The rich user
interface makes it easy to visualize pipelines running in production,
monitor progress, and troubleshoot issues when needed.
## Issues:
## Activity:
- Apache Airflow just graduated from the incubator. We are in the process of
moving our assets to top level.
## Health report:
- We have grown from 580 contributors to 674 since October 2018
- Since the November release 729 commits have been made
## PMC changes:
- Currently 17 PMC members.
- No new PMC members added in the last 3 months
## Committer base changes:
- Currently 18 committers.
- No changes (the PMC was established in the last 3 months)
## Releases:
- Latest release, v1.10.1, is from November 2018. This was during incubation.
## Mailing list activity:
- dev@airflow.apache.org:
- 578 subscribers (up 24 in the last 3 months)
## JIRA activity:
- 476 JIRA tickets created in the last 3 months
- 325 JIRA tickets closed/resolved in the last 3 months
WHEREAS, the Board of Directors deems it to be in the best interests of the Foundation and consistent with the Foundation's purpose to establish a Project Management Committee charged with the creation and maintenance of open-source software, for distribution at no charge to the public, related to workflow automation and scheduling that can be used to author and manage data pipelines. NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to be known as the "Apache Airflow Project", be and hereby is established pursuant to Bylaws of the Foundation; and be it further RESOLVED, that the Apache Airflow Project be and hereby is responsible for the creation and maintenance of software related to workflow automation and scheduling that can be used to author and manage data pipelines; and be it further RESOLVED, that the office of "Vice President, Apache Airflow" be and hereby is created, the person holding such office to serve at the direction of the Board of Directors as the chair of the Apache Airflow Project, and to have primary responsibility for management of the projects within the scope of responsibility of the Apache Airflow Project; and be it further RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Airflow Project: * Alex Guziel <saguziel@apache.org> * Alex Van Boxel <alexvanboxel@apache.org> * Arthur Wiedmer <arthur@apache.org> * Ash Berlin-Taylor <ash@apache.org> * Bolke de Bruin <bolke@apache.org> * Chris Riccomini <criccomini@apache.org> * Dan Davydov <davydov@apache.org> * Fokko Driesprong <fokko@apache.org> * Hitesh Shah <hitesh@apache.org> * Jakob Homan <jghoman@apache.org> * Jeremiah Lowin <jlowin@apache.org> * Joy Gao <joygao@apache.org> * Kaxil Naik <kaxilnaik@apache.org> * Maxime Beauchemin <maximebeauchemin@apache.org> * Siddharth Anand <sanand@apache.org> * Sumit Maheshwari <msumit@apache.org> * Tao Feng <tfeng@apache.org> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Bolke de Bruin be appointed to the office of Vice President, Apache Airflow, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed; and be it further RESOLVED, that the Apache Airflow Project be and hereby is tasked with the migration and rationalization of the Apache Incubator Airflow podling; and be it further RESOLVED, that all responsibilities pertaining to the Apache Incubator Airflow podling encumbered upon the Apache Incubator Project are hereafter discharged. Special Order 7F, Establish the Apache Airflow Project, was approved by Unanimous Vote of the directors present.
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. Once we make a release with the licensing fix we will move forward with
graduation.
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
No!
How has the community developed since the last report?
1. Since our last podling report 1 month ago (i.e. between September 4 &
October 1, inclusive), we grew our contributors from 548 to 580 (32 new
contributors)
2. Since our last podling report 1 month ago (i.e. between September 4 &
October 1, inclusive), we resolved 151 pull requests (currently at 3114
closed PRs)
3. Since being accepted into the incubator, the number of companies
officially using Apache Airflow has risen from 193 to 206, 13 new from
the last podling report 1 month ago.
How has the project developed since the last report?
See above : 151 PRs resolved, 32 new contributors, & 13 new companies
officially using it.
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2018-08-27
When were the last committers or PPMC members elected?
Tao Feng on Aug 3
Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.
One of our mentors is pretty active and helpful, one of our mentors has
never been involved, and one is in the middle.
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[x](airflow) Hitesh Shah
Comments:
[x](airflow) Jakob Homan
Comments:
IPMC/Shepherd notes:
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1.Once we make a release with the licensing fix we will move forward
with graduation.
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of? None
How has the community developed since the last report?
1. Since our last podling report 1 month ago (i.e. between August 5 &
September 4, inclusive), we grew our contributors from 512 to 548 (36 new
contributors)
2. Since our last podling report 1 month ago (i.e. between August 5 &
September 4, inclusive), we resolved 129 pull requests (currently at 2963
closed PRs)
3. Since being accepted into the incubator, the number of companies
officially
using Apache Airflow has risen from 30 to 193, 10 new from the last
podling report 1 month ago.
How has the project developed since the last report?
See above : 129 PRs resolved, 36 new contributors, & 10 new companies
officially using it. We also released 1.10.0, our 4th Apache Release while
in the incubator as seen on https://pypi.org/project/apache-airflow/ #history
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2018-08-27
When were the last committers or PPMC members elected?
Kaxil Naik on May 7 & Tao Feng on Aug 3
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[ ](airflow) Hitesh Shah
Comments:
[x](airflow) Jakob Homan
Comments:
IPMC/Shepherd notes:
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1.Once we make a release with the licensing fix we will move forward with
graduation.
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of? None
How has the community developed since the last report?
1. Since our last podling report 5 months ago (i.e. between March
28 & August 5, inclusive), we grew our contributors from 427 to 512
2. Since our last podling report 5 months ago (i.e. between March
28 & August 5, inclusive), we resolved 554 pull requests (currently at
2834 closed
PRs)
3. Since being accepted into the incubator, the number of companies
officially
using Apache Airflow has risen from 30 to 183, 34 new from the last
podling
report 5 months ago.
How has the project developed since the last report?
See above : 554 PRs resolved, 85 new contributors, & 34 new companies
officially using it. We have also added 2 new committers : Kaxil Naik on
May 7 & Tao Feng on Aug 3.
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2017-12-15
When were the last committers or PPMC members elected?
2 new committers : Kaxil Naik on May 7 & Tao Feng on Aug 3
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[ ](airflow) Hitesh Shah
Comments:
[ ](airflow) Jakob Homan
Comments:
IPMC/Shepherd notes:
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. Once we make a release with the licensing fix we will move forward with
graduation.
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
of?
None
How has the community developed since the last report?
1. Since our last podling report 1.5 months ago (i.e. between Feb 8 & March
28, inclusive), we grew our contributors from 375 to 427
2. Since our last podling report 1.5 months ago (i.e. between Feb 8 & March
28, inclusive), we resolved 152 pull requests (currently at 2280 closed
PRs)
3. Since being accepted into the incubator, the number of companies officially
using Apache Airflow has risen from 30 to 149, 17 new from the last podling
report 1.5 months ago.
How has the project developed since the last report?
See above : 152 PRs resolved, 52 new contributors, & 17 new companies
officially using it.
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2018-01-02
When were the last committers or PPMC members elected?
2017-11-30 joygao a.k.a Joy Gao (committer/PMC)
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[x](airflow) Hitesh Shah
Comments:
[x](airflow) Jakob Homan
Comments: Ready for graduation
IPMC/Shepherd notes:
I checked into the licensing issue and it is resolved, but it seems that a
release will be needed before graduation. Dave Fisher
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. We have had 4 releases and are working toward our 5th. We are getting better at releases.
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None
How has the community developed since the last report?
1. We had our forth official release 1.9.0 on Jan 2, 2018.
2. Since our last podling report 4 months ago (i.e. between Sept 25 & Feb 8,
inclusive), we grew our contributors from 315 to 385
3. Since our last podling report 4 months ago (i.e. between Sept 25 & Feb 8,
inclusive), we resolved 331 pull requests (currently at 2128 closed PRs)
4. Since being accepted into the incubator, the number of companies officially
using Apache Airflow has risen from 30 to 132, 18 new from the last podling
report.
How has the project developed since the last report?
See above : 331 PR resolved, 70 new contributors, & 18 new companies
officially using it.
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2018-01-02
When were the last committers or PPMC members elected?
2017-11-30 joygao a.k.a Joy Gao (committer/PMC)
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[x](airflow) Hitesh Shah
Comments:
[x](airflow) Jakob Homan
Comments: Ready for graduation
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. We are working on our 4th apache release 1.9.0 and are getting better at releases
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None
How has the community developed since the last report?
1. We had our third official release 1.8.2 on Aug 7th 2017.
2. Since our last podling report 3 months ago (i.e. between July
5 & Sept 25, inclusive), we grew our contributors from 280 to 315
4. Since our last podling report 3 months ago (i.e. between July
5 & Sept 25, inclusive), we resolved 167 pull requests (currently at 1797 closed
PRs)
5. Since being accepted into the incubator, the number of companies
officially using Apache Airflow has risen from 30 to 114, 16 new from the last podling report.
How has the project developed since the last report?
See above : 167 PR resolved, 35 new contributors, & 16 new companies officially using it.
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2017-08-07
When were the last committers or PPMC members elected?
2017-01-01 fokko a.k.a Fokko Driespong (committer/PMC)
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[x](airflow) Hitesh Shah
Comments:
[ ](airflow) Jakob Homan
Comments:
IPMC/Shepherd notes:
A quick look at this month's email and it looks like Airflow is ready to graduate.
Dave Fisher
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. We are working on our third apache release 1.8.2 to get more experience
with the process
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None
How has the community developed since the last report?
1. We had our second official release. 1.8.1 on May 9th 2017.
2. Since our last podling report 3 months ago (i.e. between Apr 1 and July
5, inclusive), we grew our contributors from 256 to 280
4. Since our last podling report 3 months ago (i.e. between Apr 1 and July
5, inclusive), we resolved 151 pull requests (currently at 1630 closed
PRs)
5. A new meet-up group formed in Tokyo, Japan - they held their first
meet-up on May 11.
6. Since being accepted into the incubator, the number of companies
officially using Apache Airflow has risen from 30 to 98, 15 new from
last podling report.
How has the project developed since the last report?
See above : 151 PR resolved, 24 new contributors, & 15 new companies
officially using it.
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[x] Nearing graduation
[ ] Other:
Date of last release:
2017-05-09
When were the last committers or PPMC members elected?
As mentioned on
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Mar14,2017
Alex Guziel joined the Apache Airflow PPMC/Committer group.
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[X](airflow) Hitesh Shah
Comments:
[ ](airflow) Jakob Homan
Comments:
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. We are working on our second apache release 1.8.1 to get more experience
with the process
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None
How has the community developed since the last report?
1. We had our first official release. 1.8.0 on March 19th 2017.
2. We elected 1 new PPMC Member/Committer: Alex Guziel (a.k.a saguziel)
3. Since our last podling report 3 months ago (i.e. between Jan 1 and Mar
31, inclusive), we grew our contributors from 224 to 256
4. Since our last podling report 3 months ago (i.e. between Jan 1 and Mar
31, inclusive), we resolved 216 pull requests (currently at 1479 closed
PRs)
5. Three meet-ups, one in San Francisco, CA hosted by Clover Health, one in
New York, NY hosted by Blue Apron and one in San Jose, CA hosted by PayPal
were held by the community.
6. Since being accepted into the incubator, the number of companies
officially using Apache Airflow has risen from 30 to 83.
How has the project developed since the last report?
1. As noted above, 216 pull requests were merged since our last report
(i.e. between Jan 1 and Mar 31, inclusive).
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[ ] Community building
[X] Nearing graduation
[ ] Other:
The Airflow community continues to grow, and we have successfully created
our first Apache release. We want to continue on this momentum and create
another release to solidify our process and tools around it, but we feel
we are nearing graduation. We are open to feedback and guidance to make
sure we can do so.
Date of last release:
2017-03-19
When were the last committers or PPMC members elected?
As mentioned on
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Mar14,2017
Alex Guziel joined the Apache Airflow PPMC/Committer group.
Signed-off-by:
[ ](airflow) Chris Nauroth
Comments:
[x](airflow) Hitesh Shah
Comments:
[x](airflow) Jakob Homan
Comments:
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. Getting an Apache release out
2.
3.
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None
How has the community developed since the last report?
1. We elected 1 new PPMC Member/Committer : Alex Van Boxel a.k.a.
alexvanboxel
2. Since our last podling report 3 months ago (i.e. between Oct 4 and Dec
31, inclusive), we grew our contributors from 191 to 224
3. Since our last podling report 3 months ago (i.e. between Oct 4 and Dec
31, inclusive), we resolved 202 pull requests (currently at 1263 closed
PRs)
4. One meet-up, hosted at WePay, was held by the community
5. Since being accepted into the incubator, the number of companies
officially using Apache Airflow has risen from 30 to 74
How has the project developed since the last report?
See above
Date of last release:
None. First ASF release currently being discussed.
When were the last committers or PPMC members elected?
As mentioned on
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Nov28,2016,
Alex Van Boxel joined the Apache Airflow PPMC/Committer group.
Signed-off-by:
[X](airflow) Chris Nauroth
[x](airflow) Hitesh Shah
[ ](airflow) Jakob Homan
Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines. Airflow has been incubating since 2016-03-31. Three most important issues to address in the move towards graduation: 1. Getting an Apache release out 2. 3. Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of? We discovered a private Gitter channel for committers that was created during the days when the Git project was owned by Airbnb. When the Github project’s ownership moved to the ASF, so did that of all the Gitter channels. We killed the channel - all committers left the channel and can no longer even see/discover it. How has the community developed since the last report? 1. We elected 2 new PPMC Members/Committers (Li Xuanji a.k.a. Zodiac & Sumit Maheshwari a.k.a. msumit) 2. Since our last podling report 3 months ago (i.e. between Jul 7 and Oct 4, inclusive ), we grew our contributors from 160 to 191 3. Since our last podling report 3 months ago (i.e. between Jul 7 and Oct 4, inclusive ) , we merged >142 pull requests 4. Two meet-ups (One for HUG Warsaw, the other an Airflow meet-up @Stripe HQ in SF) and several public talks (e.g. DataDaySeattle, PyData SF) were held by the community 5. Since being accepted into the incubator, the number of companies using Apache Airflow has risen from 30 to 58 How has the project developed since the last report? 1. Podling name search approved : https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-110 2. Very active discussion list, public Gitter channels and JIRA (more than 500 issues) Date of last release: None. First ASF release currently being discussed. When were the last committers or PMC members elected? As mentioned on https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements, 1. On Aug 16, Sumit Maheshwari (a.k.a. msumit) was elected to the PPMC and Committer group 2. On Aug 19, Li Xuanji (a.k.a. zodiac) was elected to the PPMC and Committer group Signed-off-by: [x](airflow) Chris Nauroth [x](airflow) Hitesh Shah [x](airflow) Jakob Homan
Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines. Airflow has been incubating since 2016-03-31. Three most important issues to address in the move towards graduation: 1. Getting an Apache release out. 2. Podling name search has to happen. 3. Verify distribution rights Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of? We've done a lot of work on distribution rights (3), but we need to wrap it up. We've been pushing out our release due to stability concerns and the need for a testing framework How has the community developed since the last report? - Since our last podling report 1 month ago, we grew our contributors from 148 to 160. - In June 2016 we had 56 commits. - We organized a meet up at WePay. We also had one before at Airbnb. - We had several committers meetings. - 1 new committer. 2 offers were given but 1 was declined. How has the project developed since the last report? - Migrated all files to contain Apache 2.0 license headers. - Replaced charting library that was non-Apache 2.0 compliant (a large effort). - Release Airflow incubator website, and added appropriate branding and disclaimers. Date of last release: None (had a non-Apache release on 2016-06-13)* * Release was discussed and approved here: https://mail-archives.apache.org/mod_mbox/incubator-general/201605.mbox/%3CEF1D08AF-80DD-4614-BC74-BBCED9251BE1@gmail.com%3E When were the last committers or PMC members elected? 2016-05-20 Signed-off-by: [X](airflow) Chris Nauroth [x](airflow) Hitesh Shah [x](airflow) Jakob Homan
Airflow is a workflow automation and scheduling system that can be used to
author and manage data pipelines.
Airflow has been incubating since 2016-03-31.
Three most important issues to address in the move towards graduation:
1. Releases
2. Grow up user and contributor communities
3. Improve and extend documentation and samples on the website
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None
How has the community developed since the last report?
* Since our last podling report 1 month ago, we grew
our contributors from 137 to 148
* Since our last podling report 1 month ago, we
accepted/merged 51 PRs
* We voted on the following matters according to Apache guidelines:
* We voted to make all current and future committers part of the PPMC
* We voted in a new committer and PPMC member : Steven Yvinec-Kruyk and
he accepted
* We voted in a commit policy of "RTC with a +1 vote from a committer
other than the author (assuming no vetos)"
How has the project developed since the last report?
* All resources have been created and all migrations are complete - code
donation included
* The website is in progress
Date of last release:
N/A - we are still working out releases as we are new to Apache.
When were the last committers or PMC members elected?
On May 20, 1 committer/PPMC member(Steven Yvinec-Kruyk) was elected and he
accepted.
Signed-off-by:
[X](airflow) Chris Nauroth
[x](airflow) Hitesh Shah
[x](airflow) Jakob Homan
Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines. Airflow has been incubating since 2016-03-31. Three most important issues to address in the move towards graduation: 1. Continued releases 2. Grow up user and contributor communities 3. Improve and extend documentation and samples on the website Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of? None How has the community developed since the last report? * Since we applied to the incubator in March 17 (proposal email), we grew our contributors from 100 to 137 * Since we applied to the incubator in March 17 (proposal email), we accepted/merged 119 PRs * Dev mailing lists activity increased How has the project developed since the last report? * All resources have been created (website, Jira, git & github mirror, ...) but not all migrations are complete * The code donation (and Pull Request migration) is in progress * The website is in progress * The previous wiki has been migrated to confluence, Jira is now used instead of the old Github issues, and the old google group is being EOLd on May 2 as users migrate to the dev mailing list Date of last release: N/A - we are still working out releases as we are new to Apache. When were the last committers or PMC members elected? N/A - we are still new to Apache and have not gotten to the point of electing new committers/PMC members. Signed-off-by: [ ](airflow) Chris Nauroth [X](airflow) Hitesh Shah [x](airflow) Jakob Homan