Community-led development "The Apache Way"
This was extracted (@ 2021-01-20 20:10) from a list of minutes
which have been approved by the Board.
Please Note
The Board typically approves the minutes of the previous meeting at the
beginning of every Board meeting; therefore, the list below does not
normally contain details from the minutes of the most recent Board meeting.
Meeting times vary, the exact schedule is available to ASF Members and Officers, search for "calendar" in the Foundation's private index page (svn:foundation/private-index.html).
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3. Create another release
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
* Attracted new contributions
* Actively involved in addressing the items in the Apache Project
Maturity model
* Applied the project for a global-scale streaming system
### How has the project developed since the last report?
* Release 0.3 in progress
* Improved streaming support in Nemo
* Added combiners in streaming
* Added in-process data communication optimization
* Made Nemo run on geo-distributed datacenters
* Dynamic task scheduling in progress
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [ ] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-03-06 (Release 0.2)
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments: There is another release in flight as this report is
prepared.
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create releases
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Committers continuing to improve code quality
- Dynamic task partitioning, cache spilling as part of GSoC
### How has the project developed since the last report?
- Implemented a simulator of job completion time estimation
- Designed and implemented cache spilling
- Designed dynamic task partitioning
- Reviewed the Apache Maturity model
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-03-06
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Are things falling through the cracks? If so, please list any
open issues that need to be addressed.
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
Are 3rd parties respecting and correctly using the podlings
name and brand? If not what actions has the PPMC taken to
correct this? Has the VP, Brand approved the project name?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Dave Fisher: Activity seems to be low since March. I'm not sure if this is
a concern or normal for this project to be slow during the Northern Summer.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create releases
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None
### How has the community developed since the last report?
- Two GSoC students working on dynamic task sizing
and automatic caching
- Committers continuing to improve code quality
### How has the project developed since the last report?
- Release 0.2 announced on March 6, 2020
- Implemented a simulator estimating job completion time
- Improved task, job metric collection methods
- Implemented task execution time estimators
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
2020-03-06
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create releases
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None
### How has the community developed since the last report?
* New committer added: Haeyoon Cho
* Committers actively involved in improving code quality
* Committers started to work on interesing features
such as dynamic task scheduling and offheap computation
### How has the project developed since the last report?
* Release 0.2 PPMC vote passed on March 3, 2020
* Developing a job simulator that estimates stage
execution time
* Started to work on dynamic task scheduling (DTS)
that changes task sizes to choose parallelism
better
* Developing offheap computation that executes UDF
functions with offheap data natively
* Supported Java 1.11
* Fixed code smells throughout the code base
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments: Great work on the release!
### IPMC/Shepherd notes:
Justin Mclean: It seems your release was placed in the release area
before the vote was complete. This can only be done after the vote
has finished.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create more releases
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Attracted contributors through the GSoC program and
class offering
### How has the project developed since the last report?
- Preparing for a new release
- Improved code quality by addressing many sonarcloud issues
- Stream processing features implemented but not merged yet
into the master
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
None yet.
We have a couple of candidates we would like to invite.
### Have your mentors been helpful and responsive?
Yes.
### Is the PPMC managing the podling's brand / trademarks?
Yes.
### Signed-off-by:
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create more releases
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
* Attracted new contributors through the GSoC program
* Two GSoC students finished up their projects successfully.
* Presented Nemo at Berlin Beam Summit 2019.
* Discussion with Google Beam, Shopify, Seznam,cz developers
At the Beam Summit Europe 2019, which occurred in Berlin from
June 19th to the 20th, there had been many discussions regarding
Nemo from engineers working in the industry, as well as
with engineers working with Apache Beam. There had been people
from various companies, as Shopify and Seznam.cz, facing problems
as large-shuffle and data skew, and had shown a lot of interest
in using the Apache Nemo runner. Also, in the following ApacheCon
summit happening in Vegas in September, the Apache Beam team will
demonstrate running their workloads using the Apache Nemo runner.
### How has the project developed since the last report?
* Integrate Apache Crail with Nemo
* Off-heap support to improve shuffle speed
* DirectByteBufferInput/OutputStream for
Off-heap SerializedMemoryStore
* Off-heap SerializedMemoryStore
* Off-heap memory management (reuse ByteBuffer)
* Separation of JVM heap region and off-heap memory region
* Enable Nemo single-stage programs to run on a serverless
framework, AWS Lambda
* Refactor Executor to support different types of execution
environment
* Add a Lambda Executor
* Add a compiler pass for Lambda
* Progress on a ML-based smart compiler
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
None yet.
Recent two contributors are good committer candidates.
### Have your mentors been helpful and responsive?
Yes.
### Signed-off-by:
- [X] (nemo) Davor Bonaci
Comments: Lots of good work on community building, awaiting results.
In-project activity lower than before.
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Add and improve features
3. Create more releases
### Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
### How has the community developed since the last report?
* Committers actively involved in sending PRs and doing code reviews
* GSoC students actively contributing to Nemo
* Engaging with developers in open source venues in June
: Beam Summit @ Berlin, KRnet in Korea
### How has the project developed since the last report?
* Improved and stablized Nemo pass interfaces
* Developed more formal treatment of expressing optimization
policies in Nemo
* Added a Machine-Learning-based policy (e.g., xgboost)
* Added offheap memory store and improved shuffle
* Added a lambda pass to enable running Nemo jobs in AWS Lambda
* A paper describing Nemo got accepted at USENIX ATC 2019,
which will be presented in July.
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
None yet.
### Have your mentors been helpful?
Mentors have been responsive on answering the questions the
community have had. They guided the community well on how to
engage with developers in the open source community.
### Signed-off-by:
- [X] (nemo) Davor Bonaci
Comments: The project is on track. I'm starting to be worried about
the decreased activity lately.
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community (committers, contributors, users)
2. More releases
3. Add and improve features
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively involved in sending PRs and doing code reviews
* Engaged with the Beam community to register Nemo as an official Beam
runner
* Started to attract new contributors
* Ran a Nemo tutorial in an open-source project venue
How has the project developed since the last report?
* Donated code to ASF
* Released 0.1 (the first release!)
* Committer’s guide added
* Added Nemo Web UI
* Created an official Beam runner
* Improved metric collection and storage
* Improved skew handling pass
* Refactored runtime pass and IR DAG
* Fixed skew handling bugs
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[X] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
2018-12-31 release 0.1
When were the last committers or PPMC members elected?
None yet.
Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.
The community has been receiving much help from our mentors!
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: Good progress; nothing to add. Focus on community building
needed going forward.
[X](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[ ](nemo) Jean-Baptiste Onofre
Comments:
[X](nemo) Markus Weimer
Comments:
IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community: add new contributors/committers
2. Grow the user base
3. Develop a clear roadmap for graduation
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively sent PRs and did code reviews
* Committers actively involved in mailing lists
* Committers started to interact with other communities
How has the project developed since the last report?
* Created the first release candidate (in the process of voting)
* Code donation to ASF
* Added Beam stream support
* Created a Beam Nemo runner
* Added Nemo Web UI
* Hardened the Nemo runtime (scheduling, data transfer, etc.)
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[X] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
In the process of voting for the first release.
When were the last committers or PPMC members elected?
None yet.
Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.
Yes. Our mentors have been helpful and responsive.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: the podling is doing really well; first release to come
shortly.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create the first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively sent PRs and did code reviews
* Committers actively involved in mailing lists
* Getting contributors including a GSoC student
How has the project developed since the last report?
* Made great progress in every aspect of the project
* Working on the first release Almost done with the features to be
included in the first release
* Solidified runtime by refactoring how to handle
tasks, stages, and jobs, and scheduling policies
* Rule-based policy support in the compiler
* Composition of passes
* Added Multiple DAG submission support in a user
program
* Added Beam 2.6, Beam SQL support
* Spark RDD caching support
* Task cloning support
* Intermediate data locality aware scheduling
* Added a WebUI (contributed by a GSoC student),
which we will polish
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[X] Working towards first release
[X] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: All great; active community contributing towards the first
release.
[X](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments: Focusing on first release would be great, it would be a
great milestone for the podling
[X](nemo) Markus Weimer
Comments: Good progress all around; but it is time for a first release
:)
[ ](nemo) Reynold Xin
Comments:
IPMC/Shepherd notes:
Our mentors mostly have been helpful and responsive.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively involved in mailing lists
* Committers actively started to send PRs and do code reviews
How has the project developed since the last report?
* Progress towards supporting Spark DSL programs
* Refactoring the Nemo runtime
* Finishing up a paper that describes the design and
implementation of Nemo
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[X] Initial setup
[X] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: Progress is probably a bit slower than before, but still
very high. Podling is doing great.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[ ](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
Comments:
IPMC/Shepherd notes:
Dave Fisher: AFAICT the SGA has not been provided and a mentor has
advised that this can wait until graduation.
I don't think that is correct and it is not the same as the guidance
given to other podlings like Heron.
See
https://lists.apache.org/thread.html/df7040d27bc8c01ec1601476b1e465226d4b1ea
99d0e5991b977fd1c@%3Cprivate.nemo.apache.org%3E
Nemo is a data processing system to flexibly control the runtime behaviors of
a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively involved in mailing lists
* Committers actively started to send PRs and do code reviews
* PPMC accepted a GSoC student to work on Nemo web UI issues
How has the project developed since the last report?
* Progress towards supporting Spark DSL programs
* Optimization policy for geo-distributed data analytics
* Optimization policy for skew handling
* Working on a paper that describes the design and
implementation of Nemo
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[X] Initial setup
[X] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: Good progress; the podling is on track.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[X](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
Comments:
Nemo is a data processing system to flexibly control the runtime behaviors of
a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
of?
None.
How has the community developed since the last report?
* Committers actively involved in mailing lists
* Committers actively started to send PRs and do code reviews
* Two committers volunteered for GSoC mentors
How has the project developed since the last report?
* SonarCloud integrated
* Community standards and documentation added
* Scheduler improved by adding a new scheduling policy and making it
reactive
* Operator pipelining implemented
* Experiments performed to demonstrate the benefits of Nemo
* Blogs posted
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[x] Initial setup
[x] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: The project is very active, moving quickly,
and doing very well.
[X](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
Comments:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of?
None.
How has the community developed since the last report?
* Committers filed ICLAs.
* Committers enrolled in mailing lists
* Committers enrolled in the slack channel
* Committers used mailing lists to discuss initial setups
* Committers started to send PRs to the apache infra
How has the project developed since the last report?
* gitbox/github set up and code transferred from snuspl/nemo to apache/incubator-nemo
* JIRA created and issues transferred from github to JIRA
* web site set up
* mailing lists set up
* The name ‘Apache Nemo’ approved (PODLINGNAMESEARCH-139)
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[x] Initial setup
[ ] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: The podling is moving very quickly.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
IPMC/Shepherd notes:
johndament: Seeing good progress with on list communication.