This was extracted (@ 2024-04-17 21:10) from a list of minutes
which have been approved by the Board.
Please Note
The Board typically approves the minutes of the previous meeting at the
beginning of every Board meeting; therefore, the list below does not
normally contain details from the minutes of the most recent Board meeting.
WARNING: these pages may omit some original contents of the minutes.
Meeting times vary, the exact schedule is available to ASF Members and Officers, search for "calendar" in the Foundation's private index page (svn:foundation/private-index.html).
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Motivate growth in the community
2. Address remaining items before graduation
### Are there any issues that the IPMC or ASF Board need to be aware of?
The issue raised in our previous podling report (Dec. 2023) still exists.
This is a project initiated from academia, and many of the committers have
graduated with their degree and moved
on to something else. The project itself is very promising, known that
many outstanding academic papers have been developed on top of Apache Nemo
[1-5], so it would be ideal
if it finds its purpose, however it seems quite challenging at this
particular moment. It is quite disappointing to see the project lose its
momentum during the covid-19
pandemics, which made it extremely challenging for us to build the
community. Nevertheless, the project itself, I believe, deserved to be in
the TLP, but it seems like if
there is no breakpoint for the project regarding the project community, I
think we could start discussing about sending it to the attic. Any
suggestions or opinions are welcome!
[1] https://www.usenix.org/conference/atc19/presentation/yang-youngseok
[2] https://dl.acm.org/doi/10.1145/3468144
[3] https://dl.acm.org/doi/10.1145/3064176.3064181
[4] https://dl.acm.org/doi/10.1145/3546591.3547524
[5] https://www.usenix.org/conference/atc23/presentation/song
### How has the community developed since the last report?
Trying to have a discussion towards the issue raised above
### How has the project developed since the last report?
Trying to have a discussion towards the issue raised above
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2022-09-02
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
Please refer to the issue raised above.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the polling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [ ] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Motivate growth in the community
2. Address remaining items before graduation
### Are there any issues that the IPMC or ASF Board need to be aware of?
This is a project initiated from academia, and the major problem here is
that many of us have graduated and moved on to something else now. Although
the project itself is very promising and is a well-developed system (proved
by a number of academic papers), the project is losing a bit of its
momentum, and there isn't a clear breakpoint where this can get itself back
on track in building its community and users. This project, I personally
think, is an unlucky one in the sense that Covid-19 hit at the time when it
was actively being promoted in ASF conferences and so, trying to build its
user base, and hasn't been able to climb up to TLP since then. I think that
there are enough resources for users to pick up on where it stands at the
moment, but it is unclear on how it should get back on track, where the
lack of community action being the major problem on its way to TLP. Any
suggestions or help would be nice, but we may also consider archiving the
project as well at this point.
### How has the community developed since the last report?
We have 1 new contributor.
### How has the project developed since the last report?
Discussions for community growth.
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2022-09-02
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
Please refer to the issue raised above.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [ ] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Motivate growth in the community
2. Address remaining items before graduation
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- We have published a paper, Sponge, at ATC 2023 conference
(https://www.usenix.org/conference/atc23/presentation/song) and promoted
Nemo in the publication in order to grow our community and work on further
potential projects based on Nemo.
### How has the project developed since the last report?
- We have published a paper on supporting features regarding offloading
bottleneck tasks to serverless frameworks with Nemo in ATC 2023.
- Introducing Nemo to researchers that were interested in the publication
through the academic conference.
- Discussions for community growth.
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2022-09-02
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
- The mentors have been helpful.
### Is the PPMC managing the podling's brand / trademarks?
- The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [ ] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
PJ Fanning: Activity on the mailing lists is quite low. It's difficult to
see if there is progress towards graduation. The podling has been
in incubation for more than 5 years.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address remaining items before graduation
2. Motivate growth in the community
3. A new release after finalizing new features in our codebase
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- In order to develop the community, we are working on reinforcing the
documentations for better usability with visual aids.
- We have also integrated Disqus on our website for better communication
with the users.
### How has the project developed since the last report?
- Discussions for graduation
- Finalizing the code cleanup and documentations for new features and
filing existing PRs (e.g., for supporting features regarding offloading
bottleneck tasks to serverless frameworks)
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2022-09-02
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
- The mentors have been helpful.
### Is the PPMC managing the podling's brand / trademarks
- The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofré
Comments:
- [ ] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address remaining items before graduation
2. Motivate growth in the community
3. A new release after finishing handling of a few existing PRs
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- We have communicated, interviewed, and have selected GSoC students in
order to grow our community and work on potential projects based on Nemo.
### How has the project developed since the last report?
- Discussions for graduation
- Organizing code for filing PRs for past work (e.g., for supporting
features regarding offloading bottleneck tasks to serverless frameworks)
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2022-09-02
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
- The mentors have been helpful.
### Is the PPMC managing the podling's brand / trademarks?
- The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [ ] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors of a job to adapt to varying deployment characteristics. Nemo has been incubating since 2018-02-04. ### Three most important unfinished issues to address before graduating: 1. Address a few remaining items before graduation 2. 3. ### Are there any issues that the IPMC or ASF Board need to be aware of? None. ### How has the community developed since the last report? - Hosted a GSoC student successfully - Community more involved for graduation ### How has the project developed since the last report? - Discussed and voted for graduation - Migrated CI to GitHub Actions - Updated the Nemo website to meet community standards - Organized and reviewed pending PRs - Streaming support in wide-area networks ### How would you assess the podling's maturity? Please feel free to add your own commentary. - [ ] Initial setup - [ ] Working towards first release - [ ] Community building - [X] Nearing graduation - [ ] Other: ### Date of last release: 2022-09-09 ### When were the last committers or PPMC members elected? September 28, 2021 ### Have your mentors been helpful and responsive? Are things falling through the cracks? If so, please list any open issues that need to be addressed. The mentors have been helpful. ### Is the PPMC managing the podling's brand / trademarks? Are 3rd parties respecting and correctly using the podlings name and brand? If not what actions has the PPMC taken to correct this? Has the VP, Brand approved the project name? The PPMC manages the podling's brand / trademarks. ### Signed-off-by: - [ ] (nemo) Hyunsik Choi Comments: - [X] (nemo) Byung-Gon Chun Comments: - [X] (nemo) Jean-Baptiste Onofré Comments: - [ ] (nemo) Markus Weimer Comments: ### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Complete the remaining items in the Apache Maturity model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Hosting a GSoC student working on stream processing
- Working on a new release
### How has the project developed since the last report?
- Added installation scripts for ease of use
- Improved stream processing metrics
- Added additional stream application examples
- Added network profiling scripts
- Updated library versions
- Upgraded the CI with new platforms
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
Are things falling through the cracks? If so, please list any
open issues that need to be addressed.
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
Are 3rd parties respecting and correctly using the podlings
name and brand? If not what actions has the PPMC taken to
correct this? Has the VP, Brand approved the project name?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Accepted a GSoC student to contribute to Nemo
### How has the project developed since the last report?
- Added stream processing experiment examples
- Explored supporting stream processing on serverless
- Improved support for WAN stream processing
- Published a paper on stream processing built on Nemo
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [ ] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
* Accepting GSoC students to contribute to Nemo
### How has the project developed since the last report?
* Updated library versions
* Added fault tolerance for stream operators
* Added latency measurement improvement for stream
* Added automatic stream input rate throttling
* Added new stream workload examples
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity Model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Successfully hosted three GSoC students to contribute to Nemo
- New committer added: Hwarim Hyun
### How has the project developed since the last report?
- Added stream processing metrics
- Added stream processing applications
- Added hierarchical aggregation
- Explored various stream processing scheduling policies
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
September 28, 2021
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofré
Comments:
- [X] (nemo) Markus Weimer
Comments: It might be time for anothe release.
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Address the items in the Apache Project Maturity model
- Hosted three GSoC students to contribute to Nemo successfully
### How has the project developed since the last report?
- Continued to develop (wide-area) streaming support
- Developing partial aggregation
- Developing how to partition a task into subtasks
- Worked on a basic Nemo simulator
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [ ] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofré
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Address the items in the Apache Project Maturity model
- Hosting three GSoC students to contribute to Nemo
### How has the project developed since the last report?
- Continued development to support (wide-area) batch and stream
processing
- Work on a simulator to predict job completion time
- Work on work stealing to improve straggler handling
- Work on stream processing adaptation with reconfiguration
and fidelity control
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [ ] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Are things falling through the cracks? If so, please list any
open issues that need to be addressed.
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
Are 3rd parties respecting and correctly using the podlings
name and brand? If not what actions has the PPMC taken to
correct this? Has the VP, Brand approved the project name?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofré
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Address the items in the Apache Project Maturity model
- Attracted new contributors
### How has the project developed since the last report?
- Created a new release
- Continued to improve stream processing support
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [ ] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-12-07
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Address a few remaining items in the Apache Project Maturity model
2. Grow the community
3. Create another release
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
* Attracted new contributions
* Actively involved in addressing the items in the Apache Project
Maturity model
* Applied the project for a global-scale streaming system
### How has the project developed since the last report?
* Release 0.3 in progress
* Improved streaming support in Nemo
* Added combiners in streaming
* Added in-process data communication optimization
* Made Nemo run on geo-distributed datacenters
* Dynamic task scheduling in progress
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [ ] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-03-06 (Release 0.2)
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments: There is another release in flight as this report is
prepared.
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create releases
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Committers continuing to improve code quality
- Dynamic task partitioning, cache spilling as part of GSoC
### How has the project developed since the last report?
- Implemented a simulator of job completion time estimation
- Designed and implemented cache spilling
- Designed dynamic task partitioning
- Reviewed the Apache Maturity model
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [X] Nearing graduation
- [ ] Other:
### Date of last release:
2020-03-06
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Are things falling through the cracks? If so, please list any
open issues that need to be addressed.
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
Are 3rd parties respecting and correctly using the podlings
name and brand? If not what actions has the PPMC taken to
correct this? Has the VP, Brand approved the project name?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Dave Fisher: Activity seems to be low since March. I'm not sure if this is
a concern or normal for this project to be slow during the Northern Summer.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create releases
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None
### How has the community developed since the last report?
- Two GSoC students working on dynamic task sizing
and automatic caching
- Committers continuing to improve code quality
### How has the project developed since the last report?
- Release 0.2 announced on March 6, 2020
- Implemented a simulator estimating job completion time
- Improved task, job metric collection methods
- Implemented task execution time estimators
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
2020-03-06
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create releases
3.
### Are there any issues that the IPMC or ASF Board need to be aware of?
None
### How has the community developed since the last report?
* New committer added: Haeyoon Cho
* Committers actively involved in improving code quality
* Committers started to work on interesing features
such as dynamic task scheduling and offheap computation
### How has the project developed since the last report?
* Release 0.2 PPMC vote passed on March 3, 2020
* Developing a job simulator that estimates stage
execution time
* Started to work on dynamic task scheduling (DTS)
that changes task sizes to choose parallelism
better
* Developing offheap computation that executes UDF
functions with offheap data natively
* Supported Java 1.11
* Fixed code smells throughout the code base
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
February 27, 2020
### Have your mentors been helpful and responsive?
Our mentors have been helpful and responsive.
### Is the PPMC managing the podling's brand / trademarks?
The PPMC manages the podling's brand / trademarks.
### Signed-off-by:
- [ ] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments: Great work on the release!
### IPMC/Shepherd notes:
Justin Mclean: It seems your release was placed in the release area
before the vote was complete. This can only be done after the vote
has finished.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create more releases
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
- Attracted contributors through the GSoC program and
class offering
### How has the project developed since the last report?
- Preparing for a new release
- Improved code quality by addressing many sonarcloud issues
- Stream processing features implemented but not merged yet
into the master
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
None yet.
We have a couple of candidates we would like to invite.
### Have your mentors been helpful and responsive?
Yes.
### Is the PPMC managing the podling's brand / trademarks?
Yes.
### Signed-off-by:
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Create more releases
### Are there any issues that the IPMC or ASF Board need to be aware of?
None.
### How has the community developed since the last report?
* Attracted new contributors through the GSoC program
* Two GSoC students finished up their projects successfully.
* Presented Nemo at Berlin Beam Summit 2019.
* Discussion with Google Beam, Shopify, Seznam,cz developers
At the Beam Summit Europe 2019, which occurred in Berlin from
June 19th to the 20th, there had been many discussions regarding
Nemo from engineers working in the industry, as well as
with engineers working with Apache Beam. There had been people
from various companies, as Shopify and Seznam.cz, facing problems
as large-shuffle and data skew, and had shown a lot of interest
in using the Apache Nemo runner. Also, in the following ApacheCon
summit happening in Vegas in September, the Apache Beam team will
demonstrate running their workloads using the Apache Nemo runner.
### How has the project developed since the last report?
* Integrate Apache Crail with Nemo
* Off-heap support to improve shuffle speed
* DirectByteBufferInput/OutputStream for
Off-heap SerializedMemoryStore
* Off-heap SerializedMemoryStore
* Off-heap memory management (reuse ByteBuffer)
* Separation of JVM heap region and off-heap memory region
* Enable Nemo single-stage programs to run on a serverless
framework, AWS Lambda
* Refactor Executor to support different types of execution
environment
* Add a Lambda Executor
* Add a compiler pass for Lambda
* Progress on a ML-based smart compiler
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
None yet.
Recent two contributors are good committer candidates.
### Have your mentors been helpful and responsive?
Yes.
### Signed-off-by:
- [X] (nemo) Davor Bonaci
Comments: Lots of good work on community building, awaiting results.
In-project activity lower than before.
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [ ] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
### Three most important unfinished issues to address before graduating:
1. Grow the community (committers, contributors, users)
2. Add and improve features
3. Create more releases
### Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
### How has the community developed since the last report?
* Committers actively involved in sending PRs and doing code reviews
* GSoC students actively contributing to Nemo
* Engaging with developers in open source venues in June
: Beam Summit @ Berlin, KRnet in Korea
### How has the project developed since the last report?
* Improved and stablized Nemo pass interfaces
* Developed more formal treatment of expressing optimization
policies in Nemo
* Added a Machine-Learning-based policy (e.g., xgboost)
* Added offheap memory store and improved shuffle
* Added a lambda pass to enable running Nemo jobs in AWS Lambda
* A paper describing Nemo got accepted at USENIX ATC 2019,
which will be presented in July.
### How would you assess the podling's maturity?
Please feel free to add your own commentary.
- [ ] Initial setup
- [ ] Working towards first release
- [X] Community building
- [ ] Nearing graduation
- [ ] Other:
### Date of last release:
Release 0.1 on Dec. 31, 2018
### When were the last committers or PPMC members elected?
None yet.
### Have your mentors been helpful?
Mentors have been responsive on answering the questions the
community have had. They guided the community well on how to
engage with developers in the open source community.
### Signed-off-by:
- [X] (nemo) Davor Bonaci
Comments: The project is on track. I'm starting to be worried about
the decreased activity lately.
- [X] (nemo) Hyunsik Choi
Comments:
- [X] (nemo) Byung-Gon Chun
Comments:
- [X] (nemo) Jean-Baptiste Onofre
Comments:
- [X] (nemo) Markus Weimer
Comments:
### IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community (committers, contributors, users)
2. More releases
3. Add and improve features
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively involved in sending PRs and doing code reviews
* Engaged with the Beam community to register Nemo as an official Beam
runner
* Started to attract new contributors
* Ran a Nemo tutorial in an open-source project venue
How has the project developed since the last report?
* Donated code to ASF
* Released 0.1 (the first release!)
* Committer’s guide added
* Added Nemo Web UI
* Created an official Beam runner
* Improved metric collection and storage
* Improved skew handling pass
* Refactored runtime pass and IR DAG
* Fixed skew handling bugs
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[X] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
2018-12-31 release 0.1
When were the last committers or PPMC members elected?
None yet.
Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.
The community has been receiving much help from our mentors!
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: Good progress; nothing to add. Focus on community building
needed going forward.
[X](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[ ](nemo) Jean-Baptiste Onofre
Comments:
[X](nemo) Markus Weimer
Comments:
IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community: add new contributors/committers
2. Grow the user base
3. Develop a clear roadmap for graduation
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively sent PRs and did code reviews
* Committers actively involved in mailing lists
* Committers started to interact with other communities
How has the project developed since the last report?
* Created the first release candidate (in the process of voting)
* Code donation to ASF
* Added Beam stream support
* Created a Beam Nemo runner
* Added Nemo Web UI
* Hardened the Nemo runtime (scheduling, data transfer, etc.)
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[ ] Working towards first release
[X] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
In the process of voting for the first release.
When were the last committers or PPMC members elected?
None yet.
Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.
Yes. Our mentors have been helpful and responsive.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: the podling is doing really well; first release to come
shortly.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
IPMC/Shepherd notes:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create the first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively sent PRs and did code reviews
* Committers actively involved in mailing lists
* Getting contributors including a GSoC student
How has the project developed since the last report?
* Made great progress in every aspect of the project
* Working on the first release Almost done with the features to be
included in the first release
* Solidified runtime by refactoring how to handle
tasks, stages, and jobs, and scheduling policies
* Rule-based policy support in the compiler
* Composition of passes
* Added Multiple DAG submission support in a user
program
* Added Beam 2.6, Beam SQL support
* Spark RDD caching support
* Task cloning support
* Intermediate data locality aware scheduling
* Added a WebUI (contributed by a GSoC student),
which we will polish
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[ ] Initial setup
[X] Working towards first release
[X] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: All great; active community contributing towards the first
release.
[X](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments: Focusing on first release would be great, it would be a
great milestone for the podling
[X](nemo) Markus Weimer
Comments: Good progress all around; but it is time for a first release
:)
[ ](nemo) Reynold Xin
Comments:
IPMC/Shepherd notes:
Our mentors mostly have been helpful and responsive.
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively involved in mailing lists
* Committers actively started to send PRs and do code reviews
How has the project developed since the last report?
* Progress towards supporting Spark DSL programs
* Refactoring the Nemo runtime
* Finishing up a paper that describes the design and
implementation of Nemo
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[X] Initial setup
[X] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: Progress is probably a bit slower than before, but still
very high. Podling is doing great.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[ ](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
Comments:
IPMC/Shepherd notes:
Dave Fisher: AFAICT the SGA has not been provided and a mentor has
advised that this can wait until graduation.
I don't think that is correct and it is not the same as the guidance
given to other podlings like Heron.
See
https://lists.apache.org/thread.html/df7040d27bc8c01ec1601476b1e465226d4b1ea
99d0e5991b977fd1c@%3Cprivate.nemo.apache.org%3E
Nemo is a data processing system to flexibly control the runtime behaviors of
a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?
None.
How has the community developed since the last report?
* Committers actively involved in mailing lists
* Committers actively started to send PRs and do code reviews
* PPMC accepted a GSoC student to work on Nemo web UI issues
How has the project developed since the last report?
* Progress towards supporting Spark DSL programs
* Optimization policy for geo-distributed data analytics
* Optimization policy for skew handling
* Working on a paper that describes the design and
implementation of Nemo
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[X] Initial setup
[X] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: Good progress; the podling is on track.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[X](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
Comments:
Nemo is a data processing system to flexibly control the runtime behaviors of
a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
of?
None.
How has the community developed since the last report?
* Committers actively involved in mailing lists
* Committers actively started to send PRs and do code reviews
* Two committers volunteered for GSoC mentors
How has the project developed since the last report?
* SonarCloud integrated
* Community standards and documentation added
* Scheduler improved by adding a new scheduling policy and making it
reactive
* Operator pipelining implemented
* Experiments performed to demonstrate the benefits of Nemo
* Blogs posted
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[x] Initial setup
[x] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: The project is very active, moving quickly,
and doing very well.
[X](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
Comments:
Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.
Nemo has been incubating since 2018-02-04.
Three most important issues to address in the move towards graduation:
1. Grow the community
2. Create a first Apache release
3. Donate code to ASF
Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of?
None.
How has the community developed since the last report?
* Committers filed ICLAs.
* Committers enrolled in mailing lists
* Committers enrolled in the slack channel
* Committers used mailing lists to discuss initial setups
* Committers started to send PRs to the apache infra
How has the project developed since the last report?
* gitbox/github set up and code transferred from snuspl/nemo to apache/incubator-nemo
* JIRA created and issues transferred from github to JIRA
* web site set up
* mailing lists set up
* The name ‘Apache Nemo’ approved (PODLINGNAMESEARCH-139)
How would you assess the podling's maturity?
Please feel free to add your own commentary.
[x] Initial setup
[ ] Working towards first release
[ ] Community building
[ ] Nearing graduation
[ ] Other:
Date of last release:
None yet.
When were the last committers or PPMC members elected?
None yet.
Signed-off-by:
[X](nemo) Davor Bonaci
Comments: The podling is moving very quickly.
[ ](nemo) Hyunsik Choi
Comments:
[X](nemo) Byung-Gon Chun
Comments:
[X](nemo) Jean-Baptiste Onofre
Comments:
[ ](nemo) Markus Weimer
Comments:
[ ](nemo) Reynold Xin
IPMC/Shepherd notes:
johndament: Seeing good progress with on list communication.