ApacheCon is Coming 9-12 Sept. 2019 - Las Vegas The Apache Software Foundation
Apache 20th Anniversary Logo

Community-led development "The Apache Way"

Apache Support Logo

This was extracted (@ 2020-09-29 22:10) from a list of minutes which have been approved by the Board.
Please Note The Board typically approves the minutes of the previous meeting at the beginning of every Board meeting; therefore, the list below does not normally contain details from the minutes of the most recent Board meeting.

Meeting times vary, the exact schedule is available to ASF Members and Officers, search for "calendar" in the Foundation's private index page (svn:foundation/private-index.html).

Nemo

16 Sep 2020

Report was filed, but display is awaiting the approval of the Board minutes.

17 Jun 2020

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

### Three most important unfinished issues to address before graduating:

 1. Grow the community (committers, contributors, users)
 2. Create releases
 3.

### Are there any issues that the IPMC or ASF Board need to be aware of?

 None

### How has the community developed since the last report?

 - Two GSoC students working on dynamic task sizing
   and automatic caching
 - Committers continuing to improve code quality

### How has the project developed since the last report?

 - Release 0.2 announced on March 6, 2020
 - Implemented a simulator estimating job completion time
 - Improved task, job metric collection methods
 - Implemented task execution time estimators

### How would you assess the podling's maturity?
 Please feel free to add your own commentary.

 - [ ] Initial setup
 - [ ] Working towards first release
 - [X] Community building
 - [ ] Nearing graduation
 - [ ] Other:

### Date of last release:

 2020-03-06

### When were the last committers or PPMC members elected?

 February 27, 2020

### Have your mentors been helpful and responsive?

 Our mentors have been helpful and responsive.

### Is the PPMC managing the podling's brand / trademarks?
 The PPMC manages the podling's brand / trademarks.

### Signed-off-by:

 - [X] (nemo) Hyunsik Choi
    Comments:
 - [X] (nemo) Byung-Gon Chun
    Comments:
 - [X] (nemo) Jean-Baptiste Onofre
    Comments:
 - [X] (nemo) Markus Weimer
    Comments:

### IPMC/Shepherd notes:

18 Mar 2020

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

### Three most important unfinished issues to address before graduating:

  1. Grow the community (committers, contributors, users)
  2. Create releases
  3.

### Are there any issues that the IPMC or ASF Board need to be aware of?

  None

### How has the community developed since the last report?

  * New committer added: Haeyoon Cho
  * Committers actively involved in improving code quality
  * Committers started to work on interesing features
    such as dynamic task scheduling and offheap computation

### How has the project developed since the last report?

  * Release 0.2 PPMC vote passed on March 3, 2020
  * Developing a job simulator that estimates stage
    execution time
  * Started to work on dynamic task scheduling (DTS)
    that changes task sizes to choose parallelism
    better
  * Developing offheap computation that executes UDF
    functions with offheap data natively
  * Supported Java 1.11
  * Fixed code smells throughout the code base

### How would you assess the podling's maturity?
Please feel free to add your own commentary.

  - [ ] Initial setup
  - [ ] Working towards first release
  - [X] Community building
  - [ ] Nearing graduation
  - [ ] Other:

### Date of last release:

  Release 0.1 on Dec. 31, 2018

### When were the last committers or PPMC members elected?

  February 27, 2020

### Have your mentors been helpful and responsive?

  Our mentors have been helpful and responsive.

### Is the PPMC managing the podling's brand / trademarks?

 The PPMC manages the podling's brand / trademarks.

### Signed-off-by:

  - [ ] (nemo) Hyunsik Choi
     Comments:
  - [X] (nemo) Byung-Gon Chun
     Comments:
  - [X] (nemo) Jean-Baptiste Onofre
     Comments:
  - [X] (nemo) Markus Weimer
     Comments:  Great work on the release!

### IPMC/Shepherd notes:
  Justin Mclean: It seems your release was placed in the release area
  before the vote was complete. This can only be done after the vote
  has finished.

18 Dec 2019

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

### Three most important unfinished issues to address before graduating:

 1. Grow the community (committers, contributors, users)
 2. Create more releases

### Are there any issues that the IPMC or ASF Board need to be aware of?

 None.

### How has the community developed since the last report?

 - Attracted contributors through the GSoC program and
   class offering

### How has the project developed since the last report?

 - Preparing for a new release
 - Improved code quality by addressing many sonarcloud issues
 - Stream processing features implemented but not merged yet
   into the master

### How would you assess the podling's maturity?
Please feel free to add your own commentary.

 - [ ] Initial setup
 - [ ] Working towards first release
 - [X] Community building
 - [ ] Nearing graduation
 - [ ] Other:

### Date of last release:

 Release 0.1 on Dec. 31, 2018

### When were the last committers or PPMC members elected?

 None yet.
 We have a couple of candidates we would like to invite.

### Have your mentors been helpful and responsive?

 Yes.

### Is the PPMC managing the podling's brand / trademarks?

 Yes.

### Signed-off-by:

 - [X] (nemo) Hyunsik Choi
    Comments:
 - [X] (nemo) Byung-Gon Chun
    Comments:
 - [ ] (nemo) Jean-Baptiste Onofre
    Comments:
 - [X] (nemo) Markus Weimer
    Comments:

### IPMC/Shepherd notes:

18 Sep 2019

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

### Three most important unfinished issues to address before graduating:

 1. Grow the community (committers, contributors, users)
 2. Create more releases

### Are there any issues that the IPMC or ASF Board need to be aware of?

 None.

### How has the community developed since the last report?

 * Attracted new contributors through the GSoC program
 * Two GSoC students finished up their projects successfully.
 * Presented Nemo at Berlin Beam Summit 2019.
 * Discussion with Google Beam, Shopify, Seznam,cz developers
  At the Beam Summit Europe 2019, which occurred in Berlin from
  June 19th to the 20th, there had been many discussions regarding
  Nemo from engineers working in the industry, as well as
  with engineers working with Apache Beam. There had been people
  from various companies, as Shopify and Seznam.cz, facing problems
  as large-shuffle and data skew, and had shown a lot of interest
  in using the Apache Nemo runner. Also, in the following ApacheCon
  summit happening in Vegas in September, the Apache Beam team will
  demonstrate running their workloads using the Apache Nemo runner.

### How has the project developed since the last report?

 * Integrate Apache Crail with Nemo
 * Off-heap support to improve shuffle speed
   * DirectByteBufferInput/OutputStream for
     Off-heap SerializedMemoryStore
   * Off-heap SerializedMemoryStore
   * Off-heap memory management (reuse ByteBuffer)
   * Separation of JVM heap region and off-heap memory region
 * Enable Nemo single-stage programs to run on a serverless
   framework, AWS Lambda
   * Refactor Executor to support different types of execution
     environment
   * Add a Lambda Executor
   * Add a compiler pass for Lambda
 * Progress on a ML-based smart compiler

### How would you assess the podling's maturity?
Please feel free to add your own commentary.

 - [ ] Initial setup
 - [ ] Working towards first release
 - [X] Community building
 - [ ] Nearing graduation
 - [ ] Other:

### Date of last release:

 Release 0.1 on Dec. 31, 2018

### When were the last committers or PPMC members elected?

 None yet.
 Recent two contributors are good committer candidates.

### Have your mentors been helpful and responsive?

 Yes.

### Signed-off-by:

 - [X] (nemo) Davor Bonaci
    Comments: Lots of good work on community building, awaiting results.
    In-project activity lower than before.
 - [X] (nemo) Hyunsik Choi
    Comments:
 - [X] (nemo) Byung-Gon Chun
    Comments:
 - [ ] (nemo) Jean-Baptiste Onofre
    Comments:
 - [X] (nemo) Markus Weimer
    Comments:

### IPMC/Shepherd notes:

19 Jun 2019

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

### Three most important unfinished issues to address before graduating:

 1. Grow the community (committers, contributors, users)
 2. Add and improve features
 3. Create more releases

### Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

 None.

### How has the community developed since the last report?

 * Committers actively involved in sending PRs and doing code reviews
 * GSoC students actively contributing to Nemo
 * Engaging with developers in open source venues in June
   : Beam Summit @ Berlin, KRnet in Korea

### How has the project developed since the last report?

 * Improved and stablized Nemo pass interfaces
 * Developed more formal treatment of expressing optimization
   policies in Nemo
 * Added a Machine-Learning-based policy (e.g., xgboost)
 * Added offheap memory store and improved shuffle
 * Added a lambda pass to enable running Nemo jobs in AWS Lambda
 * A paper describing Nemo got accepted at USENIX ATC 2019,
   which will be presented in July.

### How would you assess the podling's maturity?
Please feel free to add your own commentary.

 - [ ] Initial setup
 - [ ] Working towards first release
 - [X] Community building
 - [ ] Nearing graduation
 - [ ] Other:

### Date of last release:

 Release 0.1 on Dec. 31, 2018

### When were the last committers or PPMC members elected?

 None yet.

### Have your mentors been helpful?

 Mentors have been responsive on answering the questions the
 community have had. They guided the community well on how to
 engage with developers in the open source community.

### Signed-off-by:

 - [X] (nemo) Davor Bonaci
    Comments: The project is on track. I'm starting to be worried about
    the decreased activity lately.
 - [X] (nemo) Hyunsik Choi
    Comments:
 - [X] (nemo) Byung-Gon Chun
    Comments:
 - [X] (nemo) Jean-Baptiste Onofre
    Comments:
 - [X] (nemo) Markus Weimer
    Comments:

### IPMC/Shepherd notes:

20 Mar 2019

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

  1. Grow the community (committers, contributors, users)
  2. More releases
  3. Add and improve features

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  None.

How has the community developed since the last report?

  * Committers actively involved in sending PRs and doing code reviews
  * Engaged with the Beam community to register Nemo as an official Beam
  runner
  * Started to attract new contributors
  * Ran a Nemo tutorial in an open-source project venue

How has the project developed since the last report?

  * Donated code to ASF
  * Released 0.1 (the first release!)
  * Committer’s guide added
  * Added Nemo Web UI
  * Created an official Beam runner
  * Improved metric collection and storage
  * Improved skew handling pass
  * Refactored runtime pass and IR DAG
  * Fixed skew handling bugs

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [ ] Working towards first release
  [X] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  2018-12-31 release 0.1

When were the last committers or PPMC members elected?

  None yet.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

 The community has been receiving much help from our mentors!

Signed-off-by:

  [X](nemo) Davor Bonaci
     Comments: Good progress; nothing to add. Focus on community building
     needed going forward.
  [X](nemo) Hyunsik Choi
     Comments:
  [X](nemo) Byung-Gon Chun
     Comments:
  [ ](nemo) Jean-Baptiste Onofre
     Comments:
  [X](nemo) Markus Weimer
     Comments:

IPMC/Shepherd notes:

19 Dec 2018

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

 1. Grow the community: add new contributors/committers
 2. Grow the user base
 3. Develop a clear roadmap for graduation

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

 None.

How has the community developed since the last report?

 * Committers actively sent PRs and did code reviews
 * Committers actively involved in mailing lists
 * Committers started to interact with other communities

How has the project developed since the last report?

 * Created the first release candidate (in the process of voting)
 * Code donation to ASF
 * Added Beam stream support
 * Created a Beam Nemo runner
 * Added Nemo Web UI
 * Hardened the Nemo runtime (scheduling, data transfer, etc.)

How would you assess the podling's maturity?
Please feel free to add your own commentary.

 [ ] Initial setup
 [ ] Working towards first release
 [X] Community building
 [ ] Nearing graduation
 [ ] Other:

Date of last release:

 None yet.
 In the process of voting for the first release.

When were the last committers or PPMC members elected?

 None yet.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

 Yes. Our mentors have been helpful and responsive.

Signed-off-by:

 [X](nemo) Davor Bonaci
    Comments: the podling is doing really well; first release to come
shortly.
 [ ](nemo) Hyunsik Choi
    Comments:
 [X](nemo) Byung-Gon Chun
    Comments:
 [X](nemo) Jean-Baptiste Onofre
    Comments:
 [ ](nemo) Markus Weimer
    Comments:

IPMC/Shepherd notes:

19 Sep 2018

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

 1. Grow the community
 2. Create the first Apache release
 3. Donate code to ASF

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

 None.

How has the community developed since the last report?

 * Committers actively sent PRs and did code reviews
 * Committers actively involved in mailing lists
 * Getting contributors including a GSoC student

How has the project developed since the last report?

 * Made great progress in every aspect of the project
 * Working on the first release Almost done with the features to be
   included in the first release
 * Solidified runtime by refactoring how to handle
   tasks, stages, and jobs, and scheduling policies
 * Rule-based policy support in the compiler
 * Composition of passes
 * Added Multiple DAG submission support in a user
   program
 * Added Beam 2.6, Beam SQL support
 * Spark RDD caching support
 * Task cloning support
 * Intermediate data locality aware scheduling
 * Added a WebUI (contributed by a GSoC student),
   which we will polish

How would you assess the podling's maturity?
Please feel free to add your own commentary.

 [ ] Initial setup
 [X] Working towards first release
 [X] Community building
 [ ] Nearing graduation
 [ ] Other:

Date of last release:

 None yet.

When were the last committers or PPMC members elected?

 None yet.

Signed-off-by:

 [X](nemo) Davor Bonaci
    Comments: All great; active community contributing towards the first
    release.
 [X](nemo) Hyunsik Choi
    Comments:
 [X](nemo) Byung-Gon Chun
    Comments:
 [X](nemo) Jean-Baptiste Onofre
    Comments: Focusing on first release would be great, it would be a
    great milestone for the podling
 [X](nemo) Markus Weimer
    Comments: Good progress all around; but it is time for a first release
    :)
 [ ](nemo) Reynold Xin
    Comments:

IPMC/Shepherd notes:

 Our mentors mostly have been helpful and responsive.

20 Jun 2018

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

 1. Grow the community
 2. Create a first Apache release
 3. Donate code to ASF

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

 None.

How has the community developed since the last report?

 * Committers actively involved in mailing lists
 * Committers actively started to send PRs and do code reviews

How has the project developed since the last report?

 * Progress towards supporting Spark DSL programs
 * Refactoring the Nemo runtime
 * Finishing up a paper that describes the design and
   implementation of Nemo

How would you assess the podling's maturity?
Please feel free to add your own commentary.

 [X] Initial setup
 [X] Working towards first release
 [ ] Community building
 [ ] Nearing graduation
 [ ] Other:

Date of last release:

 None yet.

When were the last committers or PPMC members elected?

 None yet.

Signed-off-by:

 [X](nemo) Davor Bonaci
    Comments: Progress is probably a bit slower than before, but still
very high. Podling is doing great.
 [ ](nemo) Hyunsik Choi
    Comments:
 [X](nemo) Byung-Gon Chun
    Comments:
 [ ](nemo) Jean-Baptiste Onofre
    Comments:
 [ ](nemo) Markus Weimer
    Comments:
 [ ](nemo) Reynold Xin
    Comments:

IPMC/Shepherd notes:
 Dave Fisher: AFAICT the SGA has not been provided and a mentor has
 advised that this can wait until graduation.
 I don't think that is correct and it is not the same as the guidance
 given to other podlings like Heron.
 See
 https://lists.apache.org/thread.html/df7040d27bc8c01ec1601476b1e465226d4b1ea
99d0e5991b977fd1c@%3Cprivate.nemo.apache.org%3E

16 May 2018

Nemo is a data processing system to flexibly control the runtime behaviors of
a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

 1. Grow the community
 2. Create a first Apache release
 3. Donate code to ASF

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

 None.

How has the community developed since the last report?

 * Committers actively involved in mailing lists
 * Committers actively started to send PRs and do code reviews
 * PPMC accepted a GSoC student to work on Nemo web UI issues

How has the project developed since the last report?

 * Progress towards supporting Spark DSL programs
 * Optimization policy for geo-distributed data analytics
 * Optimization policy for skew handling
 * Working on a paper that describes the design and
   implementation of Nemo

How would you assess the podling's maturity?
Please feel free to add your own commentary.

 [X] Initial setup
 [X] Working towards first release
 [ ] Community building
 [ ] Nearing graduation
 [ ] Other:

Date of last release:

 None yet.

When were the last committers or PPMC members elected?

 None yet.

Signed-off-by:

 [X](nemo) Davor Bonaci
    Comments: Good progress; the podling is on track.
 [ ](nemo) Hyunsik Choi
    Comments:
 [X](nemo) Byung-Gon Chun
    Comments:
 [X](nemo) Jean-Baptiste Onofre
    Comments:
 [X](nemo) Markus Weimer
    Comments:
 [ ](nemo) Reynold Xin
    Comments:

18 Apr 2018

Nemo is a data processing system to flexibly control the runtime behaviors of
a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

 1. Grow the community
 2. Create a first Apache release
 3. Donate code to ASF

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
of?

 None.

How has the community developed since the last report?

 * Committers actively involved in mailing lists
 * Committers actively started to send PRs and do code reviews
 * Two committers volunteered for GSoC mentors

How has the project developed since the last report?

 * SonarCloud integrated
 * Community standards and documentation added
 * Scheduler improved by adding a new scheduling policy and making it
   reactive
 * Operator pipelining implemented
 * Experiments performed to demonstrate the benefits of Nemo
 * Blogs posted

How would you assess the podling's maturity?
Please feel free to add your own commentary.

 [x] Initial setup
 [x] Working towards first release
 [ ] Community building
 [ ] Nearing graduation
 [ ] Other:

Date of last release:

 None yet.

When were the last committers or PPMC members elected?

 None yet.

Signed-off-by:

 [X](nemo) Davor Bonaci
    Comments: The project is very active, moving quickly,
  and doing very well.
 [X](nemo) Hyunsik Choi
    Comments:
 [X](nemo) Byung-Gon Chun
    Comments:
 [X](nemo) Jean-Baptiste Onofre
    Comments:
 [ ](nemo) Markus Weimer
    Comments:
 [ ](nemo) Reynold Xin
    Comments:

21 Mar 2018

Nemo is a data processing system to flexibly control the runtime behaviors
of a job to adapt to varying deployment characteristics.

Nemo has been incubating since 2018-02-04.

Three most important issues to address in the move towards graduation:

  1.  Grow the community
  2.  Create a first Apache release
  3.  Donate code to ASF

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of?

  None.

How has the community developed since the last report?

  * Committers filed ICLAs.
  * Committers enrolled in mailing lists
  * Committers enrolled in the slack channel
  * Committers used mailing lists to discuss initial setups
  * Committers started to send PRs to the apache infra

How has the project developed since the last report?

  * gitbox/github set up and code transferred from snuspl/nemo to apache/incubator-nemo
  * JIRA created and issues transferred from github to JIRA
  * web site set up
  * mailing lists set up
  * The name ‘Apache Nemo’ approved (PODLINGNAMESEARCH-139)

How would you assess the podling's maturity?

Please feel free to add your own commentary.

  [x] Initial setup
  [ ] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  None yet.

When were the last committers or PPMC members elected?

 None yet.

Signed-off-by:

  [X](nemo) Davor Bonaci
     Comments: The podling is moving very quickly.
  [ ](nemo) Hyunsik Choi
     Comments:
  [X](nemo) Byung-Gon Chun
     Comments:
  [X](nemo) Jean-Baptiste Onofre
     Comments:
  [ ](nemo) Markus Weimer
     Comments:
  [ ](nemo) Reynold Xin

IPMC/Shepherd notes:

  johndament:  Seeing good progress with on list communication.