14:01:10 <bramwelt> #startmeeting OPNFV Release
14:01:10 <collabot> Meeting started Tue Aug  7 14:01:10 2018 UTC.  The chair is bramwelt. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:10 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:01:10 <collabot> The meeting name has been set to 'opnfv_release'
14:10:18 <bramwelt> #info Review of Release Schedule and changes
14:16:13 <bramwelt> #info Discussion on adding OPNFV release schedule to LFN
14:16:17 <bramwelt> #link https://gerrit.linuxfoundation.org/infra/#/c/11567/
14:22:01 <bramwelt> As it's just me here now, I'll be ending the meeting in ~15 minutes unless others show up.
14:23:24 <mbeierl> sorry, bramwelt, I am stuck in another meeting at the moment and cannot join zoon
14:23:27 <mbeierl> zoom even
14:24:14 <mbeierl> wanted to know where things are headed with CI/CD
14:25:39 <mbeierl> we also need to determine who owns the test results db and how that is being maintained as we are using it for release gating, but I don't know how to raise an issue with it if I need help?
14:28:17 <bramwelt> mbeierl: That's alright.
14:28:32 <mbeierl> bramwelt: sorry about that :(
14:28:50 <bramwelt> Now that the dust has settled with ZTE, I believe Serena is back and continuing to run that.
14:29:00 <bramwelt> If there are issues the best place is generally the Test-WG
14:29:19 <bramwelt> The Infra-WG works as well :)
14:29:20 <mbeierl> the test wg brought it up as a concern
14:29:29 <bramwelt> This last week?
14:29:42 * bramwelt goes to look at the minutes
14:29:44 <mbeierl> we are starting to talk about gathering an overall test document and approach and the test results db was brought up
14:29:47 <mbeierl> ya, last week
14:30:36 <mbeierl> admittedly, those of us in the meeting were not hugely familiar with the test results db and how to manage it
14:30:58 <mbeierl> and that brought up the concern that we might need to raise it as an issue
14:31:34 <bramwelt> mbeierl: So JIRA ticket for the testresults-db go in RELENG
14:31:45 <mbeierl> ok, good
14:31:51 <bramwelt> Yeah it has been an issue
14:32:24 <bramwelt> We had an agreement I think a few weeks back that we'd get more people involved with it to help aleviate that concern.
14:32:39 <bramwelt> Not sure where we stand there though given it's the Summer months.
14:33:12 <mbeierl> ok, so others have seen it as a potential gap and are looking at it, this is good
14:35:57 <bramwelt> #topic Release Process for Test Tooling
14:36:29 <bramwelt> mbeierl: Since I have you here ;) What are your thought on how we should release the testing tools (storperf, nfvbench, etc)?
14:36:45 <mbeierl> #info StorPerf has been requested to provide periodic stable releases as checkpoints
14:37:27 <mbeierl> #info a proposal is to be able to use the tag process, but on master, to publish artifacts off a stable, tagged release
14:37:51 <mbeierl> #info alternatively, we open a "gambia" branch now, and use that to cherry pick and create releases off that
14:39:40 <bramwelt> To clarify, the options your thinking of are tag release off master, then create a branch starting at that tag, or start a 'stable' branch and cherry pick things in there?
14:39:45 <mbeierl> #info so the basic ask, is how do we, as OPNFV, provide a mechanism for periodic releases, and how do we manage purging old artifacts once they are no longer relevant?
14:40:24 <bramwelt> I don't think we should be purging artifacts...unless they're only using for something intermediary
14:40:40 <mbeierl> don't need to create a branch off master - just simply have the PTL decide when master is releasable and tag that, to cause a tagged build and artifact to be produced
14:41:07 <mbeierl> if we release every 2 weeks, then when does the release from 6 months ago become irrelevant?
14:41:32 <mbeierl> do we keep all the 2 week releases forevery?
14:41:54 <bramwelt> Maybe irrelevant for the current usage, but very important when tracking down regressions.
14:42:08 <mbeierl> if we keep them, then what should the naming convention be?  By date for the release tag?
14:44:02 <bramwelt> date is fine, unless you expect to have multiple releases in a day (or perhaps need to release a hotfix the same day), but that can be defined and built into the naming scheme.
14:44:50 <mbeierl> I think we have a problem with the project if we end up having to release twice in the same day ;)
14:45:08 <bramwelt> There is a choice here to be made between time based releases (two weeks) vs. feature based releases (when it's ready)
14:45:16 <mbeierl> true
14:45:51 <bramwelt> Either approach is fine, but you have to wait the overhead of doing a release with the development work.
14:46:20 <bramwelt> If the release itself is automated, then maybe that's not such an issue.
14:47:12 <bramwelt> As an example, Jenkins does a release every two weeks, and a 'stable' release every 3 months, which contains the most stable 2-week release from the last few months.
14:47:38 <mbeierl> stable would be defined by the PTL
14:47:53 <bramwelt> That may not include all the features needed, but it's guaranteed the release is 'stable'
14:47:57 <mbeierl> and it is, like you said, when the feature is ready or on cadense
14:48:08 <bramwelt> No, I think stable should be defined by the release criteria
14:48:38 <bramwelt> in OPNFV we have these implied gates of verified, virtual, baremetal.
14:48:50 <bramwelt> with testing layered on top of them
14:49:54 <bramwelt> I think it's fine if the test projects want to define their own release schedule, as long as they stick to their criteria and are doing some sort of promotion in terms of expected stablity
14:50:09 <mbeierl> ok, then how do we define CI/CD?
14:50:22 <bramwelt> That's the question. :)
14:50:59 <bramwelt> So for storperf, you have things like 'does the container build', 'does the container run', 'does the run complete successfully'
14:51:28 <mbeierl> especially as the testing projects don't necessarily have the same level of gating.  Are we able to define gates for testing projects?
14:51:41 <bramwelt> Do you have any stricter requirements you think should be placed on the runs of the project?
14:51:50 <mbeierl> Right now, releases are based on installers and scenarios
14:52:00 <bramwelt> Right
14:52:14 <mbeierl> Projects should have their own code coverage/unit tests/sanity tests
14:52:39 <mbeierl> but I don't know how we can have a general OPNFV statement that is applicable to each testing project
14:53:10 <mbeierl> NFVbench, StorPerf and VSPerf all measure performance.  What makes a performance project "releasable"?
14:53:12 <bramwelt> I don't think we can either, especially if projects want to maintain their own release cycle, etc/
14:53:39 <bramwelt> mbeierl: I have no clue. :)
14:53:59 <bramwelt> And I don't think you want me to define that for you either, right?
14:54:14 <mbeierl> which, perhaps, brings us back to the point that it is at the PTL's discretion when their project is considered 'stable' and they leverage a tag or some release gate built into OPNFV to publish their artifacts
14:55:07 <mbeierl> and then, OPNFV also needs to decide, do these projects then issue a main Gambia release that is considered even more stable (LTS like?) at the end of the OPNFV community release cycle?
14:55:47 <bramwelt> I don't think so, unless projects want to go and define what their 'release support' looks like.
14:55:55 <mbeierl> I am ok with deciding when to create a "stable" release, but I also like the idea of having a longer term artifact that I can point people back at for regression
14:55:56 <bramwelt> and define a backporting process, etc.
14:56:32 <mbeierl> which is why I brought up the point of removing the interim artifacts - there is only 1 stable, and it keeps moving forward
14:56:45 <mbeierl> but, that is only one idea and others need to weigh in on this
14:57:01 <bramwelt> One point I'd like to make is that publishing an artifact should not be the end of the process.
14:57:22 <mbeierl> please elaborate
14:57:23 <bramwelt> If you don't go and verify the published thing can be downloaded, installed, run, then it's not actually stable.
14:57:28 <mbeierl> correct
14:57:46 <mbeierl> unfortunately, we don't do that for any of the published artifacts right now, afaik.
14:58:15 <bramwelt> Right, but I think that's an important criteria whatever artifact project deem 'stable' should meet.
14:58:26 <mbeierl> I agree.
14:59:07 <mbeierl> where is this conversation going to be documented and taken to the community?  Start a wiki, etherpad...?
14:59:10 <bramwelt> Yes, definintely fine removing the non-released artifacts (assuming there is no downstream of your project), as long as we're not removing previously released versions :)
14:59:51 <bramwelt> I'm going to start a thread on the list soon so we can summarize some of the point here and get more input.
15:00:14 <mbeierl> ok, sounds good, and thanks for your help today.  Sorry I had to be IRC only.
15:00:25 <bramwelt> No, this is totally fine. :)
15:00:32 <mbeierl> cool!
15:00:40 <bramwelt> To summarize, I think the process should be like this.
15:02:18 <bramwelt> Releng/Release Manager define a template projects use to specify their release process / timeline / promotion criteria. Projects fill this out and submit it for review. Release ensures some specific criteria are met (promotions, etc). approves, then works with the project to hold them accountable.
15:03:37 <bramwelt> Some groups works to define what these minimum requirements are for projects to meet for being a release and this is approved by the TSC and reevaluated as neeeded.
15:03:50 <bramwelt> Maybe that's too much tough, I'm not sure.
15:04:22 <mbeierl> sounds like a good start, though
15:05:27 <bramwelt> Cool. :) I just want to ensure projects aren't just pushing things out to push things out. There needs to be some accountability, some requirements, while also being as light touch as possible.
15:06:14 <bramwelt> K, I'll follow up on the list here later today.
15:06:21 <mbeierl> sounds good, thanks
15:06:26 <bramwelt> Thanks for talking, Mark!
15:06:40 <bramwelt> #endmeeting