15:00:28 <uli-k_> #startmeeting Octopus weekly meeting
15:00:28 <collabot`> Meeting started Mon Dec 21 15:00:28 2015 UTC.  The chair is uli-k_. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:28 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:00:28 <collabot`> The meeting name has been set to 'octopus_weekly_meeting'
15:01:00 <uli-k_> Hi, welcome to octopus meeting!
15:01:13 <uli-k_> #topic roll call
15:01:34 <fdegir> #info Fatih Degirmenci
15:01:45 <uli-k_> #info Uli Kleber
15:02:17 <uli-k_> #chair fdegir
15:02:17 <collabot`> Current chairs: fdegir uli-k_
15:03:35 <uli-k_> nobody else here?
15:04:00 <jose_lausuch> I am spying :)
15:04:33 <fdegir> people are already on holiday
15:04:46 <uli-k_> like me....
15:05:18 <uli-k_> I am happy about that etherpad, Frank started. But it shows more gaps to solve than clear plan.
15:05:26 <trozet> #info Tim Rozet
15:05:26 <trozet> im here
15:05:26 <uli-k_> I mean it shows clear plan.
15:05:50 <uli-k_> and by the plan, it becomes more clear what's missing....
15:06:17 <uli-k_> Hi Tim!
15:07:15 <uli-k_> #link https://etherpad.opnfv.org/p/steps_to_brahmaputra
15:08:30 <uli-k_> However most of the unclear parts are "outside octopus". We cannot do much if these things don't move.
15:08:44 <fdegir> too many things that could go wrong
15:08:50 <trozet> hi uli-k
15:08:54 <fdegir> or not even happen
15:08:56 <uli-k_> right.
15:09:46 <uli-k_> Is there any news about the PODs?
15:10:16 <fdegir> Ericsson POD2 is operational - enabling jobs to run on it
15:10:20 <fdegir> Intel PODs are not
15:10:36 <fdegir> and LF POD1 is not either
15:10:53 <fdegir> so 4 PODs are operational, 4 not
15:11:14 <fdegir> and really
15:11:36 <fdegir> this makes even simple stuff almost impossible
15:12:36 <fdegir> if we delay branching to release date, then we can perhaps do something
15:12:46 <fdegir> if the PODs are not ready by January 7th
15:12:46 <uli-k_> agree. I think we cannot really branch out, before we don't have some POD in addition to 4 PODs for the stable branch.
15:13:07 <fdegir> but then again
15:13:11 <fdegir> we have no POD for joid
15:13:37 <fdegir> according to current plan since both PODs reserved for joid come from Intel
15:14:55 <uli-k_> The question is, whether it would help if we start deciding to leave things out of Brahmaputra.
15:15:49 <uli-k_> How stable is Joid?
15:16:08 <fdegir> I think we should not move stuff out of the release because the community failed to provide resources
15:16:10 <fdegir> it's not fair
15:16:21 <fdegir> joid seems to be deploying fine
15:16:22 <uli-k_> Understand.
15:16:34 <fdegir> 4 node opnfv
15:16:36 <uli-k_> ok. so it is stable enough.
15:16:53 <fdegir> jose_lausuch might enlighten us regarding the test results
15:17:06 <fdegir> jose_lausuch: how functest looks for joid?
15:17:35 <jose_lausuch> so
15:17:39 <uli-k_> How stable is our clean-up script thing to mix installers on one POD?
15:17:51 <fdegir> do we have cleanup script? :D
15:17:56 <jose_lausuch> we were succesful to run functest, but with a lot of errors I can remember
15:18:00 <jose_lausuch> from Morgan
15:18:17 <fdegir> the script was for foreman and fuel and for LF POD
15:18:23 <fdegir> foreman is not there anymore
15:18:29 <uli-k_> Could we do 2 Installer streams on 2 PODs (each 2 installers)?
15:18:34 <jose_lausuch> in R1, the "cleanup" script in LF PODs messed up a lot the network and we had only 2 installers
15:18:37 <fdegir> so I have doubts regarding if we call it the script
15:18:53 <fdegir> uli-k_: what about the scenarions/combinations?
15:18:59 <jose_lausuch> but we could try again that
15:19:06 <uli-k_> Could we do 2 Installer streams on 2 PODs (each 2 installers)?
15:19:28 <fdegir> uli-k_: same question; what about the combinations?
15:19:31 <uli-k_> If it was a clean-up script, it should clean-up - independent of the mess it is cleaning up
15:19:43 <fdegir> uli-k_: what scenario should we run?
15:19:54 <fdegir> we couldn't even get a base scenario in order to set things up
15:20:41 <fdegir> if we go with installer x + controller a
15:20:52 <fdegir> what controller b and c will say
15:21:03 <fdegir> or similar thing for features is valid as well
15:21:31 <fdegir> I believe our 8 POD plan is a bit challenging
15:22:01 <fdegir> to run stuff properly
15:22:18 <uli-k_> So we are back to the first etherpad when we tried to identify the matrix of controllers and features.
15:22:30 <uli-k_> and we would need even more than 8 PODs.
15:22:34 <uli-k_> ?
15:22:43 <fdegir> I'm not saying we need more PODs
15:23:01 <fdegir> I'm saying we will learn how good is having 8 PODs to run things
15:23:21 <uli-k_> :D
15:23:29 <fdegir> cause
15:23:40 <fdegir> people will continue working and cherrypicking until the release
15:24:01 <fdegir> and in order for us to protect brahmaputra branch, we can't let them put unverified stuff to brahmaputra branch
15:24:08 <fdegir> so you have 4 PODs
15:24:14 <fdegir> and 4 for stable branch
15:24:21 <fdegir> if you reduce this number by half
15:24:36 <fdegir> it can barely support the automated runs
15:24:49 <fdegir> where people will do the leftover work or bugfixing?
15:25:03 <fdegir> if you skip creating the branch until the release date
15:25:14 <fdegir> we will block people who want to start working on next release
15:25:37 <fdegir> and make ourselves open to last minute tooling, development surprises since everything goes to same branch
15:26:45 <uli-k_> True.
15:26:58 * fdegir felt bad to talk negative
15:27:33 <fdegir> each sentence is longer and longer, looks nice :D
15:27:37 <uli-k_> no problem! Last time we pushed something and something moved..... Let's do it again!
15:27:38 <jose_lausuch> are there many people working for C release?
15:27:55 <fdegir> jose_lausuch: we don't know
15:27:57 <uli-k_> I try to find out, whether there is something we can tell the TSC tomorrow.
15:28:17 <fdegir> sorry to say but we failed with certain things
15:28:22 <fdegir> we have zero visibilirt
15:28:25 <fdegir> visibility
15:28:42 <uli-k_> Yes.
15:28:55 <fdegir> who does what, who needs what
15:28:58 <uli-k_> But nobody has a good view on what is reallyhappening.
15:29:01 <fdegir> and on top of this, all these delays
15:29:22 <fdegir> yep
15:29:28 <trozet> sorry got disconnected
15:29:29 <fdegir> learning
15:30:00 <fdegir> anyway
15:30:01 <uli-k_> What happens, if Octopus reports red for milestone E (or earlier)?
15:30:48 <uli-k_> Since we cannot kick features or combinations out at the moment,
15:30:50 <fdegir> basically we will get questions which you already typed answers for
15:30:57 <fdegir> - delay branching
15:31:00 <fdegir> - share PODs
15:31:15 <uli-k_> would it help if we ask them these questions?
15:32:03 <fdegir> it would help
15:32:13 <uli-k_> Would it help if we tell them we need a decision?
15:32:14 <fdegir> at the same time
15:32:17 <fdegir> makes things harder
15:32:27 <fdegir> but
15:32:30 <uli-k_> Then let's do that rather earlier than later.
15:32:36 <fdegir> now, we clearly specified our requirements
15:32:46 <fdegir> and we have this time based release
15:33:15 <fdegir> changed my mind, not typing more
15:33:29 <fdegir> uli-k_: please raise this issue during TSC
15:33:41 <uli-k_> I will.
15:33:44 <fdegir> uli-k_: with plan B, C, or whatever we have
15:34:02 <uli-k_> So plan A is on etherpad.
15:34:10 <uli-k_> And we know it will not happen.
15:34:44 <uli-k_> So what is plan B or C? Is it enough to ask them about sharing PODs and delay branching?
15:34:48 <trozet> uli-k, fdegir: guys shouldnt this be decided by relops?
15:35:01 <fdegir> release operations you mean?
15:35:05 <fdegir> or ?
15:35:07 <trozet> yeah
15:35:12 <fdegir> yeah
15:35:19 <fdegir> we say we have these things missing
15:35:41 <fdegir> I should really stop
15:35:58 <uli-k_> It's not happening atrelops
15:36:05 <trozet> fdegir: I mean i think we are all in the same boat right? not enough time but release date wont change :)
15:36:13 <fdegir> docs get more time than the actual stuff
15:37:17 <uli-k_> We tried to run OPNFV without planning. Just bottom up.
15:37:19 <fdegir> trozet: we're all in the same boat and we've been trying to say we need oars
15:37:44 <uli-k_> :D
15:37:44 <fdegir> trozet: but instead we talk which color the boat should be :D
15:37:57 <trozet> fdegir: :) yeah but its probably not going to change given the current circumstances
15:38:03 <fdegir> yes
15:38:05 <trozet> fdegir: lets just try to get what we can working, for the release
15:38:22 <fdegir> so when you say relops should decide this, I say it won't happen
15:38:25 <uli-k_> WOuld you have an idea for plan B?
15:38:27 <fdegir> we have to bring this to tsc
15:38:50 <fdegir> uli-k_: I still think we should branch off earlier than the release date
15:38:53 <trozet> fdegir: i think focus right now at least for us is what can we do and what is the top priority - so for Apex its getting virtual and baremetal deployments wiht parity to Arno running in CI
15:38:59 <fdegir> so we can stretch thing
15:39:17 <trozet> fdegir: so we are close I think to having liberty+lithium running in the same fashion that we did in Arno, so thats what im focused on
15:39:29 <fdegir> say we branch of January 19th
15:39:35 <fdegir> this could be the plan B
15:39:42 <fdegir> Plan A: etherpad
15:40:08 <fdegir> Plan B: delay branching by 2 weeks, leaving 2 weeks for stabilizing the brahmaputra and final touches
15:40:43 <trozet> fdegir: but what im saying is planning doesnt change anything at this point
15:40:53 <uli-k_> All other micromilestons as on etherpad? (test/doc stuff)?
15:41:04 <trozet> fdegir: we know what needs to be done at least between installers and CI, functest, we are rushing to get it done
15:41:14 <fdegir> trozet: it is not about what you can develop
15:41:32 <fdegir> it is about how the stuff can be tested, troubleshooted, reported, fixes provided, restested
15:41:51 <uli-k_> trozet: I think the problem is that Brahmaputra > Liberty+Lithium
15:42:30 <fdegir> trozet: by delaying branch date; we basically reduce the no of PODs we need
15:42:40 <fdegir> so we don't run stuff against 2 branches
15:42:57 <trozet> fdegir: yeah we should hold off on pulling the branch as long as possible
15:43:05 <fdegir> as I said, it's really stretching since test activities and your activities will happen on the same POD
15:43:12 <trozet> fdegir: liek you said pulling early just hurts us time wise
15:43:27 <trozet> fdegir: and nobody is going to be developing C release in jan :)
15:43:45 <fdegir> trozet: I was the one who put that date to etherpad
15:43:59 <fdegir> trozet: and I didn't put it there out of the blue
15:44:14 <fdegir> trozet: there was a question during one of the release management meetings
15:44:40 <fdegir> but anyway
15:44:44 <trozet> fdegir: ok
15:45:19 <fdegir> and trozet; I agree with you, it reduces the overhead you might get by branching off early
15:45:27 <fdegir> but opens us to issues
15:46:47 <trozet> fdegir: what kind of issues?
15:47:26 <fdegir> trozet: the stability of the master branch won't be good enough
15:47:59 <fdegir> trozet: causing issues for testing - it might fail deploying causing loss of hours and delay in testing
15:49:09 <uli-k_> We have to tell people that everybody MUST focus on stability now.
15:49:29 <uli-k_> But telling usually doesn't help really much..... :(
15:50:08 <jose_lausuch> uli-k_: there are a lot of projects still developing the test cases...
15:50:34 <uli-k_> Yes. we have to reduce new work.
15:51:04 <uli-k_> we have 5 weeks to the release. There cannot be new test cases.
15:51:07 <jose_lausuch> its not actually new work, it is defined in the DB already, so it has been "defined" since milestone D
15:51:15 <trozet> fdegir: so what makes you think stability is better in the stable branch than the master ;)
15:51:16 <jose_lausuch> or C, I dont know now
15:51:28 <trozet> fdegir: who is going to gate code so that they are only fixes to stable :) ?
15:51:31 <fdegir> trozet: that's my hope
15:52:17 <uli-k_> If we don't branch, we have no means to enforce Milestone E's meaning.
15:52:19 <fdegir> trozet: not me obviously
15:52:39 <fdegir> trozet: but at least in this scenario; have branch and run CI for both branches
15:52:45 <debrascott> jose_lausuch: who is still developing test cases? Are these new or fleshing out details on test cases that were in jira already?
15:52:45 <fdegir> trozet: we can at least have feedback to you
15:53:00 <trozet> fdegir: right normally release ops and CI would have to approve patches at that point
15:53:02 <fdegir> trozet: so you can judge yourself based on facts not just the feeling
15:53:20 <trozet> fdegir: but thats not goign to happen, so really its just a good faith notion that people will only submit minor fixes
15:53:22 <fdegir> trozet: that's what's in uli-k_ proposal already
15:53:25 <fdegir> trozet: won't ahppen
15:53:33 <jose_lausuch> debrascott: they were declared already, with description and so on. But they are not "finished" ,or at least not all of them
15:53:44 <jose_lausuch> for some of them I am not sure if they are even started
15:54:06 <frankbrockners> from past experience I tend to agree with trozet
15:54:10 <fdegir> obviously if we do things properly, policing is unncecessary
15:54:13 <debrascott> jose_lausuch: OK. understand
15:54:38 <trozet> so I would say the only difference between master and stable/b is just the name
15:54:47 <fdegir> nope trozet
15:55:02 <trozet> and you just tell people to not develop and only submit fixes, and hope that they do the right thing
15:55:05 <uli-k_> So even if we don't branch, we must enforce code freeze
15:55:08 <trozet> since you cannot police it
15:55:20 <fdegir> this is the normal way of working
15:55:30 <fdegir> you must trust people and expect them to do what is proper
15:55:41 <fdegir> but at the same time, you enable them to make proper judgement
15:56:02 <fdegir> if we go to master branch
15:56:07 <fdegir> noone will police there either
15:56:10 <frankbrockners> fdegir - per our prior discussion - unless you have a policeman, you won't enforce behavior (remember arno)
15:56:11 <fdegir> and things will be a lot worse
15:56:32 <fdegir> since people will have problems due to continuing development work
15:56:40 <fdegir> frankbrockners: I didn't say we police stuff
15:57:01 <uli-k_> frankbrockners has a good point. we don't have the policeman - even if we branch out....
15:57:02 <fdegir> frankbrockners: what I said, having branches and giving feedback on these branches will reduce the no of possible problems
15:57:24 <fdegir> it is trozet who said release ops and CI would have to approve patches
15:57:27 <trozet> fdegir: thats just my opinon
15:57:33 <fdegir> which I've never done in my entire career
15:57:42 <fdegir> who am I to approve people's work
15:58:04 <frankbrockners> fdegir: I agree in principle, but I'm not sure whether we get that behavior unless you have a policeman to makes sure that folks don't continue to simply merge to stable
15:58:17 <trozet> fdegir: if you trust people to do what is proper, then why not trust them to do what is proper on master?
15:58:34 <uli-k_> I think we won't find a solution here until the hour.
15:58:39 <frankbrockners> that might allow for a solution...
15:58:44 <fdegir> trozet: trusting people won't mean you to not improve things
15:59:37 <uli-k_> I think we have to close,so we give the necessary time to bgs.
15:59:43 <fdegir> yep
16:00:01 <fdegir> thanks for different opinions btw
16:00:14 <uli-k_> I will point TSC to our plan A etherpad and issue the need to have a plan B.
16:00:26 <frankbrockners> if you like we can continue the chat on Genesis meeting
16:00:41 <frankbrockners> 1st thing on the agenda is to see what scenarios we have so far..
16:00:45 <frankbrockners> https://global.gotomeeting.com/join/713520677
16:01:19 <uli-k_> frankbrockners: should I #endmeet so you can start meetbot?
16:01:30 <uli-k_> then we split the discussion to two files
16:01:51 <frankbrockners> yes please
16:01:59 <uli-k_> #endmeeting