14:31:14 #startmeeting release daily sync 14:31:14 Meeting started Fri Sep 26 14:31:14 2014 UTC. The chair is gzhao. Information about MeetBot at http://ci.openstack.org/meetbot.html. 14:31:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:31:14 The meeting name has been set to 'release_daily_sync' 14:31:28 #topic roll call 14:31:30 #info tbachman for groupbasedpolicy 14:31:46 #info gzhao 14:33:02 #info gershon for defense4all 14:34:06 edwarnicke: colindixon_ are you around? 14:34:32 #info colindixon for things 14:34:37 #chair tbachman colindixon edwarnicke 14:34:37 Current chairs: colindixon edwarnicke gzhao tbachman 14:34:47 gzhao: I’m not sure ed’s around today - I think he’s travelling? 14:35:12 tbachman: i.c 14:35:41 although I did just see a patch from him — maybe airline wifi :o 14:36:00 #info Phil Robb 14:36:09 he also verified a couple of bugs 14:36:22 #topic blocking issue 14:37:04 #info we currently reduced to only one blocking issues 2047 blocking vtn that is still outstanding 14:38:25 Need verify queue also reduced by almost half, and most likely 2091 and 2089 are the same cause 14:38:49 gzhao: we should probably re-link the spreadsheet — do you have that handy? 14:39:08 #link https://docs.google.com/spreadsheets/d/1TRYposNDFPaKcySlvwkOXvfR6Anx2EFujlIjoTthhRY/edit#gid=670022163 14:39:31 #undo 14:39:31 Removing item from minutes: 14:39:43 #link https://docs.google.com/spreadsheets/d/1TRYposNDFPaKcySlvwkOXvfR6Anx2EFujlIjoTthhRY/edit#gid=670022163 spreadsheet tracking oustanding blocking issues 14:39:46 gzhao: We also have one more fix for 2049 14:39:49 Still being tested 14:39:51 thanks tykeal 14:39:54 #info gzhao says that the only remaining blocker is 2047 14:39:55 thanks tbachman I mean 14:39:57 Added to spreadsheet 14:39:59 sorry tykeal, ignore that 14:40:04 so, what about 2074? 14:40:10 #undo 14:40:10 Removing item from minutes: 14:40:12 colindixon: I looked at it briefly 14:40:14 * regXboi is around but talking to rovarga about bgpcep jenkins 14:40:19 edwarnicke: are you in CA already? 14:40:19 colindixon: It looks like a bundle start order bug perhaps 14:40:20 #info gzhao says that the only remaining blocker is 2074 14:40:20 I recruited Giovanni, but I don’t know if he’ll be able to kill it dead 14:40:34 colindixon: It looks subtle 14:40:41 #undo 14:40:41 Removing item from minutes: 14:40:51 edwarnicke: fair enough 14:40:57 colindixon: We may have a bundle load order issue that Felix DM can't prevent :( 14:40:59 * colindixon watches the meetbot battle unfold 14:41:07 lol 14:41:12 tbachman: edwarnicke just added 2049 14:41:14 * tbachman waits for gzhao’s next move ;) 14:41:16 edwarnicke: would a Thread.sleep work? 14:41:24 colindixon: Disasterously 14:41:26 * regXboi wonders if we could coin a new show.... MDM - meetbot death match :) 14:41:32 #info gzhao says that the only remaining blockers are 2049 and 2074 14:41:38 colindixon: Because it would be sleeping the Felix DM thread 14:41:49 Startup would probably slow to a crawl 14:41:50 edwarnicke: I see 14:41:57 which issue is this... 2074? 14:42:01 regXboi: yes 14:42:06 * regXboi pads off 14:42:16 edwarnicke: so, is it easy to tell if it’s good or broken when it comes up? 14:42:29 edwarnicke: we could give users instructions on how to tell and they could relaunch? 14:42:40 um 14:42:42 colindixon: If you want 14:42:44 wait 14:42:46 colindixon: Try the sleep 14:42:52 See if it screws up felixDM 14:43:00 edwarnicke: I don’t know where to put it 14:43:01 edwarnicke: why isn't this a variation of what I found on 2007? 14:43:02 And startup speed 14:43:15 colindixon: Any chance Madhu_offline could work on this? He'd be good at it 14:43:16 edwarnicke: but I’ll see if I can get to it today (unlikely) 14:43:23 regXboi: Might be 14:43:28 regXboi: Actually 14:43:29 regXboi: Well 14:43:30 edwarnicke: we can try, I think he might be burned out 14:43:31 Madhu_offline will likely be offline for a bit longer 14:43:33 Yes 14:43:39 But only sort of 14:43:45 ? 14:43:45 The NSF intedependencies are a snarl 14:43:55 he should be on in a couple of hours is my guess 14:43:56 that's being..... polite ..... 14:43:59 The interdepend on each other in loops 14:44:40 I was hoping we wouldn't need to undo the snarl before Helium 14:44:44 it looks like no such luck 14:44:56 * regXboi goes and looks for detangler 14:45:02 * tbachman goes back to catching up minutes 14:45:15 #info gzhao says verify queue also reduced by almost half, and most likely 2091 and 2089 are the same cause 14:45:44 #info GiovanniMeo looking into 2074, but it’s a tricky problem, as it involves working out Felix DM issues 14:46:10 #info regXboi asks if this is related to BUG 2007 14:46:29 thanks tbachman 14:46:47 #info colindixon wonders if this can be addressed with a Thread.sleep(), and if so, where it should be placed 14:46:49 colindixon: np! 14:47:44 #info possible last ditch solutions: (i) a long thread.sleep() somewhere, (ii) instructions for users on how to tell if a boot of VTN is good or bad so they can reboot if not, (iii) a watchdog script that does (ii) automatically 14:48:23 ok, we’re good on bugs otherwise? 14:48:25 Please note: This is a random bundle start order bug.. it looks like its been here for over a year 14:48:33 We are just less lucky in hitting it right now 14:48:44 2049 needs to be verified and merged 14:48:51 edwarnicke: I can verify 14:49:06 edwarnicke: you think Madhu_offline will be a good choice to take a look at it? 14:49:12 tbachman: Thank you 14:49:17 edwarnicke: np! 14:49:18 tbachman: Your testing is deeply welcome :) 14:49:23 lol 14:49:30 tbachman: Jan will also be doing scale testing with it 14:49:40 edwarnicke: just to make sure, verification is just that the log messages are no longer there? 14:49:52 LuisGomez: Could you run the integration tests against https://git.opendaylight.org/gerrit/#/c/11598/4 14:49:54 tbachman: I can assist, meeting with Jan this morning. 14:50:00 alagalah_: sounds good :) 14:50:21 Bugfix for 2049 and 2101 should be same 14:50:25 #info tbachman and jmedved to verify 2049 14:50:32 tbachman: Not quite... turns out the exception was a canary in the coal mine for a hash collision... this should fix that collosion 14:50:36 #info ttkacik says 2049 and 2101 are the same bug 14:51:01 #info ttkacik notes bug fix for 2049 and 2101 should be the same 14:51:06 #undo 14:51:06 Removing item from minutes: 14:51:11 edwarnicke: ACK. Just trying to figure out what I do to verify… i.e. if I don’t see the log messages, likely that it’s not happening 14:51:25 * regXboi watches the flying meetbot 14:51:31 (note: I’d get the log messages every time I run my app) 14:51:38 Both are hash collisions (hash used instead of proper key) 14:52:01 ttkacik: thx! :) 14:52:20 thats vaclav demcak who fixed it 14:52:28 thx both then :) 14:52:38 (messenger and fixer) 14:53:25 Guys I am going to largely be away from IRC today, Sat, Sun... please text me at 919-621-0958 in case of serious need 14:53:44 edwarnicke: understood 14:53:48 edwarnicke: are you still going to be the person to triger the “final” release? 14:53:57 tbachman: Yes, txt me 14:54:04 okay 14:54:05 When we are ready 14:54:18 gzhao, colindixon, and I can work with you on that 14:54:23 (and any others :) ) 14:54:27 tbachman: you rock 14:54:38 lol 14:54:47 colindixon: community r0cks :) 14:54:49 I will be similarly off IRC come about 1p pacific today (text me at 301.455.5083 as well) 14:55:01 I’ll be on IRC 14:55:05 unrelated to edwarnicke, we are not plotting 14:55:08 but tomorrow will be tricky (on and off) 14:55:12 and am travelling Sunday 14:55:17 it’s my 5th wedding anniversary 14:55:18 colindixon: lol 14:55:20 wow! 14:55:26 so... I can be around if needed 14:55:30 how nice of them to schedule the summit around it ;) 14:55:34 do this mean we will do final cut on Monday? 14:55:46 gzhao: would be good for us to have time to regression test 14:55:57 gzhao: I'd like to do an RC before then so we have time to test it 14:56:03 gzhao: over the weekend 14:56:09 edwarnicke: ok 14:56:15 I will be around tomorrow, and working with tbachman on #2049 and suspected related #2101 14:56:31 edwarnicke: shall we push every time then select final one 14:56:36 If we can get things in, I'd like to cut sometime today 14:56:55 LuisGomez: will you be around to run some regression test over the weekend? 14:56:57 gzhao: Something like that 14:56:59 bug 2098? 14:57:22 edwarnicke, gzhao: exactly? 14:57:31 are we still cutting the every 4-5 hour releases? 14:57:48 * colindixon notes we still need to talk docs and (maybe) downloader 14:58:07 colindixon: we run build every 4-5 hours, but don't push artifacts for every build 14:58:19 Apologies guys, I have to go now :( 14:58:31 edwarnicke: later! 14:58:33 edwarnicke: go, we’ll be fine :-) 14:58:33 edwarnicke: see you on monday 14:58:57 safe travels edwarnicke 14:59:34 https://bugs.opendaylight.org/show_bug.cgi?id=2098 14:59:42 gzhao: are you tracking that? 14:59:52 filed by VTN as blocker 15:01:11 colindixon: missed it 15:01:27 I just marked 2101 as a duplicate of 2049 15:02:10 tbachman: do we know that it is? we still need to verify that 2101 is fixed 15:03:37 according to ttkacik it’s the same 15:03:56 ok, still Jan shoudl verify both are fixed 15:04:04 colindixon: good plan 15:04:45 gzhao: we need somebody to merge https://git.opendaylight.org/gerrit/#/c/11598 15:04:51 gzhao, i will be available over the weekend 15:04:55 * tbachman waits for gerrit 11598 to finish 15:05:07 it doesn't look like 2074 is a easy fix, is there a walk around for it? 15:05:10 gzhao: can you ping the list of coontroller committers I sent yesterday who have MD-SAL experience 15:05:13 LuisGomez: thanks 15:05:17 Jan and ed were reviewers — does anyone know how to reach Jan? 15:05:32 tbachman: e-mail him, but he’s not a comitter 15:05:34 I can try via email — just curious if anyone has closer contacts 15:05:35 k 15:05:37 ah 15:05:48 i will meet jan this afternoon 15:06:03 For MD-SAL, I'd add: 15:06:03 Devin, Ed, Tony and TomP as reviewers to get their attention 15:06:10 I’ll do it 15:06:43 colindixon: thx! 15:07:06 LuisGomez: Jan has quite a few bugs to verify 15:07:06 done 15:07:18 gzhao, LuisGomez: he can farm out 15:07:42 are there any special setups involved (VMs, HW switches, etc.)? 15:08:20 tbachman: no idea 15:08:37 gzhao: in regards to cutting the release, we should consult LuisGomez about regression test time 15:08:50 tbachman: yes 15:08:53 LuisGomez: how long does a complete regression take, typically (assuming no bugs of course ;) ) 15:09:03 gzhao: I need you to ping Jan or somebody about 2098 15:09:29 2098 seems like there may be some code to write there :( 15:09:34 dunno 15:09:41 colindixon: I can ping him, but the only way for me is email 15:10:12 tbachman, around 1 hour to pass all tests we have 15:10:20 wow! 15:10:22 nice :) 15:11:58 I started a build now, should include all the merged fix. I will start another one once change 11598 is merged. and others if necessary 15:12:09 gzhao: which merged fix? 15:12:33 tbachman: fixes on the to be verified section. 15:12:53 tbachman: we may make VTN fix 2098, I’ve been putting comments on the bug 15:12:59 is regXboi still around? 15:13:03 colindixon: thx 15:13:05 ? 15:13:09 but somebody other than me needs to manage it 15:13:12 but 1860 15:13:17 mentions something about permgen on windows 15:13:22 somebody looking for me? 15:13:22 were you our windows test bed ;)? 15:13:36 * regXboi pads off to look at 1860 15:13:47 gerrit 11337 15:13:56 gzhao: I #info about ways to try to fix it 15:14:01 this is resolved fixed 15:14:12 regXboi: yeah — we need the verify 15:14:21 (i.e. someone needs to *prove* it :) ) 15:14:22 colindixon: great, thanks 15:14:29 * tbachman wonders if we have something that can do that 15:14:29 oh yeah that 15:14:32 lol 15:14:33 you have 50% of it 15:14:43 of the verify? 15:14:47 I can verify that the gerrit changes the environment 15:14:51 ah 15:14:54 #info colindixon is working a fix for 2098 15:14:58 I can't verify that the gerrit provides enough memory 15:15:11 because I can't run with that much allocated in my windows VM 15:15:15 ah 15:15:19 so... 15:15:24 can we just run it on windows? 15:15:25 yeah it does what it set out to do 15:15:37 and yeah that's the amount set aside in linux 15:15:41 * tbachman has a windows laptop lying around somewhere 15:15:43 so I believe that verifies it 15:15:54 but if you want 100% .... 15:16:05 100% is good :) 15:16:13 #undo 15:16:13 Removing item from minutes: 15:16:15 I am not 15:16:30 * tbachman has been burnt before on things looking correct until used ;) 15:16:31 let me see what I can do ... hold on 15:16:37 regXboi: awesome — thx! 15:16:39 * regXboi goes off to mess with his VM settings 15:16:40 #info colindixon has added extra info and help for others to work on 2074 and 2098 (he has no time) 15:16:42 lol 15:16:48 * regXboi notes we need more windows people 15:16:57 regXboi: RobDolin? 15:17:07 um 15:17:15 I was thinking more "boots on the ground" 15:17:18 lol 15:17:31 he might be able to find boots on the ground 15:17:46 regXboi: is it just a matter of running this on a windows machine? 15:17:54 pretty much yes 15:17:54 If so — I can probably do that :) 15:17:58 okay 15:18:07 If I can start it on windows and get it to run with 11337 we are good 15:18:09 colindixon: saw your comments. 15:18:15 don't need to load anything 15:18:33 * regXboi shuts down windows VM 15:18:44 regXboi: again, thx bunches! 15:19:04 what's 6G - 6144M? 15:19:38 * tbachman notes we started this IRC meeting with only 1 blocker, and are now up to 4 15:19:41 regXboi: AKC 15:19:42 ACK 15:19:55 * regXboi regXboi throws another 2G into the windows VM and figures that *ought* to be enough :) 15:20:03 :) 15:20:09 * regXboi then mutters "famous last words" 15:20:16 lol 15:20:42 we still on for SFC review in 10? 15:20:52 okay — so I’m verifying 2049 is fixed once it’s merged and built, Jan will verify 2101 then? 15:21:03 regXboi: that was what we scheduled 15:21:07 thx 15:21:17 what;s the latest RC distro URL? 15:22:10 regXboi: let me get it 15:22:17 thx 15:22:33 regXboi: you’re verifying 1860? 15:22:40 colindixon: yes 15:22:42 it might be here: http://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.2.0-SNAPSHOT/ 15:22:46 or at least trying to again 15:22:50 regXboi: good 15:22:52 http://nexus.opendaylight.org/content/repositories/opendaylight.daily/org/opendaylight/integration/distribution-karaf/0.2.0-Helium-RC2-test-v201409252113/ 15:22:56 um 15:23:03 gzhao: is the snapshot more recent? 15:23:20 snapshot has Fri Sep 26 11:37:30 UTC 2014 15:23:23 shouldn't we update the HE simultaneous relase plan with the Karaf only release vehicle in the list? 15:23:26 we don’t know who needs to verify 2068, ed says he filed it by proxy 15:23:30 gzhao: do you know? 15:24:06 and away we go... skating away on the download of a new day (sorry folks) 15:24:23 tbachman: auto-release's last push was 9/25 midnight 15:24:39 those might from integration build 15:24:49 gzhao: that’s correct 15:24:57 * regXboi is amazed the reference bounced off the wall 15:25:03 lol 15:25:12 * tbachman actually missed it as well 15:25:51 (chuckles to self) 15:26:20 * regXboi extracts distribution 15:26:34 ok, two things: (1) is 2078 blocking? do we know? 15:26:41 colindixon: 2068 should be a easy verify, right? 15:26:45 (2) do we know who can verify 2068? 15:26:48 * regXboi mumbles we're extracting, we're extracting, we're extracting 15:27:00 * tbachman searches for hidden meaning in regXboi’s mumblings 15:27:29 tbachman: yes - that is a reference - but I'll be *VERY* impressed if anybody gets it 15:27:33 gzhao: I don’t know, the bug body doesn’t give instructions for how to reproduce 15:28:00 colindixon: true 15:28:05 ok 15:28:12 folks, that RC2 may be DOA 15:28:18 ? 15:28:32 can't extract it on windows 15:28:36 :( 15:28:46 we've got names that are too long 15:28:52 ouch 15:29:01 that’s going to be consistent on all of the releases then 15:29:07 trying again to see if I did something stupid 15:29:08 * tbachman is glad regXboi decided to test this 15:29:13 ouch 15:29:18 that’s a much bigger change 15:29:34 yep 15:29:44 regXboi: if you move it to the root (C:\) and extract from there, does that fix it? 15:29:47 oddly, I didn't have this problem with previous RCs 15:30:01 so I'm thinking I did something .... different .... 15:30:01 regXboi: the final release won't have timestamp 15:30:15 regXboi: that has the mmddyyyy in it 15:30:23 give it time folks 15:30:27 lol 15:30:32 if not, there’s also this: http://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.2.0-SNAPSHOT/ 15:30:37 it's a VM - it's not the quickest bulb in the sconce 15:30:51 * regXboi now mixes my references horribly 15:30:52 It’s still a snapshot, but I thinik it would have the same updates 15:30:56 regXboi: lol 15:31:16 ok, trying colin's idea 15:31:44 regXboi: previous one probably doesn't have timestamp 15:31:59 gzhao: that may indeed be the problem 15:32:11 regXboi: in which case the final release will be fine 15:32:15 but we’ve had this problem before 15:32:19 and suggested the fix I did 15:32:22 at least Tony did 15:32:30 * regXboi could pull latest tip and *build* - that would take about 40 minutes though 15:32:32 um 15:32:35 time check? 15:32:40 lol 15:32:41 SFC 15:32:49 Should we endmeeting then? 15:33:02 yeah 15:33:11 and then pick up again after SFC? 15:33:16 let me info in regXboi’s finding 15:33:18 or we can just topic interruption 15:33:23 and then come back 15:33:28 ok, I need to run for a bit anyway 15:33:33 #info regXboi discovered that file names in mmddyyyy release aren’t happy in windows 15:33:38 ok... Colin's idea worked 15:33:42 colindixon: nice! 15:33:47 we can extract with mmddyyyy into / 15:33:47 * colindixon does a dance 15:33:50 (the fix, not the running away) 15:34:23 #info regXboi notes that this can be addressed by extracting into the top directory (e.g C:/) 15:34:26 endmeeting folks? 15:34:36 bingo 15:34:42 1860 is verified 15:34:47 regXboi: awesome! 15:35:06 #info regXboi verifies that the fix for BUG 1860 has been done 15:35:10 ugly 15:35:14 moving defect to verified now 15:35:17 let’s try something cleaner 15:35:19 #undo 15:35:19 Removing item from minutes: 15:35:27 and so done 15:35:28 #info regXboi verfied BUG 1860 is fixed 15:35:34 regXboi: thanks! 15:35:37 no worries 15:35:40 now for SFC 15:35:44 okay — gzhao regXboi colindixon are we good to endmeeting? 15:35:54 I am 15:35:56 15:36:02 15:36:09 #endmeeting