22:01:33 <jamoluhrsen> #startmeeting integration 22:01:33 <odl_meetbot> Meeting started Thu Dec 6 22:01:33 2018 UTC. The chair is jamoluhrsen. Information about MeetBot at http://ci.openstack.org/meetbot.html. 22:01:33 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:33 <odl_meetbot> The meeting name has been set to 'integration' 22:04:02 <dfarrell07> #info pybot CLI has been deprecated, moved to robot CLI 22:07:38 <jamoluhrsen> #info abelur notes that most infra issues seem to be resolved except the IO issue and that might mostly be related to our log backup job especially since our logs are mostly a bunch of small files 22:11:06 <jamoluhrsen> #info dfarrell07 notes that we may get budget for interns that we can use to help trim the jobs down to help save money 22:13:02 <jamoluhrsen> #info LuisGomez wondering if we have any jobs that build docker images for ODL. we do have the jobs but they were not maintained for a year or so. They probably need some love to get back working 22:25:33 <vorburger> jamoluhrsen: LuisGomez: hey I'm about to be very interested in that (jobs that build docker images for ODL) as well... keep in touch? actually, watch this issue: 22:26:04 <vorburger> jamoluhrsen: LuisGomez: dfarrell07: https://jira.opendaylight.org/browse/COE-30 22:27:07 <dfarrell07> vorburger: I've had logic for years and years to build containers/VMs using Packer, using Ansible-ODL under the hood 22:27:23 <dfarrell07> vorburger: but no one was using them so I haven't really been maintaining recently 22:28:05 <dfarrell07> vorburger: I also tried to make a job to do this on Jenkins and push to Dockerhub, but ran into blocking infra limits 22:28:15 <dfarrell07> vorburger: I have an issue, let me find it 22:28:19 <vorburger> dfarrell07: well looks like that's about to come in handy again, for (apparently) both LuisGomez and me. In fact, can I internally ping you for COE-30 some time? Possibly more of a Jan than Dec issue. 22:28:50 <dfarrell07> vorburger: yeah, perfect, I'd love a good reason to be able to give cycles to that logic 22:28:59 <vorburger> dfarrell07: FYI in last week's TSC, zxiiro said that whenever we (CoE folks) were ready and really wanted this, to open a helpdesk ticket and he/someone would assist from the infra side 22:29:36 <vorburger> dfarrell07: sounds like a plan. It's not something for tomorrow morning as I haven't even got to being able to build that image locally, and I first need to progress on where I actually use that image - but I'll get back to you! 22:31:18 <dfarrell07> vorburger: cool, just let me know when you're ready :) 22:31:21 <jamoluhrsen> #action LuisGomez to email LF folks to reduce our number of parallel jobs to see if we can get more reliable jobs 22:31:23 <dfarrell07> vorburger: I linked this issue to yours https://jira.opendaylight.org/browse/INTPAK-12 22:33:09 <abelur> #info https://gerrit.linuxfoundation.org/infra/#/c/13828/ 22:34:44 <jamoluhrsen> LuisGomez: https://git.opendaylight.org/gerrit/#/c/78490/ 22:37:10 <jamoluhrsen> #action jamoluhrsen to test zxiiro's draft patches for global-jjb macros in the sandbox 22:52:19 <LuisGomez> vorburger, count on me as well 22:53:02 <LuisGomez> i was not aware of COE-30 but I can definitely spend some cycles there too 00:29:38 <zxiiro> jamoluhrsen: dfarrell07 looks like the bad URLs from autorelease is fixed https://jenkins.opendaylight.org/releng/job/integration-distribution-test-neon/118/parameters/ 00:29:54 <zxiiro> let me know if you still see anything broken 00:29:56 <jamoluhrsen> zxiiro: yeah. looks fine now. I was checking it out earlier 00:29:59 <jamoluhrsen> thanks for that 17:05:42 <jamoluhrsen> LuisGomez: did you forget to mark this OKAY? https://docs.google.com/spreadsheets/d/1dYOY99twqHV_Q0YorAOOxmL0aFc3icNXg8qA_zGwKyA/edit#gid=1286339548 17:56:09 <jamoluhrsen> zxiiro: abelur: LuisGomez: is every job failing right now to upload logs? 17:57:02 <LuisGomez> just updated jamoluhrsen 18:54:59 <jamoluhrsen> LuisGomez: zxiiro: abelur: dfarrell07: CSIT is broken. patch submitted and email sent. 18:59:39 <LuisGomez> ok jamoluhrsen 22:08:26 <abelur> LuisGomez: jamoluhrsen do you know why we dont have these files anymore under Oxygen? 22:09:06 <LuisGomez> which files? 22:09:09 <abelur> integration/distribution/opendaylight/pom.xml? 22:09:42 <LuisGomez> in oxygen there was no "common" distribution 22:09:53 <LuisGomez> that started in fluorine 22:11:20 <abelur> LuisGomez: so we dont have the integration/dist project in AR for oxygen? 22:11:52 <LuisGomez> only the opendaylight/pom.xml is missing 22:12:09 <LuisGomez> the rest should be there 22:13:46 <LuisGomez> just to be clear, the opendaylight/pom.xml builds the common distribution = managed distribution + SM projects 22:14:30 <abelur> LuisGomez: thanks for confirming 22:14:31 <LuisGomez> in oxygen and before there is no such common distribution 22:20:42 <abelur> LuisGomez: so this change should be only applicable for non-Oxygen AR builds? 22:20:43 <abelur> https://git.opendaylight.org/gerrit/#/c/78224/ 22:22:00 <abelur> LuisGomez: so from fluorine onwards?? 22:28:48 <LuisGomez> abelur, yes from florine, you can modify above patch command to just ignore failure 22:29:01 <LuisGomez> in case the folder is not there 22:29:12 <abelur> LuisGomez: patch incoming 22:35:25 <abelur> LuisGomez: https://git.opendaylight.org/gerrit/78661 22:56:25 <abelur> LuisGomez: done, its updated now 23:02:48 <abelur> jamoluhrsen: dfarrell07: anyone able to merge that please ? 23:03:14 <abelur> we are blocked on the Oxygen SR2 release and version bumping with that merged? 23:06:38 <dfarrell07> abelur: ack 23:07:54 <dfarrell07> abelur: okay, you have a +2 for when it passes verify 23:08:02 <abelur> dfarrell07: thanks :) 02:13:45 <zxiiro> jamoluhrsen: LuisGomez you guys may see a 403 - Forbidden in some jobs failing to upload logs at the end of the job. I screweed up a permission. sorry about that. It should be fixed now. 02:14:13 <zxiiro> jamoluhrsen: LuisGomez it was broken for about 10 minutes before I caught it so hopefully didn't affect too many jobs 03:51:52 <abelur> does anyone know why the version-bump jobs are failing on dluxapps? 03:51:59 <abelur> 00:36:08 [INFO] ODL :: dluxapps :: yangutils-module 0.7.5-SNAPSHOT . SUCCESS [ 0.635 s] 03:51:59 <abelur> 00:36:08 [INFO] ODL :: dluxapps :: yangutils-bundle 0.7.5-SNAPSHOT . FAILURE [ 2.126 s] 03:51:59 <abelur> 00:36:08 [INFO] ODL :: dluxapps :: odl-dluxapps-yangman 0.7.5-SNAPSHOT SKIPPED 03:53:08 <abelur> zxiiro: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-version-bump-oxygen/12/console.log.gz 03:59:49 <abelur> jamoluhrsen: dfarrell07 ^^ 04:01:03 <abelur> Is this a temporary failure? 06:30:10 <zxiiro> abelur: Return code is: 401, ReasonPhrase: Unauthorized. 06:30:24 <zxiiro> opendaylight.snapshot/org/opendaylight/dluxapps/yangutils-bundle/0.7.5-SNAPSHOT/yangutils-bundle-0.7.5-20181212.003608-1.jar 06:31:48 <zxiiro> I don't know why it's unauthorized for that bundle 06:32:03 <abelur> 401 is very strange? 06:48:37 <abelur> zxiiro: I think thoese failure are likely because of this change: https://git.opendaylight.org/gerrit/#/c/78192/ 06:51:47 <abelur> zxiiro: are we setting these vars inside of $GLOBAL_SETTINGS_FILE or SETTINGS_FILE 06:52:52 <zxiiro> abelur: i'm not sure how it's related to a macro that just installs Maven? 06:53:49 <zxiiro> abelur: the lf-provide-maven-settings after it is what passes down the settings files to the job 06:54:12 <zxiiro> abelur: that patch just touches the call that installs mvn, the fact that maven successfully runs means the binary was installed and works. 06:55:58 <abelur> hmmm 06:56:39 <abelur> ok, 401 unauthoried clearly means something to do with invalid creds, unless something changed on Nexus repo. I think these projects are all quite old and we've not touched them since. ... the only place we pass the creds to this version bump job is the config-file-provider 06:56:57 <abelur> diff --git a/jjb/autorelease/autorelease-macros.yaml b/jjb/autorelease/autorelease-macros.yaml 06:56:57 <abelur> index b9edf7fe..a119315b 100644 06:56:57 <abelur> --- a/jjb/autorelease/autorelease-macros.yaml 06:56:57 <abelur> +++ b/jjb/autorelease/autorelease-macros.yaml 06:56:57 <abelur> @@ -39,16 +39,6 @@ 06:56:57 <abelur> git clean -dff 06:56:57 <abelur> fi 06:56:58 <abelur> 06:56:58 <abelur> -- builder: 06:56:59 <abelur> - name: autorelease-cfp 06:56:59 <abelur> - builders: 06:57:00 <abelur> - - config-file-provider: 06:57:42 <abelur> s/invalid/missing/ ^^^ PS 06:57:51 <zxiiro> abelur: so why did many modules complete building before until it got to that one? 06:57:59 <zxiiro> abelur: if it's credentials related then wouldn't it fial on the first module it built? 07:00:11 <abelur> zxiiro: yeah, that's true aswell :~( 07:01:49 <zxiiro> abelur: have you checked nexus logs to see if it has more details? 07:01:52 <abelur> zxiiro: the only other thing I can think off is some issue with ldap 07:02:04 <abelur> i'll also check with the IT team 07:03:41 * abelur checks hte logs first 07:16:48 <abelur> zxiiro: sure if we can see anything from the logs? 07:17:08 <abelur> wondering if this was restarted earlier around the time when the job were going through 07:18:03 <zxiiro> abelur: I don't know. It was a suggestion. 07:19:36 <zxiiro> abelur: nexus was restarted this morning yes but it happened before this job ran 07:20:02 <zxiiro> build #13 https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-version-bump-oxygen/13/ 07:20:09 <zxiiro> which failed in the same way 07:20:32 <abelur> zxiiro: that was the second time I ran the job 07:20:39 <abelur> this was the first time: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-version-bump-oxygen/12/console.log.gz 07:21:11 <zxiiro> abelur: considering they both failed with the exact same error I would say the restart was not a factor in this 07:21:22 <abelur> but we hitting the same error both the times ... is not related to the restart 07:21:24 <abelur> yeah 07:23:57 <abelur> zxiiro: looks like the artifacts should have the prefix -SNAPSHOT, which is missing 07:24:06 <abelur> 00:36:08 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project yangutils-bundle: ArtifactDeployerException: Failed to deploy artifacts: Could not transfer artifact org.opendaylight.dluxapps:yangutils-bundle:jar:0.7.5-20181212.003608-1 from/to opendaylight-snapshot::default (https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot): Failed to 07:24:06 <abelur> transfer file: https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/dluxapps/yangutils-bundle/0.7.5-SNAPSHOT/yangutils-bundle-0.7.5-20181212.003608-1.jar. Return code is: 401, ReasonPhrase: Unauthorized. -> [Help 1] 07:24:49 <abelur> so that probably the reason the its returning a 401 unauthorized ... since its a snapshot repo 07:49:21 <zxiiro> abelur: looks like both sigmatopology-bundle and yangutils-bundle both fail to deploy. I can reproduce it 07:49:23 <zxiiro> not sure why though 07:49:56 <zxiiro> the sigmatopology-module and yangutil-module in the same directory deploys successfully no problem though 07:50:11 <zxiiro> very strange... 08:07:39 <zxiiro> abelur: I'd do a manual version bump at this point. dluxapps isn't even a maintained project anymore. 08:40:44 <abelur> zxiiro: ack 16:31:47 <jamoluhrsen> abelur, zxiiro, gerrit question if either of you is around? 16:48:28 <zxiiro> jamoluhrsen: what's the question? 16:48:43 <jamoluhrsen> zxiiro: I am on a review that's already been merged. 16:48:54 <jamoluhrsen> zxiiro: I can't figure out how to remove myself now? not possible 16:48:56 <jamoluhrsen> ? 16:50:17 <zxiiro> jamoluhrsen: yeah seems like it. 16:50:32 <jamoluhrsen> zxiiro: ack. no biggie just thought I was missing something. 16:50:58 <zxiiro> Might want to raise that to the Gerrit folks. Maybe it's intentional. 21:28:44 <abelur> zxiiro: jamoluhrsen trying to merge all these version bump changes ... 21:29:34 <abelur> zxiiro: jamoluhrsen: we have to do this in order ... however the verify is not going through at all 21:29:39 <abelur> for controller 21:29:40 <abelur> https://git.opendaylight.org/gerrit/#/c/78686/ 21:30:21 <abelur> *maven-verify* and *validate-autorelease* is not going through? 21:36:53 <jamoluhrsen> abelur: not sure on this, tbh. is it because of snapshot versions for mdsal? don't we just want the release version? 21:37:48 <abelur> nah, the release versions are already promoted :-) 21:38:11 <abelur> this is version bump assuming there is going to be a next release 21:38:43 <jamoluhrsen> abelur: well, you know will know better, but I thought we only consumed released versions of release integrated projects like mdsal. 21:38:46 <abelur> jamoluhrsen: generally we use the version bump job to deploy the snapshots into the Nexus 21:38:49 <jamoluhrsen> anyway, I saw this line in the consolelog: 21:38:50 <jamoluhrsen> 05:39:27 [ERROR] Non-resolvable import POM: Could not find artifact org.opendaylight.mdsal:mdsal-artifacts:pom:2.4.5-SNAPSHOT in opendaylight-snapshot (https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/) @ org.opendaylight.mdsal:binding-parent:0.12.5-SNAPSHOT, /tmp/r/org/opendaylight/mdsal/binding-parent/0.12.5-SNAPSHOT/binding-parent-0.12.5-SNAPSHOT.pom, line 53, column 25 21:41:26 <abelur> hmmm ... that sounds right, let me check with Thanh or Luis once 21:41:54 <abelur> zxiiro: LuisGomez: dfarrell07 anyone has any ideas please ? 21:42:49 * dfarrell07 is trying to figure out the question 21:45:18 <dfarrell07> I think we do always integrate with release versions of MRI projects, if that's the question 21:45:39 <dfarrell07> like MSI project builds should be referencing MRI release artifacts 21:53:45 <zxiiro> abelur: it's not gonna pass until mdsal is merged first. 21:54:04 <zxiiro> dfarrell07: mdsal is not mri in oxygen 21:54:52 <abelur> zxiiro: that's what I was missing ... dint even look for mdsal as its to be top in chart/order 21:55:51 <abelur> thanks :-) 21:56:12 <zxiiro> Alright It's 5 am and the sun is about to rise. Time for bed. 21:56:34 <zxiiro> 6 am rather 22:11:03 <dfarrell07> zxiiro: yikes man, yeah sleepy time 22:13:23 <zxiiro> dfarrell07: the downside of working on the otherside of the world but still have meetings to attend in the US 09:50:58 <abelur> dfarrell07: jamoluhrsen: can I get a merge on these changes please 09:51:04 <abelur> https://git.opendaylight.org/gerrit/78732 09:51:05 <abelur> https://git.opendaylight.org/gerrit/78731 09:51:05 <abelur> https://git.opendaylight.org/gerrit/#/c/78679/ 10:11:28 <zxiiro> abelur: merged 10:12:00 <abelur> Zxirro thanks 10:12:11 <zxiiro> abelur: can you cherry-pick to stable/oxygen too the downloads.rst changes? 10:12:24 <abelur> Ok 10:12:37 <abelur> From the master? 10:12:48 <zxiiro> doesn't matter which one you cherry-pick from 10:13:02 <abelur> I think that file is not in stable/oxygen 10:13:12 <abelur> And was brought in later? 10:13:13 <zxiiro> abelur: ok nevermind then 10:13:26 <zxiiro> I don't remember 14:48:19 <zxiiro> jamoluhrsen: dfarrell07 FYI I think I'm going to pass on todays integration meeting. I have too many meetings tonight and would like to get at least some sleep in between. 14:48:45 <dfarrell07> zxiiro: yeah, please do, sleep > meeting 14:48:46 <zxiiro> jamoluhrsen: dfarrell07 updates from me are 1) I think we hopefully shouldn't see gateway timeouts anymore (RELENG-1215) but please keep reporting them if anyone's still seeing them 14:49:35 <zxiiro> jamoluhrsen: dfarrell07 2) Please continue to test https://git.opendaylight.org/gerrit/#/q/topic:global-jjb-stack and give me any feedback. It does require the latest JJB 2.9.2 to work. 14:49:56 <zxiiro> We'll look into rolling it out in the new year 14:52:34 <zxiiro> In other news I'll be back in Ottawa next week (Wednesday) so expect me to be back in usual timezone soon :) 14:55:18 <dfarrell07> zxiiro: coool, thanks for all the good work and have fun :) 22:01:58 <odl_meetbot> jamoluhrsen: Error: Can't start another meeting, one is in progress. Use #endmeeting first. 22:02:05 <jamoluhrsen> #endmeeting